An AI model data storage management method based on a meta number mark and a storage medium

Abstract

The application provides an AI model data storage management method based on metadata and a storage medium, the method comprising: data screening from the initial metadata, when a new training task needs to be started, checking out a new feature branch from the current main branch to obtain metadata in a feature 1 state, if the metadata in the feature 1 state is suitable for all AI model training task requirements, the metadata can be used for the training task; if the metadata in the feature 1 state is not suitable for all AI model training task requirements, data screening is performed on the metadata in the feature 1 state to obtain metadata in a feature 2 state; and the process is continued until it is confirmed that the metadata is suitable for all AI model training task requirements. The data storage management method is data-centered, can provide efficient AI model development iteration capability, integrates data and model version control, and can realize parallel and rapid iteration of multiple task-specific data sets.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of metaverse, in particular to an AI model data storage management method based on meta number label and a storage medium. BACKGROUND

[0002] At present, the commercialization of artificial intelligence has basically reached a stage of maturity in computing power, algorithms and technology. To truly solve the specific difficulties and pain points of the industry through algorithms and application landing, a large amount of original data related to artificial intelligence needs to be collected, and the data sets collected by manual collection, camera collection and various sensors are labeled through the MetaDataLabeler platform, and after labeling and processing, algorithm training support can be performed. It can be said that data determines the landing degree of AI. Algorithm engineers use labeled data to train models in the platform to produce AI models for the current data set.

[0003] With the development of the AI industry, the demand for AI data in landing scenarios is more diversified and customized, and higher requirements for the professionalism and quality of AI data services are put forward. For example, the fields of intelligent driving and intelligent terminals in the AI industry are constantly developing, and application landing is accelerating. Taking computer vision as an example, the development of a new scenario requires tens of thousands or even hundreds of thousands of pictures collected and labeled. With the enrichment of AI application scenarios, there will be a long-term and massive demand for AI data services.

[0004] However, the current AI application often encounters bottlenecks in large-scale expansion during product landing, such as high development cost and long iteration cycle. SUMMARY

[0005] Therefore, the purpose of the present application is to provide efficient model development and iteration capabilities based on data, integrate active learning methods, data and model version control, and integrate concepts such as workspace to achieve parallel and rapid iteration of multiple task-specific data sets.

[0006] MetaDataLabeler is a data-driven algorithm training platform that can achieve data management, data labeling, data mining, model training, model verification and other functions in a code-free development manner.

[0007] The present application integrates data import, data cleaning, data labeling, labeling review, label management, initial model import, automatic training, model iteration, model export and other functions on the MetaDataLabeler platform based on traditional data collection and labeling solutions, not only solving the problem of data management and labeling, but also solving the repetitive problems that algorithm engineers spend a long time to solve, such as model training and model iteration.

[0008] The application provides an AI model data storage management method based on metadata, which filters data from the initial metadata, selects and uses data meeting the requirements of AI model tasks, and includes the following steps:

[0009] 11. When a new AI model training task needs to be started, a new feature branch is checked out from the current master branch to obtain metadata in a feature 1 state, and if the metadata in the feature 1 state meets the requirements of all AI model training tasks, the metadata can be used for AI model training tasks; if the metadata in the feature 1 state does not meet the requirements of all AI model training tasks, step 12 is entered.

[0010] 12. Data is filtered from the metadata in the feature 1 state to obtain metadata in a feature 2 state, and this process is repeated until the metadata meets the requirements of all AI model training tasks.

[0011] At this time, other users make changes to the metadata of the master branch, which does not affect the training data being used by the user.

[0012] The application provides another AI model data storage management method based on metadata, which searches for a metadata by a tag or keywords, starts the filtering process, and obtains data meeting the requirements of AI model tasks, and then uses the data meeting the requirements of AI model tasks, including the following steps:

[0013] 22. When a new AI model training task needs to be started, a metadata meeting the requirements is searched for by keywords and / or tags.

[0014] 23. A new branch is checked out based on the metadata meeting the requirements.

[0015] 24. Data filtering or cleaning is continued on the new branch until data meeting the requirements of AI model training tasks is obtained.

[0016] Further, if an AI model training task has been completed using a metadata, the metadata of the resource library and the master branch is updated, and the updated part is merged into the currently used metadata, including the following steps:

[0017] 31. Suppose the user is currently in state 2 (feature#2), switch back to the master branch;

[0018] 32. For the incremental part from master#2 to master#1, repeat the previously performed task to obtain feature 2 + feature#2 + state;

[0019] 33. Switch back to feature #2 state, merge feature #2 state and feature #2 state to get feature #3 state.

[0020] Furthermore, the backend service of the MetaDataLabeler employs the following methods:

[0021] The data storage service uses MySQL 8.0, a relational database that emphasizes strong relationships between data and supports strong consistency in data creation, deletion, modification, and querying.

[0022] Furthermore, the method used by the backend service of the MetaDataLabeler also includes:

[0023] Using Redis, a non-relational database, as a data caching service is advantageous because non-relational databases offer better support for large-scale concurrent read and write operations and offer higher performance.

[0024] Furthermore, the method used in the backend service of the MetaDataLabeler also includes:

[0025] ClickHouse, a columnar storage database, is used as the data analysis result storage service. Columnar databases are databases that store data using a column-related storage architecture, which is mainly suitable for batch data processing and real-time querying. Because each column of a columnar database is stored separately, it is easy to apply different compression algorithms to the characteristics of each column. Columnar storage databases do not decompress data during the operation process until the original data value is restored at the end. This reduces CPU consumption, memory consumption, network transmission consumption, and the final storage space requirement.

[0026] Furthermore, the method of using Redis as a data caching service includes:

[0027] The data to be stored is stored in key-value format, and the data is kept in memory;

[0028] Key-value pairs improve data access performance by using an index structure, transforming random write operations into sequential writes, thus significantly enhancing write performance.

[0029] Further, the ClickHouse has a writing speed of 50-200M / S as a data analysis result storage service.

[0030] Clickhouse is a database with very fast query, but attention should be paid to low-frequency and low-volume writing. In actual application, data needs to be accumulated first and then written when a certain amount is reached.

[0031] The application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the AI model data storage management method based on the MetaDataLabeler platform.

[0032] The application also provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the AI model data storage management method based on the MetaDataLabeler platform.

[0033] In order to reduce the cost of enterprises using AI technology and accelerate the promotion of AI-enabled enterprises, the MetaDataLabeler platform provides end-to-end algorithm development tools for algorithm personnel, provides one-stop service around the data processing and model training required in the AI development process, and promotes algorithm technology application. The MetaDataLabeler platform designs open API to integrate third-party tools, supports enterprises to invest limited manpower into application development, and realizes AI landing in industry at lower cost.

[0034] Compared with the prior art, the application has the beneficial effects that:

[0035] Based on the MetaDataLabeler platform, the application can provide efficient AI model development iteration capability, integrate data and model version control, and realize parallel rapid iteration of multiple task-specific data sets. BRIEF DESCRIPTION OF DRAWINGS

[0036] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included solely for their for illustrative purposes, and are not deemed to be a limitation of the application.

[0037] In the drawings:

[0038] Figure 1 A flowchart of the AI model data storage management method based on the MetaDataLabeler platform of the application;

[0039] Figure 2 A schematic diagram of the computer device of the embodiment of the application

[0040] Figure 3 Flow chart of the AI model data storage management method based on the meta number mark of another embodiment of the present application;

[0041] Figure 4 Flow chart of the AI model data storage management method based on the meta number mark of another embodiment of the present application;

[0042] Figure 5 Flow chart of the AI model data storage management method based on the meta number mark of another embodiment of the present application;

[0043] Figure 6 Flow chart of the AI model data storage management method based on the meta number mark of another embodiment of the present application;

[0044] Figure 7 Flow chart of the AI model data storage management method based on the meta number mark of another embodiment of the present application. DETAILED DESCRIPTION

[0045] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements throughout the description. The following exemplary embodiments are described in the context of the implementation of the present application. However, the present application is not limited to the embodiments described herein, but can be applied to other embodiments as well.

[0046] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0047] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used only to distinguish one from another. For example, a first information can be termed a second information, and, similarly, a second information can also be termed a first information, without departing from the scope of the present disclosure. As used herein, the word "if' can be interpreted to mean "when" or "upon" or "in response to determining" taking into account the context in which the term is used.

[0048] The embodiments of the present application will be further described in detail with reference to the accompanying drawings.

[0049] The embodiment of the application provides an AI model data storage management method based on a metadata tag, as shown in Figure 1 The data filtering from the initial metadata is performed, and data meeting the AI model task requirements is selected and used, including the following steps:

[0050] 11. When a new AI model training task needs to be started, a new feature branch is checked out from the current master branch, and metadata in a feature 1 state is obtained, if the metadata in the feature 1 state is suitable for all AI model training task requirements, the metadata can be used for the AI model training task, and if the metadata in the feature 1 state is not suitable for all AI model training task requirements, step 12 is entered.

[0051] 12. Data filtering is performed from the metadata in the feature 1 state, metadata in a feature 2 state is obtained, and the process is continued until it is confirmed that the metadata is suitable for all AI model training task requirements.

[0052] At this time, other users make changes to the metadata of the master branch, and the training data used by the user is not affected.

[0053] Figure 5 The AI model data storage management method of the embodiment of the application directly filters from the initial metadata, and a block diagram is as shown in the figure.

[0054] The embodiment of the application provides another AI model data storage management method based on a metadata tag, as shown in Figure 3 A metadata is searched through a tag or a keyword, a filtering process is started, data meeting the AI model task requirements is obtained, and the data is used, including the following steps:

[0055] 22. When a new AI model training task needs to be started, a metadata basically meeting the requirements is searched through a keyword and / or a tag.

[0056] 23. A new branch is checked out on the basis of the metadata basically meeting the requirements.

[0057] 24. Data filtering or cleaning is continued on the new branch until data meeting the AI model training task requirements is obtained.

[0058] Figure 6 The block diagram of the AI model data storage management method for searching a metadata for screening of an embodiment of the present application is shown in the figure.

[0059] Referring to Figure 4 As shown in the figure, the training task of the AI model has been completed using a metadata, at this time the metadata of the resource library and the master branch is updated, and the updated part is merged into the currently used metadata, including the following steps:

[0060] 31. Assuming that the user is now in the feature 2 feature#2 state, the master branch is switched back;

[0061] 32. For the incremental part of master#2-master#1, the task done before is repeated to obtain the feature 2+feature#2+state.

[0062] 33. Switching back to the feature 2 feature#2 state, merging the feature 2+feature#2+state to obtain the feature 3 feature#3 state.

[0063] The method used by the backend service of the MetaDataLabeler includes:

[0064] The relational database MySQL 8.0 is used as the data storage service, the relationship emphasizes the strong association between data, and supports the strong consistency of data addition, deletion, modification and query.

[0065] Figure 7 The block diagram of the AI model data storage management method for merging the updated part of the metadata of the master branch into the currently used metadata of an embodiment of the present application is shown in the figure.

[0066] The method used by the backend service of the MetaDataLabeler further includes:

[0067] The non-relational database Redis is used as the data cache service, and the non-relational database supports better for a large number of concurrent read and write, and has higher performance;

[0068] The method of using Redis as a data cache service includes:

[0069] The data to be stored is stored in the form of Key-Value, and the data is saved in the memory;

[0070] Key-Value improves the access performance of data by using an index structure, and converts random write operations into sequential write operations, greatly improving the write performance.

[0071] The method adopted by the backend service of the MetaDataLabeler also includes:

[0072] The columnar storage database ClickHouse is used as a data analysis result storage service, the columnar database is a database for storing data in a column-related storage architecture, mainly suitable for batch data processing and instant query, and the columnar database is easy to use different compression algorithms according to the characteristics of each column, and the columnar storage database does not decompress data during operation until the last data is restored to the original data value, thereby reducing CPU consumption, reducing memory consumption, reducing network transmission consumption, and reducing the need for final storage space.

[0073] The writing speed of the ClickHouse as the data analysis result storage service is 50-200M / S.

[0074] The clickhouse is a database with very fast query, but attention should be paid to low-frequency and low-volume writing, and in actual application, data needs to be accumulated first and then written when a certain amount is reached.

[0075] The application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the AI model data storage management method based on the MetaDataLabeler.

[0076] The application further provides a computer device, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor realizes the AI model data storage management method based on the MetaDataLabeler when executing the program.

[0077] In order to reduce the cost of enterprises using AI technology and accelerate the promotion of AI-enabled enterprises, the embodiment of the application provides an end-to-end algorithm development tool for algorithm personnel based on the MetaDataLabeler platform, provides one-stop service around the data processing and model training required in the AI development process, and promotes the application of algorithm technology.

[0078] The embodiment of the application is based on the MetaDataLabeler platform and takes data as the center, can provide efficient AI model development iteration capability, integrates data and model version control, and can realize parallel and rapid iteration of multiple task-specific data sets.

[0079] The embodiment of the application further provides a computer device, Figure 2is a structural schematic diagram of a computer device provided by an embodiment of the present application; see the accompanying drawings Figure 2 As shown in the accompanying drawings, the computer device comprises an input device 23, an output device 24, a memory 22 and a processor 21; the memory 22 is used for storing one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the AI model data storage management method based on the meta number mark provided by the above-mentioned embodiment; wherein the input device 23, the output device 24, the memory 22 and the processor 21 can be connected by a bus or other means, Figure 2 For example, the connection by the bus is taken as an example in the description.

[0080] The memory 22 is a readable and writable storage medium of a computer device, which can be used for storing software programs, computer executable programs, such as program instructions corresponding to the AI model data storage management method based on the meta number mark described in the embodiment of the present application; the memory 22 can mainly include a storage program area and a storage data area, wherein the storage program area can store an operating system and at least one application program required by a function; the storage data area can store data created according to the use of the device and the like; in addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device or other non-volatile solid-state storage device; in some examples, the memory 22 can further include a memory remotely arranged with respect to the processor 21, and these remote memories can be connected to the device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0081] The input device 23 can be used for receiving input digital or character information, and generating key signal input related to user settings and function control of the device; the output device 24 can include a display device such as a display screen.

[0082] The processor 21 executes various function applications and data processing of the device by running the software programs, instructions and modules stored in the memory 22, that is, implements the AI model data storage management method based on the meta number mark described above.

[0083] The computer device provided above can be used to execute the AI model data storage management method based on the meta number mark provided by the above-mentioned embodiment, and has corresponding functions and beneficial effects.

[0084] The embodiments of the present application also provide a storage medium containing computer executable instructions for performing the AI model data storage management method based on the meta-number mark provided by the above embodiments when executed by a computer processor. The storage medium is any various type of memory device or storage device, and includes: installation media such as CD-ROM, floppy disks or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements; and the like. The storage medium can also include other types of storage or combinations of them. In addition, the storage medium can be located in the first computer system where the program is executed, or in a second computer system different from the first computer system, and the second computer system is connected to the first computer system through a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (e.g., in different computer systems connected through a network). The storage medium can store program instructions (e.g., computer programs) executable by one or more processors.

[0085] Of course, the storage medium containing computer executable instructions provided by the embodiments of the present application is not limited to the AI model data storage management method based on the meta-number mark described in the above embodiments, and can also perform the related operations in the AI model data storage management method based on the meta-number mark provided by any embodiment of the present application.

[0086] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical solutions after the changes or replacements will fall within the protection scope of the present application.

[0087] The above description is only the preferred embodiments of the present application and is not intended to limit the present application; for those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for AI model data storage management based on meta-number tags, characterized in that, From the beginning of the metadata data screening, select and use the data that meet the requirements of AI model task, including the following steps: (11) When a new AI model training task needs to be started, a new feature branch is checked out from the current master branch, and metadata in feature 1 state is obtained. If the metadata in feature 1 state is suitable for all AI model training task requirements, the metadata can be used for AI model training task; if the metadata in feature 1 state is not suitable for all AI model training task requirements, go to step 12; (12) Data screening is performed from the metadata in feature 1 state to obtain metadata in feature 2 state; and so on until it is confirmed that the metadata is suitable for all AI model training task requirements; Search for a metadata metadata by tag or keyword keywords, start the screening process, and then use the data that meet the requirements of AI model task, including the following steps: (21) When a new AI model training task needs to be started, search for a metadata that basically meets the requirements by keywords and / or tags; (22) On the basis of the metadata that basically meets the requirements, a new branch is checked out; (23) Continue data screening or cleaning on the new branch until the data that truly meet the requirements of AI model training task are obtained; If an AI model training task has been completed using a metadata metadata, the metadata metadata of the resource library and the master branch is updated, and the updated part is merged into the currently used metadata metadata, including the following steps: (31) Set the user to feature 2 state and return to master branch; (32) Repeat the AI model training task for the increment part of branch master#2-master#1 to obtain feature 2+ state; (33) Return to feature 2 state, merge feature 2+ state to obtain feature 3 state; The method used by the backend service of the metadata label MetaDataLabeler further includes: Using columnar storage database ClickHouse as data analysis result storage service. 2.The AI model data storage management method based on meta-number tags according to claim 1, characterized in that, The method used by the backend service of the metadata label MetaDataLabeler includes: Using relational database MySQL 8.0 as data storage service. 3.The AI model data storage management method based on meta-number tags according to claim 1, characterized in that, The method used by the backend service of the metadata label MetaDataLabeler further includes: Using non-relational database Redis as data cache service.

4. The AI model data storage management method based on meta-number tags according to claim 3, characterized in that, The method of using Redis as data cache service includes: The data to be stored is stored in the form of Key-Value, and the data is saved in the memory.

5. The meta number flag-based AI model data storage management method of claim 1, wherein, The writing speed of ClickHouse as data analysis result storage service is 50-200 M / S.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the meta-number-based AI model data storage management method of any one of claims 1-5.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the meta-number-based AI model data storage management method of any one of claims 1-5.

Images