Model data version processing method and device

Abstract

Embodiments of the present specification provide a model data version processing method and device, wherein a model data version processing method includes: creating a data directory of the current version of a target model; obtaining a first model code and first metadata of the target model from a first remote warehouse; based on a data identifier contained in the first metadata, obtaining a first data set corresponding to the data identifier from a second remote warehouse, and adding the first data set to the data directory; storing training indicator data and model files in a local cache, and generating metadata for each of the training indicator data and the model files; wherein the training indicator data and the model files are obtained by model training based on the second data set and the second model code in the data directory, storing the metadata in a local warehouse, and pushing the metadata stored in the local warehouse to the first remote warehouse.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a model data version processing method and device. Background Art

[0002] The rise of mobile Internet and the implementation of big data platforms have provided artificial intelligence with a large amount of data, which has enabled artificial intelligence to be widely used in almost every field. At the same time, challenges also arise. When developing and training machine learning models, there are many changing factors, such as data sets, model training codes, model experiment indicator evaluation data, model files, etc. Therefore, version control of data files at various stages of the machine learning model life cycle is becoming increasingly important. Summary of the Invention

[0003] In a first aspect, an embodiment of the present application provides a method for processing model data versions, the method comprising:

[0004] Create a data directory for the current version of the target model;

[0005] Acquire first model code and first metadata of the target model from a first remote repository;

[0006] Based on the data identifier included in the first metadata, obtaining a first data set corresponding to the data identifier from a second remote repository, and adding the first data set to the data directory;

[0007] Storing training indicator data and a model file in a local cache, and generating metadata for the training indicator data and the model file; wherein the training indicator data and the model file are obtained by performing model training based on a second data set and a second model code in the data directory, the second data set is obtained by annotating the first data set, and the second model code is obtained by adjusting the first model code;

[0008] The metadata is stored in a local warehouse, and the metadata stored in the local warehouse is pushed to the first remote warehouse.

[0009] In a second aspect, an embodiment of the present application provides a method for processing model data versions, the method comprising:

[0010] Creating a data directory for the current version of the target model, and copying historical data files in the historical data directory of the historical version of the target model to the data directory;

[0011] Adding a data file to the local cache and generating metadata for the data file; the data file includes training indicator data and a model file obtained by model training based on the data set and model code in the data directory, the data set and the model code being generated based on the historical data file;

[0012] Storing the metadata in a temporary storage area, and synchronizing the metadata from the temporary storage area to a local warehouse;

[0013] The metadata stored in the local warehouse is pushed to the first remote warehouse, and the data file stored in the local cache is pushed to the second remote warehouse.

[0014] In a third aspect, an embodiment of the present application provides a model data version processing device, the device comprising:

[0015] The data directory creation module is used to create the data directory of the current version of the target model;

[0016] A data acquisition module, configured to acquire a first model code and first metadata of the target model from a first remote repository;

[0017] a data set adding module, configured to obtain, based on the data identifier included in the first metadata, a first data set corresponding to the data identifier from a second remote repository, and add the first data set to the data directory;

[0018] a metadata generation module, configured to store the training indicator data and the model file in a local cache, and to generate metadata for each of the training indicator data and the model file; wherein the training indicator data and the model file are obtained by performing model training based on a second data set and a second model code in the data directory, the second data set being obtained by annotating the first data set, and the second model code being obtained by adjusting the first model code;

[0019] The metadata storage module is used to store the metadata in a local warehouse and push the metadata stored in the local warehouse to the first remote warehouse.

[0020] In a fourth aspect, an embodiment of the present application provides a model data version processing device, the device comprising:

[0021] A data directory creation module is used to create a data directory of the current version of the target model and copy historical data files under the historical data directory of the historical version of the target model to the data directory;

[0022] A metadata generation module, configured to add data files to a local cache and generate metadata for the data files; the data files include training indicator data and model files obtained by performing model training based on the data sets and model codes in the data directory, the data sets and model codes being generated based on the historical data files;

[0023] A metadata storage module, configured to store the metadata in a temporary storage area and synchronize the metadata from the temporary storage area to a local warehouse;

[0024] A data push module is used to push the metadata stored in the local warehouse to the first remote warehouse, and push the data files stored in the local cache to the second remote warehouse.

[0025] In a fifth aspect, an embodiment of the present application provides a model training device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to execute the model data version processing method described in the second aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a model training device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to execute the model data version processing method described in the second aspect.

[0027] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the model data version processing method as described in the first aspect.

[0028] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the model data version processing method as described in the second aspect.

[0029] It can be seen that in an embodiment of the present application, by creating a data directory of the current version of the target model, the first model code and the first data set of the target model obtained from the first remote warehouse and the second remote warehouse are added to the data directory of the current version, and the second model code and the second data set are obtained on the basis of the first model code and the first data set, and the training index data and the model file obtained by model training based on the second model code and the second data set are pushed to the first remote warehouse and the second remote warehouse, thereby realizing centralized management and control of model data, and at the same time realizing synchronization of the model data stored in the first remote warehouse and the second remote warehouse with the locally stored model data, and ensuring disaster recovery of model data through the data storage model backed up by local storage and remote warehouses.

[0030] It can be seen that in an embodiment of the present application, by creating a data directory of the current version of the target model, the historical data files under the historical data directory of the historical version of the target model are copied to the data directory, and the model code and data set are obtained based on the historical data files. The training index data and model files obtained by model training based on the model code and data set are pushed to the first remote warehouse and the second remote warehouse, thereby realizing centralized management and control of the model data. At the same time, the model data stored in the first remote warehouse and the second remote warehouse are synchronized with the locally stored model data, and the disaster recovery of the model data is guaranteed through the data storage model backed up by the local storage and remote warehouse. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in this specification. Those skilled in the art can also derive other drawings based on these drawings without inventive work.

[0032] Figure 1 A processing flow chart of a model data version processing method provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of a model data storage provided in an embodiment of the present application;

[0034] Figure 3 A model data version management block diagram provided in an embodiment of the present application;

[0035] Figure 4 A flowchart of a model data version processing method applied to a multi-tenant scenario provided in an embodiment of the present application;

[0036] Figure 5 A block diagram of model data version management for multi-tenant scenarios provided in an embodiment of the present application;

[0037] Figure 6 A flowchart of another model data version processing method provided in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of a model data version processing device provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of another model data version processing device provided in an embodiment of the present application;

[0040] Figure 9A schematic diagram of the structure of a model data version processing device provided in an embodiment of the present application;

[0041] Figure 10 A schematic structural diagram of another model data version processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0043] The model data version processing method provided in this embodiment creates a data directory of the current version of the target model, and adds the first model code and the first data set of the target model obtained from the first remote warehouse and the second remote warehouse to the created data directory, so as to obtain the second data set by annotating the first data set in the data directory, and obtain the second model code by adjusting the first model code, thereby improving the convenience and flexibility of model data adjustment; and pushes the training indicator data and model files obtained by model training based on the second data set and the second model code in the data directory to the first remote warehouse and the second remote warehouse, so as to achieve synchronization of the model data of the first remote warehouse and the second remote warehouse with the local model data managed with the help of the data directory, and at the same time realizes centralized management and control of the data set, model code, training indicator data, and model files, thereby improving the efficiency of model data version management; specifically, in the process of pushing the training indicator data and model files to the first remote warehouse and the second remote warehouse, the training indicator data and model files are stored in the local warehouse in the form of metadata, thereby forming a data storage model of local storage and remote warehouse backup, thereby ensuring disaster recovery of the model data.

[0044] Reference Figure 1 The model data version processing method provided in this embodiment specifically includes steps S102 to S110.

[0045] Step S102: Create a data directory of the current version of the target model.

[0046] In this embodiment, the target model refers to the object of model data version management. Specifically, in the model training scenario, the target model refers to the model to be trained. The current version refers to the current version created for the target model. For example, if the model is adjusted during model training, the version corresponding to the adjusted model is the current version, and the version corresponding to the model before the adjustment is the historical version.

[0047] In the actual model training process, the training of the target model can be iteratively trained based on the historical version of the target model, or it can be independent of the historical version of the target model by creating a new branch to train the target model accordingly. In addition, there is also a case where there is no historical version of the target model, that is, the target model is trained for the first time. In this case, the model training also does not rely on the historical version.

[0048] Here, the current version of the target model is divided into two categories, one is the current version that depends on the historical version of the target model (the current version for model training based on the historical version of the target model), and the other is the current version that does not depend on the historical version of the target model (the current version for model training does not depend on the historical version of the target model). This embodiment takes the current version that does not depend on the historical version of the target model as an example to illustrate the data management process of the current version of the target model.

[0049] The data directory described in this embodiment is created for the current version of the target model and is used to manage the model data of the current version of the target model, such as the model data involved in the training process of the current version of the target model. The model data includes data sets, model codes, training indicator data and / or model files. During the specific execution process, the data directory can be used to centrally manage the data sets, model codes, training indicator data, and model files involved in the training process of the current version of the target model, thereby improving the efficiency of model data management. At the same time, by defining a unified data directory, not only can the standardized storage of model data be achieved with the help of the data directory, but the application flexibility of model data can also be improved on the basis of standardized storage.

[0050] For example, the directory structure of the data directory created for the current version of model A contains information related to data management of the current version of model A, information related to code management of the current version of model A, subdirectories of data sets, subdirectories of model codes, subdirectories of training indicator data, and subdirectories of model files. Through this directory structure, the corresponding model data of model A in the current version can be quickly located.

[0051] Step S104: Acquire the first model code and first metadata of the target model from the first remote warehouse.

[0052] The first remote warehouse refers to a data warehouse deployed at a remote end for storing model data. The data warehouse is used to store this type of model data (for example, model code) that can be locally modified and locally submitted. Specifically, when the model data includes a data set, model code, training indicator data, and a model file, the first remote warehouse in this embodiment is used to store each version of the model code, metadata of the data set, metadata of the training indicator, and metadata of the model file. In order to improve the efficiency and convenience of version management of model data, the model data can be version controlled through a code management tool. In this case, the first remote warehouse can also be a data warehouse accessed by the code management tool. For example, if the code management tool Git is used to manage the model code of model A, the Gitlab, Gitee, or Github remote code warehouse accessed by the code management tool Git is used as the first remote warehouse.

[0053] In a specific implementation, after obtaining the first model code of the target model from the first remote repository, the first model code is added to the data directory of the current version; the first metadata obtained from the first remote repository refers to the metadata of the first data set of the target model, wherein the metadata of the first data set consists of a data identifier obtained by encrypting the first data set of the target model and the configuration information of the first data set. The first data set of the target model refers to a data set consisting of training samples for model training.

[0054] Step S106 : Based on the data identifier included in the first metadata, obtain a first data set corresponding to the data identifier from the second remote warehouse, and add the first data set to the data directory.

[0055] When the first remote warehouse is introduced to remotely store and version control the model data, for binary files, each time a change is made to the binary file, the modified full binary file needs to be submitted, which will lead to a large increase in the amount of data. To address this, this embodiment introduces a second remote warehouse to remotely store and version control this type of model data where local modifications will result in full submission, thereby reducing the size of the first remote warehouse. Specifically, when the model data includes data sets, model codes, training indicator data, and model files, the second remote warehouse in this embodiment is used to store various versions of the stored data sets, data set training indicators, and model files, such as Figure 2 shown.

[0056] During specific implementation, in order to improve the efficiency of remote storage and version control of model data in the second remote warehouse, the model data stored in the second remote warehouse can be version controlled through data version management tools (such as DVC (Data Version Control)). On this basis, the second remote warehouse can also use file storage for remote storage of model data, such as using NFS (Network File System), HDFS (Hadoop Distribute File System), NAS (Network Attached Storage) and other file storage systems as the second remote warehouse.

[0057] It should be noted that, in addition to the above-mentioned acquisition of the first metadata from the first remote warehouse, and then acquisition of the first data set corresponding to the data identifier from the second remote warehouse based on the data identifier contained in the first metadata and adding it to the data directory, in order to improve the flexibility of model training, the first data set can also be added to the data directory by local addition as a training sample for training the current version of the model.

[0058] Step S108: storing the training indicator data and the model file in a local cache, and generating metadata for the training indicator data and the model file.

[0059] After obtaining the first model code of the target model from the first remote warehouse and the first data set from the second remote warehouse, since the obtained first model code and the first data set are the historical model code and data set of the target model stored remotely, therefore, in the process of performing model training for the current version of the target model, the first model code can be adjusted to obtain the version currently being trained, that is, the second model code of the current version is obtained by adjusting the first model code; similarly, the first data set can also be adjusted to obtain the version currently being trained, that is, the second data set of the current version is obtained by labeling the first data set.

[0060] During the specific execution process, in the process of model training based on the second data set and the second model code in the data directory, the data contained in the second data set is used as training sample data, and the second model code is used as the trained target model for model training, and the training index data and model file are obtained through training; the training index data refers to the corresponding index data determined according to the input and output in the training process, and the model file refers to the runnable file or executable file generated based on the model code.

[0061] After obtaining the training indicator data and the model file, metadata for each of the training indicator data and the model file is generated. In the process of generating the metadata for each of the training indicator data and the model file, a data version management tool can be called to generate metadata. In the process of calling the data version management tool to generate the metadata for each of the training indicator data and the model file, the data version management tool responds to the call and first stores the training indicator data and the model file in a local cache. Optionally, the local cache includes a local cache area occupied by the data version management tool when it is running. Based on this, the data version management tool can more quickly generate metadata for each of the training indicator data and the model file based on the training indicator data and the model file stored in its own local cache area, avoiding the problem of slow efficiency in the metadata generation process caused by the data version management tool reading the training indicator data and the model file from the outside. At the same time, the data version management tool stores the training indicator data and the model file in the local cache area, which also facilitates the subsequent data version management tool to push the training indicator data and the model file to the second remote warehouse, thereby improving the efficiency of pushing the training indicator data and the model file to the second remote warehouse.

[0062] Specifically, in the process of generating the metadata of the training indicator data and the model file respectively, an encryption algorithm is used to encrypt the training indicator data to obtain the data identification of the training indicator data, and the training indicator data is renamed according to the data identification of the training indicator data, and finally the configuration information of the training indicator data and the data identification of the training indicator data are used as the metadata of the training indicator data; similarly, an encryption algorithm is used to encrypt the model file to obtain the data identification of the model file, and the model file is renamed according to the data identification of the model file, and finally the configuration information of the model file and the data identification of the model file are used as the metadata of the model file.

[0063] For example, Figure 3 The model data version management block diagram shown in the figure obtains the first model code of model A from the first remote warehouse to the local and adds it to the data directory of the current version. The algorithm engineer modifies the data directory to obtain the second model code of the current version, and obtains the first data set from the second remote warehouse to the local and adds it to the data directory of the current version. The data annotation personnel modify the data directory to obtain the second data set of the current version; the model is trained using the obtained second model code of the current version and the second data set of the current version to obtain training indicator data and model files, and generate metadata for both the training indicator data and the model files.

[0064] In addition, the first model code and the first data set can also be used as the second model code and the second data set of the current version, and model training can be performed on this basis, and corresponding training indicator data and model files can be obtained through training, as well as metadata for both the training indicator data and the model files can be generated.

[0065] Step S110: storing the metadata in a local warehouse, and pushing the metadata stored in the local warehouse to the first remote warehouse.

[0066] After generating metadata for the training indicator data and metadata for the model file, the metadata for the training indicator data and the metadata for the model file are stored in a local repository, and ultimately pushed to the first remote repository. In a scenario where a code management tool is used to perform version control on the model data and the first remote repository is a data warehouse accessed by the code management tool, the temporary storage area may be a storage area (temporary storage area) occupied by the code management tool during operation.

[0067] After generating the metadata of the training indicator data and the metadata of the model file, this embodiment pushes the model file of the training indicator data stored in the local cache to the second remote warehouse for remote storage, forming a data storage model in which the local storage and the remote warehouse back up each other, thereby ensuring disaster recovery of the model data; at the same time, with the help of the data directory of the current version of the target model, centralized management and control of the data set, model code, training indicator data, and model files during the model training process is realized, thereby improving the efficiency of model data version management.

[0068] As mentioned above, in order to improve the efficiency of model data version control, corresponding tools can also be used to perform version control on the model data. Specifically, the corresponding model data stored in the first remote warehouse is version controlled by the code management tool. Optionally, the metadata is stored in the local warehouse, including: calling the code management tool to store the metadata in a temporary storage area; calling the code management tool to synchronize the metadata stored in the temporary storage area to the local warehouse.

[0069] On this basis, the corresponding model data stored in the second remote warehouse can also be version controlled through the data version management tool. Optionally, the temporary storage area includes the storage area occupied by the code management tool when it is running; the local warehouse includes the local storage space accessed by the code management tool; the local cache includes the local cache area occupied by the data version management tool when it is running.

[0070] Furthermore, on the basis of version control of the model data stored in the first remote warehouse through the code management tool, in order to improve the flexibility of model training, this embodiment provides an optional implementation method, after obtaining the first model code and first metadata of the target model from the first remote warehouse, and before obtaining the first data set corresponding to the data identifier from the second remote warehouse based on the data identifier contained in the first metadata and adding it to the data directory, a branch of the current version is created and switched to the branch; the model training is performed based on the branch.

[0071] In actual applications, the first data set corresponding to the data identifier is obtained from the second remote warehouse and added to the data directory, and the current version of the data set is obtained through annotation processing based on the first data set. In order to enhance the comprehensiveness of the management of the model data of the target model under the current version, this embodiment provides an optional implementation method of adding the model code to the local cache, and pushing the model code stored in the local cache to the second remote warehouse.

[0072] In addition, the second data set may be stored remotely. In an optional implementation provided by this embodiment, the second data set is stored remotely in the following manner:

[0073] adding the second data set to the local cache, and generating metadata of the second data set;

[0074] storing the metadata of the second data set in a temporary storage area, and synchronizing the metadata of the second data set from the temporary storage area to the local warehouse;

[0075] Push the metadata of the second data set stored in the local warehouse to the first remote warehouse.

[0076] Optionally, generating metadata of the second data set includes:

[0077] encrypting the second data set using an encryption algorithm, and renaming the second data set based on a data identifier of the second data set, wherein the data identifier of the second data set is obtained by encrypting the second data set;

[0078] The configuration information of the second data set and the data identifier of the second data set are used as the metadata.

[0079] It should be noted that the remote storage process of the second data set and the second model code can be performed synchronously with the remote storage process of the training indicator data and the model file, or can be performed independently of the remote storage process of the training indicator data and the model file. More specifically, the remote storage process of the second data set and the storage process of the second model code can be performed synchronously or independently. For example, the first data set corresponding to the data identifier is obtained from the second remote warehouse and added to the data directory, and after the current version of the data set is obtained through annotation processing based on the first data set, and after the first model code under the data target is subjected to the current version of the data set, the remote storage of the second model code and the second data set is performed; or, during the remote storage process of the training indicator data and the model file, the remote storage of the second model code and the second data set is performed synchronously.

[0080] Using the above example, Figure 3 In the model data version management block diagram shown, the current version of the second model code, the metadata of the current version of the second data set, the metadata of the training indicator data generated after the model training, and the metadata of the model file are pushed to the first remote warehouse for storage, and the current version of the second data set, the training indicator data and the model file generated after the model training are pushed to the second remote warehouse for storage.

[0081] As described above, the current version created for the target model is divided into two types: the current version that depends on the historical version of the target model (the current version for model training based on the historical version of the target model) and the current version that does not depend on the historical version of the target model (the current version for model training that does not depend on the historical version of the target model). On this basis, in an optional implementation manner provided by this embodiment, if the current version is created based on the historical version of the target model, the following operations are performed:

[0082] Copy the historical data set and historical model code in the historical data directory of the historical version of the target model to the data directory;

[0083] storing second training indicator data and a second model file obtained by model training based on a third data set and a third model code in the data directory into the local cache, and generating metadata for the second training indicator data and the second model file; the third data set is obtained by annotating the historical data set, and the third model code is obtained by adjusting the historical model code;

[0084] Storing the second training indicator data and the metadata of the second model file in a temporary storage area, and synchronizing them from the temporary storage area to the local warehouse;

[0085] The second training indicator data and the metadata of the second model file stored in the local warehouse are pushed to the first remote warehouse, and the data file stored in the local cache is pushed to the second remote warehouse.

[0086] In addition, after creating the data directory of the current version of the target model and before obtaining the first model code and first metadata of the target model from the first remote repository, it is determined whether the current version is created for a historical version of the target model;

[0087] If the current version is not created for the historical version of the target model, it indicates that the current version created for the target model is a current version that does not depend on the historical version of the target model, and the above-mentioned processing process from step S104 to step S110 is executed;

[0088] If the current version is created for a historical version of the target model, to improve the efficiency of model data version control during model training, perform the following operations:

[0089] Copy the historical data set and historical model code in the historical data directory of the historical version of the target model to the data directory;

[0090] storing second training indicator data and a second model file obtained by model training based on a third data set and a third model code in the data directory into the local cache, and generating metadata for the second training indicator data and the second model file; the third data set is obtained by annotating the historical data set, and the third model code is obtained by adjusting the historical model code;

[0091] Storing the second training indicator data and the metadata of the second model file in a temporary storage area, and synchronizing them from the temporary storage area to the local warehouse;

[0092] The second training indicator data and the metadata of the second model file stored in the local warehouse are pushed to the first remote warehouse, and the data file stored in the local cache is pushed to the second remote warehouse.

[0093] In this embodiment, in order to improve the efficiency of model data version control, a data storage mapping relationship of the current version can also be created, and the mapping relationship between local storage and remote storage of each version of model data can be maintained through the data storage mapping relationship, wherein the storage mapping relationship carries the local storage path, tenant identifier, version identifier of the current version, and remote warehouse storage path.

[0094] The following takes the application of a model data version processing method provided in this embodiment in a multi-tenant scenario as an example. Figure 4 and Figure 5 , further explanation of the model data version processing method provided in this embodiment is given in Figure 4 , a model data version processing method applied to a multi-tenant scenario specifically includes the following steps.

[0095] Step S402: Create a data directory of the current version of the target model under the tenant's project.

[0096] Step S404, determining whether the current version is created for a historical version of the target model;

[0097] If yes, go to step S406;

[0098] If not, execute step S408.

[0099] Step S406: copy the first model code and the first metadata in the historical data directory of the historical version to the data directory of the current version.

[0100] Step S408: Invoke the code management tool to obtain the first model code and first metadata of the target model from the first remote warehouse, where the first remote warehouse is a remote data warehouse accessed by the code management tool.

[0101] Step S410: Call a code management tool to create a branch of the current version, and switch to the branch of the current version.

[0102] Step S412, determining whether the current version is created for a historical version of the target model;

[0103] If yes, go to step S416;

[0104] If so, execute steps S414 and S416.

[0105] Step S414: Invoke the data version management tool to obtain the first data set corresponding to the data identifier included in the first metadata from the second remote warehouse and add it to the data directory. The second remote warehouse is a remote data warehouse accessed by the data version management tool.

[0106] like Figure 5As shown, the data labeler modifies the first data set to obtain the data set (the current version of the data set), and the algorithm engineer adjusts the first model code to obtain the model code (the current version of the model code); in addition, the data labeler can also add a new data set as the current version of the data set.

[0107] Step S416: Store the training indicator data and model file obtained by model training based on the current version of the model code and data set into the local cache, and generate metadata for both the training indicator data and the model file.

[0108] When the algorithm engineer begins model training, remote storage of the current version of the dataset and model code is triggered. For details, see the detailed implementation of the current version of the dataset and model code provided above. Specifically, data version management tools can be used to store the training metric data and model files obtained during model training in the local cache and generate metadata for both the training metric data and model files.

[0109] Step S418: calling the code management tool to store the metadata of the two in a temporary storage area and synchronize them to the local warehouse, and pushing the metadata of the two stored in the local warehouse to the first remote warehouse.

[0110] Step S420, determining whether the current version is created for a historical version of the target model;

[0111] If yes, go to step S422;

[0112] If not, no action is required.

[0113] Step S422: Call the data version management tool to push the training indicator data and model files stored in the local cache to the second remote warehouse.

[0114] Another embodiment of a model data version processing method provided in this specification is as follows:

[0115] The model data version processing method provided in this embodiment creates a data directory of the current version of the target model, copies the historical data files under the historical data directory of the historical version of the target model to the data directory, generates a data set and model code based on the historical data files under the data directory, and pushes the training indicator data and model files obtained by model training based on the data set and model code under the data directory to the first remote warehouse and the second remote warehouse, thereby realizing synchronization of the model data of the first remote warehouse and the second remote warehouse with the local model data managed with the help of the data directory, and at the same time realizing centralized management and control of the model data, thereby improving the efficiency of model data version management; specifically, in the process of pushing the training indicator data and model files to the first remote warehouse and the second remote warehouse, the training indicator data and model files are stored in the local warehouse in the form of metadata, thereby forming a data storage model of local storage and remote warehouse backup, thereby ensuring disaster recovery of the model data.

[0116] Reference Figure 6 The model data version processing method provided in this embodiment specifically includes steps S602 to S608.

[0117] Step S602 : creating a data directory of the current version of the target model, and copying historical data files in the historical data directory of the historical version of the target model to the data directory.

[0118] In this embodiment, the target model refers to the object of model data version management. Specifically, in the model training scenario, the target model refers to the model to be trained. The current version refers to the current version created for the target model. For example, if the model is adjusted during model training, the version corresponding to the adjusted model is the current version, and the version corresponding to the model before the adjustment is the historical version.

[0119] The data directory described in this embodiment is created for the current version of the target model and is used to manage the model data of the current version of the target model, such as the model data involved in the training process of the current version of the target model. The model data specifically includes data sets, model codes, training indicator data and / or model files. During the specific execution process, the data sets, model codes, training indicator data, and model files involved in the training process of the current version of the target model can be centrally managed through the data directory, thereby improving the efficiency of model data management. At the same time, by defining a unified data directory, not only can the standardized storage of model data be achieved with the help of the data directory, but the application flexibility of model data can also be improved on the basis of standardized storage.

[0120] The historical data files in the historical data directory of the historical version include the historical data set and historical model code of the historical version. The historical data set refers to a data set composed of training samples in the process of model training of the historical version.

[0121] Step S604: adding the data file to the local cache and generating metadata of the data file.

[0122] The data file includes training index data and model files obtained by model training based on the data set and model code in the data directory, and the data set and the model code are generated based on the historical data file; as mentioned above, the historical data file in the historical data directory of the historical version includes the historical data set and historical model code of the historical version, and on this basis, the data set is obtained by annotating the historical data set. In addition, the data set can also be obtained by adding, deleting or modifying the historical data set, and the model code is obtained by adjusting the historical model code.

[0123] During the specific execution process, in the process of model training based on the data set and model code in the data directory, the data contained in the data set is used as training sample data, and the model code is used as the target model to be trained for model training, and the training indicator data and model file are obtained through training; the training indicator data refers to the corresponding indicator data determined according to the input and output in the training process, and the model file refers to the runnable file or executable file generated based on the model code.

[0124] In order to improve the management efficiency of the data files, the data files can also be managed by a data version management tool. When the data version management tool is introduced, the local cache includes the local cache area occupied by the data version management tool when it is running.

[0125] In an optional implementation manner provided by this embodiment, generating metadata of the data file includes:

[0126] Encrypting the data file using an encryption algorithm to obtain a data identifier of the data file, and renaming the data file according to the data identifier of the data file;

[0127] The configuration information of the data file and the data identifier of the data file are used as the metadata.

[0128] Step S606: storing the metadata in a temporary storage area, and synchronizing the metadata from the temporary storage area to a local warehouse.

[0129] The temporary storage area is used to temporarily store the metadata, and the temporary storage area can be a temporary storage area or a cache area; the local warehouse is used to persistently store the metadata locally. In specific implementation, to improve the efficiency of metadata version control of the data file, a corresponding version management tool can be used to version control the model data, specifically using a code management tool to version control the metadata of the data file. Based on this, the temporary storage area includes the storage area occupied by the code management tool during operation; the local warehouse includes the local storage space accessed by the code management tool.

[0130] Step S608: Push the metadata stored in the local warehouse to the first remote warehouse, and push the data files stored in the local cache to the second remote warehouse.

[0131] The first remote warehouse refers to a data warehouse deployed at a remote end for storing model data. The data warehouse is used to store this type of model data (for example, model code) that can be locally modified and locally submitted. Specifically, when the model data includes a data set, model code, training indicator data, and a model file, the first remote warehouse in this embodiment is used to store each version of the model code, metadata of the data set, metadata of the training indicator, and metadata of the model file. In order to improve the efficiency and convenience of version management of model data, the model data can be version controlled through a code management tool. In this case, the first remote warehouse can also be a data warehouse accessed by the code management tool. For example, if the code management tool Git is used to manage the model code of model A, the Gitlab, Gitee, or Github remote code warehouse accessed by the code management tool Git is used as the first remote warehouse.

[0132] When the first remote warehouse is introduced for remote storage and version control of model data, for binary files, each time a change is made to the binary file, the modified full binary file needs to be submitted, which will result in a significant increase in the amount of data. To address this, this embodiment introduces a second remote warehouse to remotely store and version control this type of model data, where local modifications will result in a full submission, thereby reducing the size of the first remote warehouse. Specifically, when the model data includes a dataset, model code, training indicator data, and model files, the second remote warehouse in this embodiment is used to store various versions of the stored dataset, dataset training indicators, and model files.

[0133] During specific implementation, in order to improve the efficiency of remote storage and version control of model data in the second remote warehouse, the model data stored in the second remote warehouse can be version controlled through data version management tools (such as DVC (Data Version Control)). On this basis, the second remote warehouse can also use file storage for remote storage of model data, such as using NFS (Network File System), HDFS (Hadoop Distribute File System), NAS (Network Attached Storage) and other file storage systems as the second remote warehouse.

[0134] In actual applications, after generating the current version of the model code based on the historical model code contained in the historical data file, in order to enhance the comprehensiveness of the management of the model data of the target model under the current version, this embodiment provides an optional implementation method of adding the model code to the local cache, and pushing the model code stored in the local cache to the second remote warehouse.

[0135] In addition, after the current version of the dataset is generated based on the historical dataset included in the historical data file, the current version of the dataset may be remotely stored. In an optional implementation provided by this embodiment, the dataset is remotely stored in the following manner:

[0136] Adding the data set to the local cache and generating metadata for the data set;

[0137] Storing the metadata of the data set in a temporary storage area, and synchronizing the metadata of the data set from the temporary storage area to the local warehouse;

[0138] Push the metadata of the data set stored in the local warehouse to the first remote warehouse.

[0139] It should be noted that the remote storage process of the data set and the model code can be carried out synchronously with the remote storage process of the training index data and the model file, or can be carried out independently of the remote storage process of the training index data and the model file. More specifically, the remote storage process of the data set can be carried out synchronously with the storage process of the model code, or can be carried out independently.

[0140] This embodiment, based on storing the metadata of the training indicator data and the metadata of the model files in the local warehouse, pushes the metadata of the training indicator data and the metadata of the model files to the first remote warehouse for remote storage, forming a data storage model in which the local storage and the remote warehouse back up each other, thereby ensuring disaster recovery of the model data; at the same time, with the help of the data directory of the current version of the target model, centralized management and control of the data set, model code, training indicator data, and model files during the model training process is realized, thereby improving the efficiency of model data version management.

[0141] As mentioned above, in order to improve the efficiency of model data version control, corresponding tools can also be used to perform version control on the model data. Specifically, the corresponding model data stored in the first remote warehouse is version controlled by the code management tool, and the corresponding model data stored in the second remote warehouse is version controlled by the data version management tool. Based on this, on the basis of performing version control on the model data stored in the first remote warehouse through the code management tool, in order to improve the flexibility of model training, this embodiment provides an optional implementation method, after creating a data directory for the current version of the target model and copying the historical data files under the historical data directory of the historical version of the target model to the data directory, and before adding the data files to the local cache and generating the metadata of the data files, a branch of the current version is created and switched to the branch; the model training is performed based on the branch.

[0142] In this embodiment, in order to improve the efficiency of model data version control, a data storage mapping relationship of the current version can also be created, and the mapping relationship between local storage and remote storage of each version of model data can be maintained through the data storage mapping relationship, wherein the storage mapping relationship carries the local storage path, tenant identifier, version identifier of the current version, and remote warehouse storage path.

[0143] An embodiment of a model data version processing device provided in this specification is as follows:

[0144] In the above embodiment, a model data version processing method is provided. Correspondingly, a model data version processing device running on a service client is also provided, which will be described below with reference to the accompanying drawings.

[0145] Reference Figure 7 , which shows a schematic diagram of a model data version processing device provided by this embodiment.

[0146] Since the device embodiment corresponds to the method embodiment, the description is relatively simple. For the relevant parts, please refer to the corresponding description of the method embodiment provided above. The device embodiment described below is only illustrative.

[0147] This embodiment provides a model data version processing device, including:

[0148] A data directory creation module 702 is used to create a data directory of the current version of the target model;

[0149] A data acquisition module 704 is configured to acquire a first model code and first metadata of the target model from a first remote repository;

[0150] A data set adding module 706 is configured to obtain, based on the data identifier included in the first metadata, a first data set corresponding to the data identifier from a second remote repository, and add the first data set to the data directory;

[0151] A metadata generation module 708 is configured to store the training indicator data and the model file in a local cache and generate metadata for each of the training indicator data and the model file; wherein the training indicator data and the model file are obtained by performing model training based on a second data set and a second model code in the data directory, wherein the second data set is obtained by annotating the first data set, and the second model code is obtained by adjusting the first model code;

[0152] The metadata storage module 710 is configured to store the metadata in a local warehouse, and push the metadata stored in the local warehouse to the first remote warehouse.

[0153] Another embodiment of a model data version processing device provided in this specification is as follows:

[0154] In the above embodiment, a model data version processing method is provided. Correspondingly, a model data version processing device running on a service client is also provided, which will be described below with reference to the accompanying drawings.

[0155] Reference Figure 8 , which shows a schematic diagram of a model data version processing device provided by this embodiment.

[0156] Since the device embodiment corresponds to the method embodiment, the description is relatively simple. For the relevant parts, please refer to the corresponding description of the method embodiment provided above. The device embodiment described below is only illustrative.

[0157] This embodiment provides a model data version processing device, including:

[0158] A data directory creation module 802 is configured to create a data directory of a current version of a target model and copy historical data files in a historical data directory of a historical version of the target model to the data directory;

[0159] Metadata generation module 804, configured to add data files to the local cache and generate metadata for the data files; the data files include training indicator data and model files obtained by model training based on the data sets and model codes in the data directory, the data sets and model codes being generated based on the historical data files;

[0160] The metadata storage module 806 is used to store the metadata in a temporary storage area and synchronize the metadata from the temporary storage area to a local warehouse;

[0161] The data push module 808 is configured to push the metadata stored in the local warehouse to the first remote warehouse, and push the data files stored in the local cache to the second remote warehouse.

[0162] An embodiment of a model data version processing device provided in this specification is as follows:

[0163] Corresponding to the model data version processing method described above, based on the same technical concept, an embodiment of the present application further provides a model data version processing device, which is used to execute the model data version processing method provided above. Figure 9 A structural diagram of a model data version processing device provided in an embodiment of the present application.

[0164] This embodiment provides a model data version processing device, including:

[0165] like Figure 9 As shown, the model data version processing device may have relatively large differences due to different configurations or performances, and may include one or more processors 901 and memory 902, and the memory 902 may store one or more storage applications or data. Among them, the memory 902 can be a temporary storage or a persistent storage. The application stored in the memory 902 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the model data version processing device. Furthermore, the processor 901 can be configured to communicate with the memory 902 and execute a series of computer executable instructions in the memory 902 on the model data version processing device. The model data version processing device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input / output interfaces 905, one or more keyboards 906, etc.

[0166] In a specific embodiment, a model data version processing device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the model data version processing device, and the one or more programs are configured to be executed by one or more processors, including computer-executable instructions for performing the following:

[0167] Create a data directory for the current version of the target model;

[0168] Acquire first model code and first metadata of the target model from a first remote repository;

[0169] Based on the data identifier included in the first metadata, obtaining a first data set corresponding to the data identifier from a second remote repository, and adding the first data set to the data directory;

[0170] Storing training indicator data and a model file in a local cache, and generating metadata for the training indicator data and the model file; wherein the training indicator data and the model file are obtained by performing model training based on a second data set and a second model code in the data directory, the second data set is obtained by annotating the first data set, and the second model code is obtained by adjusting the first model code;

[0171] The metadata is stored in a local warehouse, and the metadata stored in the local warehouse is pushed to the first remote warehouse.

[0172] An embodiment of a model data version processing device provided in this specification is as follows:

[0173] Corresponding to the model data version processing method described above, based on the same technical concept, an embodiment of the present application further provides a model data version processing device, which is used to execute the model data version processing method provided above. Figure 10 A structural diagram of a model data version processing device provided in an embodiment of the present application.

[0174] This embodiment provides a model data version processing device, including:

[0175] like Figure 10As shown, the model data version processing device may have relatively large differences due to different configurations or performances, and may include one or more processors 1001 and memory 1002, and the memory 1002 may store one or more storage applications or data. Among them, the memory 1002 can be a temporary storage or a persistent storage. The application stored in the memory 1002 may include one or more modules (not shown in the figure), and each module may include a series of computer-executable instructions in the model data version processing device. Furthermore, the processor 1001 can be configured to communicate with the memory 1002 and execute the series of computer-executable instructions in the memory 1002 on the model data version processing device. The model data version processing device may also include one or more power supplies 1003, one or more wired or wireless network interfaces 1004, one or more input / output interfaces 1005, one or more keyboards 1006, etc.

[0176] In a specific embodiment, a model data version processing device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the model data version processing device, and the one or more programs are configured to be executed by one or more processors, including computer-executable instructions for performing the following:

[0177] Creating a data directory for the current version of the target model, and copying historical data files in the historical data directory of the historical version of the target model to the data directory;

[0178] Adding a data file to the local cache and generating metadata for the data file; the data file includes training indicator data and a model file obtained by model training based on the data set and model code in the data directory, the data set and the model code being generated based on the historical data file;

[0179] Storing the metadata in a temporary storage area, and synchronizing the metadata from the temporary storage area to a local warehouse;

[0180] The metadata stored in the local warehouse is pushed to the first remote warehouse, and the data file stored in the local cache is pushed to the second remote warehouse.

[0181] An embodiment of a computer-readable storage medium provided in this specification is as follows:

[0182] Corresponding to the model data version processing method described above, based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium.

[0183] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented:

[0184] Create a data directory for the current version of the target model;

[0185] Acquire first model code and first metadata of the target model from a first remote repository;

[0186] Based on the data identifier included in the first metadata, obtaining a first data set corresponding to the data identifier from a second remote repository, and adding the first data set to the data directory;

[0187] Storing training indicator data and a model file in a local cache, and generating metadata for the training indicator data and the model file; wherein the training indicator data and the model file are obtained by performing model training based on a second data set and a second model code in the data directory, the second data set is obtained by annotating the first data set, and the second model code is obtained by adjusting the first model code;

[0188] The metadata is stored in a local warehouse, and the metadata stored in the local warehouse is pushed to the first remote warehouse.

[0189] It should be noted that the embodiment of the computer-readable storage medium in this specification and the embodiment of the model data version processing method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method mentioned above, and the repeated parts will not be repeated.

[0190] An embodiment of a computer-readable storage medium provided in this specification is as follows:

[0191] Corresponding to the model data version processing method described above, based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium.

[0192] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented:

[0193] Creating a data directory for the current version of the target model, and copying historical data files in the historical data directory of the historical version of the target model to the data directory;

[0194] Adding a data file to the local cache and generating metadata for the data file; the data file includes training indicator data and a model file obtained by model training based on the data set and model code in the data directory, the data set and the model code being generated based on the historical data file;

[0195] Storing the metadata in a temporary storage area, and synchronizing the metadata from the temporary storage area to a local warehouse;

[0196] The metadata stored in the local warehouse is pushed to the first remote warehouse, and the data file stored in the local cache is pushed to the second remote warehouse.

[0197] It should be noted that the embodiment of the computer-readable storage medium in this specification and the embodiment of the model data version processing method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding method mentioned above, and the repeated parts will not be repeated.

[0198] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0199] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable model data version processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable model data version processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions can also be stored in a computer readable memory that can guide a computer or other programmable model data version processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable model data version processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0203] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0204] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0205] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0206] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0207] The embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0208] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0209] The foregoing description is merely an example of the present invention and is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims herein.

Claims

1. A model data version processing method, characterized in that: The method comprises: Create a data directory for the current version of the target model; Acquire first model code and first metadata of the target model from a first remote repository; Based on the data identifier included in the first metadata, obtaining a first data set corresponding to the data identifier from a second remote repository, and adding the first data set to the data directory; Storing training indicator data and a model file in a local cache, and generating metadata for the training indicator data and the model file; wherein the training indicator data and the model file are obtained by performing model training based on a second data set and a second model code in the data directory, the second data set is obtained by annotating the first data set, and the second model code is obtained by adjusting the first model code; The metadata is stored in a local warehouse, and the metadata stored in the local warehouse is pushed to the first remote warehouse.

2. The model data version processing method according to claim 1, characterized in that: If the current version is created based on a historical version of the target model, perform the following operations: Copy the historical data set and historical model code in the historical data directory of the historical version of the target model to the data directory; storing second training indicator data and a second model file obtained by model training based on a third data set and a third model code in the data directory into the local cache, and generating metadata for the second training indicator data and the second model file; the third data set is obtained by annotating the historical data set, and the third model code is obtained by adjusting the historical model code; Storing the second training indicator data and the metadata of the second model file in a temporary storage area, and synchronizing them from the temporary storage area to the local warehouse; The second training indicator data and the metadata of the second model file stored in the local warehouse are pushed to the first remote warehouse, and the data file stored in the local cache is pushed to the second remote warehouse.

3. The model data version processing method according to claim 1, characterized in that: Also includes: adding the second model code to the local cache; Push the second model code stored in the local cache to the second remote warehouse.

4. The model data version processing method according to claim 1, characterized in that: Also includes: adding the second data set to the local cache, and generating metadata of the second data set; storing the metadata of the second data set in a temporary storage area, and synchronizing the metadata of the second data set from the temporary storage area to the local warehouse; Push the metadata of the second data set stored in the local warehouse to the first remote warehouse.

5. The model data version processing method according to claim 4, characterized in that: Generating metadata of the second data set includes: encrypting the second data set using an encryption algorithm, and renaming the second data set based on a data identifier of the second data set, wherein the data identifier of the second data set is obtained by encrypting the second data set; The configuration information of the second data set and the data identifier of the second data set are used as the metadata.

6. The model data version processing method according to claim 1, wherein storing the metadata in a local repository comprises: Calling a code management tool to store the metadata in a temporary storage area; The code management tool is called to synchronize the metadata stored in the temporary storage area to the local warehouse.

7. The model data version processing method according to claim 6, characterized in that: The temporary storage area includes a storage area occupied by the code management tool when it is running; The local warehouse includes a local storage space accessed by the code management tool; The local cache includes a local cache area occupied by the data version management tool when it is running.

8. The model data version processing method according to claim 7, characterized in that: After the step of obtaining the first model code and first metadata of the target model from the first remote repository is executed, and before the step of obtaining the first data set corresponding to the data identifier from the second remote repository based on the data identifier included in the first metadata is executed, and before the step of adding the first data set to the data catalog is executed, the method further includes: Create a branch of the current version and switch to the branch, wherein the model training is performed based on the branch.

9. A model data version processing method, characterized in that: The method comprises: Creating a data directory for the current version of the target model, and copying historical data files in the historical data directory of the historical version of the target model to the data directory; Adding a data file to the local cache and generating metadata for the data file; the data file includes training indicator data and a model file obtained by model training based on the data set and model code in the data directory, the data set and the model code being generated based on the historical data file; Storing the metadata in a temporary storage area, and synchronizing the metadata from the temporary storage area to a local warehouse; The metadata stored in the local warehouse is pushed to the first remote warehouse, and the data file stored in the local cache is pushed to the second remote warehouse.

10. A model data version processing device, characterized in that: The device comprises: The data directory creation module is used to create the data directory of the current version of the target model; A data acquisition module, configured to acquire a first model code and first metadata of the target model from a first remote repository; a data set adding module, configured to obtain, based on the data identifier included in the first metadata, a first data set corresponding to the data identifier from a second remote repository, and add the first data set to the data directory; a metadata generation module, configured to store the training indicator data and the model file in a local cache, and to generate metadata for each of the training indicator data and the model file; wherein the training indicator data and the model file are obtained by performing model training based on a second data set and a second model code in the data directory, the second data set being obtained by annotating the first data set, and the second model code being obtained by adjusting the first model code; The metadata storage module is used to store the metadata in a local warehouse and push the metadata stored in the local warehouse to the first remote warehouse.

11. A model data version processing device, characterized in that: The device comprises: A processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to execute the model data version processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the model data version processing method according to any one of claims 1 to 9 is implemented.

Images