Method, apparatus, computer device, and storage medium for managing a machine learning model

Abstract

The present invention relates to the field of artificial intelligence technology, and particularly to a management method, device, computer device and storage medium for machine learning models. The method includes determining a data set and a model corresponding to the configuration information of a process node; the process node is any process node in a process, and the configuration information includes execution conditions pointing to the data set and the model; establishing an association relationship between the process node, the data set and the model; wherein the association relationship can be used to indicate the data set and the model called when the process node is executed. By matching the data sets and models required when each process node is executed throughout the process and establishing an association relationship between each process node, the data set and the model to form a closed loop, the present invention can closely associate data production and data use in an environment where multiple models share data sets, improve the efficiency and effect of data use, and further improve the management efficiency of long-cycle processes.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a management method, device, computer device and storage medium for machine learning models. Background Art

[0002] The machine learning model itself has strong adaptability and may find problem domains and corresponding solutions in various scenarios. However, it is still difficult to provide a complete solution for specific problems in many long-term research fields.

[0003] Based on this, the inventors of the present application found in the research and practice of the prior art that the existing research on machine learning generally focuses on the optimization of the model itself. Although there are also engineering practices on model management, there is little research and practice on the training, verification and iteration of multiple models on a shared data set. Taking intelligent warehousing as an example, machine learning can solve the placement problem of high-demand SKUs (Stock Keeping Unit) through modeling and can also provide the future demand for warehousing of long-tail demands, but it cannot completely plan the storage cycle, placement location, etc. of all commodities as a whole.

[0004] For another example, in industries such as pharmaceuticals, materials, and chemicals, machine learning models have been applied in many different directions. The research cycles in these fields are long and the directions are wide, and the situation of using models in multiple scenarios will be faced. However, due to the large differences in these models themselves, the training processes and output results are different, and the output cycles are also different. Using the existing machine learning model management is likely to lead to the disconnection between data production and data use, reducing the data use efficiency and effect. Summary of the Invention

[0005] Based on this, in view of the problem in the prior art that in long-term research fields, data production and data use are disconnected, reducing data use efficiency and effect, it is necessary to provide a management method, device, computer device and storage medium for machine learning models.

[0006] In a first aspect, an embodiment of the present application provides a management method for a machine learning model, including:

[0007] Determine a data set and a model for the configuration information corresponding to a process node; wherein, the process node is any process node in a process, and the configuration information includes execution conditions pointing to the data set and the model;

[0008] Establish an association relationship between the process node and the data set and the model; wherein, the association relationship can be used to indicate the data set and the model called by the process node when it is executed.

[0009] Optionally, before determining the dataset and model of the configuration information of the corresponding process node, it further includes:

[0010] Obtain a dataset;

[0011] Store the dataset in a dataset library.

[0012] Optionally, before determining the dataset and model of the configuration information of the corresponding process node, it further includes:

[0013] Obtain a model;

[0014] Store the model in a model library; wherein,

[0015] The model carries a model description for determining the usage scenario of the model.

[0016] Optionally, after storing the dataset in the dataset library and storing the model in the model library, it further includes:

[0017] Perform versioning on the dataset in the dataset library and the model in the model library through a DVC tool;

[0018] Establish an association relationship between a single dataset and all related models for the versioned dataset and model.

[0019] Optionally, after establishing the association relationship between the process node and the dataset and the model, it further includes:

[0020] When the dataset is updated, update all models associated with the dataset;

[0021] Update the association relationship where the updated dataset and / or any updated model is located according to the updated dataset and all updated models.

[0022] Optionally, when the dataset is updated, updating all models associated with the dataset includes:

[0023] Use the updated dataset as input to train all models associated with the dataset;

[0024] Determine whether the training effect of the currently trained model meets a first preset condition;

[0025] When it is met, update the currently trained model, otherwise do not perform the operation of updating the currently trained model.

[0026] Optionally, updating the association relationship where the updated dataset and / or any updated model is located according to the updated dataset and all updated models includes:

[0027] Invoke the updated dataset and all updated models, and execute all process nodes associated with the updated dataset and / or any of the updated models;

[0028] After determining that all process nodes have been executed, determine whether the overall process effect meets the second preset condition;

[0029] When it is met, update the association relationship where the updated dataset and / or any of the updated models are located, otherwise do not perform the operation of updating the association relationship where the updated dataset and / or any of the updated models are located.

[0030] Optionally, the overall process effect is represented by weighted summation of the execution results of each process node.

[0031] In a second aspect, an embodiment of the present application provides a management device for a machine learning model, including:

[0032] A screening module, configured to determine a dataset and a model corresponding to the configuration information of a process node; wherein, the process node is any process node in the process, and the configuration information includes execution conditions pointing to the dataset and the model;

[0033] An association module, configured to establish an association relationship between the process node and the dataset and the model; wherein, the association relationship can be used to indicate the dataset and the model determined to be invoked when the process node is executed.

[0034] Optionally, the management device for the machine learning model further includes:

[0035] An acquisition module, configured to acquire a dataset and store the dataset in a dataset library.

[0036] Optionally, the acquisition module is further configured to acquire a model and store the model in a model library; wherein, the model carries a model description for determining the usage scenario of the model.

[0037] Optionally, the management device for the machine learning model further includes:

[0038] A preprocessing module, configured to perform versioning processing on the dataset in the dataset library and the model in the model library through DVC tools;

[0039] The preprocessing module is further configured to establish an association relationship between a single dataset and all related models for the versioned dataset and model.

[0040] Optionally, the management device for the machine learning model further includes:

[0041] An update module, configured to update all models associated with the data set when the data set is updated;

[0042] The update module is further configured to update the association relationship where the updated data set and / or any updated model are located according to the updated data set and all updated models.

[0043] Optionally, the update module includes:

[0044] A training module, configured to use the updated data set as input to train all models associated with the data set;

[0045] A judgment module, configured to judge whether the training effect of the currently trained model meets a first preset condition; when it is met, prompt to update the currently trained model, otherwise do not perform the operation of updating the currently trained model.

[0046] Optionally, the update module further includes:

[0047] An execution module, configured to call the updated data set and all updated models, and execute all process nodes associated with the updated data set and / or any updated model;

[0048] The judgment module is further configured to judge whether the overall process effect meets a second preset condition after all process nodes are executed; when it is met, prompt to update the association relationship where the updated data set and / or any updated model are located, otherwise do not perform the operation of updating the association relationship where the updated data set and / or any updated model are located.

[0049] In a third aspect, an embodiment of the present application provides a computer device, including: a processor, a memory, and a computer program stored on the memory, the processor is coupled to the memory, and the processor executes the computer program during operation to implement the management method of the machine learning model as described above.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, the computer storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is caused to execute the instructions of the management method of the machine learning model as described above.

[0051] One of the above technical solutions has the following advantages and beneficial effects:

[0052] In each embodiment of the present application, by matching the data sets and models required when each process node is executed throughout the process, and establishing the association relationships between each process node, the data set, and the model to form a closed loop, it is possible to closely associate data production and data use in an environment where multiple models share data sets, improve the efficiency and effect of data use, and further enhance the management efficiency of long-cycle processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present application will describe the embodiments in conjunction with the drawings. The drawings of the present application are only used to describe the embodiments for display purposes. Without departing from the principles of the present application, those skilled in the art can easily make other embodiments according to the following description based on the steps.

[0054] Figure 1 It is a schematic flowchart of a machine learning model management method in an embodiment of the present application.

[0055] Figure 2 It is a schematic flowchart of a machine learning model management method in an embodiment of the present application.

[0056] Figure 3 It is a schematic flowchart of a machine learning model management method in an embodiment of the present application.

[0057] Figure 4 It is a schematic flowchart of a machine learning model management method in an embodiment of the present application.

[0058] [[ID=2)]] Figure 5 It is a schematic flowchart of a machine learning model management method in an embodiment of the present application.

[0059] Figure 6 It is a schematic flowchart of a machine learning model management system for molecular property prediction in an embodiment of the present application.

[0060] Figure 7 It is a schematic diagram of the system framework corresponding to the machine learning model management method in an embodiment of the present application.

[0061] Figure 8 It is a schematic structural diagram of a machine learning model management device in an embodiment of the present application.

[0062] Figure 9 It is a schematic structural diagram of a machine learning model management method in an embodiment of the present application.

[0063] Figure 10 It is a schematic structural diagram of a machine learning model management method in an embodiment of the present application.

[0064] Figure 11 It is a schematic structural diagram of a machine learning model management method in an embodiment of the present application. Detailed implementation manners

[0065] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. In addition, it should be noted that, for the sake of description, only the parts related to the present application are shown in the accompanying drawings, rather than all the structures. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0066] The terms "first", "second", etc. in the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0067] Referring to "embodiments" herein means that specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0068] The following explains some of the technical terms involved herein to facilitate the understanding of those skilled in the art.

[0069] 1. Machine learning: A branch of artificial intelligence. The research history of artificial intelligence has a natural and clear context from focusing on "reasoning", to focusing on "knowledge", and then to focusing on "learning". Obviously, machine learning is a way to achieve artificial intelligence, that is, to use machine learning as a means to solve problems in artificial intelligence.

[0070] 2. Graph database: A database that uses a graph structure for semantic queries. It uses nodes, edges, and attributes to represent and store data. The key concept of this system is the graph, which directly associates the data items in the storage with the set of data nodes and the edges representing relationships between the nodes.

[0071] 3. Data version management: A software engineering technique that can ensure that the same program files edited by different people are synchronized during the software development process.

[0072] 4. Data Warehouse: A central repository for integrated data from one or more different sources, storing current and historical data together.

[0073] 5. Model: A file that can recognize specific types of patterns after being trained.

[0074] Due to the strong adaptability of machine learning models themselves, problem domains and corresponding solutions can be found in a variety of scenarios. Therefore, in fields with long research cycles, such as the pharmaceutical, materials, and chemical industries, machine learning models have been applied in many different directions. Especially in the preparation of new drugs and new materials, deep learning models are widely used. However, due to the long research cycle and wide range of directions, for example, in new drug research, it is necessary to study the interaction between drug molecules and receptors, as well as the physical properties of the molecules themselves, such as solubility, toxicity, etc., which will face the situation of using models in multiple scenarios. The models used in multiple scenarios are quite different from each other in terms of training processes, output results, and output cycles. Moreover, there is a lack of solutions in the prior art for training, validating, and iterating multiple models on a shared dataset. As a result, in a complete process, even if there are many models using the same dataset, there is a disconnection between data production and data use, with low data use efficiency and insignificant effects.

[0075] Based on this, in long-cycle research fields, there are multi-dimensional models that rely on a common dataset. These models, either based on scenarios or on processes, solve a certain problem. The management method, device, computer device, and storage medium for machine learning models provided by the embodiments of the present application can, in the whole process, match the datasets and models required when each process node is executed, and establish the association relationships between each process node, the dataset, and the model to form a closed loop. In an environment where multiple models share a dataset, it can closely associate data production and data use, improve data use efficiency and effects, and further enhance the management efficiency of long-cycle processes. Further, the management method can use the update of the dataset as a trigger condition. When the dataset is updated, or the model is updated, it triggers the update of the association relationships between the dataset, the model, and the process (such as DAG, Directed Acyclic Graph, a commonly used data structure in the computer field. Due to its excellent characteristics brought by its unique topological structure, it is often used in various algorithm scenarios such as dynamic programming, finding the shortest path in navigation, and data compression) and the update of the calculation results, providing a basis for model training, or process effect evaluation, or data quality evaluation.

[0076] To better understand the technical solutions provided by the embodiments of the present application, some embodiments of the present application will be described in detail below.

[0077] The first aspect.

[0078] As Figure 1 shown, an embodiment of the present application provides a method for managing a machine learning model, including step S100 and step S200.

[0079] Step S100: Determine the dataset and model of the configuration information corresponding to the process node.

[0080] Wherein, the process node is any process node in the process, and the configuration information includes the execution conditions pointing to the dataset and the model.

[0081] In one embodiment, a full process (complete process, or the overall process) includes several process nodes, and each process node has different configuration requirements based on different application scenarios. For example: In new drug research, it is necessary to study the interaction between drug molecules and receptors, and also study the physical properties of the molecules themselves, such as solubility, toxicity, etc., which will make new drug research face the situation of using machine learning models in multiple different application scenarios. However, these models themselves vary greatly, with different training processes, output results, and output cycles. In a full process, there will also be cases where many models use the same dataset. Based on this, for long-cycle research fields, a suitable dataset is the key for the model to take effect; for the effective execution of process nodes, it is necessary to select a dataset and model that match the process node. Among them, the configuration information of the process node is used to determine which dataset and model are needed when the process node is executed.

[0082] Step S200: Establish an association relationship between the process node, the dataset, and the model.

[0083] Wherein, the association relationship can be used to indicate the dataset and the model that the process node determines to call when being executed.

[0084] In one embodiment, in order to prompt each process node to be associated with the dataset and the model to form a closed loop, and enable the tight association between data production and data use in an environment where multiple models share a dataset, improve the data use efficiency and effect, a DAG is constructed by taking the dataset, the model, and the process as the relationship main bodies, as the points in the DAG, and taking the respective versions of the dataset, the model, and the process as the connections, as the edges in the DAG.

[0085] In this way, it is possible to achieve connecting each model to the DAG through the process nodes of the full process.

[0086] Exemplarily, the relationship entities are points and their respective corresponding versions are edges, creating a record in the graph. In a completed workflow, there is one or more undirected graphs (Graph). There are multiple node-edge relationships in the graph. Let G represent the entire undirected graph, ds_mol_20210421 represent the dataset stored in a certain system, and wf_mol_property_0721 represent a certain stored model. Then G(ds_mol_20210421, wf_mol_property_0721) represents a set of relationships, and the corresponding versions are stored on the relationships. For example, G(ds_mol_20210421, wf_mol_property_0721) = (3.12_beta, 2.41_alpha), indicating that the versions of the dataset ds_mol_20210421 and the model wf_mol_property_0721 when establishing the connection are 3.12_beta and 2.41_alpha respectively.

[0087] It should be noted that establishing the association relationship between the process nodes, the dataset, and the model includes using a scheduling tool to create and change the process, for establishing the model relationships on different nodes in the process. Among them, the scheduling tool can be selected from luigi (based on the Python language, which can help establish a complex streaming batch task management system), airflow (an open-source scheduling tool written in Python by Airbnb), or celery (a distributed asynchronous message task queue developed based on Python, through which the asynchronous processing of tasks can be easily achieved).

[0088] As Figure 2 shown, in one embodiment, before step S100, the management method of the machine learning model further includes step S111 and step S112.

[0089] Step S111: Obtain the dataset.

[0090] Step S112: Store the dataset in the dataset library.

[0091] All the obtained datasets are structured data and can directly use mature file version management software.

[0092] Exemplarily, DVC (data version control) tools and git (an open-source distributed version control system) are usually used together to manage machine learning experiment data, i.e., data sets. When training each model, the training data / test data used may change. When reproducing experimental results, in addition to using the same code and config (computer system configuration), it is also very important to use the same data. DVC tools can easily store data on many storage systems, such as local disks, SSH (Secure Shell) servers, or cloud systems like aws S3 (amazon web service Simple Storage Service), GCP (Google Cloud Platform), etc. The data managed by DVC tools can be easily shared with other users using this storage system.

[0093] Using DVC tools and git can achieve file version management based on local / remote data warehouses. Then, add meta information to the versioned data set (meta information records the basic information of the data set, including creation time, creator, size, etc., and also contains some artificial tags, such as the number of features, the total number of entries, etc. Based on this meta information, the data set required for model training or process node execution can be quickly matched) and store it in a structured database for easy query when needed. Since DVC tools support both local file system storage and remote file object storage, they can meet the data management requirements in different scenarios. If the data set has encryption requirements, it can also be encrypted using methods such as the AES algorithm during file storage.

[0094] Above, it is possible to manage the data sets in the data set library, including version management, data compression and storage, data extraction, integrity verification, caching, etc.

[0095] As Figure 2 shown, in one embodiment, before step S100, the management method of the machine learning model further includes step S121 and step S122.

[0096] Step S121: Obtain a model.

[0097] Step S122: Store the model in the model library.

[0098] Among them, the model carries a model description, which refers to a textual description for the model to be trained and is used to determine the usage scenario of the model.

[0099] In different application scenarios, datasets are shared, and similarly, models are reused. To facilitate subsequent optimization and iteration of the model itself and the associated relationship of the process-dataset-model where the model is located, it is necessary to manage the model version (such as through a data warehouse + DVC tool + version meta). During the model version management process, the model is serialized. Model serialization refers to converting the model object existing in memory into a data stream / file object that can be stored in a persistent storage device through software engineering means. The specific implementation of model serialization depends on the way the model object exists. Taking the model trained by the python machine learning framework sklearn as an example: Through the built-in pickle module in python, the model can be stored as a pickle object and saved to a file. This file can be transferred to other hardware with the same environment and deserialized into a model object through the pickle module and used in the new environment. The process of verifying the training effect of model training needs to be completed through CI tools, such as the jenkins continuous integration tool.

[0100] Based on the above embodiments, by combining the DVC tool and git, version management of data, models, and code is carried out: easy to install (pip install DVC tool); convenient to use (DVC tool push; dev pull); fast speed. After the DVC tool adds a file, a new file will be generated. For example, when the DVC tool adds data.sql, it will generate data.sql.DVC tool (in kb level). Git will upload the data.sql.DVC tool file, and the DVC tool can pull the corresponding file (such as the data file saved remotely) according to the _dvc tool file. If you need to specify the versions of data, model, and code, just use git checkout version number and then DVC pull.

[0101] Connect the various process nodes of a full process and the machine learning models used in each stage through datasets, and manage them comprehensively to improve the data usage efficiency, optimize the model performance, and timely discover potential problems.

[0102] As Figure 2 shown, in one embodiment, after steps S112 and S122, the management method of the machine learning model further includes steps S131 and S132.

[0103] Step S131: Use the DVC tool to perform versioning on the datasets in the dataset library and the models in the model library.

[0104] Step S132: Establish the association relationship between a single dataset and all related models for the versioned dataset and model.

[0105] The datasets stored in the dataset library can be used for the training of the models in the model library and for being called when the process nodes are executed. Based on the situation that there may be many models using the same dataset in the whole process, it is necessary to establish the association relationship between a single dataset and all related models in advance to improve the usage efficiency of the dataset.

[0106] As Figure 3 shown, in one embodiment, after step S200, the management method of the machine learning model further includes step S300 and step S400.

[0107] Step S300: Update all models associated with the dataset when the dataset is updated.

[0108] Step S400: Update the association relationship where the updated dataset and / or any updated model is located according to the updated dataset and all updated models.

[0109] Based on the lack of engineering practices on model version management and iteration in the prior art, in one embodiment, for the training, validation and iteration of multiple models on a shared dataset: construct a process based on the application scenario, and integrate the models required for different application scenarios and at different stages into a DAG through the process. Taking the dataset update as the trigger condition, iterate and update the model itself and the association relationship of the process-dataset-model where the model is located. Specifically, taking the dataset update as the trigger condition, calculate the overall effect of the whole process. Through the changes in the node task results in the DAG, the impact of the dataset change on each stage can be analyzed and fed back, so as to provide directional guidance for stage optimization.

[0110] Please refer to Figure 4 .

[0111] As Figure 4 shown, in one embodiment, step S300 includes step S310, step S320, step S321 and step S322.

[0112] Step S310: Use the updated dataset as the input to train all models associated with the dataset.

[0113] It can be understood that when the dataset is updated, the multiple models mapped by it also need to be updated synchronously: train - iterate.

[0114] Step S320: Determine whether the training effect of the currently trained model meets the first preset condition.

[0115] In one embodiment, whether the model is trained well is set according to the actual application scenario and training purpose of each model, that is, the first preset condition.

[0116] Step S321: When satisfied, update the currently trained model.

[0117] Step S322: When not satisfied, do not perform the operation of updating the currently trained model.

[0118] In one embodiment, if the training effect of the currently trained model does not meet the first preset condition, terminate the current model update and output an update report indicating the reason for terminating the current model update.

[0119] As Figure 4-5 shown, in one embodiment, step S400 includes step S410, step S420, step S421, and step S422.

[0120] Step S410: Invoke the updated dataset and all updated models, and execute all process nodes associated with the updated dataset and / or any updated model.

[0121] In one embodiment, when the dataset is updated, the associated relationship where it is located also needs to be updated synchronously. Similarly, since all models associated with the dataset are updated due to the update of the dataset, the associated relationships where the updated models are located also need to be updated. It can be understood that among the associated relationships (process - dataset - model) that need to be updated, there are those where both the dataset and the model are updated, and those where only the model is updated.

[0122] Step S420: Determine whether the overall process effect meets the second preset condition after all process nodes are executed.

[0123] Among them, the overall process effect is represented by weighted summation of the execution results of each process node.

[0124] Exemplarily, in the process of studying the properties of small molecule drugs, as Figure 6 shown, the evaluation of the overall process effect is a final value obtained by adding the effects of each node in the DAG object and the empirical weight. For molecular property prediction, a goal can be preset as the prediction accuracy. Then for each step (step1 - step6), a proportion can be set in the goal, such as 10%, 20%, 20%, 25%, 15%, 10%. In this way, based on the training results and the validation set, an overall accuracy based on the current training set can be obtained. The setting of this proportion requires certain experience at the beginning, but as the data accumulates and the model effect improves, the proportion can be adjusted according to the final effect to achieve the expected overall effect.

[0125] Step S421: When satisfied, update the association relationship where the updated dataset and / or any updated model is located.

[0126] Step S422: When not satisfied, do not perform the operation of updating the association relationship where the updated dataset and / or any updated model is located.

[0127] Combined with the DVC tool for version management of datasets, models, and processes, by connecting datasets with models and constructing processes based on application scenarios, it is possible to improve the overall sensitivity of the process to data changes, facilitate the unified management of effective datasets, and trace the final quality changes brought about by changes at each node.

[0128] To further enable those skilled in the art to understand the technical solutions provided by the embodiments of the present application, the following takes the research process of small molecule drug properties as a specific embodiment of the present application for detailed description.

[0129] The drug R & D process is a long-cycle R & D process, which involves quite a number of small links. A property prediction process is established for key drug-like properties for the success rate of small molecule drug R & D, including solubility, potency, selectivity, solubility, bioavailability, permeability, toxicity, etc. These properties do not have a complete advantage or disadvantage in selection, but need to be comprehensively weighed. The datasets for their model training mainly come from the experimental accumulation of property determinations for various small molecule drug molecules.

[0130] First, manage data files through the DVC tool + git + aws S3. After configuring the aws S3 access permission of the DVC tool locally, the local data files can be uploaded to aws S3 through the DVC tool command line, and a definite version number can be obtained (the version number is generated by the DVC tool and obtained through the DVC tool push operation), with the default name in the form of name + md5. Here it is mol_property:87h5d41998yniinkfj348haj01ghp41urqp.

[0131] Using the same tools (DVC tool + git + aws S3), manage the first version of the model trained by data scientists or AI algorithm experts. Similarly, use the DVC tool command line to upload the model file to the pre-configured aws S3 bucket (the pre-configuration is mainly for the configuration of the DVC tool, and it is necessary to tell this tool the aws S3 address to be used. This aws S3 address needs to be applied for in advance and the access permission is set locally), and a series of versions encoded in the form of name + md5 are obtained.

[0132] The obtained version names can be simply recorded using text or Jupyter Notebook. However, for subsequent management and display, they are uniformly stored through database alignment. Here, the relational database postgresDB is selected for storage. A version table is created, and the corresponding version names are stored in the table as the primary key. For some datasets and models that require additional descriptions, they are stored in the relevant meta fields, such as the size, fields, source, maintainer, generation date, etc. of the dataset.

[0133] After completing the file version management, a graph database is needed to associate them. Here, ArangoDB provided by AWS S3 is used for relationship management. First, nodes are created for the stored version names, and the names and required attributes (for data management, to ensure data integrity, traceability, etc., generally including but not limited to: total file size, total number of files, MD5 code, uploader, maintainer, upload time, data description, main features, etc.) are inserted into the graph database to obtain a series of node objects. Each object has an _id (the _id is generated by ArangoDB to ensure the global uniqueness of each node) as the unique index. This _id can be filled back into the version table as one of the attributes for version management. After completing the node construction, based on the obtained node objects, edge relationships are constructed, that is, through graph management, the node objects are connected. Here, the connection relationship uses the version numbers obtained during version management to establish edge connections. Since there is a one-way dependency relationship between the dataset and the model, the current number of established edges is the relationship between a single dataset {m} and all related models {n}, which is represented by the set E{m,n}. The edge relationship set can be stored back in the version table or not. As long as there is one node _id, all the edge relationships and nodes connected to this node can be queried.

[0134] Next, the R & D process is constructed. The process of studying the properties of small molecule drugs can be divided into the following steps: activity, physicochemical properties, metabolic properties, toxicological properties, and drug-likeness. During the research process of these properties, a group of machine learning models with certain prediction effects have gradually been formed for properties such as drug activity screening, dissociation constant (pKa) prediction, octanol-water partition coefficient prediction, and drug toxicity screening.

[0135] Then, the processes in the R & D process are converted into scripts that can be recognized and executed by the process executor. When using Airflow to build a process, a new process corresponds to a DAG, and each process is put into an Operator (here, the BashOperator is used). Then, the entire DAG object contains 6 Operators. The first five Operators are executed independently, and the last comprehensive screening node combines the results of the first 5 nodes for prediction and needs to wait until all operations are completed before execution. Here, a DAG script file is constructed, and this DAG script file is also stored in the repository through file version management to obtain a version-related number.

[0136] After completing the conversion from the process to the DAG script file, the construction of the relationship between the process, data, and model is entered. The construction of the relationship between the process and the model has been reflected in the writing process of the DAG script file, mainly the model specifically executed in the Operator. Here, the corresponding version of the model is also written into the DAG script file as a variable, and the data is obtained by querying the version table for the current version. Thus, a new set of edge relationships is obtained, including the DAG (v) of the current version, the model Model (v) of the specified version, and the data set DataSet (v) of the current version, forming two sets of edge relationships {DAG (v), Model (v)} and {DAG (v), Dataset (v)}. Based on these two sets, it is possible to further establish the DAG update and calculation result update triggered when the data set is updated or the model is updated.

[0137] For the results of each DAG object run, a relationship of {Dataset, DAG, Model} -> {result} can also be established. The establishment of this relationship can provide, through process management, the result changes brought about by each data set change or model change, providing a basis for model training or data quality assessment.

[0138] Exemplarily, in the process of studying the properties of small molecule drugs, as Figure 6 shown, the evaluation of the overall effect of the process is a final value evaluated by the effects of each node in the DAG object plus empirical weights. For molecular property prediction, a goal can be preset as the prediction accuracy. Then, for each step (step1-step6), a proportion can be set in the goal, such as 10%, 20%, 20%, 25%, 15%, 10%. In this way, based on the training results and the validation set, an overall accuracy based on the current training set can be obtained. The setting of this proportion requires certain experience at the beginning, but as the data accumulates and the model effect improves, the proportion can be adjusted according to the final effect to achieve the expected overall effect.

[0139] In the above embodiments of the drug small molecule property research process, version management of datasets, models, and processes was carried out in combination with the DVC tool. By connecting datasets with models and constructing processes based on application scenarios, dataset updates can be fed back more quickly to the overall training process and overall quality. Through changes in the results of node tasks in the DAG, the impact of dataset changes on each stage can be analyzed and fed back, thereby providing directional guidance for stage optimization.

[0140] In the above embodiments, the management method of the machine learning model provides a solution for version management of datasets + models + processes in combination with the DVC tool.

[0141] To further enable those skilled in the art to understand the technical solutions provided by the embodiments of the present application, please refer to Figure 7 .

[0142] For version management of datasets + models + processes, in one embodiment, a system framework as shown in Figure 7 is built, including: a process management module, a data management module, a model management module, a relationship management module, as well as a dataset storage module, a model storage module, and an association relationship storage module.

[0143] The dataset management module is used to manage data files through the DVC tool + git + aws S3 and store the data files in the dataset storage module connected to the dataset management module.

[0144] Exemplarily, the dataset management module is used for the data management module to provide functions such as data storage in the database, data integrity verification, multi-version management of data, and data acquisition through version management of data.

[0145] The model management module is used to perform version management of models for the models trained by data scientists or AI algorithm experts using the same tools (DVC tool + git + aws S3) and store the module in the model storage module connected to the module management module.

[0146] Exemplarily, the model management module provides partial or complete model management functions in combination with the life cycle of the machine learning model, mainly including functions such as model storage, model version management, and model verification.

[0147] The process management module is used to construct a R & D process based on the application scenario to obtain processes applicable to various application scenarios, where the process management module is respectively connected to the dataset management module and the model management module.

[0148] Exemplarily, the process management module constructs the relationships between the models and data used in different stages of the complete R & D process for a long R & D cycle, including functions such as process construction, data set addition and maintenance, model addition and maintenance, and process result management.

[0149] After the relationship management module is used to complete the file version management, it is necessary to use a graph database to associate it. Here, ArangoDB provided by AWS S3 is used for relationship management.

[0150] Exemplarily, the relationship management module, on the basis of the process management module, uses a graph database or other methods to manage the association relationships between processes and data sets, processes and models, and data sets and models.

[0151] It can be understood that the relationship management module is respectively connected to the data set management module, the process management module, and the model management module, and is used to construct the association relationships between processes and data sets, processes and models, and data sets and models, and store the association relationships in the association relationship storage module connected to the relationship management module.

[0152] In the second aspect.

[0153] As Figure 8 shown, based on the same inventive concept, the present application provides a management device for a machine learning model, including a screening module 10 and an association module 20.

[0154] The screening module 10 is used to determine the data sets and models corresponding to the configuration information of the process nodes.

[0155] Wherein, the process node is any process node in the process, and the configuration information includes the execution conditions pointing to the data set and the model.

[0156] In one embodiment, a full process (complete process, or overall process) includes several process nodes, and each process node has different configuration requirements based on different application scenarios. For example: In new drug research, it is necessary to study the interaction between drug molecules and receptors, and also study the physical properties of the molecules themselves, such as solubility, toxicity, etc., which will make new drug research face the situation of using machine learning models in multiple different application scenarios. However, these models themselves are quite different, with different training processes and output results, and different output cycles. In a full process, there will also be a situation where many models use the same data set. Based on this, for long-cycle research fields, a suitable data set is the key for the model to take effect; for the effective execution of process nodes, it is necessary to select data sets and models that match the process nodes. Among them, the configuration information of the process node is used to determine which data sets and models are required when the process node is executed.

[0157] An association module 20 is used to establish an association relationship between the process node, the data set, and the model.

[0158] Among them, the association relationship can be used to indicate the data set and the model that the process node determines to call when being executed.

[0159] In one embodiment, in order to prompt each process node to be associated with the data set and the model to form a closed loop, and enable the close association between data production and data use in an environment where multiple models share a data set, improve the data use efficiency and effect, a DAG is constructed by taking the data set, the model, and the process as the relationship main bodies, which are used as the points in the DAG, and taking the respective corresponding versions of the data set, the model, and the process as the connections, which are used as the edges in the DAG.

[0160] In this way, it can be realized that each model is connected in series to the DAG through the process nodes of the entire process.

[0161] Exemplarily, the relationship main body is the point, and the respective corresponding versions are the edges. A record in the graph is established. In a completed workflow, there is one or more undirected graphs (Graphs). There are multiple node-edge relationships in the graph. Let G represent the entire undirected graph, ds_mol_20210421 represents the data set stored in a certain system, and wf_mol_property_0721 represents a certain stored model. Then G(ds_mol_20210421, wf_mol_property_0721) represents a set of relationships, and the corresponding versions are stored on the relationships. For example, G(ds_mol_20210421, wf_mol_property_0721) = (3.12_beta, 2.41_alpha), indicating that the versions of the data set ds_mol_20210421 and the model wf_mol_property_0721 when establishing the connection are 3.12_beta and 2.41_alpha respectively.

[0162] It should be noted that establishing the association relationship between the process node, the data set, and the model includes using a scheduling tool to create and change the process to establish the model relationship on different nodes in the process. Among them, the scheduling tool can be selected from luigi (based on the Python language, which can help establish a complex streaming batch processing task management system), airflow (an open-source scheduling tool written in Python by Airbnb), or celery (a distributed asynchronous message task queue developed based on Python, through which the asynchronous processing of tasks can be easily realized).

[0163] As Figure 9 shown, in one embodiment, the management device of the machine learning model further includes:

[0164] The acquisition module 01 is used to obtain a data set and store the data set in a data set library.

[0165] All the obtained data sets are structured data and can directly use mature file version management software.

[0166] Exemplarily, the DVC (data version control) tool and git (an open-source distributed version control system) are usually used together to manage machine learning experiment data, that is, data sets. When each model is being trained, the training data / test data used may change. When reproducing experimental results, in addition to using the same code and config ((computer system) configuration), using the same data is also very important. The DVC tool can easily store data on many storage systems, such as local disks, SSH (Secure Shell, a network protocol) servers, or cloud systems such as aws S3 (amazon web service Simple Storage Service), GCP (Google Cloud Platform), etc. The data managed by the DVC tool can be easily shared with other users using this storage system.

[0167] Using the DVC tool and git, file version management based on a local / remote data warehouse can be achieved. Then, the versioned data set is added with meta information (the meta information records the basic information of the data set, including creation time, creator, size, etc., and also contains some artificial tags, such as the number of features, total number of entries, etc. According to these meta information, the data set required for model training or process node execution can be quickly matched) and stored in a structured database for easy query when needed. Since the DVC tool supports both local file system storage and remote file object storage, it can meet the data management requirements in different scenarios. If the data set has an encryption requirement, it can also be encrypted using methods such as the AES algorithm during file storage.

[0168] The above can manage the data sets in the data set library, including version management, data compression and storage, data extraction, integrity verification, caching, etc.

[0169] As Figure 9 shown, in one embodiment, the acquisition module 01 is further used to obtain a model and store the model in a model library.

[0170] Among them, the model carries a model description for determining the usage scenario of the model.

[0171] Among them, the model carries a model description, which refers to a textual description for the model to be trained and is used to determine the usage scenario of the model.

[0172] In different application scenarios, there are cases where datasets are shared. Similarly, there are also cases where models are reused. To facilitate subsequent optimization and iteration of the model itself and the association relationship of the process-dataset-model where the model is located, version management of the model is required (such as management through a data warehouse + DVC tool + version meta). During the version management process of the model, the model is serialized. Model serialization refers to converting the model object existing in memory into a data stream / file object that can be stored in a persistent storage through software engineering means. The specific implementation of model serialization depends on the way the model object exists. Taking the model trained by the python machine learning framework sklearn as an example: through the built-in pickle module in python, the model can be stored as a pickle object and saved to a file. This file can be transferred to hardware in the same environment and deserialized into a model object through the pickle module and used in the new environment. And for the process of verifying the training effect of model training, it needs to be completed through a CI tool, such as the jenkins continuous integration tool.

[0173] Based on the above embodiments, by combining the DVC tool and git, version management of data, models, and code is performed: easy to install (pip install DVC tool); convenient to use (DVC tool push; dev pull); fast speed. After the DVC tool adds a file, a new file will be generated. For example, when the DVC tool adds data.sql, a data.sql.DVC tool file (kb level) will be generated. Git will upload this data.sql.DVC tool file, and the DVC tool can pull the corresponding file (such as the data file saved remotely) according to the _.dvc file. If you need to specify the versions of data, model, and code, just git checkout the version number and then DVC pull.

[0174] Connect the various process nodes of a full process and the machine learning models used in each stage through datasets for comprehensive management, so as to improve the usage efficiency of data, optimize the model performance, and timely discover potential problems.

[0175] Such as Figure 9 shown, in one embodiment, the management device of the machine learning model further includes:

[0176] The preprocessing module 02 is used to version the datasets in the dataset library and the models in the model library through DVC tools.

[0177] The preprocessing module 02 is also used to establish the association relationship between a single dataset and all related models for the datasets and models after versioning.

[0178] The datasets stored in the dataset library can be used for training the models in the model library and for being called when the process nodes are executed. Based on the situation that there may be many models using the same dataset in the whole process, it is necessary to establish the association relationship between a single dataset and all related models in advance to improve the usage efficiency of the dataset.

[0179] As Figure 10 shown, in one embodiment, the management device of the machine learning model further includes:

[0180] The update module 30 is used to update all models associated with the dataset when the dataset is updated.

[0181] The update module 30 is also used to update the association relationship where the updated dataset and / or any updated model are located according to the updated dataset and all updated models.

[0182] Based on the lack of engineering practices on model version management and iteration in the prior art, in one embodiment, for the training, validation and iteration of multiple models on a shared dataset: construct a process based on the application scenario, integrate the models required for different application scenarios and at different stages into a DAG through the process, and use the dataset update as the trigger condition to iteratively update the model itself and the association relationship of the process - dataset - model where the model is located. Specifically, taking the dataset update as the trigger condition, calculate the overall effect of the whole process. Through the changes in the node task results in the DAG, the impact of the dataset change on each stage can be feedback - analyzed, so as to provide directional guidance for stage optimization.

[0183] As Figure 11 shown, in one embodiment, the update module 30 includes:

[0184] The training module 31 is used to use the updated dataset as input to train all models associated with the dataset.

[0185] It can be understood that when the dataset is updated, the multiple models mapped to it also need to be updated synchronously: training - iteration.

[0186] A judgment module 32, configured to judge whether the training effect of the currently trained model meets a first preset condition; when it is met, prompt to update the currently trained model, otherwise do not perform the operation of updating the currently trained model.

[0187] Set whether the model is trained well according to the actual application scenarios and training purposes of each model, that is, the first preset condition. If the training effect of the currently trained model does not meet the first preset condition, terminate the current model update and output an update report indicating the reason for terminating the current model update.

[0188] As Figure 11 shown, in one embodiment, the update module further includes:

[0189] An execution module 33, configured to call the updated data set and all updated models, and execute all process nodes associated with the updated data set and / or any updated model.

[0190] When the data set is updated, the associated relationship where it is located also needs to be updated synchronously. Similarly, since all models associated with the data set are updated due to the update of the data set, the associated relationships where the updated models are located also need to be updated. It can be understood that among the associated relationships (process - data set - model) that need to be updated, there are those where both the data set and the model are updated, and those where only the model is updated.

[0191] The judgment module 32 is further configured to judge whether the overall process effect meets a second preset condition after all process nodes are executed; when it is met, prompt to update the associated relationships where the updated data set and / or any updated model are located, otherwise do not perform the operation of updating the associated relationships where the updated data set and / or any updated model are located.

[0192] Among them, the overall process effect is represented by weighted summation of the execution results of each process node.

[0193] Exemplarily, in the process of studying the properties of small molecule drugs, as Figure 6 shown, the evaluation of the overall process effect is a final value obtained by adding the effects of each node in the DAG object and the empirical weights. For molecular property prediction, a goal can be preset as the prediction accuracy rate. Then for each step (step1 - step6), a proportion can be set in the goal, such as 10%, 20%, 20%, 25%, 15%, 10%. In this way, according to the training results and the validation set, an overall accuracy rate based on the current training set can be obtained. The setting of this proportion requires certain experience at the beginning, but as the data accumulates and the model effect improves, the proportion can be adjusted according to the final effect to achieve the expected overall effect.

[0194] Combined with DVC tools for version management of datasets, models, and processes, by connecting datasets with models and constructing processes based on application scenarios, it is possible to improve the overall sensitivity of the process to data changes, facilitate the unified management of effective datasets, and trace the final quality changes brought about by changes at each node.

[0195] In a third aspect.

[0196] An embodiment of the present application provides a computer device, including: a processor, a memory, and a computer program stored on the memory. The processor is coupled to the memory, and when the processor is working, it executes the computer program to implement the management method of the machine learning model as described above.

[0197] In a fourth aspect.

[0198] An embodiment of the present application provides a computer-readable storage medium. The computer storage medium stores computer instructions. When the computer instructions are executed by a computer, the computer is caused to execute the instructions of the management method of the machine learning model as described above.

[0199] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center to another website, a computer, a server, or a data center in a wired manner (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center that includes one or more available media integrated. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as Digital Versatile Disc (DVD)), or semiconductor media (such as Solid State Disk (SSD)), etc.

[0200] The above are the embodiments provided by the present application, which are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A management method for a machine learning model, characterized in that Including: A data set and a model for determining the configuration information of a corresponding process node; wherein, the process node is any process node in the process, and the configuration information includes execution conditions pointing to the data set and the model, and the configuration information is used to determine the data set and the model required when the process node is executed; Establishing an association relationship between the process node, the data set, and the model; wherein, the association relationship can be used to indicate the data set and the model that the process node determines to call when executed; The establishing of the association relationship between the process node, the data set, and the model is achieved by taking the data set, the model, and the process as the main bodies of the relationship, serving as the nodes in the DAG, and taking the respective versions of the data set, the model, and the process as the connections, serving as the edges in the DAG, to construct a DAG; Wherein, before determining the data set and the model of the configuration information of the corresponding process node, it further includes: obtaining the data set; storing the data set in the data set library; And, obtaining the model; storing the model in the model library; wherein, the model carries a model description for determining the usage scenario of the model; Wherein, after storing the data set in the data set library and storing the model in the model library, it further includes: performing versioning processing on the data set in the data set library and the model in the model library through the DVC tool; establishing an association relationship between a single data set and all related models for the versioned data set and model.

2. The management method of the machine learning model according to claim 1, wherein, The establishing of the association relationship between the process node, the data set, and the model includes: using a scheduling tool to perform the creation and modification of the process and establish the model relationship on different nodes in the process; the scheduling tool is selected from luigi or celery.

3. The management method of the machine learning model according to any one of claims 1 or 2, characterized in that After establishing the association relationship between the process node, the data set, and the model, it further includes: When the data set is updated, updating all models associated with the data set; According to the updated data set and all updated models, updating the association relationship where the updated data set and / or any updated model is located.

4. The management method of the machine learning model according to claim 3, wherein The updating all models associated with the data set when the data set is updated includes: Taking the updated data set as the input and training all models associated with the data set; Judging whether the training effect of the currently trained model meets the first preset condition; When it is met, updating the currently trained model, otherwise not performing the operation of updating the currently trained model.

5. The management method of the machine learning model according to claim 4, wherein The updating the association relationship where the updated data set and / or any updated model is located according to the updated data set and all updated models includes: Invoking the updated data set and all updated models and executing all process nodes associated with the updated data set and / or any updated model; Judging whether the overall effect of the process meets the second preset condition after all process nodes are executed; When it is met, updating the association relationship where the updated data set and / or any updated model is located, otherwise not performing the operation of updating the association relationship where the updated data set and / or any updated model is located.

6. The management method of the machine learning model according to claim 5, wherein, The overall effect of the process is represented by weighted summation of the execution results of each process node.

7. A management device for a machine learning model, characterized in that, It includes: A screening module, configured to determine a data set and a model for the configuration information corresponding to a process node; wherein, the process node is any process node in the process, and the configuration information includes execution conditions pointing to the data set and the model, and the configuration information is used to determine the data set model required when the process node is executed; An association module, configured to establish an association relationship between the process node, the data set, and the model; wherein, the association relationship can be used to indicate the data set and the model that the process node determines to call when being executed; The establishment of the association relationship between the process node, the data set, and the model is achieved by taking the data set, the model, and the process as the main bodies of the relationship, serving as the nodes in the DAG, and taking the respective versions of the data set, the model, and the process as the connections, serving as the edges in the DAG, to construct a DAG; Wherein, the management device of the machine learning model further includes: a collection module, configured to obtain a data set and store the data set in a data set library; the collection module is further configured to obtain a model and store the model in a model library; wherein, the model carries a model description for determining the usage scenario of the model; A preprocessing module, configured to perform versioning processing on the data set in the data set library and the model in the model library through a DVC tool; the preprocessing module is further configured to establish an association relationship between a single data set and all related models for the versioned data set and model.

8. The management device of the machine learning model according to claim 7, characterized in that, The establishment of the association relationship between the process node, the data set, and the model includes: using a scheduling tool to create and change a process and establish a relationship with models on different nodes in the process; the scheduling tool is selected from luigi or celery.

9. The management device of the machine learning model according to claim 7 or 8, characterized in that It further includes: An update module, configured to update all models associated with the data set when the data set is updated; The update module is further configured to update the association relationship where the updated data set and / or any updated model is located according to the updated data set and all updated models.

10. The management device for a machine learning model according to claim 9, characterized in that, The update module includes: A training module, configured to use the updated data set as input to train all models associated with the data set; A judgment module, configured to judge whether the training effect of the currently trained model meets a first preset condition; when it is met, prompt to update the currently trained model, otherwise do not perform the operation of updating the currently trained model.

11. The management device of the machine learning model according to claim 10, characterized in that, The update module further includes: An execution module, configured to call the updated data set and all updated models to execute all process nodes associated with the updated data set and / or any updated model; The judgment module is further configured to judge whether the overall effect of the process meets a second preset condition after all process nodes are executed; when it is met, prompt to update the association relationship where the updated data set and / or any updated model is located, otherwise do not perform the operation of updating the association relationship where the updated data set and / or any updated model is located.

12. A computer device, characterized in that, It includes: A processor, a memory, and a computer program stored on the memory, the processor being coupled to the memory, and the processor, when operating, executing the computer program to implement the method for managing a machine learning model according to any one of claims 1-6.

13. A computer-readable storage medium, characterized in that, The computer storage medium stores computer instructions, which, when executed by a computer, cause the computer to execute the instructions for the method for managing a machine learning model according to any one of claims 1-6.

Images