A method and device for full-process traceability management

Abstract

The present invention discloses a method and device for full-process traceability management, and relates to the field of computer technology. A specific implementation of the method includes: dividing the full process into multiple sub-processes; performing version management on the job information corresponding to each sub-process; and implementing full-process traceability management by establishing a corresponding relationship between the sub-processes and according to the version information corresponding to the job information of each sub-process. This implementation method can conveniently implement full-process traceability management. When an abnormality occurs in the service, it can quickly trace and locate which link has a problem and warn and notify which relevant people, thereby reducing the burden on model construction and service maintenance personnel, and improving the efficiency and quality of model construction and service.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for full-process traceability management. Background Art

[0002] With the rapid development of modern information technology, human society has entered the era of big data, and big data is profoundly changing the direction of human social development. The core of big data is to utilize the value of data, and artificial intelligence and machine learning are key technologies for utilizing the value of data. In recent years, more and more companies and institutions have used artificial intelligence and machine learning technologies to extract business insights from big data, assist business decisions, promote modern business operations, solve many problems that are difficult to solve with traditional methods, and create huge value.

[0003] With the accumulation of big data, the improvement of machine learning algorithms and the introduction of new algorithms, the increasing computing power provided by advanced hardware resources, and the development of various basic frameworks and toolboxes for machine learning and deep learning, it has gradually become easier to develop and deploy a new complex machine learning system in a production environment. However, after several years of continuous accumulation of various machine learning models in real systems, users have found that maintaining them is very difficult and costly.

[0004] According to the CACE principle proposed by Google scientist D. Sculley and others in the article "Hidden Technical Debt in Machine Learning Systems": Changing Anything Changes Everything, it can be seen that each link in the machine learning process (for example, including: data input, data processing methods, feature engineering, model parameters, etc.) may generate a new data model due to the change of any parameter.

[0005] Therefore, it is necessary and important to conduct full-process traceability management from model construction to online service process and its various links in machine learning projects. At present, some products and systems have put forward some technical solutions from different angles, such as:

[0006] (1) Managed through Git. Git is a free, open source, distributed version control system that efficiently handles version management for projects ranging from very small to very large. It can track and view changes and evolutions of software systems during the project life cycle, and facilitate collaborative development by developers, so that each developer in the project can complete development locally and eventually merge the work of each developer together for management through the version control system. This tool is widely used in software development and deployment;

[0007] (2) Manage through Comet. Comet is a machine learning project version management solution that tracks and visually compares machine learning experiment history, code changes, and model performance. It allows data scientists and developers to easily monitor, compare, and optimize their machine learning models, and can be integrated with Git to support team collaboration.

[0008] (3) Manage through Huawei Cloud Model Arts. Model Arts is an AI full-process development platform and tool for AI developers released by Huawei Cloud at the HUAWEI CONNECT 2018 conference in October 2018. It provides full-process visual management for raw data, labeled data, training jobs, algorithm models, reasoning services, etc. throughout the entire life cycle of AI development.

[0009] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0010] (1) Git is an excellent tool for code version management, but it is not suitable for full-process traceability management of machine learning models, although code is a major component;

[0011] (2) Comet focuses on solving the problems of change history tracing, version comparison management, and collaborative development in the model test process, but it is unable to match the results of the model test with the model service in the production environment. Users are required to maintain the corresponding relationship in the description document. When the online model service is abnormal or the version changes, it is not possible to associate and trace it with the model test well. In addition, there is a lack of management of the runtime parameters of the model online service, which is not conducive to reducing the difficulty and cost of online model maintenance.

[0012] (3) Huawei Cloud Model Arts focuses on the management of modeling processes, computing software and hardware resources, and runtime parameter configuration, but lacks version management of the "code", the main component of machine learning projects. Users are required to distinguish different versions of the code by file names. In addition, model service is not an isolated process. The operation of online services often requires the triggering of upstream jobs, and the output will also serve as the input and startup trigger condition of downstream jobs. Moreover, the operation cycles of different model services are different. Model Arts does not yet support the management and monitoring of online model service dependency configurations, making it difficult to trace the impact of upstream changes on model services and the impact of model version changes on downstream. Summary of the invention

[0013] In view of this, an embodiment of the present invention provides a method and device for full-process traceability management, which can conveniently implement full-process traceability management. When an abnormality occurs in the service, the problem can be quickly traced to locate which link has the problem and warn and notify which relevant people, thereby reducing the burden on model construction and service maintenance personnel and improving the efficiency and quality of model construction and services.

[0014] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for full-process traceability management is provided.

[0015] A method for full-process traceability management includes: dividing the full process into multiple sub-processes; performing version management on the job information corresponding to each sub-process; establishing a corresponding relationship between the sub-processes, and realizing full-process traceability management according to the version information corresponding to the job information of each sub-process.

[0016] Optionally, dividing the entire process into a plurality of sub-processes includes: dividing the entire process once or multiple times to obtain one or more sub-processes at different levels.

[0017] Optionally, the full process is the full process of the machine learning model, and the multiple sub-processes obtained by division include: model training sub-process, model prediction sub-process and model online service sub-process; wherein, the model training sub-process includes four next-level sub-processes of raw data cleaning and processing, first feature engineering, first model training and model evaluation; the model prediction sub-process includes three next-level sub-processes of to-be-predicted data cleaning and processing, second feature engineering and first model prediction.

[0018] Optionally, it also includes: selecting the best model from the models trained by the model training sub-process for model online service in one of the following three ways: Way 1: the modeling personnel compare the different versions of the models trained by the model training sub-process to select the best model; Way 2: the best model is selected from the models according to preset selection rules, and the selection rules include: 1) regularly selecting a model with the best model performance evaluation index from the current many versions; 2) regularly selecting a model with the latest iteration whose model performance evaluation index is above a certain threshold from the current many versions; Way 3: according to the selection rules, multiple models with model performance evaluation index above a certain threshold are selected from the model, and then the modeling personnel compare the different versions of the selected models to select the best model.

[0019] Optionally, the model online service sub-process performs an indicator evaluation on the model performance, and when the indicator evaluation result does not meet the preset requirements, a model that meets the preset requirements is screened out from the models trained by the model training sub-process and replaces the model of the online service.

[0020] According to another aspect of an embodiment of the present invention, a device for full-process traceability management is provided.

[0021] A device for full-process traceability management, comprising: a process division module, used to divide the full process into multiple sub-processes; a version management module, used to perform version management on the job information corresponding to each of the sub-processes; a traceability management module, used to establish a corresponding relationship between the sub-processes and realize the traceability management of the full process according to the version information corresponding to the job information of each sub-process.

[0022] Optionally, the process division module is also used to divide the entire process once or multiple times to obtain one or more sub-processes at different levels.

[0023] Optionally, the full process is the full process of the machine learning model, and the multiple sub-processes obtained by division include: model training sub-process, model prediction sub-process and model online service sub-process; wherein, the model training sub-process includes four next-level sub-processes of raw data cleaning and processing, first feature engineering, first model training and model evaluation; the model prediction sub-process includes three next-level sub-processes of to-be-predicted data cleaning and processing, second feature engineering and first model prediction.

[0024] Optionally, it also includes a model screening module, which is used to: select the best model from the models trained by the model training sub-process for model online service in one of the following three ways: Method 1: the modeling personnel compare the different versions of the models trained by the model training sub-process to select the best model; Method 2: select the best model from the models according to preset selection rules, and the selection rules include: 1) regularly screening out a model with the best model performance evaluation index from the current many versions; 2) regularly screening out a model of the latest iteration whose model performance evaluation index is above a certain threshold from the current many versions; Method 3: according to the selection rules, multiple models whose model performance evaluation index is above a certain threshold are screened out from the model, and then the modeling personnel compare the different versions of the screened models to select the best model.

[0025] Optionally, the model online service sub-process performs an indicator evaluation on the model performance, and when the indicator evaluation result does not meet the preset requirements, a model that meets the preset requirements is screened out from the models trained by the model training sub-process and replaces the model of the online service.

[0026] According to another aspect of an embodiment of the present invention, an electronic device with full-process traceability management is provided.

[0027] An electronic device for full-process traceability management, comprising: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for full-process traceability management provided by an embodiment of the present invention.

[0028] According to still another aspect of the embodiments of the present invention, a computer readable medium is provided.

[0029] A computer-readable medium stores a computer program, which, when executed by a processor, implements the method for full-process traceability management provided by an embodiment of the present invention.

[0030] An embodiment of the above invention has the following advantages or beneficial effects: by dividing the whole process into multiple sub-processes; then performing version management on the job information corresponding to each sub-process; then, by establishing a corresponding relationship between the sub-processes, and realizing the traceability management of the whole process according to the version information corresponding to the job information of each sub-process, it is possible to conveniently trace which version of the prediction script and model file a certain version of the online model comes from, which upstream and downstream jobs it depends on and is depended on, which model training job the model file comes from, which version of the training script and which version of the training data the model training job comes from, and when an abnormality occurs in the model prediction service, the source can be traced to locate which link has the problem and warn which relevant people to notify, thereby reducing the burden on model construction and service maintenance personnel and improving the efficiency and quality of model construction and services.

[0031] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0033] Figure 1 Schematic diagram of main steps of the method for full-process traceability management according to an embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of the traceability management principle of the model training sub-process and the model prediction sub-process of an embodiment of the present invention;

[0035] Figure 3 It is a schematic diagram of the traceability table structure of the model training sub-process and the association relationship between the tables in one embodiment of the present invention;

[0036] Figure 4 It is a schematic diagram of comparing model version information of an embodiment of the present invention;

[0037] Figure 5It is a schematic diagram of the traceability table structure of the model prediction sub-process and the association relationship between the tables in one embodiment of the present invention;

[0038] Figure 6 It is a schematic diagram of the process of model screening by the model online service sub-process of an embodiment of the present invention;

[0039] Figure 7 Schematic diagram of main modules of a device for full-process traceability management according to an embodiment of the present invention;

[0040] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0041] Fig. 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0043] In order to solve the problems existing in the prior art, the present invention has conducted a detailed analysis and summary of the prior art and found that the maintenance of machine learning models is very difficult and costly mainly because:

[0044] (1) The construction and online service of machine learning models is a complex process of rapid experimentation and multiple explorations and studies. To build a model service with good performance indicators that can be used in a production environment, it is necessary to conduct multiple explorations and studies, comparative verifications, and design the next round of experiments based on the performance of previous experiments. However, there is currently a lack of methods and tools for the reasonable recording and management of these processes, making it difficult to compare experimental designs and reproduce model results;

[0045] (2) Model services in a real system often include data cleaning and preprocessing, feature engineering, model development and construction, model performance evaluation, model online services, and model updates and iterations. The process is long and involves many links. Each link is closely related. Changes in the upstream links will directly have a complex impact on the downstream links, making it difficult to trace the root cause.

[0046] (3) Due to the above characteristics of a complete machine learning model construction and online service process, relatively independent links in the process are generally split out and assigned to corresponding roles in the team, with multiple people collaborating to shorten the development cycle and launch services for related products as soon as possible. However, if the version correspondence and interface specifications of each link lack unified and scientific management, the communication cost between team members will increase, collaboration will be difficult, and even the availability of model output results will be affected;

[0047] (4) Due to product and business needs, models will be iteratively updated, and model services may have multiple versions in the production environment. Multi-version management of model services is more difficult than conventional code management. On the one hand, model services are different from traditional software. They not only contain code, but also large files such as model files and data that model services rely on, as well as their runtime software and hardware environment parameters and model parameters. On the other hand, online models in production environments may become abnormal due to various reasons. It is necessary to reproduce the results and trace the root causes conveniently and promptly, and switch to another version of the model service. When switching versions, it is necessary to have a comprehensive understanding of the performance indicators, runtime software and hardware environment parameters, and model parameters of the version to ensure that the version rollback is normal.

[0048] As for the three methods proposed in the prior art for full-process traceability management from model construction to online service process and its various links in machine learning projects, each method has defects to varying degrees, mainly in the following aspects:

[0049] Regarding (1) management through Git, Git is an excellent tool for code version management, but it is not very suitable for full-process traceability management of machine learning models, although code is a major component. The model building and online service process often involves important dependencies in multiple other links, such as upstream and downstream trigger jobs, software and hardware environment parameters for model operation, model runtime parameters and hyperparameters, and model performance, etc. Many of these parameters are configured by modelers during the training or online model process, and are not written in code or files. Each run comparison may have partial changes, making it difficult to manage versions through Git. If it is mandatory to write all configuration parameters to files, the maintenance cost of modelers will increase, and these model configurations are not convenient for intuitive visual management and comparison when they are versioned through Git, and are difficult to apply to the entire model development process;

[0050] For (2) management through Comet, Comet focuses on solving the problems of change history tracing, version comparison management, and collaborative development in the model test phase, but it is unable to match the results of the model test with the model service in the production environment. Users are required to maintain the corresponding relationship in the description document. When the online model service is abnormal or the version is changed, it is not possible to associate and trace it with the model test. In addition, there is a lack of management of the runtime parameters of the model online service, which is not conducive to reducing the difficulty and cost of online model maintenance.

[0051] Regarding (3) management through Huawei Cloud Model Arts, Huawei Cloud Model Arts focuses on the management of modeling processes, computing software and hardware resources, and runtime parameter configuration, but lacks version management of the "code", the main component of machine learning projects. Users are required to distinguish different versions of the code by file names. In addition, model service is not an isolated process. The operation of online services often requires the triggering of upstream jobs, and the output will also serve as the input and startup trigger condition of downstream jobs. Moreover, the operation cycles of different model services are different. Model Arts does not yet support the management and monitoring of online model service dependency configurations, making it difficult to trace and analyze the impact of upstream changes on model services and the scope of impact of changes in the model version itself on downstream.

[0052] In combination with the above analysis, the present invention provides a method and device for full-process traceability management, which can manage the entire process of machine learning models from model construction to online service. Specifically,

[0053] (1) In the process of machine learning model construction, data cleaning, feature engineering, model training, software and hardware environment parameters, runtime parameters and hyperparameters during model construction, model evaluation, model performance, and the data sets involved are managed through full-process version traceability and smooth collaboration of all links;

[0054] (2) In the machine learning model prediction phase, the whole process version traceability management and smooth collaboration of data cleaning, feature engineering, model prediction, software and hardware environment parameters, runtime parameters and hyperparameters during model prediction, and the data sets involved are carried out, and the model file version output by the model prediction phase and the previous model building phase, as well as the next step of the production environment model service configuration, traceability, collaboration, and multi-version model screening and optimization are carried out;

[0055] (3) Process traceability management of the operating software and hardware environment parameters, operating cycle, upstream and downstream dependent operations, model prediction result monitoring, and early warning message transmission of the production environment model service, as well as automatic or semi-automatic rollback of online models, so as to trace the root cause of model anomalies and analyze the scope of impact;

[0056] (4) The present invention aims to reduce the maintenance cost of modelers and make the complex processing process of the entire model construction and online service process clear, traceable, comparable and reproducible. By managing and visually comparing the versions of each step in the whole process, it is convenient for modelers to clearly understand which steps and adjustments have what impact on the model performance evaluation results, which can shorten the iteration cycle of a better model and facilitate the reproduction of the results of each version.

[0057] The specific implementation process of the embodiment of the present invention is described below with reference to the accompanying drawings.

[0058] Figure 1 Schematic diagram of the main steps of the method for full-process traceability management according to an embodiment of the present invention. Figure 1 As shown, the method for full-process traceability management of an embodiment of the present invention mainly includes the following steps S101 to S103.

[0059] Step S101: Divide the whole process into multiple sub-processes;

[0060] Step S102: performing version management on the job information corresponding to each sub-process;

[0061] Step S103: Establishing corresponding relationships between sub-processes and implementing traceability management of the entire process based on version information corresponding to the job information of each sub-process.

[0062] According to an embodiment of the present invention, when dividing the whole process into multiple sub-processes, step S101 may specifically be:

[0063] The entire process is divided once or multiple times to obtain one or more sub-processes at different levels.

[0064] According to one embodiment of the present invention, the full process is the full process of the machine learning model, and the multiple sub-processes obtained by division include:

[0065] Model training sub-process, model prediction sub-process and model online service sub-process;

[0066] Among them, the model training sub-process includes four next-level sub-processes: raw data cleaning and processing, first feature engineering, first model training, and model evaluation; the model prediction sub-process includes three next-level sub-processes: cleaning and processing of data to be predicted, second feature engineering, and first model prediction.

[0067] Figure 2 It is a schematic diagram of the traceability management principle of the model training subprocess and the model prediction subprocess of an embodiment of the present invention.

[0068] The model training sub-process generally includes four steps: raw data cleaning and processing, feature engineering, model training, and model evaluation (next-level sub-process). These four steps are closely related and will be iterated multiple times as the model is explored and verified. Therefore, the codes corresponding to the four steps of raw data cleaning and processing, feature engineering, model training, and model evaluation will have different versions. Different codes will also generate corresponding versions of the output data files, model files, and model evaluation performance indicators. Manage the multiple iterations of each step, provide tools for comparing different versions and tools for tracing the corresponding relationships between steps, so that modelers can analyze the differences between versions and the relationship between model evaluation results, shorten the cycle of iterating a better model, and reproduce the results of each version.

[0069] like Figure 2 The model training part (the upper half of the dotted line) is shown in the figure. In the process of model training, the data cleaning module, feature engineering module, model training module, and parameter configuration module will generate one or more versions (indicated by v1, v2, and v3 on the module) due to the need for exploration and iteration; accordingly, the output of each module also has a corresponding version (indicated by v1, v2, and v3 on the output file). The dotted rectangular box is the upstream and downstream operations of model training, which is not within the scope of consideration of the present invention. Git and DVC (Data Version Control) are used to manage the code (data cleaning code, feature engineering code, model training code, model evaluation code), data files (original data set, cleaned and processed data set, model training set) and model files in the model training process. The data version information (including code, data files, model files, etc.) that each step of the operation depends on is configured on the page, and the database (for example: relational database) records the corresponding relationship between each step, the version of the code or data selected for each step, the model training parameters and software and hardware environment parameters configured on the page, the model files trained by the model, and the performance indicators of the model evaluation. A database record corresponds to a model training job version. When one or more steps of the current version are modified, a new record will be generated in the database after saving, corresponding to a new version of the model training job.

[0070] Figure 3 Schematic diagram of the traceability table structure and the association relationship between tables of a model training sub-process according to an embodiment of the present invention. Figure 3 In the table, the field marked with "PK" is the primary key; the connection symbol Indicates a "one-to-one" relationship, connecting symbols Indicates a one-to-many relationship. Figure 3The detailed implementation process of tracing the model training sub-process is as follows: When the user configures a model training job on the model training page, fill in the model training job name and version number, select the model training feature set and label set and version number, select the model training code and version number from the Git repository, configure the model training parameters (including the training set / validation set split ratio, the number of model iteration training times, the batch data size, the learning rate and other operating parameters, etc.) and software and hardware environment parameters (including the computing engine, software image, hardware specifications, the number of distributed training nodes and other parameters), output model file path, model evaluation code and version number and model evaluation performance indicator value.

[0071] The information involved in the model training subprocess is divided into Figure 3The traceability table structure shown is stored in the database. When tracing the model training sub-process, it can be specifically carried out from the following four aspects. Taking Table 1-4 as the main table, (1) the model training feature table (uniquely identified by the "model training job ID") corresponding to a model training job (uniquely identified by the "model training job ID") can be associated with the "parent job ID" in Table 1-4 and the "feature engineering job ID" in Table 1-3 to obtain the feature engineering code Repo, Tag (label), person in charge and the upstream data (cleaned feature data set ID) corresponding to the model training feature table. Then, the "parent job ID" in Table 1-3 is associated with the "data cleaning job ID" in Table 1-2 to obtain the data cleaning code Repo, Tag, person in charge and the upstream data (original feature data set ID) corresponding to the cleaned feature data set. Then, the "parent job ID" in Table 1-2 is associated with the "upstream job ID" in Table 1-1 to obtain the relevant information of the original data set. (2) The model training label table (uniquely identified by "model training label table ID") corresponding to a model training job (uniquely identified by "model training job ID") can be associated with the "parent job ID" in Table 1-4 and the "label processing job ID" in Table 1-9 to obtain the label processing code Repo, Tag, person in charge and the upstream data (original label data set ID) that processed the label table corresponding to the model training label table, and then the "parent job ID" in Table 1-9 is associated with the "upstream job ID" in Table 1-8 to obtain the relevant information of the original label data set. (3) The running parameters (uniquely identified by "Runconfig ID") corresponding to a model training job (uniquely identified by "model training job ID") can be associated with the "Runconfig ID" in Table 1-4 and the "Runconfig ID" in Table 1-10 to obtain the model training parameters and software and hardware environment parameters corresponding to the model training job. (4) For each version of the model, the page configuration related information and the performance indicator related information of the model evaluation can be recorded in the form of Table 1-11, so that the model prediction link can select the optimal model for the online prediction service. Through the traceability mapping of the above four aspects of dependency and the record maintenance of related information, the entire upstream and downstream of the entire process are fully recorded in the database. It should be noted that in the above tables, although Table 1-1 and Table 1-8 store the same fields, the contents are different. Table 1-1 stores the dependency relationship between the data cleaning job and the parent job; Table 1-8 records the dependency relationship between the label processing job and the parent job.

[0072] The above technical solution saves the complete upstream and downstream configurations of each step of each version to the database. In both scenarios of model iterative exploration and regular automatic model update, the records of the current model version and the records of one or more historical model versions that the modeler wants to compare can be retrieved from the database and presented in the form of Table 1 (including Table 1-1 to Table 1-11), so that the modeler can easily compare versions and trace upstream and downstream.

[0073] Taking the model iteration exploration scenario as an example, assuming that data cleaning, feature processing, model training and runtime parameters are iteratively generated into m, n, p, q versions respectively, then the entire model training Pipeline (sub-process) will have m*n*p*q ​​versions. In the absence of the full-process traceability management solution proposed by the present invention, modelers need to maintain the correspondence between the steps of the m*n*p*q ​​versions, which is time-consuming and labor-intensive, and prone to errors. It is also very difficult to reproduce the results of a certain version. However, through the technical solution of the present invention, the number of versions that modelers need to maintain is greatly reduced, and it is not easy to make mistakes, making the reproduction and combination of versions simple and convenient.

[0074] Taking the scenario of regular automatic model updates as an example, after multiple model iterations and explorations, a model with excellent and stable performance and its corresponding model training pipeline can be selected. However, due to changes in data over time, the model needs to be retrained and evaluated regularly. In this scenario, the selected model training pipeline can be configured as a regular training job. Each training generates a model training job version and records a complete record according to the above database and table design. In the model prediction stage, a version can be selected from multiple versions for model prediction according to certain rules.

[0075] After the technical solution of the embodiment of the present invention is used to perform version traceability management on each step of the model training process, modelers can conveniently trace the corresponding relationship between different version comparison tools and each step. For example, modelers can compare multiple versions of the same model and perform multi-dimensional sorting, screening and filtering.

[0076] Figure 4 Schematic diagram of model version information comparison in one embodiment of the present invention. Figure 4 As shown in the figure, it shows the record form in which the modeler compares the specific information of different versions of the model. Different background colors in the table represent different comparison items. In actual implementation, different comparison items can be selected for comparison as needed. Modelers can compare multiple comparison items in four aspects: data cleaning, feature engineering, model training, and model service, so as to analyze the relationship between the differences between versions and the model evaluation results, and have a clearer and deeper understanding of the impact of a certain step modification on the model performance evaluation indicators.

[0077] like Figure 2 As shown in the model prediction part (the lower half of the dotted line), the model prediction stage generally includes three steps: data cleaning and processing, feature engineering, and model prediction. Compared with the model training stage, these three steps have relatively few changes, but the versions of each step and the corresponding relationship between the steps also need to be traced.

[0078] The principle of traceability management for the model prediction subprocess is similar to that of the model training subprocess, but it depends on the model file of the model training subprocess model. It is necessary to select an optimal model file from multiple versions for model prediction based on the performance indicators of the model training subprocess model evaluation. Git and DVC are used to manage the code (data cleaning code, feature engineering code, model prediction code), data files (original data set, cleaned data set, model prediction set) and model prediction results in the model prediction process. The data version information (including code, data files and model files, etc.) that each step of the operation depends on is configured on the page. The relational database records the corresponding relationship between each step, the version of the code or data selected for each step, the model prediction parameters and software and hardware environment parameters configured on the page, the model file trained by the model, and the model prediction result set. A database record corresponds to a model prediction job version. When one or more steps of the current version are modified, a new record will be generated in the database after saving, corresponding to a new version of the model prediction job.

[0079] Figure 5 FIG. 1 is a schematic diagram of the traceability table structure of the model prediction sub-process and the association relationship between the tables in one embodiment of the present invention. Figure 5 As shown, the information involved in the model prediction subprocess is as follows Figure 5The traceability table structure shown (Table 2-1 to Table 2-10) is stored in the database. When tracing the model prediction sub-process, it can be specifically carried out from the following four aspects. Taking Table 2-4 as the main table, (1) the model prediction feature table (uniquely identified by the "model prediction job ID") corresponding to a model prediction job (uniquely identified by the "model prediction job ID") can be associated with the "parent job ID" in Table 2-4 and the "feature engineering job ID" in Table 2-3 to obtain the feature engineering code Repo, Tag, person in charge and upstream data (cleaned feature data set ID) corresponding to the model prediction feature table. Then, the "parent job ID" in Table 2-3 is associated with the "data cleaning job ID" in Table 2-2 to obtain the data cleaning code Repo, Tag, person in charge and upstream data (original feature data set ID) corresponding to the cleaned feature data set. Then, the "parent job ID" in Table 2-2 is associated with the "upstream job ID" in Table 2-1 to obtain the relevant information of the original data set. (2) The running parameters (uniquely identified by "Runconfig ID") corresponding to a model prediction job (uniquely identified by "Model Prediction Job ID") can be associated with the "Runconfig ID" in Table 2-4 and the "Runconfig ID" in Table 1-8 to obtain the model prediction parameters and software and hardware environment parameters corresponding to the model prediction job. (3) The model version used by a model prediction job (uniquely identified by "Model Prediction Job ID") and the related information of the model version can be obtained by associating the "Model ID" in Table 2-4 with the "Model ID" in Table 2-9. Figure 2 The traceability path shown can trace back the entire process of model training for this version. (4) The model prediction results output by a model prediction job (uniquely identified by the "model prediction job ID") are recorded and stored in the form of Table 2-10. Through the traceability mapping of the above four aspects of dependency and the record maintenance of related information, the entire upstream and downstream of the entire process are fully recorded in the database.

[0080] Similar to the model training sub-process, models with the same prediction function have different versions, with different precision, model file sizes, corresponding data preprocessing scripts, and prediction script versions. Comparison is needed to decide which version to deploy as a certain type of online service (batch service, online service, or edge service). This is because different service forms have different requirements for models (precision, model file size, corresponding data preprocessing script, etc.).

[0081] Finally, in the model prediction phase, it is also necessary to select an optimal model for the online service subprocess for online service. According to an embodiment of the present invention, the optimal model is selected from the model trained by the model training subprocess for model online service in one of the following three ways:

[0082] Method 1: Modelers compare different versions of models trained by the model training sub-process and select the best model;

[0083] Method 2: Select the best model from the models through preset selection rules. The selection rules include: 1) regularly select a model with the best model performance evaluation index from the current many versions; 2) regularly select a model with the latest iteration whose model performance evaluation index is above a certain threshold from the current many versions;

[0084] Method 3: Combine method 1 with method 12. According to the selection rules, multiple models with model performance evaluation indicators above a certain threshold are selected from the model. The modeling personnel then compare different versions of the selected models to select the optimal model.

[0085] After the optimal model is selected, the selected optimal model can be deployed online for service. Figure 6 Schematic diagram of the process of model screening by the model online service sub-process of the embodiment of the present invention. Figure 6 As shown in the figure, the model online service is to deploy the process of the model prediction link as an online service, receive the data to be predicted generated by the upstream job as input, and output the prediction result through the model prediction pipeline as the input of the downstream job. For an online service, it is necessary to monitor the status and quality of the service. When an abnormality occurs, the error link and cause are quickly located and analyzed, and the relevant personnel are notified of an early warning. At the same time, according to the user-defined strategy, it is decided whether to adopt the "fuse trigger" mechanism for the downstream job, that is, when it is detected that the quality index related to the prediction result is lower than the set threshold, whether to continue to output to the downstream job. On the other hand, when it is detected that the quality index related to the prediction result is lower than the set threshold, the model screening module can automatically or semi-automatically select the backup model from Table 1-11 according to the configured strategy, replace the online abnormal model, and perform version rollback. That is: the model online service sub-process evaluates the index of model performance, and when the index evaluation result does not meet the preset requirements, the model that meets the preset requirements is selected from the models trained by the model training sub-process and replaces the model of the online service. Among them, when screening the model, it can be executed according to the above three methods of selecting the best model from the models trained by the model training sub-process.

[0086] Combine the following Figure 6This paper introduces the process of tracing the source of the model online service sub-process. For the convenience of explanation, the three steps of model prediction link, namely, data cleaning and processing, feature engineering, and model prediction, are abstracted into a model prediction pipeline. The version number is consistent with the model prediction job version number. The input of the model prediction pipeline comes from the output of the upstream job or the feature parameters passed in the model prediction request; the actual label processing process required for the model performance index evaluation is abstracted into a label processing pipeline. The model performance index evaluation model integrates the prediction results and the actual labels, and calculates the corresponding model performance evaluation indicators according to the model type. For example, the classification model indicators include accuracy, recall rate, F1 score (an indicator used in statistics to measure the accuracy of the binary classification model. It takes into account both the accuracy and recall rate of the classification model), etc. The regression model indicators include mean absolute error, mean square error, R square value, etc. The difference between the monitoring quality index and the set threshold determines whether to trigger the warning notification module and the model screening module.

[0087] Through the integration of the traceability management of the three links of the aforementioned model training sub-process, model prediction sub-process and model online service sub-process and the association between the three links, the present invention realizes the traceability management of each step of the whole process of machine learning model from model construction to online service and the comparison of parameter configuration between versions, that is, conveniently tracing which version of prediction script and model file a certain version of online model comes from, which upstream and downstream jobs it depends on and is depended on respectively, which model training job the model file comes from, which version of training script and which version of training data the model training job comes from, and when the model prediction service is abnormal, the traceability locates which link has problems and warns which relevant people. Applying the technical principles and solutions of full-process traceability provided by the present invention to model development platform products can reduce the burden on model construction and service maintenance personnel and improve the efficiency and quality of model construction and services.

[0088] Figure 7 Schematic diagram of the main modules of the device for full-process traceability management according to an embodiment of the present invention. Figure 7 As shown, the device 700 for full-process traceability management according to an embodiment of the present invention mainly includes a process division module 701 , a version management module 702 and a traceability management module 703 .

[0089] The process division module 701 is used to divide the whole process into multiple sub-processes;

[0090] A version management module 702 is used to perform version management on the job information corresponding to each sub-process;

[0091] The traceability management module 703 is used to establish a corresponding relationship between each sub-process and implement the traceability management of the whole process according to the version information corresponding to the job information of each sub-process.

[0092] According to an embodiment of the present invention, the process division module 701 may also be used for:

[0093] The entire process is divided once or multiple times to obtain one or more sub-processes at different levels.

[0094] According to another embodiment of the present invention, the full process is the full process of the machine learning model, and the multiple sub-processes obtained by division include:

[0095] Model training sub-process, model prediction sub-process and model online service sub-process;

[0096] Among them, the model training sub-process includes four next-level sub-processes: raw data cleaning and processing, first feature engineering, first model training, and model evaluation; the model prediction sub-process includes three next-level sub-processes: cleaning and processing of data to be predicted, second feature engineering, and first model prediction.

[0097] According to another embodiment of the present invention, the whole process traceability management device 700 may further include a model screening module (not shown in the figure), which is used to select the best model from the models trained by the model training sub-process for model online service in one of the following three ways:

[0098] Method 1: Modelers compare different versions of models trained by the model training sub-process and select the best model;

[0099] Method 2: Select the best model from the models through preset selection rules. The selection rules include: 1) regularly select a model with the best model performance evaluation index from the current many versions; 2) regularly select a model with the latest iteration whose model performance evaluation index is above a certain threshold from the current many versions;

[0100] Method 3: According to the selection rules, multiple models with model performance evaluation indicators above a certain threshold are selected from the models, and then the modeling personnel compare different versions of the selected models to select the optimal model.

[0101] According to another embodiment of the present invention, the model online service sub-process performs an indicator evaluation on the model performance, and when the indicator evaluation result does not meet the preset requirements, a model that meets the preset requirements is screened out from the models trained by the model training sub-process and replaces the model of the online service.

[0102] According to the technical solution of the embodiment of the present invention, the whole process is divided into multiple sub-processes; the job information corresponding to each sub-process is version managed; then, the corresponding relationship between the sub-processes is established, and the traceability management of the whole process is realized according to the version information corresponding to the job information of each sub-process. It is convenient to trace which version of the prediction script and model file a certain version of the online model comes from, which upstream and downstream jobs it depends on and is depended on, which model training job the model file comes from, which version of the training script and which version of the training data the model training job comes from, and when the model prediction service has an abnormality, the traceability can locate which link has the problem and warn which relevant people to notify, thereby reducing the burden of model construction and service maintenance personnel and improving the efficiency and quality of model construction and service.

[0103] Figure 8 An exemplary system architecture 800 is shown to which the method for full-process traceability management or the apparatus for full-process traceability management according to an embodiment of the present invention may be applied.

[0104] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. Network 804 is used to provide a medium for communication links between terminal devices 801, 802, 803 and server 805. Network 804 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0105] Users can use terminal devices 801, 802, and 803 to interact with server 805 through network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0106] The terminal devices 801 , 802 , and 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0107] The server 805 may be a server that provides various services, such as a backend management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 801, 802, and 803. The backend management server may analyze and process the received data such as product information query requests, and feed back the processing results (such as target push information, product information - only an example) to the terminal device.

[0108] It should be noted that the method for full-process traceability management provided in the embodiment of the present invention is generally executed by the server 805 , and accordingly, the device for full-process traceability management is generally set in the server 805 .

[0109] It should be understood that Figure 8 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0110] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of a terminal device or a server suitable for implementing an embodiment of the present invention. Fig. 9 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0111] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0112] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0113] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are executed.

[0114] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] The units or modules involved in the embodiments of the present invention may be implemented in software or in hardware. The units or modules described may also be provided in a processor. For example, they may be described as follows: a processor includes a process division module, a version management module, and a traceability management module. Among them, the names of these units or modules do not, in certain cases, constitute limitations on the units or modules themselves. For example, the traceability management module may also be described as "a module for dividing the entire process into multiple sub-processes."

[0117] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: dividing the whole process into multiple sub-processes; performing version management on the job information corresponding to each of the sub-processes; and achieving traceability management of the whole process by establishing a corresponding relationship between the sub-processes and according to the version information corresponding to the job information of each sub-process.

[0118] According to the technical solution of the embodiment of the present invention, the whole process is divided into multiple sub-processes; the job information corresponding to each sub-process is version managed; then, the corresponding relationship between the sub-processes is established, and the traceability management of the whole process is realized according to the version information corresponding to the job information of each sub-process. It is convenient to trace which version of the prediction script and model file a certain version of the online model comes from, which upstream and downstream jobs it depends on and is depended on, which model training job the model file comes from, which version of the training script and which version of the training data the model training job comes from, and when the model prediction service has an abnormality, the traceability can locate which link has the problem and warn which relevant people to notify, thereby reducing the burden of model construction and service maintenance personnel and improving the efficiency and quality of model construction and service.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for full-process traceability management, characterized in that: include: Divide the whole process into multiple sub-processes; Performing version management on the job information corresponding to each of the sub-processes; By establishing the corresponding relationship between each sub-process and realizing the traceability management of the whole process according to the version information corresponding to the job information of each sub-process; The fields in the traceability table of each sub-process are associated through connection symbols to establish a corresponding relationship between the sub-processes.

2. The method according to claim 1, characterized in that Dividing the whole process into multiple sub-processes includes: The entire process is divided once or multiple times to obtain one or more sub-processes at different levels.

3. The method according to claim 1 or 2, characterized in that: The full process is the full process of the machine learning model, and the multiple sub-processes obtained by division include: Model training sub-process, model prediction sub-process and model online service sub-process; Among them, the model training sub-process includes four next-level sub-processes: original data cleaning and processing, first feature engineering, first model training and model evaluation; the model prediction sub-process includes three next-level sub-processes: cleaning and processing of data to be predicted, second feature engineering and first model prediction.

4. The method according to claim 3, characterized in that Also includes: The best model is selected from the models trained by the model training subprocess for model online service in one of the following three ways: Method 1: The modeling personnel compare different versions of the models trained by the model training sub-process and select the optimal model; Method 2: Selecting the best model from the models by using a preset selection rule, wherein the selection rule includes: 1) regularly selecting a model with the best model performance evaluation index from the current many versions; 2) Regularly select the latest iteration of a model whose model performance evaluation index is above a certain threshold from the current many versions; Method three: According to the selection rule, multiple models with model performance evaluation indicators above a certain threshold are selected from the models, and then the modeling personnel compare different versions of the selected models to select the optimal model.

5. The method according to claim 3, characterized in that: The model online service sub-process performs an index evaluation on the model performance, and when the index evaluation result does not meet the preset requirements, a model that meets the preset requirements is screened out from the models trained by the model training sub-process and replaces the model of the online service.

6. A device for full-process traceability management, characterized in that: include: Process division module, used to divide the whole process into multiple sub-processes; A version management module, used for performing version management on the job information corresponding to each of the sub-processes; The traceability management module is used to establish a corresponding relationship between each sub-process and realize the traceability management of the whole process according to the version information corresponding to the job information of each sub-process; wherein, the fields in the traceability table of each sub-process are associated through connection symbols to establish the corresponding relationship between each sub-process.

7. The device according to claim 6, characterized in that The process division module is also used for: The entire process is divided once or multiple times to obtain one or more sub-processes at different levels.

8. The device according to claim 6 or 7, characterized in that The full process is the full process of the machine learning model, and the multiple sub-processes obtained by division include: Model training sub-process, model prediction sub-process and model online service sub-process; Among them, the model training sub-process includes four next-level sub-processes: original data cleaning and processing, first feature engineering, first model training and model evaluation; the model prediction sub-process includes three next-level sub-processes: cleaning and processing of data to be predicted, second feature engineering and first model prediction.

9. The device according to claim 8, characterized in that It also includes a model screening module, which is used to select the best model from the models trained by the model training sub-process for model online service in one of the following three ways: Method 1: The modeling personnel compare different versions of the models trained by the model training sub-process and select the optimal model; Method 2: Select the best model from the models by using a preset selection rule, wherein the selection rule includes: 1) regularly selecting a model with the best model performance evaluation index from the current many versions; 2) regularly selecting a model with the latest iteration whose model performance evaluation index is above a certain threshold from the current many versions; Method three: According to the selection rule, multiple models with model performance evaluation indicators above a certain threshold are selected from the models, and then the modeling personnel compare different versions of the selected models to select the optimal model.

10. The device according to claim 8, characterized in that The model online service sub-process performs an index evaluation on the model performance, and when the index evaluation result does not meet the preset requirements, a model that meets the preset requirements is screened out from the models trained by the model training sub-process and replaces the model of the online service.

11. An electronic device with full-process traceability management, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images