An efficient method and system for managing machine learning models
By automating the management of machine learning model configuration information, the problems of repetitive work, low efficiency, and poor reliability in traditional methods are solved, achieving efficient and reliable model management and updates, and meeting the needs of increasing model numbers.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING QUANYING TECH CO LTD
- Filing Date
- 2022-10-31
- Publication Date
- 2026-05-26
AI Technical Summary
The traditional machine learning model building process involves a lot of repetitive work, low manual efficiency, lack of standardized evaluation, and poor controllability of model updates, leading to frequent instances of model unavailability.
By receiving the model type and name input by the user, the system automatically determines the pre-set model configuration information, including the identifier ID, feature parameters, training duration, training algorithm, evaluation duration, and update cycle, thereby achieving automated management and evaluation of the model. The system uses a pre-set algorithm for training and evaluation, automatically selects the optimal model, and updates it periodically.
It enables efficient modeling in scenarios with a rapidly increasing number of models, reduces labor costs, improves the reliability and availability of model selection, reduces repetitive work, and enhances management capabilities.
Smart Images
Figure CN115688938B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to an efficient method and system for managing machine learning models. Background Technology
[0002] Traditional machine learning model building requires manual evaluation of models constructed with various inputs, parameters, and fitting methods, followed by selection of the best-performing model for deployment. However, this process involves a significant amount of repetitive work, resulting in extremely low manual efficiency.
[0003] Meanwhile, the lack of standards in evaluating models leads to low reliability; and the reliance on manual updates during model updates results in poor controllability, with instances of models becoming unusable during the update process. Summary of the Invention
[0004] (a) Technical problems to be solved
[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an efficient machine learning model management method and system, which solves the technical problem of a large amount of repetitive work and extremely low manual efficiency when building models.
[0006] (II) Technical Solution
[0007] To achieve the above objectives, the main technical solutions adopted by the present invention include:
[0008] In a first aspect, embodiments of the present invention provide an efficient machine learning model management method, comprising:
[0009] S1. Receive the model type and model name input by the user, and determine the corresponding pre-set model configuration information based on the model type and model name input by the user;
[0010] The model configuration information includes: a pre-defined model identifier ID corresponding to the model type and model name, pre-defined feature parameters, pre-defined training duration, pre-defined training algorithm, pre-defined evaluation duration, and pre-defined model update cycle;
[0011] S2. Based on the model configuration information, obtain the model corresponding to the model configuration information;
[0012] S3. Evaluate each model in the model group according to the preset evaluation mode and obtain the evaluation results;
[0013] The model group includes: the model corresponding to the configuration information and several other different versions of the model corresponding to the configuration information that are pre-stored.
[0014] S4. Store the models that meet the preset conditions from the evaluation results.
[0015] Preferably, S1 specifically includes:
[0016] S11. Obtain the model type and model name input by the user, and determine the pre-set model identifier ID corresponding to the model type and model name based on the model type and model name input by the user;
[0017] S12. Based on the model type, model name, and model identifier ID, determine the pre-set model configuration information corresponding to the model type, model name, and model identifier ID.
[0018] Preferably,
[0019] The pre-specified feature parameters are the input and output parameters in the pre-set training algorithm.
[0020] Preferably, S2 specifically includes:
[0021] S21. Based on the model configuration information, the feature parameters with a pre-specified training duration before the current time point from the pre-specified feature parameters stored in the database are used as training data.
[0022] S22. Use the training data to train the training algorithm preset in the model configuration information to obtain the training result; the preset training algorithm is a pre-specified basic model algorithm;
[0023] If the training result is a model, then the model is stored in the database;
[0024] This model corresponds to the model identifier ID in the configuration information;
[0025] The pre-specified basic model algorithm is one of the following: regression machine learning algorithm, classification machine learning algorithm, or ensemble algorithm.
[0026] Preferably, S3 specifically includes:
[0027] S31. If the training result is a model, then according to the model identifier ID corresponding to the model, obtain multiple versions of the model that are stored in advance and correspond to the model identifier ID.
[0028] S32. Evaluate the model and multiple versions of the model corresponding to the model identifier ID that are stored in advance, and obtain the evaluation result of each model.
[0029] Preferably, S32 specifically includes:
[0030] S321. Based on the preset evaluation duration in the model configuration information, use the feature parameters with the preset evaluation duration before the current time point from the preset specified feature parameters stored in the database as evaluation data.
[0031] S322. Based on the evaluation data, a pre-set evaluation algorithm is used to calculate the values of the quantitative indicators of the models based on the evaluation data to obtain the values of the quantitative indicators of each model in the model group and use them as the evaluation results.
[0032] Specifically, the model group includes: a model obtained by training the training data using a training algorithm pre-set in the model configuration information, and multiple different versions of the model pre-stored corresponding to the model identifier ID.
[0033] Preferably, S4 specifically includes:
[0034] Based on the evaluation results, determine the best and worst models in the model group, store the best model, and delete the worst model.
[0035] Among them, the model with the largest value of the quantitative index in the model group is the best model, and the model with the smallest value of the quantitative index in the model group is the worst model.
[0036] Preferably,
[0037] The quantitative indicators specifically include: accuracy, volatility, and fit.
[0038] The numerical values of the quantitative indicators are the sum of the values representing accuracy, volatility, and fit.
[0039] Preferably, the method further includes:
[0040] S5. Based on the preset model update cycle, S1-S4 will be executed automatically once at regular intervals.
[0041] On the other hand, this embodiment also provides a system for implementing the efficient machine learning model management method as described above, comprising:
[0042] The model management module is used to obtain the model type and model name input by the user, and determine the pre-set model identifier ID corresponding to the model type and model name based on the model type and model name input by the user;
[0043] The model configuration factory module is used to determine the pre-set model configuration information corresponding to the model type, model name, and model identifier ID based on the model type, model name, and model identifier ID.
[0044] The model configuration information includes: the pre-set model identifier ID, pre-specified feature parameters, pre-set training duration, pre-set training algorithm, pre-set evaluation duration, and pre-set model update cycle;
[0045] The model management module is also used to, based on the model configuration information, take the feature parameters with a pre-specified training duration before the current time point from the pre-specified feature parameters pre-stored in the database as training data; and call the task management module, which in turn calls the model training module to use the training data to train the training algorithm pre-set in the model configuration information to obtain the training result; the pre-set training algorithm is a pre-specified basic model algorithm; if the training result is a model, then the model is stored in the database;
[0046] The model management module is also used to retrieve multiple versions of the model corresponding to the model identifier ID from the database when the training result is a model;
[0047] The model management module is also used to take the feature parameters of the pre-specified evaluation time before the current time point from the pre-specified feature parameters stored in the database as evaluation data, based on the pre-set evaluation time in the model configuration information.
[0048] The model management module is also used to call the model evaluation module. The model evaluation module obtains the value of the quantitative index of each model in the model group based on the evaluation data using a pre-set evaluation algorithm for calculating the value of the quantitative index of the model based on the evaluation data.
[0049] The model group includes: a model obtained by training the training algorithm pre-set in the model configuration information using the training data, and multiple different versions of the model pre-stored corresponding to the model identifier ID;
[0050] The model management module calls the model selection module to select the model with the largest quantitative index value in the model group as the best model and the model with the smallest quantitative index value in the model group as the worst model.
[0051] The model management module is also used to set the pre-set model update cycle in the model configuration information to the task management module. The task management module automatically updates the model according to the pre-set model update cycle.
[0052] (III) Beneficial Effects
[0053] The beneficial effects of this invention are as follows: This invention provides an efficient machine learning model management method and system. By acquiring the model type and model name input by the user, determining a pre-set model identifier ID corresponding to the model type and model name based on the user-input model type and model name, further determining pre-set model configuration information corresponding to the model type, model name, and model identifier ID, and finally acquiring the model corresponding to the model configuration information based on the model configuration information, compared with the prior art, it can automatically acquire models according to the above steps, thereby meeting the modeling efficiency requirements in scenarios with a rapid increase in the number of models, while also reducing costs. Attached Figure Description
[0054] Figure 1 This is a flowchart of an efficient machine learning model management method according to the present invention;
[0055] Figure 2 This is an efficient machine learning model management system according to Embodiment 2 of the present invention. Detailed Implementation
[0056] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0057] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.
[0058] Example 1
[0059] See Figure 1 This embodiment provides an efficient machine learning model management method, including:
[0060] S1. Receive the model type and model name input by the user, and determine the corresponding pre-set model configuration information based on the model type and model name input by the user.
[0061] The model configuration information includes: a pre-defined model identifier ID corresponding to the model type and model name, pre-defined feature parameters, pre-defined training duration, pre-defined training algorithm, pre-defined evaluation duration, and pre-defined model update cycle.
[0062] S1 specifically includes: S11, obtaining the model type and model name input by the user, and determining a pre-set model identifier ID corresponding to the model type and model name based on the model type and model name input by the user.
[0063] S12. Based on the model type, model name, and model identifier ID, determine the pre-set model configuration information corresponding to the model type, model name, and model identifier ID.
[0064] The pre-specified feature parameters are the input and output parameters in the pre-set training algorithm.
[0065] S2. Based on the model configuration information, obtain the model corresponding to the model configuration information.
[0066] S2 specifically includes: S21, according to the model configuration information, using the pre-specified feature parameters of the pre-specified feature parameters stored in the database before the current time point as the training data.
[0067] S22. Use the training data to train the training algorithm preset in the model configuration information to obtain the training result; the preset training algorithm is a pre-specified basic model algorithm.
[0068] If the training result is a model, then the model is stored in the database; the model corresponds to the model identifier ID in the configuration information; the pre-specified basic model algorithm is one of the following: regression machine learning algorithm, classification machine learning algorithm, and ensemble algorithm.
[0069] S3. Evaluate each model in the model group according to the preset evaluation mode and obtain the evaluation results.
[0070] The model group includes: the model corresponding to the configuration information and several other different versions of the model corresponding to the configuration information that are pre-stored.
[0071] S3 specifically includes: S31, if the training result is a model, then according to the model identifier ID corresponding to the model, obtain multiple versions of the model that are stored in advance and correspond to the model identifier ID.
[0072] S32. Evaluate the model and multiple versions of the model corresponding to the model identifier ID that are stored in advance, and obtain the evaluation result of each model.
[0073] S32 specifically includes: S321, according to the preset evaluation duration in the model configuration information, using the feature parameters of the preset evaluation duration before the current time point from the preset specified feature parameters stored in the database as evaluation data.
[0074] S322. Based on the evaluation data, a pre-set evaluation algorithm is used to calculate the values of the quantitative indicators of the models based on the evaluation data to obtain the values of the quantitative indicators of each model in the model group and use them as the evaluation results.
[0075] The process of calculating the model's quantitative indicators based on the evaluation data is existing, and the evaluation algorithm that implements this process is pre-set.
[0076] Specifically, the model group includes: a model corresponding to the pre-set training algorithm and the training data, and multiple different versions of the model pre-stored corresponding to the model identifier ID.
[0077] S4. Store the models that meet the preset conditions from the evaluation results.
[0078] S4 specifically includes: based on the evaluation results, determining the best and worst models in the model group, storing the best model, and deleting the worst model.
[0079] Among them, the model with the largest value of the quantitative index in the model group is the best model, and the model with the smallest value of the quantitative index in the model group is the worst model.
[0080] The quantitative indicators specifically include: accuracy, volatility, and fit; the value of the quantitative indicator is the sum of the accuracy value, volatility value, and fit value.
[0081] The method further includes: S5, according to the preset model update cycle, automatically executing S1-S4 once at regular intervals.
[0082] This embodiment presents an efficient machine learning model management method. It obtains the user-inputted model type and name, determines a pre-defined model identifier ID corresponding to the model type and name, further determines pre-defined model configuration information corresponding to the model type, model name, and model identifier ID, and finally obtains the model corresponding to the model configuration information. Compared to existing technologies, this method can automatically acquire models by following the above steps, meeting the modeling efficiency requirements in scenarios with a rapidly increasing number of models, while also reducing costs.
[0083] Example 2
[0084] See Figure 2 This embodiment also provides a system for implementing the efficient machine learning model management method described above, comprising:
[0085] The module includes: model management, task management, model configuration factory, model training, model evaluation, model selection, message module, alarm management, and database.
[0086] The model management module is used to obtain the model type and model name input by the user, and determine the pre-set model identifier ID corresponding to the model type and model name based on the model type and model name input by the user.
[0087] In this embodiment, the model management module is used for model management scheduling, which enables the model management system to operate normally by scheduling other modules or services.
[0088] The model configuration factory module is used to determine the pre-set model configuration information corresponding to the model type, model name, and model identifier ID based on the model type, model name, and model identifier ID.
[0089] The model configuration information includes: the pre-set model identifier ID, pre-specified feature parameters, pre-set training duration, pre-set training algorithm, pre-set evaluation duration, and pre-set model update cycle.
[0090] The model management module is further configured to, based on the model configuration information, use the feature parameters with a pre-specified training duration prior to the current time point from the pre-specified feature parameters pre-stored in the database as training data; and call the task management module, which in turn calls the model training module to use the training data to train the training algorithm pre-set in the model configuration information to obtain the training result; the pre-set training algorithm is a pre-specified basic model algorithm; if the training result is a model corresponding to the pre-set training algorithm and the training data, then the model is stored in the database.
[0091] In this embodiment, if the training result is not a model corresponding to the pre-set training algorithm and the training data, the model training module calls the message module to send the message (the training result is not a model corresponding to the pre-set training algorithm and the training data) to the model management module. After receiving the information, the model management module calls the alarm module to notify the user of modeling failure. In this embodiment, the alarm module is used to issue a warning message.
[0092] In this embodiment, the model training module can be used to train the training algorithm preset in the model configuration information using the training data to obtain training results.
[0093] The model management module is also used to retrieve multiple versions of the model corresponding to the model identifier ID from the database when the training result is a model corresponding to the pre-set training algorithm and the training data.
[0094] The model management module is further configured to, based on the model configuration information, use the feature parameters with a pre-specified training duration prior to the current time point from the pre-specified feature parameters pre-stored in the database as training data; and call the task management module, which in turn calls the model training module to use the training data to train the training algorithm pre-set in the model configuration information to obtain the training result; the pre-set training algorithm is a pre-specified basic model algorithm; if the training result is a model corresponding to the pre-set training algorithm and the training data, then the model is stored in the database.
[0095] The model management module is also used to retrieve multiple versions of the model corresponding to the model identifier ID from the database when the training result is a model corresponding to the pre-set training algorithm and the training data.
[0096] The model management module is also used to take the pre-specified feature parameters from the pre-stored feature parameters in the database before the current time point as the evaluation data, based on the pre-set evaluation duration in the model configuration information.
[0097] The model management module is also used to call the model evaluation module. The model evaluation module obtains the value of the quantitative index of each model in the model group based on the evaluation data and uses a pre-set evaluation algorithm to calculate the value of the quantitative index of the model based on the evaluation data.
[0098] The model group includes: a model corresponding to the pre-set training algorithm and the training data, and multiple versions of the model pre-stored corresponding to the model identifier ID.
[0099] The model management module calls the model selection module to select the model with the largest quantitative index value in the model group as the best model and the model with the smallest quantitative index value in the model group as the worst model.
[0100] The model management module is also used to set the pre-set model update cycle in the model configuration information to the task management module. The task management module automatically updates the model according to the pre-set model update cycle.
[0101] The efficient machine learning model management system in this embodiment can meet the modeling efficiency requirements in scenarios with a rapidly increasing number of models, while reducing costs. Through the evaluation function of the model evaluation module in the model management system, a unified model evaluation standard is established, improving the reliability of model selection. The automated model update function of the model management system enables seamless updates, improving model usability. By uniformly managing all models through the model management system and automating modeling, evaluation, and updates, manual labor consumption in simple, repetitive tasks is reduced, enhancing the ability to manage models.
[0102] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0103] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.
[0104] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.
[0105] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0106] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0107] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.
Claims
1. An efficient machine learning model management method, characterized in that, include: S1. Receive the model type and model name input by the user, and determine the corresponding pre-set model configuration information based on the model type and model name input by the user; The model configuration information includes: a pre-defined model identifier ID corresponding to the model type and model name, pre-defined feature parameters, pre-defined training duration, pre-defined training algorithm, pre-defined evaluation duration, and pre-defined model update cycle; S2. Based on the model configuration information, obtain the model corresponding to the model configuration information; S3. Evaluate each model in the model group according to the preset evaluation mode and obtain the evaluation result; wherein, the model group includes: the model corresponding to the configuration information and multiple other different versions of the model corresponding to the configuration information that are pre-stored. S4. Store the models that meet the preset conditions from the evaluation results; The pre-specified feature parameters are: the input parameters and output parameters in the pre-set training algorithm; S2 specifically includes: S21, according to the model configuration information, using the feature parameters of the pre-specified training duration before the current time point from the pre-specified feature parameters pre-stored in the database as training data; S22, using the training data to train the training algorithm pre-set in the model configuration information to obtain the training result; the pre-set training algorithm is a pre-specified basic model algorithm. If the training result is a model, then the model is stored in the database; This model corresponds to the model identifier ID in the configuration information; The pre-specified basic model algorithm is one of the following: regression machine learning algorithm, classification machine learning algorithm, and ensemble algorithm; S3 specifically includes: S31. If the training result is a model, then according to the model identifier ID corresponding to the model, obtain multiple versions of the model that are stored in advance and correspond to the model identifier ID. S32. Evaluate the model and multiple pre-stored versions of the model corresponding to the model identifier ID, and obtain the evaluation result for each model; S32 specifically includes: S321. Based on the preset evaluation duration in the model configuration information, use the feature parameters with the preset evaluation duration before the current time point from the preset specified feature parameters stored in the database as evaluation data. S322. Based on the evaluation data, a pre-set evaluation algorithm is used to calculate the values of the quantitative indicators of the models based on the evaluation data to obtain the values of the quantitative indicators of each model in the model group and use them as the evaluation results. Specifically, the model group includes: a model obtained by training the training data using a training algorithm pre-set in the model configuration information, and multiple different versions of the model pre-stored corresponding to the model identifier ID.
2. The efficient machine learning model management method according to claim 1, characterized in that, S1 specifically includes: S11. Obtain the model type and model name input by the user, and determine the pre-set model identifier ID corresponding to the model type and model name based on the model type and model name input by the user; S12. Based on the model type, model name, and model identifier ID, determine the pre-set model configuration information corresponding to the model type, model name, and model identifier ID.
3. The efficient machine learning model management method according to claim 2, characterized in that, S4 specifically includes: Based on the evaluation results, determine the best and worst models in the model group, store the best model, and delete the worst model. Among them, the model with the largest value of the quantitative index in the model group is the best model, and the model with the smallest value of the quantitative index in the model group is the worst model.
4. The efficient machine learning model management method according to claim 3, characterized in that, The quantitative indicators specifically include: accuracy, volatility, and fit. The numerical values of the quantitative indicators are the sum of the values representing accuracy, volatility, and fit.
5. The efficient machine learning model management method according to claim 4, characterized in that, The method further includes: S5. Based on the preset model update cycle, S1-S4 will be executed automatically once at regular intervals.
6. A system for implementing the efficient machine learning model management method as described in claim 1, characterized in that, include: The model management module is used to obtain the model type and model name input by the user, and determine the pre-set model identifier ID corresponding to the model type and model name based on the model type and model name input by the user; The model configuration factory module is used to determine the pre-set model configuration information corresponding to the model type, model name, and model identifier ID based on the model type, model name, and model identifier ID. The model configuration information includes: the pre-set model identifier ID, pre-specified feature parameters, pre-set training duration, pre-set training algorithm, pre-set evaluation duration, and pre-set model update cycle; The model management module is also used to, based on the model configuration information, take the feature parameters with a pre-specified training duration before the current time point from the pre-specified feature parameters pre-stored in the database as training data; and call the task management module, which in turn calls the model training module to use the training data to train the training algorithm pre-set in the model configuration information to obtain the training result; the pre-set training algorithm is a pre-specified basic model algorithm; if the training result is a model, then the model is stored in the database; The model management module is also used to retrieve multiple versions of the model corresponding to the model identifier ID from the database when the training result is a model; The model management module is also used to take the feature parameters of the pre-specified evaluation time before the current time point from the pre-specified feature parameters stored in the database as evaluation data, based on the pre-specified evaluation time in the model configuration information. The model management module is also used to call the model evaluation module. The model evaluation module obtains the value of the quantitative index of each model in the model group based on the evaluation data using a pre-set evaluation algorithm for calculating the value of the quantitative index of the model based on the evaluation data. The model group includes: a model obtained by training the training algorithm pre-set in the model configuration information using the training data, and multiple different versions of the model pre-stored corresponding to the model identifier ID; The model management module calls the model selection module to select the model with the largest quantitative index value in the model group as the best model and the model with the smallest quantitative index value in the model group as the worst model. The model management module is also used to set the pre-set model update cycle in the model configuration information to the task management module. The task management module automatically updates the model according to the pre-set model update cycle.