A model updating method and device, electronic equipment, storage medium and product

Abstract

The application discloses a model updating method and device, electronic equipment, a storage medium and a product. The method obtains prediction data and verification data of a current model in an application process, determines an index sequence of the current model in the application process based on the prediction data and the verification data, performs degradation detection on the current model based on the index sequence, obtains a preset configuration file in the case that the current model meets a degradation condition, and updates the current model based on the preset configuration file to obtain an updated target model. The method realizes less manual intervention, improves the model iteration updating efficiency while reducing the labor cost, automatically triggers the iteration updating operation by using the degradation detection, further reduces the loss of benefits caused by the untimely model iteration updating, updates the current model based on the preset configuration file, realizes the measurement error caused by fewer objective factors, and ensures that the iterated model has better performance.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model update method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence, machine learning models have been widely used in many fields such as data analysis, credit approval, decision inference, and customer management, and various artificial intelligence models have emerged one after another.

[0003] As the number of models increases, how to use simpler, faster, and lower-cost model monitoring measures to monitor the performance of models and to iterate and update models with declining performance in a timely manner has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a model update method, apparatus, electronic device, storage medium, and product to achieve automatic iterative updates of the model.

[0005] According to one aspect of the present invention, a model update method is provided, comprising:

[0006] Obtain the prediction data and validation data of the current model during the application process, and determine the indicator sequence of the current model during the application process based on the prediction data and the validation data;

[0007] Decline detection is performed on the current model based on the aforementioned indicator sequence;

[0008] If the current model meets the decay condition, a preset configuration file is obtained, and the current model is updated based on the preset configuration file to obtain the updated target model.

[0009] Optionally, based on the predicted data and the validation data, a sequence of indicators for the current model during application is determined, including:

[0010] Based on the corresponding prediction data and verification data, the corresponding indicator data is determined, wherein the timestamp of the indicator data is the same as the timestamp of the prediction data;

[0011] Based on the timestamps and sequence lengths of the indicator data, each indicator data is filtered, and an indicator sequence is generated based on the filtered indicator data.

[0012] Optionally, the step of performing decay detection on the current model based on the indicator sequence includes:

[0013] The index sequence is input into a pre-trained decay detection model to obtain the decay parameters corresponding to the index sequence;

[0014] If the decay parameter is greater than the preset decay threshold, then the current model is determined to meet the decay condition.

[0015] Optionally, the preset configuration file includes one or more of the following: feature range, sample selection range, and iteration termination condition.

[0016] Optionally, updating the current model based on the preset configuration file to obtain the updated target model includes:

[0017] Target features are selected based on the aforementioned feature range;

[0018] Based on the aforementioned sample selection range, training samples are determined;

[0019] Based on the training samples and the target features, the current model is updated. During the update process, if the iteration termination condition is met, the iteration stops and the updated target model is obtained.

[0020] Optionally, the step of filtering target features based on the feature range includes:

[0021] Candidate features are determined within the specified feature range, and each candidate feature is evaluated to obtain evaluation parameters for each candidate feature.

[0022] The target features are selected based on the evaluation parameters of each candidate feature.

[0023] Optionally, the step of filtering the target features based on the evaluation parameters of each candidate feature includes:

[0024] The candidate features are ranked based on the evaluation parameters of each candidate feature;

[0025] Based on the feature data volume parameter, a target feature group is determined from the ranking of the candidate features, and the target feature group includes the target features corresponding to the feature data volume parameter.

[0026] Optionally, the feature data volume parameter can be multiple, and the obtained target feature group can be multiple;

[0027] Accordingly, based on the training samples and the target features, the current model is updated. During the update process, if the iteration termination condition is met, the iteration stops, and the updated target model is obtained, including:

[0028] Based on the training samples and each target feature group, the current model is updated to obtain a candidate model.

[0029] If at least one of the candidate models satisfies the iteration termination condition, the iteration stops, and the target model is determined based on the candidate models that satisfy the iteration termination condition.

[0030] Optionally, the iteration termination condition includes the updated target model's model metrics being better than the current model's model metrics before the update.

[0031] Optionally, determining the target model based on the candidate models that satisfy the iteration termination condition includes:

[0032] Determine the model metrics of each candidate model for a preset type, and determine the comprehensive metrics based on the model metrics;

[0033] The model indicators and the comprehensive indicators are evaluated based on multiple evaluation models to obtain the evaluation data of each evaluation model for the candidate model;

[0034] The target evaluation result of the candidate model is determined based on the evaluation data of each of the evaluation models, and the target model is determined based on the target evaluation result of each of the candidate models.

[0035] Optionally, determining the comprehensive index based on the model index includes:

[0036] Obtain at least some of the model indicators of the preset type, perform preset processing on the at least some model indicators, and obtain the comprehensive indicator.

[0037] According to one aspect of the present invention, a model updating apparatus is provided, comprising:

[0038] The indicator sequence determination module is used to acquire the prediction data and validation data of the current model during the application process, and determine the indicator sequence of the current model during the application process based on the prediction data and the validation data.

[0039] A decay detection module is used to perform decay detection on the current model based on the indicator sequence;

[0040] The target model update module is used to obtain a preset configuration file when the current model meets the decay condition, and update the current model based on the preset configuration file to obtain the updated target model.

[0041] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0042] At least one processor; and

[0043] A memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the model update method according to any embodiment of the present invention.

[0045] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the model update method described in any embodiment of the present invention.

[0046] According to another aspect of the present invention, a computer program product is provided, characterized in that the computer program product includes a computer program that, when executed by a processor, implements the model update method described in any embodiment of the present invention.

[0047] The technical solution of this invention obtains the prediction and validation data of the current model during application, and determines the indicator sequence of the current model during application based on the prediction and validation data; performs decay detection on the current model based on the indicator sequence; when the current model meets the decay conditions, obtains a preset configuration file, and updates the current model based on the preset configuration file to obtain the updated target model. This approach minimizes manual intervention, improving the efficiency of model iteration and reducing labor costs. The use of decay detection to automatically trigger iteration and update operations further reduces the loss of benefits caused by untimely model iteration and updates based on the preset configuration file, minimizing measurement errors caused by objective factors and ensuring better performance of the iterated model.

[0048] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of a model update method provided in Embodiment 1 of the present invention;

[0051] Figure 2 This is a graph showing the ROC evaluation results of a model provided in Embodiment 1 of the present invention;

[0052] Figure 3 This is a flowchart of a model update method provided in Embodiment 1 of the present invention;

[0053] Figure 4 This is a schematic diagram of the structure of a model update device provided in Embodiment 2 of the present invention;

[0054] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0055] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0056] It should be noted that the terms "first feature data," "second feature data," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0057] The acquisition, storage, and / or processing of data in the technical solutions involved in this application comply with the relevant provisions of national laws and regulations.

[0058] Example 1

[0059] Figure 1 This is a flowchart of a model update method provided in Embodiment 1 of the present invention. This embodiment is applicable to the automatic iterative update of models. The method can be executed by a model update device, which can be implemented in hardware and / or software. This model update device can be configured in the electronic device provided in the embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] S110. Obtain the prediction data and validation data of the current model during the application process, and determine the indicator sequence of the current model during the application process based on the prediction data and validation data.

[0061] The current model can be an artificial intelligence model that has been trained and put into use using a preset method. This current model can be any type of machine learning model, including but not limited to logistic regression models and neural network models. The application process can be the actual use of an artificial intelligence model trained using a preset method, such as acquiring input information, feeding the input information into the current model, and obtaining the current model's prediction data. Furthermore, the processing object of the current model and the type of prediction data are not limited. Optionally, the processing object of the current model can be images, videos, audio, text, etc., and the corresponding current model can be, but is not limited to, image classification models, image segmentation models, image feature extraction models, image compression models, image enhancement models, image denoising models, image label generation models, text classification models, text translation models, text summarization extraction models, text prediction models, keyword conversion models, text semantic analysis models, speech recognition models, audio denoising models, audio synthesis models, audio equalizer conversion models, weather prediction models, product recommendation models, article recommendation networks, action recognition models, face recognition models, facial expression recognition models, and other machine learning models. Accordingly, any of the above-mentioned current models are trained using corresponding sample data.

[0062] Predictive data can be the output result obtained by the current model through predictive processing of input data, while validation data can be the actual data of the predicted object. Taking the current model as a product recommendation model as an example, the predictive data can be the product data recommended to users based on the product recommendation model, while the validation data can be the product data selected or transacted by users. Taking the current model as a sales forecast module as an example, the predictive data can be the predicted sales data obtained by the sales forecast model for a future time period, while the validation data can be the actual sales data within the future time period.

[0063] In this embodiment, multiple indicators that characterize the quality of the current model can be obtained from the prediction data and validation data that match the current model during application. The indicator data corresponding to the prediction data and validation data corresponding to different timestamps can be obtained. For any indicator, the indicator data corresponding to each timestamp are integrated to obtain an indicator sequence sorted over time. This indicator sequence can be a time series dataset.

[0064] Optionally, corresponding indicator data can be determined based on the corresponding prediction data and validation data. Based on the timestamps and sequence lengths of the indicator data, each indicator data can be filtered, and an indicator sequence can be generated based on the filtered indicator data.

[0065] The metrics data can include, but are not limited to, AUC, KS, GINI, AR, PSI, precision, accuracy, recall, F1 score, and false positive rate. A timestamp can be a complete and verifiable record indicating that a piece of data existed at a specific point in time. A timestamp can provide users with electronic evidence to prove when certain data was generated. Correspondingly, the timestamps of the metrics data are the same as those of the predicted data. Sequence length can be data characterizing the length of the sequence's time span, such as one week or one month.

[0066] Optionally, the indicator sequence can be updated based on the sequence length, where the sequence length can be the time length or the amount of indicator data in the sequence (e.g., the sequence length could be the length of 300 indicator data points). For example, if the sequence length is one week, and the prediction and validation data used in the current model application are from day 8, then the data from day 1 would be deleted, and the data from day 8 would be placed at the end of the indicator sequence, thus updating the indicator sequence. The indicator data are then sorted based on their corresponding timestamps, and the indicator data whose timestamps are closest to the current time and satisfy the sequence length requirement are selected to form the indicator sequence.

[0067] For each indicator, a separate indicator sequence can be generated; correspondingly, there can be multiple indicator sequences. Establishing indicator sequences by filtering indicator data facilitates subsequent model testing using these sequences.

[0068] S120. Perform decay detection on the current model based on the indicator sequence.

[0069] Among them, decay detection can be a process of detection based on a model trained by a preset algorithm.

[0070] Optionally, the decay detection process can be as follows: input the indicator sequence into a pre-trained decay detection model to obtain the decay parameters corresponding to the indicator sequence; if the decay parameters are greater than a preset decay threshold, then the current model is determined to meet the decay conditions.

[0071] The decay detection model can be a model trained using a pre-defined deep learning algorithm, such as an LSTM (Long Short-Term Memory) model. The decay parameter can be a value between 0 and 1, representing the degree of model decay, and can be obtained through evaluation using the decay detection model. See details... Figure 2 , Figure 2It can be a graph of the model's ROC evaluation results, where the horizontal axis is the false positive rate (FPR), which is the proportion of the corresponding decay parameter that is less than the preset decay threshold, and the vertical axis is the true positive rate (TPR), which is the proportion of the corresponding decay parameter that is greater than the preset decay threshold.

[0072] It should be noted that, assuming the recession threshold is 0.6, those with a probability greater than or equal to 0.6 are classified as positive, and those less than 0.6 as negative. A corresponding (FPR, TPR) pair can be calculated, yielding the corresponding coordinates on a plane. Ideally, TPR should be close to 1, and FPR should be close to 0.

[0073] For example, firstly, an indicator sequence is constructed based on prediction and validation data. For different types of indicator data, since the data ranges of different indicator data are inconsistent, StandardScaler can be used for scaling. An LSTM model is constructed, and the n_past parameter is set to 30 (n_past represents the number of past steps to predict the next target value). The LSTM neural model is trained, and the decay parameters are adjusted using GridSearchCV to obtain a decay detection model. Based on the model performance during training, the decay parameters of the decay detection model are optimized. The decay detection model with the optimal decay parameters is used to detect the decay status of the current model. Here, the decay detection model can be used to perform decay detection on different indicator sequences separately, obtaining multiple decay detection results. Correspondingly, the degree of decay of the current model can be determined by combining multiple decay detection results. For example, if the decay detection result corresponding to any indicator meets the decay condition, it can be determined that the current model meets the decay condition; for example, multiple decay detection results can be weighted using the weights corresponding to multiple indicators to obtain comprehensive decay data, and it can be determined whether the comprehensive decay data meets the decay condition.

[0074] By predicting the model's decline risk through a pre-trained decline detection model, a triggering basis is provided for whether to proceed with the next model iteration, achieving objective prediction of decline risk, having a certain degree of foresight, making the assessment more accurate, and triggering iterations more timely.

[0075] S130. If the current model meets the decay condition, obtain the preset configuration file, update the current model based on the preset configuration file, and obtain the updated target model.

[0076] This process involves pre-initializing the current model's parameters to obtain a preset configuration file. This configuration includes settings such as the feature range, sample selection range, and iteration termination conditions, which are not specifically limited here. The preset configuration file is stored and invoked when the current model meets the decay conditions. The preset configuration file can be set up by displaying a configuration page on the interface, receiving configuration information based on user actions on the page, and then generating the preset configuration file. This simple parameter initialization allows for automatic model iteration, making it easy to operate, highly usable, and requiring minimal modeling skills from the user.

[0077] The preset configuration file may include one or more of the following: feature range, sample selection range, and iteration termination condition. Optionally, the feature range can be a selectable set of features, such as a feature wide table name. The sample selection range may include the time range for selecting sample data and / or the dataset range of the sample data.

[0078] The update process for the current model can be iterative based on preset rules to obtain a more optimized target model. The target model can be an iterative model that meets the iteration termination condition. Optionally, the update process for the current model can be as follows: filter target features based on feature range, determine training samples based on sample selection range, update the current model based on training samples and target features, and stop the iteration if the iteration termination condition is met during the update process, thus obtaining the updated target model.

[0079] The target features can be randomly obtained using a random algorithm; for example, several features can be randomly selected from a feature range as target features. Feature filtering can be a process of removing bad features through a preset method. Correspondingly, the target features can be good features within the feature range. The good and bad features here can be determined through preset evaluation methods.

[0080] The sample selection range can be a preset time span. Training samples can be determined from the historical data of the current model. Correspondingly, the preset samples can include historical data within the preset time span. For example, based on the sample selection range, historical data within a time span N before the current moment can be selected as sample data. The time information of each historical data point is then divided into 30% of the total sample data to form a validation set, and the remaining 70% is further divided into 30% to form a test set. The remaining sample data forms the training set. Updating the current model can be a process where, after determining that the current model meets the decay condition, the target features are extracted from the training samples in the determined training, test, and validation sets, and the current model is automatically iterated and trained. The iteration termination condition can include the updated target model's metrics being better than the previous model's metrics.

[0081] The model is iteratively updated by using a preset configuration file to ensure the excellence of the current model's metrics during application, reduce reliance on human intervention, and optimize the performance of the updated target model based on the set iteration termination conditions.

[0082] Optionally, candidate features are determined within the feature range, each candidate feature is evaluated to obtain evaluation parameters for each candidate feature, and target features are selected based on the evaluation parameters of each candidate feature.

[0083] Candidate features can be data after initial feature screening using a random algorithm, for example, randomly selecting 50% of the features within a feature range as candidate features. The evaluation parameter can be the IV value of the candidate features.

[0084] By preprocessing the data within the feature range to form candidate features and evaluating the candidate features to obtain their evaluation parameters, target features are selected based on the evaluation parameters, thereby further improving the accuracy and quality of the target features.

[0085] Optionally, based on the evaluation parameters of each candidate feature, the candidate features are sorted, and the target feature group is determined from the sorted candidate features according to the feature data volume parameter. The target feature group includes the target features corresponding to the feature data volume parameter.

[0086] The ranking can be based on prioritizing candidate features according to evaluation parameters, such as arranging candidate features from largest to smallest based on their IV values. The feature data volume parameter can be a parameter representing the size or proportion of the feature data. The target feature group can be a set of target features selected from the candidate features.

[0087] For example, the top 50% of candidate features (i.e., the IV values) are evaluated to form a target feature group, i.e., the feature data volume parameter is 50%. In some embodiments, the feature data volume parameter can be 60%, 70%, 80%, 90%, or 100%, without specific limitation here. Optionally, there can be multiple feature data volume parameters, and multiple target feature groups are obtained.

[0088] By sorting and grouping candidate features, target features are obtained based on the feature data volume parameter to select target features that are effective for model optimization. Furthermore, different numbers of target features are selected from the candidate features to form multiple target feature groups. The current model can be updated simultaneously based on different numbers of target features to obtain different updated models. The optimal target features have been selected. In one round of updates, multiple updated models are trained, simplifying the update process and shortening the update time.

[0089] In this embodiment, the feature data volume parameter and the number of target feature groups can be set according to actual needs, and are not specifically limited here. The feature data volume parameter and the number of target feature groups are the same.

[0090] The process of evaluating candidate features can be as follows: The candidate features are binned to obtain multiple groups of candidate features, for example, N groups of candidate features (see Table 1). Table 1 can be the binning results of the candidate features, i.e., the evaluation results.

[0091] Table 1

[0092]

[0093] Where "good" represents the number of good samples in the model, and "bad" represents the number of bad samples in the model. The proportion of good samples and the proportion of bad samples in each group are calculated, and the formula for calculating the WOE value of each group is as follows:

[0094]

[0095] Among them, G i B represents the number of good samples in the i-th group (Group i). i Let G represent the number of bad samples in the i-th group (Group i), G represent the total number of good samples, and B represent the total number of bad samples.

[0096] Based on the WOE value, the grouped IV value for each candidate feature can be calculated:

[0097]

[0098] The IV value of a feature is the sum of the IV values ​​for each group:

[0099]

[0100] Features with an IV value less than or equal to 0.02 are discarded according to the following criteria. See Table 2, which shows the correspondence between the IV values ​​of candidate features and their variable discriminant power.

[0101] Table 2

[0102] IV value Variable Discrimination <＝0.02 The variable has no predictive power and is unusable. 0.02-0.1 Weak predictability 0.1-0.2 It has a certain predictive power >0.2 High predictability High IV There may be potential risks

[0103] Here, the evaluation parameters, i.e., IV values, of each candidate feature are calculated to sort and filter the candidate features, resulting in at least one target feature group for model updating. Accordingly, the current model is updated based on the training samples and each target feature group, resulting in a candidate model. If at least one of the candidate models satisfies the iteration termination condition, the iteration stops, and the target model is determined based on the candidate models that satisfy the iteration termination condition.

[0104] Candidate models can be the results of model training based on different feature data volume parameters. For example, the top 50%, 60%, 70%, 80%, 90% of IV values ​​and all features can be used as inputs for model training, outputting the first, second, third, fourth, fifth, and sixth sub-models, which are the candidate models. These candidate models are then evaluated to determine the target model.

[0105] In some embodiments, evaluating candidate models can involve scoring their performance and selecting the candidate model with the highest score as the target model. Specifically, this can involve obtaining the combined scores Score1, Score2, Score3, Score4, Score5, and Score6 from the six candidate models mentioned above. The target model can be the candidate model with the highest score in this round of updates and iterations.

[0106] By setting multiple feature data quantity parameters and obtaining multiple target feature groups, the current model is updated and iterated based on the training samples and each target feature group to generate multiple candidate models and give them a comprehensive score. The candidate model with the highest score in this round of iteration is output as the target model, which reduces the measurement error caused by objective factors and ensures that the model after iterative update has better performance.

[0107] Optionally, the model indicators of the preset type for each candidate model are determined, and the comprehensive indicators are determined based on the model indicators. The model indicators and comprehensive indicators are evaluated based on multiple evaluation models to obtain the evaluation data of each evaluation model on the candidate model. The target evaluation result of the candidate model is determined based on the evaluation data of each evaluation model on the candidate model, and the target model is determined based on the target evaluation result of each candidate model.

[0108] The model metrics can be one or more of the metric data. The comprehensive metric can be at least a portion of the model metrics obtained from a preset type, such as the well-performing AUC, KS, and PSI, obtained by pre-processing at least a portion of the model metrics, for example, by weighting AUC, KS, and PSI. The evaluation model can be a trained logistic regression, xgboot, or lightGBM model, etc., without specific limitations. Correspondingly, there can be one or more evaluation models, and each evaluation model can output a target evaluation result.

[0109] For example, the evaluation index S is derived from the following comprehensive function using the three indicators that perform well: AUC, KS, and PSI.

[0110] S=αA+βB+γC

[0111] Where A represents the model AUC value, B represents the model KS value, and C represents the model PSI stability. α is set to 0.3, β to 0.3, and γ to 0.4.

[0112] Eleven metrics—AUC, KS, GINI, AR, PSI, precision, accuracy, recall, F1 score, false positive rate, and S—are used as input feature data. A weighted average is calculated using evaluation models—trained logistic regression, xgboot, and lightGBM—to assess the performance score of the iteratively generated models, which serves as the target evaluation result. Models are then ranked based on their performance scores, and the model with the highest score is selected as the target model output.

[0113] By using scores to measure model performance from multiple dimensions, the optimal iterative model can be selected more efficiently and accurately.

[0114] It should be noted that the selected target model is checked to see if it meets the iteration termination condition. If not, the process of selecting target features based on feature range, determining training samples based on sample selection range, and updating the current model based on training samples and target features is repeated until the obtained target model meets the iteration termination condition.

[0115] In an optional embodiment, see details below. Figure 3First, the current model is initialized with parameter configuration to obtain a preset configuration file. Based on prediction and validation data, the indicator sequence of the current model in application is determined, and an evaluation model (decay detection model) is built using the LSTM algorithm of deep learning. Based on the decline detection model, the decline risk of the model is predicted, and the model that meets the decline conditions is automatically iterated in a timely manner. When iteration is triggered, a random algorithm is first used to select 50% of the features as the feature range for the first round of iteration, i.e., feature initial screening. Then, the training set, test set, and validation set are automatically divided. The data is preprocessed. Based on the feature preprocessing results, further feature selection is performed. Using the forward iteration method, six sub-models, i.e., candidate models, are trained. Each sub-model is automatically tuned using the Bayesian optimization algorithm for hyperparameters. The performance of the sub-models is evaluated using 11 index data. The index data is used as new feature inputs to the logistic regression model, the XGBoost model, and the LightGBM model to form a comprehensive evaluation model, i.e., the evaluation model. The three scores output by the three models are weighted and averaged to obtain the comprehensive score of each sub-model. The sub-model with the highest comprehensive score is selected as the target model output. The output target model is compared to see if it meets the iteration exit condition. If the target model meets the condition, it is output as a new model version; otherwise, a second round of iteration is performed until the iteration exit condition is met.

[0116] The technical solution of this embodiment obtains the prediction and validation data of the current model during application, and determines the indicator sequence of the current model during application based on the prediction and validation data; performs decay detection on the current model based on the indicator sequence; when the current model meets the decay conditions, obtains a preset configuration file, and updates the current model based on the preset configuration file to obtain the updated target model. This approach minimizes manual intervention, improving the efficiency of model iteration and reducing labor costs. The use of decay detection to automatically trigger iteration and update operations further reduces the loss of benefits caused by untimely model iteration and updates based on the preset configuration file, minimizing measurement errors caused by objective factors and ensuring better performance of the iterated model.

[0117] Example 2

[0118] Figure 4 This is a schematic diagram of a model update device provided in Embodiment 2 of the present invention. Figure 4 As shown, the device includes:

[0119] The indicator sequence determination module 410 is used to acquire the prediction data and validation data of the current model during the application process, and determine the indicator sequence of the current model during the application process based on the prediction data and the validation data.

[0120] The decay detection module 420 is used to perform decay detection on the current model based on the indicator sequence;

[0121] The target model update module 430 is used to obtain a preset configuration file when the current model meets the decay condition, and update the current model based on the preset configuration file to obtain the updated target model.

[0122] Optionally, the indicator sequence determination module 410 includes:

[0123] The indicator data determination module is used to determine the corresponding indicator data based on the corresponding prediction data and verification data, wherein the timestamp of the indicator data is the same as the timestamp of the prediction data;

[0124] The indicator sequence generation module is used to filter each indicator data based on the timestamp of the indicator data and the sequence length, and generate an indicator sequence based on the filtered indicator data.

[0125] Optionally, the decay detection module 420 includes:

[0126] The decay parameter acquisition module is used to input the indicator sequence into a pre-trained decay detection model to obtain the decay parameters corresponding to the indicator sequence.

[0127] The decay condition satisfaction module is used to determine that the current model satisfies the decay condition if the decay parameter is greater than a preset decay threshold.

[0128] Optionally, the preset configuration file includes one or more of the following: feature range, sample selection range, and iteration termination condition.

[0129] Optionally, the target model update module 430 includes:

[0130] The target feature filtering module is used to filter target features based on the feature range;

[0131] The training sample determination module is used to determine training samples based on the sample selection range;

[0132] The first update module is used to update the current model based on the training samples and the target features. During the update process, if the iteration termination condition is met, the iteration is stopped and the updated target model is obtained.

[0133] Optionally, the target feature filtering module includes:

[0134] An evaluation parameter acquisition module is used to determine candidate features within the feature range, evaluate each candidate feature, and obtain evaluation parameters for each candidate feature.

[0135] The first screening module is used to screen the target features based on the evaluation parameters of each candidate feature.

[0136] Optionally, the first filtering module includes:

[0137] The sorting module is used to sort the candidate features based on the evaluation parameters of each candidate feature;

[0138] The target feature group determination module is used to determine a target feature group from the ranking of the candidate features based on the feature data volume parameter, wherein the target feature group includes the target features corresponding to the feature data volume parameter.

[0139] Optionally, the feature data volume parameter can be multiple, and the obtained target feature group can be multiple;

[0140] Accordingly, the first update module includes:

[0141] The candidate model acquisition module is used to update the current model based on the training samples and each target feature group to obtain a candidate model.

[0142] The target model determination module is used to stop the iteration if at least one of the candidate models satisfies the iteration termination condition, and to determine the target model based on the candidate models that satisfy the iteration termination condition.

[0143] Optionally, the iteration termination condition includes the updated target model's model metrics being better than the current model's model metrics before the update.

[0144] Optional, the target model determination module includes:

[0145] The comprehensive index determination module is used to determine the model index of the preset type for each candidate model, and to determine the comprehensive index based on the model index;

[0146] The evaluation data acquisition module is used to evaluate the model indicators and the comprehensive indicators based on multiple evaluation models to obtain the evaluation data of each evaluation model for the candidate model.

[0147] The first target model determination module is used to determine the target evaluation result of the candidate model based on the evaluation data of each evaluation model, and to determine the target model based on the target evaluation result of each candidate model.

[0148] Optionally, the comprehensive index determination module is specifically used for:

[0149] Obtain at least some of the model indicators of the preset type, perform preset processing on the at least some model indicators, and obtain the comprehensive indicator.

[0150] The model update apparatus provided in this embodiment of the invention can execute the model update method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0151] Example 3

[0152] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0153] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0154] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0155] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as model update methods.

[0156] In some embodiments, the model update method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the model update method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the model update method by any other suitable means (e.g., by means of firmware).

[0157] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0159] Example 4

[0160] Embodiment 4 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a model update method, the method comprising:

[0161] Obtain the prediction data and validation data of the current model during the application process, and determine the indicator sequence of the current model during the application process based on the prediction data and the validation data;

[0162] Decline detection is performed on the current model based on the aforementioned indicator sequence;

[0163] If the current model meets the decay condition, a preset configuration file is obtained, and the current model is updated based on the preset configuration file to obtain the updated target model.

[0164] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0165] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0166] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0167] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0168] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0169] Example 5

[0170] Embodiment 5 of the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the model update method according to any embodiment of the present invention.

[0171] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A model update method, characterized in that, include: Obtain the prediction data and validation data of the current model during the application process, and determine the indicator sequence of the current model during the application process based on the prediction data and the validation data; Decline detection is performed on the current model based on the aforementioned indicator sequence; If the current model meets the decay condition, a preset configuration file is obtained, and the current model is updated based on the preset configuration file to obtain the updated target model. The preset configuration file includes one or more of the following: feature range, sample selection range, and iteration termination condition. The step of determining the indicator sequence of the current model in the application process based on the predicted data and the validation data includes: Based on the corresponding prediction data and verification data, the corresponding indicator data is determined, wherein the timestamp of the indicator data is the same as the timestamp of the prediction data; Based on the timestamps and sequence lengths of the indicator data, each indicator data is filtered, and an indicator sequence is generated based on the filtered indicator data. The step of performing decay detection on the current model based on the indicator sequence includes: The index sequence is input into a pre-trained decay detection model to obtain the decay parameters corresponding to the index sequence; If the decay parameter is greater than the preset decay threshold, then the current model is determined to meet the decay condition; The step of updating the current model based on the preset configuration file to obtain the updated target model includes: Target features are selected based on the aforementioned feature range; Based on the aforementioned sample selection range, training samples are determined; Based on the training samples and the target features, the current model is updated. During the update process, if the iteration termination condition is met, the iteration stops and the updated target model is obtained.

2. The method according to claim 1, characterized in that, The process of filtering target features based on the feature range includes: Candidate features are determined within the specified feature range, and each candidate feature is evaluated to obtain evaluation parameters for each candidate feature. The target features are selected based on the evaluation parameters of each candidate feature.

3. The method according to claim 2, characterized in that, The process of filtering the target features based on the evaluation parameters of each candidate feature includes: The candidate features are ranked based on the evaluation parameters of each candidate feature; Based on the feature data volume parameter, a target feature group is determined from the ranking of the candidate features, and the target feature group includes the target features corresponding to the feature data volume parameter.

4. The method according to claim 3, characterized in that, The feature data quantity parameter is multiple, and the obtained target feature group is multiple; Accordingly, based on the training samples and the target features, the current model is updated. During the update process, if the iteration termination condition is met, the iteration stops, and the updated target model is obtained, including: Based on the training samples and each target feature group, the current model is updated to obtain a candidate model. If at least one of the candidate models satisfies the iteration termination condition, the iteration stops, and the target model is determined based on the candidate models that satisfy the iteration termination condition.

5. The method according to claim 1 or 4, characterized in that, The iteration termination condition includes the updated target model's model metrics being better than the current model's model metrics before the update.

6. The method according to claim 4, characterized in that, Determining the target model based on the candidate models that satisfy the iteration termination condition includes: Determine the model metrics of each candidate model for a preset type, and determine the comprehensive metrics based on the model metrics; The model indicators and the comprehensive indicators are evaluated based on multiple evaluation models to obtain the evaluation data of each evaluation model for the candidate model; The target evaluation result of the candidate model is determined based on the evaluation data of each of the evaluation models, and the target model is determined based on the target evaluation result of each of the candidate models.

7. The method according to claim 6, characterized in that, The determination of the comprehensive index based on the model index includes: Obtain at least some of the model indicators of the preset type, perform preset processing on the at least some model indicators, and obtain the comprehensive indicator.

8. A model update device, characterized in that, include: The indicator sequence determination module is used to acquire the prediction data and validation data of the current model during the application process, and determine the indicator sequence of the current model during the application process based on the prediction data and the validation data. A decay detection module is used to perform decay detection on the current model based on the indicator sequence; The target model update module is used to obtain a preset configuration file when the current model meets the decay condition, and update the current model based on the preset configuration file to obtain the updated target model. The preset configuration file includes one or more of the following: feature range, sample selection range, and iteration termination condition. The indicator sequence determination module includes: The indicator data determination module is used to determine the corresponding indicator data based on the corresponding prediction data and verification data, wherein the timestamp of the indicator data is the same as the timestamp of the prediction data; The indicator sequence generation module is used to filter each indicator data based on the timestamp of the indicator data and the sequence length, and generate an indicator sequence based on the filtered indicator data. The degradation detection module includes: The decay parameter acquisition module is used to input the indicator sequence into a pre-trained decay detection model to obtain the decay parameters corresponding to the indicator sequence. The decay condition satisfaction module is used to determine that the current model satisfies the decay condition if the decay parameter is greater than a preset decay threshold. The target model update module includes: The target feature filtering module is used to filter target features based on the feature range; The target feature filtering module is used to determine training samples based on the sample selection range; The first update module is used to update the current model based on the training samples and the target features. During the update process, if the iteration termination condition is met, the iteration is stopped and the updated target model is obtained.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the model update method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the model update method according to any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the model update method according to any one of claims 1-7.

Images