Model processing method and related device

Abstract

The invention discloses a model processing method, which is used for improving the performance of a model obtained by training in processing new data when concept drift occurs in the model. In the method, when it is detected that concept drift occurs in a first model in use, input data obtained after the concept drift occurs in the first model is adopted to train a second model used for replacing the first model, and the first model still continues to be used for a period of time. Moreover, when the second model is trained, the degree of concept drift of the first model can be continuously detected, and the training hyper-parameters of the second model are dynamically adjusted based on the degree of concept drift of the first model, so that the second model can more quickly and effectively adapt to the distribution of new data, and the accuracy of the data distribution is improved. Therefore, the performance of the second model obtained by training in processing new data is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model processing method and related devices. Background Art

[0002] In practical machine learning scenarios, the distribution of data processed by a machine learning model can change over time. This phenomenon is often referred to as concept drift. Concept drift can cause the distribution of the training data used by the machine learning model to be inconsistent with the distribution of the actual data processed, resulting in a decrease in the performance of the machine learning model in real-world applications.

[0003] Currently, related technologies detect concept drift in real time during the operation of machine learning models, and retrain the machine learning models when concept drift is detected to improve the performance of the machine learning models when processing new data.

[0004] However, the retrained machine learning models in related technologies are often still difficult to adapt to the distribution of new data, resulting in limited performance improvement after retraining of the machine learning models. Summary of the Invention

[0005] The present application provides a model processing method for improving the performance of a trained model in processing new data when concept drift occurs in the model.

[0006] In a first aspect, the present application provides a model processing method for processing a running model to improve the model's ability to adapt to concept drift. The method specifically comprises: first, detecting a first concept drift value of a first model at a first time point, where the first concept drift value indicates the extent to which the performance of the first model has degraded due to a change in the distribution of the input data of the first model. The first model is a machine learning model used in actual application scenarios, and is used to process user-provided data.

[0007] In response to the first concept drift value satisfying a first condition, a second model is trained using input data from the first model acquired after the first time point, with the second model replacing the first model. Specifically, the first condition can be established by setting a concept drift warning value, and the satisfaction of the first condition is determined by comparing the concept drift value with the concept drift warning value. When the first concept drift value satisfies the first condition, it indicates that the performance of the first model has degraded due to concept drift, reaching a warning level, thereby triggering the start of training for the backup second model.

[0008] Furthermore, during the training of the second model, its training hyperparameters are dynamically adjusted based on the concept drift values ​​of the first model. This means that during the training of the second model, the concept drift values ​​of the first model are continuously monitored and the training hyperparameters of the second model are dynamically adjusted based on the detected concept drift values, rather than being fixed.

[0009] In this solution, when concept drift is detected in the first model in use, the input data obtained after the first model's concept drift is used to train a second model to replace the first model, while the first model continues to be used for a period of time. Furthermore, while training the second model, the degree of concept drift in the first model is continuously monitored, and the training hyperparameters of the second model are dynamically adjusted based on the degree of concept drift in the first model. This allows the second model to adapt to the distribution of new data more quickly and effectively, thereby improving the performance of the trained second model when processing new data.

[0010] In one possible implementation, the method further includes detecting a second concept drift value of the first model at a second time point, and a performance value of the second model at the second time point, the second time point being subsequent to the first time point. That is, after training the second model begins, in addition to continuing to detect the concept drift value of the first model, the performance value of the trained second model may also be detected in real time or periodically.

[0011] Then, based on the second concept drift value and the performance value, a replacement decision value is determined, wherein the replacement decision value is used to decide whether to adopt the second model to replace the first model currently in use.

[0012] When the replacement decision value satisfies the second condition, the second model is used to replace the first model. For example, when the replacement decision value is higher than a preset alarm value, the second model can be used to replace the first model.

[0013] In this solution, the degree of concept drift of the first model currently in use and the performance of the second model trained later are comprehensively considered to decide whether the second model trained later needs to replace the original first model. This enables a better decision on the timing of replacing the first model with the second model, thereby ensuring that the accuracy of data processing can be effectively improved after the model replacement is performed.

[0014] In one possible implementation, the process of determining the replacement decision value may specifically include: performing normalization on the second concept drift value and the performance value respectively to obtain a normalized second concept drift value and a normalized performance value; and performing weighted summation on the normalized second concept drift value and the normalized performance value to obtain the replacement decision value.

[0015] In this solution, by normalizing the second concept drift value and the performance value of the second model respectively and then performing weighted summation, the concept drift values ​​and performance values ​​in different dimensions can be normalized to the same dimension to achieve fusion, so as to obtain a replacement decision value related to the concept drift value and the model performance value, ensuring that the calculated replacement decision value can well comprehensively reflect the degree of concept drift of the first model and the performance of the second model.

[0016] In one possible implementation, the training hyperparameter includes a learning rate. If the concept drift value is positively correlated with the degree of performance degradation of the first model, the learning rate is positively correlated with the concept drift value. That is, a larger concept drift value indicates a more pronounced concept drift (i.e., a greater difference in data distribution). Therefore, a larger learning rate is required to more quickly learn the data distribution.

[0017] In the case where the concept drift value is negatively correlated with the degree of degradation of the first model's performance, the learning rate is negatively correlated with the concept drift value.

[0018] In this solution, by setting the learning rate to change dynamically following the degree of concept drift reflected by the concept drift value, the learning speed of the second model during training can be made consistent with the distribution of training data, thereby improving the training efficiency and accuracy of the second model.

[0019] In one possible implementation, the second model is a decision tree model, and the training hyperparameters include the number of feature selections of the decision tree.

[0020] While the concept drift value is positively correlated with the degree of performance degradation of the first model, the number of features selected is negatively correlated with the concept drift value. That is, the larger the concept drift value, the more obvious the concept drift (i.e., the greater the data distribution difference), so the number of features selected by the decision tree during the decision-making process can be reduced, thereby effectively preventing the decision tree model from overfitting and helping the decision tree model combat concept drift. The smaller the concept drift value, the less obvious the concept drift (i.e., the smaller the data distribution difference), so the number of features selected by the decision tree during the decision-making process can be increased, thereby executing the decision process based on more features, which is conducive to learning deep data patterns and thus improving the accuracy of the decision tree model.

[0021] Similarly, while the concept drift value is negatively correlated with the degree of performance degradation of the first model, the number of feature selections is positively correlated with the concept drift value.

[0022] In this solution, by setting the number of feature selections of the decision tree to change dynamically with the degree of concept drift reflected by the concept drift value, the learning situation of the second model during the training process can be consistent with the distribution of the training data, thereby improving the training efficiency and accuracy of the second model.

[0023] In one possible implementation, to monitor a model, a user-entered calculation method for a monitoring indicator may be obtained. The monitoring indicator is used to characterize the model's performance. During the operation of a target model, the value of the monitoring indicator corresponding to the target model is calculated based on the calculation method. The target model is either the first model or the second model. When the value of the monitoring indicator reaches an indicator alarm value, a model performance alarm is triggered.

[0024] In this solution, by obtaining user-defined monitoring indicators and continuously calculating the values ​​of monitoring indicators during model operation, the performance monitoring of the model can be implemented in a user-defined manner, ensuring that the performance of the monitored model is consistent with the actual situation in the current field, so that the model performance monitored in various business scenarios can effectively reflect the true prediction quality of the model.

[0025] In one possible implementation, the above-mentioned monitoring indicators are calculated from multiple performance indicators, and the multiple performance indicators are calculated from one or more sub-indicators. Then, when calculating the value of the monitoring indicator corresponding to the target model, a sub-indicator set can be determined based on multiple performance indicators, and the sub-indicator set includes multiple sub-indicators used to calculate the multiple performance indicators; then the values ​​of the multiple sub-indicators corresponding to the target model are calculated, and the value of the monitoring indicator is determined based on the values ​​of the multiple sub-indicators. Specifically, since the monitoring indicator is actually composed of multiple performance indicators, each performance indicator is decomposed according to the calculation method of the sub-indicator. Then, after the values ​​of each sub-indicator are calculated, the values ​​of the multiple sub-indicators can be further integrated to determine the value of the monitoring indicator.

[0026] In this solution, by decomposing the performance indicator into sub-indicators, calculating the value of each sub-indicator, and combining the values ​​of multiple sub-indicators to calculate the value of the monitoring indicator, it is possible to deduplicate the sub-indicators and avoid repeatedly calculating the values ​​of some of the same sub-indicators when calculating each performance indicator separately, thereby speeding up the indicator calculation and improving the efficiency of model monitoring.

[0027] In a possible implementation, the first model and the second model are both classification models for performing classification tasks, such as a decision tree model, a convolutional neural network, or a fully connected network.

[0028] In one possible implementation, the input data for the first and second models is tabular data. That is, the first model actually processes data with a certain distribution range. When the first model's input data is tabular data, it can effectively detect the distribution differences between the actual data processed by the first model and the training data, thereby obtaining accurate concept drift values, which facilitates subsequent monitoring and replacement of the first model.

[0029] A second aspect of the present application provides a model processing device, comprising: a processing module, configured to detect a first concept drift value of a first model at a first time point, the first concept drift value being used to indicate the extent to which the performance of the first model has deteriorated due to a change in the distribution of the input data of the first model; the processing module, further configured to, in response to the first concept drift value satisfying a first condition, use the input data of the first model acquired after the first time point to train a second model, the second model being used to replace the first model; wherein, during the training of the second model, the training hyperparameters of the second model are dynamically adjusted as the concept drift value of the first model changes during operation.

[0030] In one possible implementation, the processing module is further used to: detect a second concept drift value of the first model at a second time point, and a performance value of the second model at the second time point, where the second time point is after the first time point; determine a replacement decision value based on the second concept drift value and the performance value; and replace the first model with the second model when the replacement decision value satisfies a second condition.

[0031] In one possible implementation, the processing module is further used to: perform normalization on the second concept drift value and the performance value respectively to obtain a normalized second concept drift value and a normalized performance value; and perform weighted summation on the normalized second concept drift value and the normalized performance value to obtain a replacement decision value.

[0032] In one possible implementation, the training hyperparameters include a learning rate; when the concept drift value is positively correlated with the degree of performance degradation of the first model, the learning rate is positively correlated with the concept drift value; when the concept drift value is negatively correlated with the degree of performance degradation of the first model, the learning rate is negatively correlated with the concept drift value.

[0033] In one possible implementation, the second model is a decision tree model, and the training hyperparameters include the number of feature selections of the decision tree; when the concept drift value is positively correlated with the degree of performance degradation of the first model, the number of feature selections is negatively correlated with the concept drift value; when the concept drift value is negatively correlated with the degree of performance degradation of the first model, the number of feature selections is positively correlated with the concept drift value.

[0034] In one possible implementation, the model processing device also includes: an acquisition module, used to obtain the calculation method of the monitoring indicator input by the user, and the monitoring indicator is used to characterize the performance of the model; the processing module is also used to calculate the value of the monitoring indicator corresponding to the target model based on the calculation method during the operation of the target model, and the target model is the first model or the second model; the processing module is also used to trigger a model performance alarm when the value of the monitoring indicator reaches the indicator alarm value.

[0035] In one possible implementation, the monitoring indicator is calculated from multiple performance indicators, and the multiple performance indicators are calculated from one or more sub-indicators; the processing module is also used to: determine a sub-indicator set based on multiple performance indicators, the sub-indicator set includes multiple sub-indicators used to calculate multiple performance indicators; calculate the values ​​of multiple sub-indicators corresponding to the target model, and determine the value of the monitoring indicator based on the values ​​of the multiple sub-indicators.

[0036] In a possible implementation, the first model and the second model are both classification models for performing classification tasks.

[0037] In a possible implementation, the input data of the first model and the second model are tabular data.

[0038] A third aspect of the present application provides a computing device cluster, comprising at least one computing device, each computing device including a processor and memory. The processor of at least one computing device is configured to execute instructions stored in the memory of at least one computing device, causing the computing device cluster to perform the method described in the first aspect or any of the implementations of the first aspect. For details regarding the steps in each possible implementation of the first aspect performed by the computing device cluster, please refer to the first aspect and will not be repeated here.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium having instructions stored therein. When the instructions are executed on a computer, the computer can execute any of the above methods.

[0040] A fifth aspect of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods described above.

[0041] In a sixth aspect, the present application provides a chip comprising a processor and a communication interface, wherein the communication interface is used to communicate with modules outside the chip, and the processor is used to run computer programs or instructions so that a device in which the chip is installed can execute any of the methods described above.

[0042] Among them, the technical effects brought about by any design method in the second to sixth aspects can refer to the technical effects brought about by different implementation methods in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a system architecture 100 provided in an embodiment of the present application;

[0044] Figure 2 A flow chart of a model processing method provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of monitoring a running model and implementing concept drift adaptation provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of a model implementing concept drift adaptation provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a process flow for processing a model in an actual application scenario provided by an embodiment of the present application;

[0048] Figure 6 A schematic diagram of the structure of a model processing device provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0051] Figure 9 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application;

[0052] Figure 10 A schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0053] Figure 11 A schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clear, the embodiments of this application are described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of this application, rather than all embodiments. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0055] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchangeable where appropriate so that the embodiments can be implemented in a sequence other than that illustrated or described in this application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or that are inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The named or numbered process steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units in this application is a logical division. In actual application, there may be other division methods. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. Moreover, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed into multiple circuit units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this application.

[0056] To facilitate understanding, some technical terms involved in the embodiments of this application are first introduced below.

[0057] (1) Machine Learning

[0058] Machine learning is a method that allows computers to automatically learn patterns and rules from data to complete specific tasks.

[0059] (2) Machine Learning Model

[0060] A machine learning model is a mathematical model built by learning from existing data and extracting features from it. By building a machine learning model based on existing data, the model can learn the underlying patterns in the existing data, allowing it to make predictions or classify new data.

[0061] Generally speaking, machine learning models are usually implemented by neural networks.

[0062] (3) Neural Network

[0063] A neural network can be composed of neural units, which can be represented by x s (i.e. input data) and intercept 1 as input operation unit, the output of the operation unit can be:

[0064]

[0065] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s is x s The weight parameter of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0066] (4) Decision Tree Model

[0067] The decision tree model is a tree-structured machine learning model, similar to a flowchart, used to solve classification and regression problems. The decision tree model recursively partitions the dataset and constructs a tree to make predictions and decisions.

[0068] Simply put, the decision tree model continuously makes conditional judgments on data samples, and ultimately distinguishes samples of different categories as much as possible.

[0069] (5) Hyperparameters

[0070] Hyperparameters are parameters that are manually set during the machine learning model building process. They are usually adjusted before or during model training. Hyperparameters do not rely on data to obtain optimal values, but are determined through expert knowledge or other experience.

[0071] Here are some specific examples of hyperparameters: the number of decision trees, the depth of the decision trees, network parameters (which may include structural elements of the network, such as the number of layers, the number and size of convolution kernels, activation functions, etc.), optimization parameters (involving learning rate, batch size, number of batches, optimizer type, and parameters of other loss functions), regularization parameters (which may include weight decay coefficients and dropout ratios).

[0072] The goal of hyperparameter tuning is to balance model optimization and regularization to find the optimal or near-optimal solution for the model. For example, the learning rate directly influences the network gradient and the magnitude of parameter updates during training, making it crucial for the model's convergence speed and accuracy.

[0073] In practical applications, the selection and use of hyperparameters is a critical step because it affects the performance and stability of the final model. Choosing appropriate hyperparameters usually involves a deep understanding of the data, experimental testing, and adaptive adjustments for different scenarios and applications.

[0074] (6) Concept Drift

[0075] Concept drift refers to a phenomenon in which the distribution of learned concepts within a learning system changes. For example, in machine learning scenarios, the distribution of data processed by a machine learning model can change over time, leading to inconsistencies between the data distribution during model training and in actual application. This translates to a change in data concepts.

[0076] (7) Accuracy (ACC)

[0077] In classification tasks, ACC refers to the proportion of correctly classified samples to the total number of samples.

[0078] (8) Precision

[0079] Precision refers to the proportion of samples predicted to be positive that actually belong to the positive class.

[0080] (9) Recall

[0081] The recall rate refers to the proportion of all positive samples that are correctly identified as positive, that is, the proportion of identified positive samples to actual positive samples.

[0082] (10)F1-SCORE

[0083] F1-SCORE refers to the harmonic mean of precision and recall.

[0084] (11) KAPPA coefficient

[0085] In machine learning, the KAPPA coefficient is a metric used to measure the performance of classification models, particularly for evaluating their performance with unbalanced samples. The KAPPA coefficient takes into account the accuracy of the model's predictions for both positive and negative examples. The KAPPA coefficient is calculated by multiplying the total number of ground truth pixels (N) by the sum of the diagonal lines of the confusion matrix (Xkk), subtracting the product of the total number of ground truth pixels in a particular class and the total number of classified pixels in that class, and dividing the result by the square of the total number of pixels minus the product of the total number of ground truth pixels in a particular class and the total number of classified pixels in that class.

[0086] If concept drift occurs during the application of a machine learning model, its performance can degrade during actual application. Therefore, related technologies perform real-time concept drift detection during the operation of the machine learning model and retrain the model when concept drift is detected to improve its performance when processing new data. However, these retrained machine learning models often still struggle to adapt to the distribution of new data, resulting in limited performance improvements after retraining.

[0087] Based on this, an embodiment of the present application provides a model processing method. When concept drift is detected in a first model in use, the input data obtained after the concept drift occurs in the first model is used to train a second model used to replace the first model, and the first model will continue to be used for a period of time. In addition, when training the second model, the degree of concept drift of the first model will be continuously detected, and the training hyperparameters of the second model will be dynamically adjusted based on the degree of concept drift of the first model, so that the second model can adapt to the distribution of new data more quickly and effectively, thereby improving the performance of the trained second model when processing new data.

[0088] See also Figure 1 , Figure 1 A schematic diagram of a system architecture 100 provided in an embodiment of the present application. Figure 1 As shown, in the system architecture 100, the execution device 110 can be implemented by at least one computing instance of a physical host (computing device), a virtual machine, or a container. When the execution device 110 is implemented by a virtual machine or a container, the execution device 110 actually exists in the form of a cloud computing product and can provide cloud services.

[0089] Optionally, in order to store data persistently, the system architecture 100 is further provided with a data storage system 120, which may be located outside the execution device 110 (e.g., Figure 2As shown, the execution device 110 exchanges data with the execution device 110 via a network. Optionally, when the execution device 110 is a physical host, the data storage system 120 may be located inside the execution device 110, such as when the data storage system 120 exchanges data with the processor via a bus. In this case, the data storage system 120 is represented by a hard disk. When the data storage system 120 is present, the execution device 110 may use the data in the data storage system 120 or call the program code in the data storage system 120 to implement the model processing method provided in the embodiments of the present application.

[0090] Users can operate their respective user devices (such as local device 101 and local device 102) to interact with execution device 110. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a laptop computer, and a smart car.

[0091] Each user's local device can interact with the execution device 110 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0092] In one implementation, execution device 110 is used to implement the model processing method provided in the embodiments of the present application, thereby detecting and processing the running model, thereby ensuring that the model can be trained to obtain a new model that is more adaptable to the new data distribution after concept drift occurs. Furthermore, when local device 101 and local device 102 need to use the model to process data, execution device 110 processes the user's request based on the newly trained model and returns the corresponding processing results to local device 101 and local device 102.

[0093] In another implementation, the execution device 110 is used to implement the model processing method provided in the embodiment of the present application, and after training a new model to replace the original model, the new model is sent to the local device 101 and the local device 102. In this way, the local device 101 and the local device 102 can deploy the new model locally, thereby ensuring higher processing accuracy for new data.

[0094] In another implementation, one or more aspects of the execution device 110 can be implemented by each local device. For example, the local device 101 can provide local data or feedback calculation results to the execution device 110, or execute the model processing method provided in the embodiment of the present application.

[0095] In general, the model processing method provided in the embodiments of the present application can be applied to electronic devices, such as the aforementioned execution device 110, local device 101 or local device 102.

[0096] See also Figure 2 , Figure 2 This is a flow chart of a model processing method provided in an embodiment of the present application. Figure 2 As shown, the model processing method provided in the embodiment of the present application includes the following steps 201-205.

[0097] Step 201 : Detect a first concept drift value of a first model at a first time point. The first concept drift value is used to indicate the extent to which the performance of the first model is degraded due to a change in the distribution of input data of the first model.

[0098] In this embodiment, the first model is a machine learning model that is put into use in an actual application scenario and is used to process data provided by the user. The first model can be a machine learning model used in various fields, and the distribution of data processed by the first model will change. For example, the first model can be a model in the financial field, used to process the user's financial information or tax information; for another example, the first model can be a model in the power system field, used to predict the cause of faults in the power system. In general, this embodiment does not limit the specific field of application of the first model.

[0099] During the use of the first model, the first model can be monitored to obtain a concept drift value of the first model in real time or periodically. In this embodiment, the first concept drift value is obtained after monitoring the operation of the first model at a first point in time and is used to indicate the extent to which the performance of the first model has degraded due to changes in the distribution of the input data of the first model.

[0100] There are multiple ways to detect the first concept drift value. Specifically, because concept drift is a phenomenon in which data distribution changes, and this phenomenon can cause model performance to degrade, the concept drift value can be determined from either the data distribution or model performance perspectives.

[0101] In one possible implementation, the first concept drift value may be determined by detecting the distribution of input data of the first model. For example, the first concept drift value may be determined by detecting changes in statistical characteristics of the input data, such as a mean, variance, or quantile distribution of the input data.

[0102] In another implementation, the first concept drift value may be determined by testing the performance of the first model, for example, by testing the accuracy of the first model when processing input data.

[0103] It should be noted that, because there are multiple ways to detect the concept drift value of the first model, the way the first concept drift value indicates the degree to which the performance of the first model has declined due to a change in the distribution of the input data of the first model may vary under different detection methods. For example, the first concept drift value and the degree of decline in the performance of the first model may be positively correlated, i.e., a larger concept drift value indicates a greater decline in the performance of the first model. For another example, the first concept drift value and the degree of decline in the performance of the first model may be negatively correlated, i.e., a larger concept drift value indicates a lesser decline in the performance of the first model.

[0104] Optionally, the input data of the first model is, for example, tabular data, that is, the first model actually processes data with a certain distribution range. For example, the first model is a classification model in the financial field, and the input data of the first model is the user's financial data (such as the number of user loans, loan amount, loan time, number of defaults, etc.), and the first model is used to output classification results based on the input data (such as whether the user has a default risk). In the case where the input data of the first model is tabular data, the difference in distribution between the actual data processed by the first model and the training data can be effectively detected, so that an accurate concept drift value can be obtained, which is conducive to the subsequent monitoring and replacement of the first model.

[0105] Step 202 : In response to the first concept drift value satisfying the first condition, the input data of the first model acquired after the first time point is used to train a second model, and the second model is used to replace the first model.

[0106] In this embodiment, when the first concept drift value detected at the first time point satisfies the first condition, the second model is trained using the input data of the first model acquired after the first time point. Furthermore, the first model is still used during the training of the second model.

[0107] The first condition is set in a manner related to the degree of degradation of the first model. For example, when the first concept drift value is positively correlated with the degree of degradation of the first model's performance, the first condition may be that the first concept drift value is greater than or equal to the concept drift warning value; for another example, when the first concept drift value is negatively correlated with the degree of degradation of the first model's performance, the first condition may be that the first concept drift value is less than or equal to the concept drift warning value. For example, when the accuracy of the first model is used to represent the first concept drift value, when the accuracy of the first model is less than or equal to 80% (i.e., the concept drift warning value), it can be considered that the first concept drift value of the first model meets the first condition, i.e., has reached the concept drift warning level.

[0108] In general, the first condition can be achieved by setting a concept drift warning value and comparing the concept drift value with the concept drift warning value to determine whether the first condition is met. When the first concept drift value meets the first condition, it indicates that the performance of the first model has degraded due to concept drift and has reached a warning level, thereby triggering the training of the backup second model.

[0109] It's important to emphasize that during the training of the second model, its training hyperparameters are dynamically adjusted based on the concept drift values ​​of the first model. This means that during the training of the second model, the concept drift values ​​of the first model are continuously monitored and the training hyperparameters of the second model are dynamically adjusted based on the detected concept drift values, rather than being fixed.

[0110] Since the concept drift value detected when the first model is running can well reflect the distribution of the data currently processed by the first model (which is also the training data of the second model), dynamically adjusting the training hyperparameters of the second model based on the concept drift value of the first model can enable the second model to adopt appropriate training hyperparameters according to the distribution characteristics of the training data, ensuring that the second model can adapt to the distribution of new data more quickly and effectively, thereby improving the performance of the trained second model when processing new data.

[0111] Optionally, the training hyperparameters of the second model may include, for example, hyperparameters such as learning rate, number of feature selections for the decision tree, number of decision trees, network parameters, batch size, number of batches, or regularization parameters, which are not specifically limited in this embodiment.

[0112] In one possible example, the training hyperparameter includes a learning rate. If the detected concept drift value is positively correlated with the degree of performance degradation of the first model, the learning rate is positively correlated with the concept drift value. That is, a larger concept drift value indicates a more pronounced concept drift (i.e., a greater difference in data distribution), and therefore a larger learning rate is required to more quickly learn the data distribution.

[0113] Similarly, just as the concept drift value is negatively correlated with the degree of performance degradation of the first model, the learning rate is also negatively correlated with the concept drift value. That is, a smaller concept drift value indicates a more pronounced concept drift (i.e., a greater difference in the data distribution), and therefore a larger learning rate is required to more quickly learn the data distribution.

[0114] In this way, by setting the learning rate to change dynamically following the degree of concept drift reflected by the concept drift value, the learning speed of the second model during training can be made consistent with the distribution of training data, thereby improving the training efficiency and accuracy of the second model.

[0115] In one possible example, assuming that the second model is a decision tree model, the training hyperparameters include the number of feature selections for the decision tree. Then, in the case where the concept drift value is positively correlated with the degree of performance degradation of the first model, the number of feature selections is negatively correlated with the concept drift value. That is, the larger the concept drift value, the more obvious the concept drift (i.e., the greater the difference in data distribution), so the number of features selected by the decision tree in the decision-making process can be reduced, thereby effectively preventing the decision tree model from overfitting and helping the decision tree model to combat concept drift. The smaller the concept drift value, the less obvious the concept drift (i.e., the smaller the difference in data distribution), so the number of features selected by the decision tree in the decision-making process can be increased, thereby executing the decision process based on more features, which is conducive to learning deep data laws and thus improving the accuracy of the decision tree model.

[0116] Similarly, when the concept drift value is negatively correlated with the degree of performance degradation of the first model, the number of feature selections of the decision tree may be positively correlated with the concept drift value.

[0117] In general, by setting the number of feature selections in the decision tree to change dynamically with the degree of concept drift reflected by the concept drift value, the learning situation of the second model during training can be consistent with the distribution of training data, thereby improving the training efficiency and accuracy of the second model.

[0118] In this embodiment, the first model and the second model are models for processing the same task, and the first model and the second model can be, for example, models with the same structure. For example, the first model and the second model are both classification models for performing classification tasks, such as decision tree models, convolutional neural networks, or fully connected networks.

[0119] Step 203 : detecting a second concept drift value of the first model at a second time point and a performance value of the second model at the second time point, where the second time point is after the first time point.

[0120] That is to say, after starting to train the second model, in addition to continuing to detect the concept drift value of the first model, the performance value of the trained second model can also be detected in real time or periodically. The performance value of the second model can be represented by the accuracy of the second model, for example. The second concept drift value of the first model at the second time point can refer to a concept drift value obtained by performing concept drift detection on the first model at the second time point, or it can refer to a concept drift value obtained by comprehensively analyzing the concept drift detection of the first model within a certain time range before the second time point; similarly, the performance value of the second model at the second time point can refer to a performance value obtained by performing performance detection on the second model at the second time point, or it can refer to a performance value obtained by comprehensively analyzing the performance detection of the second model within a certain time range before the second time point.

[0121] It should be noted that when performing concept drift detection on the first model and performing performance detection on the second model, the input data of the first model and the second model are actually the same.

[0122] Step 204: Determine a replacement decision value based on the second concept drift value and the performance value.

[0123] In this embodiment, since the second concept drift value indirectly indicates the performance of the first model, and the performance value indicates the performance of the second model, a replacement decision value can be determined by combining the second concept drift value and the performance value. This replacement decision value is used to determine whether to adopt the second model to replace the first model currently in use.

[0124] Optionally, in the process of determining the replacement decision value, the second concept drift value and the performance value may be normalized to obtain a normalized second concept drift value and a normalized performance value. The normalized second concept drift value and the normalized performance value may both be distributed within the same interval (e.g., the interval [0, 1]).

[0125] Then, the normalized second concept drift value and the normalized performance value are weighted and summed to obtain the replacement decision value. There are many ways to perform weighted summation of the normalized second concept drift value and the normalized performance value, such as a weighted summation method with individually set corresponding weights, an exponential function weighted summation method, a sliding window weighted summation method, or a threshold weighted summation method, etc. This embodiment does not specifically limit this. The sliding window weighted summation method means that the second concept drift value and the performance value of the second model are both obtained based on the performance of the first model and the second model in processing data within a sliding time window (i.e., within the most recent period of time). The threshold weighted summation method means that the concept drift value and the performance value are assigned corresponding weights according to the interval range in which the concept drift value and the performance value are located to perform weighted summation.

[0126] In this way, by normalizing the second concept drift value and the performance value of the second model respectively and then performing weighted summation, the concept drift values ​​and performance values ​​in different dimensions can be normalized to the same dimension to achieve fusion, so that the replacement decision value related to the concept drift value and the model performance value can be obtained, ensuring that the calculated replacement decision value can well comprehensively reflect the degree of concept drift of the first model and the performance of the second model.

[0127] Step 205: When the replacement decision value satisfies the second condition, the first model is replaced by the second model.

[0128] In this embodiment, if the replacement decision value satisfies the second condition, the second model can be put into the actual application scenario and the first model can be discontinued in that application scenario, thereby replacing the first model with the second model. For example, if the replacement decision value exceeds a preset alarm value, the second model can be used to replace the first model.

[0129] In this solution, the degree of concept drift of the first model currently in use and the performance of the second model trained later are comprehensively considered to decide whether the second model trained later needs to replace the original first model. This enables a better decision on the timing of replacing the first model with the second model, thereby ensuring that the accuracy of data processing can be effectively improved after the model replacement is performed.

[0130] Specifically, the difference between the first and second models lies primarily in the amount of training data. The first model was initially trained based on a large amount of training data, so it uses a relatively large amount of training data. The second model, on the other hand, only uses training data acquired after the concept drift warning was issued. Therefore, the first model learns from a more diverse data distribution, and theoretically, generalizes better when presented with different data distributions.

[0131] Therefore, when the first model experiences concept drift, even though the second model performs better than the first, it may not be suitable for replacement. For example, the second model's prediction performance on the current data is better than the first model, but only to a limited extent. For example, the first model's accuracy is 80% while the second model's is 81%. However, when faced with future data distributions, which the second model may not have learned, the first model's prediction performance will often be better. Therefore, simply comparing the performance of the first and second models on the current data is inappropriate to replace the first model with the second model. Alternatively, if the first model experiences concept drift, but the second model has not yet learned the data patterns that caused the concept drift, its prediction performance may even be worse than the first model. In this case, replacing the first model with the second model because the first model has reached the concept drift warning level is also unreasonable. The best approach is to continue training the second model until its performance exceeds that of the first. Therefore, this solution comprehensively considers the degree of concept drift and the performance of the second model to calculate a comprehensive metric to determine when to perform model replacement, thereby determining the optimal timing for model replacement.

[0132] The above describes the process of training a new model by detecting the concept drift of the model and replacing the original model with the new model. In some embodiments, due to user needs, users often expect to continuously monitor the running model to ensure that the performance of the model always remains at the user's expected level. Moreover, in some specific fields, users often do not want to use conventional performance indicators to measure the performance of the model, but rather want to use some customized performance indicators to measure the performance of the model, so as to ensure that the performance of the monitored model is consistent with the actual situation in the current field.

[0133] Based on this, this solution proposes a solution in which human experts customize the monitoring indicators of the model and implement model monitoring based on the monitoring indicators.

[0134] For example, in order to implement model monitoring, a calculation method of a monitoring indicator input by a user may be obtained, where the monitoring indicator is used to characterize the performance of the model.

[0135] Then, during the operation of the target model, the value of the monitoring indicator corresponding to the target model is calculated based on the calculation method of the monitoring indicator. The target model is the first model or the second model described above. That is, during the operation of the first model or the second model, the above-mentioned monitoring indicator can be used to implement performance monitoring of the first model or the second model.

[0136] Furthermore, when the calculated monitoring indicator value reaches the indicator alarm value, a model performance alarm is triggered, thereby notifying the user to process the currently used model.

[0137] In this solution, by obtaining user-defined monitoring indicators and continuously calculating the values ​​of monitoring indicators during model operation, the performance monitoring of the model can be implemented in a user-defined manner, ensuring that the performance of the monitored model is consistent with the actual situation in the current field, so that the model performance monitored in various business scenarios can effectively reflect the true prediction quality of the model.

[0138] For example, let's assume the target model is a binary classification model (for example, to determine whether a user is at risk of default). Typically, there are many metrics that can be used to evaluate binary classification problems. In this scenario, business experts believe that a single metric is too simplistic and inadequate to evaluate the effectiveness of the target model. Therefore, a custom evaluation formula is provided: Monitoring metric = 0.3*ACC + 0.3*F1-score + 0.3*KAPPA + 0.1*Recall rate of customers at risk of default.

[0139] Optionally, the above-mentioned monitoring indicators can be calculated from multiple performance indicators, and the multiple performance indicators are calculated from one or more sub-indicators. Moreover, for different performance indicators, different performance indicators may be calculated from some of the same sub-indicators, that is, the sub-indicators required to calculate two different performance indicators may overlap. For example, assuming that performance indicator A is calculated from sub-indicator 1, sub-indicator 2, and sub-indicator 3, and performance indicator B is calculated from sub-indicator 2, sub-indicator 3, and sub-indicator 4, then sub-indicator 2 and sub-indicator 3 are overlapping in the sub-indicators required to calculate performance indicator A and performance indicator B, that is, sub-indicator 2 and sub-indicator 3 are required to calculate performance indicator A and performance indicator B.

[0140] Thus, when calculating the value of the monitoring indicator corresponding to the target model, a sub-indicator set can be determined based on multiple performance indicators. The sub-indicator set includes multiple sub-indicators used to calculate the multiple performance indicators. Since each performance indicator has one or more corresponding sub-indicators, by combining the sub-indicators corresponding to each performance indicator, it is possible to determine which sub-indicators (i.e., the sub-indicator set) are actually required to calculate the multiple performance indicators.

[0141] Then, the values ​​of multiple sub-indicators corresponding to the target model are calculated, and the value of the monitoring indicator is determined based on the values ​​of the multiple sub-indicators. Specifically, since the monitoring indicator is actually composed of multiple performance indicators, each performance indicator is decomposed according to the calculation method of the sub-indicators. After the values ​​of each sub-indicator are calculated, the values ​​of the multiple sub-indicators can be further combined to determine the value of the monitoring indicator.

[0142] In this solution, by decomposing the performance indicator into sub-indicators, calculating the value of each sub-indicator, and combining the values ​​of multiple sub-indicators to calculate the value of the monitoring indicator, it is possible to deduplicate the sub-indicators and avoid repeatedly calculating the values ​​of some of the same sub-indicators when calculating each performance indicator separately, thereby speeding up the indicator calculation and improving the efficiency of model monitoring.

[0143] For example, taking the above-mentioned monitoring indicator as an example, that is, the monitoring indicator = 0.3*ACC + 0.3F1-score + 0.3*KAPPA + 0.1*recall rate of customers with default risk. As can be seen from the formula, this monitoring indicator can be broken down into four performance indicators: ACC, F1-SCORE, KAPPA, and the recall rate of customers with default risk. Among them, F1-SCORE can be broken down into two sub-indicators: precision and recall rate. The KAPPA coefficient (kappa = (p0-pe) / (1-pe), where p0 equals ACC) can be broken down into ACC and pe. In this way, by breaking down each performance indicator into sub-indicators, it can be seen that the indicators that actually need to be calculated are ACC, pe, precision, and recall rate. This achieves deduplication of indicators and avoids the situation where sub-indicators are repeatedly calculated when calculating the same performance indicator.

[0144] To facilitate understanding, the model processing method provided in this embodiment will be described in detail below with reference to specific examples.

[0145] For example, see Figure 3 , Figure 3 This is a schematic diagram of a method for monitoring a running model and implementing concept drift adaptation provided in an embodiment of the present application. Figure 3 As shown in Figure 1, during normal use of a machine learning model, the model processes the input data after data preprocessing to produce corresponding prediction results. Furthermore, if concept drift occurs in the distribution of the data processed by the machine learning model, the model will be adapted to the concept drift to ensure the model's prediction accuracy.

[0146] As the machine learning model processes data, the prediction results it generates are stored in a designated database for subsequent model performance evaluation. At this point, the model monitoring platform calculates the performance metrics of the machine learning model in real time based on the prediction results stored in the database and records the results in the database. By setting thresholds for monitoring metrics on the model monitoring platform, it is possible to trigger an alert to staff when the performance metrics of the machine learning model exceed a specific threshold, thereby indicating that the performance of the machine learning model has deteriorated.

[0147] Specifically, see Figure 4 , Figure 4 A schematic diagram of a model implementing concept drift adaptation provided in an embodiment of the present application. Figure 4As shown in the figure, the machine learning model normally used in actual application scenarios can be called the primary model. During the operation of the primary model, concept drift detection is performed on the primary model in real time or periodically. If the concept drift value of the primary model exceeds the drift warning value, training of the backup model begins. During the training of the backup model, the training hyperparameters of the backup model are dynamically adjusted based on the concept drift value of the primary model.

[0148] Furthermore, after training the backup model, a replacement decision value is calculated based on the backup model's performance and the primary model's concept drift value to determine whether the backup model should replace the primary model. If the calculated replacement decision value exceeds the alarm threshold, the backup model can be used to replace the primary model, effectively putting the backup model into practical application scenarios. If the calculated replacement decision value does not exceed the alarm threshold, training of the backup model continues, and the primary model continues to be used in practical application scenarios.

[0149] See also Figure 5 , Figure 5 The following is a flow chart of a processing model in an actual application scenario provided by the embodiment of the present application. Figure 5 As shown in the figure, in actual application scenarios, the specific process of implementing model processing is divided into four stages: model training, concept drift adaptation, custom calculation, and model performance monitoring. The model training stage includes steps 1-3 below, the concept drift adaptation stage includes steps 4-8 below, the custom calculation stage includes steps 9-14 below, and the model performance monitoring stage includes steps 15-17 below.

[0150] Step 1: Obtain the raw data required for model training from the data source.

[0151] Step 2: Perform data preprocessing and feature engineering on the acquired raw data. The specific process includes but is not limited to processing missing values, processing outliers, standardizing and normalizing the data, etc. For details, please refer to existing data preprocessing and feature engineering, and no specific restrictions are made here.

[0152] Step 3: Use the preprocessed data to build a training dataset, select an appropriate machine learning algorithm, train the initial model, and obtain a trained model. The trained model is then used as the main model in actual application scenarios.

[0153] Step 4: During the operation of the master model, monitor the distribution of the master model's input data or the performance of the master model in real time to detect whether the master model has experienced concept drift. For example, by monitoring changes in the statistical characteristics of the input data (such as mean, variance, and quantiles), or by monitoring the performance of the master model in processing real-time data, it can be determined whether the master model has experienced concept drift.

[0154] In step 5, if the concept drift value of the primary model exceeds the drift warning value, a backup model is prepared. This backup model is independently trained starting from the time when the concept drift value of the primary model exceeds the drift warning value. If the concept drift value of the primary model does not exceed the drift warning value, jump to step 9.

[0155] Step 6: During the training of the backup model, dynamically adjust the training hyperparameters of the backup model based on the detected concept drift values ​​of the primary model. Dynamic adjustment methods include, but are not limited to, dynamically adjusting the learning rate based on the concept drift values ​​and dynamically adjusting the number of features selected in the decision tree.

[0156] Step 7: After the training of the backup model begins, a replacement decision value is calculated by weighted summation based on the performance value of the backup model and the concept drift value of the primary model to determine whether the backup model needs to be used to replace the primary model. The weighted summation calculation methods corresponding to the performance value of the backup model and the concept drift value of the primary model include, but are not limited to, simple weighted averaging of the concept drift values ​​and performance values ​​at different time points, exponential function weighting, sliding window weighting (using a sliding window method to only consider the concept drift values ​​and performance values ​​within the most recent period), threshold weighting (based on a set threshold, the concept drift value / performance value is divided into different levels, and then different weighting weights are assigned to the concept drift values / performance values ​​of different levels), etc.

[0157] Step 8: Determine whether the calculated replacement decision value exceeds the alarm threshold. If the replacement decision value exceeds the alarm threshold, the backup model is used to replace the current main model.

[0158] Step 9: Perform model prediction on the currently used model and store the prediction results in a specified location in the database for subsequent model effect monitoring.

[0159] In step 10, business experts enter the indicators of business concern based on business characteristics and define the calculation method and purpose of each indicator.

[0160] Step 11: Store the indicator formulas and uses entered by the experts to build an expert-defined indicator library.

[0161] Step 12: Obtain the monitoring indicators used in the current business scenario from the expert custom indicator library and perform formula analysis on the monitoring indicators.

[0162] Step 13: parse the monitoring indicator into multiple sub-indicators, and deduplicate the sub-indicators to remove duplicate sub-indicators to obtain all sub-indicators required for calculating the monitoring indicator.

[0163] Step 14: Calculate the value of each sub-indicator, and calculate the final result of the monitoring indicator by summarizing multiple sub-indicators to obtain the model performance evaluation value.

[0164] Step 15: Set an indicator alarm value on the model monitoring platform and determine whether the calculated model performance evaluation value exceeds the indicator alarm value.

[0165] Step 16: If the model performance evaluation value exceeds the indicator alarm value, the alarm system is triggered to notify relevant personnel.

[0166] Step 17: Record the model performance evaluation value obtained by real-time calculation into the database for subsequent reference by relevant personnel.

[0167] The above describes in detail the method provided by the embodiment of the present application. Next, the device provided by the embodiment of the present application for executing the above method will be introduced.

[0168] See also Figure 6 , Figure 6 This is a structural diagram of a model processing device provided in an embodiment of the present application. Figure 6 As shown, the model processing device includes: a processing module 601, which is used to detect a first concept drift value of a first model at a first time point, where the first concept drift value is used to indicate the extent to which the performance of the first model has declined due to a change in the distribution of the input data of the first model; the processing module 601 is also used to, in response to the first concept drift value satisfying a first condition, use the input data of the first model obtained after the first time point to train a second model, where the second model is used to replace the first model; wherein, during the training process of the second model, the training hyperparameters of the second model are dynamically adjusted as the concept drift value of the first model changes when the first model is running.

[0169] In one possible implementation, the processing module 601 is further used to: detect a second concept drift value of the first model at a second time point, and a performance value of the second model at the second time point, where the second time point is subsequent to the first time point; determine a replacement decision value based on the second concept drift value and the performance value; and replace the first model with the second model when the replacement decision value satisfies a second condition.

[0170] In one possible implementation, the processing module 601 is further configured to: respectively normalize the second concept drift value and the performance value to obtain a normalized second concept drift value and a normalized performance value; and perform a weighted summation on the normalized second concept drift value and the normalized performance value to obtain a replacement decision value.

[0171] In one possible implementation, the training hyperparameters include a learning rate; when the concept drift value is positively correlated with the degree of performance degradation of the first model, the learning rate is positively correlated with the concept drift value; when the concept drift value is negatively correlated with the degree of performance degradation of the first model, the learning rate is negatively correlated with the concept drift value.

[0172] In one possible implementation, the second model is a decision tree model, and the training hyperparameters include the number of feature selections of the decision tree; when the concept drift value is positively correlated with the degree of performance degradation of the first model, the number of feature selections is negatively correlated with the concept drift value; when the concept drift value is negatively correlated with the degree of performance degradation of the first model, the number of feature selections is positively correlated with the concept drift value.

[0173] In one possible implementation, the model processing device also includes: an acquisition module 602, which is used to obtain a calculation method for the monitoring indicator input by the user, where the monitoring indicator is used to characterize the performance of the model; a processing module 601, which is also used to calculate the value of the monitoring indicator corresponding to the target model based on the calculation method during the operation of the target model, where the target model is the first model or the second model; and the processing module 601, which is also used to trigger a model performance alarm when the value of the monitoring indicator reaches an indicator alarm value.

[0174] In one possible implementation, the monitoring indicator is calculated from multiple performance indicators, and the multiple performance indicators are calculated from one or more sub-indicators; the processing module 601 is also used to: determine a sub-indicator set based on multiple performance indicators, the sub-indicator set including multiple sub-indicators used to calculate the multiple performance indicators; calculate the values ​​of multiple sub-indicators corresponding to the target model, and determine the value of the monitoring indicator based on the values ​​of the multiple sub-indicators.

[0175] In a possible implementation, the first model and the second model are both classification models for performing classification tasks.

[0176] In a possible implementation, the input data of the first model and the second model are tabular data.

[0177] The acquisition module 602 and the processing module 601 can be implemented by software or hardware. For example, the implementation of the processing module 601 will be described below using the processing module 601 as an example. Similarly, the implementation of the acquisition module 602 can refer to the implementation of the processing module 601.

[0178] The processing module 601 is an example of a software functional unit. The processing module 601 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the processing module 601 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Generally, a region may include multiple AZs.

[0179] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0180] As an example of a hardware functional unit, processing module 601 may include at least one computing device, such as a server. Alternatively, processing module 601 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0181] The multiple computing devices included in processing module 601 can be distributed in the same region or in different regions. The multiple computing devices included in processing module 601 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in processing module 601 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0182] It should be noted that the information interaction, implementation process, etc. between the modules / units of the above-mentioned device are based on the same concept as the method embodiment of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the embodiment of the present application, and no further details will be given here.

[0183] See also Figure 7 , Figure 7 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. Figure 7 The computing device 700 shown can be used to execute the model processing method provided in this embodiment. Figure 7 As shown, computing device 700 includes a bus 702, a processor 704, a memory 706, and a communication interface 708. Processor 704, memory 706, and communication interface 708 communicate with each other via bus 702. Computing device 700 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 700.

[0184] The bus 702 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus 702 may include a path for transmitting information between various components of the computing device 700 (eg, memory 706, processor 704, communication interface 708).

[0185] The processor 704 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0186] The memory 706 may include volatile memory, such as random access memory (RAM). The processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0187] The memory 706 stores executable program codes, and the processor 704 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module and processing module, thereby implementing the aforementioned model processing method. That is, the memory 706 stores instructions for executing the model processing method.

[0188] The communication interface 708 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or a communication network.

[0189] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0190] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 706 in one or more computing devices 700 in the computing device cluster may store the same instructions for executing the model processing method.

[0191] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store some instructions for executing the model processing method. In other words, the combination of one or more computing devices 700 can jointly execute the instructions for executing the model processing method.

[0192] It should be noted that the memory 706 in different computing devices 700 in the computing device cluster can store different instructions, each for executing a portion of the functions of the data processing apparatus. In other words, the instructions stored in the memory 706 in different computing devices 700 can implement the functions of one or more of the above-mentioned receiving module and processing module.

[0193] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 9 A possible implementation is shown. Figure 9 This is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. Figure 9 As shown, in computing device cluster 900, two computing devices 700A and 700B are connected via a network. Specifically, the connection to the network is achieved through a communication interface within each computing device. In this possible implementation, memory 706 within computing device 700A stores instructions for executing the functions of a receiving module. Simultaneously, memory 706 within computing device 700B stores instructions for executing the functions of a processing module.

[0194] It should be understood that Figure 9 The functionality of the computing device 700A shown in FIG. 7 may also be implemented by multiple computing devices 700. Similarly, the functionality of the computing device 700B may also be implemented by multiple computing devices 700.

[0195] An embodiment of the present application further provides a chip, which includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit. The processing unit may execute computer-executable instructions stored in the storage unit so that the chip in the electronic device executes the method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), etc.

[0196] For details, please refer to Figure 10 , Figure 10This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 1000. NPU 1000 is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1003, which is controlled by controller 1004 to extract matrix data from memory and perform multiplication operations.

[0197] In some implementations, the arithmetic circuit 1003 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1003 is a two-dimensional systolic array. The arithmetic circuit 1003 can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1003 is a general-purpose matrix processor.

[0198] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1002 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1001 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1008.

[0199] Unified memory 1006 is used to store input and output data. Weight data is directly transferred to weight memory 1002 through the Direct Memory Access Controller (DMAC) 1005. Input data is also transferred to unified memory 1006 through the DMAC.

[0200] BIU stands for Bus Interface Unit, i.e., bus interface unit 1010 , which is used for interaction between the AXI bus, DMAC, and instruction fetch buffer (IFB) 1009 .

[0201] The bus interface unit 1010 (BIU) is used for the instruction fetch memory 1009 to obtain instructions from the external memory, and is also used for the storage unit access controller 1005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0202] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1006 or transfer weight data to the weight memory 1002 or transfer input data to the input memory 1001.

[0203] The vector calculation unit 1007 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 1003, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0204] In some implementations, the vector calculation unit 1007 can store the processed output vector to the unified memory 1006. For example, the vector calculation unit 1007 can apply a linear function or a nonlinear function to the output of the operation circuit 1003, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1007 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1003, for example, for use in subsequent layers in a neural network.

[0205] An instruction fetch buffer 1009 connected to the controller 1004 is used to store instructions used by the controller 1004;

[0206] Unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch memory 1009 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0207] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0208] See Figure 11 , Figure 11 This application also provides a computer-readable storage medium. In some embodiments, the above Figure 2 The disclosed methods may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of manufacture.

[0209] Figure 11 Schematically illustrates a conceptual partial view of an example computer-readable storage medium including a computer program for executing a computer process on a computing device, arranged in accordance with at least some embodiments presented herein.

[0210] In one embodiment, the computer readable storage medium 1100 is provided using a signal bearing medium 1101. The signal bearing medium 1101 may include one or more program instructions 1102, which when executed by one or more processors may provide the above-mentioned instructions for Figure 2 Describes the functionality or part of the functionality.

[0211] In some examples, signal bearing medium 1101 may include computer readable medium 1103 such as, but not limited to, a hard drive, compact disk (CD), digital video disk (DVD), digital tape, memory, ROM or RAM, and the like.

[0212] In some embodiments, the signal-bearing medium 1101 may include a computer-recordable medium 1104, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, or the like. In some embodiments, the signal-bearing medium 1101 may include a communication medium 1105, such as, but not limited to, a digital and / or analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communication link, a wireless communication link, or the like). Thus, for example, the signal-bearing medium 1101 may be communicated by a wireless form of the communication medium 1105 (e.g., a wireless communication medium conforming to the IEEE 802.X standard or other transmission protocol).

[0213] The one or more program instructions 1102 may be, for example, computer-executable instructions or logic-implemented instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to the program instructions 1102 communicated to the computing device via one or more of computer-readable media 1103, computer-recordable media 1104, and / or communication media 1105.

[0214] It should also be noted that the device embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0215] The present application also provides a computer program product containing instructions. The computer program product may be software or a program product containing instructions that can be run on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method described in the above embodiments.

[0216] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods of each embodiment of the present application.

[0217] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0218] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, training equipment or data center to another website, computer, training equipment or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training equipment, data center, etc. that includes one or more available media integrations. Available media can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (SSD)).

Claims

1. A model processing method, characterized in that: include: detecting a first concept drift value of the first model at a first time point, where the first concept drift value indicates the extent to which performance of the first model has degraded due to a change in distribution of input data of the first model; In response to the first concept drift value satisfying a first condition, using input data of the first model acquired after the first time point to train a second model, where the second model is used to replace the first model; During the training process of the second model, the training hyperparameters of the second model are dynamically adjusted as the concept drift value of the first model changes during operation.

2. The method according to claim 1, characterized in that The method further comprises: detecting a second concept drift value of the first model at a second time point and a performance value of the second model at the second time point, where the second time point is after the first time point; determining a replacement decision value based on the second concept drift value and the performance value; When the replacement decision value satisfies a second condition, the first model is replaced by the second model.

3. The method according to claim 2, characterized in that The determining a replacement decision value based on the second concept drift value and the performance value includes: Normalizing the second concept drift value and the performance value respectively to obtain a normalized second concept drift value and a normalized performance value; A weighted sum is performed on the normalized second concept drift value and the normalized performance value to obtain the replacement decision value.

4. The method according to any one of claims 1 to 3, characterized in that The training hyperparameters include learning rate; In a case where the concept drift value is positively correlated with the degree of degradation of the first model performance, the learning rate is positively correlated with the concept drift value; In a case where the concept drift value is negatively correlated with the degree of degradation of the first model performance, the learning rate is negatively correlated with the concept drift value.

5. The method according to any one of claims 1 to 4, characterized in that The second model is a decision tree model, and the training hyperparameter includes the number of feature selections of the decision tree; In the case where the concept drift value is positively correlated with the degree of performance degradation of the first model, the number of selected features is negatively correlated with the concept drift value; In a case where the concept drift value is negatively correlated with the degree of performance degradation of the first model, the number of selected features is positively correlated with the concept drift value.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Obtaining a calculation method for a monitoring indicator input by a user, wherein the monitoring indicator is used to characterize the performance of the model; During the operation of the target model, calculating the value of the monitoring indicator corresponding to the target model based on the calculation method, the target model being the first model or the second model; When the value of the monitoring indicator reaches the indicator alarm value, a model performance alarm is triggered.

7. The method according to claim 6, characterized in that The monitoring indicator is calculated from a plurality of performance indicators, and each of the plurality of performance indicators is calculated from one or more sub-indicators; Calculating the value of the monitoring indicator corresponding to the target model based on the calculation method includes: Determining a sub-indicator set based on the multiple performance indicators, the sub-indicator set including multiple sub-indicators used to calculate the multiple performance indicators; The values ​​of the multiple sub-indicators corresponding to the target model are calculated, and the value of the monitoring indicator is determined based on the values ​​of the multiple sub-indicators.

8. The method according to any one of claims 1 to 7, characterized in that Both the first model and the second model are classification models for performing classification tasks.

9. The method according to any one of claims 1 to 8, characterized in that The input data of the first model and the second model are tabular data.

10. A model processing device, characterized in that: include: a processing module, configured to detect a first concept drift value of the first model at a first time point, wherein the first concept drift value is used to indicate the extent to which the performance of the first model has deteriorated due to a change in the distribution of input data of the first model; The processing module is further configured to, in response to the first concept drift value satisfying a first condition, use input data of the first model acquired after the first time point to train a second model, where the second model is used to replace the first model; During the training process of the second model, the training hyperparameters of the second model are dynamically adjusted as the concept drift value of the first model changes during operation.

11. The device according to claim 10, characterized in that The processing module is further configured to: detecting a second concept drift value of the first model at a second time point and a performance value of the second model at the second time point, where the second time point is after the first time point; determining a replacement decision value based on the second concept drift value and the performance value; When the replacement decision value satisfies a second condition, the first model is replaced by the second model.

12. The device according to claim 11, characterized in that The processing module is further configured to: Normalizing the second concept drift value and the performance value respectively to obtain a normalized second concept drift value and a normalized performance value; A weighted sum is performed on the normalized second concept drift value and the normalized performance value to obtain the replacement decision value.

13. The device according to any one of claims 10 to 12, characterized in that: The training hyperparameters include learning rate; In a case where the concept drift value is positively correlated with the degree of degradation of the first model performance, the learning rate is positively correlated with the concept drift value; In a case where the concept drift value is negatively correlated with the degree of degradation of the first model performance, the learning rate is negatively correlated with the concept drift value.

14. The device according to any one of claims 10 to 13, characterized in that: The second model is a decision tree model, and the training hyperparameter includes the number of feature selections of the decision tree; In the case where the concept drift value is positively correlated with the degree of performance degradation of the first model, the number of selected features is negatively correlated with the concept drift value; In a case where the concept drift value is negatively correlated with the degree of performance degradation of the first model, the number of selected features is positively correlated with the concept drift value.

15. The device according to any one of claims 10 to 14, characterized in that: The device further comprises: An acquisition module is used to obtain a calculation method of a monitoring indicator input by a user, where the monitoring indicator is used to characterize the performance of the model; The processing module is further configured to calculate, during operation of the target model, a value of the monitoring indicator corresponding to the target model based on the calculation method, where the target model is the first model or the second model; The processing module is further configured to trigger a model performance alarm when the value of the monitoring indicator reaches an indicator alarm value.

16. The device according to claim 15, characterized in that The monitoring indicator is calculated from a plurality of performance indicators, and each of the plurality of performance indicators is calculated from one or more sub-indicators; The processing module is further configured to: Determining a sub-indicator set based on the multiple performance indicators, the sub-indicator set including multiple sub-indicators used to calculate the multiple performance indicators; The values ​​of the multiple sub-indicators corresponding to the target model are calculated, and the value of the monitoring indicator is determined based on the values ​​of the multiple sub-indicators.

17. The device according to any one of claims 10 to 16, characterized in that Both the first model and the second model are classification models for performing classification tasks.

18. A computing device cluster, characterized in that: comprising at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.

19. A computer storage medium, characterized in that The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 9.

20. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 9.

Images