A model stability detection method, device and equipment

Abstract

The present application discloses a method, device and equipment for detecting model stability, which obtains base period labeled samples, base period unlabeled samples and current period unlabeled samples at a base period time point, where the base period time point is earlier than the current period time point. The base period labeled samples are used to train a base period decision tree model. The base period decision tree model includes at least one path from the root node to the leaf node. The base period decision tree model is applied to predict the base period unlabeled samples, and the first sample statistics corresponding to each path after the prediction are obtained to form a first numerical set. The base period decision tree model is applied to predict the current period unlabeled samples, and the second sample statistics corresponding to each path after the prediction are obtained to form a second numerical set. Since the sample statistics will be different when the distribution of the input data in the sample changes, it can be determined whether data drift has occurred based on the first distribution difference result between the first numerical set and the second numerical set to detect the stability of the model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and equipment for detecting model stability. Background Art

[0002] In machine learning and deep learning, models are typically trained and then used in specific application scenarios. Typically, over time, the distribution of the model's input data may change, resulting in data drift. The relationship between the input data and the model's target may also change, resulting in concept drift.

[0003] It can be seen that data drift, concept drift, etc. will cause the stability of the trained model to decrease. Therefore, how to detect the stability of the trained model is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the present application provides a model stability detection method, device and equipment, which can detect the stability of a trained model and determine where its problems lie.

[0005] To solve the above problems, the technical solutions provided by this application are as follows:

[0006] In a first aspect, the present application provides a model stability detection method, the method comprising:

[0007] Obtaining a base period labeled sample and a base period unlabeled sample at a base period time point, as well as a current period unlabeled sample at a current period time point; the base period time point is earlier than the current period time point;

[0008] Training a decision tree model based on the base period labeled samples to obtain a base period decision tree model; the base period decision tree model includes at least one path from a root node to a leaf node;

[0009] Applying the base period decision tree model to predict the base period unlabeled samples, obtaining a first sample statistic corresponding to each of the paths after prediction, and forming a first value set from the first sample statistics corresponding to each of the paths;

[0010] Applying the base period decision tree model to predict the unlabeled samples of the current period, obtaining a second sample statistic corresponding to each of the paths after the prediction, and forming a second numerical value set by the second sample statistics corresponding to each of the paths; the first sample statistic and the second sample statistic are statistics of the same type;

[0011] A first distribution difference result between the first value set and the second value set is obtained, and when the first distribution difference result meets a first preset condition, it is determined that data drift occurs.

[0012] In a second aspect, the present application provides a model stability detection device, the device comprising:

[0013] A first acquisition unit is configured to acquire a base period labeled sample and a base period unlabeled sample at a base period time point, and a current period unlabeled sample at a current period time point; the base period time point is earlier than the current period time point;

[0014] A training unit, configured to train a decision tree model based on the base period labeled samples to obtain a base period decision tree model; the base period decision tree model includes at least one path from a root node to a leaf node;

[0015] a first prediction unit, configured to apply the base period decision tree model to predict the base period unlabeled samples, obtain a first sample statistic corresponding to each of the paths after prediction, and form a first value set from the first sample statistics corresponding to each of the paths;

[0016] a second prediction unit, configured to apply the base period decision tree model to predict the unlabeled samples of the current period, obtain a second sample statistic corresponding to each of the paths after the prediction, and form a second numerical value set from the second sample statistics corresponding to each of the paths; the first sample statistic and the second sample statistic are statistics of the same type;

[0017] The second acquiring unit is configured to acquire a first distribution difference result between the first value set and the second value set, and determine that data drift occurs when the first distribution difference result meets a first preset condition.

[0018] In a third aspect, the present application provides an electronic device, comprising:

[0019] one or more processors;

[0020] a storage device having one or more programs stored thereon,

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-described model stability detection methods.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-described model stability detection methods.

[0023] It can be seen that this application has the following beneficial effects:

[0024] The present application provides a model stability detection method, apparatus, and device, which obtain base period labeled samples and base period unlabeled samples at a base period time point, as well as current period unlabeled samples at a current period time point, where the base period time point is earlier than the current period time point. A decision tree model is trained based on the base period labeled samples to obtain a base period decision tree model. The base period decision tree model includes at least one path from a root node to a leaf node. To detect whether sample data at the current period time point has drifted, the base period decision tree model is used to predict the base period unlabeled samples, and a first sample statistic corresponding to each predicted path is obtained. The first sample statistics corresponding to each path are respectively formed into a first value set. Furthermore, the base period decision tree model is used to predict the current period unlabeled samples, and a second sample statistic corresponding to each predicted path is obtained. The second sample statistics corresponding to each path are respectively formed into a second value set. The first sample statistic and the second sample statistic are statistics of the same type. A first distribution difference result is obtained between the first value set and the second value set, and the first distribution difference result can represent the distribution difference between the first sample statistic and the second sample statistic corresponding to each path. When the first distribution difference result satisfies the first preset condition, it is determined that data drift has occurred. That is, relative to the sample data at the base period time point, the sample data at the current period time point has drifted, which may reduce the stability of the model.

[0025] It can be seen that data drift refers to the change in the distribution of the model's input data over time. The input data corresponds to input data features, which are internal nodes in the base decision tree model path. When the base decision tree model is used to predict samples, the input data features in the path are passed through during the prediction process. As time goes by, when the distribution of input data features changes, the paths of the base unlabeled samples and the current unlabeled samples through the base decision tree model will be different, resulting in changes in the sample statistics corresponding to the same path. Therefore, based on the distribution difference results of the first sample statistic and the second sample statistic corresponding to each path, it is possible to more accurately determine whether data drift has occurred and thus detect the stability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of a framework of an exemplary application scenario provided in an embodiment of the present application;

[0027] Figure 2 A flow chart of a model stability detection method provided in an embodiment of the present application;

[0028] Figure 3 A schematic diagram of a base period decision tree model provided in an embodiment of the present application;

[0029] Figure 4A flowchart of another model stability detection method provided in an embodiment of the present application;

[0030] Figure 5 A schematic structural diagram of a model stability detection device provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the basic structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0033] In order to facilitate understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described below.

[0034] It is understandable that before using the technical solutions of each embodiment of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0035] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the disclosed technical solution based on the prompt message.

[0036] As an optional but non-limiting implementation, in response to a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0037] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0038] In machine learning and deep learning, models are typically trained and used in specific application scenarios. For example, a model could be used to predict whether a watermelon is good quality or whether a product is high-quality. A model typically represents the mapping relationship between input data and output data. Input data corresponds to input data features, and output data is the data output by the model after the input data is input into the model. For example, for a model that predicts whether a product is high-quality, the input data features might include the product's page views and click-through rate, and the output data could be two output results: "High-quality product" or "Not high-quality product." The trained model can then represent the mapping relationship between input data features such as page views and click-through rate and the output data indicating whether a product is high-quality. That is, once a product's page views and click-through rate are determined, the model can determine whether the product is high-quality.

[0039] Typically, as time goes by, the distribution of the model's input data may change, resulting in data drift. It is understandable that since the model represents the mapping relationship between the input data and the output data at the time of training. If the input data shifts over time, the trained model's prediction accuracy for the input data with data shift will decrease. Taking the model that predicts whether a product is a high-quality product as an example, the input data during model training includes the product's page views, click-through rate, etc. The products in the input data used during model training may have been popular at the time, so their page views and click-through rates are both high (they can be considered high-quality products). However, as time goes by, the popularity of this type of product decreases, and its page views, click-through rate, etc. all decrease. At this time, when predicting this type of product based on the trained model, the prediction accuracy may be reduced.

[0040] Over time, the relationship between input data and model targets may change, resulting in concept drift. The model target is the model's output data (or output result). That is, the mapping relationship between the model's input data and output data may change. For example, for a model that predicts whether a product is a high-quality product, during model training, if the product's page views exceed 600 times per minute and its click-through rate exceeds 400 times per minute, the product is considered a high-quality product. However, over time, the method for determining whether a product is a high-quality product changes (indicating a change in the relationship between the input data and the model target). For example, if a product's page views exceed 500 times per minute and its click-through rate exceeds 300 times per minute, the product is considered a high-quality product. Therefore, the trained model's prediction accuracy for input data that has undergone concept drift will decrease.

[0041] It can be seen that data drift, concept drift, etc. will cause the stability of the trained model to decrease. Therefore, how to detect the stability of the trained model is an urgent problem to be solved.

[0042] Based on this, embodiments of the present application provide a model stability detection method, apparatus, and device. These methods obtain base period labeled samples and base period unlabeled samples at a base period time point, as well as current period unlabeled samples at a current period time point, where the base period time point is earlier than the current period time point. A decision tree model is trained based on the base period labeled samples to obtain a base period decision tree model. The base period decision tree model includes at least one path from a root node to a leaf node. To detect whether sample data at the current period time point has drifted, the base period decision tree model is used to predict the base period unlabeled samples, obtaining a first sample statistic corresponding to each predicted path, and forming a first value set from the first sample statistics corresponding to each path. Furthermore, the base period decision tree model is used to predict the current period unlabeled samples, obtaining a second sample statistic corresponding to each predicted path, and forming a second value set from the second sample statistics corresponding to each path. The first sample statistic and the second sample statistic are statistics of the same type. A first distribution difference result is obtained between the first value set and the second value set, and the first distribution difference result can represent the distribution difference between the first sample statistic and the second sample statistic corresponding to each path. When the first distribution difference result satisfies the first preset condition, it is determined that data drift has occurred, that is, data drift has occurred in the sample data at the current time point relative to the sample data at the base time point.

[0043] It can be seen that data drift refers to the change in the distribution of the model's input data over time. The input data corresponds to input data features, which are internal nodes in the base decision tree model path. When the base decision tree model is used to predict samples, the input data features in the path are passed through during the prediction process. As time goes by, when the distribution of input data features changes, the paths of the base unlabeled samples and the current unlabeled samples through the base decision tree model will be different, resulting in changes in the sample statistics corresponding to the same path. Therefore, based on the distribution difference results of the first sample statistic and the second sample statistic corresponding to each path, it is possible to more accurately determine whether data drift has occurred and thus detect the stability of the model.

[0044] It is understandable that the defects in the above solutions are the results obtained by the applicant after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present application below should be the applicant's contribution to the embodiments of the present application during the application process.

[0045] In order to facilitate understanding of the model stability detection method provided in the embodiment of the present application, the following Figure 1 See the example scenario shown. Figure 1As shown, this figure is a framework diagram of an exemplary application scenario provided by an embodiment of the present application. The method can be applied to terminal devices and / or servers, and is not limited here. Among them, terminal devices include but are not limited to smartphones, tablets, laptops, smart watches, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms. Specifically, it can be applied to the back end of the target application installed in the terminal device and / or server. The target application is an application installed in the terminal device and / or server, and the target application is not limited here.

[0046] In actual applications, the terminal device and / or server obtains the base period labeled samples and the base period unlabeled samples at the base period time point, as well as the current period unlabeled samples at the current period time point. The base period time point is earlier than the current period time point. The samples at the base period time point include the base period labeled samples and the base period unlabeled samples. The current period unlabeled samples at the current period time point belong to the samples at the current period time point. The input data features included in the samples at the base period time point and the samples at the current period time point are the same. The terminal device and / or server trains a decision tree model based on the base period labeled samples to obtain a base period decision tree model. The base period decision tree model includes at least one path from the root node to the leaf node. When using the base period decision tree model for prediction, the path in the base period decision tree model will be executed / passed through.

[0047] The terminal device and / or server uses the base period decision tree model to predict the base period unlabeled samples. During the prediction process, the paths in the base period decision tree model are executed to obtain the first sample statistics corresponding to each path after the prediction. The first sample statistics corresponding to each path respectively constitute a first numerical set. In addition, the terminal device and / or server uses the base period decision tree model to predict the current period unlabeled samples. During the prediction process, the paths in the base period decision tree model are also executed to obtain the second sample statistics corresponding to each path after the prediction, and the second sample statistics corresponding to each path respectively constitute a second numerical set.

[0048] To detect whether sample data at a given time point has drifted, the terminal device and / or server obtains a first distribution difference result between the first value set and the second value set. Whether data drift has occurred is determined based on the first distribution difference result. For example, if the first distribution difference result satisfies a first preset condition, data drift is determined to have occurred.

[0049] Those skilled in the art will understand that Figure 1 The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.

[0050] To facilitate understanding of the present application, a model stability detection method provided in an embodiment of the present application is described below with reference to the accompanying drawings.

[0051] See also Figure 2 As shown in FIG, this figure is a flow chart of a model stability detection method provided by an embodiment of the present application, as shown in FIG. Figure 2 As shown, the method may include S201-S205:

[0052] S201: Obtain base period labeled samples and base period unlabeled samples at the base period time point, as well as current period unlabeled samples at the current period time point; the base period time point is earlier than the current period time point.

[0053] To test the stability of a model, you need to determine whether data drift occurs between the input data used during model training and the input data to be predicted after the model is deployed, and / or whether concept drift occurs between model training and after the model is deployed.

[0054] Based on this, it is necessary to compare the input data and / or model effects at two time points. In an embodiment of the present application, these two time points are respectively expressed as a base period time point and a current period time point. Wherein, the base period time point is a preceding time point, i.e., the base period time point is earlier than the current period time point. It is understandable that the time length represented by the base period time point and the current period time point is not limited here. For example, the base period time point and the current period time point can be two moments or two time periods. The time interval between the base period time point and the current period time point is not limited here either, and can be determined according to actual conditions.

[0055] The labeled samples obtained at the base period time point are called base period labeled samples. The unlabeled samples obtained through the actual online use of the model at the base period time point are called base period unlabeled samples. In addition, the labeled samples obtained at the current period time point are called current period labeled samples. The unlabeled samples obtained through the actual online use of the model at the current period time point are called current period unlabeled samples. The difference between labeled samples and unlabeled samples is that unlabeled samples (such as base period unlabeled samples and current period unlabeled samples) include input data, while labeled samples (such as base period labeled samples and current period labeled samples) include not only input data but also the labels corresponding to the input data.

[0056] To improve the accuracy of model stability testing, the sample sizes of the base period labeled samples, base period unlabeled samples, current period labeled samples, and current period unlabeled samples can be as large and as identical as possible. However, this application does not limit the sample sizes of the base period labeled samples, base period unlabeled samples, current period labeled samples, and current period unlabeled samples. That is, the sample sizes of the base period labeled samples, base period unlabeled samples, current period labeled samples, and current period unlabeled samples are also allowed to be different.

[0057] In the embodiments of the present application, whether at the base period time point or the current period time point, whether used for model training or model prediction, the data used are referred to as sample data, referred to as samples for short. The sample data includes input data. It is understandable that when training the model, the input data in the base period labeled samples are used to train the model. When predicting the model (i.e., using the model for prediction after the model is deployed), the trained model is used to predict the input data in the unlabeled sample data (e.g., base period unlabeled samples and / or current period unlabeled samples).

[0058] The input data corresponds to input data features. Input data features are data obtained after feature representation (such as vector representation) of the input data, which is not limited here. It can be understood that the input data features in the sample data used are the same, whether at the base time point or at the current time point, whether for model training or model prediction, except that the feature values of the input data features may be different. For example, when using a model to predict high-quality goods, the input data features are features such as page views and click-through rate. The values of page views, click-through rate, etc. may be different at the base time point and the current time point.

[0059] S202: Training a decision tree model based on base period labeled samples to obtain a base period decision tree model; the base period decision tree model includes at least one path from a root node to a leaf node.

[0060] In the embodiments of the present application, the model to be tested for stability specifically refers to a model trained on labeled samples from a base period. Since the stability of the model is related to data drift and concept drift that occur over time, it is necessary to test whether the distribution of the input data has experienced data drift and whether the mapping relationship between the input data and the output data has experienced concept drift.

[0061] Based on this, the model to be tested for stability can be the base period decision tree model in this step, or other models that have been trained based on base period labeled samples (such as other neural network models, etc.). It should be understood that regardless of whether the model to be tested for stability is the base period decision tree model in this step, or other models that have been trained based on base period labeled samples, the base period decision tree model in this step serves the purpose of testing the stability of the model.

[0062] Labeled samples (e.g., labeled samples from the base period and labeled samples from the current period) include not only the input data but also the labels corresponding to the input data. For example, when the input data is the number of page views or click-through rate of a product, the corresponding labels may be two labels, such as "high-quality product" or "low-quality product." When training a decision tree model based on labeled samples from the base period, the input data from the labeled samples from the base period and the labels corresponding to the input data are used to train the decision tree model, resulting in a base period decision tree model.

[0063] Decision tree models have a tree-like structure, consisting of internal nodes and leaf nodes. Decision tree models can predict output outcomes corresponding to input data. Decision tree models include classification tree models and regression tree models, which are not limited here. For example, in classification problems, decision tree models are used to classify input data features, with leaf nodes representing a category.

[0064] See also Figure 3 , Figure 3 A schematic diagram of a base period decision tree model provided in an embodiment of the present application, Figure 3 This is only for illustration and does not constitute a limitation. Figure 3 As shown, the boxes represent the internal nodes of the base decision tree model, and the ovals represent the leaf nodes of the base decision tree model. The internal nodes include input data feature 1 and input data feature 2. The leaf nodes include output 1 and output 2. If the base decision tree is a classification tree, output 1 and output 2 represent two classification results. For example, when judging the quality of a product, input data feature 1 is the number of views, input data feature 2 is the click-through rate, output 1 is a high-quality product, and output 2 is a non-high-quality product. It can be understood that output 1 corresponds to label 1, for example, both are high-quality products, and output 2 corresponds to label 2, for example, both are non-high-quality products. It is just that output 1, output 2, etc. are the output results of the base decision tree model, which are predicted values, and the labels represent the actual expected results, which are used in the training process of the base decision tree model.

[0065] When predicting input data based on the base period decision tree model, conditional judgment will be performed. Figure 3 As shown, first determine whether input data feature 1 in the input data satisfies feature value 1. If the feature value of input data feature 1 satisfies feature value 1 of input data feature 1, output 1 is obtained. Otherwise, when the feature value of input data feature 1 is feature value 2 of input data feature 1, determine input data feature 2 in the input data. If the feature value of input data feature 2 is feature value 1 of input data feature 2, output 1 is obtained. If the feature value of input data feature 2 is feature value 2 of input data feature 2, output 2 is obtained. It is understandable that the judgment condition in the base period decision tree model can also be a feature value range, which is not limited here.

[0066] The base decision tree model includes at least one path from the root node to the leaf node. The number of paths is the same as the number of leaf nodes. Figure 3 The base period decision tree model shown includes three paths. The first path (path 1) is the path from input data feature 1 to output 1; the second path (path 2) is the path from input data feature 1 to input data feature 2, and then to output 1; the third path (path 3) is the path from input data feature 1 to input data feature 2, and then to output 2.

[0067] As an optional example, each path from the root node to the leaf node of the base period decision tree model can be numbered, for example, the paths are numbered path_0, path_1, ..., path_n, where n is the number of paths and n is a positive integer. Figure 3 The three paths in the example may be numbered path_0, path_1, and path_2, respectively, to represent path 1, path 2, and path 3. In addition, path_x may be used to represent any one of the paths.

[0068] Each path of the base decision tree model corresponds to at least one single feature and / or at least one cross feature. The path is associated with the single feature and / or cross feature on the path. Among them, the single feature is the input data feature, and the cross feature can also be understood as a combination feature, specifically the combination feature of the input data feature. Figure 3 As shown, on the first path, the single feature is input data feature 1. On the second path, there is a cross feature consisting of input data feature 1 and input data feature 2. The input data features associated with the second path are input data feature 1 and input data feature 2. The second path is also associated with a cross feature consisting of input data feature 1 and input data feature 2.

[0069] S203: Applying the base period decision tree model to predict the base period unlabeled samples, obtaining a first sample statistic corresponding to each path after prediction, and forming a first value set from the first sample statistics corresponding to each path.

[0070] After obtaining the base-period decision tree model, we use it to predict each base-period unlabeled sample. This effectively means predicting the input data for each base-period unlabeled sample. During the prediction process, the base-period decision tree model evaluates the input data for each base-period unlabeled sample, ultimately reaching the leaf node of the path. After the prediction is complete, each base-period unlabeled sample corresponds to a path in the base-period decision tree model. The same path may correspond to one or more base-period unlabeled samples.

[0071] For example, the path executed corresponding to the base period unlabeled sample 1 is path 1, the path executed corresponding to the base period unlabeled sample 2 is path 1, the path executed corresponding to the base period unlabeled sample 3 is path 2, the path executed corresponding to the base period unlabeled sample 4 is path 3, the path executed corresponding to the base period unlabeled sample 5 is path 3, and so on.

[0072] Thus, the sample statistics corresponding to each path after the base period unlabeled sample prediction is completed can be obtained, which is called the first sample statistic, and the first sample statistics corresponding to each path are composed of the first value set. In addition, the first value set can also include the path number of the path. That is, the first value set can include the path number of the path and the first sample statistics corresponding to each path (which can also be called the first sample statistics corresponding to each path number). In the first value set, the corresponding first sample statistics can also be displayed in order according to the path number.

[0073] The first sample statistic corresponding to the path is used to indicate the number of base period unlabeled samples that execute the path during the prediction process.

[0074] As an optional example, the first sample statistic corresponding to the path may be the number of base period unlabeled samples executing the path. It can be seen that the number of base period unlabeled samples executing the path can represent the number of base period unlabeled samples executing the path.

[0075] As another optional example, the first sample statistic corresponding to the path can also be the ratio of the number of base period unlabeled samples executing the path to the total number of base period unlabeled samples. It can be seen that this ratio can also represent the number of base period unlabeled samples executing the path.

[0076] Based on this, the embodiment of the present application provides a specific implementation method for obtaining the first sample statistic corresponding to each path after prediction, and forming a first value set from the first sample statistics corresponding to each path, including:

[0077] Calculate the first ratio corresponding to each path after prediction, and form a first numerical set by the first ratios corresponding to each path; the first ratio corresponding to the path is the ratio of the number of base period unlabeled samples passing through the path after prediction to the total number of base period unlabeled samples.

[0078] Let path_x represent any of the paths. The first ratio corresponding to path_x is the quotient of the number of base-period unlabeled samples passing through the path divided by the total number of base-period unlabeled samples. The first ratio corresponding to path_x can also be understood as the proportion of base-period unlabeled samples used by path_x during the prediction process.

[0079] It can be understood that compared to using the number of base period unlabeled samples of the execution path to represent the first sample statistic, the proportion value can intuitively represent the distribution of unlabeled samples used in each path. Using the first proportion to represent the first sample statistic and performing subsequent difference comparisons will make the stability test results of the subsequent model more accurate.

[0080] S204: Apply the base period decision tree model to predict the unlabeled samples of this period, obtain the second sample statistic corresponding to each path after the prediction, and form a second numerical set by the second sample statistics corresponding to each path; the first sample statistic and the second sample statistic are statistics of the same type.

[0081] To test the model's stability, the base-period decision tree model is used to predict each current-period unlabeled sample. Similar to S203 , after the prediction is complete, each current-period unlabeled sample corresponds to a path in the base-period decision tree model. The same path may correspond to one or more current-period unlabeled samples.

[0082] Thus, the sample statistics corresponding to each path after the unlabeled sample prediction of this period are obtained, which are called second sample statistics. The second sample statistics corresponding to each path respectively form a second value set. In addition, the second value set can also include the path number of the path. That is, the second value set includes the path number of the path and the second sample statistics corresponding to each path respectively (which can also be called the second sample statistics corresponding to each path number). In the second value set, the second sample statistics can also be displayed in order by path number.

[0083] The second sample statistic corresponding to the path is used to indicate the number of unlabeled samples in the current period that execute the path during the prediction process.

[0084] As an optional example, the second sample statistic corresponding to the path may be the number of unlabeled samples in the current period when the path is executed.

[0085] As another optional example, the second sample statistic corresponding to the path may also be the ratio of the number of unlabeled samples in the current period that execute the path to the total number of unlabeled samples in the current period.

[0086] Based on this, the embodiment of the present application provides a specific implementation method for obtaining the second sample statistic corresponding to each path after prediction, and forming a second value set from the second sample statistics corresponding to each path, including:

[0087] Calculate the second ratio corresponding to each path after prediction, and form a second numerical set of the second ratios corresponding to each path; the second ratio corresponding to the path is the ratio of the number of unlabeled samples in this period that pass through the path after prediction to the total number of unlabeled samples in this period.

[0088] Let path_x represent any of the paths. The second ratio corresponding to path_x is the quotient of the number of unlabeled samples in the current period that pass through the path and the total number of unlabeled samples in the current period. The second ratio corresponding to path_x can also be understood as the proportion of unlabeled samples in the current period that path_x uses during the prediction process.

[0089] It can be understood that compared to using the number of unlabeled samples in the current execution path to represent the second sample statistic, the proportion value can intuitively represent the distribution of unlabeled samples used in each path. Using the second proportion to represent the second sample statistic and performing subsequent difference comparisons will make the stability test results of the subsequent model more accurate.

[0090] S205: Obtain a first distribution difference result between the first value set and the second value set, and determine that data drift occurs when the first distribution difference result meets a first preset condition.

[0091] The first distribution difference result of the first value set and the second value set is mainly a distribution difference result between the first sample statistics corresponding to each path in the first value set and the second sample statistics corresponding to each path in the second value set.

[0092] It is understandable that as time goes by, when the distribution of input data features changes, the paths that the base period unlabeled samples and the current period unlabeled samples take through the base period decision tree model will be different, causing the sample statistics corresponding to the same path to change. Therefore, based on the distribution difference results of the first sample statistics and the second sample statistics corresponding to each path, it is possible to more accurately determine whether data drift has occurred. For example, when the first distribution difference result meets the first preset condition, it is determined that a model stability problem of the data drift type has occurred. This data drift refers to the change in the distribution of the sample data at the current time point relative to the sample data at the base period time point over time.

[0093] In one possible implementation, an embodiment of the present application provides a method for obtaining a first distribution difference result between a first set of values and a second set of values, and determining that data drift has occurred when the first distribution difference result satisfies a first preset condition, including:

[0094] Based on the first sample statistics corresponding to each path in the first value set and the second sample statistics corresponding to each path in the second value set, a first KL divergence of the first value set and the second value set is calculated; when the first KL divergence meets a first threshold range, it is determined that data drift has occurred.

[0095] KL divergence can also be called KL distance or relative entropy. In the embodiment of the present application, KL divergence is used to measure the difference between two distributions. That is, the first KL divergence, as the first distribution difference result, is used to measure the distribution difference between the first sample statistic and the second sample statistic corresponding to each path. When the first distribution difference result is the first KL divergence, the first preset condition is that the first KL divergence satisfies a first threshold range.

[0096] Specifically, the calculation formula of KL divergence is:

[0097]

[0098] Among them, P and Q represent two distributions, i represents each possible event, and D KL (P||Q) is the KL divergence, P i and Q i represent the probability of event i in distribution P and Q respectively.

[0099] In the embodiment of the present application, any path path_x can be used as the value of i. For example, when there are 3 paths in total, the values of i are 1, 2, and 3. The first sample statistic corresponding to path_x can be used as P i , take the second sample statistic corresponding to path_x as Q i Substitute into the calculation formula to calculate the first KL divergence.

[0100] It can be seen that the larger the first KL divergence is, the greater the difference between the first value set and the second value set is. As an optional example, a first KL threshold is set, and the first threshold range is a range greater than the first KL threshold.

[0101] In addition, the maximum mean difference (MMD) may be used to represent the first distribution difference between the first value set and the second value set. The maximum mean difference (MMD) may also be used to measure the similarity (or difference) between the two distributions, which is not limited here.

[0102] As an optional example, when the first sample statistic is a first proportion and the second sample statistic is a second proportion, the first KL divergence of the first value set and the second value set is calculated based on the first sample statistic corresponding to each path in the first value set and the second sample statistic corresponding to each path in the second value set. Specifically, the first KL divergence of the first value set and the second value set is calculated based on the first proportion and the second proportion corresponding to each path. In this example, the first proportion corresponding to path_x is used as P i , the second ratio corresponding to path_x is used as Q i Substitute into the calculation formula to calculate the first KL divergence.

[0103] For example, the number of unlabeled samples in the base period and the number of unlabeled samples in the current period are both 1000. Figure 3 As shown, the paths are path_0, path_1, ..., path_2, that is, i takes the values of 1, 2, and 3. In S203, the base period unlabeled samples for executing path_0 are 200, the base period unlabeled samples for executing path_1 are 300, and the base period unlabeled samples for executing path_2 are 500. Then the first ratio corresponding to path_0 is 200 / 1000=0.2, the first ratio corresponding to path_1 is 0.3, and the first ratio corresponding to path_2 is 0.5. In S204, the base period unlabeled samples for executing path_0 are 400, the base period unlabeled samples for executing path_1 are 400, and the base period unlabeled samples for executing path_2 are 200. Then the first ratio corresponding to path_0 is 0.4, the first ratio corresponding to path_1 is 0.4, and the first ratio corresponding to path_2 is 0.2. Therefore, by substituting each value into the calculation formula of KL divergence, the first KL divergence can be calculated.

[0104] Based on the relevant contents of S201-S205 above, it can be seen that the present application provides a model stability detection method, which obtains base period labeled samples and base period unlabeled samples at the base period time point, as well as current period unlabeled samples at the current period time point, where the base period time point is earlier than the current period time point. A decision tree model is trained based on the base period labeled samples to obtain a base period decision tree model. The base period decision tree model includes at least one path from the root node to the leaf node. In order to detect whether the sample data at the current period time point has drifted, the base period decision tree model is applied to predict the base period unlabeled samples, and the first sample statistics corresponding to each path after the prediction are obtained, and the first sample statistics corresponding to each path are respectively formed into a first numerical set. In addition, the base period decision tree model is applied to predict the current period unlabeled samples, and the second sample statistics corresponding to each path after the prediction are obtained, and the second sample statistics corresponding to each path are respectively formed into a second numerical set. Wherein, the first sample statistic and the second sample statistic are statistics of the same type. A first distribution difference result is obtained between the first set of values and the second set of values. The first distribution difference result can represent the distribution difference between the first sample statistic and the second sample statistic corresponding to each path. When the first distribution difference result satisfies a first preset condition, data drift is determined to have occurred. That is, data drift has occurred in the sample data at the current time point relative to the sample data at the base time point, which may reduce the stability of the model.

[0105] It can be seen that data drift refers to the change in the distribution of the model's input data over time. The input data corresponds to input data features, which are internal nodes in the base decision tree model path. When the base decision tree model is used to predict samples, the input data features in the path are passed through during the prediction process. As time goes by, when the distribution of input data features changes, the paths of the base unlabeled samples and the current unlabeled samples through the base decision tree model will be different, resulting in changes in the sample statistics corresponding to the same path. Therefore, based on the distribution difference results of the first sample statistic and the second sample statistic corresponding to each path, it is possible to more accurately determine whether data drift has occurred and thus detect the stability of the model.

[0106] Based on the above, we can determine whether a data drift-type model stability issue has occurred. Based on this, we can also attribute features to the input data characteristics that caused the data drift.

[0107] In one possible implementation, the model stability detection method provided in the embodiment of the present application further includes the following steps:

[0108] A1: Calculate the difference between the first sample statistic and the second sample statistic corresponding to each path.

[0109] As an optional example, when the first sample statistic corresponding to the path is the number of unlabeled samples in the base period of executing the path, and the second sample statistic corresponding to the path is the number of unlabeled samples in the current period of executing the path, the gap can be the difference in quantity.

[0110] As another optional example, when the first sample statistic is a first proportion and the second sample statistic is a second proportion, the gap may be a difference between the proportions. Based on this, step A1 specifically calculates the gap between the first proportion and the second proportion corresponding to each path.

[0111] For any path path_x, the difference between its first ratio and second ratio can be represented by path_x_diff. Then, path_x_diff = |first ratio - second ratio|. Here, |·| represents an absolute value.

[0112] A2: Determine a path whose gap satisfies the gap range as a first target path, and determine whether the input data feature associated with the first target path is related to the data drift.

[0113] As an optional example, when the first sample statistic corresponding to a path is the number of unlabeled samples in the base period of executing the path, and the second sample statistic corresponding to the path is the number of unlabeled samples in the current period of executing the path, the difference range is a quantity range. A quantity threshold can be set, and the quantity range is a range greater than the quantity threshold.

[0114] As another optional example, when the first sample statistic is a first ratio and the second sample statistic is a second ratio, the difference range is a ratio range. A ratio threshold can be set, and the quantity range is a range greater than the ratio threshold.

[0115] Each path corresponds to a calculated gap, and a gap that meets the gap range is determined from multiple gaps, and the path corresponding to the gap is determined as the first target path. It can be understood that when the gap is larger, the number of samples that execute the path changes greatly, which may be due to the change in the distribution of input data features on the path over time. Therefore, the input data features associated with the first target path are related to data drift. Furthermore, single features, cross features, etc. associated with the first target path may be a cause of model instability. The number of first target paths may be one or more. When the number of first target paths is multiple, the larger the value of the gap corresponding to the path, the greater the influence of the input data features associated with the path on data drift.

[0116] Based on the relevant contents of A1-A2 above, it can be seen that the embodiments of the present application can not only accurately detect the model stability problems caused by data drift, but also perform feature attribution on data drift to determine the input data features that cause data drift.

[0117] The above content enables model stability testing, provides a method for determining data drift, and implements feature attribution for data drift. However, model stability is also related to concept drift. The following, combined with the accompanying figures, provides a method for determining concept drift to further facilitate model stability testing.

[0118] See also Figure 4 , Figure 4 This is a flow chart of another model stability detection method provided in an embodiment of the present application. Figure 4 As shown, the method includes S401-S404:

[0119] S401: Obtain the labeled samples of this period at the current time point.

[0120] As can be seen from the above S201 content, the labeled samples obtained at the current time point are called the current labeled samples.

[0121] S402: Applying the base period decision tree model to predict the base period labeled samples, and forming a first label set based on the number of sample labels corresponding to each path after prediction.

[0122] After obtaining the base-period decision tree model, the base-period decision tree model can be used to predict the base-period labeled samples, which is actually to predict the input data in the base-period labeled samples. Similarly, during the prediction process, the corresponding path will be executed. After the prediction is completed, each base-period labeled sample will correspond to a path in the base-period decision tree model. The same path may correspond to one or more base-period labeled samples. Since the base-period labeled samples include the labels of the input data, it can be considered that each path may correspond to one or more labels of input data. It can be understood that in the classification problem, since the category represented by the leaf node corresponding to a path is only one category, the labels of one or more input data corresponding to each path should be the same label. In this way, the number of labels of the input data corresponding to each path can be counted, that is, the number of sample labels.

[0123] It can be seen that the number of sample labels corresponding to the path is the same as the number of labeled samples in the base period corresponding to the path. Figure 3 The base period decision tree model shown is for base period labeled sample 1, base period labeled sample 2, base period labeled sample 3, and base period labeled sample 4. After the prediction is completed, base period labeled sample 1 corresponds to path 1, base period labeled sample 2 corresponds to path 1, base period labeled sample 3 corresponds to path 2, and base period labeled sample 4 corresponds to path 3. The label corresponding to path 1 is label 1, and the number of corresponding sample labels is 2; the label corresponding to path 2 is also label 1, and the number of corresponding sample labels is 1; the label corresponding to path 3 is label 2, and the number of corresponding sample labels is 1.

[0124] The first label set also includes the path number of the path. That is, the first label set may include the path number of the path and the number of sample labels corresponding to each path after predicting the labeled samples in the base period (also referred to as the number of sample labels corresponding to each path number). In the first label set, the corresponding number of sample labels may also be displayed in order of path number.

[0125] S403: Apply the base period decision tree model to predict the labeled samples of this period, and form a second label set based on the number of sample labels corresponding to each path after prediction.

[0126] After obtaining the base-period decision tree model, it can be used to predict the current-period labeled samples. This effectively means predicting the input data within the current-period labeled samples. After the prediction is complete, each current-period labeled sample corresponds to a path in the base-period decision tree model. The same path may correspond to one or more current-period labeled samples. The current-period labeled samples include the labels of the input data, and the number of input data labels corresponding to each path can also be counted, i.e., the number of sample labels. In this step, the number of sample labels corresponding to a path is the same as the number of labeled samples in the current period corresponding to that path.

[0127] The second label set also includes the path number of the path. That is, the second label set includes the path number of the path and the number of sample labels corresponding to each path after the prediction of the labeled samples in the current period (also known as the number of sample labels for the labeled samples in the current period corresponding to each path number). In the first label set, the corresponding sample label number can also be displayed in order by path number.

[0128] S404: Obtain a second distribution difference result between the first label set and the second label set, and determine that concept drift occurs when the second distribution difference result meets a second preset condition.

[0129] The second distribution difference result of the first label set and the second label set is mainly the distribution difference result between the number of sample labels corresponding to each path in the first label set and the number of sample labels corresponding to each path in the second label set.

[0130] It is understandable that as time passes from the base period time point to the current period time point, when the mapping relationship between the input data and the model output data changes, the input data in the base period labeled samples and the input data in the current period labeled samples reach different leaf nodes after passing through the base period decision tree model, that is, the output results obtained will be different, which can also be understood as the corresponding labels being different (in the labeled samples, the output results and labels are the same). This causes the number of sample labels corresponding to the same path to change. Therefore, based on the number of sample labels corresponding to each path in the first label set and the number of sample labels corresponding to each path in the second label set, it is possible to more accurately determine whether concept drift has occurred, so as to detect the stability of the model.

[0131] For example, when the second distribution difference result satisfies the second preset condition, concept drift is determined to have occurred. Concept drift refers to a change in the distribution of the mapping relationship between the unlabeled sample data at the current time point and the output data of the decision tree model in the base period, relative to the mapping relationship between the unlabeled sample data at the base period time point and the output data of the decision tree model in the base period, over time. The output data of the decision tree model in the base period is obtained by inputting the unlabeled sample data into the decision tree model in the base period.

[0132] Based on the relevant contents of S401-S404, it can be seen that the embodiment of the present application can not only detect model stability issues caused by data drift, but also detect model stability issues caused by concept drift. In this way, the cause of the model stability issue can be determined more accurately.

[0133] In one possible implementation, an embodiment of the present application provides a specific implementation of obtaining a second distribution difference result between the first label set and the second label set in S404, and determining that concept drift has occurred when the second distribution difference result meets a second preset condition, including:

[0134] B1: Calculate the third ratio of the number of sample labels corresponding to each path in the first label set to the total number of labeled samples in the base period.

[0135] like Figure 3 As shown in the figure, if the base-period decision tree model predicts base-period labeled samples 1, 2, 3, and 4, and the number of base-period labeled samples is 4, then the total number of sample labels is 4. After the prediction is complete, base-period labeled sample 1 corresponds to path 1, base-period labeled sample 2 corresponds to path 1, base-period labeled sample 3 corresponds to path 2, and base-period labeled sample 4 corresponds to path 3. Path 1 corresponds to label 1, and the number of sample labels is 2; path 2 also corresponds to label 1, and the number of sample labels is 1; path 3 corresponds to label 2, and the number of sample labels is 1. Therefore, the third ratio corresponding to path 1 is 1 / 2. The rest is similar and will not be repeated here.

[0136] B2: Calculate the fourth ratio of the number of sample labels corresponding to each path in the second label set to the total number of labeled samples in this period.

[0137] The calculation process of the fourth ratio is similar to that of the third ratio and will not be repeated here.

[0138] B3: Based on the third ratio and the fourth ratio corresponding to each path, calculate the second KL divergence corresponding to each path.

[0139] It is understandable that the calculation formula of the second KL divergence is the same as the above embodiment, and it is only necessary to use the third ratio corresponding to path_x as P i , the fourth ratio corresponding to path_x is taken as Q i Substitute it into the calculation formula, and the resulting KL divergence is the second KL divergence.

[0140] B4: Select the maximum KL divergence among the second KL divergences corresponding to each path.

[0141] After calculating the second KL divergence corresponding to each path, the second KL divergences are compared to obtain the maximum KL divergence. It can be understood that the maximum KL divergence is the second distribution difference result.

[0142] B5: When the maximum KL divergence satisfies the second threshold range, it is determined that concept drift occurs.

[0143] As an optional example, a second KL threshold is set, and the second threshold range is a range larger than the second KL threshold. The magnitude relationship between the first KL threshold and the second KL threshold, as well as the range relationship between the first threshold range and the second threshold range, are not limited here and can be set according to actual conditions.

[0144] In this step, the maximum KL divergence is used to measure the distribution difference between the third proportion and the fourth proportion corresponding to each path. When the maximum KL divergence meets the second threshold range, it indicates that the distribution difference is large, and it is determined that concept drift has occurred.

[0145] Based on the contents of B1-B5, an embodiment of the present application provides a specific implementation method for obtaining the second distribution difference result of the first label set and the second label set, that is, the ratio of the number of sample labels corresponding to the path to the total number of labeled samples in the base period and the ratio of the number of sample labels corresponding to the path to the total number of labeled samples in the current period are compared in distribution, and the distribution comparison result is expressed by KL divergence to determine whether concept drift has occurred.

[0146] Based on the above, we can determine whether a concept drift-type model stability issue has occurred. Based on this, we can also attribute features to the input data characteristics that caused the concept drift.

[0147] In one possible implementation, the model stability detection method provided in the embodiment of the present application further includes the following steps:

[0148] C1: Determine the second target path corresponding to the maximum KL divergence.

[0149] It is understandable that each path corresponds to a second KL divergence. The maximum KL divergence is selected from the second KL divergences, and the path corresponding to the maximum KL divergence is determined as the second target path.

[0150] C2: When the maximum KL divergence satisfies the third threshold range, it is determined that the input data feature associated with the second target path is related to concept drift.

[0151] Since the second KL divergence corresponding to the second target path is the largest, this indicates that the mapping relationship between the input data and the output data of the base-period decision tree model when executing this path at the base-period time point and the current-period time point has significantly changed. Therefore, the second target path has the greatest impact on concept drift. In this case, the input data features associated with the second target path are determined to be related to concept drift. Furthermore, single features and cross-features associated with the second target path may also contribute to model instability.

[0152] As an optional example, a third KL threshold is set, and the third threshold range is a range greater than the third KL threshold.

[0153] Based on the relevant contents of C1-C2 above, it can be seen that the embodiments of the present application can not only accurately detect the model stability problem caused by concept drift, but also perform feature attribution on concept drift to determine the input data features that cause concept drift.

[0154] In one possible implementation, the embodiment of the present application further provides a specific implementation of obtaining a second distribution difference result between the first label set and the second label set in S404, and determining that concept drift has occurred when the second distribution difference result meets a second preset condition, including:

[0155] Calculate the fifth ratio corresponding to each sample label in the first label set, and calculate the sixth ratio corresponding to each sample label in the second label set; the fifth ratio is the ratio of the number of sample labels in the first label set to the total number of labeled samples in the base period, and the sixth ratio is the ratio of the number of sample labels in the second label set to the total number of labeled samples in the current period;

[0156] Calculating a third KL divergence between the first label set and the second label set based on the fifth ratio and the sixth ratio;

[0157] When the third KL divergence satisfies a fourth threshold range, it is determined that concept drift occurs.

[0158] As above, the number of each sample label in the first label set and the second label set can be counted, and the fifth ratio and the sixth ratio can be calculated. The number of each sample label is obtained by the number of sample labels corresponding to each path. For example, Figure 3As shown in the figure, if the label corresponding to path 1 in the first label set is label 1, the number of corresponding sample labels is 2; the label corresponding to path 2 is also label 1, the number of corresponding sample labels is 1; the label corresponding to path 3 is label 2, the number of corresponding sample labels is 1. Then the fifth ratio corresponding to label 1 in the first label set is three-quarters (that is, the sum of the number of sample labels corresponding to path 1 and the number of sample labels corresponding to path 2 is calculated, and the quotient of this sum and the number of labeled samples in the base period is the fifth ratio), and the fifth ratio corresponding to label 2 is one-quarter.

[0159] When calculating the third KL divergence based on the fifth and sixth ratios, the calculation formula of the third KL divergence is the same as the above embodiment, and only the fifth ratio corresponding to path_x is used as P i , the sixth ratio corresponding to path_x is taken as Q i Substitute it into the calculation formula, and the resulting KL divergence is the third KL divergence.

[0160] As an optional example, a fourth KL threshold is set, and the fourth threshold range is a range greater than the fourth KL threshold.

[0161] It's understandable that when the mapping between the input data and model output data at the base and current time points changes, the number of sample labels corresponding to the same path will change, which will also cause the number of the same sample labels to change. Consequently, the distributions of the fifth and sixth ratios corresponding to various sample labels will differ. The third KL divergence calculated based on the fifth and sixth ratios corresponding to various sample labels can reflect these distribution differences and, therefore, can be used to indicate whether concept drift has occurred.

[0162] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods.

[0163] Those skilled in the art will understand that, in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not impose any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0164] Based on the model stability detection method provided in the above method embodiment, the present application also provides a model stability detection device. The model stability detection device is described below with reference to the accompanying drawings. Since the principle of the problem solved by the device in the embodiment of the present disclosure is similar to that of the model stability detection method in the embodiment of the present application, the implementation of the device can be referred to as the implementation of the method, and the repeated parts will not be repeated.

[0165] See also Figure 5As shown in the figure, this figure is a structural schematic diagram of a model stability detection device provided in an embodiment of the present application.

[0166] like Figure 5 As shown, the model stability detection device includes:

[0167] The first acquisition unit 501 is configured to acquire a base period labeled sample and a base period unlabeled sample at a base period time point, and a current period unlabeled sample at a current period time point; the base period time point is earlier than the current period time point;

[0168] A training unit 502 is configured to train a decision tree model based on the base period labeled samples to obtain a base period decision tree model; the base period decision tree model includes at least one path from a root node to a leaf node;

[0169] A first prediction unit 503 is configured to apply the base period decision tree model to predict the base period unlabeled samples, obtain a first sample statistic corresponding to each of the paths after prediction, and form a first value set based on the first sample statistics corresponding to each of the paths;

[0170] The second prediction unit 504 is configured to apply the base period decision tree model to predict the unlabeled samples of the current period, obtain a second sample statistic corresponding to each of the paths after the prediction, and form a second value set based on the second sample statistics corresponding to each of the paths; the first sample statistic and the second sample statistic are statistics of the same type;

[0171] The second acquiring unit 505 is configured to acquire a first distribution difference result between the first value set and the second value set, and determine that data drift occurs when the first distribution difference result meets a first preset condition.

[0172] In a possible implementation, the second acquiring unit 505 includes:

[0173] a first calculation subunit, configured to calculate a first KL divergence of the first value set and the second value set based on a first sample statistic corresponding to each of the paths in the first value set and a second sample statistic corresponding to each of the paths in the second value set;

[0174] The first determining subunit is configured to determine that data drift occurs when the first KL divergence satisfies a first threshold range.

[0175] In a possible implementation, the first prediction unit 503 is specifically configured to:

[0176] Calculating a first ratio corresponding to each of the paths after prediction, and forming a first numerical value set by the first ratios corresponding to the paths respectively; the first ratio corresponding to the path is the ratio of the number of base period unlabeled samples passing through the path after prediction to the total number of base period unlabeled samples;

[0177] The second prediction unit 504 is specifically configured to:

[0178] Calculate the second ratio corresponding to each of the paths after prediction, and form a second numerical set of the second ratios corresponding to the paths; the second ratio corresponding to the path is the ratio of the number of unlabeled samples in this period that pass through the path after prediction to the total number of unlabeled samples in this period.

[0179] In a possible implementation, the sample includes input data features, and the apparatus further includes:

[0180] a calculation unit, configured to calculate the difference between the first ratio and the second ratio corresponding to each of the paths;

[0181] A first determining unit is configured to determine a path whose gap satisfies a gap range as a first target path, and determine that an input data feature associated with the first target path is related to the data drift.

[0182] In a possible implementation, the apparatus further includes:

[0183] A third acquisition unit is used to acquire the current period labeled samples at the current time point;

[0184] A third prediction unit is configured to apply the base period decision tree model to predict the base period labeled samples, and form a first label set based on the number of sample labels corresponding to each of the paths after prediction;

[0185] a fourth prediction unit, configured to apply the base period decision tree model to predict the labeled samples of the current period, and form a second label set based on the number of sample labels corresponding to each of the paths after prediction;

[0186] The fourth acquiring unit is configured to acquire a second distribution difference result between the first label set and the second label set, and determine that concept drift occurs when the second distribution difference result meets a second preset condition.

[0187] In a possible implementation, the fourth acquiring unit includes:

[0188] a second calculation subunit, configured to calculate a third ratio of the number of sample labels corresponding to each of the paths in the first label set to the total number of labeled samples in the base period;

[0189] a third calculation subunit, configured to calculate a fourth ratio of the number of sample labels corresponding to each of the paths in the second label set to the total number of labeled samples in the current period;

[0190] a fourth calculation subunit, configured to calculate a second KL divergence corresponding to each of the paths based on the third ratio and the fourth ratio corresponding to each of the paths;

[0191] A selection subunit, configured to select the maximum KL divergence among the second KL divergences corresponding to the paths;

[0192] The second determining subunit is configured to determine that concept drift occurs when the maximum KL divergence satisfies a second threshold range.

[0193] In a possible implementation, the apparatus further includes:

[0194] A second determining unit, configured to determine a second target path corresponding to the maximum KL divergence;

[0195] A third determining unit is configured to determine, when the maximum KL divergence satisfies a third threshold range, that the input data feature associated with the second target path is related to the concept drift.

[0196] It should be noted that the specific implementation of each unit in this embodiment can refer to the relevant description in the above method embodiment. The division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional units in the embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. For example, in the above embodiment, the processing unit and the sending unit can be the same unit or different units. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0197] Based on the model stability detection method provided in the above method embodiment, the present application also provides an electronic device, including: one or more processors; a storage device on which one or more programs are stored, when the one or more programs are executed by the one or more processors, the one or more processors implement the model stability detection method described in any of the above embodiments.

[0198] Reference below Figure 6, which shows a schematic structural diagram of an electronic device 600 suitable for implementing an embodiment of the present application. The terminal device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable Android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs (televisions) and desktop computers. Figure 6 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0199] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0200] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0201] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present application are performed.

[0202] The electronic device provided in the embodiment of the present application and the model stability detection method provided in the above embodiment belong to the same inventive concept. Technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0203] Based on a model stability detection method provided in the above method embodiment, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the model stability detection method as described in any of the above embodiments is implemented.

[0204] It should be noted that the computer-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0205] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0206] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0207] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the model stability detection method.

[0208] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0209] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0210] The units involved in the embodiments described in this application may be implemented by software or hardware. In some cases, the name of a unit / module does not constitute a limitation of the unit itself. For example, a voice data acquisition module may also be described as a "data acquisition module."

[0211] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0212] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0213] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0214] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0215] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0216] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0217] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model stability detection method, characterized in that: The method comprises: Obtaining a base period labeled sample and a base period unlabeled sample at a base period time point, as well as a current period unlabeled sample at a current period time point; the base period time point is earlier than the current period time point; Training a decision tree model based on the base period labeled samples to obtain a base period decision tree model; the base period decision tree model includes at least one path from a root node to a leaf node; Applying the base period decision tree model to predict the base period unlabeled samples, obtaining a first sample statistic corresponding to each of the paths after prediction, and forming a first value set from the first sample statistics corresponding to each of the paths; Applying the base period decision tree model to predict the unlabeled samples of the current period, obtaining a second sample statistic corresponding to each of the paths after the prediction, and forming a second numerical value set by the second sample statistics corresponding to each of the paths; the first sample statistic and the second sample statistic are statistics of the same type; A first distribution difference result between the first value set and the second value set is obtained, and when the first distribution difference result meets a first preset condition, it is determined that data drift in concept drift and data drift occurs.

2. The method according to claim 1, characterized in that The obtaining a first distribution difference result between the first value set and the second value set, and determining that data drift occurs in the concept drift and data drift when the first distribution difference result satisfies a first preset condition, includes: Calculating a first KL divergence of the first value set and the second value set based on a first sample statistic corresponding to each of the paths in the first value set and a second sample statistic corresponding to each of the paths in the second value set; When the first KL divergence satisfies a first threshold range, it is determined that data drift occurs.

3. The method according to claim 1, characterized in that The first sample statistic corresponding to each of the paths after the prediction is obtained, and a first value set is formed by the first sample statistics corresponding to each of the paths, including: Calculating a first ratio corresponding to each of the paths after prediction, and forming a first numerical value set by the first ratios corresponding to the paths respectively; the first ratio corresponding to the path is the ratio of the number of base period unlabeled samples passing through the path after prediction to the total number of base period unlabeled samples; The second sample statistics corresponding to each of the paths after the prediction are obtained, and a second value set is formed by the second sample statistics corresponding to each of the paths, including: Calculate the second ratio corresponding to each of the paths after prediction, and form a second numerical set of the second ratios corresponding to the paths; the second ratio corresponding to the path is the ratio of the number of unlabeled samples in this period that pass through the path after prediction to the total number of unlabeled samples in this period.

4. The method according to claim 3, characterized in that The sample includes input data features, and the method further includes: Calculate the difference between the first ratio and the second ratio corresponding to each of the paths; A path whose gap satisfies a gap range is determined as a first target path, and an input data feature associated with the first target path is determined to be related to the data drift.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Obtain the labeled samples of the current period at the current time point; Applying the base period decision tree model to predict the base period labeled samples, and forming a first label set based on the number of sample labels corresponding to each of the paths after prediction; Applying the base period decision tree model to predict the labeled samples of the current period, and forming a second label set based on the number of sample labels corresponding to each of the paths after prediction; A second distribution difference result between the first label set and the second label set is obtained, and when the second distribution difference result meets a second preset condition, it is determined that concept drift occurs.

6. The method according to claim 5, characterized in that The obtaining of a second distribution difference result between the first label set and the second label set, and determining that concept drift occurs when the second distribution difference result satisfies a second preset condition, includes: Calculating a third ratio of the number of sample labels corresponding to each of the paths in the first label set to the total number of labeled samples in the base period; Calculating a fourth ratio of the number of sample labels corresponding to each of the paths in the second label set to the total number of labeled samples in the current period; Calculating a second KL divergence corresponding to each path based on the third ratio and the fourth ratio corresponding to each path; Selecting the maximum KL divergence among the second KL divergences corresponding to each of the paths; When the maximum KL divergence satisfies a second threshold range, it is determined that concept drift occurs.

7. The method according to claim 6, characterized in that The sample includes input data features, and the method further includes: Determining a second target path corresponding to the maximum KL divergence; When the maximum KL divergence satisfies a third threshold range, it is determined that the input data feature associated with the second target path is related to the concept drift.

8. A model stability detection device, characterized in that: The device comprises: A first acquisition unit is configured to acquire a base period labeled sample and a base period unlabeled sample at a base period time point, and a current period unlabeled sample at a current period time point; the base period time point is earlier than the current period time point; A training unit, configured to train a decision tree model based on the base period labeled samples to obtain a base period decision tree model; the base period decision tree model includes at least one path from a root node to a leaf node; a first prediction unit, configured to apply the base period decision tree model to predict the base period unlabeled samples, obtain a first sample statistic corresponding to each of the paths after prediction, and form a first value set from the first sample statistics corresponding to each of the paths; a second prediction unit, configured to apply the base period decision tree model to predict the unlabeled samples of the current period, obtain a second sample statistic corresponding to each of the paths after the prediction, and form a second numerical value set from the second sample statistics corresponding to each of the paths; the first sample statistic and the second sample statistic are statistics of the same type; The second acquisition unit is configured to acquire a first distribution difference result between the first value set and the second value set, and determine that data drift occurs in the concept drift and data drift when the first distribution difference result meets a first preset condition.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the model stability detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the model stability detection method according to any one of claims 1 to 7 is implemented.

Images