Data consistency detection method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a data consistency detection method and device, electronic equipment, a computer program product and a computer readable storage medium. The method comprises the following steps: acquiring a training data set for training a machine learning model and a prediction data set of the machine learning model in an application process; respectively detecting the consistency of the training data set and the prediction data set from at least two detection dimensions to obtain detection results respectively corresponding to the detection dimensions; and determining a consistency detection result in combination with the detection results corresponding to the at least two detection dimensions, the consistency detection result being used for indicating whether the data of the training data set is consistent with the data of the prediction data set. By means of the method and device, the consistency of the data set can be comprehensively and accurately detected, and data exception can be found in time.

Description

Technical Field

[0001] The present application relates to machine learning technology, and in particular to a data consistency detection method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] In the field of machine learning, data determines the upper limit of machine learning, and the algorithm model only approaches this upper limit as much as possible. The consistency between the training data set and the test data set will greatly affect the training effect of the algorithm. The consistency between the offline data set and the online data set will also greatly affect the effect of the algorithm application. In related technologies, the detection process mainly relies on the experience of algorithm personnel, and cannot achieve automatic detection and optimization, and the effect of data consistency detection is poor. Summary of the Invention

[0003] The embodiments of the present application provide a data consistency detection method, device, electronic device, computer-readable storage medium, and computer program product, which can comprehensively and accurately detect the consistency of a data set and promptly detect data anomalies.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides a data consistency detection method, the method comprising:

[0006] Obtaining a training data set for training a machine learning model and a prediction data set for the machine learning model during application;

[0007] Testing the consistency of the training dataset and the prediction dataset from at least two detection dimensions, respectively, to obtain detection results corresponding to each detection dimension;

[0008] The detection results corresponding to the at least two detection dimensions are combined to determine a consistency detection result, where the consistency detection result is used to indicate whether the data of the training dataset and the prediction dataset are consistent.

[0009] In the above solution, the consistency of the training dataset and the prediction dataset is tested from at least two detection dimensions to obtain the detection results corresponding to each detection dimension, including:

[0010] From at least two detection dimensions of feature shift detection, label shift detection, and pattern change detection, respectively, the consistency of the training dataset and the prediction dataset is detected to obtain detection results corresponding to each detection dimension;

[0011] The feature offset detection is used to indicate the detection of consistency of feature distributions of the training dataset and the prediction dataset;

[0012] The label shift detection is used to instruct to detect the consistency of the label distribution of the training data set and the prediction data set;

[0013] The pattern change detection is used to detect the consistency of the importance of the same features in the training dataset and the prediction dataset.

[0014] In the above solution, the feature shift detection includes single feature shift detection and adversarial verification. The consistency of the training dataset and the prediction dataset is detected from at least two detection dimensions of feature shift detection, label shift detection, and pattern change detection, respectively, to obtain the detection results corresponding to each detection dimension, including:

[0015] When the at least two detection dimensions include the feature offset detection, performing single feature offset detection on the training dataset and the prediction dataset to obtain a single feature offset result;

[0016] The single feature offset detection is used to indicate the detection of the consistency of the distribution of a single feature in the training data set and the prediction data set;

[0017] Performing adversarial verification on the training data set and the prediction data set to obtain an adversarial verification result;

[0018] The adversarial verification is used to instruct to detect the distribution difference between the training data set and the prediction data set;

[0019] The single feature shift result and the adversarial verification result are jointly determined as the detection result corresponding to the feature shift detection dimension.

[0020] In the above solution, performing single feature offset detection on the training dataset and the prediction dataset to obtain a single feature offset result includes:

[0021] Calculate the JS divergence of each feature that has a corresponding relationship between the training dataset and the prediction dataset, and obtain the JS divergence value of each feature;

[0022] Comparing each of the JS divergence values ​​with the JS divergence threshold value to obtain a comparison result;

[0023] When the comparison result indicates that the JS divergence value is greater than the JS divergence threshold, it is determined that a single feature offset exists in the feature corresponding to the JS divergence value, and the existence of the single feature offset is used as the single feature offset result.

[0024] In the above solution, performing adversarial verification on the training dataset and the prediction dataset to obtain adversarial verification results includes:

[0025] Reconstructing the labels of each feature in the training dataset and the prediction dataset to obtain an adversarial verification dataset, wherein the adversarial verification dataset includes positive sample labels and negative sample labels;

[0026] The positive sample label indicates that the corresponding feature comes from the training dataset, and the negative sample label indicates that the corresponding feature comes from the prediction dataset;

[0027] Determining the information gain of each of the features in the adversarial verification dataset respectively;

[0028] Determining a first feature importance of each feature based on the information gain of the feature;

[0029] When the first feature importance of the feature is greater than the feature importance threshold, determining that the predicted dataset has a feature offset compared to the training dataset;

[0030] The presence of feature shift in the predicted dataset compared to the training dataset is taken as an adversarial verification result.

[0031] In the above solution, after reconstructing the labels of each feature in the training dataset and the prediction dataset to obtain an adversarial verification dataset, the method further includes:

[0032] Dividing the adversarial validation dataset into an adversarial validation training set and an adversarial validation test set;

[0033] Training the binary classification model based on the adversarial verification training set to obtain a target binary classification model;

[0034] Determining the classification test accuracy of the target binary classification model based on the adversarial validation test set;

[0035] When the classification test accuracy is greater than or equal to a classification test accuracy threshold, determining that there is a feature shift in the prediction data set compared to the training data set;

[0036] Determining the classification test accuracy of the target binary classification model based on the adversarial validation test set includes:

[0037] When the adversarial validation test set includes multiple test samples, using the target binary classification model to predict each of the test samples respectively to obtain multiple prediction results;

[0038] According to the prediction result of each test sample and the corresponding sample label, the classification test accuracy of the target binary classification model is obtained.

[0039] In the above solution, the consistency of the training dataset and the prediction dataset is detected from at least two detection dimensions of feature offset detection, label offset detection, and pattern change detection, respectively, to obtain the detection results corresponding to each detection dimension, including:

[0040] When the at least two detection dimensions include the label shift detection, counting a first positive sample rate of the entire training dataset, a second positive sample rate of the entire prediction dataset, a third positive sample rate of each feature in the training dataset, and a fourth positive sample rate of each feature in the prediction dataset;

[0041] Compare the first positive sample rate and any two of the third positive sample rates to obtain a first comparison result;

[0042] Comparing any two of the second positive sample rate and each of the fourth positive sample rates to obtain a second comparison result;

[0043] When the first comparison result indicates that the ratio of the two arbitrary items satisfies a first ratio condition, or the second comparison result indicates that the ratio of the two arbitrary items satisfies a second ratio condition, determining that there is a label offset in the prediction dataset compared to the training dataset;

[0044] Determining whether the predicted dataset has a label shift compared to the training dataset is a detection result corresponding to the label shift detection dimension.

[0045] In the above solution, when the at least two detection dimensions include the tag offset detection, the method further includes:

[0046] Determining a prediction accuracy of the machine learning model based on the prediction data set;

[0047] Dividing the prediction data set into a target training set and a target validation set;

[0048] Training the machine learning model based on the target training set to obtain a target machine learning model;

[0049] Determining the validation accuracy of the target machine learning model based on the target validation set;

[0050] Determining a test accuracy of the target machine learning model based on the training data set;

[0051] Determining the prediction accuracy of the machine learning model based on the prediction data set includes:

[0052] When the prediction data set includes multiple prediction samples, using the machine learning model to predict each of the prediction samples separately to obtain multiple prediction results;

[0053] Obtaining the prediction accuracy of the machine learning model according to the prediction result and the corresponding sample label of each prediction sample;

[0054] Determining the verification accuracy of the target machine learning model based on the target verification set includes:

[0055] When the target validation set includes multiple validation samples, using the target machine learning model to predict each of the validation samples separately to obtain multiple prediction results;

[0056] Obtaining the verification accuracy of the target machine learning model according to the prediction result of each of the verification samples and the corresponding sample label;

[0057] Determining the test accuracy of the target machine learning model based on the training data set includes:

[0058] When the training data set includes multiple test samples, using the target machine learning model to predict each of the test samples separately to obtain multiple prediction results;

[0059] The test accuracy of the target machine learning model is obtained based on the prediction results of each test sample and the corresponding sample labels.

[0060] In the above solution, the consistency of the training dataset and the prediction dataset is detected from at least two detection dimensions of feature offset detection, label offset detection, and pattern change detection, respectively, to obtain the detection results corresponding to each detection dimension, including:

[0061] When the at least two detection dimensions include the pattern change detection, determining the information gain of each feature in the training data set respectively;

[0062] Determining a second feature importance of each feature in the training data set according to the information gain of the corresponding feature;

[0063] Determining the information gain of each feature in the prediction data set respectively;

[0064] Determining a third feature importance of each feature in the prediction data set according to the information gain of the corresponding feature;

[0065] Based on the second feature importance and the third feature importance, a detection result corresponding to the pattern change detection dimension is determined.

[0066] In the above solution, determining the detection result corresponding to the pattern change detection dimension based on the second feature importance and the third feature importance includes:

[0067] Based on the second feature importance, sorting multiple features in the training data set to obtain a first rank for each feature;

[0068] Based on the third feature importance, sorting multiple features in the prediction data set to obtain a second rank for each feature;

[0069] Comparing the first rank and the second rank of the same feature in the training dataset and the prediction dataset to obtain a rank comparison result;

[0070] When the ranking comparison result indicates that there is a difference between the first ranking and the second ranking of at least one feature, it is determined that there is a pattern change in the machine learning model, and the pattern change in the machine learning model is used as the detection result corresponding to the pattern change detection dimension.

[0071] In the above solution, combining the detection results corresponding to the at least two detection dimensions to determine the consistency detection result includes:

[0072] Obtaining weights of the detection results corresponding to the at least two detection dimensions;

[0073] The detection results corresponding to the at least two detection dimensions are weightedly summed according to the weights to obtain the consistency detection result.

[0074] In the above solution, after determining the consistency detection result, the method further includes:

[0075] Generate a consistency test report, wherein the consistency test report includes the test results corresponding to each of the test dimensions and the consistency test result;

[0076] Output the consistency detection report.

[0077] In the above solution, after determining the consistency detection result, the method further includes:

[0078] When the at least two detection dimensions include the feature offset detection, acquiring first information;

[0079] The first information includes a classification test accuracy of a binary classification model, wherein the binary classification model is trained based on an adversarial verification dataset, and the adversarial verification dataset is constructed based on the training dataset and the prediction dataset;

[0080] When the at least two detection dimensions include the mode change detection, acquiring second information;

[0081] The second information includes the prediction accuracy of the machine learning model, the verification accuracy of the target machine learning model, and the test accuracy of the target machine learning model; wherein the target machine learning model is trained based on the prediction data set;

[0082] Based on at least one of the first information and the second information, when it is determined that the machine learning model update condition is met, the model parameter update of the machine learning model is automatically triggered.

[0083] In the above solution, when determining, based on at least one of the first information and the second information, that the machine learning model update condition is satisfied, automatically triggering the update of the model parameters of the machine learning model includes:

[0084] When the at least two detection dimensions include the feature offset detection, comparing the classification test accuracy with a preset threshold to obtain a classification test accuracy comparison result;

[0085] When the classification test accuracy comparison result indicates that the classification test accuracy is greater than or equal to the preset threshold, automatically triggering the model parameter update of the machine learning model;

[0086] When the at least two detection dimensions include the pattern change detection, comparing the prediction accuracy with the test accuracy to obtain a third comparison result;

[0087] Comparing the prediction accuracy with the verification accuracy to obtain a fourth comparison result;

[0088] When the third comparison result indicates that the prediction accuracy is greater than the test accuracy, and the fourth comparison result indicates that the prediction accuracy is less than the verification accuracy, the model parameter update of the machine learning model is automatically triggered.

[0089] The present invention provides a data consistency detection device, comprising:

[0090] An acquisition module, configured to acquire a training data set for training a machine learning model and a prediction data set of the machine learning model during application;

[0091] A detection module, configured to detect the consistency of the training dataset and the prediction dataset from at least two detection dimensions, and obtain detection results corresponding to each detection dimension;

[0092] The determination module is used to determine a consistency detection result based on the detection results corresponding to the at least two detection dimensions, and the consistency detection result is used to indicate whether the data of the training data set and the prediction data set are consistent.

[0093] An embodiment of the present application provides an electronic device, comprising:

[0094] a memory for storing computer-executable instructions;

[0095] The processor is configured to implement the data consistency detection method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0096] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data consistency detection method provided in the embodiment of the present application when executed by a processor.

[0097] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the data consistency detection method provided in the embodiment of the present application is implemented.

[0098] The embodiments of the present application have the following beneficial effects:

[0099] Obtain a training data set for training a machine learning model, as well as a prediction data set during the application process, and test the consistency of the training data set and the prediction data set from at least two detection dimensions, respectively, to obtain the test results corresponding to each detection dimension, and combine the test results corresponding to at least two detection dimensions to obtain the result of the data consistency test. Through this application, data consistency testing is performed from multiple detection dimensions, and the final result of the consistency test is determined by combining the test results of multiple detection dimensions. In this way, the consistency of the training data set and the prediction data set can be comprehensively and accurately tested, and data anomalies can be discovered in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 This is a schematic diagram of the architecture of the data consistency detection system provided in an embodiment of the present application;

[0101] Figure 2 is a schematic diagram of the structure of the terminal provided in an embodiment of the present application;

[0102] Figure 3AThis is a schematic diagram of the first flow chart of the data consistency detection method provided in an embodiment of the present application;

[0103] Figure 3B This is a second flow chart of the data consistency detection method provided in an embodiment of the present application;

[0104] Figure 3C 3 is a schematic diagram of a third flow chart of the data consistency detection method provided in an embodiment of the present application;

[0105] Figure 3D 4 is a schematic diagram of a fourth flow chart of the data consistency detection method provided in an embodiment of the present application;

[0106] Figure 3E This is a fifth flow chart of the data consistency detection method provided in an embodiment of the present application;

[0107] Figure 3F This is a sixth flow chart of the data consistency detection method provided in an embodiment of the present application;

[0108] Figure 3G This is a seventh flow chart of the data consistency detection method provided in an embodiment of the present application;

[0109] Figure 3H This is a schematic diagram of the eighth flow chart of the data consistency detection method provided in an embodiment of the present application;

[0110] Figure 4A This is a schematic diagram of the first interface of an exemplary user purchase intention rating provided in an embodiment of the present application;

[0111] Figure 4B This is a schematic diagram of a second interface for an exemplary user purchase intention rating provided in an embodiment of the present application;

[0112] Figure 4C This is a flowchart of an exemplary user purchase intention rating service provided in an embodiment of the present application;

[0113] Figure 4D is a schematic diagram of a first principle diagram of an exemplary data consistency detection method provided in an embodiment of the present application;

[0114] Figure 4E is a second schematic diagram of an exemplary data consistency detection method provided in an embodiment of the present application;

[0115] Figure 4F This is a schematic diagram of a report template for an exemplary data consistency detection method provided in an embodiment of the present application;

[0116] Figure 4G This is a first report schematic diagram of an exemplary data consistency detection method provided in an embodiment of the present application;

[0117] Figure 4H This is a second report schematic diagram of an exemplary data consistency detection method provided in an embodiment of the present application;

[0118] Figure 4I This is a third report schematic diagram of the exemplary data consistency detection method provided in the embodiments of the present application.

[0119] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0120] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0121] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0122] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0123] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0124] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0125] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0126] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0127] 1) Data consistency. This refers to the consistency of the training and prediction datasets used in machine learning models. During machine learning model training, both the training and prediction datasets must be independent and identically distributed (IID). This assumption is fundamental to the effective application of all machine learning models. In real-world scenarios, sample data consistency can vary due to a variety of factors, significantly impacting model performance.

[0128] 2) Training dataset refers to the training set and test set used in the offline training process, collectively referred to as the training dataset.

[0129] 3) Prediction dataset, which refers to the prediction set used in the online prediction process, is called the prediction dataset, where the samples in the prediction dataset can obtain labels within a set time.

[0130] 4) Confusion matrix. The confusion matrix is ​​composed of true positive examples, true negative examples, false positive examples, and false negative examples.

[0131]

[0132]

[0133] The specific meanings of the four values ​​are:

[0134] True Positive (TP): a positive example that is correctly predicted, that is, the true value of the data is a positive example and the predicted value is also a positive example;

[0135] True Negative (TN): a correctly predicted counterexample, that is, the true value of the data is a counterexample and the predicted value is also a counterexample;

[0136] False Positive (FP): a positive example that is incorrectly predicted, that is, the true value of the data is a negative example, but it is mistakenly predicted as a positive example;

[0137] False Negative (FN) is a falsely predicted negative example, that is, the true value of the data is a positive example, but it is mistakenly predicted to be a negative example.

[0138] 5) Accuracy, also known as the model evaluation metric (Area Under Curve, AUC). AUC is defined as the area under the receiver operating characteristic (ROC) curve and the coordinate axes. It is often used as a performance metric to measure the quality of a binary classification model and is calculated by summing the areas under the ROC curve.

[0139] The horizontal axis of the ROC curve is the false positive rate, and the vertical axis is the true positive rate. The calculation method of AUC is as follows:

[0140] False Positive Rate (FPR): The probability that a sample is considered positive but not a true positive, that is, the probability that a sample is considered positive among true negative samples. The calculation formula is:

[0141]

[0142] True Positive Rate (TPR): The probability that a positive example is also a true positive example, that is, the probability that a true positive example is a positive example (also known as the positive recall rate). The calculation formula is:

[0143]

[0144] The ROC curve is based on the FPR and TPR. A binary classification model may have a high or low threshold. Each threshold setting results in a different FPR and TPR. Plotting the (FPR, TPR) coordinates for each threshold for the same model in ROC space creates the ROC curve for that specific model.

[0145] AUC stands for Area Under the Curve (ROC). When comparing different classification models, the ROC curve for each model can be plotted and the area under the curve compared as an indicator of model quality. Because the area is calculated within a 1x1 grid, the AUC must be between 0 and 1. Assume that values ​​above the threshold are positive and values ​​below the threshold are negative. If a positive sample and a negative sample are randomly drawn, the probability that the classifier correctly determines that the positive sample has a higher value than the negative sample is equal to the AUC value. In other words, a classifier with a larger AUC value has a higher accuracy rate.

[0146] 6) Information gain. Information gain is defined as the difference between the empirical entropy H(D) of set D and the empirical conditional entropy H(D|A) of D given the condition of feature A. That is, information entropy minus conditional information entropy, which indicates the degree to which the uncertainty of information D is reduced by knowing the information of feature A (the greater the information gain, the more certain D is when A is known). The calculation formula is:

[0147] g(D,A)=H(D)-H(D|A) (3)

[0148] in,

[0149] When performing data consistency testing, related technologies classify abnormal results into three categories: covariate shift, prior probability shift, and concept shift. These technologies can perform consistency testing on a specific category of abnormal results. However, in actual applications, the following technical issues still exist:

[0150] 1. The detection of data consistency is not comprehensive enough and can only be performed on offline data sets, resulting in poor applicability of the detection solution.

[0151] 2. The detection process mainly relies on the experience of algorithm personnel and cannot be automatically detected.

[0152] 3. The machine learning model cannot be automatically optimized after the detection is completed.

[0153] The embodiments of the present application provide a data consistency detection method, device, electronic device, computer-readable storage medium, and computer program product, which can comprehensively and accurately detect the consistency of a data set and promptly detect data anomalies.

[0154] The following describes the data consistency detection system provided by the embodiment of the present application. Figure 1 , Figure 1 This is an architectural diagram of a data consistency detection system 100 provided in an embodiment of the present application. To support a data consistency detection application, a terminal 400 is connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0155] The server 200 is used to obtain a training data set and a prediction data set from the database 500, and to detect the consistency of the training data set and the prediction data set from at least two detection dimensions, respectively, to obtain the detection results corresponding to each detection dimension, to determine the consistency detection result by combining the detection results corresponding to at least two detection dimensions, and to transmit the consistency detection result to the terminal 400 through the network 300.

[0156] The terminal 400 is used to receive the consistency detection result and display the obtained consistency detection result on the graphical interface 410.

[0157] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0158] The electronic device provided in the embodiments of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and car terminals, and can also be implemented as servers.

[0159] Next, the electronic device that implements the data consistency detection method provided in the embodiment of the present application is described. In practical applications, the electronic device can be a terminal or a server. Figure 2 , Figure 2 is a structural diagram of an electronic device 600 provided in an embodiment of the present application, Figure 2 The electronic device 600 shown includes: at least one processor 610, a memory 650, at least one network interface 620 and a user interface 630. The various components in the electronic device 600 are coupled together via a bus system 640. It is understood that the bus system 640 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 640 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 640 is not described in detail. Figure 2 Various buses are labeled as bus system 640 .

[0160] The processor 610 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0161] The user interface 630 includes one or more output devices 631 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 630 also includes one or more input devices 632, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0162] The memory 650 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 650 may optionally include one or more storage devices that are physically remote from the processor 610.

[0163] The memory 650 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 650 described in the embodiments of the present application is intended to include any suitable type of memory.

[0164] In some embodiments, the memory 650 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0165] Operating system 651, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0166] A network communication module 652 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 620. Exemplary network interfaces 620 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0167] a presentation module 653 for enabling presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 631 (e.g., a display screen, a speaker, etc.) associated with the user interface 630;

[0168] The input processing module 654 is configured to detect one or more user inputs or interactions from one of the one or more input devices 632 and to translate the detected inputs or interactions.

[0169] In some embodiments, the data consistency detection device provided in the embodiments of the present application can be implemented in a software manner. Figure 2The data consistency detection device 655 stored in the memory 650 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 6551, a detection module 6552, and a determination module 6553. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0170] In other embodiments, the data consistency detection device provided in the embodiments of the present application can be implemented in hardware. As an example, the data consistency detection device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute a data consistency detection method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0171] Based on the above description of the data consistency detection system and electronic device, the following describes the data consistency detection method provided by the embodiment of the present application. In practical applications, the method can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Taking the server alone as an example, see Figure 3A , Figure 3A This is a first flow chart of the data consistency detection method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0172] In step 101, a training data set for training a machine learning model and a prediction data set for the machine learning model during application are obtained.

[0173] In some embodiments, the machine learning model may be a classification model that classifies objects, where the objects may be at least one of user objects and item objects. Accordingly, the multiple features included in the training dataset may be object features, i.e., sample features used to train the classification model. These sample features have a one-to-one correspondence with the objects and carry a category label indicating the category to which the corresponding object belongs. In practical applications, the training dataset may also include a test set for model testing and a validation set for model validation.

[0174] Here, when the above-mentioned object is a user object, the corresponding object features may include at least one of the following: user object ID, gender, age, etc.; when the above-mentioned object is an item object, the corresponding object features may include at least one of the following: texture, shape, size, etc.

[0175] When the machine learning model is a classification model for object classification, the multiple features included in the prediction data set are multiple object features of the machine learning model during the application process. The object features can also carry corresponding category labels. For example, during the model application process, object features are collected periodically (such as once a month) and category labels corresponding to the object features are constructed.

[0176] In actual applications, the above classification model can be applied to multiple different application scenarios such as advertising recommendation, games, shopping, etc. For example, in the advertising recommendation scenario, the classification model can be used to predict whether the user object clicks to view the displayed advertisement; in the game scenario, the classification model can be used to predict whether the user object is an active user object; in the shopping scenario, the classification model can be used to predict whether the user object will purchase.

[0177] In step 102, the consistency of the training data set and the prediction data set is tested from at least two detection dimensions to obtain the detection results corresponding to each detection dimension.

[0178] In some embodiments, see Figure 3B , Figure 3B This is a second flow chart of the data consistency detection method provided in an embodiment of the present application. Figure 3A The illustrated step 102 can be implemented by the following step 1021 , which is described in detail below.

[0179] In step 1021, the consistency of the training dataset and the prediction dataset is detected from at least two detection dimensions of feature offset detection, label offset detection, and pattern change detection, respectively, to obtain detection results corresponding to each detection dimension.

[0180] Among them, feature shift detection is used to indicate the detection of the consistency of feature distribution of the training dataset and the prediction dataset; label shift detection is used to indicate the detection of the consistency of label distribution of the training dataset and the prediction dataset; pattern change detection is used to indicate the detection of the consistency of the importance of the same features in the training dataset and the prediction dataset.

[0181] In some embodiments, the consistency of the training dataset and the prediction dataset can be detected from two detection dimensions, namely, feature offset detection and label offset detection, to obtain the detection results corresponding to each detection dimension. The consistency of the training dataset and the prediction dataset can also be detected from two detection dimensions, namely, feature offset detection and pattern change detection, to obtain the detection results corresponding to each detection dimension. The consistency of the training dataset and the prediction dataset can also be detected from two detection dimensions, namely, label offset detection and pattern change detection, to obtain the detection results corresponding to each detection dimension. The consistency of the training dataset and the prediction dataset can also be detected from three detection dimensions, namely, feature offset detection, label offset detection, and pattern change detection, to obtain the detection results corresponding to each detection dimension.

[0182] In some embodiments, see Figure 3C , Figure 3C This is a third flow chart of the data consistency detection method provided in the embodiment of the present application. Figure 3B The illustrated step 1021 can be implemented by following steps 10211A to 10213A, which are described in detail below.

[0183] In step 10211A, when at least two detection dimensions include feature offset detection, single feature offset detection is performed on the training dataset and the prediction dataset to obtain a single feature offset result.

[0184] Among them, single feature offset detection is used to detect the consistency of the distribution of a single feature in the training dataset and the prediction dataset.

[0185] In some embodiments, when at least two detection dimensions include feature shift detection, single feature shift detection can be performed on the training dataset and the prediction dataset in the following manner to obtain a single feature shift result:

[0186] Calculate the JS divergence of each feature that has a corresponding relationship between the training data set and the prediction data set, and obtain the JS divergence value of each feature.

[0187] For example, taking feature A as an example, the JS divergence value of feature A is calculated using the following formula:

[0188]

[0189] Among them, P(x) is the probability distribution of feature A in the training data set, Q(x) is the probability distribution of feature A in the prediction data set, For P(x) The relative entropy of For Q(x) The relative entropy of JS(P(x)||Q(x)) is the JS divergence of feature A, which is used to measure the similarity of the probability distribution of the current feature A in the training set and the prediction set. For example, the larger the value of JS divergence, the smaller the similarity of the probability distribution of feature A in the training set and the prediction set, that is, the greater the degree of deviation of feature A.

[0190] Each JS divergence value is compared with the JS divergence threshold to obtain a comparison result.

[0191] In some embodiments, the JS divergence threshold may be set according to actual conditions, for example, the JS divergence threshold may be set to 0.05.

[0192] When the comparison result indicates that the JS divergence value is greater than the JS divergence threshold, it is determined that a single feature offset exists in the feature corresponding to the JS divergence value, and the existence of the single feature offset is taken as a single feature offset result.

[0193] In some embodiments, if the JS divergence value exceeds the set JS divergence threshold, it is considered that the feature offset corresponding to the JS divergence value is more serious. It can be determined that the feature corresponding to the JS divergence value has a single feature offset, and the existence of the single feature offset is used as the single feature offset result.

[0194] According to the above method, the specific features of the anomaly can be clearly pointed out, making it easier for algorithm personnel to quickly discover and locate abnormal data.

[0195] In step 10212A, adversarial verification is performed on the training data set and the prediction data set to obtain adversarial verification results.

[0196] Among them, adversarial verification is used to detect the distribution differences between the training data set and the prediction data set.

[0197] In some embodiments, the following methods can be used to perform adversarial verification on the training dataset and the prediction dataset to obtain adversarial verification results:

[0198] Reconstruct the labels of each feature in the training dataset and the prediction dataset to obtain an adversarial validation dataset, which includes positive sample labels and negative sample labels.

[0199] Among them, the positive sample label indicates that the corresponding feature comes from the training dataset, and the negative sample label indicates that the corresponding feature comes from the prediction dataset.

[0200] Add a label column to each sample in the training dataset and the prediction dataset, mark the samples in the training dataset as 1, and mark the samples in the prediction dataset as 0. The features of the samples are the existing features to generate an adversarial verification dataset.

[0201] Determine the information gain for each feature in the adversarial validation dataset separately.

[0202] First, calculate the empirical entropy H(X) of the adversarial validation dataset X, which can be calculated using the following formula:

[0203]

[0204] Where n is the number of possible values ​​of X, x i is a possible value in X, P(x i ) is X and its value is x i The unit of empirical entropy is usually bits (or logarithmic units with base 2).

[0205] Next, calculate the empirical conditional entropy H(X|Y) of each feature. Take feature A as an example. Y represents the set of every possible value of feature A. The empirical conditional entropy H(X|Y) of feature A can be calculated using the following formula:

[0206]

[0207] Where n is the number of possible values ​​of X, x i is a possible value in X, P(x i ) is X and its value is x i The probability of m is the number of possible values ​​of Y, y i is a possible value in Y, P(y i ) is Y and its value is y i The probability, P(x i ,y i ) is X and its value is x i And Y takes the value y i The unit of the empirical conditional entropy is also bits (or logarithmic units with base 2).

[0208] Finally, the information gain of feature A can be calculated by the following formula:

[0209] g(X,Y)=H(X)-H(X|Y) (7)

[0210] Among them, g(X,Y) is the information gain of feature A, that is, the total gain. The gain of feature A can be obtained by dividing the total gain by the number of possible values ​​of feature A.

[0211] The information gain of each feature is obtained in the same way as above and will not be described here.

[0212] The greater the information gain, the greater the reduction in information uncertainty, that is, the higher the purity of the information. However, in the adversarial validation dataset, the information gain corresponding to each feature must be balanced; otherwise, feature shift will occur. To determine whether the information gain corresponding to each feature is balanced, we can calculate the ratio of the information gain corresponding to each feature to the sum of the information gains corresponding to all features, which is referred to as the first feature importance of the feature below. A threshold can be set for the ratio. When the ratio of at least one feature exceeds the threshold, the information gain corresponding to each feature is considered unbalanced, resulting in feature shift in the adversarial validation dataset.

[0213] Based on the information gain of each feature, the first feature importance of the corresponding feature is determined.

[0214] Taking feature A as an example, first, obtain the information gain of each feature, that is, the total gain, and add them up to get the sum of the total gains. Then, divide the total gain of feature A by the sum of the total gains of all features. Finally, get the first feature importance of feature A. Feature importance can reflect the importance of feature A in the adversarial verification dataset.

[0215] The method for obtaining the first feature importance of each feature is the same as the above method and will not be repeated here.

[0216] When the first feature importance of the feature is greater than the feature importance threshold, it is determined that there is a feature shift in the prediction dataset compared to the training dataset.

[0217] In some embodiments, the feature importance threshold can be set according to actual conditions. For example, the feature importance threshold can be set to 0.1. If the first feature importance exceeds the set feature importance threshold, it is considered that there is a high risk of feature shift, and it can be determined that there is feature shift in the predicted dataset compared to the training dataset.

[0218] The presence of feature shift in the predicted dataset compared to the training dataset is used as the adversarial verification result.

[0219] Through the above methods, we can comprehensively and quantitatively point out the feature offset of the predicted dataset compared to the training dataset.

[0220] In step 10213A, the single feature shift result and the adversarial verification result are jointly determined as the detection result corresponding to the feature shift detection dimension.

[0221] Through steps 10211A to 10213A, the two aspects of single feature offset and anti-verification can be integrated to jointly realize the detection of feature offset detection dimension, thereby improving the comprehensiveness of data consistency detection.

[0222] See also Figure 3D , Figure 3D This is a fourth flow chart of the data consistency detection method provided in an embodiment of the present application. Figure 3C The illustrated step 1021 may also be implemented by following steps 10211B to 10215B, which are described in detail below.

[0223] In step 10211B, when at least two detection dimensions include label offset detection, the first positive sample rate of the entire training dataset, the second positive sample rate of the entire prediction dataset, the third positive sample rate of each feature in the training dataset, and the fourth positive sample rate of each feature in the prediction dataset are counted.

[0224] In some embodiments, a statistical method is used to calculate a first positive sample rate of the entire training dataset, a second positive sample rate of the entire prediction dataset, a third positive sample rate of each feature in the training dataset, and a fourth positive sample rate of each feature in the prediction dataset.

[0225] In step 10212B, any two of the first positive sample rate and each third positive sample rate are compared to obtain a first comparison result.

[0226] In some embodiments, the first positive sample rate is compared with each third positive sample rate, and any two third positive sample rates are compared to obtain a first ratio, ie, a first comparison result.

[0227] In step 10213B, any two of the second positive sample rate and each fourth positive sample rate are compared to obtain a second comparison result.

[0228] In some embodiments, the second positive sample rate is compared with each fourth positive sample rate, and any two fourth positive sample rates are compared to obtain a second ratio, ie, a second comparison result.

[0229] In step 10214B, when the first comparison result indicates that the ratio of any two items meets the first ratio condition, or the second comparison result indicates that the ratio of any two items meets the second ratio condition, it is determined that there is a label offset in the prediction dataset compared to the training dataset.

[0230] In some embodiments, the first ratio condition and the second ratio condition can be set to a ratio greater than 1. When the first ratio is greater than 1 or the second ratio is greater than 1, it can be determined that there is a label offset in the prediction dataset compared to the training dataset.

[0231] In step 10215B, the presence of a label offset in the predicted dataset compared to the training dataset is determined as a detection result corresponding to the label offset detection dimension.

[0232] See also Figure 3E , Figure 3E This is a fifth flow chart of the data consistency detection method provided in an embodiment of the present application. Figure 3C The illustrated step 1021 may also be implemented by following steps 10211C to 10215C, which are described in detail below.

[0233] In step 10211C, when at least two detection dimensions include pattern change detection, the information gain of each feature in the training dataset is determined separately.

[0234] In some embodiments, the implementation method of separately determining the information gain of each feature in the training data set can be referred to step 10212A, which will not be repeated here.

[0235] In step 10212C, the second feature importance of each feature in the training data set is determined based on the information gain of the corresponding feature.

[0236] In some embodiments, the implementation method of determining the second feature importance of the corresponding feature based on the information gain of each feature in the training data set can refer to step 10212A, which will not be repeated here.

[0237] In step 10213C, the information gain of each feature in the prediction data set is determined separately.

[0238] In some embodiments, the implementation of separately determining the information gain of each feature in the prediction data set can refer to step 10212A, which will not be repeated here.

[0239] In step 10214C, the third feature importance of each feature in the prediction data set is determined.

[0240] In some embodiments, the implementation method of determining the third feature importance of the corresponding feature based on the information gain of each feature in the prediction data set can refer to step 10212A, which will not be repeated here.

[0241] In step 10215C, based on the second feature importance and the third feature importance, the detection result corresponding to the pattern change detection dimension is determined.

[0242] In some embodiments, the detection result corresponding to the pattern change detection dimension may be determined based on the second feature importance and the third feature importance in the following manner:

[0243] Based on the importance of the second feature, multiple features in the training data set are sorted to obtain the first rank of each feature.

[0244] Sort the importance of the second feature of each feature from large to small, and the sequence number of each feature is the first rank of each feature.

[0245] Based on the importance of the third feature, multiple features in the prediction data set are ranked to obtain the second rank of each feature.

[0246] Sort the importance of the third feature of each feature from large to small, and the sequence number of each feature is the second rank of each feature.

[0247] Compare the first and second ranks of the same feature in the training dataset and the prediction dataset to obtain the rank comparison result.

[0248] The following example illustrates that, for example, the first rank of each of the top 10 features in the training dataset is obtained, and the first ranks of these 10 features are subtracted from the corresponding second ranks in the prediction dataset to obtain the difference results, and each difference result is added together to obtain the sum of the difference results, and then the sum of the difference results is divided by 10 to obtain the rank comparison result. Feature importance can reflect the importance of the feature in the training dataset and the prediction dataset, and the importance can be quantified by the rank.

[0249] When the rank comparison result indicates that there is a difference between the first rank and the second rank of at least one feature, it is determined that there is a pattern change in the machine learning model, and the pattern change in the machine learning model is used as the detection result corresponding to the pattern change detection dimension.

[0250] The following example illustrates that, for example, the rank comparison result threshold can be set according to actual conditions. For example, the rank comparison result threshold can be set to 3. When the rank comparison result is greater than the rank comparison result threshold, it is considered that there is a difference between the first and second ranks of at least one feature, and it is determined that there is a pattern change in the machine learning model. The pattern change in the machine learning model is used as the detection result corresponding to the pattern change detection dimension.

[0251] Through step 102, data consistency can be tested from at least two detection dimensions, solving the above-mentioned technical problem 1) "the detection of data consistency is not comprehensive enough, and can only be performed on offline data sets, resulting in poor applicability of the detection solution." It achieves the technical effect of comprehensively and accurately detecting the consistency of the data set, timely discovering data anomalies, and being able to detect online data sets.

[0252] Continue to see Figure 3A , continue with step 102 above for explanation.

[0253] In step 103, the consistency detection result is determined by combining the detection results corresponding to at least two detection dimensions.

[0254] The consistency test result is used to indicate whether the data in the training dataset and the prediction dataset are consistent.

[0255] In some embodiments, see Figure 3F , Figure 3F This is a sixth flow chart of the data consistency detection method provided in an embodiment of the present application. Figure 3A The illustrated step 103 can also be implemented by following steps 1031 to 1032 , which are described in detail below.

[0256] In step 1031, the weights of the detection results corresponding to at least two detection dimensions are obtained.

[0257] In some embodiments, the weights may be set according to actual conditions. The same weights may be assigned to the detection results corresponding to different detection dimensions, or different weights may be assigned to the detection results corresponding to different detection dimensions.

[0258] In step 1032, the detection results corresponding to at least two detection dimensions are weighted and summed according to the weights to obtain a consistency detection result.

[0259] Through step 103, the detection results of multiple dimensions can be combined to comprehensively evaluate the detection results of data consistency, thereby solving the above-mentioned technical problem 1) "the detection of data consistency is not comprehensive enough, and can only be performed on offline data sets, resulting in poor applicability of the detection solution." It achieves the technical effect of comprehensively and accurately detecting the consistency of data sets, timely discovering data anomalies, and being able to detect online data sets.

[0260] See also Figure 3G , Figure 3G This is a seventh flow chart of the data consistency detection method provided in the embodiment of the present application. Figure 3A After step 103 shown, a consistency detection report may be generated, which will be described in detail below in conjunction with steps 201 and 202 .

[0261] In step 201, a consistency test report is generated, which includes the test results corresponding to each test dimension and the consistency test result.

[0262] In some embodiments, through a quantitative method, the detection results corresponding to each detection dimension used in the detection process and the consistency detection results are displayed in the same report to generate a consistency detection report.

[0263] In step 202, a consistency detection report is output.

[0264] Through steps 201 to 202, the abnormal points of data consistency can be automatically given through quantitative methods, solving technical problem 2) "the detection process mainly relies on the experience of algorithm personnel and cannot be automatically detected, thus achieving the effect of automatic detection of data consistency."

[0265] See also Figure 3H , Figure 3H This is an eighth flow chart of the data consistency detection method provided in the embodiment of the present application. Figure 3A After step 103 shown, the model parameter update of the machine learning model can also be automatically triggered, which is specifically explained in conjunction with steps 301 to 303.

[0266] In step 301 , when at least two detection dimensions include feature offset detection, first information is acquired.

[0267] The first information includes the classification test accuracy of the binary classification model, where the binary classification model is trained based on an adversarial verification dataset, and the adversarial verification dataset is constructed based on a training dataset and a prediction dataset.

[0268] In some embodiments, the first information may be obtained by:

[0269] When at least two detection dimensions include feature offset detection, the adversarial verification dataset is divided into an adversarial verification training set and an adversarial verification test set; the binary classification model is trained based on the adversarial verification training set to obtain a target binary classification model; based on the adversarial verification test set, the classification test accuracy of the target binary classification model is determined; when the classification test accuracy is greater than or equal to the classification test accuracy threshold, it is determined that there is a feature offset in the predicted dataset compared to the training dataset; based on the adversarial verification test set, the classification test accuracy of the target binary classification model is determined, including: when the adversarial verification test set includes multiple test samples, each test sample is predicted separately using the target binary classification model to obtain multiple prediction results; according to the prediction result of each test sample and the corresponding actual result, the classification test accuracy of the target binary classification model is obtained.

[0270] The classification test accuracy can be used as an indicator to evaluate the classification effect of the target binary classification model, that is, the AUC of the target binary classification model. The classification test AUC of the target binary classification model can be calculated by the following formula:

[0271]

[0272] Among them, FPR is the probability that the target binary classification model is judged as a positive sample but not a true sample, that is, the probability of judging a true negative sample as a positive sample, FP is the number of true negative samples that are incorrectly predicted as positive samples, and TN is the number of true negative samples that are correctly predicted as negative samples.

[0273]

[0274] Among them, TPR is the probability that the target binary classification model determines that the positive sample is the true sample, that is, the probability of the true sample being judged as a positive sample, TP is the number of true samples correctly predicted as positive samples, and FN is the number of true samples incorrectly predicted as negative samples.

[0275] The ROC curve corresponding to the target binary classification model is constructed according to the coordinates (FPR, TPR). The AUC value is the area under the ROC curve corresponding to the target binary classification model, which can be calculated by the following formula:

[0276]

[0277] in, is the serial number of the i-th sample (the probability score is arranged from small to large, ranked in the rank position), M is the number of positive samples, N is the number of negative samples, and Σinsi∈positive means that only the serial numbers of positive samples are added.

[0278] In step 302 , when at least two detection dimensions include pattern change detection, second information is acquired.

[0279] The second information includes the prediction accuracy of the machine learning model, the verification accuracy of the target machine learning model, and the test accuracy of the target machine learning model; wherein the target machine learning model is trained based on the prediction data set.

[0280] In some embodiments, the second information may be obtained by:

[0281] When at least two detection dimensions include pattern change detection, the prediction accuracy of the machine learning model is determined based on the prediction data set; the prediction data set is divided into a target training set and a target validation set; the machine learning model is trained based on the target training set to obtain a target machine learning model; the validation accuracy of the target machine learning model is determined based on the target validation set; the test accuracy of the target machine learning model is determined based on the training data set; the prediction accuracy of the machine learning model is determined based on the prediction data set, including: when the prediction data set includes multiple prediction samples, each prediction sample is predicted separately using the machine learning model to obtain multiple prediction results; according to the prediction result of each prediction sample and the corresponding sample label, the machine learning model is obtained. The prediction accuracy of the learning model; based on the target validation set, the validation accuracy of the target machine learning model is determined, including: when the target validation set includes multiple validation samples, each validation sample is predicted separately by the target machine learning model to obtain multiple prediction results; according to the prediction result of each validation sample and the corresponding sample label, the validation accuracy of the target machine learning model is obtained; based on the training data set, the test accuracy of the target machine learning model is determined, including: when the training data set includes multiple test samples, each test sample is predicted separately by the target machine learning model to obtain multiple prediction results; according to the prediction result of each test sample and the corresponding sample label, the test accuracy of the target machine learning model is obtained.

[0282] The prediction accuracy can be used as an indicator to evaluate the prediction effect of the machine learning model, that is, the AUC of the machine learning model. The implementation method of obtaining the prediction AUC of the machine learning model can refer to step 301; the verification accuracy can be used as an indicator to evaluate the verification effect of the target machine learning model, that is, the verification AUC of the target machine learning model. The implementation method of obtaining the verification AUC of the machine learning model can refer to step 301; the test accuracy can be used as an indicator to evaluate the test effect of the target machine learning model, that is, the test AUC of the target machine learning model. The implementation method of obtaining the test AUC of the machine learning model can refer to step 301, which will not be repeated here.

[0283] In step 303, based on at least one of the first information and the second information, when it is determined that the machine learning model update condition is met, the model parameter update of the machine learning model is automatically triggered.

[0284] In some embodiments, when at least two detection dimensions include feature offset detection, the classification test accuracy is compared with a preset threshold to obtain a classification test accuracy comparison result; when the classification test accuracy comparison result indicates that the classification test accuracy is greater than or equal to the preset threshold, the model parameter update of the machine learning model is automatically triggered; when at least two detection dimensions include pattern change detection, the prediction accuracy is compared with the test accuracy to obtain a third comparison result; the prediction accuracy is compared with the verification accuracy to obtain a fourth comparison result; when the third comparison result indicates that the prediction accuracy is greater than the test accuracy, and the fourth comparison result indicates that the prediction accuracy is less than the verification accuracy, the model parameter update of the machine learning model is automatically triggered.

[0285] Through steps 301 to 303, the machine learning model update suggestions can be automatically given through quantitative methods, and the model parameter update can be automatically triggered, which solves the technical problem 3) "the machine learning model cannot be automatically optimized after the detection is completed" and achieves the effect of automatically updating the model parameters after the detection is completed.

[0286] The embodiment of the present application detects the consistency of the training data set used to train the machine learning model and the prediction data set of the machine learning model during the application process from at least two detection dimensions, and determines the consistency detection result after collecting the detection results corresponding to each detection dimension. This solves the technical problem that "the detection of data consistency is not comprehensive enough, and can only be performed on offline data sets, resulting in poor applicability of the detection scheme". In addition, by outputting the detection results corresponding to each detection dimension and the consistency detection results in the form of a consistency detection report, the technical problem that "the detection process mainly relies on the experience of algorithm personnel and cannot be automatically detected" is solved. In addition, by setting the machine learning model update conditions, the model parameter update of the machine learning model is automatically triggered, which solves the technical problem that "the machine learning model cannot be automatically optimized after the detection is completed". Finally, comprehensive and accurate data consistency detection is achieved, adaptability is improved, data consistency anomalies and machine learning model update suggestions are automatically given, and online algorithms are evaluated in real time.

[0287] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0288] The application scenario of the embodiment of the present application can be a data intelligence scenario, and the specific product implementation includes lead rating (that is, rating the user's intention based on the user's characteristic data, for example: the intention of purchasing behavior, the intention of entering the store, the intention of clicking on the page, etc.), defeat activation (that is, activating the defeated user, the defeated user is the user who has not generated purchasing behavior, store-entering behavior, or clicking behavior within the set time. The intention of the defeated user is re-rated based on the defeated user's characteristic data, so as to increase the possibility of the defeated user generating purchasing behavior, store-entering behavior, click on the page, etc.) defeat activation, etc., which are mainly used for rating the user's intention to purchase behavior.

[0289] It is understandable that in the embodiments of the present application, when data related to user information or user behavior is involved and is applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0290] See also Figure 4A , Figure 4A This is a schematic diagram of the first interface of an exemplary user purchase intention rating provided in an embodiment of the present application. Figure 4A For a diagram of inputting a user list and performing purchase intention rating, see Figure 4B , Figure 4B This is a schematic diagram of the second interface of an exemplary user purchase intention rating provided in an embodiment of the present application. Figure 4B This is a diagram showing how to sort a user list based on their purchase intention. The specific steps for ranking user purchase intention are as follows:

[0291] Step 1): Enter the list of users to be rated, represented by a unique identification id;

[0292] Step 2): The machine learning model performs calculations, ratings, and intention rankings;

[0293] Step 3): Output the rating results of each user, which can be the rating score of each user or a list of users sorted from large to small by intention.

[0294] For customers, the effectiveness of a product is primarily determined by the accuracy of user purchase intent ratings, in addition to the normal return results for all routine calls. The product's core functionality—rating and intent ranking—is implemented by the underlying machine learning model. Therefore, the effectiveness of the product is directly correlated with the effectiveness of the machine learning model, specifically its accuracy.

[0295] Data determines the upper limit of machine learning, and machine learning models only try to approximate this upper limit as much as possible. Whether the training dataset is consistent with the test dataset will greatly affect the training effect of the machine learning model. Whether the offline dataset (training dataset) is consistent with the online dataset (prediction dataset) will also greatly affect the effect of the machine learning model application. Therefore, how to detect data anomalies and guide the update of the machine learning model is the core problem to be solved in the embodiments of this application.

[0296] See also Figure 4C , Figure 4C This is a flowchart of an exemplary user purchase intention rating service provided by an embodiment of the present application, wherein the "data consistency check" is the actual application link of the embodiment of the present application, and the specific steps are as follows:

[0297] Step 1): The customer opens the client page and enters the list of users to be rated

[0298] Step 2): The client sends the user list back to the backend server

[0299] Step 3): The backend server uses the user list to match the corresponding object features from the feature database

[0300] Step 4): Input object features into the rating model

[0301] Step 5): The model outputs each user's intention score and returns it to the backend server

[0302] Step 6): The backend server returns the intention score to the customer

[0303] Step 7): Data consistency detection evaluates data consistency by comparing and calculating the training set, test set, and prediction set to detect data anomalies and guide model updates.

[0304] Among them, the rating model (i.e., the machine learning model) is used to predict the probability of user purchase based on the received object features (the probability value range is 0-1). The probability can be used as the basis for each user's intention score. The back-end server returns each user's intention score to the customer based on the score corresponding to the different pre-set probabilities. The probability can also be used as the basis for each user's intention level. The back-end server returns the level corresponding to each user's intention to the customer based on the level corresponding to the different pre-set probabilities.

[0305] See also Figure 4D , Figure 4DIt is a schematic diagram of the first principle diagram of an exemplary data consistency detection method provided by an embodiment of the present application. The detection dimensions of the embodiment of the present application may include three dimensions, namely feature offset, label offset, and pattern change, wherein feature offset is jointly implemented by single feature offset detection and adversarial verification, label offset is implemented by label offset detection, and pattern change is implemented by reverse verification. The input of the embodiment of the present application is a training data set (the training data set may also include a training set for model training, a test set for model testing, and a verification set for model verification), a prediction data set (i.e., a prediction set), and an existing model (i.e., a machine learning model), and the output is a signal of data anomalies and whether a model update is required (the training set in the training data set is more representative, and the training set can be used for data consistency detection to obtain a detection result of the consistency between the training data set and the prediction data set. The training sets mentioned below all refer to the training sets in the training data set).

[0306] First dimension: feature offset

[0307] Feature drift refers to changes in the distribution of features between different datasets in the same scenario due to factors such as the passage of time or sampling bias. Feature drift manifests as independent and different distributions of features. Due to the definition of business scenarios, features here often have multiple dimensions. Therefore, detecting feature drift requires considering both single-feature drift and cross-feature drift. This embodiment of the application evaluates these two scenarios separately through single-feature detection and adversarial verification.

[0308] 1) Single feature detection

[0309] In the data intelligence scenario, features include both discrete and continuous features. In order to uniformly evaluate them, this embodiment of the application selects JS divergence as the calculation indicator. The JS divergence threshold of this embodiment of the application is 0.05. If it exceeds 0.05, it is considered that the feature deviation is relatively serious.

[0310]

[0311] Among them, P(x) is the probability distribution of the feature in the training data set, Q(x) is the probability distribution of the feature in the prediction data set, For P(x) The relative entropy of For Q(x) The relative entropy of , JS(P(x)||Q(x)) is the JS divergence of the feature, which is used to measure the similarity of the probability distribution of the current feature in the training set and the prediction set.

[0312] 2) Adversarial Verification

[0313] Adversarial verification is to build a two-class model, and the task of the two-class model is to distinguish whether the sample comes from the training set or the set. The original labels of the two data sets are hidden, and a new training set is generated with whether the sample comes from the set as the new sample label. The optimal two-class model (i.e., the target two-class model) is trained to calculate the accuracy performance (i.e., classification test accuracy) and feature importance (i.e., first feature importance) of the two-class model. The embodiment of the present application adopts a tree model as a two-class model, and the adversarial verification AUC (i.e., the test AUC when testing the test set during the training of the two classifier is also the classification test accuracy) is used as the accuracy index. The information gain of the feature when the tree model node is split is calculated as the feature importance index. Among them, the tree model refers to a model with a tree structure. The mathematical description of the tree model is a "piecewise function". The training of the tree model includes selecting features in turn, determining the internal nodes for dividing the feature threshold, and determining the branches or categories of the leaf nodes. The threshold value of the feature importance in the embodiment of the present application is 0.1. If it exceeds 0.1, there is a higher risk of feature offset.

[0314] The information gain of a feature can be calculated using the following formula:

[0315] g(X,Y)=H(X)-H(X|Y) (12)

[0316] Here, g(X,Y) is the information gain of the feature, or the total gain. Dividing the total gain by the number of possible values ​​of the feature gives the gain of the feature. H(X) is the empirical entropy of the new training set, and H(X|Y) is the empirical conditional entropy of the feature, which can be calculated using the following formula:

[0317]

[0318] Where n is the number of possible values ​​of X, x i is a possible value in X, P(x i ) is X and its value is x i The unit of empirical entropy is usually bits (or logarithmic units with base 2).

[0319]

[0320] Where n is the number of possible values ​​of X, x i is a possible value in X, P(x i ) is X and its value is x i The probability of Y is represented by the set of each possible value of the feature, m is the number of possible values ​​of Y, y i is a possible value in Y, P(y i ) is Y and its value is y i The probability, P(x i ,yi ) is X and its value is x i And Y takes the value y i The unit of the empirical conditional entropy is also bits (or logarithmic units with base 2).

[0321] The test AUC is calculated as follows:

[0322]

[0323] Among them, FPR is the probability that the binary classification model judges it as a positive sample but is not a true sample, that is, the probability of judging it as a positive sample among true negative samples, FP is the number of true negative samples that are incorrectly predicted as positive samples, and TN is the number of true negative samples that are correctly predicted as negative samples.

[0324]

[0325] Among them, TPR is the probability that the binary classification model determines that the positive sample is the true sample, that is, the probability of the true sample being judged as a positive sample, TP is the number of true samples correctly predicted as positive samples, and FN is the number of true samples incorrectly predicted as negative samples.

[0326] The ROC curve corresponding to the binary classification model is constructed according to the coordinates (FPR, TPR). The AUC value is the area under the ROC curve corresponding to the binary classification model, which can be calculated by the following formula:

[0327]

[0328] in, is the serial number of the i-th sample (the probability score is arranged from small to large, ranked in the rank position), M is the number of positive samples, N is the number of negative samples, and Σinsi∈positive means that only the serial numbers of positive samples are added.

[0329] As long as one feature shifts, it means that the overall prediction dataset has feature shift compared to the training dataset.

[0330] Second dimension: label offset

[0331] Label shift is generally caused by changes in the external environment or biased data sampling, and is usually manifested as a difference in the ratio of positive and negative samples between different data sets. The embodiment of the present application uses statistical methods to detect label shift, and locates the risk of data anomaly by counting the positive and negative sample ratios of the overall and each feature value. If there is a feature value that is much higher than other values ​​(the embodiment of the present application is set to a feature value that is more than twice the value of other features), it is considered that there is a label shift problem.

[0332] Use statistical methods to calculate the overall positive sample rate of the training set (training set in the training data set), the overall positive sample rate of the prediction set (prediction set in the prediction data set), the positive sample rate of each feature in the training set, and the positive sample rate of each feature in the prediction set. Compare the overall positive sample rate of the training set with the positive sample rate of each feature in the training set, and the positive sample rate between each feature in the training set. Compare the overall positive sample rate of the prediction set with the positive sample rate of each feature in the prediction set, and the positive sample rate between each feature in the prediction set. As long as the positive sample rate value of a certain item is more than twice the positive sample rate value of other items, it is considered that there is a label shift problem.

[0333] The third dimension: pattern change

[0334] The reasons for schema changes are generally complex, including the influence of feature and label shifts, as well as implicit changes in the business logic itself. This can manifest as the model's performance on the test set being significantly lower than on the training set, or the model's online application performance being significantly lower than during training. This embodiment of the application uses reverse validation to detect schema changes.

[0335] Reverse validation first uses the existing model (i.e., machine learning model) to predict the prediction set (samples in the prediction set can obtain labels within a set time), calculate the predicted AUC value (i.e., prediction accuracy) and feature importance (i.e., the second feature importance), and then use the prediction set to train the existing model to obtain a new model (i.e., target machine learning model) to obtain the reverse validation AUC (i.e., verification accuracy) and reverse validation feature importance (i.e., the third feature importance). In turn, the training set is tested to calculate the reverse validation test AUC value (i.e., test accuracy). The threshold value of the feature importance of the embodiment of the present application is 0.1. If it exceeds 0.1, there is a higher risk of feature offset. Finally, the activation detection is calculated based on the adversarial verification AUC, prediction AUC, reverse validation verification AUC, and reverse validation test AUC. The logic is shown in the code below.

[0336] Data anomalies and model updates

[0337] This step includes analyzing the results of the above tests and outputting data anomalies and model update signals to the overall framework. The output includes a test report and automatically triggers the model update process.

[0338] When a model update is triggered, a new training data set is first obtained to retrain the model to obtain a new machine learning model. The above steps are repeated to recheck the data consistency corresponding to the new machine learning model until the machine learning model update is no longer triggered and the update is stopped. If the machine learning model is an online model, the new training data set can be an online data set (the samples in the online data set can obtain the corresponding labels within the set time). If the model is an offline model, the new training data set can be a prediction set (the prediction set is the prediction set used in the reverse verification process).

[0339] When an anomaly occurs in the indicative detection, but the activation detection result does not trigger an automatic update, business personnel can further determine whether the model needs to be updated based on the indicative detection result.

[0340] See also Figure 4E , Figure 4E is a second schematic diagram of an exemplary data consistency detection method provided in an embodiment of the present application, Figure 4E Including the output information corresponding to the detection dimension used in the embodiment of this application, that is, the entire content of the detection report. The embodiment of this application will include indicative detection and activation detection. Indicative detection is mainly used to locate anomalies and provide guidance for business personnel to further analyze the problem; activation detection is mainly used as a basis for whether to update the machine learning model, see Figure 4F , Figure 4F This is a schematic diagram of a report template for an exemplary data consistency detection method provided in an embodiment of the present application. Figure 4F This is a template for data consistency check report.

[0341] The code logic to trigger the update of the machine learning model is:

[0342]

[0343] Among them, advalid_auc is the adversarial validation AUC, pred_auc is the prediction AUC, revalid_auc is the validation AUC of the reverse validation process, and revalid_pred_auc is the test AUC of the reverse validation process.

[0344] If the output signal is -1, there is no need to update the model. If it is 0, further analysis by business personnel is required. If it is 1, it is recommended to update the model.

[0345] See also Figure 4G , Figure 4G This is a first report diagram of an exemplary data consistency detection method provided in an embodiment of the present application, see Figure 4H , Figure 4H This is a second report diagram of an exemplary data consistency detection method provided in an embodiment of the present application, see Figure 4I , Figure 4I is the third report schematic diagram of the exemplary data consistency detection method provided by the embodiments of the present application. Figures 4G to 4I It is a data consistency detection report obtained by invoking the embodiments of the present application to perform data consistency detection on the most recent data set in a certain data intelligence scenario.

[0346] Taking the data in Figure 4I as an example, where advalid_auc is the adversarial validation AUC = 0.70, pred_auc is the prediction AUC = 0.64, revalid_auc is the validation AUC in the reverse validation process = 0.66, and revalid_pred_auc is the test AUC in the reverse validation process = 0.6. According to the code logic, revalid_pred_auc < pred_auc and pred_auc < revalid_auc, and at the same time advalid_auc >= 0.7. After calculation, the output signal is 1, so the machine learning model update of the overall framework will be triggered.

[0347] Next, continue to illustrate the exemplary structure of the software module implementation of the data consistency detection device 655 provided by the embodiments of the present application. In some embodiments, as Figure 2 shown, the software module in a data consistency detection device 655 stored in the memory 650 may include:

[0348] An acquisition module 6551, configured to acquire a training data set for training a machine learning model, and a prediction data set during the application of the machine learning model.

[0349] A detection module 6552, configured to detect the consistency of the training data set and the prediction data set from at least two detection dimensions respectively, and obtain detection results corresponding to each detection dimension.

[0350] A determination module 6553, configured to determine a consistency detection result by combining the detection results corresponding to the at least two detection dimensions, and the consistency detection result is used to indicate whether the data of the training data set and the prediction data set is consistent.

[0351] In some embodiments, the detection module 6552 is further used to detect the consistency of the training data set and the prediction data set from at least two detection dimensions of feature shift detection, label shift detection and pattern change detection, respectively, to obtain detection results corresponding to each detection dimension; wherein, the feature shift detection is used to indicate the detection of the consistency of the feature distribution of the training data set and the prediction data set; the label shift detection is used to indicate the detection of the consistency of the label distribution of the training data set and the prediction data set; the pattern change detection is used to indicate the detection of the consistency of the importance of the same features in the training data set and the prediction data set.

[0352] In some embodiments, the detection module 6552 is also used to perform single feature offset detection on the training data set and the prediction data set to obtain a single feature offset result when the at least two detection dimensions include the feature offset detection; wherein the single feature offset detection is used to indicate the detection of the consistency of the distribution of a single feature in the training data set and the prediction data set; perform adversarial verification on the training data set and the prediction data set to obtain an adversarial verification result; wherein the adversarial verification is used to indicate the detection of the distribution difference between the training data set and the prediction data set; and jointly determine the single feature offset result and the adversarial verification result as the detection result corresponding to the feature offset detection dimension.

[0353] In some embodiments, the detection module 6552 is further used to calculate the JS divergence of each feature that has a corresponding relationship between the training data set and the prediction data set, and obtain the JS divergence value of each feature; compare each of the JS divergence values ​​with the JS divergence threshold to obtain a comparison result; when the comparison result indicates that the JS divergence value is greater than the JS divergence threshold, it is determined that the feature corresponding to the JS divergence value has a single feature offset, and the existence of the single feature offset is used as the single feature offset result.

[0354] In some embodiments, the detection module 6552 is also used to reconstruct the labels of each feature in the training dataset and the prediction dataset to obtain an adversarial verification dataset, wherein the adversarial verification dataset includes positive sample labels and negative sample labels; wherein the positive sample label indicates that the corresponding feature comes from the training dataset, and the negative sample label indicates that the corresponding feature comes from the prediction dataset; the information gain of each feature in the adversarial verification dataset is determined respectively; based on the information gain of each feature, the first feature importance of the corresponding feature is determined; when the first feature importance of the feature is greater than the feature importance threshold, it is determined that the prediction dataset has a feature offset compared to the training dataset; and the presence of a feature offset in the prediction dataset compared to the training dataset is used as the adversarial verification result.

[0355] In some embodiments, the detection module 6552 is also used to, when the at least two detection dimensions include the label shift detection, count the first positive sample rate of the entire training dataset, the second positive sample rate of the entire prediction dataset, the third positive sample rate of each feature in the training dataset, and the fourth positive sample rate of each feature in the prediction dataset; compare the first positive sample rate with any two of the third positive sample rates to obtain a first comparison result; compare the second positive sample rate with any two of the fourth positive sample rates to obtain a second comparison result; when the first comparison result indicates that the ratio of the any two items satisfies the first ratio condition, or the second comparison result indicates that the ratio of the any two items satisfies the second ratio condition, determine that the prediction dataset has a label shift compared to the training dataset; determine that the label shift of the prediction dataset compared to the training dataset is the detection result corresponding to the label shift detection dimension.

[0356] In some embodiments, the detection module 6552 is also used to, when the at least two detection dimensions include the pattern change detection, respectively determine the information gain of each feature in the training data set; determine the second feature importance of the corresponding feature based on the information gain of each feature in the training data set; respectively determine the information gain of each feature in the prediction data set; determine the third feature importance of the corresponding feature based on the information gain of each feature in the prediction data set; and determine the detection result corresponding to the pattern change detection dimension based on the second feature importance and the third feature importance.

[0357] In some embodiments, the detection module 6552 is also used to sort the multiple features in the training data set based on the second feature importance to obtain the first rank of each feature; sort the multiple features in the prediction data set based on the third feature importance to obtain the second rank of each feature; compare the first rank and the second rank of the same feature in the training data set and the prediction data set to obtain a rank comparison result; when the rank comparison result indicates that there is a difference between the first rank and the second rank of at least one feature, it is determined that there is a pattern change in the machine learning model, and the existence of the pattern change in the machine learning model is used as the detection result corresponding to the pattern change detection dimension.

[0358] In some embodiments, the determination module 6553 is further used to obtain the weights of the detection results corresponding to the at least two detection dimensions; and perform weighted summation on the detection results corresponding to the at least two detection dimensions according to the weights to obtain the consistency detection result.

[0359] In some embodiments, the determination module 6553 is further used to generate a consistency detection report, wherein the consistency detection report includes the detection results corresponding to each of the detection dimensions and the consistency detection result; and output the consistency detection report.

[0360] In some embodiments, the detection module 6552 is further used to obtain first information when the at least two detection dimensions include the feature offset detection; wherein the first information includes the classification test accuracy of the binary classification model, and the binary classification model is trained based on the adversarial verification dataset, and the adversarial verification dataset is constructed based on the training dataset and the prediction dataset; when the at least two detection dimensions include the pattern change detection, obtain second information; wherein the second information includes the prediction accuracy of the machine learning model, the verification accuracy of the target machine learning model and the test accuracy of the target machine learning model; wherein the target machine learning model is trained based on the prediction dataset.

[0361] In some embodiments, the determination module 6553 is also used to automatically trigger the update of model parameters of the machine learning model when it is determined that the machine learning model update condition is met based on at least one of the first information and the second information.

[0362] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data consistency detection method described above in the embodiment of the present application.

[0363] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the data consistency detection method provided in the embodiment of the present application, for example, Figures 3A to 3H The data consistency detection method is shown.

[0364] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0365] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0366] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0367] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0368] In summary, through the embodiments of the present application, the consistency of data can be jointly detected from multiple dimensions, making the data consistency more comprehensive and improving adaptability; through a quantitative method, it is possible to automatically provide anomalies in data consistency and update suggestions for machine learning models; and it is possible to realize real-time evaluation of online algorithms; it can be seen that the embodiments of the present application are more comprehensive, more accurate, more widely applicable, more suitable for the needs of real business scenarios, and have clearer guiding significance for the actual implementation of the algorithm.

[0369] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data consistency detection method, characterized in that: The method comprises: Obtaining a training data set for training a machine learning model and a prediction data set for the machine learning model during application; Testing the consistency of the training dataset and the prediction dataset from at least two detection dimensions, respectively, to obtain detection results corresponding to each detection dimension; The detection results corresponding to the at least two detection dimensions are combined to determine a consistency detection result, where the consistency detection result is used to indicate whether the data of the training dataset and the prediction dataset are consistent.

2. The method according to claim 1, characterized in that The consistency of the training dataset and the prediction dataset is detected from at least two detection dimensions to obtain detection results corresponding to each detection dimension, including: From at least two detection dimensions of feature shift detection, label shift detection, and pattern change detection, respectively, the consistency of the training dataset and the prediction dataset is detected to obtain detection results corresponding to each detection dimension; The feature offset detection is used to indicate the detection of consistency of feature distributions of the training dataset and the prediction dataset; The label shift detection is used to instruct to detect the consistency of the label distribution of the training data set and the prediction data set; The pattern change detection is used to detect the consistency of the importance of the same features in the training dataset and the prediction dataset.

3. The method according to claim 2, characterized in that The feature shift detection includes single feature shift detection and adversarial verification. The consistency of the training dataset and the prediction dataset is detected from at least two detection dimensions of feature shift detection, label shift detection, and pattern change detection, respectively, to obtain the detection results corresponding to each detection dimension, including: When the at least two detection dimensions include the feature offset detection, performing single feature offset detection on the training dataset and the prediction dataset to obtain a single feature offset result; The single feature offset detection is used to indicate the detection of the consistency of the distribution of a single feature in the training data set and the prediction data set; Performing adversarial verification on the training data set and the prediction data set to obtain an adversarial verification result; The adversarial verification is used to instruct to detect the distribution difference between the training data set and the prediction data set; The single feature shift result and the adversarial verification result are jointly determined as the detection result corresponding to the feature shift detection dimension.

4. The method according to claim 3, characterized in that The performing single feature offset detection on the training data set and the prediction data set to obtain a single feature offset result includes: Calculate the JS divergence of each feature that has a corresponding relationship between the training dataset and the prediction dataset, and obtain the JS divergence value of each feature; Comparing each of the JS divergence values ​​with the JS divergence threshold value to obtain a comparison result; When the comparison result indicates that the JS divergence value is greater than the JS divergence threshold, it is determined that a single feature offset exists in the feature corresponding to the JS divergence value, and the existence of the single feature offset is used as the single feature offset result.

5. The method according to claim 3, characterized in that The performing adversarial verification on the training data set and the prediction data set to obtain an adversarial verification result includes: Reconstructing the labels of each feature in the training dataset and the prediction dataset to obtain an adversarial verification dataset, wherein the adversarial verification dataset includes positive sample labels and negative sample labels; The positive sample label indicates that the corresponding feature comes from the training dataset, and the negative sample label indicates that the corresponding feature comes from the prediction dataset; Determining the information gain of each of the features in the adversarial verification dataset respectively; Determining a first feature importance of each feature based on the information gain of the feature; When the first feature importance of the feature is greater than the feature importance threshold, determining that the predicted dataset has a feature offset compared to the training dataset; The presence of feature shift in the predicted dataset compared to the training dataset is taken as an adversarial verification result.

6. The method according to claim 2, characterized in that The consistency of the training dataset and the prediction dataset is detected from at least two detection dimensions of feature shift detection, label shift detection, and pattern change detection, respectively, to obtain detection results corresponding to each detection dimension, including: When the at least two detection dimensions include the label shift detection, counting a first positive sample rate of the entire training dataset, a second positive sample rate of the entire prediction dataset, a third positive sample rate of each feature in the training dataset, and a fourth positive sample rate of each feature in the prediction dataset; Compare the first positive sample rate and any two of the third positive sample rates to obtain a first comparison result; Comparing any two of the second positive sample rate and each of the fourth positive sample rates to obtain a second comparison result; When the first comparison result indicates that the ratio of the two arbitrary items satisfies a first ratio condition, or the second comparison result indicates that the ratio of the two arbitrary items satisfies a second ratio condition, determining that there is a label shift in the prediction dataset compared to the training dataset; Determining whether the predicted dataset has a label shift compared to the training dataset is a detection result corresponding to the label shift detection dimension.

7. The method according to claim 2, characterized in that The consistency of the training dataset and the prediction dataset is detected from at least two detection dimensions of feature shift detection, label shift detection, and pattern change detection, respectively, to obtain detection results corresponding to each detection dimension, including: When the at least two detection dimensions include the pattern change detection, determining the information gain of each feature in the training dataset respectively; Determining a second feature importance of each feature in the training data set according to the information gain of the corresponding feature; Determining the information gain of each feature in the prediction data set respectively; Determining a third feature importance of each feature in the prediction data set according to the information gain of the corresponding feature; Based on the second feature importance and the third feature importance, a detection result corresponding to the pattern change detection dimension is determined.

8. The method according to claim 7, characterized in that The determining, based on the second feature importance and the third feature importance, a detection result corresponding to the pattern change detection dimension includes: Based on the second feature importance, sorting multiple features in the training data set to obtain a first rank for each feature; Based on the third feature importance, sorting multiple features in the prediction data set to obtain a second rank for each feature; Comparing the first rank and the second rank of the same feature in the training dataset and the prediction dataset to obtain a rank comparison result; When the ranking comparison result indicates that there is a difference between the first ranking and the second ranking of at least one feature, it is determined that there is a pattern change in the machine learning model, and the pattern change in the machine learning model is used as the detection result corresponding to the pattern change detection dimension.

9. The method according to claim 1, characterized in that The combining the detection results corresponding to the at least two detection dimensions to determine the consistency detection result includes: Obtaining weights of the detection results corresponding to the at least two detection dimensions; The detection results corresponding to the at least two detection dimensions are weightedly summed according to the weights to obtain the consistency detection result.

10. The method according to any one of claims 1 to 9, characterized in that After determining the consistency test result, the method further includes: Generate a consistency test report, wherein the consistency test report includes the test results corresponding to each of the test dimensions and the consistency test result; Output the consistency detection report.

11. The method according to any one of claims 1 to 9, characterized in that After determining the consistency test result, the method further includes: When the at least two detection dimensions include the feature offset detection, acquiring first information; The first information includes a classification test accuracy of a binary classification model, wherein the binary classification model is trained based on an adversarial verification dataset, and the adversarial verification dataset is constructed based on the training dataset and the prediction dataset; When the at least two detection dimensions include the mode change detection, acquiring second information; The second information includes the prediction accuracy of the machine learning model, the verification accuracy of the target machine learning model, and the test accuracy of the target machine learning model; wherein the target machine learning model is trained based on the prediction data set; Based on at least one of the first information and the second information, when it is determined that the machine learning model update condition is met, the model parameter update of the machine learning model is automatically triggered.

12. A data consistency detection device, characterized in that: The device comprises: An acquisition module, configured to acquire a training data set for training a machine learning model and a prediction data set of the machine learning model during application; A detection module, configured to detect the consistency of the training dataset and the prediction dataset from at least two detection dimensions, and obtain detection results corresponding to each detection dimension; The determination module is used to determine a consistency detection result based on the detection results corresponding to the at least two detection dimensions, and the consistency detection result is used to indicate whether the data of the training data set and the prediction data set are consistent.

13. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions; The processor is configured to implement the data consistency detection method according to any one of claims 1 to 11 when executing the computer-executable instructions stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data consistency detection method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the data consistency detection method according to any one of claims 1 to 11 is implemented.

Images