Data processing method and apparatus
By obtaining predicted samples to calculate model quality scores, determining training labels, and training a quality detection model, the problem of decreased accuracy of machine learning models when data changes is solved, and the quality monitoring of models and samples is realized, thus improving the efficiency of model quality assurance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG E COMMERCE BANK CO LTD
- Filing Date
- 2023-02-13
- Publication Date
- 2026-07-14
AI Technical Summary
Existing machine learning models suffer from decreased accuracy and difficulty in effectively monitoring changes in data, resulting in unstable model quality. Furthermore, the monitoring metrics do not align with business needs, making it impossible to detect anomalies in models and samples in a timely manner.
By acquiring prediction samples, calculating model quality scores, and determining training labels when the quality scores exceed a threshold, an initial quality detection model is trained. The target quality detection model is then used to monitor the output results and input data of the business model, ensuring the quality of the model and samples.
It enables quality monitoring of business models, improves the efficiency of model quality assurance, and ensures the accuracy of model output results and the stability of input data.
Smart Images
Figure CN116468144B_ABST
Abstract
Description
Technical Field
[0001] The embodiments in this specification relate to the field of computer technology, and in particular to data processing methods. Background Technology
[0002] Currently, an increasing number of platforms are using machine learning models as tools for prediction or scoring. As platform network traffic increases, so does the number of customers. For the machine learning models currently in use, changes in the input data can affect the model's accuracy, so it's necessary to monitor the model's quality. Therefore, a better solution is urgently needed. Summary of the Invention
[0003] In view of this, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, to address technical deficiencies in the prior art.
[0004] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:
[0005] Obtain prediction samples for the business model, and determine the model quality score of the business model based on the prediction samples;
[0006] If the model quality score is greater than the quality score threshold, determine the training label of the predicted sample;
[0007] Based on the predicted samples and the training labels of the predicted samples, the first initial quality detection model is trained to obtain the first target quality detection model;
[0008] Obtain at least two output results of the business model for the target dataset, and input the at least two output results into the first target quality detection model to determine the quality data of the business model.
[0009] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:
[0010] The sample acquisition module is configured to acquire prediction samples for the business model and determine the model quality score of the business model based on the prediction samples.
[0011] The label determination module is configured to determine the training label of the predicted sample when the model quality score is greater than a quality score threshold.
[0012] The model training module is configured to train the first initial quality detection model based on the predicted sample and the training label of the predicted sample to obtain the first target quality detection model.
[0013] The quality determination module is configured to acquire at least two output results of the business model for the target dataset, and input the at least two output results into the first target quality detection model to determine the quality data of the business model.
[0014] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:
[0015] Memory and processor;
[0016] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.
[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.
[0018] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.
[0019] This specification provides a data processing method and apparatus. The data processing method includes: acquiring prediction samples for a business model; determining a model quality score for the business model based on the prediction samples; determining training labels for the prediction samples if the model quality score is greater than a quality score threshold; training a first initial quality detection model based on the prediction samples and the training labels to obtain a first target quality detection model; acquiring at least two output results of the business model for a target dataset, and inputting the at least two output results into the first target quality detection model to determine the quality data of the business model. By acquiring at least two output results of the business model for a target dataset and inputting the at least two output results into the first target quality detection model to determine the quality data of the business model, the quality data of the business model is determined, thereby achieving the goal of scoring the business model, monitoring the quality of the business model, and improving the efficiency of model quality assurance. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of the architecture of a data processing method provided in one embodiment of this specification.
[0021] Figure 2a This is a flowchart illustrating a data processing method provided in one embodiment of this specification;
[0022] Figure 2bThis is a problem classification model architecture diagram of a data processing method provided in one embodiment of this specification;
[0023] Figure 2c This is an overall architecture diagram of a data processing method provided in one embodiment of this specification;
[0024] Figure 2d This is a sample quality assurance system architecture diagram of a data processing method provided in one embodiment of this specification;
[0025] Figure 3 This is a schematic diagram of the structure of a data processing device provided in one embodiment of this specification;
[0026] Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation
[0027] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.
[0028] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0029] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."
[0030] First, the terms and concepts used in one or more embodiments of this specification will be explained.
[0031] PSI: Sample Stability Index is an important indicator that measures the offset caused by sample changes, and is usually used to measure the stability of a sample.
[0032] Sample information content: measures the degree of difference between samples. The higher the sample information content, the greater the difference between samples.
[0033] Training samples: A set of samples used for training the business model.
[0034] Prediction Samples: After the business model is trained, the online data is input into the model.
[0035] Null value rate: The percentage of empty samples in a set of samples.
[0036] Single value rate: The percentage of samples with the highest percentage of values in a set of samples.
[0037] Currently, deep learning model-based algorithms are facing increasing quality and technical risks. These risks include the lack of transparency in decision-making due to the algorithm's "black box" nature, and decision bias caused by data discrimination. Furthermore, technical risks also include numerous online issues arising from data problems during algorithm development. Specifically, these include insufficient sensitivity of the algorithm model to data changes, difficulty in detecting abnormal null and single-value rates in feature metrics, discrepancies between the distribution of training data and new sample distributions leading to unexpected score distributions for new data, and errors in model judgment caused by the presence of data types not present in the training data in new samples.
[0038] Furthermore, potential technical risks also include a misalignment between the focus of the indicator monitoring system and business needs. Specifically, indicator monitoring requires setting verification rules based on requirements, and monitoring can only be applied to a single field at a time. Currently used indicator monitoring methods are unsuitable for algorithmic needs, such as changes in sample distribution. The monitoring of indicator data is difficult to align with the user dimension, resulting in insensitivity to indicator fluctuations and time-series anomalies.
[0039] Based on this, a data processing method is provided in this specification. This specification also relates to a data processing apparatus, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.
[0040] See Figure 1 , Figure 1 A schematic diagram of the architecture of a data processing method provided according to one embodiment of this specification is shown.
[0041] The task of ensuring the quality of algorithm services mainly includes two aspects: model quality assurance and sample quality assurance.
[0042] Model quality assurance primarily ensures the stable computational capability of the business model for business problems. It mainly focuses on two aspects: long-term model performance stability and stability assessment of newly added prediction samples. Performance stability is specifically determined through PSI score stability calculations, enabling business performance awareness. Stability assessment of newly added prediction samples is achieved through predicting problems with these samples, providing early warnings of business issues.
[0043] Sample quality assurance primarily ensures the stability of the predicted sample distribution for the input business model. Sample quality assurance is achieved through three aspects: comparing the distribution of predicted samples with training samples, statistical indicator analysis, and calculating sample information content. The calculation of sample information content between predicted and training samples is implemented through training sample evaluation. The comparison of the distribution of predicted and training samples and the statistical indicator analysis can be achieved by calculating the null value rate or single value rate, or by calculating indicators or features (PSI), enabling the detection of model anomalies.
[0044] The data processing method described in this specification enables the scoring of business models, thereby achieving the goal of monitoring the quality of business models and improving the efficiency of model quality assurance.
[0045] See Figure 2a , Figure 2a A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.
[0046] Step 202: Obtain prediction samples for the business model, and determine the model quality score of the business model based on the prediction samples.
[0047] The business model can be a machine learning model used for business operations; it is also called a business algorithm model, such as a model for scoring borrowers in a lending scenario or a model for scoring users in a wealth management scenario. The prediction sample can be a sample collected over a period of time, such as data samples collected from the business platform over a month. The model quality score is used to evaluate the performance quality of the model.
[0048] In practical applications, model quality assurance is primarily used to ensure the stable computational capability of the business model for solving business problems. Model quality assurance mainly focuses on two aspects: long-term model performance stability and the stability assessment of newly added prediction samples. A sample stability index can be used to evaluate the stability of the input samples to the model, thereby determining whether the model needs adjustment to better match the current sample data.
[0049] For example, in a lending scenario, data on borrowers collected from a lending platform yields a prediction sample. This prediction sample consists of data from 5,000 users. Based on this data, a quality score is determined for the borrower rating model. This borrower rating model can score borrowers, and based on this score, a borrower's repayment credit score can be determined.
[0050] The embodiments in this specification demonstrate that by obtaining prediction samples for a business model, the model quality score of the business model can be determined, thereby enabling the detection of the quality of the business model.
[0051] Specifically, data can be periodically obtained from the business platform, and the samples can be input into the business model to determine the quality score of the model. The specific implementation method is as follows.
[0052] In one possible implementation, obtaining prediction samples for the business model and determining the model quality score of the business model based on the prediction samples includes:
[0053] The business platform acquires business data within a preset period from the business platform and uses the business data within the preset period as the prediction sample, wherein the business platform includes the business model;
[0054] The predicted sample is input into the business model to obtain the sample score distribution for the predicted sample;
[0055] The model quality score of the business model is determined based on the sample score distribution.
[0056] Here, the business platform can be a platform for business processing, such as a wealth management platform or a lending platform. The preset period can be a time period, such as one month. Business data can be data related to the business platform, such as user data, including users' ages and birthdays. The sample score distribution can be the distribution of rating results; for example, if each user's rating is between 0 and 1, the sample score distribution could be the score distribution of 5000 users.
[0057] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
[0058] In practical applications, after the predicted sample is input into the business algorithm model, its PSI value can be calculated through the score distribution output by the model.
[0059] For example, in a lending scenario, data on borrowers is collected from the lending platform to obtain a prediction sample. The prediction sample consists of data from 5,000 users. This data is then input into a lending user rating model to determine the ratings for these 5,000 users. Based on these ratings, the model quality score of the business model is determined.
[0060] In the embodiments described in this specification, the model quality score is determined by calculating the model results, thereby enabling the monitoring of the model quality score.
[0061] Specifically, the model's quality score can be determined using the model stability calculation formula, and the specific implementation method is as follows.
[0062] In one feasible approach, determining the model quality score of the business model based on the sample score distribution includes:
[0063] The first and second samples in the predicted samples are determined based on the sample score distribution;
[0064] Based on the model stability calculation formula, the model quality score of the business model is determined according to the first sample and the second sample.
[0065] Here, the first sample and the second sample are samples with two different attribute characteristics in the predicted samples. For example, the age distribution corresponding to the first sample is over sixty years old, and the age distribution corresponding to the second sample is under eighteen years old. The model stability calculation formula can be the PSI calculation formula.
[0066] In practical applications, the PSI (Power Score Index) formula can be used to determine the model quality score. When a batch of samples is input into the model, business personnel or sample labelers have certain distributional expectations for the model's output results, that is, they have a rough estimate of the proportion of each score interval. The PSI calculation can assess the deviation between the model's output score distribution and the expected distribution. The higher the PSI score, the greater the deviation between the model's output score distribution and the expected distribution. Furthermore, setting more PSI intervals results in more accurate distribution deviation calculations, but also makes the model more susceptible to sample noise.
[0067] For example, in a lending scenario, data on borrowers is collected from lending platforms to obtain a prediction sample. The prediction sample consists of data from 5,000 users. This data is then input into a lending user rating model to determine the ratings for these 5,000 users. Based on these ratings, the PSI calculation formula is used to determine the model quality score.
[0068] The embodiments in this specification determine the stability of the business model output results through the PSI calculation formula, thereby achieving the effect of scoring the business model.
[0069] Furthermore, quality assurance can be performed not only based on the output of the business model, but also based on the data input to the business model. The specific implementation method is as follows.
[0070] In one possible implementation, after obtaining the prediction samples for the business model, the method further includes:
[0071] The predicted samples are input into the distribution feature extraction network layer to obtain the distribution features;
[0072] The distribution features are input into the second initial quality detection model to obtain the predicted problem label for the predicted sample;
[0073] Based on the training labels for the prediction question labels of the prediction samples, the model parameters of the second initial quality detection model are adjusted to obtain the second target quality detection model.
[0074] Among them, the distributed feature extraction network layer can be a network layer for feature extraction, and the second initial quality detection model can be a model for monitoring the data quality of the business model.
[0075] In practical applications, see Figure 2b , Figure 2b This is a problem classification model architecture diagram of a data processing method provided in one embodiment of this specification. The problem category prediction link based on the predicted sample is as follows: the predicted sample is input into the distribution feature extractor to obtain the distribution features of the predicted sample, the distribution features are input into the trained problem prediction network to predict the problem, and finally the problem category output by the model is compared with the manually judged category to calculate the model performance.
[0076] For example, in a lending scenario, data from borrowers collected from a lending platform yields a prediction sample. This prediction sample consists of data from 5,000 users. This data is then input into a distribution feature extraction network layer to obtain distribution features. These distribution features are then input into a second initial quality detection model to obtain the problem classification output by the second initial quality detection model. It is then determined whether the problem classification output by the second initial quality detection model corresponds to the actual problem classification. If they do not correspond, it indicates that the model's prediction results are inaccurate, and the model parameters need to be adjusted further to obtain the second target quality detection model.
[0077] In this embodiment, by training a second initial quality detection model, the parameters of the second initial quality detection model are changed, resulting in a more accurate second target quality detection model. This improves the accuracy of problem classification.
[0078] After training the second target quality detection model, the input data of the business model can be detected. The specific implementation method is as follows.
[0079] In one possible implementation, after obtaining the second target quality detection model, the method further includes:
[0080] The predicted sample is sent to the second target quality detection model to obtain the predicted problem label for the predicted sample;
[0081] The quality data of the business model is determined based on the prediction question labels for the prediction samples.
[0082] The second objective quality detection model can be a pre-trained detection model. The predicted question label can be the label output by the second objective quality detection model, that is, the output question classification.
[0083] In practical applications, a well-trained secondary target quality detection model can be used to ensure the quality of the input data of the model.
[0084] For example, in a lending scenario, data from borrowers collected from a lending platform is used to obtain a prediction sample. The prediction sample consists of data from 5,000 users. This data from 5,000 users is then input into a distribution feature extraction network layer to obtain distribution features. These distribution features are then input into a second objective quality detection model to obtain the problem classification corresponding to the data.
[0085] In this embodiment of the specification, by inputting the data of the input business model into a second target quality inspection model for problem detection, the problems of the data in the business model are identified, thereby achieving quality assurance.
[0086] Step 204: If the model quality score is greater than the quality score threshold, determine the training label of the predicted sample.
[0087] The quality score threshold can be a set threshold used to determine whether the predicted sample is abnormal.
[0088] In practical applications, see Figure 2b After the predicted sample is input into the business algorithm model, its PSI value can be calculated through the score distribution output by the model. When the PSI value is greater than our preset threshold, it is determined that the predicted sample is abnormal. Business personnel will manually locate the problem category and then put it into the database.
[0089] For example, in a lending scenario, data from borrowers collected from lending platforms yields a prediction sample. This prediction sample consists of data from 5,000 users. This data is then input into a lending user rating model to determine the ratings for each of the 5,000 users. Based on these ratings, the PSI (Power Score Index) formula is used to calculate the model's quality score. If the model's quality score exceeds a certain threshold, the prediction sample is assigned a corresponding training label.
[0090] In the embodiments described in this specification, samples are labeled when anomalies occur in the predicted samples, thereby enabling the training of a model for quality assurance.
[0091] Specifically, different question labels can be determined by setting corresponding classification rules, and these labels can be used as training labels for the prediction samples. The specific implementation method is as follows.
[0092] In one possible implementation, determining the training label of the predicted sample includes:
[0093] The features of the predicted sample are determined, and the problem is classified according to the features of the predicted sample using a preset classification rule to determine the training label of the predicted sample.
[0094] The features of the predicted samples can be characteristics such as the user's age. The preset classification rules can be pre-defined problem classification rules. For example, for financial projects, users under the age of 18 are considered to be underage.
[0095] In practical applications, see Figure 2b The problem category prediction link based on model score is as follows: After the prediction sample is input into the business algorithm model, the distribution features of the model output score are obtained through the distribution feature extractor. The distribution features are then input into the trained problem prediction network for problem prediction. Finally, the problem category output by the model is compared with the manually judged category to calculate the model performance.
[0096] For example, in a lending scenario, data from borrowers collected from lending platforms yields a prediction sample. This prediction sample consists of data from 5000 users. This data is then input into a lending user rating model to determine the ratings for each of the 5000 users. Based on these ratings, the PSI (Power Score Index) formula is used to determine the model's quality score. If the model quality score exceeds a certain threshold, the features of the prediction sample are determined. For instance, in the case of financial projects, users under 18 years old are considered to be underage, so corresponding training labels are assigned to users under 18 years old in the prediction sample.
[0097] The embodiments in this specification train a first initial quality detection model by labeling the predicted samples.
[0098] Step 206: Based on the predicted samples and the training labels of the predicted samples, train the first initial quality detection model to obtain the first target quality detection model.
[0099] The first initial quality detection model can be an untrained detection model used to detect the output results of the business model. Correspondingly, the first target quality detection model can be a trained detection model used to detect the output results of the business model.
[0100] In practical applications, before using the first target quality detection model, it is also necessary to train the first initial quality detection model so that the first initial quality detection model can judge the results of the business model.
[0101] For example, in a lending scenario, data from borrowers collected from a lending platform yields a prediction sample. This prediction sample consists of data from 5000 users. This data is then input into a user rating model to determine the ratings for each of the 5000 users. Based on these ratings, a distribution feature extraction network layer is used to obtain distribution features. These distribution features are then input into a first initial quality detection model to obtain the corresponding problem classification. It is then determined whether the problem classification output by the first initial quality detection model corresponds to the actual problem classification. If they do not correspond, the model's prediction is inaccurate, and further adjustments to the model parameters are needed to obtain the first target quality detection model.
[0102] In one possible implementation, training the first initial quality detection model based on the predicted samples and the training labels of the predicted samples to obtain the first target quality detection model includes:
[0103] The predicted sample is input into the business model to obtain the sample score distribution of the business model for the predicted sample;
[0104] The distribution feature extraction network layer extracts features from the sample score distribution to obtain the distribution features;
[0105] The distribution features are input into the first initial quality detection model to obtain the predicted problem labels for the distribution features;
[0106] Based on the predicted question label for the distribution characteristics and the training label, the model parameters of the first initial quality detection model are adjusted to obtain the first target quality detection model.
[0107] In practical applications, the predicted samples are input into a distribution feature extractor to obtain the distribution features of the predicted samples. The distribution features are then input into a trained question prediction network to predict the question. Finally, the question category output by the model is compared with the manually determined category to calculate the model performance.
[0108] For example, in a lending scenario, data from borrowers collected from a lending platform yields a prediction sample. This prediction sample consists of data from 5000 users. This data is then input into a user rating model to determine the ratings for each of the 5000 users. Based on these ratings, a distribution feature extraction network layer is used to obtain distribution features. These distribution features are then input into a first initial quality detection model to obtain the corresponding problem classification. It is then determined whether the problem classification output by the first initial quality detection model corresponds to the actual problem classification. If they do not correspond, the model's prediction is inaccurate, and further adjustments to the model parameters are needed to obtain the first target quality detection model.
[0109] The embodiments in this specification train a first initial quality detection model to obtain a first target quality detection model, so as to achieve quality assurance of the output results of the business model.
[0110] Step 208: Obtain at least two output results of the business model for the target dataset, and input the at least two output results into the first target quality detection model to determine the quality data of the business model.
[0111] The target dataset can be a real-time dataset obtained by the business platform.
[0112] In practical applications, after training the first initial quality detection model to obtain the first target quality detection model, it can be applied.
[0113] For example, in a lending scenario, data from borrowers collected from a lending platform yields a prediction sample of 5000 users. This data is then input into a user rating model to determine the ratings for each user. Based on these ratings, a distribution feature extraction network layer is used to obtain distribution features. These features are then input into a first initial quality detection model to obtain the corresponding problem classification. It is then determined whether the problem classification output by the first initial quality detection model corresponds to the actual problem classification. If they do not correspond, the model's prediction is inaccurate, and the model parameters need further adjustment to obtain the first target quality detection model. Input data from the lending platform is acquired in real-time and input into a business model, generating at least two output results. These at least two output results are then input into the first target quality detection model to obtain the problem classifications corresponding to at least two output results.
[0114] The embodiments in this specification use a first target quality detection model to ensure the quality of the output results of the business model, thereby ensuring that the output results of the business model can be monitored.
[0115] In one possible implementation, inputting the at least two output results into the first target quality detection model to determine the quality data of the business model includes:
[0116] The at least two output results are input into the first target quality detection model to obtain the predicted problem label for the predicted sample;
[0117] The quality data of the business model is determined based on the prediction question labels for the prediction samples.
[0118] In practical applications, the quality data of the business model can be determined after obtaining the output of the first target quality detection model.
[0119] For example, in a lending scenario, data from borrowers collected from a lending platform yields a prediction sample of 5000 users. This sample is then input into a user rating model to determine user scores. These scores are then fed into a feature extraction network layer to obtain distribution features. These distribution features are then input into a first initial quality detection model to obtain the corresponding problem classification. It is determined whether the problem classification output by the first initial quality detection model corresponds to the actual problem classification. If they do not correspond, the model's prediction is inaccurate, and the model parameters need further adjustment to obtain the first target quality detection model. Input data from the lending platform is acquired in real-time and input into a business model. The output of the business model is then input into a feature extraction network layer to obtain distribution features. These distribution features are then input into the first target quality detection model to obtain at least two problem classifications corresponding to the output results.
[0120] In one possible implementation, obtaining at least two outputs of the business model for the target dataset includes:
[0121] The target dataset is obtained from the business platform, and the target dataset is input into the business model to obtain the at least two output results.
[0122] In practical applications, data can be acquired in real time through a business platform and processed through a business model to obtain output results.
[0123] For example, in a lending scenario, the target dataset corresponding to the borrowers is obtained in real time from the lending platform. The target dataset consists of data from 5,000 users. The data of these 5,000 users is input into the lending user rating model to determine the rating of the 5,000 users, thus obtaining the output result.
[0124] Further, see Figure 2c , Figure 2c This is an overall architecture diagram of a data processing method provided in one embodiment of this specification. In summary, the business model is first trained using training samples. Then, using the predicted samples and the model score output by the business model, a classification network is used to predict the problem category. Before predicting the problem category using the classification network, HaiXun trains the classification network. Specifically, a PSI score is calculated based on the model score output by the business model for the predicted sample. It is then determined whether the PSI score is greater than a PSI threshold. If the PSI score is greater than the threshold, it indicates that the corresponding predicted sample has a problem; it is then labeled and stored in the database. If the PSI score is not greater than the threshold, it indicates that the corresponding predicted sample is a good sample, and it is also stored in the database. The classification network is trained using labeled samples from the database to obtain a classification network capable of accurately predicting the problem category.
[0125] It should be noted that, in addition to the model quality assurance mentioned above, sample quality assurance is also included. Sample quality assurance is mainly used to ensure the stability of the distribution of predicted samples input to the business model. Sample quality assurance is achieved through three aspects: comparing the distribution of predicted samples with training samples, statistical indicator analysis, and calculating the sample information content. The core of sample quality assurance is to ensure that the model training data and predicted data are consistent in the overall data distribution, and that the training samples should completely cover the types of predicted samples. See [link to relevant documentation]. Figure 2d The sample quality assurance system architecture in the embodiments of this specification is as follows: Figure 2d As shown, the sample quality assurance module mainly includes four indicators: single value rate, null value rate, feature PSI, and sample information content. Only after the predicted sample meets these four indicators can it be used as a docile sample to train the business model; otherwise, it will be labeled and classified as a problem.
[0126] In one possible implementation, after obtaining the prediction samples for the business model, the method further includes:
[0127] Obtain training samples of the business model and determine the degree of difference between the training samples and the prediction samples;
[0128] The sample quality data of the predicted sample is determined based on the degree of difference.
[0129] The difference can be the difference in information content between the training sample and the predicted sample, or the difference in PSI score between the training sample and the predicted sample.
[0130] In practical applications, model score stability monitoring can only reflect the deviation between the score distribution and the preset distribution. During the process of batch accessing user data, it cannot be guaranteed that the samples in each batch are evenly distributed. Therefore, score distribution early warning needs to be evaluated by combining the information content of the predicted samples and the distribution of the predicted samples.
[0131] Specifically, there are two main logics: If the information content of the predicted sample is greater than that of the training sample, it means that the training sample cannot fully contain the user situation of the predicted sample, and the model needs to be evaluated and additional training data added. If the PSI of the predicted sample distribution is greater than 0.1 compared with the training sample distribution, it means that the distributions of the predicted sample and the training sample are inconsistent, which may lead to errors in the score PSI, and manual investigation is required.
[0132] In one possible implementation, after obtaining the prediction samples for the business model, the method further includes:
[0133] The target data in the prediction sample is determined, and the sample quality data of the prediction sample is determined according to the proportion of the target data.
[0134] The target data can be null values or identical values in the data corresponding to the sample.
[0135] In practical applications, this also includes monitoring the predicted sample null value rate and single value rate. The null value rate and single value rate monitoring primarily involve two deployment logics: monitoring changes in the indicator itself and monitoring changes in indicator processing. Monitoring changes in the indicator itself: Since some indicators may change during calculation, such as changing the indicator name, this type of indicator change can be detected by monitoring the null value rate. Monitoring changes in indicator processing: Changes in the indicator processing logic may alter the definition of some values, such as changing the indicator value from [0,1] to [0,10]. This type of change can be detected by monitoring the single value rate.
[0136] This specification provides a data processing method and apparatus. The data processing method includes: acquiring prediction samples for a business model; determining a model quality score for the business model based on the prediction samples; determining training labels for the prediction samples if the model quality score is greater than a quality score threshold; training a first initial quality detection model based on the prediction samples and the training labels to obtain a first target quality detection model; acquiring at least two output results of the business model for a target dataset, and inputting the at least two output results into the first target quality detection model to determine the quality data of the business model. By acquiring at least two output results of the business model for a target dataset and inputting the at least two output results into the first target quality detection model to determine the quality data of the business model, the quality data of the business model is determined, thereby achieving the goal of scoring the business model, monitoring the quality of the business model, and improving the efficiency of model quality assurance.
[0137] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 3 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 3 As shown, the device includes:
[0138] The sample acquisition module 302 is configured to acquire prediction samples for the business model and determine the model quality score of the business model based on the prediction samples.
[0139] The label determination module 304 is configured to determine the training label of the predicted sample when the model quality score is greater than the quality score threshold.
[0140] The model training module 306 is configured to train the first initial quality detection model based on the predicted sample and the training label of the predicted sample to obtain the first target quality detection model.
[0141] The quality determination module 308 is configured to acquire at least two output results of the business model for the target dataset, and input the at least two output results into the first target quality detection model to determine the quality data of the business model.
[0142] In one possible implementation, the sample acquisition module 302 is further configured to:
[0143] The business platform acquires business data within a preset period from the business platform and uses the business data within the preset period as the prediction sample, wherein the business platform includes the business model;
[0144] The predicted sample is input into the business model to obtain the sample score distribution for the predicted sample;
[0145] The model quality score of the business model is determined based on the sample score distribution.
[0146] In one possible implementation, the sample acquisition module 302 is further configured to:
[0147] The first and second samples in the predicted samples are determined based on the sample score distribution;
[0148] Based on the model stability calculation formula, the model quality score of the business model is determined according to the first sample and the second sample.
[0149] In one possible implementation, the tag determining module 304 is further configured to:
[0150] The features of the predicted sample are determined, and the problem is classified according to the features of the predicted sample using a preset classification rule to determine the training label of the predicted sample.
[0151] In one possible implementation, the model training module 306 is further configured as follows:
[0152] The predicted sample is input into the business model to obtain the sample score distribution of the business model for the predicted sample;
[0153] The distribution feature extraction network layer extracts features from the sample score distribution to obtain the distribution features;
[0154] The distribution features are input into the first initial quality detection model to obtain the predicted problem labels for the distribution features;
[0155] Based on the predicted question label for the distribution characteristics and the training label, the model parameters of the first initial quality detection model are adjusted to obtain the first target quality detection model.
[0156] In one possible implementation, the quality determination module 308 is further configured to:
[0157] The at least two output results are input into the first target quality detection model to obtain the predicted problem label for the predicted sample;
[0158] The quality data of the business model is determined based on the prediction question labels for the prediction samples.
[0159] In one possible implementation, the quality determination module 308 is further configured to:
[0160] The target dataset is obtained from the business platform, and the target dataset is input into the business model to obtain the at least two output results.
[0161] In one possible implementation, the model training module 306 is further configured as follows:
[0162] The predicted samples are input into the distribution feature extraction network layer to obtain the distribution features;
[0163] The distribution features are input into the second initial quality detection model to obtain the predicted problem label for the predicted sample;
[0164] Based on the training labels for the prediction question labels of the prediction samples, the model parameters of the second initial quality detection model are adjusted to obtain the second target quality detection model.
[0165] In one possible implementation, the model training module 306 is further configured as follows:
[0166] The predicted sample is sent to the second target quality detection model to obtain the predicted problem label for the predicted sample;
[0167] The quality data of the business model is determined based on the prediction question labels for the prediction samples.
[0168] In one possible implementation, the quality determination module 308 is further configured to:
[0169] Obtain training samples of the business model and determine the degree of difference between the training samples and the prediction samples;
[0170] The sample quality data of the predicted sample is determined based on the degree of difference.
[0171] In one possible implementation, the quality determination module 308 is further configured to:
[0172] The target data in the prediction sample is determined, and the sample quality data of the prediction sample is determined according to the proportion of the target data.
[0173] This specification provides a data processing method and apparatus, wherein the data processing apparatus includes: acquiring prediction samples for a business model; determining a model quality score for the business model based on the prediction samples; determining training labels for the prediction samples if the model quality score is greater than a quality score threshold; training a first initial quality detection model based on the prediction samples and the training labels to obtain a first target quality detection model; acquiring at least two output results of the business model for a target dataset, and inputting the at least two output results into the first target quality detection model to determine the quality data of the business model. By acquiring at least two output results of the business model for a target dataset and inputting the at least two output results into the first target quality detection model to determine the quality data of the business model, the quality data of the business model is determined, thereby achieving the goal of scoring the business model, monitoring the quality of the business model, and improving the efficiency of model quality assurance.
[0174] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.
[0175] Figure 4 A structural block diagram of a computing device 400 according to one embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.
[0176] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.
[0177] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0178] Computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). Computing device 400 can also be a mobile or stationary server.
[0179] The processor 420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the aforementioned data processing method. The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the aforementioned data processing method belong to the same concept; details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the aforementioned data processing method.
[0180] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.
[0181] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.
[0182] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.
[0183] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the data processing method described above.
[0184] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0185] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0186] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.
[0187] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0188] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.
Claims
1. A data processing method, comprising: Obtain prediction samples for the business model, and determine the model quality score of the business model based on the prediction samples; If the model quality score is greater than the quality score threshold, determine the training label of the predicted sample; Based on the predicted samples and the training labels of the predicted samples, the first initial quality detection model is trained to obtain the first target quality detection model; Obtain at least two output results of the business model for the target dataset, and input the at least two output results into the first target quality detection model to determine the quality data of the business model.
2. The method according to claim 1, wherein obtaining prediction samples for the business model and determining the model quality score of the business model based on the prediction samples includes: The business platform acquires business data within a preset period from the business platform and uses the business data within the preset period as the prediction sample, wherein the business platform includes the business model; The predicted sample is input into the business model to obtain the sample score distribution for the predicted sample; The model quality score of the business model is determined based on the sample score distribution.
3. The method according to claim 2, wherein determining the model quality score of the business model based on the sample score distribution includes: The first and second samples in the predicted samples are determined based on the sample score distribution; Based on the model stability calculation formula, the model quality score of the business model is determined according to the first sample and the second sample.
4. The method according to claim 1, wherein determining the training label of the predicted sample comprises: The features of the predicted sample are determined, and the problem is classified according to the features of the predicted sample using a preset classification rule to determine the training label of the predicted sample.
5. The method according to claim 1, wherein training the first initial quality detection model based on the predicted sample and the training labels of the predicted sample to obtain the first target quality detection model includes: The predicted sample is input into the business model to obtain the sample score distribution of the business model for the predicted sample; The distribution feature extraction network layer extracts features from the sample score distribution to obtain the distribution features; The distribution features are input into the first initial quality detection model to obtain the predicted problem label for the distribution features; Based on the predicted question label for the distribution characteristics and the training label, the model parameters of the first initial quality detection model are adjusted to obtain the first target quality detection model.
6. The method according to claim 1, wherein inputting the at least two output results into the first target quality detection model to determine the quality data of the business model includes: The at least two output results are input into the first target quality detection model to obtain the predicted problem label for the predicted sample; The quality data of the business model is determined based on the prediction question labels for the prediction samples.
7. The method according to claim 1, wherein obtaining at least two output results of the business model for the target dataset includes: The target dataset is obtained from the business platform, and the target dataset is input into the business model to obtain the at least two output results.
8. The method according to claim 1, further comprising, after obtaining the prediction samples for the business model: The predicted samples are input into the distribution feature extraction network layer to obtain the distribution features; The distribution features are input into the second initial quality detection model to obtain the predicted problem label for the predicted sample; Based on the training labels for the prediction question labels of the prediction samples, the model parameters of the second initial quality detection model are adjusted to obtain the second target quality detection model.
9. The method according to claim 8, further comprising, after obtaining the second target quality detection model: The predicted sample is sent to the second target quality detection model to obtain the predicted problem label for the predicted sample; The quality data of the business model is determined based on the prediction question labels for the prediction samples.
10. The method according to claim 1, further comprising, after obtaining the prediction samples for the business model: Obtain training samples of the business model and determine the degree of difference between the training samples and the prediction samples; The sample quality data of the predicted sample is determined based on the degree of difference.
11. The method according to claim 1, further comprising, after obtaining the prediction samples for the business model: The target data in the prediction sample is determined, and the sample quality data of the prediction sample is determined according to the proportion of the target data.
12. A data processing apparatus, comprising: The sample acquisition module is configured to acquire prediction samples for the business model and determine the model quality score of the business model based on the prediction samples. The label determination module is configured to determine the training label of the predicted sample when the model quality score is greater than a quality score threshold. The model training module is configured to train the first initial quality detection model based on the predicted sample and the training label of the predicted sample to obtain the first target quality detection model. The quality determination module is configured to acquire at least two output results of the business model for the target dataset, and input the at least two output results into the first target quality detection model to determine the quality data of the business model.
13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 11.
14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.