Data cleaning method, data cleaning device, equipment and storage medium

Abstract

The embodiment of the invention provides a data cleaning method, a data cleaning device, equipment and a storage medium, and relates to the field of artificial intelligence. The method comprises the steps that a training data set and model training parameters corresponding to a first data classification model are acquired, the data stability degree and the data information amount of the training data set are determined according to the training data set and the model training parameters, the training data set is classified, and a first type of training data and a second type of training data are obtained; training a first data classification model according to the training data to obtain a second data classification model, and classifying the target training data set to obtain fourth class training data and fifth class training data; and if the quality verification of the fifth type of training data fails, retraining the second data classification model according to the fourth type of training data and the fifth type of training data until the quality verification of the fifth type of training data succeeds. The invention aims to solve the problem that an existing machine learning model is poor in effect in most financial classification tasks.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a data cleaning method, a data cleaning device, a computer device, and a computer-readable storage medium. Background Art

[0002] Currently, in the financial field, data quality is crucial to the prediction accuracy and reliability of machine learning models. Currently, data quality screening methods are mainly used to identify low-quality data in training data, mainly dividing data into three categories: easy-to-predict (Easy) data, ambiguous (Ambiguous) data and hard-to-predict (Hard) data, and retaining the Easy data to improve the performance of machine learning models.

[0003] However, for training data in the financial field, if only Easy data is retained, high-quality data in Ambiguous may be filtered out. For example, in the four-category customer value classification task, data in certain classification categories will be completely lost, making it impossible to guarantee the amount of data for model training. This will result in low prediction accuracy of the trained machine learning model and poor performance in most financial classification tasks. Summary of the invention

[0004] The present application provides a data cleaning method, a data cleaning device, a computer device and a computer-readable storage medium, which aim to solve the problem of low data quality and small data volume of training data in the financial field, and further solve the problem of poor effect of learning models in most financial classification tasks.

[0005] To achieve the above objectives, the present application provides a data cleaning method, the method comprising:

[0006] Acquire a training data set and model training parameters corresponding to the first data classification model, and determine the data stability and data information volume of the training data set according to the training data set and the model training parameters;

[0007] Based on the first data classification model, classifying the training data set according to the data stability and data information amount of the training data set to obtain first-category training data, second-category training data, and third-category training data, wherein the data quality of the third-category training data is lower than that of the second-category training data, and the data quality of the second-category training data is lower than that of the first-category training data;

[0008] Generate a target training data set according to the first type of training data and the second type of training data, and train the first data classification model based on the target training data set to obtain a second data classification model;

[0009] Based on the second data classification model, classifying the target training data set according to the data stability and data information amount of the target training data set to obtain fourth category training data and fifth category training data, wherein the data quality of the fifth category training data is lower than that of the fourth category training data;

[0010] If the quality verification of the fifth category training data fails, the second data classification model is retrained according to the fourth category training data and the fifth category training data until the quality verification of the fifth category training data classified according to the retrained second data classification model succeeds.

[0011] To achieve the above object, the present application provides a data cleaning device, the data cleaning device comprising:

[0012] A data analysis module, used to obtain a training data set and model training parameters corresponding to the first data classification model, and determine the data stability and data information volume of the training data set according to the training data set and the model training parameters;

[0013] a first data classification module, configured to classify the training data set based on the first data classification model and according to the data stability and data information volume of the training data set to obtain first-category training data, second-category training data, and third-category training data, wherein the data quality of the third-category training data is lower than that of the second-category training data, and the data quality of the second-category training data is lower than that of the first-category training data;

[0014] A model training module, used to generate a target training data set according to the first type of training data and the second type of training data, and train the first data classification model based on the target training data set to obtain a second data classification model;

[0015] a second data classification module, used in the second data classification model, classifying the target training data set according to the data stability and data information amount of the target training data set to obtain fourth-category training data and fifth-category training data, wherein the data quality of the fifth-category training data is lower than that of the fourth-category training data;

[0016] A data iteration module is used to retrain the second data classification model according to the fourth category of training data and the fifth category of training data if the quality verification of the fifth category of training data fails, until the quality verification of the fifth category of training data classified according to the retrained second data classification model succeeds.

[0017] To achieve the above-mentioned purpose, the present application provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the data cleaning method described above when executing the computer program.

[0018] To achieve the above objectives, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the data cleaning method as described above.

[0019] The data cleaning method, data cleaning device, computer equipment and computer-readable storage medium disclosed in the embodiments of the present application obtain a training data set and model training parameters corresponding to a first data classification model, and determine the data stability and data information volume of the training data set according to the training data set and the model training parameters; based on the first data classification model, classify the training data set according to the data stability and data information volume of the training data set to obtain first-category training data, second-category training data and third-category training data; generate a target training data set according to the first-category training data and the second-category training data, and train the first data classification model based on the target training data set to obtain a second data classification model; based on the second data classification model, classify the target training data set according to the data stability and data information volume of the target training data set to obtain fourth-category training data and fifth-category training data; if the quality verification of the fifth-category training data fails, retrain the second data classification model according to the fourth-category training data and the fifth-category training data until the quality verification of the fifth-category training data obtained by classification according to the retrained second data classification model succeeds. This can increase the training data used for model training, and can dynamically and flexibly adjust the classification threshold. It can also effectively improve data quality and ensure data quantity, improve the generalization ability and prediction accuracy of the model, and improve the applicability and effectiveness of the model in various financial tasks, providing more reliable support for financial services such as insurance claims processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a scenario diagram of a data cleaning method provided in an embodiment of the present application;

[0022] Figure 2It is a flow chart of a data cleaning method provided in an embodiment of the present application;

[0023] Figure 3 is a schematic block diagram of a data cleaning device provided in one embodiment of the present application;

[0024] Figure 4 It is a schematic block diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0027] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] In the financial field, for example, in customer churn and loan approval tasks, if the existing data cleaning methods are used to clean and train the training data in the financial field, the following disadvantages will occur:

[0029] 1. There may be good data in the Hard data that performs poorly in the initial training cycle but performs steadily in the subsequent training cycles. However, these good data will be classified as Hard data in the existing data cleaning methods.

[0030] 2. If only Hard data is discarded and Easy and Ambiguous data are retained for model training, although a large amount of data loss can be avoided, a large amount of noise in the Ambiguous data is retained, which affects the data quality and fails to achieve the best effect.

[0031] 3. If only the Easy class data is retained, some good data existing in the Ambiguous class data will be lost, especially in the four-category task, some classification category data will be completely lost.

[0032] To address the above-mentioned defects, the present application proposes a data cleaning method, a data cleaning device, a computer device and a computer-readable storage medium, thereby increasing the training data used for model training, and being able to dynamically and flexibly adjust the classification threshold. It can also effectively improve the data quality and ensure the data quantity, improve the generalization ability and prediction accuracy of the model, and improve the applicability and effectiveness of the model in various financial tasks, providing more reliable support for financial services such as insurance claims processes.

[0033] The method can be applied to a server, and of course can also be applied to a terminal device, wherein the terminal device may include a fixed terminal such as a mobile phone, a tablet computer, a personal digital assistant (PDA), etc. The server may be, for example, a separate server or a server cluster. However, for ease of understanding, the following embodiment will be described in detail with a data cleaning method applied to a server.

[0034] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0035] like Figure 1 As shown, the data cleaning method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the application environment includes a terminal device 110 and a server 120, wherein the terminal device 110 can communicate with the server 120 through a network. Specifically, the server 120 obtains a training data set and model training parameters corresponding to the first data classification model, and determines the data stability and data information volume of the training data set according to the training data set and the model training parameters; based on the first data classification model, the training data set is classified according to the data stability and data information volume of the training data set to obtain first-category training data, second-category training data, and third-category training data; a target training data set is generated according to the first-category training data and the second-category training data, and the first data classification model is trained based on the target training data set to obtain a second data classification model; based on the second data classification model, the target training data set is classified according to the data stability and data information volume of the target training data set to obtain fourth-category training data and fifth-category training data; if the quality verification of the fifth-category training data fails, the second data classification model is retrained according to the fourth-category training data and the fifth-category training data until the quality verification of the fifth-category training data classified according to the retrained second data classification model succeeds, and the fifth-category training data is sent to the terminal device 110. Among them, the server 120 can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 110 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0036] See also Figure 2 , Figure 2 This is a schematic flow chart of a data cleaning method provided in an embodiment of the present application. The data cleaning method can increase the training data used for model training, and can dynamically and flexibly adjust the classification threshold, and can also effectively improve the data quality and ensure the data quantity, improve the generalization ability and prediction accuracy of the model, and improve the applicability and effect of the model in various financial tasks, providing more reliable support for financial services such as insurance claims process.

[0037] The data cleaning method can be applied to claims prediction models, customer risk assessment models, customer churn prediction models, fraud detection models, and credit scoring models, without specific limitation here.

[0038] like Figure 2 As shown, the data cleaning method includes steps S101 to S105.

[0039] S101. Obtain a training data set and model training parameters corresponding to a first data classification model, and determine the data stability and data information volume of the training data set according to the training data set and the model training parameters.

[0040] The first data classification model is a data classification model trained based on the training data set. The model training parameters may include the total number of epochs set by the first data classification model. The data stability of the training data set is used to represent the average variance of the data of the training data set in subsequent epochs, reflecting the stability of the data. The data information volume of the training data set is used to represent the average information entropy of the data of the training data set in subsequent stable epochs, reflecting the information volume of the data.

[0041] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0042] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0043] Specifically, before obtaining the training data set and model training parameters corresponding to the first data classification model, a preset data classification model and the training data set are first obtained, and the preset data classification model is trained based on the training data set to generate the first data classification model.

[0044] Among them, the preset data classification model can be an untrained data classification model, specifically a claims prediction model, a customer risk assessment model, a customer churn prediction model, a fraud detection model and a credit scoring model. Exemplarily, the preset data classification model can be a transformer table model.

[0045] In some embodiments, the prediction probability variance and average information entropy of each training data in the training data set are determined based on the training data set and the model training parameters; the data stability of the training data set is determined based on the prediction probability variance of each training data in the training data set; and the data information volume of the training data set is determined based on the average information entropy of each training data in the training data set. In this way, the data stability and data information volume of the training data set can be accurately determined to ensure that potential high-quality data is not discarded, thereby improving the generalization ability and prediction performance of the model.

[0046] Among them, the prediction probability variance is used to represent the variance of the prediction probability of each training data at the e-th epoch, and the average information entropy is used to represent the average amount of information of each training data.

[0047] For example, if the prediction probability variance of each training data in the training data set is smaller, it means that the prediction changes in each round are smaller, which means that the data stability is higher; if the prediction probability variance of each training data in the training data set is larger, it means that the prediction changes in each round are larger, which means that the data stability is lower.

[0048] For example, if the average information entropy of each training data in the training data set is smaller, it means that its prediction accuracy is high and the data quality is higher.

[0049] In some embodiments, the prediction probability of each training data in the training data set is obtained; according to the prediction probability of each training data in the training data set, the average prediction probability and the average information entropy of the training data set are determined; according to the model training parameters, the prediction probability of each training data in the training data set and the average prediction probability, the prediction probability variance of the training data set is determined. In this way, the average information entropy and the prediction probability variance can be accurately determined to ensure that potential high-quality data is not discarded, thereby improving the generalization ability and prediction performance of the model.

[0050] Among them, the prediction probability of each training data in the training data set is the prediction probability of the training data at the e-th epoch, the average prediction probability is the average prediction probability of the training data in the stable prediction training cycle starting from the subsequent i-th epoch, and the model training parameters may include the total number of epochs set by the first data classification model.

[0051] Specifically, the prediction probability of each training data in the training data set can be calculated first, and then the average prediction probability and the average information entropy can be calculated through the prediction probability of each training data. Finally, the prediction probability variance of the training data set can be calculated through the model training parameters, the prediction probability of each training data in the training data set and the average prediction probability.

[0052] For example, the predicted probability variance of the training data set can be expressed as:

[0053]

[0054] Among them, v ep (x) is the predicted probability variance of the training data set, E is the total number of epochs set for the first data classification model, and P e (x) is the predicted probability of the training data at the e-th epoch, It is the average prediction probability of the training data in the subsequent epoch i to start the stable prediction training cycle.

[0055] For example, the average information entropy of the training data set can be expressed as:

[0056] v al =(1-P e (x))*P e (x)

[0057] Among them, v al is the average information entropy of the training data set, P e (x) is the predicted probability of the training data at the e-th epoch.

[0058] S102. Based on the first data classification model, classify the training data set according to the data stability and the amount of data information of the training data set to obtain first-category training data, second-category training data, and third-category training data, wherein the data quality of the third-category training data is lower than that of the second-category training data, and the data quality of the second-category training data is lower than that of the first-category training data.

[0059] Among them, the data quality of the third category of training data is lower than that of the second category of training data, and the data quality of the second category of training data is lower than that of the first category of training data. The first category of training data can be Easy data, that is, data that is easy to predict, the second category of training data can be Ambiguous data, that is, unclear data, and the third category of training data can be Hard data, that is, data that is difficult to predict.

[0060] Although the embodiment of the present application also classifies the training data set based on the data classification model to obtain the first category of training data, the second category of training data, and the third category of training data, the embodiment of the present application takes into account the good data in the second category of training data that performs poorly in the initial training cycle but performs steadily in the subsequent training cycles that tend to stabilize, and redefines the data classification method to ensure that potential high-quality data is not discarded, thereby improving the generalization ability and prediction performance of the model.

[0061] In some embodiments, for any training data in the training data set, the prediction probability variance and average information entropy corresponding to the training data are obtained; if the prediction probability variance corresponding to the training data is less than the preset variance threshold, and the average information entropy corresponding to the training data is less than the preset information entropy threshold, the training data is determined to be the first type of training data; if the prediction probability variance corresponding to the training data is greater than the preset variance threshold, and the average information entropy corresponding to the training data is greater than the preset information entropy threshold, the training data is determined to be the third type of training data; if it is determined that the training data is neither the first type of training data nor the third type of training data, the training data is determined to be the second type of training data. In this way, the training data can be accurately classified to ensure that potential high-quality data is not discarded, and the generalization ability and prediction performance of the model are improved.

[0062] The preset variance threshold may be any variance value, specifically the median of the variance values ​​of each training data in the training data set, or the average value, which is not specifically limited here. The preset information entropy threshold may be any information entropy value, specifically the information entropy value of the training data corresponding to the 80th position in the training data set according to the information entropy value, or the average value, which is not specifically limited here.

[0063] Exemplarily, for any training data in the training data set, the prediction probability variance and average information entropy corresponding to the training data are obtained; if the prediction probability variance corresponding to the training data is less than the preset variance threshold, it means that the prediction results of these data do not change much during the training process; if the average information entropy corresponding to the training data is less than the preset information entropy threshold, it means that these data are relatively simple and the model is relatively easy to predict. At this time, it can be determined that the training data is the first type of training data, namely, Easy type data.

[0064] Exemplarily, for any training data in the training data set, the prediction probability variance and average information entropy corresponding to the training data are obtained; if the prediction probability variance corresponding to the training data is greater than the preset variance threshold, it means that the prediction results of these data vary greatly during the training process; if the average information entropy corresponding to the training data is greater than the preset information entropy threshold, it means that these data are relatively complex and difficult to predict by the model. At this time, it can be determined that the training data is the third type of training data, namely, Hard data;

[0065] Exemplarily, for any training data in the training data set, the prediction probability variance and average information entropy corresponding to the training data are obtained; if it is determined that the training data is not the first type of training data or the third type of training data, then the training data is determined to be the second type of training data, i.e., Ambiguous type data.

[0066] Therefore, all the training data in the training data set can be classified to obtain the first category of training data, the second category of training data and the third category of training data.

[0067] S103: Generate a target training data set according to the first type of training data and the second type of training data, and train a first data classification model based on the target training data set to obtain a second data classification model.

[0068] The target training data set is a training data set generated by optimizing the first type of training data and the second type of training data. The second data classification model is a data classification model obtained by training the first data classification model based on the target training data set.

[0069] The present application trains a first data classification model through a target training data set to obtain a second data classification model. The second type of training data, namely Ambiguous data, is retained in the target training data set, thereby retaining some good data samples, and iteratively generates and screens on the basis of Easy data and Ambiguous data to generate target training data, thereby achieving a balance between data quality and quantity, avoiding the data loss or noise problem caused by simply relying on Easy data or discarding Hard data in traditional data cleaning methods, thereby improving the applicability and effectiveness of the model in various financial tasks.

[0070] In some embodiments, a preset adversarial network is trained based on the first type of training data and the second type of training data to generate a target adversarial network; optimized training data is generated based on the target adversarial network; and the optimized training data, the first type of training data, and the second type of training data are mixed to generate a target training data set. Thus, the target training data set can be accurately generated from the first type of training data and the second type of training data.

[0071] The optimized training data may be optimized training data based on the first type of training data and the second type of training data.

[0072] Specifically, a preset adversarial network is trained based on the first type of training data and the second type of training data to generate a target adversarial network, and optimized training data are iteratively generated and screened through the target adversarial network. The optimized training data, the first type of training data, and the second type of training data are mixed and screened to finally generate a target training data set.

[0073] S104. Based on the second data classification model, classify the target training data set according to the data stability and data information volume of the target training data set to obtain fourth category training data and fifth category training data, wherein the data quality of the fifth category training data is lower than that of the fourth category training data.

[0074] The data quality of the fifth type of training data is lower than that of the fourth type of training data. The fourth type of training data may be the reclassified Easy type data, and the fifth type of training data may be the reclassified Ambiguous type data.

[0075] Specifically, the prediction probability variance and average information entropy of each training data in the second data classification model can be determined according to the target training data set and the model training parameters of the second data classification model; the data stability of the second data classification model can be determined according to the prediction probability variance of each training data in the second data classification model; and the data information amount of the target training data set can be determined according to the average information entropy of each training data in the target training data set.

[0076] It should be noted that the method for generating the data stability and data information volume of the target training data set is similar to the method for generating the data stability and data information volume of the training data set. For details, please refer to the above embodiment, and no repetitive elaboration will be given here.

[0077] In some embodiments, for any training data in the target training data set, the prediction probability variance and average information entropy corresponding to the training data are obtained; if the prediction probability variance corresponding to the training data is less than the preset variance threshold, and the average information entropy corresponding to the training data is less than the preset information entropy threshold, the training data is determined to be the fourth type of training data; if the prediction probability variance corresponding to the training data is greater than the preset variance threshold, and the average information entropy corresponding to the training data is greater than the preset information entropy threshold, the training data is determined to be the sixth type of training data; if it is determined that the training data is not the fourth type of training data and the sixth type of training data, the training data is determined to be the fifth type of training data. In this way, the training data can be accurately classified to ensure that potential high-quality data is not discarded, and the generalization ability and prediction performance of the model are improved.

[0078] Among them, the preset variance threshold can be any variance value, specifically the median of the variance values ​​of each training data in the training data set, or the average value, which is not specifically limited here. The preset information entropy threshold can be any information entropy value, specifically the information entropy value of the training data corresponding to the 80th position in the training data set according to the information entropy value, or the average value, which is not specifically limited here. The sixth category of training data can be the reclassified Hard category data.

[0079] Exemplarily, for any training data in the target training data set, the prediction probability variance and average information entropy corresponding to the training data are obtained; if the prediction probability variance corresponding to the training data is less than the preset variance threshold, it means that the prediction results of these data do not change much during the training process; if the average information entropy corresponding to the training data is less than the preset information entropy threshold, it means that these data are relatively simple and the model is relatively easy to predict. At this time, it can be determined that the training data is the fourth type of training data, namely, Easy type data.

[0080] Exemplarily, for any training data in the target training data set, the prediction probability variance and average information entropy corresponding to the training data are obtained; if the prediction probability variance corresponding to the training data is greater than the preset variance threshold, it means that the prediction results of these data vary greatly during the training process; if the average information entropy corresponding to the training data is greater than the preset information entropy threshold, it means that these data are relatively complex and difficult to predict by the model. At this time, it can be determined that the training data is the sixth type of training data, namely, Hard data;

[0081] Exemplarily, for any training data in the target training data set, the prediction probability variance and the average information entropy corresponding to the training data are obtained; if it is determined that the training data is not the fourth type of training data or the sixth type of training data, then the training data is determined to be the fifth type of training data, i.e., Ambiguous type data.

[0082] Therefore, all the training data in the target training data set can be classified to obtain the fourth category of training data, the fifth category of training data, and the sixth category of training data.

[0083] In some embodiments, based on the second data classification model, the target training data set is classified according to the data stability and data information volume of the target training data set to obtain the fourth category of training data and the fifth category of training data, and then the fifth category of training data is classified and processed to obtain data of different categories; it is determined whether the cosine similarity between the data of each category is greater than the preset similarity threshold; if the cosine similarity between the data of each category is greater than the preset similarity threshold, it is determined that the quality verification of the fifth category of training data is successful; if there are two categories of data whose cosine similarity is not greater than the preset similarity threshold, it is determined that the quality verification of the fifth category of training data has failed. In this way, the quality of the fifth category of training data can be accurately verified, so that the target training data set can be dynamically adjusted according to the performance of the model during the training process, ensuring that the data can be optimally classified and processed at different training stages, and flexibly responding to changes and diversity of data.

[0084] The different types of data may be different types of data in the Ambiguous data class. The preset similarity threshold may be any cosine similarity, which is not specifically limited here.

[0085] Specifically, determine whether the cosine similarity between the data of each category is greater than the preset similarity threshold; if the cosine similarity between the data of each category is greater than the preset similarity threshold, then determine that the quality verification of the fifth category training data is successful; if there are two categories of data whose cosine similarity is not greater than the preset similarity threshold, then determine that the quality verification of the fifth category training data has failed.

[0086] Exemplarily, if the preset similarity threshold is 80%, and if the cosine similarities between the data of each category are 90%, 85%, 95% and 88% respectively, it can be determined that the cosine similarities between the data of each category are greater than the preset similarity threshold, then the quality verification of the fifth category training data is determined to be successful.

[0087] Exemplarily, if the preset similarity threshold is 80%, and if the cosine similarities between the data of each category are 70%, 85%, 95% and 88% respectively, it can be determined that the cosine similarity between the data of two categories is not greater than the preset similarity threshold, then it is determined that the quality verification of the fifth category of training data has failed.

[0088] S105: If the quality verification of the fifth type of training data fails, retrain the second data classification model according to the fourth type of training data and the fifth type of training data, until the quality verification of the fifth type of training data classified according to the retrained second data classification model succeeds.

[0089] Specifically, the target adversarial network is retrained based on the fourth category of training data and the fifth category of training data to update the target adversarial network; the optimized training data is regenerated based on the updated target adversarial network; the optimized training data, the fourth category of training data and the fifth category of training data are mixed to regenerate the target training data set, and the second data classification model is trained based on the regenerated target training data set to obtain an updated second data classification model; finally, based on the updated second data classification model, the regenerated target training data set is classified according to the data stability and data information amount of the regenerated target training data set to obtain the fourth category of training data and the fifth category of training data; finally, the quality of the regenerated fifth category training data is checked until the quality of the regenerated fifth category training data is successfully checked.

[0090] The data cleaning method disclosed in the embodiment of the present application obtains a training data set and model training parameters corresponding to a first data classification model, and determines the data stability and data information volume of the training data set according to the training data set and the model training parameters; based on the first data classification model, the training data set is classified according to the data stability and data information volume of the training data set to obtain first-category training data, second-category training data, and third-category training data; a target training data set is generated according to the first-category training data and the second-category training data, and a first data classification model is trained based on the target training data set to obtain a second data classification model; based on the second data classification model, the target training data set is classified according to the data stability and data information volume of the target training data set to obtain fourth-category training data and fifth-category training data; if the quality verification of the fifth-category training data fails, the second data classification model is retrained according to the fourth-category training data and the fifth-category training data until the quality verification of the fifth-category training data classified according to the retrained second data classification model succeeds. This can increase the training data used for model training, and can dynamically and flexibly adjust the classification threshold. It can also effectively improve data quality and ensure data quantity, improve the generalization ability and prediction accuracy of the model, and improve the applicability and effectiveness of the model in various financial tasks, providing more reliable support for financial services such as insurance claims processes.

[0091] See also Figure 3 , Figure 3 It is a schematic block diagram of a data cleaning device provided in an embodiment of the present application. The data cleaning device can be configured in a server to execute the aforementioned data cleaning method.

[0092] like Figure 3 As shown, the data cleaning device 200 includes: a data analysis module 201, a first data classification module 202, a model training module 203, a second data classification module 204 and a data iteration module 205.

[0093] The data analysis module 201 is used to obtain a training data set and model training parameters corresponding to the first data classification model, and determine the data stability and data information volume of the training data set according to the training data set and the model training parameters;

[0094] A first data classification module 202 is used to classify the training data set based on the first data classification model and according to the data stability and data information amount of the training data set to obtain first-category training data, second-category training data and third-category training data, wherein the data quality of the third-category training data is lower than that of the second-category training data, and the data quality of the second-category training data is lower than that of the first-category training data;

[0095] A model training module 203 is used to generate a target training data set according to the first type of training data and the second type of training data, and train the first data classification model based on the target training data set to obtain a second data classification model;

[0096] A second data classification module 204, used in the second data classification model, classifies the target training data set according to the data stability and data information volume of the target training data set to obtain fourth-category training data and fifth-category training data, wherein the data quality of the fifth-category training data is lower than that of the fourth-category training data;

[0097] The data iteration module 205 is used to retrain the second data classification model according to the fourth category training data and the fifth category training data if the quality verification of the fifth category training data fails, until the quality verification of the fifth category training data classified according to the retrained second data classification model succeeds.

[0098] In some embodiments, the data analysis module 201 is also used to determine the prediction probability variance and the average information entropy of each training data in the training data set based on the training data set and the model training parameters; determine the data stability of the training data set based on the prediction probability variance of each training data in the training data set; and determine the data information amount of the training data set based on the average information entropy of each training data in the training data set.

[0099] In some embodiments, the data analysis module 201 is also used to obtain the prediction probability of each training data in the training data set; determine the average prediction probability and average information entropy of the training data set based on the prediction probability of each training data in the training data set; determine the prediction probability variance of the training data set based on the model training parameters, the prediction probability of each training data in the training data set and the average prediction probability.

[0100] In some embodiments, the first data classification module 202 is also used to obtain the predicted probability variance and average information entropy corresponding to any one training data in the training data set; if the predicted probability variance corresponding to the training data is less than a preset variance threshold, and the average information entropy corresponding to the training data is less than the preset information entropy threshold, then the training data is determined to be first-category training data; if the predicted probability variance corresponding to the training data is greater than the preset variance threshold, and the average information entropy corresponding to the training data is greater than the preset information entropy threshold, then the training data is determined to be third-category training data; if it is determined that the training data is neither the first-category training data nor the third-category training data, then the training data is determined to be second-category training data.

[0101] In some embodiments, the model training module 203 is also used to train a preset adversarial network based on the first category of training data and the second category of training data to generate a target adversarial network; generate optimized training data based on the target adversarial network; and mix the optimized training data, the first category of training data, and the second category of training data to generate a target training data set.

[0102] In some embodiments, the second data classification module 204 is also used to obtain the predicted probability variance and average information entropy corresponding to any one training data in the target training data set; if the predicted probability variance corresponding to the training data is less than a preset variance threshold, and the average information entropy corresponding to the training data is less than the preset information entropy threshold, then the training data is determined to be fourth-category training data; if the predicted probability variance corresponding to the training data is greater than the preset variance threshold, and the average information entropy corresponding to the training data is greater than the preset information entropy threshold, then the training data is determined to be sixth-category training data; if it is determined that the training data is neither the fourth-category training data nor the sixth-category training data, then the training data is determined to be fifth-category training data.

[0103] In some embodiments, the data cleaning device 200 also includes a data quality verification module 206, which is used to classify the fifth category of training data to obtain data of different categories; determine whether the cosine similarity between the data of each category is greater than a preset similarity threshold; if the cosine similarity between the data of each category is greater than the preset similarity threshold, it is determined that the quality verification of the fifth category of training data is successful; if there are two categories of data whose cosine similarity is not greater than the preset similarity threshold, it is determined that the quality verification of the fifth category of training data has failed.

[0104] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0106] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Figure 4 Runs on the computer device shown.

[0107] See also Figure 4 , Figure 4 1 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0108] like Figure 4 As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a volatile storage medium, a non-volatile storage medium and an internal memory.

[0109] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any data cleaning method.

[0110] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0111] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any data cleaning method.

[0112] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure of the computer device is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0113] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0114] Wherein, in some embodiments, the processor is used to run a computer program stored in the memory to implement the following steps: obtaining a training data set and a model training parameter corresponding to a first data classification model, and determining the data stability and data information volume of the training data set according to the training data set and the model training parameter; based on the first data classification model, classifying the training data set according to the data stability and data information volume of the training data set to obtain first-category training data, second-category training data, and third-category training data, wherein the data quality of the third-category training data is lower than that of the second-category training data, and the data quality of the second-category training data is lower than that of the first-category training data; according to the first-category training data The method comprises the following steps: generating a target training data set based on the first data classification model and the second-category training data, and training the first data classification model based on the target training data set to obtain a second data classification model; based on the second data classification model, classifying the target training data set according to the data stability and data information amount of the target training data set to obtain fourth-category training data and fifth-category training data, wherein the data quality of the fifth-category training data is lower than that of the fourth-category training data; if the quality verification of the fifth-category training data fails, retraining the second data classification model based on the fourth-category training data and the fifth-category training data until the quality verification of the fifth-category training data obtained by classification according to the retrained second data classification model succeeds.

[0115] In some embodiments, the processor is further used to determine the predicted probability variance and the average information entropy of each training data in the training data set based on the training data set and the model training parameters; determine the data stability of the training data set based on the predicted probability variance of each training data in the training data set; and determine the data information amount of the training data set based on the average information entropy of each training data in the training data set.

[0116] In some embodiments, the processor is also used to obtain the predicted probability of each training data in the training data set; determine the average predicted probability and the average information entropy of the training data set based on the predicted probability of each training data in the training data set; determine the predicted probability variance of the training data set based on the model training parameters, the predicted probability of each training data in the training data set and the average predicted probability.

[0117] In some embodiments, the processor is also used to obtain the predicted probability variance and average information entropy corresponding to any one training data in the training data set; if the predicted probability variance corresponding to the training data is less than a preset variance threshold, and the average information entropy corresponding to the training data is less than the preset information entropy threshold, then the training data is determined to be first-category training data; if the predicted probability variance corresponding to the training data is greater than the preset variance threshold, and the average information entropy corresponding to the training data is greater than the preset information entropy threshold, then the training data is determined to be third-category training data; if it is determined that the training data is neither the first-category training data nor the third-category training data, then the training data is determined to be second-category training data.

[0118] In some embodiments, the processor is further used to train a preset adversarial network based on the first category of training data and the second category of training data to generate a target adversarial network; generate optimized training data based on the target adversarial network; and mix the optimized training data, the first category of training data, and the second category of training data to generate a target training data set.

[0119] In some embodiments, the processor is also used to obtain the predicted probability variance and average information entropy corresponding to any one training data in the target training data set; if the predicted probability variance corresponding to the training data is less than a preset variance threshold, and the average information entropy corresponding to the training data is less than the preset information entropy threshold, then the training data is determined to be fourth-category training data; if the predicted probability variance corresponding to the training data is greater than the preset variance threshold, and the average information entropy corresponding to the training data is greater than the preset information entropy threshold, then the training data is determined to be sixth-category training data; if it is determined that the training data is not the fourth-category training data or the sixth-category training data, then the training data is determined to be fifth-category training data.

[0120] In some embodiments, the processor is further used to classify the fifth category of training data to obtain data of different categories; determine whether the cosine similarity between the data of each category is greater than a preset similarity threshold; if the cosine similarity between the data of each category is greater than the preset similarity threshold, then determine that the quality verification of the fifth category of training data is successful; if there are two categories of data whose cosine similarity is not greater than the preset similarity threshold, then determine that the quality verification of the fifth category of training data has failed.

[0121] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any one of the data cleaning methods provided in the embodiments of the present application is implemented.

[0122] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0123] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0124] The present invention refers to a new application model of computer technologies such as storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. of the blockchain language model. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.

[0125] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data cleaning method, characterized in that: The method comprises: Acquire a training data set and model training parameters corresponding to the first data classification model, and determine the data stability and data information volume of the training data set according to the training data set and the model training parameters; Based on the first data classification model, classifying the training data set according to the data stability and data information amount of the training data set to obtain first-category training data, second-category training data, and third-category training data, wherein the data quality of the third-category training data is lower than that of the second-category training data, and the data quality of the second-category training data is lower than that of the first-category training data; Generate a target training data set according to the first type of training data and the second type of training data, and train the first data classification model based on the target training data set to obtain a second data classification model; Based on the second data classification model, classifying the target training data set according to the data stability and data information amount of the target training data set to obtain fourth category training data and fifth category training data, wherein the data quality of the fifth category training data is lower than that of the fourth category training data; If the quality verification of the fifth category training data fails, the second data classification model is retrained according to the fourth category training data and the fifth category training data until the quality verification of the fifth category training data classified according to the retrained second data classification model succeeds.

2. The method according to claim 1, characterized in that: The determining the data stability and data information amount of the training data set according to the training data set and the model training parameters includes: Determine the prediction probability variance and average information entropy of each training data in the training data set according to the training data set and the model training parameters; Determining the data stability of the training data set according to the prediction probability variance of each training data in the training data set; The data information amount of the training data set is determined according to the average information entropy of each training data in the training data set.

3. The method according to claim 2, characterized in that The step of determining the prediction probability variance and the average information entropy of each training data in the training data set according to the training data set and the model training parameters includes: Obtaining the predicted probability of each training data in the training data set; Determining an average prediction probability and an average information entropy of the training data set according to the prediction probability of each training data in the training data set; The prediction probability variance of the training data set is determined according to the model training parameters, the prediction probability of each training data in the training data set and the average prediction probability.

4. The method according to claim 1, characterized in that The method of classifying the training data set based on the first data classification model according to the data stability and data information volume of the training data set to obtain first-category training data, second-category training data, and third-category training data includes: For any training data in the training data set, obtaining the prediction probability variance and average information entropy corresponding to the training data; If the prediction probability variance corresponding to the training data is less than a preset variance threshold, and the average information entropy corresponding to the training data is less than a preset information entropy threshold, then the training data is determined to be first-category training data; If the prediction probability variance corresponding to the training data is greater than the preset variance threshold, and the average information entropy corresponding to the training data is greater than the preset information entropy threshold, then the training data is determined to be the third type of training data; If it is determined that the training data is not the first type of training data and the third type of training data, then the training data is determined to be the second type of training data.

5. The method according to claim 1, characterized in that The generating a target training data set according to the first type of training data and the second type of training data comprises: Training a preset adversarial network based on the first type of training data and the second type of training data to generate a target adversarial network; Generate optimized training data based on the target adversarial network; The optimized training data, the first type of training data and the second type of training data are mixed to generate a target training data set.

6. The method according to claim 1, characterized in that The method of classifying the target training data set based on the second data classification model according to the data stability and data information volume of the target training data set to obtain the fourth category of training data and the fifth category of training data includes: For any training data in the target training data set, obtaining the prediction probability variance and average information entropy corresponding to the training data; If the prediction probability variance corresponding to the training data is less than a preset variance threshold, and the average information entropy corresponding to the training data is less than a preset information entropy threshold, then the training data is determined to be fourth-category training data; If the prediction probability variance corresponding to the training data is greater than the preset variance threshold, and the average information entropy corresponding to the training data is greater than the preset information entropy threshold, then the training data is determined to be the sixth type of training data; If it is determined that the training data is not the fourth type of training data or the sixth type of training data, then the training data is determined to be the fifth type of training data.

7. The method according to claim 1, characterized in that After classifying the target training data set based on the second data classification model according to the data stability and data information volume of the target training data set to obtain the fourth category of training data and the fifth category of training data, the method further includes: Classifying the fifth category of training data to obtain data of different categories; Determine whether the cosine similarity between the data of each category is greater than a preset similarity threshold; If the cosine similarity between the data of each category is greater than the preset similarity threshold, it is determined that the quality verification of the fifth category training data is successful; If the cosine similarity between the data of two categories is not greater than the preset similarity threshold, it is determined that the quality check of the fifth category training data has failed.

8. A data cleaning device, characterized in that: The data cleaning device comprises: A data analysis module, used to obtain a training data set and model training parameters corresponding to the first data classification model, and determine the data stability and data information volume of the training data set according to the training data set and the model training parameters; a first data classification module, configured to classify the training data set based on the first data classification model and according to the data stability and data information volume of the training data set to obtain first-category training data, second-category training data, and third-category training data, wherein the data quality of the third-category training data is lower than that of the second-category training data, and the data quality of the second-category training data is lower than that of the first-category training data; A model training module, used to generate a target training data set according to the first type of training data and the second type of training data, and train the first data classification model based on the target training data set to obtain a second data classification model; a second data classification module, used in the second data classification model, classifying the target training data set according to the data stability and data information amount of the target training data set to obtain fourth-category training data and fifth-category training data, wherein the data quality of the fifth-category training data is lower than that of the fourth-category training data; A data iteration module is used to retrain the second data classification model according to the fourth category of training data and the fifth category of training data if the quality verification of the fifth category of training data fails, until the quality verification of the fifth category of training data classified according to the retrained second data classification model succeeds.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement, when executing the computer program: The data cleaning method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the data cleaning method according to any one of claims 1 to 7.

Images