Downstream analysis feedback-oriented data cleaning method and system and electronic equipment

Abstract

The invention discloses a downstream analysis feedback-oriented data cleaning method, which comprises the following steps of: constructing a multi-layer perceptron classifier to execute classification, and generating a soft label; dirty data is combined with a soft label to input an error detection model, a preliminary error detection result in a feature vector form is output, and a clustering result is obtained through clustering weighting; selecting partial data for training, inputting a plurality of isomorphic error detection models after stratified sampling, and voting an integration result; processing error detection output and dirty data by using a neural network repairing model of a cross attention mechanism, and outputting correction data through feature extraction and optimization fusion; and the downstream model feeds back an optimization repair process to obtain final cleaning data. According to the method, data is detected mainly through a method based on deep learning, dirty data in the data are restored, and the data are more accurate and credible. According to the method, customization processing can be carried out on the data according to the requirements of downstream tasks, and it is ensured that the cleaned data can be directly applied to the tasks.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data cleaning method and system and electronic equipment for downstream analysis feedback. Background Art

[0002] With the development of big data technology, the amount of data has exploded, the sources of data have diversified, the complexity and diversity of data have continued to increase, and the quality of data has a direct impact on the accuracy and effectiveness of data analysis and decision-making. High-quality data is the basis of data analysis, but real-world data often has various problems, such as missing values, outliers, duplicate values, inconsistencies, etc. Data with these problems is usually called "dirty data". Dirty data will affect the accuracy and credibility of data analysis, and even lead to wrong conclusions and decisions.

[0003] Existing data cleaning methods often fail to fully consider the specific needs of downstream tasks. This lack of targeted processing may result in the cleaned data still not meeting the precise requirements of specific analysis, thus affecting the quality and credibility of the analysis results. Existing methods usually focus on the general cleaning and standardization of data, while ignoring the applicability and effectiveness of data in specific analysis tasks. In order to improve the pertinence and practicality of data cleaning, it is urgent to research and develop data cleaning models for feedback on downstream analysis tasks. These models should be able to customize the data according to the specific requirements of downstream tasks, ensuring that the cleaned data is not only of high quality and consistency, but also directly meets the needs of downstream analysis tasks. Summary of the invention

[0004] In order to overcome the above technical problems, the embodiment of the present application provides a data cleaning method, system, and electronic device for downstream analysis feedback to solve the dirty data problem. The method mainly detects the data based on a deep learning method and repairs the dirty data therein to make the data more accurate and reliable. The method can customize the data according to the needs of downstream tasks to ensure that the cleaned data can be directly applied to these tasks.

[0005] According to a first aspect of an embodiment of the present application, a data cleaning method for downstream analysis feedback is provided, comprising:

[0006] Construct a multi-layer perceptron classifier as a downstream task model;

[0007] Input the dirty data into the downstream task model, perform a classification task, and output the confidence of the classification result, and use the confidence as a soft label, wherein the soft label includes feedback information of the downstream task;

[0008] The dirty data and the soft labels are input into the error detection model. The error detection model first organizes the error detection results in the form of feature vectors, and clusters the generated feature vectors. The soft labels are used for weighting in the clustering process. Finally, a part of the data samples are selected as training samples according to the clustering results.

[0009] The training samples are subjected to stratified sampling and then input into a plurality of identical error detection models, the results of different error detection models are collected, and the results are integrated using a voting method to obtain a final error detection result;

[0010] A Transformer neural network model is constructed based on the cross-attention mechanism between samples as a data repair model to represent the features of the dirty data;

[0011] The final error detection result and dirty data are input into a repair model, and the repair model is trained through missing data interpolation tasks and original data reconstruction tasks to obtain preliminary data repair results;

[0012] The repair result is continuously optimized according to the feedback information, and the final cleaning result is obtained when the repair model converges.

[0013] According to a second aspect of an embodiment of the present application, a data cleaning system for downstream analysis feedback is provided, including:

[0014] The first construction module is used to construct a multi-layer perceptron classifier as a downstream task model;

[0015] A downstream prediction module, used to input dirty data into the downstream task model, perform classification tasks, and output the confidence of the classification results, using the confidence as a soft label, and the soft label contains feedback information of the downstream task;

[0016] A data processing module, used to input the dirty data and the soft label into the error detection model, wherein the error detection model first organizes the error detection results in the form of feature vectors, and clusters the generated feature vectors. The soft label is used for weighting in the clustering process, and finally, a part of the data samples is selected as training samples according to the clustering results;

[0017] An error detection module is used to perform stratified sampling on the training samples and input them into multiple identical error detection models, collect the results of different error detection models, integrate them using a voting method, and obtain a final error detection result;

[0018] The second construction module is used to construct a Transformer neural network model based on a cross-attention mechanism between samples, which is used as a data repair model to perform feature representation on the dirty data;

[0019] A data patching module is used to input the final error detection result and dirty data into a patching model for patching, obtain preliminary data patching results, and perform original data reconstruction tasks and missing data completion tasks to train the patching model;

[0020] The feedback optimization module is used to continuously optimize the repair result according to the feedback information, and obtain the final cleaning result when the repair model converges.

[0021] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including:

[0022] one or more processors;

[0023] A memory for storing one or more programs;

[0024] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0025] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0026] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0027] Compared with the prior art, the present invention optimizes the data patching strategy by combining downstream task feedback, thereby improving the quality and applicability of data patching and the performance of data on downstream tasks; integrates the results of multiple models through a majority voting mechanism, thereby improving the accuracy of error detection and the adaptability of the model; and the designed cross-attention mechanism between samples enables the data patching model to not only focus on a single data sample, but also consider the relationship between data samples. The patching model integrates more contextual information to obtain more accurate data cleaning results.

[0028] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0030] Figure 1 The present invention is a flowchart of a data cleaning method for downstream analysis feedback according to an exemplary embodiment.

[0031] Figure 2It is an overall block diagram of a model of a data cleaning method for downstream analysis feedback according to an exemplary embodiment.

[0032] Figure 3 It is a block diagram of a data cleaning system for downstream analysis feedback according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of systems and methods consistent with some aspects of the present application as detailed in the appended claims.

[0034] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0035] Existing data cleaning methods are mainly designed independently of downstream tasks and cannot meet the specific needs of downstream tasks. This lack of targeted processing may result in the cleaned data still not meeting the precise requirements of specific analysis, thus affecting the quality and credibility of the analysis results. Specifically, existing methods usually focus on general cleaning and standardization of data, while ignoring the applicability and effectiveness of data in specific analysis tasks. This will result in the performance of downstream tasks not being significantly improved when the cleaned data is used for downstream tasks, especially when there are complex data errors, which will also reduce the performance of downstream tasks.

[0036] In order to overcome the above technical problems, the embodiments of the present application provide a data cleaning method and system, and an electronic device for downstream analysis, which improve the pertinence and practicality of data cleaning and ultimately improve the accuracy of downstream tasks.

[0037] The data cleaning method for downstream analysis feedback provided by the present application can be generally applied to scenarios where task data is organized in a tabular form, such as patient health status prediction, power consumption, meteorological data classification, etc., and can improve the accuracy of these downstream tasks. In order to facilitate the description of the method of the present application, meteorological data classification is used as a downstream task for illustration in this embodiment.

[0038] Figure 1is a flow chart of a data cleaning method for downstream analysis provided by an embodiment of the present application. Figure 1 As shown, the method may include the following steps:

[0039] S1: Construct a multi-layer perceptron classifier as the downstream task model;

[0040] Specifically, the input layer of the multilayer perceptron contains m input units, where m is the number of features of the input data. The number of hidden layers n and the number of nodes in each hidden layer are specified by hyperparameters. The output layer contains c nodes, where c is the number of target classifications for the classification task. When the downstream model provides guidance for error detection, the output layer will directly output the confidence of all classification results as soft labels; when the downstream model performs downstream tasks or provides feedback for the completion model, the output layer also needs to output specific classifications through the SoftMax operation. When performing regression tasks, the output layer will be additionally connected to an output node to output the regression value, and the output layer will degenerate into the last hidden layer.

[0041] In order to improve the generalization ability of the downstream model, the multi-layer perceptron adds a batch normalization layer and a Dropout layer after each hidden layer. The batch normalization layer can accelerate the convergence of the model, reduce the gradient vanishing problem, and improve the generalization ability of the model. The Dropout layer can reduce the overfitting problem of the model and improve the generalization ability of the model.

[0042] S2: Input the dirty data into the downstream task model, perform the classification task, and output the confidence of the classification result, and use the confidence as a soft label, wherein the soft label includes feedback information of the downstream task;

[0043] Specifically, soft labels are a form of label representation relative to hard labels. In traditional classification tasks, hard labels are usually discrete category indications, such as directly labeling meteorological data samples as "sunny" or "cloudy". However, soft labels not only provide category indications, but also include the confidence of the classification model for each category, that is, the probability distribution of each category, such as the confidence of "sunny" is 0.92, the confidence of "cloudy" is 0.06, and the confidence of "rainy" is 0.02.

[0044] Let x i is the data point, y i is its corresponding hard label, and is the output of the MLP classifier about x i The soft label of Define the loss function to optimize the classifier.

[0045] S3: inputting the dirty data and the soft label into the error detection model together. The error detection model first organizes the error detection results in the form of feature vectors and clusters the generated feature vectors; the soft label is used for weighting in the clustering process; finally, according to the clustering results, a part of the data samples are selected as training samples; this step may include the following sub-steps:

[0046] S31: Perform preliminary error detection on input dirty data using built-in rules, and use three types of traditional error detection techniques to characterize data errors; the error detection techniques include outlier detection, pattern violation detection, and rule violation detection techniques; for each type of error detection technique, automatically generate a set of potential, controllable configurations, covering different thresholds of anomaly detection algorithms and different patterns and rules of corresponding violation detection algorithms; apply these error detection techniques to determine preliminary error detection results and obtain feature vectors;

[0047] Specifically, outlier detection uses a histogram model, based on the data item frequency threshold θ tf Detect outliers, and the threshold is selected in the range of {0.1, 0.2, ..., 0.9}. Pattern violation detection uses the bag of characters model to encode all possible data patterns and detect pattern violations. Rule violation detection limits single attribute dependencies and considers all attribute pairs as potential functional dependencies.

[0048] Each data unit is mapped to a feature vector composed of the output of the error detection strategy. Suppose there is a set of error detection strategies S = {s1, s2, …, s |S|}, run each strategy s∈S for each data unit in the data set d. Strategy s marks whether the data unit d[i,j] is a data error. If s marks the data unit d[i,j] as an error, then s(d[i,j])=1, otherwise s(d[i,j])=0. The feature vector of the data unit d[i,j] is the vector output by all error detection strategies s∈S on the data unit, which can be formally expressed as

[0049] During data processing, the model will perform a j The feature vector is post-processed to remove features that are numerical for all data units. Finally, each data unit is assigned a feature vector. Each component of the vector represents the binary output of a specific configuration of a specific error detection algorithm.

[0050] S32: hierarchical clustering is performed on the data samples using the feature vectors, where the distance between two classes is defined as the average distance between data points of the classes; after weighting the distance between the classes using the soft labels, the data samples are divided into a number of clusters as clustering results;

[0051] Specifically, suppose the feature vectors of two samples i and j are u i and u j , and the corresponding soft label probabilities are a i and a j The weighted distance between data points i and j can be defined as follows:

[0052]

[0053] For example, in meteorological data, the distance between data samples with similar meteorological conditions in the same region is smaller than that between data samples with different meteorological conditions in different regions;

[0054] S33: Determine a sample weighted sampling strategy based on the clustering results. In each round of iteration, calculate the number of unlabeled data points and the weight of the soft label in each cluster, and allocate sampling probability accordingly. Clusters with more unlabeled data points and larger weights will have higher sampling probabilities. Use the weighted sampling strategy to select data samples, and after the user marks them, obtain training samples.

[0055] In the specific implementation, we first count the number of unlabeled data points u in each cluster. k and the corresponding soft label weight w k , and combine this information to calculate the sampling probability p of each cluster according to the following formula k :

[0056]

[0057] Where m is the total number of clusters. Through this weighted strategy, clusters with a large number of unlabeled data points and high soft label weights will be preferred so that samples can be drawn from these clusters for user labeling.

[0058] Then, representative samples are selected from each cluster based on the calculated sampling probability. After the user labels the selected samples, the model uses this label information to propagate labels to other data points in the same cluster. By iteratively selecting samples and propagating labels, the number of unlabeled data points is gradually reduced, and the coverage and accuracy of labeling are improved.

[0059] S4: After stratified sampling, the training samples are input into multiple identical error detection models, the results of different error detection models are collected, and the results are integrated using a voting method to obtain a final error detection result; this step may include the following sub-steps:

[0060] S41: performing stratified sampling on the training samples, extracting subsamples from the original training data set at a predetermined ratio each time according to the category and characteristic distribution factors of the samples, and obtaining a total of several groups of training samples;

[0061] Specifically, taking meteorological data as an example, the training samples are first divided into four layers according to geographical regions: north, south, east, and west. Then, simple random sampling is performed in each layer. Finally, the samples drawn from all layers are merged to form a set of meteorological data samples across the country. This is repeated several times to obtain several sets of training samples.

[0062] S42: inputting the data samples obtained by stratified sampling into a plurality of identical error detection models for model training;

[0063] Specifically, each error detection model trains a classification model for each data column based on the feature vector of the data unit and the propagated data label. The trained classification model is used to predict the labels of the remaining data units in other unlabeled clusters. The classification results are the two states of normal and abnormal data. By integrating the results of each column, each model obtains the error detection results.

[0064] S43: Collect the detection results of different error detection models, integrate the results using the voting method, and obtain the final error detection result. Based on the group training, the results from each independent training model are integrated through ensemble learning to improve the accuracy and robustness of the final decision;

[0065] Specifically, the output of each model is a 0-1 matrix, where 1 indicates that the data is wrong and 0 indicates no error. In order to integrate the outputs of each grouping model, voting is used as an integration strategy. The voting method adopts the form of simple majority voting. In simple majority voting, the prediction result of each model for each element will be regarded as a "vote", and the final prediction result is determined by the category with the most votes. Mathematically, for the output matrices O1, O2, ..., O N , the integrated result R at the element position (i,j) ij It can be expressed as follows:

[0066]

[0067] Among them, O nij Represents the prediction result of the nth model at position (i, j).

[0068] S5: constructing a Transformer neural network model based on the cross-attention mechanism between samples as a data repair model to perform feature representation on the dirty data; this step may include the following sub-steps:

[0069] S51: Based on the cross-attention mechanism between samples, data information is dynamically shared between samples, and two cross-attention blocks between samples are constructed; the two cross-attention blocks between samples are connected in series so that each data sample unit can obtain the data information of the remaining positions of its row and column;

[0070] Specifically, in the cross-attention mechanism between samples, the dimension of the input tensor X is defined as b×n×d, where b represents the batch size, n represents the number of samples, and d is the dimension of the feature. Next, three learnable weight matrices W are used q ,W k ,W v , respectively calculate the query matrix Q, key matrix K, and value matrix V; the calculation formula for these matrices is Q = W q X, K = W k X, and V = W v X. In the self-attention mechanism, the product of query and key QK T By dividing To scale, and then apply the SoftMax operation to calculate the final attention score matrix A; the calculation method of the inter-sample attention score matrix A is the same as the calculation method of the score matrix of the existing self-attention mechanism;

[0071] The inter-sample attention mechanism is the result of the cross-fusion of the row attention mechanism and the column attention mechanism. The scope of row attention calculation is the different attributes of the same sample, so that the target data cell can be connected to other cells in its row. The calculation scope of column attention is other cell data in the column where the cell is located, that is, the attributes corresponding to other samples. By cross-using row and column attention, the target cell can be connected to the row and column data to obtain the corresponding semantic information. By using this cross-attention twice in series, the target cell can be connected to any cell to obtain more semantic information. For example, for the "temperature" value in a meteorological data sample, under the action of the inter-sample attention mechanism, when processing this cell, the model will not only consider other attributes of the current sample, such as "humidity" and "wind speed", but also connect the "temperature" values ​​of other different samples.

[0072] The output O is calculated through the score matrix A and the value matrix V. The specific operation is O = AV, that is, each value vector is weighted by the attention score to obtain a weighted feature representation. Finally, the output O is reshaped into the original b×n×d shape to complete the calculation process of the entire inter-sample attention.

[0073] S52: Perform basic feature repair through the cross-attention block between samples, introduce missing mask vectors to create a mechanism that can directly deal with and handle data incompleteness, and use the existing complete data to supplement and improve the representation of the missing parts;

[0074] Specifically, the feature vector and its missing mask vector After connecting through a linear layer, we get e, which is then input into an N-layer stacked sample-to-sample attention and feed-forward network to be converted to z, and then converted to z by the formula Get the learned feature representation Here W z and b z All are trainable parameters;

[0075] The eigenvector is obtained by the following formula:

[0076]

[0077] in is the eigenvector, is the missing mask vector, is the learned feature representation, ⊙ represents element-wise multiplication;

[0078] S53: Based on the basic feature restoration, the data is processed and additional nonlinear transformations are performed through a second deeper attention block to capture more complex data relationships and patterns and further refine the feature representation;

[0079] Specifically, the feature vector obtained in the previous stage Instead of missing mask vector After linear connection, we get α, which is input into the N-layer stacked sample-to-sample attention and feed-forward network and output β. We use two linear projections on β through the following formula, and finally get the enhanced feature representation through the ReLU activation function and bias.

[0080]

[0081] Where W β , b β , W γ , b γ is a trainable parameter.

[0082] S54: Combine the outputs of the two attention blocks in the previous two steps ( and ), and dynamically adjust their contribution according to the missing data to generate the final feature representation

[0083] Specifically, in order to dynamically balance and according to The attention weights A output by multiple heads in the last layer of the second attention block are averaged. Using the average attention weight and missing mask A linear layer is used to generate the combined weight η, and the weight η is used to combine and form The missing values ​​in Replace the corresponding values ​​in to generate the patch vector That is, the patched data

[0084] S6: Input the final error detection result and dirty data into a repair model, and train the repair model through missing data interpolation tasks and original data reconstruction tasks to obtain preliminary data repair results; this step may include the following sub-steps:

[0085] S61: Perform missing value interpolation tasks at missing locations in the data set; the patching model predicts the data value of each missing location and fills it in, and forms a loss function based on the feedback of the downstream task model for gradient descent optimization;

[0086] Specifically, the loss function of the missing value interpolation task receives three input parameters: estimated value, target value and mask, and calculates the mean absolute error L between the estimated value and the target value marked by the mask MIT as follows:

[0087]

[0088] Where L MIT The input estimate, target value, and mask are the patch feature vectors The original feature vector X without artificial masking and the indicator mask vector I;

[0089] S62: Perform the original data reconstruction task at the non-missing position of the data set; the patching model predicts any position of the data set, and performs gradient descent optimization based on the loss function composed of the error between the predicted value and the actual value;

[0090] Specifically, the loss function of the original data reconstruction task receives three input parameters: estimated value, target value and mask, and calculates the mean absolute error L between the estimated value and the target value marked by the mask ORT as follows:

[0091]

[0092] Where L ORT The target value and mask are the input feature vector and its missing mask vector and is the feature vector learned in the previous step, which is input here as an estimate.

[0093] S7: Use the downstream task model to generate feedback information for the repair model and continuously optimize the repair result. When the repair model converges, the final cleaning result is obtained. In this process, the error generated by the downstream model in the classification or regression task can be calculated by L DT It means that the optimization goal of the data patching model is to combine the loss of and the downstream task feedback through a weighted sum L, as follows:

[0094] L=L ORT +λL MIT +μL DT

[0095] Where λ and μ are adjustable weight coefficients; the patching model is updated and optimized by minimizing the final loss L;

[0096] Corresponding to the aforementioned embodiment of the data cleaning method for downstream analysis, the present application also provides an embodiment of a data cleaning system for downstream analysis.

[0097] Figure 3 is a block diagram of the data cleaning method system for downstream analysis of the present application. Figure 3 , the system comprises:

[0098] The first construction module 1 is used to construct a multi-layer perceptron classifier as a downstream task model;

[0099] Downstream prediction module 2, used to input dirty data into the downstream task model, perform classification tasks, and output the confidence of the classification results, using the confidence as a soft label, and the soft label contains feedback information of the downstream task;

[0100] Data processing module 3, used to input the dirty data and soft labels into the error detection model, the error detection model first organizes the error detection results in the form of feature vectors, and clusters according to the generated feature vectors, and uses the soft labels for weighting in the clustering process, and finally selects a part of the data samples as training samples according to the clustering results;

[0101] The error detection module 4 is used to perform stratified sampling on the training samples and input them into multiple identical error detection models, collect the results of different error detection models, integrate them using a voting method, and obtain a final error detection result;

[0102] The second construction module 5 is used to construct a Transformer neural network model based on a cross-attention mechanism between samples, which is used as a data repair model to perform feature representation on the dirty data;

[0103] Data patching module 6, used for inputting the final error detection result and dirty data into the patching model for patching, obtaining preliminary data patching results, and performing original data reconstruction tasks and missing data completion tasks to train the patching model;

[0104] The feedback optimization module 7 is used to continuously optimize the repair result according to the feedback information, and obtain the final cleaning result when the repair model converges.

[0105] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0106] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0107] Correspondingly, the present application also provides an electronic device, comprising: one or more sensors; one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors can implement the method described in the first aspect.

[0108] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the data cleaning method for downstream analysis as described above is implemented.

[0109] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0110] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data cleaning method for downstream analysis feedback, characterized in that: include: S1: Construct a multi-layer perceptron classifier as the downstream task model; S2: Input the dirty data into the downstream task model, perform the classification task, and output the confidence of the classification result, and use the confidence as a soft label, wherein the soft label includes feedback information of the downstream task; S3: inputting the dirty data and the soft label into the error detection model together. The error detection model first organizes the error detection results in the form of feature vectors, and clusters the generated feature vectors. The soft label is used for weighting in the clustering process. Finally, according to the clustering results, a part of the data samples are selected as training samples. S4: performing stratified sampling on the training samples and inputting them into multiple identical error detection models, collecting the results of different error detection models, integrating them using a voting method, and obtaining a final error detection result; S5: A Transformer neural network model is constructed based on the cross-attention mechanism between samples as a data repair model to represent the features of the dirty data; S6: inputting the final error detection result and dirty data into a repair model, the repair model is trained through missing data interpolation tasks and original data reconstruction tasks, and a preliminary data repair result is obtained; S7: Continuously optimize the repair result according to the feedback information, and obtain the final cleaning result when the repair model converges.

2. The method according to claim 1, characterized in that The multi-layer perceptron classifier comprises an input layer, an output layer, a normalization layer, a Dropout layer and an output layer; when the downstream model provides guidance for error detection, the output layer will directly output the confidence of all classification results as soft labels; when the downstream model performs downstream tasks or provides feedback for the completion model, the output layer also needs to output specific classifications through a SoftMax operation; when performing regression tasks, the output layer will be additionally connected to an output node to output regression values, and the output layer at this time will degenerate into the last hidden layer.

3. The method according to claim 1, characterized in that S3 specifically includes: S31: Perform preliminary error detection on the input dirty data and characterize data errors using three types of error detection techniques; the error detection techniques include outlier detection, pattern violation detection, and rule violation detection techniques; for each type of error detection technique, automatically generate a set of potential, controllable configurations, covering different thresholds of anomaly detection algorithms and different patterns and rules of corresponding violation detection algorithms; apply these error detection techniques to determine preliminary error detection results and obtain feature vectors; S32: hierarchical clustering is performed on the data sample using the feature vector, the distance between two classes is defined as the average distance between the data points of the classes, and after weighting the distance between the classes using the soft label, the data sample will be divided into a number of clusters as the clustering result; S33: Determine the sample weighted sampling strategy based on the clustering results. In each round of iteration, calculate the number of unlabeled data points and the weight of the soft label in each cluster, and allocate the sampling probability accordingly. Clusters with more unlabeled data points and larger weights will have higher sampling probabilities. Use the weighted sampling strategy to select data samples, and obtain training samples after the user marks them.

4. The method according to claim 1, characterized in that: S4 specifically includes: S41: performing stratified sampling on the training samples, extracting subsamples from the original training data set at a predetermined ratio each time according to the category and characteristic distribution factors of the samples, and obtaining a total of several groups of training samples; S42: inputting the data samples obtained by stratified sampling into a plurality of identical error detection models for model training; S43: Collect the detection results of different error detection models, integrate the results using a voting method, and obtain the final error detection result.

5. The method according to claim 1, characterized in that S5 specifically includes: S51: Based on the cross-attention mechanism between samples, data information is dynamically shared between samples, and two cross-attention blocks between samples are constructed. The two cross-attention blocks between samples are connected in series, so that each data sample unit can obtain the data information of the remaining positions of its row and column; S52: Perform basic feature repair through the cross-attention block between samples, introduce missing mask vectors to create a mechanism that can directly deal with and handle data incompleteness, and use the existing complete data to supplement and improve the representation of the missing parts; S53: Based on the basic feature restoration, the data is processed and additional nonlinear transformations are performed through a second deeper attention block to capture more complex data relationships and patterns and further refine the feature representation; S54: Combine the outputs of the two attention blocks in the previous two steps and dynamically adjust their contributions according to the missing data to generate the final feature representation.

6. The method according to claim 1, characterized in that S6 specifically includes: S61: Perform missing value interpolation tasks at missing locations in the data set; predict the data value of each missing location and fill it in through the patching model, and perform gradient descent optimization based on the loss function formed by the feedback of the downstream task model; S62: Perform the original data reconstruction task at the non-missing position of the data set; the patching model predicts any position of the data set, and performs gradient descent optimization based on the loss function composed of the error between the predicted value and the actual value.

7. The method according to claim 1, characterized in that In S7, the feedback of the downstream task model to the data patching model occurs throughout the entire process of data patching model training; the optimization goal of the data patching model is a weighted loss value combining the learned data representation of the two stages with the loss of the downstream task feedback; when this weighted loss value converges to a smaller range, the patching model obtains the final cleaning result.

8. A data cleaning system for downstream analysis, characterized in that: Specifically include: Construction module, used to construct a multi-layer perceptron classifier as a downstream task model; A downstream prediction module, used to input dirty data into the downstream task model, perform classification tasks, and output the confidence of the classification results, using the confidence as a soft label, and the soft label contains feedback information of the downstream task; A data processing module, used to input the dirty data and the soft label into the error detection model, wherein the error detection model first organizes the error detection results in the form of feature vectors, and clusters the generated feature vectors. The soft label is used for weighting in the clustering process, and finally, a part of the data samples is selected as training samples according to the clustering results; An error detection module is used to perform stratified sampling on the training samples and input them into multiple identical error detection models, collect the results of different error detection models, integrate them using a voting method, and obtain a final error detection result; A construction module is used to construct a Transformer neural network model based on a cross-attention mechanism between samples, which is used as a data repair model to perform feature representation on the dirty data; A data patching module is used to input the final error detection result and dirty data into a patching model for patching, obtain preliminary data patching results, and perform original data reconstruction tasks and missing data completion tasks to train the patching model; The feedback optimization module is used to continuously optimize the repair result according to the feedback information, and obtain the final cleaning result when the repair model converges.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images