Data cleaning method, system, and electronic equipment for downstream analysis feedback

Abstract

The present invention discloses a data cleaning method for downstream analysis feedback, including: constructing a multi-layer perceptron classifier to perform classification and generate soft labels; dirty data is combined with soft labels to input an error detection model, and preliminary error detection results in the form of feature vectors are output, and clustering results are obtained through clustering weighting; part of the data is selected for training, and after stratified sampling, multiple isomorphic error detection models are input, and the results are integrated by voting; a neural network repair model with a cross-attention mechanism is used to process error detection output and dirty data, and corrected data is output through feature extraction, optimization and fusion; downstream model feedback optimizes the repair process to obtain the final cleaned data. This method mainly detects data and repairs dirty data therein through a deep learning-based method, making the data more accurate and reliable. This method can customize the data according to the needs of downstream tasks to ensure that the cleaned data can be directly applied to these tasks.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data cleaning method and system, and electronic equipment for downstream analysis feedback. Background Art

[0002] With the development of big data technology, data volumes are exploding, data sources are diversifying, and data complexity and diversity are increasing. Data quality directly impacts the accuracy and effectiveness of data analysis and decision-making. High-quality data is the foundation of data analysis, but real-world data often contains various issues, such as missing values, outliers, duplicate values, and inconsistencies. Data with these issues is often referred to as "dirty data." Dirty data can compromise the accuracy and credibility of data analysis and even lead to erroneous conclusions and decisions.

[0003] Existing data cleaning methods often fail to fully consider the specific needs of downstream tasks. This lack of targeted processing may result in the cleaned data still not meeting the precise requirements of specific analyses, thereby affecting the quality and credibility of the analysis results. Existing methods often focus on general data cleaning and standardization, while ignoring the applicability and effectiveness of the data in specific analytical tasks. To improve the targetedness and practicality of data cleaning, it is urgent to research and develop data cleaning models that are oriented towards feedback from downstream analytical tasks. These models should be able to customize the data according to the specific requirements of downstream tasks, ensuring that the cleaned data is not only of high quality and consistency, but also directly meets the needs of downstream analytical tasks. Summary of the Invention

[0004] To overcome the above technical problems, the present invention provides a data cleaning method, system, and electronic device for downstream analysis feedback to address the dirty data problem. This method primarily uses a deep learning-based approach to detect and repair dirty data, making the data more accurate and reliable. This method can customize data processing based on the needs of downstream tasks, ensuring that the cleaned data can be directly applied to these tasks.

[0005] According to a first aspect of an embodiment of the present application, a data cleaning method for downstream analysis feedback is provided, comprising:

[0006] Construct a multi-layer perceptron classifier as the downstream task model;

[0007] Inputting dirty data into the downstream task model, performing a classification task, and outputting the confidence of the classification result, using the confidence as a soft label, wherein the soft label includes feedback information of the downstream task;

[0008] The dirty data and soft labels are input into the error detection model. The error detection model first organizes the error detection results in the form of feature vectors and clusters them based on the generated feature vectors. The soft labels are used for weighting in the clustering process. Finally, based on the clustering results, a portion of data samples are selected as training samples.

[0009] The training samples are subjected to stratified sampling and then input into multiple identical error detection models, the results of different error detection models are collected, and the results are integrated using a voting method to obtain a final error detection result;

[0010] A Transformer neural network model is constructed based on the cross-attention mechanism between samples as a data repair model to represent the features of the dirty data;

[0011] The final error detection result and dirty data are input into the repair model, and the repair model is trained through the missing data interpolation task and the original data reconstruction task to obtain a preliminary data repair result;

[0012] The repair result is continuously optimized according to the feedback information, and the final cleaning result is obtained when the repair model converges.

[0013] According to a second aspect of an embodiment of the present application, a data cleaning system for downstream analysis feedback is provided, comprising:

[0014] The first construction module is used to construct a multi-layer perceptron classifier as a downstream task model;

[0015] A downstream prediction module is configured to input dirty data into the downstream task model, perform classification, and output the confidence level of the classification result, using the confidence level as a soft label containing feedback information for the downstream task;

[0016] A data processing module is configured to input the dirty data and soft labels into an error detection model. The error detection model first organizes the error detection results in the form of feature vectors, clusters the generated feature vectors, and uses the soft labels for weighting during the clustering process. Finally, based on the clustering results, a portion of data samples is selected as training samples.

[0017] An error detection module is used to perform stratified sampling on the training samples and input them into multiple identical error detection models, collect the results of different error detection models, integrate them using a voting method, and obtain a final error detection result;

[0018] The second construction module is used to build a Transformer neural network model based on the cross-attention mechanism between samples, which serves as a data repair model to represent the features of the dirty data;

[0019] A data patching module is used to input the final error detection result and dirty data into the patching model for patching, obtain preliminary data patching results, and perform original data reconstruction tasks and missing data completion tasks to train the patching model;

[0020] The feedback optimization module is used to continuously optimize the repair result according to the feedback information, and obtain the final cleaning result when the repair model converges.

[0021] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including:

[0022] one or more processors;

[0023] a memory for storing one or more programs;

[0024] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0025] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0026] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0027] Compared with the existing technology, the present invention optimizes the data patching strategy by combining downstream task feedback, thereby improving the quality and applicability of data patching, as well as the performance of data on downstream tasks; integrating the results of multiple models through a majority voting mechanism, improving the accuracy of error detection and the adaptability of the model; the designed cross-attention mechanism between samples enables the data patching model to focus not only on a single data sample, but also consider the relationship between data samples. By integrating more contextual information, the patching model can obtain more accurate data cleaning results.

[0028] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0030] Figure 1 The present invention is a flowchart showing a data cleaning method for downstream analysis feedback according to an exemplary embodiment.

[0031] Figure 2It is an overall block diagram of a model of a data cleaning method for downstream analysis feedback according to an exemplary embodiment.

[0032] Figure 3 The present invention is a block diagram of a data cleaning system for downstream analysis feedback according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0034] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0035] Existing data cleaning methods are primarily designed independently of downstream tasks and fail to meet their specific needs. This lack of targeted processing can result in cleaned data still failing to meet the precise requirements of specific analyses, impacting the quality and credibility of the analysis results. Specifically, existing methods often focus on general data cleaning and standardization, while ignoring the data's applicability and effectiveness in specific analytical tasks. This results in minimal performance improvement for downstream tasks after the cleaned data is used, and can even degrade performance in the presence of complex data errors.

[0036] In order to overcome the above technical problems, the embodiments of the present application provide a data cleaning method and system, and an electronic device for downstream analysis, which improve the pertinence and practicality of data cleaning and ultimately improve the accuracy of downstream tasks.

[0037] The data cleaning method for downstream analysis feedback provided in this application can be generally applied to scenarios where task data is organized in a tabular form, such as patient health status prediction, power consumption, meteorological data classification, and other scenarios, and can improve the accuracy of these downstream tasks. To facilitate the description of the method of this application, meteorological data classification is used as a downstream task in this embodiment.

[0038] Figure 1This is a flow chart of a data cleaning method for downstream analysis provided by an embodiment of the present application. Figure 1 As shown, the method may include the following steps:

[0039] S1: Construct a multi-layer perceptron classifier as the downstream task model;

[0040] Specifically, the input layer of the multilayer perceptron contains m input units, where m is the number of features in the input data. The number of hidden layers, n, and the number of nodes in each hidden layer are specified by hyperparameters. The output layer contains c nodes, where c is the number of target categories for the classification task. When the downstream model provides guidance for error detection, the output layer directly outputs the confidence level of all classification results as soft labels. When the downstream model performs downstream tasks or provides feedback for model completion, the output layer also needs to undergo a SoftMax operation to output specific classifications. When performing regression tasks, the output layer will be additionally connected to an output node to output the regression value, and the output layer will then degenerate into the last hidden layer.

[0041] To improve the generalization of downstream models, this multilayer perceptron adds batch normalization and dropout layers after each hidden layer. Batch normalization accelerates model convergence, reduces the vanishing gradient problem, and improves model generalization. Dropout layers reduce overfitting and improve model generalization.

[0042] S2: Input the dirty data into the downstream task model, perform the classification task, and output the confidence of the classification result. The confidence is used as a soft label, and the soft label contains feedback information of the downstream task;

[0043] Specifically, soft labels are a form of label representation relative to hard labels. In traditional classification tasks, hard labels are typically discrete category indicators, such as directly labeling weather data samples as "sunny" or "cloudy." However, soft labels not only provide category indicators but also include the classification model's confidence in each category, that is, the probability distribution of each category, such as the confidence of "sunny" is 0.92, the confidence of "cloudy" is 0.06, and the confidence of "rainy" is 0.02.

[0044] Let x i is the data point, y i is its corresponding hard tag, and is the output of the MLP classifier about x i The soft label is, using the formula Define the loss function to optimize the classifier.

[0045] S3: The dirty data and soft labels are input into the error detection model. The error detection model first organizes the error detection results in the form of feature vectors and clusters them based on the generated feature vectors. The soft labels are used for weighting during the clustering process. Finally, based on the clustering results, a portion of data samples is selected as training samples. This step may include the following sub-steps:

[0046] S31: Perform preliminary error detection on the input dirty data using built-in rules and characterize data errors using three traditional error detection techniques; the error detection techniques include outlier detection, pattern violation detection, and rule violation detection techniques; for each error detection technique, automatically generate a set of potential, controllable configurations covering different thresholds for the anomaly detection algorithm and different patterns and rules for the corresponding violation detection algorithm; apply these error detection techniques to determine preliminary error detection results and obtain feature vectors;

[0047] Specifically, outlier detection uses a histogram model, based on the data item frequency threshold θ tf Outlier detection uses a threshold range of {0.1, 0.2, ..., 0.9}. Pattern violation detection uses a bag-of-characters model to encode all possible data patterns and detect pattern violations. Rule violation detection restricts single-attribute dependencies and considers all attribute pairs as potential functional dependencies.

[0048] Each data unit is mapped to a feature vector composed of the output of the error detection strategy. Suppose there is a set of error detection strategies S = {s1,s2,…,s |S|}, run each strategy s∈S for each data unit in the dataset d. Strategy s marks whether the data unit d[i,j] is a data error. If s marks the data unit d[i,j] as an error, then s(d[i,j])=1, otherwise s(d[i,j])=0. The feature vector of the data unit d[i,j] is the vector output by all error detection strategies s∈S on this data unit, which can be formally expressed as

[0049] During data processing, the model will perform j The feature vector is post-processed to remove features that are numerical for all data units. Ultimately, each data unit is assigned a feature vector. Each component of the vector represents the binary output of a specific configuration of a specific error detection algorithm.

[0050] S32: hierarchical clustering is performed on the data samples using the feature vectors, where the distance between two classes is defined as the average distance between data points in the classes; after weighting the distances between the classes using the soft labels, the data samples are divided into a number of clusters as clustering results;

[0051] Specifically, suppose the feature vectors of two samples i and j are u i and u j , and the corresponding soft label probabilities are a i and a j The weighted distance between data points i and j can be defined as follows:

[0052]

[0053] For example, in meteorological data, the distance between data samples with similar meteorological conditions in the same region is smaller than that between data samples with different meteorological conditions in different regions;

[0054] S33: Determine a weighted sampling strategy based on the clustering results. In each iteration, calculate the number of unlabeled data points and the weight of the soft label in each cluster, and assign sampling probabilities accordingly. Clusters with more unlabeled data points and larger weights will have higher sampling probabilities. Use the weighted sampling strategy to select data samples, which are labeled by the user to obtain training samples.

[0055] In the specific implementation, we first count the number of unlabeled data points u in each cluster k and the corresponding soft label weight w k , and combine this information to calculate the sampling probability p of each cluster according to the following formula k :

[0056]

[0057] Where m is the total number of clusters. Through this weighted strategy, clusters with a large number of unlabeled data points and high soft label weights will be preferentially selected so that samples can be extracted from these clusters for user labeling.

[0058] Based on the calculated sampling probability, a representative sample is then selected from each cluster. After the user labels the selected sample, the model uses this label information to propagate labels to other data points in the same cluster. By iteratively selecting samples and propagating labels, the number of unlabeled data points is gradually reduced, improving labeling coverage and accuracy.

[0059] S4: The training samples are subjected to stratified sampling and then input into multiple identical error detection models, the results of different error detection models are collected, and the results are integrated using a voting method to obtain a final error detection result. This step may include the following sub-steps:

[0060] S41: Perform stratified sampling on the training samples, extracting subsamples from the original training data set at a predetermined ratio each time according to the sample categories and feature distribution factors, and obtaining a total of several groups of training samples;

[0061] Specifically, taking meteorological data as an example, the training samples are first divided into four layers according to geographical regions: north, south, east, and west. Then, simple random sampling is performed within each layer. Finally, the samples drawn from all layers are merged to form a set of national meteorological data samples. Repeat this several times to obtain several sets of training samples.

[0062] S42: inputting the data samples obtained by stratified sampling into multiple identical error detection models for model training;

[0063] Specifically, each error detection model trains a classification model for each data column based on the data unit's feature vector and the propagated data labels. The trained classification model is then used to predict the labels of the remaining data units in the unlabeled clusters. The classification results are either normal or abnormal. By integrating the results from each column, each model generates its own error detection result.

[0064] S43: Collect the detection results of different error detection models and integrate them using voting to obtain the final error detection result. Based on group training, integrating the results from each independently trained model through ensemble learning can improve the accuracy and robustness of the final decision.

[0065] Specifically, the output of each model is a 0-1 matrix, where 1 indicates that the data has errors and 0 indicates that there are no errors. In order to integrate the outputs of each grouping model, voting is used as an integration strategy. The voting method adopts the form of simple majority voting. In simple majority voting, the prediction result of each model for each element will be regarded as a "vote", and the final prediction result is determined by the category with the most votes. Mathematically, for the output matrices O1, O2, ..., O N , the integration result R at element position (i,j) ij It can be expressed as follows:

[0066]

[0067] Among them, O nij Represents the prediction result of the nth model at position (i, j).

[0068] S5: Construct a Transformer neural network model based on the cross-attention mechanism between samples as a data repair model to perform feature representation on the dirty data. This step may include the following sub-steps:

[0069] S51: Based on the inter-sample cross-attention mechanism, data information is dynamically shared between samples, and two inter-sample cross-attention blocks are constructed. The two inter-sample cross-attention blocks are serially connected so that each data sample unit can obtain the data information of the remaining positions in its row and column.

[0070] Specifically, in the cross-attention mechanism between samples, the dimension of the input tensor X is defined as b×n×d, where b represents the batch size, n represents the number of samples, and d is the dimension of the feature. Next, three learnable weight matrices W q ,W k ,W v , respectively calculate the query matrix Q, key matrix K, and value matrix V; the calculation formula of these matrices is Q = W q X, K = W k X, and V = W v X. In the self-attention mechanism, the product of query and key QK T By dividing To perform scaling, the SoftMax operation is then applied to calculate the final attention score matrix A. The calculation method of the inter-sample attention score matrix A is the same as the calculation method of the score matrix of the existing self-attention mechanism.

[0071] The inter-sample attention mechanism is the result of the cross-fusion of the row attention mechanism and the column attention mechanism. The scope of row attention calculation is the different attributes of the same sample, so that the target data cell can be connected to other cells in its row. The calculation scope of column attention is other cell data in the column where the cell is located, that is, the attributes corresponding to other samples. By cross-using row and column attention, the target cell can be connected with the row and column data to obtain the corresponding semantic information. By using this cross-attention twice in series, the target cell can be connected with any cell to obtain more semantic information. For example, for the "temperature" value in a meteorological data sample, under the action of the inter-sample attention mechanism, when processing this cell, the model will not only consider other attributes of the current sample, such as "humidity" and "wind speed", but also connect the "temperature" values ​​of other different samples.

[0072] The output O is calculated using the score matrix A and the value matrix V. The specific operation is O = AV, that is, each value vector is weighted by the attention score to obtain a weighted feature representation. Finally, the output O is reshaped into the original b×n×d shape, completing the entire inter-sample attention calculation process.

[0073] S52: Perform basic feature restoration through the cross-attention block between samples, introduce missing mask vectors to create a mechanism that can directly cope with and handle data incompleteness, and use the existing complete data to supplement and improve the representation of the missing parts;

[0074] Specifically, the feature vector and its missing mask vector After connecting through a linear layer, we get e, which is then input into an N-layer stacked inter-sample attention and feedforward network to be converted to z, and then converted to z by the formula Get the learned feature representation The W here z and b z These are all trainable parameters;

[0075] The eigenvector is obtained by the following formula:

[0076]

[0077] in is the eigenvector, is the missing mask vector, is the learned feature representation, ⊙ represents element-wise multiplication;

[0078] S53: Based on the basic feature restoration, a second deeper attention block is used to process the data and perform additional nonlinear transformations to capture more complex data relationships and patterns and further refine the feature representation;

[0079] Specifically, the feature vector obtained in the previous stage Instead of missing mask vector After linear connection, we get α, which is input into the N-layer stacked inter-sample attention and feedforward network and output β. Then, we use two linear projections on β through the following formula, and finally get the enhanced feature representation through the ReLU activation function and bias.

[0080]

[0081] Where W β , b β , W γ , b γ is a trainable parameter.

[0082] S54: Combine the outputs of the two attention blocks in the first two steps ( and ), and dynamically adjust their contribution according to the missing data to generate the final feature representation

[0083] Specifically, in order to dynamically balance the missing information and according to The attention weights A output by multiple heads in the last layer of the second attention block are averaged. Using the averaged attention weights and missing mask Generate combined weights η through a linear layer, and use weights η to combine and form The missing values ​​in Replace the corresponding values ​​in to generate a patch vector That is, the patched data

[0084] S6: Input the final error detection result and dirty data into the repair model. The repair model is trained through missing data interpolation tasks and original data reconstruction tasks to obtain preliminary data repair results. This step may include the following sub-steps:

[0085] S61: Perform missing value interpolation tasks at missing locations in the dataset; the patching model predicts the data value of each missing location and fills it in, and forms a loss function based on the feedback of the downstream task model and performs gradient descent optimization;

[0086] Specifically, the loss function of the missing value imputation task receives three input parameters: estimated value, target value and mask, and calculates the mean absolute error L between the estimated value and the target value marked by the mask MIT as follows:

[0087]

[0088] Among them L MIT The input estimate, target value and mask are the patch feature vectors The original feature vector X without artificial masking and the indicator mask vector I;

[0089] S62: Perform the original data reconstruction task at the non-missing position of the dataset; the patching model predicts any position of the dataset and performs gradient descent optimization based on the loss function composed of the error between the predicted value and the actual value;

[0090] Specifically, the loss function of the original data reconstruction task receives three input parameters: estimated value, target value and mask, and calculates the mean absolute error L between the estimated value and the target value of the mask mark ORT as follows:

[0091]

[0092] Among them L ORT The target value and mask are the input feature vector and its missing mask vector and is the feature vector learned in the previous step, which is input here as an estimate.

[0093] S7: Use the downstream task model to generate feedback information for the repair model and continuously optimize the repair results; when the repair model converges, the final cleaning result is obtained; in this process, the error generated by the downstream model in the classification or regression task can be calculated using L DT Indicates that the optimization goal of the data patching model is to combine the loss of and the downstream task feedback through a weighted sum L as follows:

[0094] L=L ORT +λL MIT +μL DT

[0095] Where λ and μ are adjustable weight coefficients; the patching model is updated and optimized by minimizing the final loss L;

[0096] Corresponding to the aforementioned embodiment of the data cleaning method for downstream analysis, the present application also provides an embodiment of a data cleaning system for downstream analysis.

[0097] Figure 3 This is a block diagram of the data cleaning method system for downstream analysis of this application. Figure 3 , the system comprises:

[0098] The first construction module 1 is used to construct a multi-layer perceptron classifier as a downstream task model;

[0099] Downstream prediction module 2, configured to input dirty data into the downstream task model, perform classification, and output the confidence of the classification result, using the confidence as a soft label containing feedback information of the downstream task;

[0100] Data processing module 3 is used to input the dirty data and soft labels into the error detection model. The error detection model first organizes the error detection results in the form of feature vectors and clusters them based on the generated feature vectors. The soft labels are used for weighting in the clustering process. Finally, based on the clustering results, a portion of data samples is selected as training samples.

[0101] The error detection module 4 is used to perform stratified sampling on the training samples and input them into multiple identical error detection models, collect the results of different error detection models, integrate them using a voting method, and obtain a final error detection result;

[0102] The second construction module 5 is used to construct a Transformer neural network model based on the cross-attention mechanism between samples, which serves as a data repair model to represent the features of the dirty data;

[0103] Data patching module 6, used to input the final error detection result and dirty data into the patching model for patching, obtain preliminary data patching results, and perform original data reconstruction tasks and missing data completion tasks to train the patching model;

[0104] The feedback optimization module 7 is used to continuously optimize the repair result according to the feedback information, and obtain the final cleaning result when the repair model converges.

[0105] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0106] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0107] Correspondingly, the present application also provides an electronic device, comprising: one or more sensors; one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors are able to implement the method described in the first aspect.

[0108] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the data cleaning method for downstream analysis as described above.

[0109] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0110] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data cleaning method for downstream analysis feedback, characterized in that: include: S1: Construct a multi-layer perceptron classifier as the downstream task model; S2: Input the dirty data into the downstream task model, perform classification, and output the confidence level of the classification result. The confidence level is used as a soft label, and the soft label contains feedback information for the downstream task. The dirty data is meteorological dirty data including temperature, humidity, and wind speed. S3: The dirty data and soft labels are input into the error detection model. The error detection model first organizes the error detection results in the form of feature vectors and clusters them based on the generated feature vectors. The soft labels are used for weighting in the clustering process. Finally, based on the clustering results, a portion of data samples is selected as training samples. S4: performing stratified sampling on the training samples and inputting them into multiple identical error detection models, collecting the results of different error detection models, integrating them using a voting method, and obtaining a final error detection result; S5: Based on the cross-attention mechanism between samples, a Transformer neural network model is constructed as a data repair model to represent the features of the dirty data; S6: Inputting the final error detection result and dirty data into the repair model, the repair model is trained through the missing data interpolation task and the original data reconstruction task to obtain a preliminary data repair result; S7: Continuously optimizing the repair result according to the feedback information, and obtaining the final cleaning result when the repair model converges; Among them, S5 specifically includes: S51: Based on the inter-sample cross-attention mechanism, data information is dynamically shared between samples, and two inter-sample cross-attention blocks are constructed. The two inter-sample cross-attention blocks are connected in series so that each data sample unit can obtain the data information of the remaining positions in its row and column; S52: Perform basic feature restoration through the cross-attention block between samples, introduce missing mask vectors to create a mechanism that can directly cope with and handle data incompleteness, and use the existing complete data to supplement and improve the representation of the missing parts; S53: Based on the basic feature restoration, a second deeper attention block is used to process the data and perform additional nonlinear transformations to capture more complex data relationships and patterns and further refine the feature representation; S54: Combine the outputs of the two attention blocks in the first two steps and dynamically adjust their contributions according to the missing data to generate the final feature representation.

2. The method according to claim 1, characterized in that The multi-layer perceptron classifier includes an input layer, an output layer, a normalization layer, a Dropout layer, and an output layer; when the downstream model provides guidance for error detection, the output layer will directly output the confidence of all classification results as soft labels; when the downstream model performs downstream tasks or provides feedback for the completion model, the output layer also needs to undergo a SoftMax operation to output a specific classification; when performing regression tasks, the output layer will be additionally connected to an output node to output the regression value, and the output layer at this time will degenerate into the last hidden layer.

3. The method according to claim 1, characterized in that S3 specifically includes: S31: Perform preliminary error detection on the input dirty data and characterize data errors using three types of error detection techniques; the error detection techniques include outlier detection, pattern violation detection, and rule violation detection techniques; for each type of error detection technique, automatically generate a set of potential, controllable configurations covering different thresholds of the anomaly detection algorithm and different patterns and rules of the corresponding violation detection algorithm; apply these error detection techniques to determine preliminary error detection results and obtain feature vectors; S32: hierarchical clustering is performed on the data samples using the feature vectors. The distance between two classes is defined as the average distance between data points in the classes. After weighting the distance between the classes using the soft labels, the data samples are divided into a number of clusters as clustering results. S33: Determine the sample weighted sampling strategy based on the clustering results. In each round of iteration, calculate the number of unlabeled data points and the weight of the soft label in each cluster, and allocate sampling probabilities accordingly. Clusters with more unlabeled data points and larger weights will have higher sampling probabilities. Use the weighted sampling strategy to select data samples, and after the user marks them, obtain training samples.

4. The method according to claim 1, wherein S4 specifically includes: S41: Perform stratified sampling on the training samples, extracting subsamples from the original training data set at a predetermined ratio each time according to the sample categories and feature distribution factors, and obtaining a total of several groups of training samples; S42: inputting the data samples obtained by stratified sampling into multiple identical error detection models for model training; S43: Collect the detection results of different error detection models, integrate the results using a voting method, and obtain the final error detection result.

5. The method according to claim 1, wherein S6 specifically includes: S61: Perform missing value interpolation tasks at missing locations in the dataset; use the patching model to predict the data value of each missing location and fill it in, and use the feedback from the downstream task model to form a loss function for gradient descent optimization; S62: Perform the original data reconstruction task at the non-missing position of the dataset; the patching model predicts any position of the dataset, and performs gradient descent optimization based on the loss function composed of the error between the predicted value and the actual value.

6. The method according to claim 1, characterized in that In S7, the feedback of the downstream task model to the data patching model occurs throughout the entire process of data patching model training; the optimization goal of the data patching model is the weighted loss value combining the learned data representation of the two stages with the loss of the downstream task feedback; when this weighted loss value converges to a smaller range, the patching model obtains the final cleaning result.

7. A data cleaning system for downstream analysis, characterized in that: Specifically include: Construction module, used to construct a multi-layer perceptron classifier as a downstream task model; A downstream prediction module is configured to input dirty data into the downstream task model, perform classification, and output the confidence level of the classification result, using the confidence level as a soft label containing feedback information for the downstream task; A data processing module is configured to input the dirty data and soft labels into an error detection model. The error detection model first organizes the error detection results in the form of feature vectors, clusters the generated feature vectors, and uses the soft labels for weighting during the clustering process. Finally, based on the clustering results, a portion of data samples is selected as training samples. An error detection module is used to perform stratified sampling on the training samples and input them into multiple identical error detection models, collect the results of different error detection models, integrate them using a voting method, and obtain a final error detection result; A construction module is used to construct a Transformer neural network model based on a cross-attention mechanism between samples, which serves as a data repair model to represent the features of the dirty data; A data patching module is used to input the final error detection result and dirty data into the patching model for patching, obtain preliminary data patching results, and perform original data reconstruction tasks and missing data completion tasks to train the patching model; A feedback optimization module is used to continuously optimize the repair result according to the feedback information, and obtain the final cleaning result when the repair model converges; A Transformer neural network model is constructed based on the cross-attention mechanism between samples as a data repair model to represent the features of the dirty data, including: Based on the inter-sample cross-attention mechanism, data information is dynamically shared between samples, and two inter-sample cross-attention blocks are constructed. The two inter-sample cross-attention blocks are connected in series, so that each data sample unit can obtain the data information of the rest of the positions in its row and column; By using the cross-attention block between samples to perform basic feature restoration, a missing mask vector is introduced to create a mechanism that can directly cope with and handle data incompleteness, using the existing complete data to supplement and improve the representation of the missing parts; On the basis of basic feature restoration, a second deeper attention block is used to process the data and perform additional nonlinear transformations to capture more complex data relationships and patterns and further refine the feature representation; The outputs of the two attention blocks in the first two steps are combined, and their contributions are dynamically adjusted according to the missing data to generate the final feature representation.

8. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images