Complex table data error detection method and device, and electronic equipment

Abstract

The present invention discloses a complex table data error detection method, comprising: obtaining complex table data samples; constructing a logical grouping module, a joint characterization module, a special pair sampling module and a classifier module corrected by an inverse propensity score; the logical grouping module logically groups attribute columns based on a distance correlation coefficient, and divides attribute columns that are most likely to have a contextual semantic relationship into a group; the joint characterization module enhances the characterization capability of the contextual logical relationship of feature vectors in each group; the special pair sampling module based on a cluster sum of squares function and a bce-rerank model can accurately locate the most informative special case data pairs in the case of a small number of labeled instances; the classifier module based on an inverse propensity score correction is intended to eliminate the selection bias problem in the sampling process; and the labeled table data samples are input into the above-mentioned error detection model for processing to obtain a final error detection result.

Description

Technical Field

[0001] The present invention relates to the field of complex table data detection, and more particularly to a complex table data error detection method and device, and electronic equipment. Background Art

[0002] With the rapid development of the internet, massive amounts of data continue to emerge and accumulate. By analyzing this massive data, we can unlock its potential value. Data cleaning is a fundamental yet challenging step in various relational data analysis tasks, such as entity matching, data integration, and analytics. This is because data errors can severely impact data quality, mislead downstream decision-making, and even cause significant financial losses to businesses. Research indicates that in the real world, approximately 5% of data units in a dataset contain errors due to various reasons.

[0003] Data error detection usually includes two stages: error detection (ED) to identify erroneous values and error correction (EC) to fix erroneous values. Currently, error detection methods can be mainly divided into the following categories: (1) outlier detection, which detects erroneous values that deviate significantly from the overall distribution of data based on statistical methods; (2) duplicate value detection, which removes duplicate values by identifying tuples pointing to the same entity; (3) format error detection, which detects format errors through predefined patterns (such as the date format 2000 / 01 / 01); (4) rule violation detection, which detects data errors that violate rules based on given data rules; (5) external knowledge violation detection, which uses external knowledge (such as master data, knowledge base) to detect data errors.

[0004] Currently, there are numerous error detection algorithms in the existing technology. However, in order to achieve high prediction scores while sampling a small number of labeled instances, the sampling strategies in these algorithms typically prioritize data with high information content and high frequency of occurrence, while sampling data with low information content and low frequency of occurrence with a very low probability. This results in an uneven distribution of error patterns in the sampled data, with the number of sampled data under different error patterns varying significantly, indicating selection bias in the sampled data. This selection bias in the sampled data can cause binary classification models to overfit samples from high-frequency categories during training, while lacking the ability to recognize samples from low-frequency categories. This bias affects the model's generalization ability. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a complex table data error detection method and device, and electronic equipment to improve the accuracy and efficiency of table data error detection.

[0006] To achieve the above object, the present invention provides the following technical solution: a complex table data error detection method, comprising the following steps:

[0007] Step 1: Obtain tabular data samples and construct tabular data matrix and tabular data label matrix;

[0008] Step 2: Build a complex table data error detection model. The error detection model includes:

[0009] A grouping module, which logically groups attribute columns based on distance correlation coefficients and groups attribute columns that are most likely to have contextual semantic relationships into one group;

[0010] Joint representation module, which uses sentence-transformer and wordvec to represent the data within the logical group, thereby strengthening the representation ability of the contextual logical relationship of the feature vectors in each group;

[0011] Special case pair sampling module, which uses the intra-cluster square sum function and the BCE-rerank model to accurately locate special case data pairs in the same cluster, thereby screening out the most informative special case data pairs;

[0012] The classifier module uses an inverse propensity score correction classifier to eliminate the influence of selection bias in the sampling process on the classifier;

[0013] Step three: Input the tabular data samples into the trained complex tabular data error detection model, and obtain the error detection results through analysis and processing by the grouping module, joint representation module, special case sampling module and classifier module.

[0014] As a further improvement of the present invention, the specific steps of obtaining the table data sample and constructing the table data matrix in step 1 are as follows:

[0015] Step 11: obtain a standardized table data sample from a financial website using an automated crawler tool or a database query interface;

[0016] Step 1 and 2: Construct a table data matrix with n data and m attribute columns based on the multi-view missing face image data samples , No. Expressed as At the same time, a tabular data label matrix with k data and m attribute columns is constructed based on the multi-view missing face image label samples { }, Y Expressed as ;

[0017] In the process of constructing the tabular data matrix, the error in the tabular data is defined as its corresponding ground truth value Any cell value that is different , and in The cell value is assigned to the positive class when , or when , is assigned to the negative class, then the cell value is considered correctly classified; if The cell value is assigned to the negative class when , or when , is assigned to the positive class, then the cell value is considered misclassified.

[0018] As a further improvement of the present invention, the analysis and processing steps of the grouping module in the complex table data error detection model in step 2 are as follows:

[0019] Step 21: For two groups of samples and , calculate the Eubian distance matrix between them respectively:

[0020] ;

[0021] Step 22: Then, double-center the distance matrix by subtracting the mean of its row and column from each element and adding back the mean of the entire matrix:

[0022]

[0023] and, is the element in the i-th row and j-th column of the original distance matrix, is the element in the i-th row and j-th column of the original distance matrix b, 、 、 Respectively The row, column and overall averages of 、 、 Respectively The row, column and overall averages of

[0024] Step 23: Based on the doubly centered distance matrix obtained in step 22, further calculate the distance covariance and distance variance:

[0025]

[0026] ;

[0027] Step 24: Calculate the distance correlation coefficient based on the distance covariance and distance variance , the distance correlation coefficient , the value range is [0,1], 0 means complete independence, and 1 means there is a deterministic relationship:

[0028]

[0029] Step 25: Using the dCor metric, the DS strategy is used for each dirty column. Select columns ,in Exceeding the threshold , forming a related column set ,Right now:

[0030] .

[0031] As a further improvement of the present invention, the analysis and processing steps of the joint characterization module in step 2 are as follows:

[0032] First, Sentence-Transformer is used to obtain sentence-level representation of the data in each logical group. Then, WordVec is used to extract word-level features. The two are concatenated or weighted summed to form a richer feature vector. Based on the fused features, the self-attention mechanism is applied to assign different weights to different parts according to the context to further optimize the feature representation.

[0033] As a further improvement of the present invention, the analysis and processing steps of the sampling module in the special case of step 2 are as follows:

[0034] First, the grouped data is converted into multiple sets of embedding vectors using the joint representation module, and then the embedding vectors are clustered using the DBSCAN algorithm to obtain If there are k labeled instances in a cluster, k / 2 samplings are performed. The sampling strategy is divided into two stages: the first stage is group sampling, and the second stage is special pair sampling to screen out the most informative special case data pairs.

[0035] As a further improvement of the present invention, the analysis and processing steps of the classifier module in step 2 are as follows:

[0036] The IPS score concept is introduced into the classifier module. By calculating the propensity score and constructing the weight matrix, the weight matrix is then applied to the standard cross-entropy loss function of the binary classifier to obtain a binary classifier corrected based on the IPS score. The binary classifier corrected based on the IPS score is then used to eliminate the influence of the selection bias problem in the sampling process on the classifier.

[0037] As a further improvement of the present invention, in step 3, before inputting the tabular data sample into the trained complex tabular data error detection model, the tabular data matrix and the tabular data label matrix are first spliced to obtain the tabular data splicing matrix C, which is specifically as follows:

[0038] C ={ }

[0039] in, Indicates the value of the mth attribute column of the i-th data, Indicates the value of the annotation label corresponding to the mth attribute column of the i-th data;

[0040] Then, the tabular data splicing matrix C is input into the complex tabular data error detection model for training to obtain a trained tabular data error detection model.

[0041] Another aspect of the present invention provides a complex table data error detection device, which is used to execute the above complex table data error detection method, comprising:

[0042] Acquisition module (1), used to acquire tabular data samples and construct tabular data matrix and tabular data label matrix;

[0043] A modeling module (2) is used to construct a complex table data error detection model, which includes a grouping module based on distance correlation coefficient, a joint representation module based on WordVec model and Sentence-transformer model, a special case pair sampling module based on intra-cluster square sum function and bce-rerank model, and a classifier module based on inverse propensity score correction;

[0044] A splicing module (3) is used to splice the table data matrix and the table data label matrix to obtain a table data splicing matrix;

[0045] A training module (4) is used to input the table data splicing matrix into the complex table data error detection model for training, thereby obtaining a trained table data error detection model;

[0046] The error detection module (5) is used to input the table data matrix into the trained complex table data error detection model for error detection to obtain the final error detection result.

[0047] On the other hand, the present invention provides an electronic device comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the above-mentioned complex table data error detection method.

[0048] Another aspect of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer-readable storage medium is run on a computer, the computer can execute the complex table data error detection method.

[0049] The beneficial effects of the present invention are as follows: compared with the prior art, the present method uses a distance correlation coefficient to logically group attribute columns, and divides the attribute columns that are most likely to have contextual semantic relationships into a group; the sentence-transformer model and the wordvec model are jointly used to represent the data after logical grouping, thereby enhancing the representation ability of the contextual semantics of the feature vectors in each group, so that a high error detection accuracy can be achieved even with a small number of labeled instances; the intra-cluster sum of squares function and the bce-rerank model are used to accurately locate special case data pairs, thereby accurately screening out the most informative special case data; the inverse propensity score correction classifier is used to eliminate the selection bias problem in the sampling process; the present method overcomes the problems of low detection accuracy and low time efficiency of the existing tabular data error detection model in the case of a small number of labeled instances. Compared with the most advanced complex tabular data error detection method, the present method improves the detection accuracy by more than 27% and the detection efficiency by more than 13%. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the complex table data error detection method of the present invention;

[0051] Figure 2 Schematic diagram of the complex tabular data error detection model;

[0052] Figure 3 The block diagram of the complex table data error detection device. DETAILED DESCRIPTION

[0053] The present invention will be further described below with reference to the embodiments shown in the accompanying drawings.

[0054] Of the five error types mentioned in the background art, the most complex is violating data context semantic relationships. The other four error types are easy to detect, but data context semantic relationships are more difficult to detect. In tabular datasets, some attribute columns often have data context semantic relationships with each other, but not with other attribute columns. In such cases, if these columns can be grouped into different groups, the resulting embedding vector representation can more accurately represent the context semantic relationships.

[0055] Based on the above concept, this embodiment provides a complex table data error detection method, referring to Figure 1 As shown, it mainly includes the following steps:

[0056] S1: Obtain tabular data samples and construct tabular data matrices and tabular data label matrices. This step includes the following substeps:

[0057] S11: Collect standard table data samples from financial websites through automated crawler tools or database query interfaces.

[0058] S12: Construct a tabular data matrix.

[0059] Construct a table data matrix of n data and m attribute columns based on the multi-view missing face image data samples . No. It can be expressed as ;

[0060] Construct a tabular data label matrix of k data and m attribute columns based on the multi-view missing face image label samples { }. It can be expressed as .

[0061] In this embodiment, the error is defined as the ground truth value corresponding to Any cell value that is different The goal is to find the value of each cell in a given dataset D. Determine the most likely positive or negative assignment. The cell value is assigned to the positive class when , or when , is assigned to the negative class, the cell value is considered to be correctly classified; if The cell value is assigned to the negative class when , or when , is assigned to the positive class, then the cell value is considered misclassified.

[0062] S2: Construct a complex table data error detection model, including a grouping module based on distance correlation coefficient, a joint representation module based on WordVec model and Sentence-transformer model, a special case pair sampling module based on the intra-cluster square sum function and bce-rerank model, and a classifier module based on inverse propensity score correction; refer to Figure 2 , building the model includes the following modules:

[0063] Grouping module: logically groups attribute columns based on distance correlation coefficients, and groups attribute columns that are most likely to have contextual semantic relationships into one group;

[0064] Joint representation module: uses sentence-transformer and wordvec to represent the data within the logical group, thereby strengthening the representation ability of the contextual logical relationship of the feature vectors within each group;

[0065] Special case pair sampling module: uses the intra-cluster square sum function and the BCE-rerank model to accurately locate special case data pairs in the same cluster, thereby accurately screening out the most informative special case data pairs;

[0066] Classifier module: The inverse propensity score is used to correct the classifier, aiming to eliminate the impact of selection bias in the sampling process on the classifier.

[0067] Next, we will further explain the above four modules:

[0068] S21: The grouping module logically groups the attribute columns based on the distance correlation coefficient. The grouping module divides the attribute columns that are most likely to have contextual semantic relationships into one group.

[0069] Specifically, this embodiment introduces a column selection strategy (DS) based on the distance correlation coefficient. This strategy uses the distance correlation coefficient (DCOR) to evaluate inter-column correlation and filter out irrelevant columns to facilitate data retrieval. Specifically, the DS strategy uses the DCOR metric to evaluate inter-column correlation. Compared to the target dirty column m, a column n with a higher DCOR value tends to contain more actual correlation information with column m, which is crucial for effective ED.

[0070] The first step of the grouping module is to construct the distance matrix:

[0071] First, for the two groups of samples and , calculate the Eubian distance matrix between them respectively:

[0072]

[0073] The distance matrix is then double-centered, which means that the average value of its row and column is subtracted from each element, and then the average value of the entire matrix is added back. This process is intended to eliminate the offset effect of the distance matrix. The formula is as follows:

[0074]

[0075] and, is the element in the i-th row and j-th column of the original distance matrix, is the element in the i-th row and j-th column of the original distance matrix b, 、 、 Respectively The row, column and overall averages of 、 、 Respectively The row, column and overall averages of .

[0076] Based on the doubly centered distance matrix, we further calculate the distance covariance and distance variance. Distance covariance is a key indicator for measuring the distance similarity between two vectors, while distance variance quantifies the distance variation within a single vector. The calculation formula is as follows:

[0077]

[0078]

[0079] Finally, the distance correlation coefficient is calculated based on the distance covariance and distance variance , which is a standardized indicator to measure the dependence of two random vectors, with a value range of [0,1], where 0 indicates complete independence and 1 indicates a deterministic relationship:

[0080]

[0081] Using the dCor metric, the DS strategy for each dirty column Select columns ,in Exceeding the threshold , forming a related column set ,Right now:

[0082]

[0083] In this embodiment, this embodiment will It is set to 0.5 to balance the coverage of relevant columns with the need for high information content, ensuring that key features are retained while minimizing information loss. This strategy ensures that the most relevant columns with high dCor values are selected to construct each group of data, so that the data columns within each group are closely related, and the embedding vector model built on this basis can contain contextual semantic information.

[0084] S22: The joint representation module uses sentence-transformer and wordvec to represent the data in the logical group, thereby enhancing the representation ability of the contextual logical relationship of the feature vectors in each group;

[0085] Specifically, Sentence-Transformer is first used to obtain sentence-level representations of the data within each logical group, followed by WordVec to extract word-level features. These features are then concatenated or weighted together to form a richer feature vector. Based on these fused features, a self-attention mechanism is applied to further optimize the feature representation by assigning different weights to different components based on the context. This approach not only captures local semantic information but also better understands global contextual logical relationships, ultimately improving the quality and expressiveness of the feature vectors within each group.

[0086] The joint representation module first processes the tabular data: Contains m rows and n columns, where the first k columns are text columns and the last nk columns are numeric columns. The specific representation is as follows:

[0087]

[0088] in Represents the data in row i and column j.

[0089] Next, use SentenceTransformer for embedding to extract text columns from the tabular data:

[0090]

[0091] Then use SentenceTransformer to encode the text column:

[0092]

[0093] in is the output dimension of SentenceTransformer. Then use WordVec to encode the text column:

[0094]

[0095] in is the output dimension of WordVec. Concatenate the embedding results of SentenceTransformer and WordVec:

[0096]

[0097] Extract numeric columns:

[0098]

[0099] Concatenate the joint embedding of the text column and the features of the numeric column:

[0100]

[0101] Define the query, key, and value matrices:

[0102]

[0103] Calculate the attention score:

[0104]

[0105] in is the dimension size of the key, usually = + + . Apply the Softmax function to get the attention weight distribution:

[0106]

[0107] Finally, the weighted sum is calculated to get the output:

[0108]

[0109] S23: The special case pair sampling module uses the intra-cluster square sum function and the BCE-rerank model to accurately locate the special case data pairs in the highest-scoring cluster, thereby accurately screening out the most informative tabular data.

[0110] Specifically, the grouped data is first converted into multiple sets of embedding vectors using the joint representation module, and then the embedding vectors are clustered using the DBSCAN algorithm to obtain If there are k labeled instances, k / 2 samplings are performed. The sampling strategy is divided into two stages: the first stage is group sampling, and the second stage is special pair sampling.

[0111] First stage sampling: The first stage of sampling is The cluster with the highest score is found based on the cluster score function. The cluster score function is as follows:

[0112]

[0113] in: Indicates the number of data items that have been labeled in the cluster. Represents the within-cluster sum of squares (Within-Cluster Sum of Squares), which is used to measure the compactness or variability of data points within a cluster.

[0114] The cluster score function evaluates the quality of each cluster by combining the above three variables. In order to evaluate the homogeneity of the data set, this embodiment introduces the concept of within-cluster sum of squares (WCSS) into the cluster score function. It is an effective measure of the variability or compactness of a data set and represents the sum of the squared differences between each data point and the center point of the data set. Specifically, The calculation formula is:

[0115]

[0116] in, represents the number of data points in the cluster, represents the number of features, represents the jth eigenvalue of the i-th data point, Represents the j-th eigenvalue of the cluster center.

[0117] The cluster score function takes into account the size of the cluster, the number of labeled data points, and the compactness of the data points within the cluster to assign a score to each cluster. A higher score indicates that the cluster is more representative and is suitable for selection as a target cluster for sampling.

[0118] The second stage of sampling: find the most relevant data and outlier data based on the bce-rerank model. Using the bce-rerank model, find the data that is most relevant to all the data in the cluster and the outlier data in the highest-scoring cluster. First, calculate the similarity. For each data point , use the bce-rerank model to calculate the similarity scores between data The specific formula is:

[0119]

[0120] Then re-sort the data: sort the data by similarity score from high to low. The data with the highest similarity score is the most relevant data. The data with the lowest similarity score is the outlier data.

[0121] Form special case pairs: The most relevant data and the outlier data are combined into special case pairs, and a sampling process is completed to obtain a special case pair containing two sampled data.

[0122] Using the bce-rerank model can significantly improve retrieval performance. Through the above two-stage sampling strategy, the most informative tabular data can be efficiently screened, thereby improving the accuracy of error detection.

[0123] S23: The classifier module uses an inverse propensity score correction classifier to eliminate the impact of selection bias in the sampling process on the classifier.

[0124] Specifically, this embodiment proposes to introduce the concept of IPS score into the classifier module. By calculating the propensity score, constructing the weight matrix, and then applying the weight matrix to the standard cross entropy loss function of the binary classifier, a binary classifier based on the IPS score correction is obtained. The corrected binary classifier can reduce excessive dependence on popular cluster data while increasing learning of unpopular cluster data, thereby achieving more balanced and comprehensive error detection. In order to prove this more rigorously, this embodiment designs an expected loss function. Assume that the true label of each sample is and predicted probability are independent and identically distributed, and the observation probability of each sample is .

[0125] Under ideal balanced sampling conditions, the expectation of the standard cross entropy loss function for the binary classifier is:

[0126]

[0127] because and is independent and identically distributed, the expectation can be split into:

[0128]

[0129] When the sampling data has selection bias, the observation probability of each sample is not uniformly distributed. Assume that the sample The observation probability is , then the expected loss function of all data is:

[0130]

[0131] However, due to the observation probability is not uniform, and the number of samples actually observed may be biased. Therefore, the expected loss function of the entire data can be expressed as:

[0132]

[0133] In order to prove that the expected sum of all loss functions in the case of deviation is not equal to the expected sum of loss functions in the case of equilibrium, this example compares and :

[0134]

[0135]

[0136] Obviously, unless For all ,otherwise To eliminate the bias, this example uses the IPS (Importance Sampling) weight:

[0137]

[0138] The expected loss function of all data after IPS weighting is:

[0139]

[0140] After simplification, we get:

[0141]

[0142] This is exactly what you would expect from a standard cross entropy loss function. .

[0143] Through the above analysis, this embodiment can draw the following conclusions: In the case of selection bias, the expected loss function of all data is Compared with the standard cross entropy loss function Different. Use the expected loss function of all data weighted by IPS score After transformation, it is consistent with the standard cross entropy loss function. same.

[0144] In other words, by introducing appropriate weights (such as IPS weights), the influence of selection bias can be eliminated, and the weighted expected loss function can be made consistent with the standard expected loss function. This solves the sampling bias problem caused by the pursuit of high-information labeled instances in the error detection algorithm, and enables the weighted corrected binary classifier to learn each error pattern in a balanced manner.

[0145] S3: Concatenate the table data matrix and the table data label matrix to obtain the table data concatenation matrix C;

[0146] Specifically, the table data matrix X and the table data label matrix Y are concatenated, which can be expressed as the table data concatenation matrix C:

[0147] C ={ }

[0148] in, Indicates the value of the mth attribute column of the i-th data, Indicates the value of the annotation label corresponding to the m-th attribute column of the i-th data.

[0149] S4: Inputting the table data splicing matrix C into the complex table data error detection model for training to obtain a trained table data error detection model;

[0150] Specifically, the tabular data splicing matrix C is input into the complex tabular data error detection model. Through the sampling strategy based on the cosine function and decision tree model, the specified number of labeled tabular data items are sampled multiple times. Through multiple sampling training, the model can be better generalized in practical applications, and the overfitting of specific training samples can be reduced, thereby enhancing the robustness of the training model.

[0151] S5: Input the table data splicing matrix C into the trained complex table data error detection model to perform error detection and obtain the final error detection result;

[0152] Corresponding to the aforementioned embodiment of the complex table data error detection method, the present application also provides an embodiment of the complex table data error detection device.

[0153] Based on the above complex table data error detection method, this embodiment further specifically provides a corresponding execution device, referring to Figure 3 As shown, the device includes:

[0154] Acquisition module 1, used to obtain tabular data samples and construct tabular data matrix and tabular data label matrix;

[0155] Modeling module 2 is used to build a complex tabular data error detection model, including a grouping module based on distance correlation coefficient, a joint representation module based on the WordVec model and the Sentence-Transformer model, a special case pair sampling module based on the within-cluster sum of squares function and the BCE-Rerank model, and a classifier module based on inverse propensity score correction;

[0156] A splicing module 3 is used to splice the table data matrix and the table data label matrix to obtain a table data splicing matrix;

[0157] A training module 4 is used to input the table data splicing matrix into the complex table data error detection model for training to obtain a trained table data error detection model;

[0158] The error detection module 5 is used to input the table data matrix into the trained complex table data error detection model for error detection to obtain the final error detection result.

[0159] The specific manner in which the operations of each of the above modules are performed has been described in detail in the above embodiments of the method and will not be elaborated on here. At the same time, based on the apparatus, corresponding electronic devices and computer storage media are provided to cooperate with the execution of the above method, so they will not be repeated here.

[0160] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A complex table data error detection method, characterized by: The steps include: Step 1: Obtain tabular data samples and construct tabular data matrix and tabular data label matrix; Step 2: Build a complex table data error detection model. The error detection model includes: A grouping module, which logically groups attribute columns based on distance correlation coefficients and groups attribute columns that are most likely to have contextual semantic relationships into one group; Joint representation module, which uses sentence-transformer and wordvec to represent the data within the logical group, thereby strengthening the representation ability of the contextual logical relationship of the feature vectors in each group; Special case pair sampling module, which uses the intra-cluster square sum function and the BCE-rerank model to accurately locate special case data pairs in the same cluster, thereby screening out the most informative special case data pairs; The classifier module uses an inverse propensity score correction classifier to eliminate the influence of selection bias in the sampling process on the classifier; Step three: Input the tabular data samples into the trained complex tabular data error detection model, and obtain the error detection results through analysis and processing by the grouping module, joint representation module, special case sampling module and classifier module.

2. The complex table data error detection method according to claim 1, characterized in that: The specific steps of obtaining the table data sample in step 1 and constructing the table data matrix are as follows: Step 11: obtain a standardized table data sample from a financial website using an automated crawler tool or a database query interface; Step 1 and 2: Construct a table data matrix with n data and m attribute columns based on the multi-view missing face image data samples , No. Expressed as At the same time, a tabular data label matrix with k data and m attribute columns is constructed based on the multi-view missing face image label samples { }, Y Expressed as ; In the process of constructing the tabular data matrix, the error in the tabular data is defined as its corresponding ground truth value Any cell value that is different , and in The cell value is assigned to the positive class when , or when , is assigned to the negative class, then the cell value is considered to be correctly classified; if The cell value is assigned to the negative class when , or when , is assigned to the positive class, then the cell value is considered misclassified.

3. The complex table data error detection method according to claim 1 or 2, characterized in that: The analysis and processing steps of the grouping module in the complex table data error detection model in step 2 are as follows: Step 21: For two groups of samples and , calculate the Eubian distance matrix between them respectively: ； Step 22: Then, double-center the distance matrix by subtracting the mean of its row and column from each element and adding back the mean of the entire matrix: ； and, is the element in the i-th row and j-th column of the original distance matrix, is the element in the i-th row and j-th column of the original distance matrix b, 、 、 Respectively The row, column and overall averages of 、 、 Respectively The row, column and overall averages of Step 23: Based on the doubly centered distance matrix obtained in step 22, further calculate the distance covariance and distance variance: ； ； Step 24: Calculate the distance correlation coefficient based on the distance covariance and distance variance , the distance correlation coefficient , the value range is [0,1], 0 means complete independence, 1 means there is a deterministic relationship: ； Step 25: Using the dCor metric, the DS strategy is used for each dirty column. Select columns ,in Exceeding the threshold , forming a related column set ,Right now: 。 4. The complex table data error detection method according to claim 1 or 2, characterized in that: The analysis and processing steps of the joint characterization module in step 2 are as follows: First, Sentence-Transformer is used to obtain sentence-level representations of the data within each logical group. Then, WordVec is used to extract word-level features. The two are concatenated or weighted summed to form a richer feature vector. Based on the fused features, the self-attention mechanism is applied to assign different weights to different parts according to the context to further optimize the feature representation.

5. The complex table data error detection method according to claim 1 or 2, characterized in that: The analysis and processing steps for the sampling module in the special case of step 2 are as follows: First, the grouped data is converted into multiple sets of embedding vectors using the joint representation module, and then the embedding vectors are clustered using the DBSCAN algorithm to obtain If there are k labeled instances in a cluster, k / 2 samplings are performed. The sampling strategy is divided into two stages: the first stage is group sampling, and the second stage is special pair sampling to screen out the most informative special case data pairs.

6. The complex table data error detection method according to claim 1 or 2, characterized in that: The analysis and processing steps of the classifier module in step 2 are as follows: The IPS score concept is introduced into the classifier module. By calculating the propensity score and constructing the weight matrix, the weight matrix is then applied to the standard cross-entropy loss function of the binary classifier to obtain a binary classifier corrected based on the IPS score. The binary classifier corrected based on the IPS score is then used to eliminate the influence of the selection bias problem in the sampling process on the classifier.

7. The complex table data error detection method according to claim 1 or 2, characterized in that: In step 3, before inputting the tabular data sample into the trained complex tabular data error detection model, the tabular data matrix and the tabular data label matrix are first concatenated to obtain the tabular data concatenation matrix C, which is specifically as follows: C ={ }； in, Indicates the value of the mth attribute column of the i-th data, Indicates the value of the annotation label corresponding to the mth attribute column of the i-th data; Then, the tabular data splicing matrix C is input into the complex tabular data error detection model for training to obtain a trained tabular data error detection model.

8. A complex table data error detection device, configured to execute the complex table data error detection method according to any one of claims 1 to 7, characterized in that: include: Acquisition module (1), used to acquire tabular data samples and construct tabular data matrix and tabular data label matrix; A modeling module (2) is used to construct a complex table data error detection model, which includes a grouping module based on distance correlation coefficient, a joint representation module based on WordVec model and Sentence-transformer model, a special case pair sampling module based on intra-cluster square sum function and bce-rerank model, and a classifier module based on inverse propensity score correction; A splicing module (3) is used to splice the table data matrix and the table data label matrix to obtain a table data splicing matrix; A training module (4) is used to input the table data splicing matrix into the complex table data error detection model for training, thereby obtaining a trained table data error detection model; The error detection module (5) is used to input the table data matrix into the trained complex table data error detection model for error detection to obtain the final error detection result.

9. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the complex table data error detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed on a computer, enable the computer to execute the complex table data error detection method according to any one of claims 1 to 7.

Images