Table data noise identification and correction method based on Stein scoring

Abstract

The invention relates to the technical field of label noise of table data, a distance measurement-based method and a neural network-based method are currently common label noise screening methods, and the methods are difficult to distinguish wrongly labeled samples and difficult samples with fuzzy categories near a decision boundary; according to the table data noise recognition and correction method based on Stein scoring, the difference of the attribution degree between sample and data feature distribution and label export distribution is analyzed, and the logarithmic probability density gradient of a sample is estimated through a diffusion model; moving the sample along a gradient represented by a scoring function by using a gradient method until the sample is converged to a centroid, forming a moving track moving from an original position of a sample space to a high-density centroid of data of a domain, and calculating the data feature distribution by comparing the differences of the directions and lengths of the moving track of the sample in the data feature distribution and the distribution exported by the label. Samples of potential tag errors in the tabular data are identified and tags thereof are corrected.

Description

Technical Field

[0001] This invention relates to the field of label noise technology for tabular data, and more specifically, to a method for identifying and correcting tabular data noise based on Stein scoring. Background Technology

[0002] Distance-based and neural network-based methods are commonly used for label noise screening. For easily distinguishable noise with significant feature differences from clean samples, distance-based methods use distance measurements to calculate the nearest neighbors of each sample. By evaluating the label consistency between a sample and its nearest neighbors, potential easily distinguishable noise can be identified quickly and effectively, such as the Edit Nearest Neighbor (ENN) algorithm and the Particle Ball method. However, for hard-to-distinguish noise located in overlapping class regions or class boundaries, the inherent class ambiguity leads to a significant decrease in the effectiveness of such noise identification methods. Neural network-based methods typically have two different frameworks. The first uses supervised learning to model the relationship between features and labels. The second uses unsupervised learning, leveraging representation learning to extract highly discriminative feature representations to improve the accuracy of label noise detection. However, hard-to-distinguish noise and hard clean samples (samples that are inherently difficult to fit) exhibit similar loss and confidence patterns. Therefore, using only supervised or unsupervised learning frameworks is insufficient to effectively distinguish hard-to-distinguish noise from hard clean samples. Furthermore, hard-to-distinguish noise simultaneously induces pattern distortion and distribution shift, significantly weakening the model's ability to fit clean samples. Therefore, this invention proposes a new framework to address this problem. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method for identifying and correcting tabular data noise based on Stein scoring by analyzing the differences in the degree of attribution between sample and data feature distributions and distributions derived from labels. This invention is mainly used to identify samples with potential label errors in a dataset, and achieves better noise identification results by dividing label noise into easily distinguishable noise and difficult-to-distinguish noise.

[0004] To achieve the above objectives, the present invention provides the following technical solution: A method for identifying and correcting tabular data noise based on Stein scoring is proposed. In both the data feature distribution and the label-derived distribution, a diffusion model is used to estimate the log probability density gradient of the samples. The gradient method is then used to move the samples along the gradient represented by the scoring function until they converge to the centroid, forming a trajectory from the original position in the sample space to the high-density centroid of the corresponding distribution. The method analyzes inter-domain differences by comparing the direction and length of the sample trajectories in the two distributions and labels noisy samples. Specifically, the method includes the following steps: Step 1. Training the diffusion model: Train an unsupervised diffusion model without label information and a supervised diffusion model with label information on data with noisy labels. The supervised and unsupervised diffusion models learn the distribution derived from the labels and the distribution of data features, respectively. The supervised diffusion model uses the gradient of the classifier to correct the scoring function so that the trajectory of the sample can converge to the region with a higher probability density of the target class. Step 2. Calculate the sample movement trajectory: After all diffusion models have completed the training in Step 1, they obtain the time... Score function for time The log probability density gradient estimated by the scoring function guides each sample to move in the sample space, forming a movement trajectory from the sample to the high-density centroid of the data. Using unsupervised diffusion model and supervised diffusion model, two movement trajectories are derived for each sample: one from the original position in the sample space to the high-density centroid of the data feature distribution, and the other to the high-density centroid of the distribution derived from the label. Step 3. Noise Identification: Based on the two movement trajectories of the acquired sample, using... Identification methods for filtering difficult-to-distinguish noise The nearest neighbor identification method is applied to help filter out easily distinguishable noise that was missed. The two types of noise are combined into a noise set. The remaining samples constitute a clean dataset. ; Step 4. Label Correction: Two classifiers with different structures are used to re-label the noisy samples identified in Step 3 to improve the accuracy of noise label correction. The corrected samples are then incorporated into the dataset. To obtain a dataset that meets the sample size requirements and has accurate labels.

[0005] Furthermore, in step 1, for the unsupervised diffusion model, the training objective function is as follows: For supervised diffusion models, the label information of samples is directly embedded into the neural network, enabling the supervised diffusion model to learn the joint information of features and labels. The training objective function is as follows: in, and These represent the network parameters of the unsupervised and supervised diffusion models, respectively. This represents the number of time steps, where T represents the maximum number of time steps. The weighting function controls the importance of different time steps. Represents time step The corresponding noise level at that time This represents the sample after adding noise at time step t. The labels corresponding to the samples. Representative model.

[0006] Furthermore, in step 2, the trajectory calculation of the unsupervised diffusion model involves using the gradient value estimated by the diffusion model for the actual data distribution. This moves the sample to a location with higher probability density, ensuring the sample... Along every step The direction is updated, eventually moving to the region where the probability density is maximized: in, Representative sample The first in the movement path One point, For the movement step size, Representative at hour The Stein score corresponding to the position. Represents the original data; after N iterations, it generates a trajectory pointing towards the centroid of the data feature distribution. ; Trajectory calculation for supervised diffusion models: The scoring function of the supervised diffusion model is adjusted using a classifier by introducing additional gradients during the Langevin dynamics iteration process. This makes it easier for samples to move to the high-density region corresponding to the target label. The adjusted Stein scoring function is expressed as: in, It comes from the supervised diffusion model with parameters of The scoring function, Then the parameter is The gradient of the classifier; to ensure that the movement path of the supervised diffusion model converges to the centroid corresponding to the label, the following formula is used for iterative updates: The generated products after iteration tend to be based on labels. Derive the movement path of the centroid corresponding to the distribution. .

[0007] Furthermore, in step 3, In the identification method, the distance of the movement path is measured using the second normal form of the scores corresponding to each point on the path. The difference between the distance of the movement trajectory of the unsupervised diffusion model and the movement trajectory of the supervised diffusion model is calculated. in The score in the vector field of the supervised diffusion model guided by the classifier represents the score value. This represents the scoring function in the vector field of the unsupervised diffusion model. Representative model, Representing the first The unsupervised and supervised movement paths calculated from each sample are used to incorporate the directional information of the movement paths from the unsupervised and supervised diffusion models, and the cosine distance between the two is added. In, it also serves as a measurement criterion: in, Representing the first The endpoints of unsupervised and supervised movement paths for each sample. Represents the cosine function; ultimately The calculation formula has been updated as follows: in, Controlling the intensity of the cosine distance, Samples exceeding the threshold are categorized as noise samples.

[0008] Furthermore, the noise sample set for the identification method is: in, Calculate the threshold size corresponding to sample set D.

[0009] Furthermore, in step 4, a clean dataset is used. Training classifier and classifier Calculate the predicted label for each noise sample. and prediction labels To avoid erroneous predictions damaging data quality, a consistency principle is adopted, and data is only considered consistent if the predicted labels of the two classifiers are identical. and prediction labels Sample label correction is only performed when the results are consistent; otherwise, the sample is removed. in, The final revised label, This indicates that the sample has been deleted.

[0010] In summary, the invention has the following beneficial effects: The method of this invention optimizes parameters for different datasets. By comparing the directions and lengths of two moving trajectories of samples and labeling noisy samples in the tabular data, the SS-NIC of this invention shows a small difference in average ACC (Accuracy) compared to GCE and SCE in downstream tasks when the noise ratio is 0.05, but achieves the best performance on 17 datasets. From the perspective of different data characteristics, SS-NIC maintains stable performance in high-dimensional and imbalanced datasets. As the noise ratio in the training data increases to 0.1, the performance advantage of SS-NIC becomes even more prominent. At this point, the SS-NIC of this invention ranks first with the highest average accuracy, significantly higher than all control methods, confirming the adaptability of this invention in scenarios with increased noise. In high-noise scenarios with noise ratios of 0.2 and 0.4, the performance differences among the control methods further widen, while the advantage of SS-NIC shows a stable and continuously expanding trend. In terms of average accuracy, SS-NIC ranked first with a significant advantage in both high noise ratio scenarios, indicating that SS-NIC performs particularly well in high noise ratio scenarios. When applied to noise recognition of tabular datasets, it can adapt to complex data distributions and demonstrates strong noise robustness. Attached Figure Description

[0011] Figure 1 This is a flowchart of the present invention; Figure 2 This is used to derive distribution plots and data feature distribution plots from the labels. Black symbols in the plot represent labels. The sample, This represents a clean sample. Represents easily distinguishable noise. The gray mark represents indistinguishable noise; the gray mark represents the label. The samples. ▲ and ★ represent the centroids of the label-derived and data feature distributions formed by samples with the same label and samples with similar features, respectively. Detailed Implementation

[0012] The present invention will now be described in further detail with reference to the accompanying drawings.

[0013] It should be noted that, for ease of description, the descriptions of direction in the following text are consistent with the directions in the accompanying drawings, but they do not limit the structure of the present invention.

[0014] like Figures 1-2As shown, this invention discloses a method for identifying and correcting tabular data noise based on Stein scoring. In the same dataset, the data feature distribution formed by samples with similar features differs from the label-derived distribution formed by samples with the same label. Noisey samples amplify this difference. High-density regions in the dataset reflect the inherent clustering characteristics and structural basis of the data. The distance between a sample and the centroid of a high-density region indicates the degree to which the sample belongs to that region; this distance reflects the correspondence between labels and features. Difficult-to-distinguish noise may exhibit different belonging relationships to high-density regions in the two distributions, with varying degrees of belonging; while clean samples are usually located within the same high-density region and have similar degrees of belonging. Therefore, a diffusion model is used to estimate the log probability density gradient of the samples. The gradient method is used to move the samples along the gradient represented by the scoring function until they converge to the centroid, forming a trajectory from the original position in the sample space to the high-density centroid of the data in the corresponding domain. By comparing the direction and length of the sample movement trajectories in the two distributions, the inter-domain differences are analyzed and noise samples are labeled. Specifically, the method includes the following steps: Step 1. Training the diffusion model: Train an unsupervised diffusion model without label information and a supervised diffusion model with label information on data with noisy labels. The supervised and unsupervised diffusion models learn the distribution derived from the labels and the distribution of data features, respectively. The supervised diffusion model uses the gradient of the classifier to correct the scoring function so that the trajectory of the sample can converge to the region with a higher probability density of the target class.

[0015] For the unsupervised diffusion model, the training objective function is as follows: (1) For supervised diffusion models, the label information of samples is directly embedded into the neural network, enabling the supervised diffusion model to learn the joint information of features and labels. The training objective function is as follows: (2) in, and Let represent the network parameters of the unsupervised and supervised diffusion models, respectively; t represents the time step; and T represents the maximum time step. The weighting function controls the importance of different time steps. This represents the noise level at time step t. y represents the sample after adding noise at time step t, and y is the label of the sample. Representative model.

[0016] Step 2. Calculate the sample movement trajectory: After all diffusion models have completed the training in Step 1, the scoring function at time t=0 is obtained. The log probability density gradient estimated by the scoring function guides each sample to move in the sample space, forming a movement trajectory from the sample to the high-density centroid of the data. Using unsupervised diffusion model and supervised diffusion model, two movement trajectories are derived for each sample: one from the original position in the sample space to the high-density centroid of the data feature distribution, and the other to the high-density centroid of the distribution derived from the label.

[0017] Calculating the trajectory of an unsupervised diffusion model: using the gradient value estimated from the real data distribution by the diffusion model. Make the sample Move to a location with higher probability density to ensure the sample Along every step The direction is updated, eventually moving to the region where the probability density is maximized: (3) in, Representative sample The first in the movement path One point, For the movement step size, Representative at hour The Stein score corresponding to the position. Represents the original data; after N iterations, it generates a trajectory pointing towards the centroid of the data feature distribution. According to Theorem 1, it can be guaranteed that the data moves towards the region with higher data density at each step until convergence. Therefore, the iteration is performed according to Formula (3). During the iteration process, a movement trajectory is generated for each sample, starting from the initial point and ending at the centroid corresponding to the data feature distribution.

[0018] Theorem 1: Hypothesis Function It satisfies the L-Lipschitz condition and is a convex function; formula (3) can ensure that That is, each iteration moves the sample to a region with higher data density until convergence.

[0019] Proof: exist Taylor expansion yields: (4) From equation (3), we get: (5) Substituting equation (5) into equation (4), we get: (6) in Step size, ,and It is positive semidefinite, therefore The term is nonnegative, therefore it is true if and only if When it is a zero vector Therefore, when Sometimes, ,Right now This proves that each update will move the sample to the position where... In higher regions, the data reaches its high-density centroid upon iteration convergence.

[0020] Trajectory calculation for supervised diffusion models: The scoring function of the supervised diffusion model is adjusted using a classifier by introducing additional gradients during the Langevin dynamics iteration process. This makes it easier for samples to converge to the high-density region corresponding to the target label. The adjusted Stein scoring function is expressed as: (7) in, It comes from the supervised diffusion model with parameters of The scoring function, Then the parameter is The gradient of the classifier, the weights and biases of each neuron in the neural network are obtained through the loss function; to ensure that the movement path of the supervised diffusion model converges to the centroid corresponding to the label, the following formula is used for iterative updates: (8) Similar to the unsupervised path generation process, samples move along the scoring function of the supervised diffusion model. Because the classifier's gradient biases the scoring function towards the high-density regions corresponding to specific labels, the iteration generates a path that tends towards the centroid corresponding to label y. .

[0021] Step 3. Noise Identification: After obtaining the unsupervised and supervised movement trajectories of the samples, noise identification is performed using the trajectory information. Based on the two movement trajectories of the acquired samples, noise is identified using... Identification methods for filtering difficult-to-distinguish noise The nearest neighbor identification method is applied to help filter out easily distinguishable noise that was missed. The two types of noise are combined into a noise set. The remaining samples constitute a clean dataset. By using the centroid of the convergence of feature-similar samples in the movement trajectory as input, nearest neighbor filtering is performed to further reduce noise that may be missed.

[0022] Difficult-to-distinguish noise is usually located in the intersection region of two clusters, and its trajectory differs significantly between supervised and unsupervised diffusion models. In the identification method, the distance of the path is measured using the second normal form of the scores corresponding to each point on the path. The difference between the distance of the unsupervised diffusion model's trajectory and the supervised diffusion model's trajectory is then calculated. in, The score in the vector field of the supervised diffusion model guided by the classifier represents the score value. This represents the scoring function in the vector field of the unsupervised diffusion model. Representative model, Representing the first The unsupervised and supervised movement paths calculated from each sample are used to incorporate the direction information of the movement paths from both the unsupervised and supervised diffusion models. The cosine distance between the movement path and the direction information is then added to the calculation. In, it also serves as a measurement criterion: (10) The cosine distance is the cosine distance between two vectors formed by two points and the origin. The resulting value measures the similarity of the directions of the two vectors, quantifying directional information into distance.

[0023] final The calculation formula has been updated as follows: (11) in Controlling the intensity of the cosine distance, Samples whose values ​​exceed the threshold are categorized as noise samples.

[0024] The noise sample set for the identification method is: (12) in, Calculate the threshold value corresponding to the sample set D, where sample set D is the entire training dataset. The set that constitutes the composition.

[0025] This invention classifies samples with large scoring modulus and inconsistent movement direction as difficult-to-distinguish noise. By introducing movement paths and their scoring information, and combining supervised and unsupervised movement characteristics, it has the ability to efficiently identify indistinguishable noise.

[0026] Step 4. Label Correction: Two classifiers with different structures are used to re-label the noisy samples identified in Step 3, improving the accuracy of noise label correction. In this embodiment, one classifier uses a random forest, and the other uses a multilayer perceptron; the corrected samples are then incorporated into the dataset. To obtain a dataset that meets the required sample size and has accurate sample labels. Specifically, use a clean dataset. Training classifier and classifier Calculate the predicted label for each noise sample. and prediction labels To avoid erroneous predictions damaging data quality, a consistency principle is adopted, and data is only considered consistent if the predicted labels of the two classifiers are identical. and prediction labels Sample label correction is only performed when the results are consistent; otherwise, the sample is removed. (13) in, The final revised label, This indicates that the sample has been deleted.

[0027] After the above steps, the corrected dataset will be merged into the clean subset: ,in It is a collection of noisy samples with corrected labels. Ultimately, we obtain a dataset that guarantees the number of samples while maintaining more accurate labels.

[0028] The method proposed in this invention is applicable to classification problems with numerical features. To verify the effectiveness of this invention, 26 commonly used datasets from the UCI and KEEL databases were used. All features in these datasets are numerical, and the labels are categorical variables. To ensure the comparability of experimental results, a detailed description of each dataset is shown in Table 1, displaying the sample size, number of features, number of labels, and label distribution. Features in each dataset were normalized using Min-Max, labels were one-hot encoded (unless otherwise specified), and random noise was used. The amount of material is evenly distributed across all classes in the dataset.

[0029] Table 1 Data Information To fairly verify the effectiveness of the SS-NIC method, this invention uses 10 baseline methods and conducts experiments on the above 26 datasets. The baseline methods are described below: CWD (Common Risk Digestion): This method constructs an unbiased risk estimator by estimating the unbiased centroid classify the classifier, making it robust to noise. The CWD method requires the noise ratio as prior knowledge; in the experiment, the true noise ratio was directly used. enter.

[0030] ENN: Based on KNN, it obtains a clean dataset by removing samples whose labels are inconsistent with the labels of each sample's k nearest neighbors.

[0031] GCE: Combines cross-entropy loss (CE) and mean absolute error (MAE) loss, leveraging symmetry to improve robustness to noise while maintaining fast convergence characteristics similar to CE. Smoothing factor. use .

[0032] f_div: It is proven that training a classifier with appropriate f-divergence is noise robust, and a set of noise robust f-divergences is collected.

[0033] SCE: Combines cross-entropy loss (CE) and inverse cross-entropy loss (RCE) to improve the classifier's robustness to noise through the symmetry of the loss function. The adjustment parameters in the experiment... , From respectively and Perform a grid parameter search.

[0034] SR: When the output of a fixed neural network is a one-hot vector, any loss function is robust to noise. Therefore, the SR method employs a sharpening function and Norm regularization makes the classifier's output closer to one-hot encoding. In the experiments, sharpening parameters were used. Regularization coefficient from A grid search is performed.

[0035] LW: This measure of a model's predictive stability during training is determined by the number of samples with inconsistent predicted labels across consecutive epochs. Based on this, it tracks trends in predictive stability and uses the first local extremum as an early stopping point for model training.

[0036] GMM-CT: This method uses Gaussian Mixture Model (GMM)

[43] to model the sample loss and selects samples with lower loss as clean samples. In addition, it will continuously track the trend of sample confidence changes and identify samples with higher loss but continuously increasing confidence as hard samples.

[0037] mCRF: This method builds a random forest model on noisy data. By counting the number of leaf nodes whose labels match those of their ancestor nodes, samples exceeding a threshold are marked as noise samples. The most suitable threshold for training is then selected through cross-validation or a 2-means algorithm.

[0038] mRD: Calculates the relative density of samples based on the k-nearest neighbor method. It calculates the ratio of the sum of the k-nearest neighbor distances of a sample to its k-nearest neighbors with the same label to the sum of the k-nearest neighbor distances of samples with different labels. When the relative density is greater than a preset parameter, the sample is considered a noise sample.

[0039] Among them, CWD, f_div, GCE, SCE, SR, and LW methods are used to train noisy robust classifiers. These classifiers all use a 3-layer MLP, and the number of neurons in the hidden layer is determined by r. Sure, To determine the dimensions of the input data, the ReLU activation function is used uniformly. ENN, GMM-CT, and mRD methods are used to filter out noisy samples in the data, and the model settings follow the recommended configurations in the literature.

[0040] This invention uses Bayesian optimization to optimize parameters for different datasets. In the first stage, the sample shifting step... Classifier weights In the second stage, the intensity of the cosine distance quantile distance setting The number of nearest neighbors is set to threshold The experiment used the PyTorch 1.11 framework, with a 24-core Intel Xeon Gold 5318Y CPU with a clock speed of 2.10GHz, 256GB of memory, and two NVIDIA Tesla A800 GPUs with 160GB of video memory and CUDA version 11.5.

[0041] In methods like CWD, f_div, GCE, SCE, and SR, which design noise-robust loss at the algorithmic level, the classifier is trained on a noisy training set, and its performance on noise-free data is tested. For data-level methods like ENN, mCRF, and mRD, this embodiment first uses a noise filtering method to obtain a relatively clean dataset, then trains a new classifier on the relatively clean dataset, and evaluates the classifier's classification ability on a test set, testing the accuracy (ACC) and F1 score of different methods in downstream tasks. To ensure the comparability of the two types of methods, this invention uses the same neural network structure as the noise-robust methods in the downstream tasks based on the noise filtering method. All methods in the experiments use ten-fold cross-validation; in each fold, all methods use the same input data, and the results shown are the mean of the ten-fold experimental results.

[0042] First, we analyze the performance of each method in terms of classification accuracy. Tables 2 to 5 show the noise ratio. The table shows the classification task accuracy results at 0.05, 0.1, 0.2, and 0.4, respectively. A value of '-' indicates that the method classifies all data in the dataset as noise, resulting in no samples available for downstream tasks. The last two rows of the table show the mean accuracy and mean accuracy ranking of each method across 26 datasets. The NAN column represents the performance of the method on downstream tasks with unprocessed noisy data. Bold text in the table indicates the best result achieved on a specific dataset.

[0043] Table 2 shows that on 26 benchmark datasets, when the label noise rate is 0.05, the traditional robust loss functions—Generalized Cross-Entropy (GCE) and Symmetric Cross-Entropy (SCE)—achieved high average classification accuracies (GCE = 0.813, SCE = 0.815). They even performed exceptionally well on some datasets (e.g., 0.977 accuracy on the transplant dataset) and significantly outperformed most other methods on some challenging tasks (e.g., 0.728 for GCE and 0.778 for SCE on the vowel dataset). These results validate their ability to learn reliable label information in mildly noisy environments. In contrast, the proposed SS-NIC method has an average accuracy of 0.811, only 0.002 lower than GCE and 0.004 lower than SCE, yet it ranks first on 18 of the 26 datasets. For example, on the hepatitis dataset, SS-NIC achieves an accuracy of 0.938, far exceeding SCE's 0.850 and GCE's 0.838. Furthermore, SS-NIC maintains stable performance under various data characteristics, such as high-dimensional scenes (optdigits, 0.983) and imbalanced class tasks (spectfheart, 0.809). In terms of average ranking (lower values ​​are better), SS-NIC leads with a ranking of 2.327, superior to SCE's 4.50 and GCE's 4.75, highlighting its overall advantage in low-noise environments. In contrast, simpler sample selection methods—such as EditedNearest Neighbors (ENN, 0.776) and mRD (0.773)—have significantly lower average accuracy than loss-based algorithms. This indicates that blindly deleting noisy samples may inadvertently delete useful samples, thus weakening classification performance. In Tables 3, 4, and 5, as the noise ratio gradually increases, the effectiveness of the SS-NIC method becomes increasingly prominent. Even at high noise ratios, the SS-NIC method maintains a high average ACC, while other methods show a significant decrease in average ACC with increasing noise ratios. Overall, the SS-NIC method outperforms the baseline method in ACC for downstream tasks.

[0044] Table 2 Accuracy of downstream classification tasks Table 3 Accuracy of downstream classification tasks Table 4 Accuracy of downstream classification tasks Table 5 Accuracy of downstream classification tasks To further evaluate the robustness of each method to noise, this invention also compares the F1 score on downstream classification tasks. Table 6 shows the ranking of each method's F1 score at different noise ratios, with values ​​representing the average of 26 data points. Bold text in the table indicates the highest F1 score among all methods at the same noise ratio. Similar to the ACC metric, the SS-NIC method leads across all noise ratios, demonstrating excellent performance on downstream tasks.

[0045] Table 6: F1 ranking of downstream classification tasks: The above analysis compares the various methods using the ACC and F1 metrics for downstream classification tasks. In general, the SS-NIC method provides better robustness to noisy samples, performs better in downstream tasks, has good generalization ability, and maintains a stable average ACC even under high noise ratios in classification tasks, indicating that the SS-NIC method has strong noise robustness and can handle various noise conditions.

[0046] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for noise identification and correction of tabular data based on Stein scoring, characterized in that: In both the data feature distribution and the label-derived distribution, the log probability density gradient of the samples is estimated using a diffusion model. The gradient method is then used to move the samples along the gradient represented by the scoring function until they converge to the centroid, forming a trajectory from the original position in the sample space to the high-density centroid of the corresponding distribution. The differences between the two distributions are analyzed by comparing the direction and length of the sample trajectories, and noisy samples are labeled. Specifically, the steps include: Step 1. Training the diffusion model: Train an unsupervised diffusion model without label information and a supervised diffusion model with label information on data with noisy labels. The supervised and unsupervised diffusion models learn the distribution derived from the labels and the distribution of data features, respectively. The supervised diffusion model uses the gradient of the classifier to correct the scoring function so that the trajectory of the sample can converge to the region with a higher probability density of the target class. Step 2. Calculate the sample movement trajectory: After all diffusion models have completed the training in Step 1, they obtain the time... Score function for time The log probability density gradient estimated by the scoring function guides each sample to move in the sample space, forming a movement trajectory from the sample to the high-density centroid of the data. Using unsupervised diffusion model and supervised diffusion model, two movement trajectories are derived for each sample: one from the original position in the sample space to the high-density centroid of the data feature distribution, and the other to the high-density centroid of the distribution derived from the label. Step 3. Noise Identification: Based on the two movement trajectories of the acquired sample, using... Identification methods for filtering difficult-to-distinguish noise The nearest neighbor identification method is applied to help filter out easily distinguishable noise that was missed during the selection process. The two types of noise are combined into a noise set. The remaining samples constitute a clean dataset. ; Step 4. Label Correction: Two classifiers with different structures are used to re-label the noisy samples identified in Step 3 to improve the accuracy of noise label correction. The corrected samples are then incorporated into the dataset. To obtain a dataset that meets the sample size requirements and has accurate labels.

2. The method for identifying and correcting tabular data noise based on Stein scoring according to claim 1, characterized in that: In step 1, for the unsupervised diffusion model, the training objective function is as follows: For supervised diffusion models, the label information of samples is directly embedded into the neural network, enabling the supervised diffusion model to learn the joint information of features and labels. The training objective function is as follows: in, and These represent the network parameters of the unsupervised and supervised diffusion models, respectively. This represents the number of time steps, where T represents the maximum number of time steps. The weighting function controls the importance of different time steps. Represents time step The corresponding noise level at that time This represents the sample after adding noise at time step t. The labels corresponding to the samples. Representative model.

3. The method for identifying and correcting tabular data noise based on Stein scoring according to claim 1, characterized in that: In step 2, the trajectory calculation of the unsupervised diffusion model involves using the gradient value estimated by the diffusion model based on the distribution of the real data. This moves the sample to a location with higher probability density, ensuring the sample... Along every step The direction is updated, eventually moving to the region where the probability density is maximized: in, Representative sample The first in the movement path One point, For the movement step size, Representative at hour The Stein score corresponding to the position. Represents the original data; after N iterations, it generates a trajectory pointing towards the centroid of the data feature distribution. ; Trajectory calculation for supervised diffusion models: The scoring function of the supervised diffusion model is adjusted using a classifier by introducing additional gradients during the Langevin dynamics iteration process. This makes it easier for samples to move to the high-density region corresponding to the target label. The adjusted Stein scoring function is expressed as: in, It comes from the supervised diffusion model with parameters of The scoring function, Then the parameter is The gradient of the classifier; to ensure that the movement path of the supervised diffusion model converges to the centroid corresponding to the label, the following formula is used for iterative updates: The generated products after iteration tend to be based on labels. Derive the movement path of the centroid corresponding to the distribution. .

4. The method for identifying and correcting tabular data noise based on Stein scoring according to claim 1, characterized in that: In step 3 In the identification method, the distance of the movement path is measured using the second normal form of the scores corresponding to each point on the path. The difference between the distance of the movement trajectory of the unsupervised diffusion model and the movement trajectory of the supervised diffusion model is calculated. in The score in the vector field of the supervised diffusion model guided by the classifier represents the score value. This represents the scoring function in the vector field of the unsupervised diffusion model. Representative model, Representing the first The unsupervised and supervised movement paths calculated from each sample are used to incorporate the directional information of the movement paths from the unsupervised and supervised diffusion models, and the cosine distance between the two is added. In, it also serves as a measurement criterion: in, Representing the first The endpoints of unsupervised and supervised movement paths for each sample. Represents the cosine function; ultimately The calculation formula has been updated as follows: in, Controlling the intensity of the cosine distance, Samples exceeding the threshold are categorized as noise samples.

5. The method for identifying and correcting tabular data noise based on Stein scoring according to claim 4, characterized in that: The noise sample set for the identification method is: in, Calculate the threshold size corresponding to sample set D.

6. The method for identifying and correcting tabular data noise based on Stein scoring according to claim 1, characterized in that: In step 4, a clean dataset is used. Training classifier and classifier Calculate the predicted label for each noise sample. and prediction labels To avoid erroneous predictions damaging data quality, a consistency principle is adopted, and data is only considered consistent if the predicted labels of the two classifiers are identical. and prediction labels Sample label correction is only performed when the results are consistent; otherwise, the sample is removed. in, The final revised label, This indicates that the sample has been deleted.

Images