Text noise label correction method based on granular ball calculation

Abstract

The invention belongs to the field of natural language processing, and particularly relates to a text noise label correction method based on granular ball calculation. The method comprises the following steps: training a Bert model by adopting a manually labeled clean text data set to obtain a trained Bert model; encoding the noisy text data set by adopting the trained Bert model to obtain a feature vector; carrying out dimension reduction processing on the feature vector to obtain a feature vector after dimension reduction; enabling the feature vectors after dimension reduction of all the samples and the original labels to form initial pellets; adding the initial pellets into a queue to be processed, and setting a purity threshold value; correcting the labels of the texts in the queue to be processed according to the purity of the pellets and the purity threshold until the queue to be processed is empty, and obtaining a text data set with the corrected labels; according to the method, the noise proportion of the text data set is reduced, the robustness of the subsequent deep learning model is enhanced, and the model can realize more accurate text classification processing.

Description

Technical Field

[0001] The invention belongs to the field of natural language processing, and in particular relates to a text noise label correction method based on granular sphere calculation. Background Art

[0002] With the continuous improvement of computing power, the emergence of large-scale data sets has enabled deep neural networks to achieve great success in various fields. It is well known that the performance of deep neural networks depends largely on large, high-quality labeled data sets. However, due to the negligence of manual labeling or the inaccuracy of automatic labeling, the labeling process will inevitably introduce errors. Label noise will seriously affect the performance of deep neural networks because the model will learn incorrect associations and patterns. If the noise samples reach a certain proportion, the impact on the model will eventually be devastating. Label correction is a natural language processing technology, which is a method of correcting the wrong labels of samples in a dataset to correct labels. As a subtask of data noise processing, the importance of noisy label correction is self-evident. Label correction based on granular spherical computing was born.

[0003] The mainstream methods for dealing with noisy labels can be roughly divided into two categories: data-level methods and model-level methods. (1) Data-level methods mainly involve data cleaning and preprocessing. These methods aim to mitigate the impact of noisy labels on model training by cleaning and correcting noisy labels using techniques such as manual review, automatic detection, or sample selection and reweighting. However, these methods usually rely on a lot of manual intervention, which is time-consuming and prone to subjectivity. (2) Model-level methods focus on enhancing the robustness of the model during training. These methods include designing robust loss functions, performing noise modeling, or adopting ensemble learning and semi-supervised learning techniques to mitigate the negative impact of noisy labels. However, these methods usually involve complex algorithm design and a lot of computing resources, which may be difficult to implement in practical applications or require major adjustments to the model. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a text noise label correction method based on granular sphere calculation, which includes:

[0005] S1: Use a manually annotated clean text dataset to train the Bert model to obtain a trained Bert model;

[0006] S2: Use the trained Bert model to encode the noisy text dataset to obtain the feature vector;

[0007] S3: Perform dimension reduction processing on the feature vector to obtain a feature vector after dimension reduction;

[0008] S4: The reduced-dimensional feature vectors and original labels of all samples form the initial sphere;

[0009] S5: Add the initial pellets to the queue to be processed and set the purity threshold;

[0010] S6: Take a ball from the queue to be processed, calculate the purity of the ball, and if the purity of the ball is less than the purity threshold, split the ball to obtain multiple sub-balls; otherwise, adjust the purity threshold and execute step S6 again;

[0011] S7: Determine the purity of each sub-ball. If the ball is greater than or equal to the purity threshold, correct the labels of all texts in the ball to the label with the largest number in the ball. Otherwise, add the sub-ball to the queue to be processed.

[0012] S8: Repeat steps S6-S7 until the queue to be processed is empty, and obtain a text dataset with corrected labels.

[0013] Preferably, the process of encoding a noisy text data set using a trained Bert model includes: inputting the noisy text data set into the trained Bert model, and extracting the hidden state of the last layer of the model to generate a feature vector.

[0014] Preferably, in step S3, a principal component analysis method is used to perform dimensionality reduction processing on the feature vector.

[0015] Preferably, the initial pellet is represented by:

[0016] GB={(x1,y1),(x2,y2),....,(x n ,y n )}

[0017] Among them, GB represents the initial ball, x n The feature vector representing the nth text, y n Represents the original label of the nth text.

[0018] Preferably, the formula for calculating the purity of the pellets is:

[0019]

[0020] Among them, P GB Indicates the purity of the pellet, x i Represents the feature vector of the i-th text, y i represents the original label of the i-th text, and |GB represents the number of texts in the sphere.

[0021] Preferably, the process of splitting the particle balls includes: clustering the particle balls using a K-means++ clustering algorithm to obtain k sub-particle balls.

[0022] Furthermore, the process of clustering the spheres using the K-means++ clustering algorithm includes:

[0023] Step 1: Randomly select a text in the sphere as the center point, and calculate the distance from the remaining texts in the sphere to the center point;

[0024] Step 2: Calculate the probability of each text being selected as the next center point based on the distance from the text to the center point;

[0025] Step 3: Calculate the cumulative probability of the text based on the probability of the text being selected as the next center point;

[0026] Step 4: Select the next center point based on the cumulative probability distribution of the text;

[0027] Step 5: Repeat steps 2 to 4 to obtain k cluster center points;

[0028] Step 6: Based on the k cluster center points, use the K-means clustering algorithm to obtain k sub-particle balls;

[0029] Step 7: Repeat steps 1 to 6 multiple times to obtain multiple particle-ball splitting results; calculate the intra-cluster sum of squares of each splitting result, and select the particle-ball splitting result with the smallest intra-cluster sum of squares as the final particle-ball splitting result, that is, to obtain the final k sub-particle balls.

[0030] Furthermore, k is the number of different labels in the sphere.

[0031] The beneficial effects of the present invention are as follows: the present invention efficiently utilizes the semantic information of the feature space of text samples, reduces the cost of manpower to correct noise data, effectively corrects the labels of all text samples, reduces the noise ratio of the text data set, enhances the robustness of subsequent deep learning models, and enables the model to achieve more accurate text classification processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of the text noise label correction method based on granular sphere calculation in the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The present invention proposes a text noise label correction method based on granular sphere calculation, such as Figure 1 As shown, the method includes:

[0035] S1: Use a manually annotated clean text dataset to train the Bert model to obtain a trained Bert model.

[0036] The training text data set can be obtained from the publicly available data set. Preprocess the text data: filter the expressions and special characters in the text data by matching the string with regular expressions to obtain the preprocessed text data; the text processed by the regular expression is the plain text that meets the requirements, i.e. the preprocessed text data.

[0037] The plain text dataset is used to train the Bert model. The Bert model can be a BERT-Base-uncase model, which consists of a 12-layer encoding network. The hidden state size of each layer is 768 and there are 12 attention heads. The trained Bert model will be used as the encoder of subsequent text datasets.

[0038] S2: Use the trained Bert model to encode the noisy text dataset to obtain the feature vector.

[0039] Each text sample is mapped to a high-dimensional feature vector, usually 768 dimensions, which represents the semantic information of the text. The feature vector is generated by inputting the text into the BERT model, extracting the hidden state of the last layer of the model, and generating a feature vector for subsequent processing. Assume that the input text x i , then the output feature vector after the BERT model is:

[0040] h i =BERT(x i )

[0041] Among them, h i It is a 768-dimensional feature vector, representing the contextual semantic information of the text.

[0042] S3: Perform dimension reduction processing on the feature vector to obtain a feature vector after dimension reduction.

[0043] Preferably, the principal component analysis (PCA) method is used to reduce the dimensionality of the feature vector. PCA projects high-dimensional data into a low-dimensional space by identifying the most significant features in the data. The processing process is as follows:

[0044] Normalize the feature matrix X:

[0045]

[0046] Among them, μ and σ are the mean and standard deviation of the feature matrix respectively.

[0047] Calculate the covariance matrix C:

[0048]

[0049] Where n represents the number of text samples.

[0050] Perform eigenvalue decomposition and select the first k principal components:

[0051] C=VΛV T

[0052] Among them, V represents the eigenvector matrix and Λ represents the diagonal eigenvalue matrix.

[0053] Project the feature matrix into a lower-dimensional space:

[0054] Y=X′V k

[0055] Among them, Y is the feature matrix after dimension reduction, V k are the eigenvectors corresponding to the first k largest eigenvalues.

[0056] S4: The reduced-dimensional feature vectors and original labels of all samples form the initial sphere.

[0057] A sphere is a collection of samples with similar characteristics, defined as:

[0058] GB={(x1,y1),(x2,y2),....,(x n ,y n )}

[0059] Among them, GB represents the initial ball, x n The feature vector representing the nth text, y n Represents the original label of the nth text.

[0060] S5: Add the initial pellets to the queue to be processed and set the purity threshold.

[0061] Create an empty queue as a waiting queue, add the initial pellet to the waiting queue, and set a purity threshold. The purity threshold is used to determine whether the pellet needs to be split (split). Preferably, the purity threshold is manually set, such as 0.95.

[0062] S6: Take a ball from the queue to be processed and calculate the purity of the ball. If the purity of the ball is less than the purity threshold, split the ball to obtain multiple sub-balls; otherwise, adjust the purity threshold and execute step S6 again.

[0063] The formula for calculating the purity of the pellets is:

[0064]

[0065] Among them, P GB Indicates the purity of the pellet, x i Represents the feature vector of the i-th text, y i represents the original label of the i-th text, and |GB represents the number of texts.

[0066] If the purity of the ball is less than the purity threshold, the ball is split; if the purity of the ball is greater than or equal to the purity threshold, the purity threshold is adjusted upward and step S6 is performed again. The process of splitting the ball includes:

[0067] For each sphere that does not meet the purity requirements, the K-means++ clustering algorithm is used to split it. The algorithm steps are:

[0068] Step 1: Randomly select a text in the sphere as the center point and calculate the distance from the remaining texts in the sphere to the center point.

[0069] First, randomly select the first center point c. This step does not involve any complex calculations. This random selection is to provide a starting point for K-means++. Different random selections may result in different initial centers and final results. For each remaining point x∈Q in the sphere i , Q i ={x1,x2,...,x n}, calculate the Euclidean distance from it to the nearest selected center point, the calculation formula is as follows:

[0070]

[0071] Among them, c j is the jth cluster center, and d is the dimension of the feature vector.

[0072] Step 2: Calculate the probability of each text being selected as the next center point based on the distance from the text to the center point.

[0073] According to the square of these distances d(x,c) 2 To calculate the probability of each point being selected as the next center point:

[0074]

[0075] in, It's Q i The sum of the squared distances between all samples in and the selected GB center c is used to normalize the probability so that its sum is 1.

[0076] Step 3: Calculate the cumulative probability of the text based on the probability of the text being selected as the next center point.

[0077] Based on the probability P(x i )Construct the cumulative probability distribution CDF(x i ), where CDF(x i ) is the selected point x i The cumulative probability of:

[0078]

[0079] Step 4: Select the next center point based on the cumulative probability distribution of the text.

[0080] Generate a random number r (satisfying 0≤r<1), and compare r with the cumulative probability distribution CDF(x i ) to find the CDF(x i-1 )≤r <CDF(x i ), select the corresponding sample point xi as the new cluster center.

[0081] Step 5: Repeat steps 2 to 4 to obtain k cluster center points.

[0082] Repeat the above steps (step 2 to step 4) until k cluster centers are selected. This selection method ensures that the center point of the new center is as far away from the existing center as possible, thus forming a good cluster distribution in different areas.

[0083] The number of sub-balls after splitting is k, where k is the number of different label types in the ball being split.

[0084] Step 6: Based on the k cluster center points, use the K-means clustering algorithm to obtain k sub-particle balls.

[0085] Step 7: The clustering quality can be evaluated by the intra-cluster sum of squares (Inertia), the formula is as follows:

[0086]

[0087] The smaller the Inertia value, the better the clustering effect and the higher the similarity between text samples; j Indicates that it belongs to c j A sample of this cluster, GB i represents the i-th ball.

[0088] Perform n_init splits in total, compare the intra-cluster sums of squares of these n_init splits, and select the split result with the smallest intra-cluster sum of squares as the final split result. Preferably, n_init is manually set, such as n_init=5.

[0089] S7: Determine the purity of each sub-ball. If the ball is greater than or equal to the purity threshold, correct the labels of all texts in the ball to the label with the largest number in the ball. Otherwise, add the sub-ball to the queue to be processed.

[0090] For sub-spheres whose purity is greater than or equal to the threshold, the labels of all its samples are corrected to the most common label in the sphere:

[0091]

[0092] If the purity does not reach the threshold, the pellet will be added back to the queue to be processed for further splitting and processing.

[0093] S8: Repeat steps S6-S7 until the queue to be processed is empty, and obtain a text dataset with corrected labels.

[0094] When the queue to be processed is empty, the labels of all spheres are corrected. The labels of all text data are finally corrected to the most common labels in the sphere. Training the model with the text dataset after the text noise label correction can improve the performance of the model and enable the model to achieve more accurate text classification processing.

[0095] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A text noise label correction method based on granular sphere calculation, characterized in that: include: S1: Use a manually annotated clean text dataset to train the Bert model to obtain a trained Bert model; S2: Use the trained Bert model to encode the noisy text dataset to obtain the feature vector; S3: Perform dimension reduction processing on the feature vector to obtain a feature vector after dimension reduction; S4: The reduced-dimensional feature vectors and original labels of all samples form the initial sphere; S5: Add the initial pellets to the queue to be processed and set the purity threshold; S6: Take a ball from the queue to be processed, calculate the purity of the ball, and if the purity of the ball is less than the purity threshold, split the ball to obtain multiple sub-balls; otherwise, adjust the purity threshold and execute step S6 again; S7: Determine the purity of each sub-ball. If the ball is greater than or equal to the purity threshold, correct the labels of all texts in the ball to the label with the largest number in the ball. Otherwise, add the sub-ball to the queue to be processed. S8: Repeat steps S6-S7 until the queue to be processed is empty, and obtain a text dataset with corrected labels.

2. According to the method for correcting text noise labels based on granular sphere calculation according to claim 1, it is characterized in that: The process of encoding a noisy text dataset using a trained Bert model includes: inputting the noisy text dataset into the trained Bert model, and extracting the hidden state of the last layer of the model to generate a feature vector.

3. The text noise label correction method based on granular sphere calculation according to claim 1 is characterized in that: In step S3, the principal component analysis method is used to perform dimension reduction processing on the feature vector.

4. The text noise label correction method based on granular sphere calculation according to claim 1 is characterized in that: The initial sphere is represented by: GB={(x1,y1),(x2,y2),....,(x n ,and n )} Among them, GB represents the initial ball, x n The feature vector representing the nth text, y n Represents the original label of the nth text.

5. The text noise label correction method based on granular sphere calculation according to claim 1 is characterized in that: The formula for calculating the purity of the pellets is: Among them, P GB Indicates the purity of the pellet, x i Represents the feature vector of the i-th text, y i represents the original label of the i-th text, and |GB| represents the number of texts in the ball.

6. The text noise label correction method based on granular sphere calculation according to claim 1 is characterized in that: The process of splitting the particle ball includes: clustering the particle ball using the K-means++ clustering algorithm to obtain k sub-particle balls.

7. The text noise label correction method based on granular sphere calculation according to claim 6 is characterized in that: The process of clustering the spheres using the K-means++ clustering algorithm includes: Step 1: Randomly select a text in the sphere as the center point, and calculate the distance from the remaining texts in the sphere to the center point; Step 2: Calculate the probability of each text being selected as the next center point based on the distance from the text to the center point; Step 3: Calculate the cumulative probability of the text based on the probability of the text being selected as the next center point; Step 4: Select the next center point based on the cumulative probability distribution of the text; Step 5: Repeat steps 2 to 4 to obtain k cluster center points; Step 6: Based on the k cluster center points, use the K-means clustering algorithm to obtain k sub-particle balls; Step 7: Repeat steps 1 to 6 multiple times to obtain multiple particle-ball splitting results; calculate the intra-cluster sum of squares of each splitting result, and select the particle-ball splitting result with the smallest intra-cluster sum of squares as the final particle-ball splitting result, that is, to obtain the final k sub-particle balls.

8. The text noise label correction method based on granular sphere calculation according to claim 6 is characterized in that: k is the number of different labels in the sphere.

Images