BERT text classification method fusing Focal Loss and Label Smoothing
By introducing Focal Loss and Label Smoothing into the BERT model, the categories imbalance and overfitting problems in text classification are solved, the classification accuracy of the model is improved, and efficient and accurate text classification is achieved.
Patent Information
- Application Number
- CN202510348031.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-24
- Publication Date
- 2025-08-12
AI Technical Summary
The existing BERT model has category imbalance and overfitting problems in text classification, resulting in a decrease in classification accuracy.
The Focal Loss function is used to enhance the attention to difficult-to-classify samples, and smooth the label distribution through Label Smoothing and trained in combination with BERT model.
It improves the classification accuracy of the model, solves the problems of category imbalance and overfitting, and improves the accuracy and efficiency of text classification.
Smart Images
Figure CN120470119A_ABST
Abstract
Description
Technical Field
[0001] The present invention is applied to the fields of natural language processing and deep learning, and specifically relates to a BERT text classification method that integrates Focal Loss and Label Smoothing. Background Art
[0002] With the development of the internet and artificial intelligence, we have entered a digital age. This has generated a vast amount of text data, which we need to effectively manage and utilize to improve the information environment and enhance people's daily lives. Traditional text classification relies on manual labor, a process that consumes significant time and resources, resulting in high costs and extremely low efficiency. Therefore, achieving automated and efficient text classification has become an urgent task, aiming to improve efficiency and reduce costs. While machine learning algorithms have advanced the automation of text classification to some extent, they still have significant drawbacks. They require manual feature engineering, which is both time-consuming and labor-intensive. More critically, these algorithms often overlook semantic relationships and contextual information within the text when processing it, making it difficult to accurately capture the text's deeper semantics, significantly limiting the accuracy and depth of classification. In recent years, deep learning technology has emerged with significant momentum and widespread application. Text classification algorithms have also gradually shifted from traditional machine learning methods to deep learning-based neural network models and pre-trained language models. Deep learning models demonstrate significant advantages over traditional machine learning models. It can automatically learn and extract text features rich in semantic information, minimize manual intervention, significantly reduce labor costs, and provide a practical path for achieving efficient and accurate text classification. However, when using the pre-trained language model BERT for text classification, text category imbalance and overfitting problems may occur, which will reduce the accuracy of classification. How to deal with these phenomena becomes a problem. The present invention first introduces the Focal Loss function to enhance the focus on difficult-to-classify samples and solve the category imbalance problem. Then, Label Smoothing is introduced to smooth the label distribution, reduce the model's overfitting of the training data, and ultimately improve the performance of the model. Summary of the Invention
[0003] The present invention provides a BERT text classification method that integrates Focal Loss and Label Smoothing to solve the problems of text category imbalance and overfitting and improve the model classification accuracy.
[0004] To achieve the above purpose, the technical solution steps adopted by the present invention are as follows:
[0005] S1 builds a text classification model: defines a hierarchical category system and constructs category labels.
[0006] S2 data preprocessing: Convert traditional Chinese characters to simplified Chinese, convert full-width numbers and letters to half-width numbers and letters, delete missing values and duplicate data, remove special characters and HTML tags, and process stop words and punctuation.
[0007] S3 divides the preprocessed dataset into two parts, one for training, the other for fine-tuning, and the final evaluation of the BERT model.
[0008] The components of the BERT model in S3 are:
[0009] Word embedding: Convert each word in the input text into a fixed-size vector.
[0010] Paragraph embedding: add a paragraph embedding vector for each word;
[0011] Positional embeddings: A unique position vector generated by sine and cosine functions is added to each word, allowing the model to capture positional information and understand the order of words. Ultimately, these positional embeddings are combined with the output of the token embeddings to form a vector representation of the word.
[0012] Word segmentation: The WordPiece algorithm is used for word segmentation. The WordPiece algorithm is a subword-based word segmentation algorithm. It combines common character combinations into subwords based on the frequency of occurrence of character combinations in the corpus, thereby retaining semantic information while controlling the size of the vocabulary. The detailed steps are as follows:
[0013] 1) Initialize the vocabulary: The vocabulary contains single Chinese characters, common words (such as common two-character words and three-character words), and special tags (such as [cls] and [sep]).
[0014] 2) Input text: Accepts Chinese text with word segmentation, such as "I love natural language processing".
[0015] 3) Character segmentation: First, the input text is segmented by characters to obtain a character list ['I', 'love', 'self', 'natural', 'language', 'words', 'handle', 'manage'].
[0016] 4) Subword merging: Find the longest matching subword from the vocabulary. For each character or character combination, check whether it exists in the vocabulary. If it does, keep it as a subword; if not, try to split it into smaller parts. The specific process is as follows:
[0017] Starting from the first character, try to match the longest subword. For "I", it is already in the vocabulary, so it is kept as a subword.
[0018] Next, look at "love". Similarly, it is retained as a sub-word in the vocabulary.
[0019] For "nature", it is also in the vocabulary and is retained as a whole sub-word, rather than being split into "self" and "nature".
[0020] And so on. The final word segmentation result may be ["I", "love", "nature", "language", "processing"].
[0021] 5) Add special tokens: Similar to character-based word segmentation, add [CLS] and [SEP] tokens at the beginning and end of the word segmentation result to get the final word segmentation result ["[CLS]", "I", "love", "nature", "language", "processing", "[SEP]"].
[0022] 6) Handle out-of-vocabulary words: If characters or character combinations that do not exist in the vocabulary are encountered during word segmentation, they are marked as [UNK]. For example, if the input text contains a rare character that is not in the vocabulary, it will be replaced with [UNK] in the word segmentation result.
[0023] Transformer Encoder: Transformer includes a multi-head attention mechanism, a feed-forward neural network, layer normalization, and residual connections. Self-attention is the core of Transformer, enabling the model to process data at different positions in the input sequence to capture the relationships between words. The feed-forward neural network can perform the same operation independently on the output vectors at each position, enhancing the model's non-linearity. The output of each sub-layer (self-attention and feed-forward network) passes through a residual connection and then undergoes layer normalization. Residual connections help avoid the problem of vanishing gradients during training in deep networks, and layer normalization helps stabilize the learning process of the network.
[0024] Feed-forward Neural Network Formula:
[0025] Where is the input vector, and are weight matrices that map the input dimensions to the hidden dimension and then back to the output dimension respectively. and are bias terms, is the ReLU activation function to enhance non-linearity.
[0026] Residual Connection Formula: , where represents the output after the multi-head self-attention layer or the feed-forward neural network layer, is the input to this layer.
[0027] Layer normalization mathematical formula: ,in , , ,variance Measures the degree of dispersion of vector elements relative to the mean. is the input vector, is the mean of the vector, is a constant usually , are learnable parameters with the same dimension as the input vector.
[0028] S4. Define loss function: Define Focal Loss function and Label Smoothing function respectively, which are used to calculate the loss between the model prediction results and the true label during the training process.
[0029] Focal Loss mathematical formula: in is the predicted probability of the true category y, is the balancing factor for category y.
[0030] Label Smoothing mathematical formula: ,in , j represents class j, is a smoothing parameter with a value range of [0,1], and C is the number of categories.
[0031] S5 model training: Use the training set data, combined with Focal Loss and Label Smoothing for model training, and continuously adjust the model parameters to minimize the loss.
[0032] S6 checks whether the preset maximum number of training rounds has been reached. If not, training continues; otherwise, it enters the model evaluation phase.
[0033] S7 uses the validation set data to evaluate the trained model and calculate relevant evaluation indicators, such as precision, recall, F1 value, etc.
[0034] Accuracy: , recall rate:
[0035] F1 value:
[0036] Where T P The number of classification results correctly classified into a certain category, F N It indicates the number of incorrectly classified results but actually belong to a certain category, and Fp indicates the number of samples that are classified into a certain category but actually do not belong to that category.
[0037] S8 uses the test set data to perform a final test on the model and outputs the test results as the final evaluation of the model performance.
[0038] The present invention has the following beneficial effects and advantages:
[0039] (1) Introducing Focal Loss to solve the problem of category imbalance
[0040] (2) Label Smoothing is introduced to reduce the overfitting of the model to the training data by smoothing the label distribution. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 The overall flow chart of the present invention is DETAILED DESCRIPTION
[0042] The specific implementation details of the present invention are further described below with reference to the accompanying drawings.
[0043] S1 builds the basic model: Use the transformers library to load the pre-trained BERT model, leveraging its powerful language understanding capabilities as a foundation. Add a fully connected layer to the pre-trained model as the classification head. The output dimension of the fully connected layer matches the number of categories in the classification task.
[0044] S2 data preprocessing: remove special characters and HTML tags, process stop words and punctuation, and delete missing values and duplicate data.
[0045] S3 divides the data set: receives the data set processed by S2 and divides the processed data set into training set, validation set and test set according to a certain ratio.
[0046] S4 defines the loss function: Define the Focal Loss function. According to the mathematical formula of Focal Loss, implement its calculation logic in the code by adjusting parameters such as and To deal with the data imbalance problem. Define the Label Smoothing function: According to the principle of Label Smoothing, implement the calculation logic of converting hard labels into soft labels, and set the smoothing parameters Improve the generalization ability of the model.
[0047] S5 model training: Introduce the Focal Loss and Label Smoothing defined in S4 into BERT and load the training set divided by S3, and then perform the model training phase of text classification. First, define a class that receives the input input_ids, attention_mask, and labels as parameters, and then obtains the data index; then creates an instance and passes in the training set data, encapsulates it, and sets the batch size; then performs forward propagation, defines a BertClassifier class, and adds a fully connected layer as the classification head. In the forward method, pass the input input_ids and attention_mask into the BERT model, obtain its output pooler_output, and input it into the fully connected layer after Dropout processing to obtain the predicted logits; then traverse each batch of data, and then input it into the model for forward propagation to obtain logits. This is followed by loss calculation. Define a FocalLoss class. In the forward method, first operate on the input to obtain logpt, then extract the corresponding value from logpt according to the true label target, and then obtain , and finally calculate the loss according to the formula and find the average. Then convert the hard labels into soft labels, define a LabelSmoothing class, and in the forward method, generate soft labels based on the target. Finally, calculate the loss and find the average. Add the losses calculated by Focal Loss and Label Smoothing to get the total loss. Then perform backpropagation and parameter update, and use the Adam optimizer to update the model parameters. Before each update, first clear the gradient in the optimizer, then call the backward method of the total loss to perform backpropagation to calculate the gradient, and finally call the step method of the optimizer to update the model parameters. Finally, control the number of training rounds and set the total number of training rounds. In each training round, set the model to training mode, traverse each batch of data for forward propagation, loss calculation, backpropagation, and parameter update, and accumulate the loss of each batch. After a round of training, calculate the average loss of that round and output it.
[0048] S6 checks whether the preset maximum number of training rounds has been reached. If not, training continues; otherwise, it enters the model evaluation phase.
[0049] S7 Model Evaluation: After preparing the validation set or test set, perform model evaluation. If the evaluation results do not meet expectations, adjust the hyperparameters and re-perform training evaluation.
[0050] S8, final testing phase: use precision, recall and F1 value to comprehensively evaluate model performance.
[0051] The above is a specific implementation process.
[0052] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A BERT text classification method integrating Focal Loss and Label Smoothing, characterized by: S1 builds the basic model, S2 preprocesses the data, S3 divides the data set, S4 defines the loss function, S5 trains the model, S6 checks whether the preset maximum number of training rounds is reached, S7 evaluates the model, and step 8 is the final test stage.
2. The BERT text classification method integrating Focal Loss and Label Smoothing according to claim 1, characterized in that: S1 builds the basic model, defines the hierarchical category system, and constructs category labels.
3. The BERT text classification method integrating Focal Loss and Label Smoothing according to claim 1, characterized in that: S2 data preprocessing: Convert traditional Chinese characters to simplified Chinese, convert full-width numbers and letters to half-width numbers and letters, delete missing values and duplicate data, remove special characters and HTML tags, and process stop words and punctuation.
4. A BERT text classification method integrating Focal Loss and Label Smoothing according to claim 1, characterized in that ,S3 divides the preprocessed data set: receives the data set processed by S2, and divides the processed data set into training set, validation set and test set according to a certain ratio. It is used for model training and verification.
5. The BERT text classification method integrating Focal Loss and Label Smoothing according to claim 1, characterized in that: S4 defines the loss function: Focal Loss and Label Smoothing are combined into one loss function to solve the class imbalance and overfitting problems.
6. The BERT text classification method integrating Focal Loss and Label Smoothing according to claim 1, wherein: S5 model training: Load the loss function defined in S4 and the training set divided in S3 into the BERT model for training.
7. The BERT text classification method integrating Focal Loss and Label Smoothing according to claim 1, characterized in that: S6 checks whether the preset maximum number of training rounds has been reached: if not, continue training; otherwise, enter the model evaluation phase.
8. The BERT text classification method integrating Focal Loss and Label Smoothing according to claim 1, wherein: S7 Model Evaluation: After preparing the validation set or test set, perform model evaluation. If the evaluation results do not meet expectations, adjust the hyperparameters and re-perform training evaluation.
9. A BERT text classification method integrating Focal Loss and Label Smoothing according to claim 1, characterized in that S8, final testing phase: use precision, recall and F1 value to fully evaluate the model performance; Accuracy: (1) Recall: (2) F1 value: (3) Among them, Tp is the number of classification results correctly classified into a certain category, F N It indicates the number of incorrect classification results but actually belong to a certain category, and Fp indicates the number of samples that are classified into a certain category but actually do not belong to that category; The higher the F1 value, the better the performance of the model.