News classification method for field pre-training based on adaptive selective mask

Abstract

The invention relates to a field pre-training news classification method based on an adaptive selective mask, and belongs to the technical field of natural language processing. The method comprises the following steps: adding a domain-specific pre-training stage between general pre-training and fine tuning of a BERT classification model, and providing an adaptive selection masking strategy in a domain-guided pre-training process; performing fine tuning on each layer of the BERT classification model by using different learning rates, specifically, setting different learning rates for different layers; and checking and deleting error label data possibly existing in the downstream supervised learning task, and re-weighting and re-training the label data. According to the invention, the pre-training efficiency of the model and the news text classification accuracy are improved.

Description

Technical Field

[0001] The invention relates to a news classification method based on domain pre-training of adaptive selective masks, belonging to the technical field of natural language processing (NPL). Background Art

[0002] Text classification is a fundamental task in natural language processing (NLP) that aims to assign predefined labels or categories to text units (such as sentences, queries, paragraphs, and documents). Text classification is widely used in question-answering systems, spam detection, sentiment analysis, news classification, user intent classification, and content auditing. Although text data comes from a wide range of sources and contains rich information, it is still challenging to extract insights from it due to its unstructured nature. Text classification methods are mainly divided into manual annotation and automatic annotation. Although manual annotation is accurate, it is inefficient. Automatic annotation uses machine learning models to classify text, saving costs and improving accuracy and efficiency. Traditional machine learning models include two steps: feature extraction and classification. Common feature extraction methods include the bag of words model (BoW), while common classification algorithms include naive Bayes and support vector machines (SVM). Although these methods are effective, they rely on domain knowledge and have difficulty capturing complex relationships and high-level abstractions in the data.

[0003] In contrast, deep learning models can automatically learn features and identify complex patterns from large-scale training data. Common deep learning text classification models include convolutional neural networks (CNNs), recurrent neural networks (RNNs), and attention mechanisms (such as Transformers). These methods have performed well in various text classification tasks and address the limitations of traditional machine learning methods in feature extraction. However, deep learning methods also face some challenges, such as the need for a large amount of annotated data for training, customization for specific data sets resulting in weak generalization capabilities, difficulty in explaining internal mechanisms, and the need for a large amount of computing resources during training.

[0004] Automatic text classification methods have been further developed with the rise of pre-trained models. These models are pre-trained on large-scale datasets and achieve better performance and generalization in downstream tasks while reducing the reliance on large-scale annotated data. BERT (Bidirectional Encoder Representations from Transformers) is one of the most widely used pre-trained language models. BERT is trained through the Masked Language Modeling (MLM) task, randomly masking some tokens in the text sequence and restoring the masked tokens through a bidirectional Transformer. BERT overcomes various shortcomings of traditional machine learning models through automatic feature representation learning, powerful context-aware and relational learning, fine-tuning on multiple tasks, and the advantage of handling unstructured data.

[0005] Radford et al. (2018) and Rajpurkar et al. (2016) showed that PLM excels in text classification tasks. The main reasons for the success of PLM are two model architectures: autoregressive models and autoencoder models. GPT, proposed by OpenAI in 2018, is one of the earliest autoregressive models, which predicts text sequences from left to right, with each word prediction dependent on the previous words. In contrast, BERT, proposed by Devlin et al. (2018), is an autoencoder model trained through the masked language model (MLM) task, which can take into account bidirectional contextual information, thereby better understanding the context and relationships in sentences.

[0006] There are usually two strategies for applying pre-trained language models to downstream text classification tasks: feature-based methods and fine-tuning methods. Feature-based methods (such as ELMo) add pre-trained word embeddings as additional features to the existing task-specific model architecture; fine-tuning methods train downstream tasks by fine-tuning all pre-trained parameters, reducing dependence on data, improving parameter learning efficiency, and providing stronger transfer learning capabilities.

[0007] Most pre-trained language models follow the paradigm of pre-training and fine-tuning, using a "two-stage pre-training + fine-tuning" approach. In the general pre-training stage, the model learns general knowledge from a large number of unsupervised samples. In the fine-tuning stage, the pre-trained general model is fine-tuned using existing supervised data to adapt to downstream domains and task requirements. In specific news text classification tasks, they show some limitations: the pre-training stage lacks targeted training, and the fine-tuning stage is limited by limited downstream data, making it difficult for the model to fully learn task-specific knowledge. In addition, the traditional random masking strategy is random when masking, and the model is less efficient during the mask pre-training process. In addition, there is often wrong label data in the downstream supervised data set, which will interfere with model training and reduce the model classification effect. Summary of the invention

[0008] In view of the above problems existing in the prior art, the present invention provides a news classification method based on domain pre-training of adaptive selective masking, which improves the pre-training efficiency of the model and the accuracy of news text classification.

[0009] The technical solution of the present invention is: a news classification method based on domain pre-training of adaptive selective mask, the method comprising:

[0010] Add a domain-specific pre-training stage between the general pre-training and fine-tuning of the BERT classification model, and use an enhanced classification model including the general pre-training model and the domain-specific pre-training model to generate the final prediction of news text classification;

[0011] An adaptive selection masking strategy is provided during domain-guided pre-training to dynamically select and mask the landmarks that are important for downstream tasks during domain-specific pre-training.

[0012] As a further embodiment of the present invention, the present invention further comprises:

[0013] Fine-tune the BERT classification model: Fine-tune each layer of the BERT classification model using different learning rates. Specifically, different learning rates are set for different layers.

[0014] In terms of optimizer and parameter adjustment, the Adam optimizer is used in combination with the warm-up strategy and the linear decay strategy;

[0015] The warm-up strategy is to use a small learning rate at the beginning of training and gradually increase it to the preset value;

[0016] The linear decay strategy is to linearly reduce the learning rate from a preset value to a low threshold during training.

[0017] As a further embodiment of the present invention, the present invention further comprises:

[0018] Clean noisy labeled data: Check and remove incorrectly labeled data in downstream supervised learning tasks, and reweight the remaining labeled data for training.

[0019] As a further solution of the present invention, the domain-specific pre-training model includes a task pre-training layer and a fully connected layer;

[0020] The task pre-training layer uses unsupervised data in the news field and an adaptive selective masking strategy for pre-training and fine-tuning;

[0021] The fully connected layer processes the output through the Softmax function to generate the final prediction of news text classification.

[0022] As a further solution of the present invention, the universal pre-trained model is a pre-trained language model based on a multi-layer Transformer encoder, which performs masked language model tasks on large-scale unlabeled data and can self-learn the bidirectional contextual relationship of vocabulary.

[0023] As a further solution of the present invention, the enhanced classification model specifically includes the following components:

[0024] (1) Input layer: The input of the BERT classification model is the original text sequence; the input layer preprocesses the text by combining token embedding, paragraph embedding, and position embedding; preprocessing the text includes tokenization, adding special tags, and converting words into vector representations that the BERT classification model can handle;

[0025] Specifically:

[0026] Token embedding: Each word or subword in the input text is converted into an embedding vector of fixed dimension using the WordPiece tokenizer; token embedding captures the semantic information of each word and represents its meaning in the vector space;

[0027] Paragraph embedding: add a paragraph embedding vector for each word;

[0028] Position embeddings: A unique position vector generated by sine and cosine functions is added to each word, allowing the model to capture position information and understand the order of words; ultimately, these position embeddings are combined with the output of the token embeddings to form a vector representation of the word;

[0029] (2) The network structure of the BERT classification model: It consists of 12 stacked Transformer encoder layers; each encoder layer includes a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization;

[0030] (3) Masked Language Model (MLM): MLM is used in the pre-training phase of the BERT classification model. MLM involves randomly masking or replacing some words in the input text and requiring the model to predict the original content of these masked or replaced words. The BERT classification model predicts the masked words in the input sequence, and the output of the model is the probability distribution of each masked word obtained through the Softmax layer.

[0031] Next Sentence Prediction (NSP) Task: The next sentence prediction (NSP) task in the BERT classification model is used to determine whether two sentences are semantically adjacent or coherent.

[0032] (4) Domain-specific pre-training models: Domain-specific pre-training models are fine-tuned based on general pre-training; specifically, between pre-training and fine-tuning, unsupervised data in the news domain is used. An intermediate pre-training stage is added; this stage uses an adaptive selective mask strategy TaskSPT to learn the language pattern of the task; then, it outputs the classification prediction result through fine-tuning on the target task.

[0033] As a further solution of the present invention, the task pre-training layer includes two processes: task-specific pre-training and target task fine-tuning;

[0034] The task-specific pre-training is based on a general pre-training model using unsupervised data from the news domain. ,During the training process, an adaptive selective masking strategy is ,adopted to dynamically mask and label the news domain data;

[0035] The target task fine-tuning uses labeled data from downstream tasks to fine-tune the BERT classification model to meet the requirements of the target task; in the news text classification task, the fully connected layer outputs the classification result predicted by the model.

[0036] As a further solution of the present invention, in domain-specific pre-training, an adaptive selective masking strategy selects important words by analyzing the impact of task vocabulary on classification results, and dynamically applies selective masking to unsupervised data in the news domain. , this strategy includes the following steps:

[0037] (1) Use downstream supervision data Train a neural model; after training, use the model to calculate The score of each word in ; Next, choose a threshold to When the word is important in , each word with a score below the threshold is marked as important; then, the adaptive selective mask is applied to , only important words are retained and all unimportant words are replaced with special values; finally, the domain-specific pre-trained model is trained on the masked Pre-training on data;

[0038] (2) Select important words and propose a simple method to find Important vocabulary: Given an input sequence , set up a sequence to record important words; then, put each important word one by one Add to and evaluated separately; at time step 0, Initialize to empty; define a function To calculate For the news classification task, this function calculates the original input sequence Chinese Vocabulary The classification confidence of Add to The difference between the classification confidences after Remove the previous important words; the function definition is as follows:

[0039] ;

[0040] in, Yes Input The target classification label, It is fine-tuned on the task Calculated classification confidence;

[0041] (3) Masking on unsupervised data in the domain, supervised datasets from downstream tasks We select a small subset of the dataset and focus on labeling the words in the subset that are relevant to the news topic and are critical to the classification task; next, we use these focused labeling datasets to fine-tune the GenePM model to learn to identify important tags through a binary classification task; then, we use the fine-tuned GenePM model as a scoring mechanism to evaluate The importance of each tag in is calculated, and the tags considered important are selected; these important tags are masked to construct new training data; finally, the masked The GenePM model is pre-trained using the data.

[0042] As a further solution of the present invention, the fine-tuning of the BERT classification model is a new fine-tuning method, called discriminative fine-tuning, which allows different learning rates to be assigned to each layer;

[0043] For the conventional stochastic gradient descent method SGD, the time step The parameter update formula is as follows:

[0044] ;

[0045] in, is the learning rate, is the gradient of the objective function of the BERT classification model; in discriminative fine-tuning, the parameter Divided into ,in Contains the BERT classification model in The parameters of the layer, is the number of layers in the BERT classification model; similarly, the parameter Divided into ,in It is The learning rate of the layer; is the model parameter at the ith iteration or update;

[0046] The stochastic gradient descent SGD update formula under discriminative fine-tuning is as follows:

[0047] ;

[0048] in, Represents the parameter value of the model at the Lth layer and the tth iteration or time step, where L represents the number of layers and t represents the number of iteration steps or time steps, and is used to mark the stage of parameter update; Describes the gradient value at the jth parameter position in the Lth layer;

[0049] First fine-tune only the last layer and choose the learning rate for this layer , and then set the bottom layer’s learning rate to = / 2.4.

[0050] As a further solution of the present invention, in the cleaning of noisy label data, a confident learning CL algorithm is used to automatically filter out potential label errors; the process of processing label data with the confident learning CL algorithm includes three steps:

[0051] (1) Estimating class-conditional label noise: Using noisy data Training the model To predict the probability ;Build a confidence joint matrix , the values ​​in the confidence joint matrix represent the noise labels Each true label The confidence joint matrix is ​​then normalized to obtain the joint probability matrix , the values ​​in the joint probability matrix represent the noise labels Each true label The probability of

[0052] (2) Filtering noisy samples: Using the estimated joint probability matrix To identify and remove samples with incorrect labels, and retain clean samples for subsequent training;

[0053] (3) Reweighting and training: Calculate each category The weight of ,in, Represents the true label The class probability of Indicates that the noise label The real label For Category The clean data samples are reweighted using the weights, and the reweighted samples are used to train the enhanced classification model.

[0054] The beneficial effects of the present invention are:

[0055] 1. This invention introduces a domain-specific pre-training stage between pre-training and fine-tuning, and uses medium-scale unsupervised domain data to learn domain-specific knowledge, thus making up for the lack of domain-specific knowledge in traditional pre-training models. In this stage, an adaptive selective masking strategy is adopted, which dynamically adjusts the mask according to the importance of each tag, significantly improving the model's pre-training efficiency and the accuracy of news text classification;

[0056] 2. Assign a specific task score to each tag and dynamically and selectively mask target words related to the news topic based on the threshold, so as to capture key information more accurately. This innovation makes up for the shortcomings of the traditional random masking strategy and provides a more effective solution for domain-specific pre-training;

[0057] 3. When processing supervised data, it was found that there was noise in the labels of all four data sets. In order to reduce the impact of these noises on model training, the present invention uses the Confident Learning (CL) algorithm to clean up the noisy labels. The specific steps include filtering out the noise samples and reweighting and retraining the remaining samples, thereby improving the quality of the label data. This process helps the model better learn and identify patterns between labels, and ultimately improves the accuracy of classification;

[0058] 4. Through the application of this overall method of the present invention, not only the model's ability to master knowledge in specific fields is enhanced, but also the quality of training data and the performance of the model are improved, providing new insights and solutions for research and applications in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Schematic diagram of the general pre-training model of the present invention;

[0060] Figure 2 Schematic diagram of masked language model;

[0061] Figure 3 Schematic diagram of domain-specific pre-training models;

[0062] Figure 4 Schematic diagram of the process of finding and cleaning incorrectly labeled data. DETAILED DESCRIPTION

[0063] Example 1: Figure 1-Figure 4 As shown, a news classification method based on domain pre-training of adaptive selective masking is provided, and the method comprises:

[0064] A domain-specific pre-training stage is added between the general pre-training and fine-tuning of the BERT classification model, and an enhanced classification model including a general pre-training model (GenePM) and a domain-specific pre-training model (TaskSPT) is used to generate the final prediction for news text classification; an adaptive selective masking strategy is provided during the domain-guided pre-training process to dynamically select and mask tags that are important for downstream tasks during domain-specific pre-training.

[0065] Domain-specific pre-training phase: This is a further pre-training for news-related texts after the general pre-training of the BERT model. The general pre-trained BERT model has learned extensive language knowledge on large-scale text data, but it is not optimized specifically for the vocabulary, grammar, or structure of the news domain. News texts often have their own expressions, common vocabulary, and topics (such as "tech news" and "financial news", etc.), so we pre-train on domain-specific datasets related to news topics to allow the model to specifically learn and adapt to the characteristics of these news domains.

[0066] The general pre-trained model is a pre-trained language model based on a multi-layer Transformer encoder, which captures bidirectional contextual information of text by combining tokenization, various embeddings, and task-specific output layers; the model performs masked language model tasks on large-scale unlabeled data, and can self-learn the bidirectional contextual relationship of vocabulary to master language rules and patterns. After pre-training, the model can be fine-tuned for specific tasks;

[0067] The domain-specific pre-training model includes a task pre-training layer and a fully connected layer; the task pre-training layer uses unsupervised data in the news domain and an adaptive selective masking strategy for pre-training and fine-tuning; the fully connected layer processes the output through a Softmax function to generate a final prediction for news text classification.

[0068] The enhanced classification model specifically includes the following components:

[0069] (1) Input layer: The input of the BERT classification model is the original text sequence. The input layer preprocesses the text by combining token embedding, paragraph embedding, and position embedding. Preprocessing the text includes tokenization, adding special tags (such as [CLS] and [SEP]), and converting words into vector representations that the BERT classification model can handle.

[0070] Specifically:

[0071] Token embedding: Each word or subword in the input text is converted into an embedding vector of fixed dimension using the WordPiece tokenizer; token embedding captures the semantic information of each word and represents its meaning in the vector space;

[0072] Paragraph embedding: A paragraph embedding vector is added to each word; usually two vectors (paragraph A and paragraph B) are used to indicate the paragraph to which the word belongs. This is used to distinguish different paragraphs or sentences in the input text and provide necessary contextual information;

[0073] Position embeddings: A unique position vector generated by sine and cosine functions is added to each word, allowing the model to capture position information and understand the order of words; ultimately, these position embeddings are combined with the output of the token embeddings to form a vector representation of the word;

[0074] (2) The network structure of the BERT classification model: It consists of 12 stacked Transformer encoder layers; each encoder layer includes a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization; enabling the model to capture complex dependencies in the input sequence. Specifically:

[0075] Multi-head self-attention: The multi-head attention mechanism uses parallel computation of multiple attention heads to capture diverse information and features of the input sequence from different angles, thereby enhancing the expressiveness of the model. This mechanism improves the stability and robustness of the model while optimizing processing efficiency through better parallel computation. The specific operation involves converting the input sequence into multiple sets of queries Q, keys K, and value vectors V through linear transformations. These vectors are dot-producted to derive the attention scores, and then normalized using the softmax function. The weighted sum of the value vectors of each attention head is calculated based on the attention weights to obtain a single output. Finally, the outputs of all attention heads are concatenated and linearly transformed to generate the final output representation, as shown below:

[0076] ;

[0077] In the self-attention mechanism, Q is the query vector, K is the key vector, and V is the value vector. is the dimension of the key vector, and Softmax is the Softmax function used to normalize the attention scores. The feedforward neural network further transforms the output of the self-attention mechanism to extract higher-level features. A feedforward neural network usually consists of two linear transformations and an activation function (such as ReLU), which allows nonlinear mapping of the input, thereby increasing the expressive power of the model. The mathematical formula of a feedforward neural network is:

[0078] ;

[0079] in, is the dimension of the key vector, is the weight matrix, and is the bias vector. yes Activation function. Residual connections and layer normalization improve the stability and effectiveness of model training and help alleviate the gradient vanishing and exploding problems. Residual connections allow information to propagate directly by skipping certain layers, while layer normalization ensures that data distribution between layers is stable, speeding up model training and improving performance. The mathematical formula for residual connections is:

[0080] ;

[0081] Represents the output after passing through a certain layer (for example, multi-head self-attention or feed-forward neural network). is the input of this layer; the mathematical formula for layer normalization is:

[0082] ;

[0083] in, is the input vector, is the average value of the input vector, calculated as: ;

[0084] is the standard deviation of the input vector, calculated as: , is a tiny constant that prevents division by zero, and are learnable parameters used to scale and offset the normalized output, is the dimension of the input vector. Combining residual connections and layer normalization, each layer in the BERT classification model can be expressed as:

[0085] ;

[0086] (3) Masked Language Model (MLM): MLM is used in the pre-training phase of the BERT classification model. MLM is one of the core training tasks of the BERT classification model. MLM involves randomly masking or replacing some words in the input text and requiring the model to predict the original content of these masked or replaced words. Specifically, during pre-training, the BERT classification model randomly selects 15% of the words in the text for masking. Among these words, 80% of the words are replaced with MASK tags, 10% of the words are replaced with random words, and the remaining 10% remain unchanged. The BERT classification model predicts the masked words in the input sequence, and the output of the model is the probability distribution of each masked word obtained through the Softmax layer. The mathematical formula of the masked language model is:

[0087] ;

[0088] is the target probability distribution, It is words, represent All the previous words, are model parameters, is an indicator function. If If a word is masked, it is equal to 1, otherwise it is 0;

[0089] For example, Figure 2 As shown in Figure 1, a masking operation is performed on a news sentence. Through MLM, the probability distribution of the masked words is obtained, where the sum of all probability distributions is 1. The word " The probability distribution value of " is 98.5%, which is much higher than the probability distribution values ​​of other words. Therefore, it is inferred that " ” is the masked word.

[0090] Next Sentence Prediction NSP Task: The next sentence prediction NSP task in the BERT classification model is used to determine whether two sentences are semantically adjacent or coherent; the mathematical formula for this task is:

[0091] ;

[0092] in, and Represents two sentences, Represents the BERT model for sentence pairs The vector representation of represents a random sentence from different documents.

[0093] The NSP task compares forward sentence pairs The probability of a random unrelated sentence pair The purpose is to determine whether the sentences are adjacent or coherent. By maximizing the probability of correct sentence pairs and minimizing the probability of incorrect sentence pairs, the BERT classification model learns the semantic relationship and contextual coherence between sentences.

[0094] (4) Domain-specific pre-training models: Domain-specific pre-training models are fine-tuned based on general pre-training; specifically, between pre-training and fine-tuning, unsupervised data in the news domain is used. An intermediate pre-training stage is added; this stage uses an adaptive selective mask strategy TaskSPT to learn the language pattern of the task; then, it outputs the classification prediction result through fine-tuning on the target task.

[0095] As a further solution of the present invention, the task pre-training layer includes two processes: task-specific pre-training and target task fine-tuning;

[0096] The task-specific pre-training is based on a general pre-training model using unsupervised data from the news domain. ,During the training process, an adaptive selective masking strategy is used to dynamically mask and annotate news domain data;,enhance the model's ability to learn task language patterns and improve pre-training efficiency;

[0097] The target task fine-tuning uses the labeled data from the downstream task to fine-tune the BERT classification model to meet the requirements of the target task; in the news text classification task, the fully connected layer outputs the classification result predicted by the model. The output of the model is a probability distribution. For example, for a given news article input to the model, the category with the highest probability in the distribution is selected as the prediction result. If the probability of the "sports" category reaches 98.5%, which is significantly higher than other categories, we predict that it is classified as "sports" news.

[0098] As a further solution of the present invention, in domain-specific pre-training, an adaptive selective masking strategy selects important words by analyzing the impact of task vocabulary on classification results, and dynamically applies selective masking to unsupervised data in the news domain. , this strategy includes the following steps:

[0099] (1) Use downstream supervision data Train a neural model; after training, use the model to calculate The scores of each word in the news vocabulary are calculated; these scores can be the classification probabilities predicted by the model or other indicators of word importance. Next, a threshold is selected, which can be determined based on the performance of the validation set or other heuristic rules. The lower the threshold, the closer the word is to the target word in the news vocabulary; therefore, words with scores below this threshold are marked as important words; specifically, in identifying When the word is important in , each word with a score below the threshold is marked as important; then, the adaptive selective mask is applied to , only important words are kept and all unimportant words are replaced with special values ​​(e.g., MASK); finally, the domain-specific pre-trained model is trained on the masked Pre-training on the data enables the model to learn the features of important words, thereby improving its classification performance in downstream tasks.

[0100] Task vocabulary: refers to the fact that in news classification tasks, certain words play an important role in the classification results or appear more frequently. For example, "weather", "culture", and "sports" may appear frequently in news texts. They are all words in the news subject area of ​​some news texts and are important for the model to judge the news type or topic. In domain-specific pre-training, the model will focus on learning these words to help it understand the news content more accurately;

[0101] (2) Select important words and propose a simple method to find Important vocabulary: Given an input sequence , set up a sequence to record important words; then, put each important word one by one Add to and evaluated separately; at time step 0, Initialize to empty; define a function To calculate For the news classification task, this function calculates the original input sequence Chinese Vocabulary The classification confidence of Add to To ensure that the current The score calculation is not affected by the previous important words, and is dynamically calculated from Remove the previous important words; the function definition is as follows:

[0102] ;

[0103] in, Yes Input The target classification label, It is fine-tuned on the task Calculated classification confidence; in the experiment, we set If the standard of important vocabulary, , which means that when adding After fine-tuning Incomplete sequence buffers can be correctly classified with confidence close to that of complete sequences .

[0104] (3) Masking on unsupervised data in the domain, supervised datasets from downstream tasks We select a small subset of data that contains only the tags that are critical to the classification task; we focus on labeling the words in this subset that are relevant to the news topic and are critical to the classification task; we then use these focused labeled datasets to fine-tune the GenePM model to learn to identify important tags through a binary classification task; we then use the fine-tuned GenePM model as a scoring mechanism to evaluate The importance of each tag in is calculated, and the tags considered important are selected; these important tags are masked to construct new training data; finally, the masked Pre-train the GenePM model with the data to enhance its learning ability for specific tasks;

[0105] The present invention also includes fine-tuning the BERT classification model: fine-tuning each layer of the BERT classification model using different learning rates, specifically setting different learning rates for different layers; this helps to retain the features learned in the pre-training phase, prevent catastrophic forgetting, and adapt to the needs of new tasks;

[0106] In terms of optimizer and parameter adjustment, the Adam optimizer is used in combination with the warm-up strategy and the linear decay strategy to improve the convergence speed and stability during fine-tuning.

[0107] The warm-up strategy is to use a small learning rate at the beginning of training and gradually increase it to the preset value;

[0108] The linear decay strategy is to linearly reduce the learning rate from a preset value to a low threshold during training.

[0109] The adaptive learning rate feature of the Adam optimizer, combined with the layered learning rate setting of discriminative fine-tuning, can further improve the fine-tuning effect of the model.

[0110] As a further solution of the present invention, in order to enable the neural network to capture different types of information in each layer, we use different learning rates to update parameters during the fine-tuning process; the fine-tuning of the BERT classification model is a new fine-tuning method, called discriminative fine-tuning. Different from the common practice of using the same learning rate to update all model parameters, discriminative fine-tuning allows us to assign different learning rates to each layer, thereby reflecting the "discriminative" characteristics of each layer;

[0111] Discriminative fine-tuning allows assigning different learning rates to each layer;

[0112] For the conventional stochastic gradient descent method SGD, the time step The parameter update formula is as follows:

[0113] ;

[0114] in, is the learning rate, is the gradient of the objective function of the BERT classification model; in discriminative fine-tuning, the parameter Divided into ,in Contains the BERT classification model in The parameters of the layer, is the number of layers in the BERT classification model; similarly, the parameter Divided into ,in It is The learning rate of the layer; is the model parameter at the i-th iteration or update; the purpose of this method is to finely control the parameter update speed of each layer to avoid the loss of shallow features or insufficient deep learning, thereby improving the overall performance of the model;

[0115] Unlike traditional methods that usually use the same learning rate for all layers, this method may lead to a loss of balance between retaining original knowledge and learning new tasks. Discriminative fine-tuning better retains the features learned in the pre-training phase by setting different learning rates for each layer, while adapting to the needs of new tasks. Specifically, shallow features require a lower learning rate to maintain stability, while deep features require a higher learning rate to adapt to new tasks; the stochastic gradient descent SGD update formula under discriminative fine-tuning is as follows:

[0116] ;

[0117] in, Represents the parameter value of the model at the Lth layer and the tth iteration or time step, where L represents the number of layers and t represents the number of iteration steps or time steps, and is used to mark the stage of parameter update; Describes the gradient value at the jth parameter position in the Lth layer;

[0118] We found through experiments that we first fine-tune only the last layer and choose the learning rate of this layer , and then set the bottom layer’s learning rate to = / 2.4. This approach works well. This layered learning rate adjustment method helps us further optimize the model fine-tuning process. Compared with traditional methods, this layered strategy better balances the learning of new features and the retention of original features, thereby enhancing the overall performance of the model;

[0119] Researchers often use benchmark datasets to evaluate the progress of the state-of-the-art and verify theoretical findings. However, if there are a large number of label errors in the dataset, it may affect the accurate evaluation of the progress of machine learning. Since manually verifying all labels is expensive and time-consuming, an automated method is needed to identify potential label errors;

[0120] Therefore, the present invention also includes cleaning up noisy label data: checking and removing erroneous label data in downstream supervised learning tasks, and re-weighting and training the remaining label data to ensure the quality of the label data, thereby improving the effectiveness of model training;

[0121] In the cleaning of noisy label data, the confident learning CL algorithm is used to automatically filter potential label errors; there is no need to manually check the entire data set, thereby reducing the cost and workload of manual verification. In addition, it can accurately quantify uncertainty and provide robustness when dealing with class imbalance in the data set; the process of confident learning CL algorithm processing label data includes three steps:

[0122] (1) Estimating class-conditional label noise: Using noisy data Training the model To predict the probability ;Build a confidence joint matrix , the values ​​in the confidence joint matrix represent the noise labels Each true label count; for example, 80 means that when labeled as "sports", there are actually 80 samples that are "sports"; then the confidence joint matrix is ​​normalized to obtain the joint probability matrix , the values ​​in the joint probability matrix represent the noise labels Each true label probability; for example, 0.20 means that when labeled as "sports", 20% of the samples are actually "sports;

[0123] (2) Filtering noisy samples: Using the estimated joint probability matrix To identify and remove samples with incorrect labels, and retain clean samples for subsequent training;

[0124] (3) Reweighting and training: Calculate each category The weight of ,in, Represents the true label The class probability of Indicates that the noise label The real label For Category The clean data samples are reweighted using the weights, and the reweighted samples are used to train the enhanced classification model.

[0125] In order to prove the effect of the present invention, the following experiments were performed:

[0126] In news text classification, precision, recall and F1 score are commonly used evaluation indicators:

[0127] Accuracy: The proportion of samples correctly predicted by the model to the total samples, reflecting the overall prediction effect.

[0128] Recall rate: The proportion of samples that are actually of a certain class that are successfully identified by the model as that class, reflecting the model's ability to identify that class.

[0129] F1 score: The weighted average of precision and recall, which comprehensively evaluates the precision and recall capabilities of the model.

[0130] Experiment 1: The enhanced classification model (BERT-S) is compared with other models on the benchmark dataset;

[0131] Comparative experiments were conducted on four popular benchmark datasets (AG'sNews, 20Newsgroups, R8 and R52) with other models. The experimental results show that the enhanced classification model (BERT-S) has an average improvement of 4.1% in accuracy and 4.0% in F1 score compared with other models. Some typical models were selected for detailed discussion, see Table 1 and Table 2, and the specific analysis is as follows:

[0132] Table 1: Comparison of the enhanced classification model and the other 9 models on the AG's News dataset (%)

[0133]

[0134] Table 2: Comparison of the enhanced classification model and the other 7 models on the 20Newsgroups, R8 and Reuters-21578 datasets (%)

[0135]

[0136] Performance on AG's News dataset: The F1 score of the enhanced classification model (BERT-S) is 3.2% higher than ULMFiT and 4.6% higher than CNN. ULMFiT has difficulty capturing long-distance dependencies due to its LSTM architecture, and CNN cannot model global relationships in semantically complex news texts. In contrast, the bidirectional Transformer architecture and adaptive mask strategy of the enhanced classification model are better at capturing global semantics. Compared with the static mask of Mask-BERT, the adaptive mask strategy of the enhanced classification model is flexible and efficient, and its F1 score is 3.5% higher than XLNet.

[0137] Performance on the 20Newsgroups dataset: The F1 score of the enhanced classification model (BERT-S) is 4.4% higher than SGC and 8.3% higher than RoBERTa. SGC is suitable for graph-structured data but has limited ability to model sequence information in text classification, while RoBERTa's static mask strategy also reduces adaptability. The adaptive mask strategy of the enhanced classification model enhances the classification performance of multi-topic text. Compared with BERT-TextCNN, BERT-S has a 7.2% higher F1 score and is better at capturing long-distance dependencies.

[0138] Performance on the R8 dataset: BERT-S's F1 score is 99.6%, 1.6% higher than RoBERTaGAT and 2.6% higher than TextCNN. RoBERTaGAT and TextCNN are limited in modeling long-distance dependencies, while BERT-S's adaptive mask strategy is more flexible and effectively captures task-specific language patterns.

[0139] Performance on the R52 dataset: The F1 score of the enhanced classification model (BERT-S) is 5.0% higher than TextGCN and 2.5% higher than RoBERTaGCN. TextGCN is good at processing graph structures but not good at subtle features in language sequences. In contrast, BERT-S better captures the semantic differences between categories through an adaptive masking strategy, outperforming RoBERTaGCN and BERT-TextCNN.

[0140] In summary, the adaptive masking strategy of the enhanced classification model effectively improves the ability to capture task-specific language patterns and global dependencies, and performs superiorly in various text classification tasks, surpassing traditional pre-trained models.

[0141] Experiment 2: Comparison of the performance of the enhanced classification model (BERT-S) and the traditional BERT model on other tasks;

[0142] The experimental results are shown in Table 3. The enhanced classification model (BERT-S) performs well in a variety of natural language processing tasks, such as sentiment analysis (IMDB), question answering (QUAD), and natural language inference (SNLI), and always outperforms the traditional BERT classification model in performance. The specific analysis is as follows:

[0143] In the sentiment analysis task, the enhanced classification model effectively retains key information through an adaptive selective masking strategy, avoiding interference from secondary content, and enabling the model to accurately capture emotional tendencies. On the IMDB dataset, BERT-S's accuracy, recall, and F1 score increased by 1.6%, 2.0%, and 1.8%, respectively.

[0144] In the question-answering task, the enhanced classification model can focus on the words most relevant to the question and improve the accuracy of information extraction. On the QUAD dataset, the accuracy of the enhanced classification model increased by 1.9%, and the recall rate and F1 score increased by 1.0% and 1.4% respectively.

[0145] In natural language reasoning tasks, the enhanced classification model can dynamically adjust the masking range and enhance logical reasoning capabilities. Experiments show that the accuracy of the enhanced classification model on the SNLI dataset has increased by 2%, and the recall rate and F1 score have increased by 1.5% and 0.9% respectively.

[0146] In summary, the enhanced classification model demonstrates strong adaptability and robustness on the IMDB, QUAD, and SNLI datasets, further verifying its excellent generalization ability.

[0147] Table 3: Performance comparison of enhanced classification model and traditional BERT model on other tasks (%)

[0148]

[0149] In summary, from Experiments 1 and 2, it can be seen that the enhanced classification model of the present invention has significant advantages over the traditional model in news classification tasks, with an average accuracy improvement of about 4.1%, and an accuracy of more than 90% in multi-task classification, and the training efficiency is also greatly improved.

[0150] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A news classification method based on domain pre-training with adaptive selective masking, characterized by: The method comprises: Add a domain-specific pre-training stage between the general pre-training and fine-tuning of the BERT classification model, and use an enhanced classification model including the general pre-training model and the domain-specific pre-training model to generate the final prediction of news text classification; An adaptive selection masking strategy is provided during domain-guided pre-training to dynamically select and mask the landmarks that are important for downstream tasks during domain-specific pre-training.

2. The news classification method based on domain pre-training of adaptive selective mask according to claim 1, characterized in that: Also includes: Fine-tune the BERT classification model: Fine-tune each layer of the BERT classification model using different learning rates. Specifically, different learning rates are set for different layers. In terms of optimizer and parameter adjustment, the Adam optimizer is used in combination with the warm-up strategy and the linear decay strategy; The warm-up strategy is to use a small learning rate at the beginning of training and gradually increase it to the preset value; The linear decay strategy is to linearly reduce the learning rate from a preset value to a low threshold during training.

3. The news classification method based on domain pre-training of adaptive selective mask according to claim 1, characterized in that: Also includes: Clean noisy labeled data: Check and remove incorrectly labeled data in downstream supervised learning tasks, and reweight the remaining labeled data for training.

4. The news classification method based on domain pre-training of adaptive selective mask according to claim 1, characterized in that: The domain-specific pre-training model includes a task pre-training layer and a fully connected layer; The task pre-training layer uses unsupervised data in the news field and an adaptive selective masking strategy for pre-training and fine-tuning; The fully connected layer processes the output through the Softmax function to generate the final prediction of news text classification.

5. The news classification method based on domain pre-training of adaptive selective mask according to claim 1, characterized in that: The general pre-trained model is a pre-trained language model based on a multi-layer Transformer encoder, which performs masked language model tasks on large-scale unlabeled data and can self-learn the bidirectional contextual relationship of vocabulary.

6. The news classification method based on domain pre-training of adaptive selective mask according to claim 1, characterized in that: The enhanced classification model specifically includes the following components: (1) Input layer: The input of the BERT classification model is the original text sequence; the input layer preprocesses the text by combining token embedding, paragraph embedding, and position embedding; Preprocessing text includes tokenization, adding special tags, and converting words into vector representations that can be processed by the BERT classification model; Specifically: Token embedding: Each word or subword in the input text is converted into an embedding vector of fixed dimension using the WordPiece tokenizer; token embedding captures the semantic information of each word and represents its meaning in the vector space; Paragraph embedding: add a paragraph embedding vector for each word; Position embeddings: A unique position vector generated by sine and cosine functions is added to each word, allowing the model to capture position information and understand the order of words; ultimately, these position embeddings are combined with the output of the token embeddings to form a vector representation of the word; (2) The network structure of the BERT classification model: It consists of 12 stacked Transformer encoder layers; each encoder layer includes a multi-head self-attention mechanism, a feedforward neural network, residual connections, and layer normalization; (3) Masked Language Model (MLM): MLM is used in the pre-training phase of the BERT classification model. MLM involves randomly masking or replacing some words in the input text and requiring the model to predict the original content of these masked or replaced words. The BERT classification model predicts the masked words in the input sequence, and the output of the model is the probability distribution of each masked word obtained through the Softmax layer. Next Sentence Prediction (NSP) Task: The next sentence prediction (NSP) task in the BERT classification model is used to determine whether two sentences are semantically adjacent or coherent. (4) Domain-specific pre-training models: Domain-specific pre-training models are fine-tuned based on general pre-training; specifically, between pre-training and fine-tuning, unsupervised data in the news domain is used. An intermediate pre-training stage is added; this stage uses an adaptive selective mask strategy TaskSPT to learn the language pattern of the task; then, it outputs the classification prediction result through fine-tuning on the target task.

7. The news classification method based on domain pre-training of adaptive selective mask according to claim 4, characterized in that: The task pre-training layer includes two processes: task-specific pre-training and target task fine-tuning; The task-specific pre-training is based on a general pre-training model using unsupervised data from the news domain. ,During the training process, an adaptive selective masking strategy is ,adopted to dynamically mask and label the news domain data; The target task fine-tuning uses labeled data from downstream tasks to fine-tune the BERT classification model to meet the requirements of the target task; in the news text classification task, the fully connected layer outputs the classification result predicted by the model.

8. The news classification method based on domain pre-training of adaptive selective mask according to claim 1, characterized in that: In domain-specific pre-training, an adaptive selective masking strategy selects important words by analyzing the impact of task vocabulary on classification results, and dynamically applies selective masking to unsupervised data in the news domain. , this strategy includes the following steps: (1) Use downstream supervision data Train a neural model; after training, use the model to calculate The score of each word in ; Next, choose a threshold to When the word is important in , each word with a score below the threshold is marked as important; then, the adaptive selective mask is applied to , only important words are retained and all unimportant words are replaced with special values; finally, the domain-specific pre-trained model is trained on the masked Pre-training on data; (2) Select important words and propose a simple method to find Important vocabulary: Given an input sequence , set up a sequence to record important words; then, put each important word one by one Add to and evaluated separately; at time step 0, Initialize to empty; define a function To calculate For the news classification task, this function calculates the original input sequence Chinese Vocabulary The classification confidence of Add to The difference between the classification confidences after Remove the previous important words; the function definition is as follows: ； in, Yes Input The target classification label, It is fine-tuned on the task Calculated classification confidence; (3) Masking on unsupervised data in the domain, supervised datasets from downstream tasks We select a small subset of the dataset and focus on labeling the words in the subset that are relevant to the news topic and are critical to the classification task; next, we use these focused labeling datasets to fine-tune the GenePM model to learn to identify important tags through a binary classification task; then, we use the fine-tuned GenePM model as a scoring mechanism to evaluate The importance of each tag in is calculated, and the tags considered important are selected; these important tags are masked to construct new training data; finally, the masked The GenePM model is pre-trained using the data.

9. The news classification method based on domain pre-training of adaptive selective mask according to claim 2, characterized in that: The fine-tuning of the BERT classification model is a new fine-tuning method called discriminative fine-tuning. Discriminative fine-tuning allows different learning rates to be assigned to each layer. The stochastic gradient descent SGD update formula under discriminative fine-tuning is as follows: ； in, Represents the parameter value of the model at the Lth layer and the tth iteration or time step, where L represents the number of layers and t represents the number of iteration steps or time steps, and is used to mark the stage of parameter update; Describes the gradient value at the jth parameter position in the Lth layer; It is The learning rate of the layer.

10. The news classification method based on domain pre-training of adaptive selective mask according to claim 3, characterized in that: In the cleaning of noisy label data, the confident learning CL algorithm is used to automatically filter out potential label errors; the process of processing label data by the confident learning CL algorithm includes three steps: (1) Estimating class-conditional label noise: Using noisy data Training the model To predict the probability ;Build a confidence joint matrix , the values ​​in the confidence joint matrix represent the noise labels Each true label The confidence joint matrix is ​​then normalized to obtain the joint probability matrix , the values ​​in the joint probability matrix represent the noise labels Each true label probability; (2) Filtering noisy samples: Using the estimated joint probability matrix To identify and remove samples with incorrect labels, and retain clean samples for subsequent training; (3) Reweighting and training: Calculate each category The weight is ,in, Represents the true label The class probability of Indicates the noise label The real label For Category The clean data samples are reweighted using the weights, and the reweighted samples are used to train the enhanced classification model.

Images