Domain difference self-learning fine-tuning text classification method based on pre-trained model

Abstract

The application relates to the field of natural language processing and the field of deep learning, in particular to a domain difference self-learning fine-tuning text classification method based on a pre-training model, which comprises the following steps: obtaining a text to be measured; adopting a fine-tuned pre-training model to perform feature extraction on the text to be measured, so as to obtain a feature vector of the text to be measured; inputting the feature vector of the text to be measured into a domain level head module, so as to obtain the confidence of the text to be measured in each category; and taking the category with the highest confidence of the text to be measured in all categories as the final prediction result of the text to be measured. The application is based on a pre-training model and combines a domain difference self-learning technology, introduces supervised contrast learning, classification learning and weak clustering learning and other strategies, realizes the learning of different levels of semantic differences, and utilizes the implicit relationship between categories to assist classification. Meanwhile, a classifier weight initialization method more suitable for text classification is adopted, so that the classification performance is improved.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and deep learning, specifically to a domain-differential self-learning fine-tuning text classification method based on a pre-trained model, which can be used for text classification and intent recognition. Background Technology

[0002] Text classification is one of the fundamental tasks in the field of natural language processing. Its main purpose is to categorize input text into one or more categories based on its content. With the rapid development of big data and artificial intelligence technologies, text classification has been widely applied in various fields.

[0003] Traditional text classification methods often require manual feature construction, resulting in high costs and semantic gaps. In recent years, the development of deep learning technology has provided new solutions for text classification tasks, especially pre-trained models, which have achieved significant results. The implementation process of existing classic classification fine-tuning methods based on pre-trained models is as follows: Figure 1 As shown, a pre-trained model is used as an encoder to extract features from the training dataset. The extracted feature vectors are then input into the SoftMax classifier to obtain the confidence score for each category. Cross-entropy loss is calculated for the confidence score of each category, and the network is updated with backward gradients to optimize the pre-trained model.

[0004] However, traditional methods using cross-entropy loss for classification fine-tuning fail to effectively utilize instance-level features. The model only learns task-level knowledge and cannot fully learn the implicit features between categories, which limits the model's learning ability and makes it difficult to alleviate overfitting due to insufficient sample size. Therefore, it is necessary to introduce instance-level knowledge to improve classification accuracy. To optimize traditional fine-tuning schemes, existing methods introduce contrastive learning to learn fine-grained features between instances. The implementation flowchart of this scheme is as follows: Figure 2 As shown, compared to traditional classification fine-tuning methods, this improved scheme adds a contrastive learning loss and combines it with the classification cross-entropy loss, then performs a backward gradient update operation on the network. This avoids the problem of decreased training accuracy caused by overfitting in traditional algorithms with few samples. However, in classic fine-tuning classification methods with contrastive learning, when there is sufficient sample data, the model learns too much instance-level knowledge, which affects the sample clustering effect and offsets the positive impact of the learned additional knowledge on classification.

[0005] Furthermore, classic fine-tuning methods for pre-trained model classification typically use Xavier to initialize a softmax classifier, treating the classifier's output as the predicted class value. However, this initialization method fails to consider the characteristics of text representations of the same class in the encoder's latent space and does not perform corresponding optimizations for the classification task.

[0006] In summary, these existing techniques, when dealing with classification problems, only need to ensure that the similarity between a sentence and the correctly classified category exceeds that of other incorrectly classified categories. Relying solely on cross-entropy loss for classification may lead to blindly training sentences to have completely different similarities with incorrect categories. This could cause the model to ignore the connections between sentences and other categories, focusing only on learning discrete features, thus reducing its generalization ability. Summary of the Invention

[0007] To address the problems existing in current technologies, this invention proposes a domain-difference self-learning fine-tuning text classification method based on a pre-trained model, comprising the following steps:

[0008] Obtain the text to be tested;

[0009] The finely tuned pre-trained model is used to extract features from the text to be tested, and the feature vector of the text to be tested is obtained.

[0010] The feature vector of the text to be tested is input into the domain-level header module to obtain the confidence score of the text to be tested in each category;

[0011] The category with the highest confidence among all categories is taken as the final prediction result for the test text.

[0012] Furthermore, in order to obtain the fine-tuned pre-trained model, this invention employs the following method for fine-tuning training:

[0013] Obtain the text training set;

[0014] A pre-trained model is used to extract features from each training text in the text training set to obtain the feature vector of the training text.

[0015] The extracted training text feature vectors are input into three modules: instance-level header, task-level header, and domain-level header, respectively, to obtain instance-level features, task-level features, and domain-level features of the training text.

[0016] Supervised contrastive learning is performed on the obtained instance-level features to learn the fine-grained semantic differences between instances, resulting in instance-level loss.

[0017] The obtained task-level features are subjected to classification learning to learn the differences between different tasks, and the task-level loss is obtained.

[0018] Weak clustering learning is performed on the obtained domain-level features to learn the potential semantic relationships between categories, resulting in a domain-level loss.

[0019] Instance-level loss L scl Mission-level loss L ce and domain-level loss L wcl The weighted summation yields the final loss;

[0020] By calculating the final loss, the network is updated using backward gradients to optimize the pre-trained model.

[0021] Compared with traditional text classification methods, this invention has the following advantages:

[0022] 1. By proposing a weak clustering loss, the loss of the incorrect class is linked to that of the correct class, allowing the model to dynamically learn the similarity target. This enables the model to learn continuous features and better learn the implicit relationship between different label vectors, thereby improving the classification accuracy.

[0023] 2. Compared with traditional methods that incorporate contrastive learning for classification fine-tuning, this method, by applying supervised contrastive learning, can better learn the features between categories, making the clustering of samples of the same category more compact.

[0024] 3. By combining fine-tuning of pre-trained models with supervised contrastive learning, classification learning, and weak clustering learning strategies, different levels of semantic differences can be learned, achieving efficient text classification with good generalization ability.

[0025] 4. To enable the model to give more attention to challenging examples in each category, this invention improves the classifier's weight initialization method for classification tasks, allowing the model to consider more difficult samples in each class, thereby improving classification performance. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the implementation of a classic text classification fine-tuning method based on a pre-trained model.

[0027] Figure 2 This is a flowchart illustrating the implementation of a traditional contrastive learning method for fine-tuning text classification based on pre-trained models.

[0028] Figure 3 This is a flowchart illustrating the fine-tuning implementation of a pre-trained model according to an embodiment of the present invention.

[0029] Figure 4 This is a flowchart of the text classification process based on a pre-trained model, according to an embodiment of the present invention.

[0030] Figure 5This is a flowchart of the domain-difference self-learning fine-tuning text classification method based on a pre-trained model, according to an embodiment of the present invention.

[0031] Figure 6 This is a flowchart of the fine-tuning training process of the pre-trained model in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] To facilitate understanding, the terms used in the embodiments of this application will be explained below.

[0034] BERT model: Released by Google in 2018, this language model trains deep bidirectional representations by jointly using bidirectional transformers across all layers. BERT combines the advantages of numerous natural language processing models, achieving superior performance in various natural language processing tasks. In related technologies, the input vector of the BERT model is the sum of token embeddings, position embeddings, and segment embeddings. Token embeddings represent the vectorized representation of characters, position embeddings represent the position of characters in the text, and segment embeddings represent the order of sentences in the text.

[0035] Pre-training is a process of training a neural network model using a large dataset, enabling the model to learn general features from the dataset. The purpose of pre-training is to provide high-quality model parameters for subsequent training of the neural network model on a specific dataset. In this embodiment, pre-training refers to the process of training a BERT model using unlabeled training text.

[0036] Fine-tuning: A process of further training a pre-trained neural network model using a specific dataset. Typically, the dataset used in the fine-tuning stage is smaller than that used in the pre-training stage, and the fine-tuning stage employs supervised learning, meaning the training samples in the dataset used in the fine-tuning stage contain labeled information. In this embodiment, the fine-tuning stage refers to the process of training a BERT model (pre-trained) using training text containing classification labels.

[0037] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP technologies typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0038] The fine-tuned text classification method based on a pre-trained model and the domain-difference self-learning fine-tuned text classification method based on a pre-trained model provided in this application are applications in semantic understanding.

[0039] like Figure 3 and Figure 4 As shown, the specific implementation process is as follows: A pre-trained model, such as BERT, is used as the encoder to extract features from the training dataset. The extracted feature vectors are input into three modules: instance-level head, task-level head, and domain-level head, respectively, to obtain instance-level features, task-level features, and domain-level features. Supervised contrastive learning is performed on the obtained instance-level features to learn the fine-grained semantic differences between instances, resulting in the instance-level loss L. scl Simultaneously, classification learning is performed on the obtained task-level features to learn the differences between different tasks, resulting in a task-level loss L. ce Furthermore, weak clustering learning is performed on the obtained domain-level features to learn the latent semantic relationships between categories, resulting in a domain-level loss L. WCl The final loss L is obtained by weighted summation of these losses. t By calculating the final loss L t The network is updated using inverse gradients to optimize the model. After optimization, the fine-tuned model is used to classify and predict new text. A trained model, such as BERT, is used as the encoder to extract features from text data in test or real-world applications. The extracted feature vectors are input into the domain-level head module to obtain the confidence score of the test text in each category. The category with the highest confidence score among all categories is taken as the final prediction result. The main difference between this process and the classic pre-trained model fine-tuning for category prediction is that this invention inputs the pooled feature vectors into the domain-level head of this invention before classification. This enables the learning of semantic differences at different levels and utilizes implicit relationships between categories to assist classification, thereby improving the accuracy of text classification.

[0040] Figure 5 This is a flowchart of the domain-difference self-learning fine-tuning text classification method based on a pre-trained model, as described in this embodiment of the invention. Figure 5As shown, in this embodiment of the invention, the fine-tuning text classification method includes:

[0041] 101. Obtain the text to be tested;

[0042] In one possible implementation of this invention, the text to be tested is search text sent by the terminal, or it is content to be published provided by the terminal, or it is webpage text content extracted by the server. This application does not limit the specific method of obtaining the text to be tested.

[0043] 102. Use the fine-tuned pre-trained model to extract features from the text to be tested, and obtain the feature vector of the text to be tested;

[0044] In this embodiment, a finely tuned pre-trained model is used to extract features from the acquired test text. The encoder in the model can be used to extract the feature vector of the test text.

[0045] 103. Input the feature vector of the text to be tested into the domain-level header module to obtain the confidence score of the text to be tested in each category;

[0046] In this embodiment, the feature vector of the text to be tested is processed through a pooling layer, and then the feature vector after pooling is input into the domain-level head module. The domain-level head processes the feature vector to obtain the confidence score for each category.

[0047] 104. The category with the highest confidence among all categories is taken as the final prediction result of the text to be tested.

[0048] In this embodiment, since the text to be tested has a corresponding confidence level in each category, for example, assuming there are three categories, the text to be tested has a confidence level of 0.3 in category A, 0.4 in category B, and 0.7 in category C, then category C will be used as the final prediction result of the text to be tested.

[0049] Figure 6 This is a flowchart of the fine-tuning training process of the pre-trained model according to an embodiment of the present invention, as follows: Figure 6 As shown, the method includes:

[0050] 201. Obtain the text training set;

[0051] In this embodiment, the text training set can be a Chinese training sample set, and the training texts contained therein include classification labels. The text training set can be represented as x1, x2, ..., x n .

[0052] Regarding the method of obtaining the text training set, in one possible implementation, when the finally trained text classification model is used to classify the test text, the text training set can be a collection of historical texts.

[0053] In another possible implementation, the text training set can also be a general Chinese corpus dataset, such as the Chinese Natural Language Corpus (ChineseNLPcorpus). This application does not limit the method of obtaining the text training set.

[0054] 202. Use a pre-trained model to extract features from each training text in the text training set to obtain the feature vector of the training text;

[0055] In this embodiment of the invention, a pre-trained model, such as the BERT model, is used as an encoder to extract features from the text training set.

[0056] 203. Input the extracted training text feature vectors into the instance-level header, task-level header, and domain-level header modules respectively to obtain the instance-level features, task-level features, and domain-level features of the training text.

[0057] In this embodiment of the invention, the instance-level header, the task-level header, and the domain-level header all employ fully connected layers. The input and output dimensions of the instance-level header are the same; taking the BERT encoder as an example, the instance-level header is a 768×768 fully connected layer. The output dimension of both the task-level header and the domain-level header is the number of text categories. Assuming the number of categories is 10, these two header modules are two 768×10 fully connected layers.

[0058] In this embodiment of the invention, the domain-level head is actually a fully connected layer, which can be represented as y = wx + b, where y represents the output feature, x represents the input feature, w represents the weight of the domain-level head, and b represents the bias. To allow the model to pay more attention to challenging examples of each category, in the encoder latent space, most sentences of the same category tend to cluster because their text representations are similar. Classifying sentences with existing clustering trends is relatively easy, but classifying sentences of the same category but far from that clustering trend is more challenging. This invention can calculate the direction with the largest variance in the encoder latent space for examples of the same category in the training set, and use this direction vector as the class representation of that category. Using the PCA method, the direction with the largest variance in the latent space for examples of the same category is obtained, and the initialization method is as follows:

[0059] w0=PCA([enc(x1),enc(x2),…,enc(x m )])

[0060] Where w0 represents the initial weight of the domain-level head, PCA represents the common principal component analysis algorithm, and here only the first common principal component is taken, that is, the direction with the largest variance, x i Let x be the i-th instance sentence in the text training set, which is also the i-th training text, and let x be the i-th instance sentence. i ∈c, i = 1, 2, ..., m, where m is the number of all instances of class c in the training dataset, and enc is the pre-trained encoder.

[0061] 204. Supervised contrastive learning is performed on the obtained instance-level features to learn the fine-grained semantic differences between instances, and the instance-level loss is obtained.

[0062] In this embodiment of the invention, the loss function for supervised contrastive learning loss is as follows:

[0063]

[0064] Among them, L scl Indicates instance-level loss; It is in this batch that category y i The total number of instances with the same label, where I is an indicator function. Represents the training text x i With training text x j They are different texts, but both belong to the same category. i and category y i With the same tag, h i Represents the training text x i The feature vector h after encoding by the encoder j Represents the training text x j The feature vector h after encoding by the encoder k Represents the training text x k The feature vector after encoding by the encoder, where τ represents the temperature hyperparameter and N represents the batch size of the training texts used in the training.

[0065] 205. Perform classification learning on the obtained task-level features to learn the differences between different tasks and obtain the task-level loss;

[0066] In this embodiment of the invention, the loss function is the same as the classic cross-entropy classification loss, and the loss function is as follows:

[0067]

[0068] Among them, L ce Indicates task-level loss; y j ∈0,1 represents the training text x j The true category; Represents the training text x jThe predicted category represents the batch size of the training texts used in the training process.

[0069] 206. Weak clustering learning is performed on the obtained domain-level features to learn the latent semantic relationships between categories, and the domain-level loss is obtained.

[0070]

[0071] Among them, L WCl Represents domain-level loss; sim represents the similarity metric, the cosine similarity function used in this invention; h i h represents the i-th feature vector of the training text. j Let h represent the j-th feature vector of the training text. k y represents the k-th feature vector of the training text, N represents the batch size of the training texts, and M is the number of text categories; j ∈0,1 represents sentence x i The true category; λ is a hyperparameter.

[0072] Understandably, the proposed weak clustering loss is one of the main innovations of this invention. By using the proposed loss function, when calculating the loss of incorrect classes, we consider the relationship between them and correct classes, thereby enabling the model to learn continuous features, better uncovering the implicit relationships between classes, and achieving the effect of domain-specific self-learning.

[0073] 207. Instance-level loss L scl Mission-level loss L ce and domain-level loss L wcl The final loss L is obtained by weighted summation. t ;

[0074] In this embodiment, the final loss L t Represented as:

[0075] L t =λ1L scl +λ2L ce +λ3L wcl

[0076] Here, λ1, λ2, and λ3 represent different weights.

[0077] 208. By calculating the final loss, the network is updated with backward gradients to optimize the pre-trained model.

[0078] In this embodiment of the invention, the backpropagation of the network includes using the propagation error with relevant weights and a neighborhood-level loss during the backpropagation process until convergence is achieved and a preset false negative rate and a preset false positive rate are reached.

[0079] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A domain-differential self-learning fine-tuning text classification method based on a pre-trained model, characterized in that, The method includes: Obtain the text to be tested; use a fine-tuned pre-trained model to extract features from the text to obtain the feature vector of the text to be tested; input the feature vector of the text to be tested into the domain-level head module to obtain the confidence of the text to be tested in each category; take the category with the highest confidence among all categories as the final prediction result of the text to be tested. The fine-tuning training process for the pre-trained model includes: Obtain the text training set; A pre-trained model is used to extract features from each training text in the text training set to obtain the feature vector of the training text. The extracted training text feature vectors are input into three modules: instance-level header, task-level header, and domain-level header, respectively, to obtain instance-level features, task-level features, and domain-level features of the training text. Supervised contrastive learning is performed on the obtained instance-level features to learn the fine-grained semantic differences between instances, resulting in instance-level loss. The obtained task-level features are subjected to classification learning to learn the differences between different tasks, and the task-level loss is obtained. Weak clustering learning is performed on the obtained domain-level features to learn the potential semantic relationships between categories, resulting in a domain-level loss. instance-level loss Mission-level losses and domain-level losses The final loss is obtained by weighted summation; The instance-level loss The calculation formula is expressed as: in, This indicates the batch size of the training texts used in the training process. It is in the batch and category The total number of instances with the same label, where I is an indicator function. Represents training text With training text They are different texts, but both fall under different categories. and categories Having the same label express The feature vector after being encoded by the encoder express The feature vector after being encoded by the encoder Represents training text The feature vector after being encoded by the encoder Indicates temperature hyperparameter; The domain-level loss The calculation formula is expressed as: in, Represents a similarity measure. It represents the number of text categories. Representative sentence The true category, It is a hyperparameter; By calculating the final loss, the network is updated using backward gradients to optimize the pre-trained model.

2. The domain-difference self-learning fine-tuning text classification method based on a pre-trained model according to claim 1, characterized in that, The instance-level header, the task-level header, and the domain-level header all use fully connected layers; the input dimension and output dimension of the instance-level header are the same; the output dimension of the task-level header and the domain-level header are both the number of text categories.

3. The domain-difference self-learning fine-tuning text classification method based on a pre-trained model according to claim 1, characterized in that, The weight initialization method for the domain-level header is expressed as follows: ； in, This indicates the initial weight of the domain-level header. This represents the common principal component analysis algorithm, where only the first common principal component is selected, i.e., the direction with the largest variance is chosen. Let i be the i-th instance sentence in the text training set, which is also the i-th training text, and have , , For all categories in the training dataset The number of instances, This is a pre-trained encoder.

Images