Text classification model training method and text classification method

Abstract

The invention relates to a text classification model training method and a text classification method. The training method comprises the following steps: for an original sample in a current training batch, inputting a constructed positive example data pair into a text classification model so as to obtain a feature vector pair of the positive example data pair through an encoder of the text classification model, the positive example data pair comprises data of an original sample and data of an expanded sample obtained after the original sample is expanded; determining a first cross entropy loss of the original sample, a second cross entropy loss of the expansion sample and a self-supervised contrast learning loss through the feature vector pair; and jointly determining a target loss through the first cross entropy loss, the second cross entropy loss and the self-supervised contrast learning loss, and training the text classification model through the target loss. The technical problems of poor generalization ability, model instability and the like caused by singly using cross entropy loss to train a text classification model in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a training method for a text classification model and a text classification method. Background Art

[0002] Text classification models have a wide range of applications in modern society, covering multiple fields, from business to technology to daily life. Text classification models usually use a pretrained language model (PLM), such as BERT, as a base encoder, and fine-tune the training on a specific dataset through the cross-entropy loss (CE). However, using only cross-entropy loss may lead to problems such as poor generalization ability and model instability. No effective solution has been proposed for this technical problem. Summary of the invention

[0003] The embodiments of the present application provide a training method for a text classification model and a text classification method, so as to at least solve the technical problems such as poor generalization ability and model instability caused by the single use of cross entropy loss to train the text classification model in the related art.

[0004] According to one aspect of an embodiment of the present application, a training method for a text classification model is provided, comprising: for original samples in a current training batch, inputting constructed positive data pairs into a text classification model to obtain a feature vector pair of the positive data pairs through an encoder of the text classification model, wherein the positive data pair includes data of the original sample and data of an expanded sample obtained by expanding the original sample; determining a first cross entropy loss of the original sample, a second cross entropy loss of the expanded sample, and a self-supervised contrastive learning loss through the feature vector pair; jointly determining a target loss through the first cross entropy loss, the second cross entropy loss, and the self-supervised contrastive learning loss, and training the text classification model through the target loss.

[0005] Optionally, the target loss is jointly determined by the first cross entropy loss, the second cross entropy loss and the self-supervised contrastive learning loss, including: weighting the first cross entropy loss, the second cross entropy loss and the self-supervised contrastive learning loss by specifying a weight to determine the target loss.

[0006] Optionally, the first cross entropy loss, the second cross entropy loss and the self-supervised contrastive learning loss are weighted by specifying weights to determine the target loss, including: multiplying the average of the first cross entropy loss and the second cross entropy loss by a first specified weight to obtain a first result; obtaining the product of the self-supervised contrastive learning loss and the second specified weight to obtain a second result; and summing the first result and the second result to determine the target loss.

[0007] Optionally, the second specified weight is greater than the first specified weight.

[0008] Optionally, the original sample is a natural language sample, and the positive data pair is constructed in the following manner: setting an expansion guide description, wherein the expansion guide description indicates an expansion task that does not change the semantics; concatenating the expansion guide description and the original sample to obtain a prompt word; inputting the prompt word into a large language model LLM to obtain the expanded sample; and combining the data of the original sample and the data of the expanded sample to obtain the positive data pair.

[0009] Optionally, the target loss satisfies the following formula: ,in, is the first cross entropy loss, For the second cross entropy loss, is the self-supervised contrastive learning loss, To specify weights.

[0010] According to another aspect of an embodiment of the present application, a text classification method is also provided, comprising: in response to receiving a text to be classified, classifying the text to be classified using a text classification model trained by the above method; and outputting a category of the text to be classified.

[0011] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: a processor, and a memory for storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the method in any of the above embodiments.

[0012] According to another aspect of the embodiments of the present application, a non-transitory machine-readable medium storing computer instructions is also provided, where the computer instructions are used to enable a computer to execute the method in any of the above embodiments.

[0013] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which, when executed by a processor of a computer, causes the computer to execute the method in any of the above embodiments.

[0014] Beneficial effects of the embodiments of the present application: In the embodiment of the present application, for the original samples in the current training batch, the constructed positive data pair is input into the text classification model to obtain the feature vector pair of the positive data pair through the encoder of the text classification model, wherein the positive data pair includes the data of the original sample and the data of the expanded sample obtained after the original sample is expanded; the first cross entropy loss of the original sample, the second cross entropy loss of the expanded sample and the self-supervised contrastive learning loss are determined through the feature vector pair; the target loss is jointly determined by the first cross entropy loss, the second cross entropy loss and the self-supervised contrastive learning loss, and the text classification model is trained by the target loss. That is to say, the embodiment of the present application uses the first cross entropy loss of the original sample, the second cross entropy loss of the expanded sample and the self-supervised contrastive learning loss to jointly train the text classification model, which solves the technical problems such as poor generalization ability and model instability caused by the single use of cross entropy loss to train the text classification model in the related art, and improves the performance of the text classification model.

[0015] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0017] Figure 1 is a flowchart of a method for training a text classification model according to an embodiment of the present application; Figure 2 is a structural diagram of a text classification model according to an embodiment of the present application; Figure 3 is a schematic diagram of a training structure of a text classification model according to an embodiment of the present application; Figure 4 is a schematic diagram of a process of expanding an original sample according to the LLM of an embodiment of the present application; Figure 5 is a schematic diagram of classification results of a text classification model according to an embodiment of the present application; Figure 6 is a structural block diagram of a training device for a text classification model according to an embodiment of the present application; Figure 7 is a flowchart of a text classification method according to an embodiment of the present application; Figure 8is a structural schematic diagram of a text classification device according to an embodiment of the present application; Fig. 9 Schematic diagram of the structure of the electronic device of this embodiment. DETAILED DESCRIPTION

[0018] Embodiments of the present embodiment will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present embodiment are shown in the accompanying drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, which are instead provided for a more thorough and complete understanding of the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not intended to limit the scope of protection of the present embodiment.

[0019] In order to better understand the embodiments of the present application, the relevant technologies involved in the embodiments of the present application are described below.

[0020] Text classification tasks based on PLM are a method that has achieved remarkable results in the field of natural language processing in recent years. It is also the most common implementation scheme at present. It is a model that performs unsupervised training on large-scale text corpora, with the goal of learning language representations in text. These models are usually based on neural networks, such as the Transformer architecture, and are pre-trained through tasks such as Masked Language Model (MLM) or Next Sentence Prediction (NSP). PLM can learn rich language knowledge and semantic representations.

[0021] Fine-tuning: When using PLM for text classification, fine-tuning is usually performed. Fine-tuning refers to the process of supervised learning on labeled text classification data based on PLM. Through fine-tuning, the model can be adapted to a specific text classification task and the model parameters can be adjusted to better meet the needs of the task.

[0022] Loss function: When training a text classification model, CE is usually used, which is calculated as the cross entropy between the probability distribution of the predicted label and the log probability of the actual label.

[0023] Using PLM for text classification tasks can leverage the language knowledge and semantic representation learned by PLM to improve the performance of text classification, especially when data is scarce or the annotation cost is high.

[0024] Large Language Model (LLM), also known as large language model, is based on transformer architecture like the above-mentioned PLM, but LLM usually refers to language models containing tens of billions (or more) parameters. They are trained on massive amounts of text data and use reinforcement learning from human feedback (RLHF) to optimize the language model to gain a deeper understanding of the language. LLM's emergent abilities. One of the most notable features that distinguishes LLM from previous pre-trained language models (PLM) is their emergent abilities. Emergent ability is a surprising ability that is not obvious in small models but is particularly prominent in large models. Similar to the phase transition phenomenon in physics, emergent ability is like the rapid improvement of model performance as the scale increases, exceeding the random level, which is what we often call quantitative change leading to qualitative change. Emergent abilities can be related to certain complex tasks, but this application is more concerned with its general capabilities. Next, we briefly introduce three typical emergent abilities of LLM: Contextual learning: The contextual learning capability was first introduced by the Generative Pre-trained Transformer (GPT-3). This capability allows a language model to perform tasks by understanding the context and generating corresponding outputs when provided with natural language instructions or multiple task examples, without the need for additional training or parameter updates.

[0025] Instruction Following: LLM is fine-tuned on multi-task data described in natural language, also known as instruction fine-tuning. LLM is shown to perform well on unseen tasks formally described using instructions. This means that LLM is able to perform tasks based on task instructions without having seen specific examples beforehand, demonstrating its strong generalization ability.

[0026] Step-by-step reasoning: Small language models often have difficulty solving complex tasks that involve multiple reasoning steps, such as math problems. However, LLMs solve these tasks by adopting a Chain of Thought (CoT) reasoning strategy, using a prompting mechanism that includes intermediate reasoning steps to arrive at the final answer. It is speculated that this ability may be acquired through training on code.

[0027] The above emergent capabilities enable LLMs to excel in handling a variety of tasks, making them powerful tools for solving complex problems and applying them in multiple fields.

[0028] Contrastive learning is a machine learning method whose theoretical basis is to learn data representation by comparing the differences between two or more samples. The core idea of ​​this method is to use the intrinsic structure of the data and the differences between the features to learn useful representations, rather than directly training through labels or guidance signals. One of the advantages of this method is that it does not require a large amount of labeled data, so it has great potential in unsupervised or semi-supervised learning tasks. In contrastive learning, a common practice is to learn representations by comparing positive samples with negative samples. Positive samples are similar pairs of samples, while negative samples are dissimilar pairs of samples. The model is trained to "pull" the representation of positive sample pairs and "push" the representation of negative sample pairs, so that the distance between similar samples in the representation space is small, while the distance between dissimilar samples is large. Among them, how to construct positive samples is the key to the algorithm. Data enhancement can be performed through methods such as rule conversion, mutual translation, and dropout to construct noisy positive sample data. In the field of natural language processing, contrastive learning has been widely used in various tasks, including text representation learning, language model pre-training, and text similarity calculation. Training PLM through contrastive learning allows the model to learn richer and more reasonable text representations, thereby achieving better performance in various natural language processing tasks.

[0029] At present, the commonly used text classification model in the industry adopts PLM, which is obtained by fine-tuning and training on a specific data set using cross entropy as the loss function. This text classification model has several problems: the cross entropy loss function is sensitive to outliers, which means that during the training process, outliers may have a greater impact on the text classification model, resulting in a decrease in the performance of the text classification model. At the same time, the boundary gap between samples of different categories in the text classification model trained with cross entropy is relatively small, which also leads to poor generalization ability of the text classification model. The fine-tuned pre-trained model has stability and robustness problems and is very easy to be attacked, resulting in prediction errors.

[0030] The shortcomings of data augmentation methods: For example, the use of dropout (a regularization technique used to prevent overfitting of neural networks. During training, dropout randomly sets the output of some neurons to zero, forcing the network to not rely too much on certain specific paths when predicting, but to make more use of diverse features) for text augmentation lacks diversity. Poor interpretability. Conventional methods lack diversity and are prone to errors.

[0031] How to effectively perform data augmentation? A key question in contrastive learning is “how to construct a positive data set In computer vision, high-quality images can be obtained by simply cropping, flipping, twisting, and rotating an image. , while in natural language processing, construct The method is similar to that of images. Generally, the original sentence is translated, rearranged, or some words are deleted, inserted, or replaced. However, due to the discreteness, data expansion in natural language processing (NLP) often introduces a lot of negative noise, making The quality of the data is difficult to guarantee. The dropout method implicitly constructs positive data pairs during model training. That is, The random dropout mask mechanism exists in the fully connected layer and attention probability of the model. The same input will get different results after passing through the model, which can be regarded as an expanded . Compared with the above methods, the effect is much better. But there are still some problems: Loss of semantic information: Dropout randomly sets some neuron outputs to zero during training, which may lead to the loss of some important semantic information, thus affecting the quality of the text representation learned by the model.

[0032] Generation instability: When generating text, due to the randomness of dropout, the same input may result in different outputs, which makes the generated text unstable.

[0033] Poor interpretability: Using dropout to expand text is done by activating neurons inside the neural network. It is difficult to analyze the accuracy of the constructed positive data.

[0034] Computational inefficiency: Dropout randomly discards the output of neurons in each forward propagation, which increases the computational overhead of the training process, especially when processing large-scale text data.

[0035] The dropout rate (usually represented by p) is not easy to determine the specific value: the impact of the size of p can be understood as the probability that the sentence vector with dropout will be offset relative to the sentence vector without dropout in the entire unit hypersphere. Because BERT is a multi-layer structure, each layer will have dropout. The accumulation of these noises will cause the sentence vector to be offset in each dimension. However, when p is small, the two vectors are still close in space, ensuring a stable alignment. However, in different data sets, how to choose the specific size of p is still uncertain and there is no theoretical support.

[0036] In order to solve the above technical problems, the embodiments of the present application provide relevant solutions, which are described in detail below. Among them, the application scenarios of the embodiments of the present application can be in the scenarios of text classification such as vertical domain classification, intent recognition, rejection, arbitration, etc. in the dialogue system. For example, including but not limited to: 1. Spam filtering. Identify and filter out spam from a large number of emails. For example, email service providers use text classification models to automatically classify spam into spam folders. Internal corporate email systems prevent spam from interfering with normal work.

[0037] 2. Social media content classification. Social media platforms generate a large amount of user-generated content every day, which needs to be classified for better management and utilization. Automatic tag recommendation: Recommend appropriate tags based on the content of the post. Sentiment analysis: Identify whether social media content is positive, negative or neutral. Content review: Automatically identify and mark harmful information such as hate speech, fake news, etc.

[0038] 3. Customer service and feedback analysis. Companies need to analyze a large amount of customer feedback in order to understand customer needs and opinions in a timely manner. Customer service email classification: Classify customer emails into complaints, inquiries, suggestions, etc. Automatic reply: Automatically generate appropriate replies based on customer questions or needs. Sentiment tendency analysis: Analyze the sentiment tendency in customer feedback to identify customer satisfaction issues in a timely manner.

[0039] 4. News classification. News websites and applications need to classify a large number of news articles so that users can quickly find the news they are interested in. News recommendation system: recommend appropriate news based on user browsing history and preferences. Content management: classify news by topic, such as sports, finance, entertainment, politics, etc.

[0040] 5. Legal document classification. There are a large number of documents in the legal field that need to be classified and managed, such as contracts, judgments, regulations, etc. Contract management: Classify contracts according to type and content for easy retrieval and management. Legal case classification: Classify legal cases by type to facilitate legal personnel to find and reference. Legal query: Automatically match relevant legal documents based on keywords or phrases entered by users.

[0041] 6. Medical and clinical text classification. In the medical field, there are many case reports, medical papers, and medical records, which need to be classified and processed. Symptom classification: Automatically identify and classify symptoms based on the descriptions in electronic medical records. Medical literature management: Classify massive amounts of medical research literature to facilitate medical researchers to find and reference. Clinical trial data management: Classify clinical trial text data for easy analysis and management.

[0042] 7. Product and service review analysis. On e-commerce platforms, a large number of user reviews need to be analyzed to understand users' views on products or services. Review classification: Classify reviews into positive, negative, and neutral reviews. Topic extraction: Extract the main issues or features that users are concerned about from the reviews, such as quality, price, logistics, etc. Automatically generate product ratings: Automatically generate comprehensive ratings for products or services based on user reviews.

[0043] 8. Automatic document classification and archiving. There will be a large number of documents within the enterprise that need to be classified, stored and archived for future search and use. Enterprise document management: Automatically classify and label documents according to their content. Intelligent filing system: Automatically archive documents to corresponding folders or databases according to document type and content. Search system: Improve the accuracy and efficiency of enterprise document search based on classification information.

[0044] 9. Automatic grading and homework classification in the education field. In the education field, teachers need to grade and classify a large number of student assignments. Homework classification: Automatically classify the homework submitted by students into different categories, such as mathematics, English, science, etc. Automatic grading: Automatically grade the homework according to the standard answers and grading rules. Feedback generation: Automatically generate personalized feedback information based on the homework content and grading.

[0045] 10. Chatbots and intelligent customer service. Intelligent customer service systems and chatbots need to identify the types of questions users ask and provide corresponding services. Question classification: Classify questions into different categories based on the content of the user's questions, such as technical support, account management, after-sales service, etc. Automatic response: Provide automatic response or transfer to relevant customer service personnel based on the classification results. Frequently Asked Questions (FAQ) system: Find relevant answers from the FAQ library and respond based on the content of the user's questions.

[0046] In the above application scenario, the present application embodiment provides the following Figure 1 The flowchart of the training method of the text classification model shown in FIG. 1 includes the following steps: Step S102, for the original samples in the current training batch, the constructed positive example data pair is input into the text classification model to obtain the feature vector pair of the positive example data pair through the encoder of the text classification model, wherein the positive example data pair includes the data of the original sample and the data of the expanded sample obtained by expanding the original sample.

[0047] It should be noted that the above text classification model framework includes PLM and classification layer network. The basic encoders of PLM include but are not limited to: BERT, RoBERTa, AlBERT, etc., among which the basic encoders share weights in the multi-task training process. Taking BERT as an example, Figure 2 As shown, BERT requires a series of words (tokens) as input (corresponding to Figure 2 When doing classification tasks, for each token sequence, BERT expects a special tag [CLS], which is the first token of each input sequence and represents the classification token. Figure 2 The E in the above example is the feature vector obtained by encoding T. The feature vector corresponding to the classification token after encoding plays a core role in integrating information and performing classification. Just like a normal encoder of Transformer, BERT takes a sequence of words as input, which continuously flows upward. Each layer applies self-attention and passes its result through a feedforward network, which is then passed to the next encoder. Each token is encoded as a hidden vector of the size of the hidden layer (768 in the hidden layer of the BERT Base version) For the text classification model of the embodiment of the present application, only the latent vector of the [CLS] token, that is, the above-mentioned feature vector, is concerned. When the original samples in the current training batch include multiple positive example data pairs, for example, Input into the text classification model to determine multiple pairs of feature vectors, for example, .

[0048] Optionally, in an embodiment of the present application, the construction method of the above-mentioned positive example data pairs includes but is not limited to: data enhancement, sliding window, clause or sentence splitting, domain knowledge, time series, repetition, sample alignment, extended context, mixed data, etc.

[0049] Step S104, determining a first cross entropy loss of the original sample, a second cross entropy loss of the expanded sample, and a self-supervised contrastive learning loss through the feature vector pair.

[0050] It should be noted that the self-supervised contrastive learning loss is a loss function with the property of self-discovery of difficult negative samples. For sample i, the contrastive learning loss will automatically give more penalties to difficult negative samples (negative samples with closer distances), that is, a larger gradient makes it far away from the positive sample. That is, the above self-supervised contrastive learning loss is to push texts of the same class closer and push texts of different classes apart. To understand the nature of the contrastive loss and the role of the temperature coefficient, one can imagine this situation as the force applied to different negative samples as point charges of the same pole at different distances. The closer the point charges are, the greater the Coulomb repulsion they receive, while the farther the point charges are, the smaller the repulsion they receive. In the text classification model of the embodiment of the present application, for a text, its positive example is the sample after expansion of the text, and the negative example can be the text expanded from other texts in the same training batch. Therefore, in the calculation formula of self-supervised contrastive learning loss, the numerator can be the similarity of the positive example pair, and the denominator is the similarity of the positive example pair plus all negative example pairs. Minimizing the above self-supervised contrastive learning loss function is to maximize the numerator while minimizing the denominator, that is, to maximize the similarity of the positive example pairs and minimize the similarity of the negative example pairs.

[0051] Cross entropy loss is a loss function widely used in classification problems, especially when dealing with multi-classification problems. It measures the difference between the probability distribution predicted by the model and the probability distribution of the true label. The cross entropy loss function guides the model training process by reducing the difference between the predicted probability distribution and the true probability distribution, so that the model can predict the target category more accurately. In an embodiment of the present application, the first cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the probability distribution of the true label for the above-mentioned original sample, and the second cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the probability distribution of the true label for the above-mentioned augmented sample.

[0052] Step S106, jointly determining a target loss by using the first cross entropy loss, the second cross entropy loss, and the self-supervised contrastive learning loss, and training the text classification model by using the target loss.

[0053] Through the above steps S102~S106, the first cross entropy loss of the original sample, the second cross entropy loss of the expanded sample and the self-supervised contrastive learning loss are used to jointly train the text classification model, which solves the technical problems of poor generalization ability and model instability caused by the single use of cross entropy loss to train the text classification model in the related art, and improves the performance of the text classification model.

[0054] In one possible implementation, the target loss is jointly determined by the first cross entropy loss, the second cross entropy loss and the self-supervised contrastive learning loss, including: step S11, the first cross entropy loss, the second cross entropy loss and the self-supervised contrastive learning loss are weighted by specifying weights to determine the target loss. It should be noted that the weighting method can be linear weighting. In addition, the above-mentioned specified weights are used to control the contribution value of each loss function in the total loss. Optionally, the above-mentioned specified weights include but are not limited to: a first specified weight, a second specified weight, and optionally, the sum of the first specified weight and the second specified weight can be 1. During the training process, the weights are dynamically adjusted according to the changes in each loss. For example, the weights can be adjusted according to the size of the loss value in each iteration to ensure that all losses have a similar impact on the overall goal.

[0055] When the above-mentioned designated weight includes the above-mentioned first designated weight and the above-mentioned second designated weight, step S11 may include: step S111, multiplying the average value of the first cross entropy loss and the second cross entropy loss by the first designated weight to obtain a first result; step S112, obtaining the product of the self-supervised contrastive learning loss and the second designated weight to obtain a second result; step S113, summing the first result and the second result to determine the target loss. Through steps S111 to S113, different types of loss functions can be weighted and combined to comprehensively optimize the performance of multiple target models.

[0056] When the sum of the first specified weight and the second specified weight is 1, the embodiment of the present application proposes that the second specified weight is greater than the first specified weight.

[0057] Optionally, the above target loss may satisfy the following formula 1: (Formula 1) in, is the first cross entropy loss, is the second cross entropy loss, is the self-supervised contrastive learning loss, is the specified weight. Optionally, the specified weight can be set to 0.3. The application architecture of the above formula 1 is as follows Figure 3 As shown. Figure 3 In the example above, after the original sample "Call dad and say we'll have dinner together in the evening" is input into the encoder BERT, the original sample feature vector can be obtained, and then the original sample feature vector can be calculated. After inputting the extended sample "Call dad and say let's have dinner together in the evening" into the encoder BERT, we can get the extended sample feature vector, and then we can calculate For all samples in the current training batch, self-supervised contrastive learning can be calculated , and finally the target loss can be determined according to the above formula 1.

[0058] Optionally, in the embodiment of the present application, the above It can be determined by the following formula 2: (Formula 2) Among them, C represents the number of categories in the data set, and N represents the number of data in a training batch. represents the true label of the i-th original sample in a batch in category c, It represents the probability that the i-th original sample output by the model belongs to the c-th category.

[0059] Above It can be determined by the following formula 3: (Formula 3) Among them, C represents the number of categories in the data set, and N represents the number of data in a training batch. represents the true label of the i-th augmented sample in category c in a batch, It represents the probability that the i-th augmented sample output by the model belongs to the c-th category.

[0060] Above It can be determined by the following formula 4: (Formula 4) in, represents the vector representation of i original samples after BERT encoding, that is, the latent vector corresponding to [CLS], that is, Figure 3 The output of BERT on the left, It represents the vector representation of the i-th augmented sample after BERT encoding, that is Figure 3 The output of BERT on the right. For the above eigenvector pair, represents the cosine similarity of two feature vectors, It is a temperature constant, which is used to control the degree of attention paid by the text classification model to difficult samples. It is not good to be too large or too small. The subscript i identifies the subscript of the sample.

[0061] High-quality positive samples should have the following characteristics: 1. Semantic information that is consistent with the original data. The newly added data and the original data need to have the same label to ensure the consistency of the data set, so that the text classification model can rely on stable and consistent annotations to learn the relationship between features and categories. In addition, it is even more necessary to ensure that there is the same semantic information. Randomly removing a word alone is likely to change the meaning of the entire sentence (such as removing a negative word). 2. Diversification is required. New data is needed in terms of replacement words, sentence patterns, and sentence patterns to enhance the generalization ability of the model. The method of exchanging words alone is relatively limited. 3. It must have correct grammatical structure and semantic information, and cannot contain obvious grammatical errors. In order to achieve the above requirements, the embodiment of the present application uses the capabilities of a large-scale language model to generate text that is different from the original text but has the same semantics, that is, the positive sample required by the embodiment of the present application. Optionally, the above original sample is a natural language sample, and the above positive example data pair is constructed in the following way: Step S21, setting an expansion guide description, the expansion guide description indicates an expansion task that does not change the semantics; step S22, splicing the expansion guide description and the original sample to obtain a prompt word; step S23, inputting the prompt word into the large language model LLM to obtain the expanded sample; step S24, combining the data of the original sample and the data of the expanded sample to obtain the positive data pair. For example, the embodiment of the present application splices the original sample with the original sample through a carefully designed expansion guide description (prompt) and inputs it into the large model. The embodiment of the present application uses ChatGPT to allow ChatGPT to automatically generate a reply. Specifically, for Chinese data, the prompt is: "Please help me rewrite the following sentence slightly, keeping the semantics unchanged:", and the original sample that needs to be expanded is spliced ​​after the colon. For English data, the prompt is: "Please rephrase the following sentence slightly: ", and the original sample that needs to be expanded is spliced ​​after the colon. As Figure 4 As shown in the figure, "call dad and say let's have dinner together in the evening" is processed into "call dad and say let's have dinner together in the evening", and "Holden Caulfield did it better" is processed into "Holden Caulfield did a better job". They are not the same original samples but have exactly the same semantics. That is, the data pair required by the embodiment of the present application . Using a large language model for data augmentation, and through carefully constructed prompts, we successfully generated high-quality similar positive samples. This positive sample was applied to contrastive learning, and the experiment proved its effectiveness. Figure 5As shown in the figure, taking the Stanford Sentiment Treebank 2 (SST-2) sentiment analysis dataset using the General Language Understanding Evaluation (GLUE) benchmark as an example, where Class A is negative movie reviews and Class B is positive movie reviews, it can be found that although only using cross entropy can correctly classify, the spatial distribution distance of the vectors of the two categories is small. After expanding the text through LLM, the model that combines cross entropy and self-supervised contrastive learning can not only correctly classify, but the spatial distribution distance of the vectors of the two categories is farther, which shows that the model will have better robustness.

[0062] After determining the above target loss, the text classification model can be trained using the target loss. Optionally, in an embodiment of the present application, it may include: step S31, calculating the gradient of the target loss with respect to the text classification model parameters; step S32, adjusting the parameters using gradient adjustment until the target loss converges. Specifically, the gradient of the loss function with respect to the model parameters can be calculated by calling the backward() function, and the model parameters can be updated through the optimizer. Through the above steps S31~S32, the gradient descent algorithm automates the parameter optimization process, reduces human intervention, and the target loss function will converge quickly and stably under appropriate settings.

[0063] The effects of the embodiments of the present application are described below in conjunction with specific experimental data.

[0064] The present application embodiment uses the SST-2 dataset and samples small data to verify the text classification model. Specifically, 200 pieces of training data and 500 pieces of verification data are randomly sampled. Using the same encoder, the results of using different loss functions and data expansion methods are compared, as shown in Table 1.

[0065]

[0066] Three models are shown in Table 1 above. Model 1 is a text classification model trained by a single cross entropy loss (+CE), Model 2 is a text classification model jointly trained by cross entropy loss, self-supervised contrastive learning and dropout (+CE+SCL+dropout), and Model 3 is a text classification model jointly trained by the first cross entropy loss, the second cross entropy loss and the self-supervised contrastive learning loss proposed in the embodiment of the present application, wherein the augmented sample related to the second cross entropy loss uses GPT-aug (+CE+SCL+GPT-aug). The accuracy of Model 1 is 0.8922, the recall rate is 0.8920, and the precision rate is 0.8925. The index considering the precision and recall rate is 0.8921. The accuracy of Model 2 is 0.7913, the recall rate is 0.7915, and the precision rate is 0.7915. The index considering the precision and recall rate is 0.7913. The accuracy of model 3 is 0.9059, the recall rate is 0.9058, and the precision rate is 0.9063. The comprehensive index of precision and recall rate is 0.9059.

[0067] According to the experiment in Table 1 above, the following conclusions can be drawn: The model (+CE+SCL+GPT-aug) proposed in the embodiment of the present application achieved the best results. It is better than the result of using cross entropy alone. The method of text expansion based on LLM in the embodiment of the present application has better results than the method of text expansion based on dropout. In the scenario of a small data set (only 200 training data), the model proposed in the embodiment of the present application also achieved a very obvious improvement in results! In summary, the embodiment of the present application proposes a text classification framework that combines the use of LLM for text enhancement to do self-supervised contrastive learning and fine-tune PLM. The main contents include: The principle of the effectiveness of contrastive learning is that semantically similar samples are sampled from the same latent class, which ensures that the embodiment of the present application can obtain high-quality vector representations by contrastive learning to train the encoder, and a reasonable, high-quality vector representation can positively help the classification model. Therefore, the embodiment of the present application no longer uses cross entropy alone to train the model, and will use self-supervised contrastive learning loss to train the encoder at the same time, and use cross entropy to train the text classification model, and combine the two parts through a specific formula, which significantly improves the performance of the model. In order to solve the problem of using dropout and rules for data enhancement, this case uses a large language model for data enhancement, and successfully generates high-quality similar positive samples through carefully constructed prompts. This positive sample is applied to contrastive learning, and the experiment proves its effectiveness.

[0068] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a training device for a text classification model. Figure 6 The figure is a structural block diagram of a training device for a text classification model according to an embodiment of the present application, comprising: An input module 602 inputs the constructed positive data pair to the text classification model for the original sample in the current training batch, so as to obtain a feature vector pair of the positive data pair through an encoder of the text classification model, wherein the positive data pair includes data of the original sample and data of an expanded sample obtained by expanding the original sample; A determination module 604 determines a first cross entropy loss of the original sample, a second cross entropy loss of the expanded sample, and a self-supervised contrastive learning loss through the feature vector pair; The training module 606 jointly determines a target loss by using the first cross entropy loss, the second cross entropy loss, and the self-supervised contrastive learning loss, and trains the text classification model by using the target loss.

[0069] In a possible implementation, the training module 606 includes a determination unit, configured to weight the first cross entropy loss, the second cross entropy loss, and the self-supervised contrastive learning loss by specifying a weight to determine the target loss.

[0070] The determination unit includes: a first acquisition subunit, which is used to multiply the average value of the first cross entropy loss and the second cross entropy loss by a first specified weight to obtain a first result; a second acquisition subunit, which is used to obtain the product of the self-supervised contrastive learning loss and the second specified weight to obtain a second result; and a determination subunit, which is used to sum the first result and the second result to determine the target loss.

[0071] Optionally, the second specified weight is greater than the first specified weight. That is to say, the embodiment of the present application focuses more on the impact of self-supervised contrastive learning loss on the model, which is more conducive to optimizing the feature distance between samples of the same category, while expanding the feature distance between samples of different categories to improve the classification performance of the model.

[0072] The above original sample is a natural language sample, and the above positive data pair is constructed in the following manner: setting an expansion guidance description, which indicates an expansion task that does not change the semantics; concatenating the expansion guidance description and the original sample to obtain a prompt word; inputting the prompt word into the large language model LLM to obtain the expanded sample; combining the data of the original sample and the data of the expanded sample to obtain the positive data pair.

[0073] The training module 606 includes: a calculation unit, used to calculate the gradient of the target loss with respect to the text classification model parameters; and an adjustment unit, used to adjust the parameters using a gradient adjustment method until the target loss converges.

[0074] The above target loss satisfies the following formula: ,in, is the first cross entropy loss, is the second cross entropy loss, is the self-supervised contrastive learning loss, To specify weights.

[0075] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0076] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a text classification method. Figure 7 As shown, including: Step S702, in response to receiving the text to be classified, classifying the text to be classified using a text classification model trained by a training method for a text classification model; Step S704: output the category of the text to be classified.

[0077] Through the above steps S702~S704, the text classification model is jointly trained using cross entropy loss and self-supervised contrastive learning. Contrastive learning aims to perform pattern recognition and feature learning by learning the similarities and differences between objects, and the data enhancement method is a very critical link in contrastive learning. Too much change will affect the semantics, and too little will have no effect, and an appropriate degree of control is required. The embodiment of the present application uses a method based on a large model (LLM), which can better enhance the data and improve the effect of the model. At the same time, the embodiment of the present application also proves the effectiveness of the model in the case of small samples through experiments. In real production, there is often a lack of a large amount of high-quality supervised data, which makes it difficult to train a high-quality classification model. In the algorithm framework proposed in the embodiment of the present application, the training data required for contrastive learning is automatically constructed through a large model (LLM), and then the training of the classification model is combined with the contrastive learning algorithm. At the same time, the model can learn a more reasonable feature representation, improve robustness, and have less dependence on labeled data. It is very suitable for small sample situations in real production and has practical significance.

[0078] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a text classification device. Figure 8 The figure is a structural block diagram of a text classification device according to an embodiment of the present application, comprising: A classification module 802 is used for classifying the text to be classified using a text classification model trained by a training method for a text classification model in response to receiving the text to be classified; The output module 804 is used to output the category of the text to be classified.

[0079] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0080] The embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program executable by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the present application when executed by the at least one processor.

[0081] An embodiment of the present application also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0082] An embodiment of the present application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present application.

[0083] refer to Fig. 9 , the structural block diagram of the electronic device that can be used as the server or client of the embodiment of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0084] like Fig. 9As shown, the electronic device includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 909. An input / output (I / O) interface 905 is also connected to the bus 909.

[0085] Multiple components in the electronic device are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 can be any type of device that can input information to the electronic device, and the input unit 906 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 907 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 can include but is not limited to a disk, an optical disk. The communication unit 909 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0086] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present application may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via a ROM 902 and / or a communication unit 909. In some embodiments, the computing unit 901 may be configured to perform the above method in any other appropriate manner (e.g., by means of firmware).

[0087] The computer program for implementing the method of the embodiment of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0088] In the context of the embodiments of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, or infrared system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0089] It should be noted that the term "including" and its variations used in the embodiments of the present application are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present application are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0090] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0091] The various steps described in the method implementation methods provided in the embodiments of the present application can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present application is not limited in this respect.

[0092] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0093] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. Training methods for text classification models, including: For the original samples in the current training batch, the constructed positive example data pairs are input into the text classification model to obtain the feature vector pairs of the positive example data pairs through the encoder of the text classification model, wherein the positive example data pairs include the data of the original samples and the data of the expanded samples obtained by expanding the original samples; Determine a first cross entropy loss of the original sample, a second cross entropy loss of the expanded sample, and a self-supervised contrastive learning loss through the feature vector pair; A target loss is jointly determined by the first cross entropy loss, the second cross entropy loss, and the self-supervised contrastive learning loss, and the text classification model is trained by the target loss.

2. The method according to claim 1, jointly determining the target loss by using the first cross entropy loss, the second cross entropy loss, and the self-supervised contrastive learning loss, comprising: The target loss is determined by weighting the first cross entropy loss, the second cross entropy loss, and the self-supervised contrastive learning loss by specifying a weight.

3. The method according to claim 2, weighting the first cross entropy loss, the second cross entropy loss, and the self-supervised contrastive learning loss by specifying a weight to determine the target loss, comprising: Multiplying an average of the first cross entropy loss and the second cross entropy loss by a first specified weight to obtain a first result; Obtaining a product of the self-supervised contrastive learning loss and a second specified weight to obtain a second result; The first result and the second result are summed to determine the target loss. The method of claim 3 , wherein the second specified weight is greater than the first specified weight.

5. According to the method of claim 1, the original sample is a natural language sample, and the positive example data pair is constructed in the following manner: Setting an expansion guide description, wherein the expansion guide description indicates an expansion task that does not change semantics; splicing the expanded guide description and the original sample to obtain a prompt word; Inputting the prompt word into the large language model LLM to obtain the expanded sample; The data of the original sample and the data of the expanded sample are combined to obtain the positive example data pair.

6. According to the method according to any one of claims 1 to 5, the target loss satisfies the following formula: ,in, is the first cross entropy loss, For the second cross entropy loss, is the self-supervised contrastive learning loss, To specify weights.

7. Text classification methods, including: In response to receiving the text to be classified, classifying the text to be classified using the text classification model trained by the method according to any one of claims 1 to 6; Output the category of the text to be classified.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images