Language model fine-tuning method, text classification method, device and equipment

Abstract

The embodiment of the present application provides a language model fine-tuning method, a text classification method, an apparatus and a device. The fine-tuning method includes: obtaining an input word vector, the input word vector includes: a training sample word vector of a training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word and a label word vector corresponding to a single label word; the training sample, the first template word and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence; inputting the input word vector into a pre-trained language model to obtain a word prediction result and an adjacent sentence judgment result for the first text sentence and the second text sentence; obtaining a first loss value based on the word prediction result and the true label word; obtaining a second loss value based on the adjacent sentence judgment result and the true judgment result; training the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model. The present application can improve the prediction performance of the language model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a language model fine-tuning method, a text classification method, an apparatus, and an equipment. Background Art

[0002] Language model pre-training and fine-tuning is a widely popular method for implementing downstream tasks. Specifically, by adding task-specific layers to an existing pre-trained model and then fine-tuning the parameters of the entire model, specific downstream tasks such as sentiment analysis, entity recognition, sentence similarity calculation, and other text classification tasks can be implemented.

[0003] Typically, the training tasks during the pre-training phase of a language model differ from those during the fine-tuning phase. Specifically, the tasks during pre-training and fine-tuning may be different. Alternatively, many language models may involve multiple training tasks during pre-training, thereby learning a variety of information contained in the text, while fine-tuning the model involves only a single training task.

[0004] The above-mentioned fine-tuning method will cause the prior knowledge learned in the pre-training stage of the language model to be forgotten. In other words, the above-mentioned fine-tuning method will make it impossible for the fine-tuning stage to make good use of the prior knowledge learned by the language model from the training samples in the pre-training stage. Therefore, the performance of the language model after fine-tuning is poor. Summary of the Invention

[0005] In view of this, embodiments of the present application provide a language model fine-tuning method, a text classification method, an apparatus, and a device to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present application, a language model fine-tuning method is provided, comprising:

[0007] Obtain an input word vector, the input word vector including: a training sample word vector of a training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence;

[0008] Inputting the input word vector into a pre-trained language model to obtain a word prediction result of the mask and an adjacent sentence judgment result for the first text sentence and the second text sentence;

[0009] A first loss value is obtained based on the word prediction result and the true label word; a second loss value is obtained based on the adjacent sentence judgment result and the true judgment result;

[0010] The pre-trained language model is trained according to the first loss value and the second loss value to obtain a trained language model.

[0011] According to a second aspect of an embodiment of the present application, a language model fine-tuning method is provided, which is applied to a server device, comprising:

[0012] receiving training samples sent by a client device;

[0013] generating an input word vector based on the training sample, the input word vector including: a training sample word vector of the training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence;

[0014] Inputting the input word vector into a pre-trained language model to obtain a word prediction result of the mask and an adjacent sentence judgment result for the first text sentence and the second text sentence;

[0015] A first loss value is obtained based on the word prediction result and the true label word; a second loss value is obtained based on the adjacent sentence judgment result and the true judgment result;

[0016] The pre-trained language model is trained according to the first loss value and the second loss value to obtain a trained language model, and the trained language model is returned to the client device.

[0017] According to a third aspect of an embodiment of the present application, a text classification method is provided, comprising:

[0018] Get the target text to be classified;

[0019] For each label word, obtain the prompt text vector corresponding to the label word, and input the prompt text vector into a pre-trained language model to obtain the word prediction result corresponding to the label word and the adjacent sentence judgment result; the prompt text vector includes: the target text word vector of the target text, the mask, the first template word vector, the second template word vector, and the label word vector of the label word;

[0020] Based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, the confidence information of the label word is obtained, and the confidence information represents the possibility that the label word is the true label word of the target text;

[0021] Determining the category label of the target text based on the confidence information of each label word;

[0022] The pre-trained language model is obtained by the method described in the first aspect or the second aspect.

[0023] According to a fourth aspect of an embodiment of the present application, a text classification method is provided, which is applied to a server device, including:

[0024] receiving a target text to be classified sent by a client device;

[0025] For each label word, obtain the prompt text vector corresponding to the label word, and input the prompt text vector into a pre-trained language model to obtain the word prediction result corresponding to the label word and the adjacent sentence judgment result; the prompt text vector includes: the target text word vector of the target text, the mask, the first template word vector, the second template word vector, and the label word vector of the label word;

[0026] Based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, the confidence information of the label word is obtained, and the confidence information represents the possibility that the label word is the true label word of the target text;

[0027] Determining a category label of the target text based on confidence information of each label word, and returning the category label to the client device;

[0028] The pre-trained language model is obtained by the method described in the first aspect or the second aspect.

[0029] According to a fifth aspect of an embodiment of the present application, a language model fine-tuning device is provided, comprising:

[0030] A first acquisition module is configured to acquire an input word vector, wherein the input word vector includes: a training sample word vector of a training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence;

[0031] A first prediction module is configured to input the input word vector into a pre-trained language model to obtain a word prediction result of the mask and an adjacent sentence judgment result for the first text sentence and the second text sentence;

[0032] A loss value obtaining module is configured to obtain a first loss value based on the word prediction result and the true label word; and obtain a second loss value based on the adjacent sentence judgment result and the true judgment result;

[0033] The first training module is used to train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model.

[0034] According to a sixth aspect of an embodiment of the present application, a language model fine-tuning apparatus is provided, which is applied to a server device and includes:

[0035] A training sample receiving module, configured to receive training samples sent by a client device;

[0036] a generating module, configured to generate an input word vector based on the training sample, the input word vector comprising: a training sample word vector of the training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence;

[0037] A second prediction module is configured to input the input word vector into a pre-trained language model to obtain a word prediction result of the mask and an adjacent sentence judgment result for the first text sentence and the second text sentence;

[0038] A loss value calculation module is configured to obtain a first loss value based on the word prediction result and the true label word; and obtain a second loss value based on the adjacent sentence judgment result and the true judgment result;

[0039] A second training module is used to train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model;

[0040] The model returning module is used to return the trained language model to the client device.

[0041] According to a seventh aspect of an embodiment of the present application, a text classification device includes:

[0042] The second acquisition module is used to obtain the target text to be classified;

[0043] A third prediction module is configured to obtain, for each tag word, a prompt text vector corresponding to the tag word, and input the prompt text vector into a pre-trained language model to obtain a word prediction result corresponding to the tag word and an adjacent sentence judgment result; the prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a tag word vector of the tag word;

[0044] A first confidence information obtaining module is used to obtain confidence information of the label word based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, wherein the confidence information represents the possibility that the label word is the true label word of the target text;

[0045] A first category label determination module, configured to determine the category label of the target text based on confidence information of each label word;

[0046] The pre-trained language model is obtained by the method described in the first aspect or the second aspect.

[0047] According to an eighth aspect of an embodiment of the present application, a text classification apparatus, applied to a server device, includes:

[0048] A target text receiving module, configured to receive a target text to be classified sent by a client device;

[0049] A fourth prediction module is configured to obtain, for each tag word, a prompt text vector corresponding to the tag word, and input the prompt text vector into a pre-trained language model to obtain a word prediction result corresponding to the tag word and an adjacent sentence judgment result; the prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a tag word vector of the tag word;

[0050] A second confidence information obtaining module is used to obtain confidence information of the label word based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, wherein the confidence information represents the possibility that the label word is the true label word of the target text;

[0051] A second category label determination module, configured to determine the category label of the target text based on the confidence information of each label word;

[0052] A category label returning module, configured to return the category label to the client device;

[0053] The pre-trained language model is obtained by the method described in the first aspect or the second aspect.

[0054] According to the ninth aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the language model fine-tuning method described in the first aspect or the second aspect, or operations corresponding to the text classification method described in the third aspect or the fourth aspect.

[0055] According to the tenth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the language model fine-tuning method as described in the first aspect or the second aspect, or the text classification method as described in the third aspect or the fourth aspect is implemented.

[0056] The language model fine-tuning method, text classification method, device and equipment provided in the embodiment of the present application, by adding prompt template words to the training sample, constructs the training sample into a mask containing the mask to be predicted and the text data of two text sentences, and then inputs the above text data into the pre-trained language model, so that the pre-trained language model simultaneously performs the word prediction (masked language modeling) task corresponding to the mask and the adjacent sentence judgment task, and retrains (fine-tunes) the language model according to the execution results of the above two tasks. That is to say, in the embodiment of the present application, by transforming the training sample, the training tasks in the language model fine-tuning stage are consistent with the training tasks in the pre-training stage, both of which include: masked language modeling tasks that tend to understand the semantics between word units within the text sentence, and adjacent sentence judgment tasks that tend to understand the semantics between adjacent text sentences. In this way, the semantic understanding ability between word units and the semantic understanding ability between adjacent text sentences learned by the language model in the pre-training stage can be fully utilized to effectively improve the performance of the language model that is finally trained. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0058] Figure 1 This is a flowchart of the steps of a language model fine-tuning method according to the first embodiment of the present application;

[0059] Figure 2 for Figure 1 A schematic diagram of an example scenario in the illustrated embodiment;

[0060] Figure 3 This is a flowchart of the steps of a language model fine-tuning method according to the second embodiment of the present application;

[0061] Figure 4 This is a flowchart of the steps of a language model fine-tuning method according to the third embodiment of the present application;

[0062] Figure 5 This is a flowchart of a text classification method according to the fourth embodiment of the present application;

[0063] Figure 6 This is a flowchart of a text classification method according to Embodiment 5 of the present application;

[0064] Figure 7 4 is a structural block diagram of a language model fine-tuning device according to Embodiment 6 of the present application;

[0065] Figure 8 1 is a structural block diagram of a language model fine-tuning device according to Embodiment 7 of the present application;

[0066] Figure 9 4 is a structural block diagram of a text classification device according to Embodiment 8 of the present application;

[0067] Figure 10 1 is a structural block diagram of a text classification device according to Embodiment 9 of the present application;

[0068] Figure 11 This is a structural diagram of an electronic device according to the tenth embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0070] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0071] Example 1

[0072] Reference Figure 1 , Figure 1 The following is a flow chart of the steps of a language model fine-tuning method according to the first embodiment of the present application. Specifically, the language model fine-tuning method provided in this embodiment includes the following steps:

[0073] Step 102, obtain the input word vector, which includes: the training sample word vector of the training sample, the first template word vector of the first template word, the mask, the second template word vector of the second template word, and the label word vector corresponding to the single label word; wherein, the training sample, the first template word and the mask constitute the first text sentence, and the second template word and the single label word constitute the second text sentence.

[0074] In the embodiments of the present application, both the first template word and the second template word can be pre-set word-grams, and the specific setting method of the first template word and the second template word is not limited here. Furthermore, discrete word-grams with clear semantics can be selected as the first template word or the second template word; continuous word-grams without clear semantics can also be selected as the first template word or the second template word, and the embodiments of the present application do not limit this.

[0075] The first template word can be the same word-gram or different word-grams. For example, the first template word and the second template word can be word-grams with similar semantics, and so on.

[0076] The label words in the embodiments of the present application are pre-set labels based on the specific downstream text classification task. For example, for the binary classification task of sentiment classification, the label words may be "positive" and "negative," or "positive" and "negative," etc. In the embodiments of the present application, there is no limitation on the method for setting the label words, and they can be set based on experience or relevant habits.

[0077] The input word vector is obtained by performing a word embedding operation on the sentence pair consisting of the first text sentence and the second text sentence.

[0078] For classification tasks, there are necessarily multiple label words. In this step, given a fixed training sample, a corresponding sentence pair can be constructed for each specific label word. This sentence pair includes: a first text sentence consisting of the training sample, a first template word, and a mask; and a second text sentence consisting of a second template word and the specific label word. Therefore, for N classification tasks, based on one training sample, N sentence pairs can be constructed for subsequent fine-tuning of the pre-trained model.

[0079] In addition, in the embodiment of the present application, the training samples, the first template words, the second template words and the label words can be Chinese words or other foreign words, such as English words, etc. Here, there is no limitation on the specific language form used by the text and words.

[0080] Step 104 : Input the input word vector into the pre-trained language model to obtain masked word prediction results and adjacent sentence judgment results for the first text sentence and the second text sentence.

[0081] The specific structure of the pre-trained language model is not limited in the embodiments of the present application. For example, it can be a transformer-based language model, such as BERT, RoBERTa, etc.

[0082] After inputting the input word vector into the pre-trained language model, the pre-trained language model can be used to perform the MLM (Masked Language Model) task and the NSP (Next Sentence Predict) task. The MLM task is used to predict the word unit at the masked position and obtain the masked word prediction result, that is, the probability of each word unit in the pre-trained model vocabulary at the masked position; the NSP task is used to determine whether the first text sentence and the second text sentence have a contextual relationship (that is, whether the second text sentence is the next sentence of the first text sentence, or whether the first text sentence is the previous sentence of the second text sentence, the latter is to say, whether the first text sentence and the second text sentence are adjacent sentences), and obtain the adjacent sentence judgment result, that is, the probability that the two text sentences have a contextual relationship, or the probability that the two text sentences do not have a contextual relationship, or the probability that the two text sentences are adjacent sentences.

[0083] Step 106: Obtain a first loss value based on the word prediction result and the true label word; obtain a second loss value based on the adjacent sentence judgment result and the true judgment result.

[0084] In this step, there is no limitation on the specific loss function used to obtain the first loss value or the second loss value. Any appropriate classification loss function can be selected according to the actual situation, such as negative log-likelihood loss function, cross entropy loss function, exponential loss function, square loss function, etc.

[0085] Step 108: Train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model.

[0086] Specifically, the first loss value and the second loss value can be fused to obtain a fused loss value, and then the pre-trained language model can be trained based on the fused loss value to obtain a trained language model.

[0087] In the embodiments of the present application, there is no limitation on the specific fusion strategy for the first loss value and the second loss value. For example, a first weight value corresponding to the first loss value and a second weight value corresponding to the second loss value can be set respectively, and then the first loss value and the second loss value are weightedly fused based on the first and second weight values ​​to obtain a fused loss value, etc.

[0088] See also Figure 2 , Figure 2 This is a schematic diagram of the scene corresponding to the first embodiment of the present application. Figure 2 The schematic diagram shown in the figure illustrates an embodiment of the present application using a specific scenario as an example:

[0089] See also Figure 2, the downstream task corresponding to this scenario is a sentiment classification task, the set label mapping is "good corresponds to positive; bad corresponds to negative", and the obtained training sample is "No reason to watch."; add the first template word: "Itwas_", the second template word: "Itwas_" and a single label word "bad" to the training sample (the label word can be any one of the preset label words "bad" and "good", and "bad" is only used as an example here, which does not constitute a limitation of the embodiment of the present application), and obtain a sentence pair: "No reason to watch. It was [MASK]. It was bad.", where "Noreason to watch. It was [MASK]" is the first text sentence, and "Itwas_" is the second template word. bad" is the second text sentence; perform word embedding operation on the above sentence pair to obtain the input word vector: "[CLS]e(No)e(reason)e(to)e(watch)e(.)e(It)e(was)e([MASK])[SEP]e(It)e(was)e(bad)[SEP]", where "[CLS]" is a special symbol representing the semantic information of the entire sentence pair, "[MASK]" is a mask symbol; "[SEP]" is a text sentence segmentation symbol; "e(No)" is the word vector corresponding to the word "No", "e(No)e(reason)e(to)e(watch)e(.)" is the training text word vector corresponding to the training text "No reason towatch.", and "e(It)e(was)" is both the first template word vector corresponding to the first template word and the second template word vector corresponding to the second template word; input the above input word vector into the pre-trained language model, and pass the "MLM The "head" module is used to predict the word "[MASK]" to obtain the word prediction result; the "NSP head" module used to perform the NSP task in the pre-trained language model is used to obtain the adjacent sentence judgment result based on "[CLS]"; a first loss value is obtained based on the word prediction result and the true label word of the training sample (in this example, "nagative"), and a second loss value is obtained based on the adjacent sentence judgment result and the true judgment result (in this example, "yes"); further, the pre-trained language model is trained based on the first loss value and the second loss value to obtain a trained language model.

[0090] According to the language model fine-tuning method provided in the embodiment of the present application, by adding prompt template words to the training samples, the training samples are constructed to include the mask to be predicted and the text data of two text sentences, and then the above text data is input into the pre-trained language model, so that the pre-trained language model simultaneously performs the word prediction (masked language modeling) task corresponding to the mask and the adjacent sentence judgment task, and the language model is re-trained (fine-tuned) according to the execution results of the above two tasks. That is to say, in the embodiment of the present application, by transforming the training samples, the training tasks in the language model fine-tuning stage are consistent with the training tasks in the pre-training stage, both of which include: masked language modeling tasks that tend to understand the semantics between word units within text sentences, and adjacent sentence judgment tasks that tend to understand the semantics between adjacent text sentences. In this way, the semantic understanding ability between word units and the semantic understanding ability between adjacent text sentences learned by the language model in the pre-training stage can be fully utilized to effectively improve the performance of the language model that is finally trained.

[0091] The language model fine-tuning method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: a server, a PC, etc.

[0092] Example 2

[0093] Reference Figure 3 , Figure 3 The following is a flow chart of the steps of a language model fine-tuning method according to the second embodiment of the present application. Specifically, the language model fine-tuning method provided in this embodiment includes the following steps:

[0094] Step 302: Acquire a training sample, a preset first template word, a preset second template word, and a plurality of preset label words.

[0095] The first template word and the second template word in the embodiment of the present application can both be pre-set word elements, and the specific setting method of the first template word and the second template word is not limited here. Furthermore, discrete word elements with clear semantics can be selected as the first template word or the second template word; or continuous word elements without clear semantics can be selected as the first template word or the second template word, and the embodiment of the present application does not limit this. The first template word can be the same word element or different word elements. For example, the first template word and the second template word can be word elements with similar semantics, and so on.

[0096] The label words in the embodiments of the present application are pre-set labels based on the specific downstream text classification task. For example, for the binary classification task of sentiment classification, the label words may be "positive" and "negative," or "positive" and "negative," etc. In the embodiments of the present application, there is no limitation on the method for setting the label words, and they can be set based on experience or relevant habits.

[0097] In addition, in the embodiment of the present application, the training samples, the first template words, the second template words and the label words can be Chinese words or other foreign words, such as English words, etc. Here, there is no limitation on the specific language form used by the text and words.

[0098] Step 304 : Selecting the real label words of the training sample and a preset number of non-real label words of the training sample from a plurality of preset label words as target label words.

[0099] Specifically, for classification tasks, there must be multiple label words, including real label words and non-real label words. In this step, whether it is a binary classification task or other multi-classification tasks, for each training sample, only all real label words and a preset number of non-real label words are selected as target label words.

[0100] The above-mentioned preset number can be any natural number that is smaller than the total number of non-real labels. For example, the total number of real label words can be taken as the above-mentioned preset number.

[0101] In this step, there is no limitation on the specific method of selecting a preset number of non-real label words from multiple non-real label words. For example, the method may be random selection or selection according to a preset rule, etc.

[0102] Step 306 : Combining word units based on the training sample, the first template word, the second template word, and the single target label word to obtain a combined sample.

[0103] For a fixed training sample, since the number of non-real label words in the target label words obtained in step 304 is reduced, the number of combined negative samples in the combined samples obtained in this step will also be reduced.

[0104] In the embodiment of the present application, through steps 304 and 306, only all combined positive samples and a preset number (a smaller number) of combined negative samples can be constructed for each training sample. In this way, in the subsequent language model fine-tuning process, the balance of the ratio of positive samples to negative samples can be maintained, avoiding the class imbalance problem caused by too many negative samples, which helps to improve the prediction performance of the final language model.

[0105] Step 308: Perform word embedding operation on the combined sample to obtain input word vector.

[0106] Among them, the input word vector includes: the training sample word vector of the training sample, the first template word vector of the first template word, the mask, the second template word vector of the second template word and the label word vector corresponding to the single target label word; among them, the training sample, the first template word and the mask constitute the first text sentence, and the second template word and the single target label word constitute the second text sentence.

[0107] Step 310 : Input the input word vector into the pre-trained language model to obtain masked word prediction results and adjacent sentence judgment results for the first text sentence and the second text sentence.

[0108] Step 312: obtain a first loss value based on the word prediction result and the true label word; obtain a second loss value based on the adjacent sentence judgment result and the true judgment result.

[0109] Step 314: Train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model.

[0110] For the specific implementation of steps 310 to 314, please refer to the corresponding contents in steps 104 to 108, which will not be repeated here.

[0111] In an embodiment of the present application, by adding a prompt template word to the training sample, the training sample is constructed to include a mask to be predicted and text data of two text sentences, and then the above text data is input into the pre-trained language model, so that the pre-trained language model simultaneously performs the word prediction (masked language modeling) task corresponding to the mask and the adjacent sentence judgment task, and the language model is retrained (fine-tuned) according to the execution results of the above two tasks. That is to say, in an embodiment of the present application, by transforming the training sample, the training tasks in the language model fine-tuning stage are consistent with the training tasks in the pre-training stage, both of which include: masked language modeling tasks that tend to understand the semantics between word units within the text sentence, and adjacent sentence judgment tasks that tend to understand the semantics between adjacent text sentences. In this way, the semantic understanding ability between word units and the semantic understanding ability between adjacent text sentences learned by the language model in the pre-training stage can be fully utilized to effectively improve the performance of the language model that is finally trained.

[0112] In addition, in an embodiment of the present application, by selecting real label words and a preset number of non-real label words from multiple label words as target label words, it is possible to construct only all combined positive samples and a smaller number of combined negative samples for each training sample. In this way, in the subsequent language model fine-tuning process, the balance between the ratio of positive samples and negative samples can be maintained, avoiding the class imbalance problem caused by too many negative samples, which helps to improve the prediction performance of the final language model.

[0113] The language model fine-tuning method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: a server, a PC, etc.

[0114] Example 3

[0115] Reference Figure 4 , Figure 4 This is a flowchart of the steps of a language model fine-tuning method according to Example 3 of the present application. This embodiment can be applied in scenarios where a user provides training samples from their specific downstream tasks, and the corresponding server-side device fine-tunes the pre-trained language model based on the user-provided training samples, thereby returning a trained language model suitable for the user's needs to the user.

[0116] Specifically, the language model fine-tuning method provided in this embodiment includes the following steps:

[0117] Step 402: Receive a training sample sent by a client device.

[0118] Step 404: Generate an input word vector based on the training sample. The input word vector includes: the training sample word vector of the training sample, the first template word vector of the first template word, the mask, the second template word vector of the second template word, and the label word vector corresponding to the single label word; wherein, the training sample, the first template word and the mask constitute the first text sentence, and the second template word and the single label word constitute the second text sentence.

[0119] Step 406 : Input the input word vector into the pre-trained language model to obtain masked word prediction results and adjacent sentence judgment results for the first text sentence and the second text sentence.

[0120] Step 408: Obtain a first loss value based on the word prediction result and the true label word; obtain a second loss value based on the adjacent sentence judgment result and the true judgment result.

[0121] Step 410: Train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model, and return the trained language model to the client device.

[0122] In an embodiment of the present application, by adding a prompt template word to the training sample, the training sample is constructed to include a mask to be predicted and text data of two text sentences, and then the above text data is input into the pre-trained language model, so that the pre-trained language model simultaneously performs the word prediction (masked language modeling) task corresponding to the mask and the adjacent sentence judgment task, and the language model is retrained (fine-tuned) according to the execution results of the above two tasks. That is to say, in an embodiment of the present application, by transforming the training sample, the training tasks in the language model fine-tuning stage are consistent with the training tasks in the pre-training stage, both of which include: masked language modeling tasks that tend to understand the semantics between word units within the text sentence, and adjacent sentence judgment tasks that tend to understand the semantics between adjacent text sentences. In this way, the semantic understanding ability between word units and the semantic understanding ability between adjacent text sentences learned by the language model in the pre-training stage can be fully utilized to effectively improve the performance of the language model that is finally trained.

[0123] The language model fine-tuning method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: a server, a PC, etc.

[0124] Example 4

[0125] Reference Figure 5 , Figure 5 1 is a flowchart of a text classification method according to Example 4 of the present application. Specifically, the text classification method provided in this embodiment includes the following steps:

[0126] Step 502: Obtain the target text to be classified.

[0127] Step 504: For each label word, obtain the prompt text vector corresponding to the label word, and input the prompt text vector into the pre-trained language model to obtain the word prediction result corresponding to the label word and the adjacent sentence judgment result.

[0128] The prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a label word vector of the label word.

[0129] The pre-trained language model is obtained by any language model fine-tuning method in the above-mentioned embodiments 1 to 3.

[0130] Step 506: Based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, the confidence information of the label word is obtained.

[0131] Among them, the confidence information represents the possibility that the label word is the true label word of the target text.

[0132] Optionally, in some embodiments, the confidence information of the tag word may be obtained in the following specific manner:

[0133] Determining a first probability that the masked predicted word is the label word based on a word prediction result corresponding to the label word;

[0134] Determining a second probability that a third text sentence and a fourth text sentence are adjacent sentences based on a result of determining adjacent sentences corresponding to the label word; wherein the third text sentence includes: the target text, a first template word corresponding to the first template word vector, and the mask; and the fourth text sentence includes: a second template word corresponding to the second template word vector and the label word;

[0135] The first probability and the second probability are combined to obtain confidence information of the label word.

[0136] Specifically, the word prediction result corresponding to the label word may include: after performing word unit prediction on the mask position, the mask positions obtained are respectively the probability values ​​of each label word, that is, the word prediction result includes multiple probability values, each probability value corresponds to a label word, and is used to represent the probability that the mask position is the label word. The adjacent sentence judgment result corresponding to the label word may include: the probability value that the third text sentence and the fourth text sentence have a contextual relationship (being adjacent sentences), or the probability value that the third text sentence and the fourth text sentence do not have a contextual relationship (being not adjacent sentences).

[0137] In this step, the probability value that the mask position is the label word can be determined from the word prediction result corresponding to the label word as the first probability; from the adjacent sentence judgment result corresponding to the label word, the probability value that the third text sentence and the fourth text sentence are adjacent sentences can be determined as the second probability, and then the first probability and the second probability are fused to obtain the confidence information of the label word.

[0138] In the embodiments of the present application, there is no limitation on the specific method for fusing the first probability and the second probability. For example, a first probability weight value corresponding to the first probability and a second probability weight value corresponding to the second probability can be set separately, and then based on the first probability weight value and the second probability weight value, the first probability and the second probability are weightedly fused, and the fusion result is used as the confidence information of the label word.

[0139] Step 508: Determine the category label of the target text based on the confidence information of each label word.

[0140] In the embodiments of the present application, the specific method for determining the category label of the target text based on the confidence information of each label word is not limited, and an appropriate determination method can be selected according to actual needs. For example, a confidence threshold can be pre-set, and then the label words with confidence information greater than the above confidence threshold are determined as the category label of the target text. When there are multiple label words with confidence information greater than the above confidence threshold, the label word corresponding to the maximum confidence information can be selected as the category label of the target text, and so on.

[0141] The language model in the embodiment of the present application is obtained by simultaneously training the word prediction (masked language modeling) task corresponding to the mask and the adjacent sentence judgment task. That is to say, in the language model fine-tuning stage, the training samples are transformed so that the training tasks in the language model fine-tuning stage are consistent with the training tasks in the pre-training stage, and both include: masked language modeling tasks that tend to understand the semantics between words within text sentences, and adjacent sentence judgment tasks that tend to understand the semantics between adjacent text sentences. In this way, the semantic understanding ability between words and the semantic understanding ability between adjacent text sentences learned by the language model in the pre-training stage can be fully utilized to effectively improve the performance of the language model that is finally trained. Therefore, the accuracy of the target text classification can be improved by classifying the target text based on the above-mentioned trained language model.

[0142] The text classification method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: a server, a PC, etc.

[0143] Example 5

[0144] Reference Figure 6 , Figure 6 This is a flowchart of the steps of a text classification method according to Example 5 of the present application. The application scenario of this embodiment can be: the user provides the target text to be classified in the classification task scenario, and the corresponding server-side device classifies the target text based on the fine-tuned (trained) language model, obtains the classification label of the target text, and returns it to the user.

[0145] Specifically, the text classification method provided in this embodiment includes the following steps:

[0146] Step 602: Receive the target text to be classified sent by the client device.

[0147] Step 604: For each label word, obtain the prompt text vector corresponding to the label word, and input the prompt text vector into the pre-trained language model to obtain the word prediction result corresponding to the label word and the adjacent sentence judgment result.

[0148] The pre-trained language model is obtained by any of the language model fine-tuning methods in the above-mentioned embodiments 1 to 3.

[0149] The prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a label word vector of the label word.

[0150] Step 606: Obtain confidence information of the label word based on the word prediction result corresponding to the label word and the adjacent sentence judgment result.

[0151] The confidence information represents the possibility that the label word is the true label word of the target text.

[0152] Step 608 : Determine the category label of the target text based on the confidence information of each label word, and return the category label to the client device.

[0153] The language model in the embodiment of the present application is obtained by simultaneously training the word prediction (masked language modeling) task corresponding to the mask and the adjacent sentence judgment task. That is to say, in the language model fine-tuning stage, the training samples are transformed so that the training tasks in the language model fine-tuning stage are consistent with the training tasks in the pre-training stage, and both include: masked language modeling tasks that tend to understand the semantics between words within text sentences, and adjacent sentence judgment tasks that tend to understand the semantics between adjacent text sentences. In this way, the semantic understanding ability between words and the semantic understanding ability between adjacent text sentences learned by the language model in the pre-training stage can be fully utilized to effectively improve the performance of the language model that is finally trained. Therefore, the accuracy of the target text classification can be improved by classifying the target text based on the above-mentioned trained language model.

[0154] The text classification method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: a server, a PC, etc.

[0155] Example 6

[0156] Reference Figure 7 , Figure 7 : This is a structural block diagram of a language model fine-tuning device according to embodiment 6 of the present application. The language model fine-tuning device provided in this embodiment of the present application includes:

[0157] A first acquisition module 702 is configured to acquire an input word vector, where the input word vector includes: a training sample word vector of the training sample, a first template word vector of the first template word, a mask, a second template word vector of the second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence;

[0158] A first prediction module 704 is configured to input the input word vector into a pre-trained language model to obtain a masked word prediction result and an adjacent sentence judgment result for the first text sentence and the second text sentence;

[0159] A loss value obtaining module 706 is configured to obtain a first loss value based on the word prediction result and the true label word; and obtain a second loss value based on the adjacent sentence judgment result and the true judgment result;

[0160] The first training module 708 is used to train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model.

[0161] Optionally, in some embodiments, the first obtaining module 702 is specifically configured to:

[0162] Obtaining a training sample, a preset first template word, a preset second template word, and a plurality of preset label words;

[0163] Combining word units based on the training sample, the first template word, the second template word, and a single label word to obtain a combined sample;

[0164] Perform word embedding operation on the combined sample to obtain the input word vector.

[0165] Optionally, in some embodiments, when the first acquisition module 702 performs the step of combining word units based on the training sample, the first template word, the second template word, and the single label word to obtain a combined sample, it is specifically configured to:

[0166] Selecting the true label words of the training samples and a preset number of non-true label words of the training samples from a plurality of preset label words as target label words;

[0167] Based on the training sample, the first template word, the second template word and the single target label word, word units are combined to obtain a combined sample.

[0168] The language model fine-tuning apparatus of the present embodiment is used to implement the corresponding language model fine-tuning method of the aforementioned method embodiment 1 or embodiment 2, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here. In addition, the functional implementation of each module in the language model fine-tuning apparatus of the present embodiment can refer to the description of the corresponding parts in the aforementioned method embodiment 1 or embodiment 2, and will not be described in detail here.

[0169] Example 7

[0170] See also Figure 8 , Figure 8This is a structural block diagram of a language model fine-tuning device according to Embodiment 7 of the present application. The language model fine-tuning device provided in the embodiment of the present application is applied to a server device, including:

[0171] The training sample receiving module 802 is configured to receive training samples sent by a client device;

[0172] A generation module 804 is configured to generate an input word vector based on the training sample, where the input word vector includes: a training sample word vector of the training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence;

[0173] A second prediction module 806 is configured to input the input word vector into the pre-trained language model to obtain a masked word prediction result and an adjacent sentence judgment result for the first text sentence and the second text sentence;

[0174] The loss value calculation module 808 is configured to obtain a first loss value based on the word prediction result and the true label word; and obtain a second loss value based on the adjacent sentence judgment result and the true judgment result;

[0175] A second training module 810 is configured to train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model;

[0176] The model returning module 812 is configured to return the trained language model to the client device.

[0177] The language model fine-tuning apparatus of the present embodiment is used to implement the corresponding language model fine-tuning method of the aforementioned method embodiment 3, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here. In addition, the functional implementation of each module in the language model fine-tuning apparatus of the present embodiment can refer to the description of the corresponding parts in the aforementioned method embodiment 3, and will not be described in detail here.

[0178] Example 8

[0179] Reference Figure 9 , Figure 9 This is a structural block diagram of a text classification device according to Example 8 of the present application. The text classification device provided in this embodiment of the application includes:

[0180] The second acquisition module 902 is used to obtain the target text to be classified;

[0181] The third prediction module 904 is configured to obtain, for each tag word, a corresponding prompt text vector of the tag word, and input the prompt text vector into a pre-trained language model to obtain a word prediction result corresponding to the tag word and an adjacent sentence judgment result; the prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a tag word vector of the tag word;

[0182] A first confidence information obtaining module 906 is configured to obtain confidence information of the label word based on the word prediction result corresponding to the label word and the adjacent sentence judgment result. The confidence information represents the possibility that the label word is the true label word of the target text;

[0183] A first category label determination module 908 is configured to determine the category label of the target text based on the confidence information of each label word;

[0184] The pre-trained language model is obtained by any of the language model fine-tuning methods in the aforementioned method embodiments 1 to 3.

[0185] Optionally, in some embodiments, the first confidence information obtaining module 906 is specifically configured to:

[0186] Determine the first probability that the masked predicted word is the label word based on the word prediction result corresponding to the label word;

[0187] Determine a second probability that the third text sentence and the fourth text sentence are adjacent sentences based on the adjacent sentence judgment result corresponding to the label word; wherein the third text sentence includes: the target text, the first template word corresponding to the first template word vector, and the mask; and the fourth text sentence includes: the second template word corresponding to the second template word vector and the label word;

[0188] The first probability and the second probability are combined to obtain the confidence information of the label word.

[0189] The text classification device of the present embodiment is used to implement the corresponding text classification method of the aforementioned fourth method embodiment and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here. In addition, the functional implementation of each module in the text classification device of the present embodiment can refer to the description of the corresponding parts in the aforementioned fourth method embodiment, and will not be described in detail here.

[0190] Example 9

[0191] See also Figure 10 , Figure 10 This is a structural block diagram of a text classification device according to Example 9 of the present application. The text classification device provided in this embodiment of the present application is applied to a server device and includes:

[0192] The target text receiving module 1002 is used to receive the target text to be classified sent by the client device;

[0193] The fourth prediction module 1004 is configured to obtain, for each tag word, a corresponding prompt text vector of the tag word, and input the prompt text vector into a pre-trained language model to obtain a word prediction result corresponding to the tag word and an adjacent sentence judgment result; the prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a tag word vector of the tag word;

[0194] A second confidence information obtaining module 1006 is configured to obtain confidence information of the label word based on the word prediction result corresponding to the label word and the adjacent sentence judgment result. The confidence information represents the possibility that the label word is the true label word of the target text;

[0195] A second category label determination module 1008 is used to determine the category label of the target text based on the confidence information of each label word;

[0196] A category label returning module 1010 is configured to return the category label to the client device;

[0197] The pre-trained language model is obtained by any of the language model fine-tuning methods in the aforementioned method embodiments 1 to 3.

[0198] The text classification device of the present embodiment is used to implement the corresponding text classification method of the aforementioned method embodiment 5 and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here. In addition, the functional implementation of each module in the text classification device of the present embodiment can refer to the description of the corresponding parts in the aforementioned method embodiment 5, and will not be described in detail here.

[0199] Example 10

[0200] Reference Figure 11 , shows a structural diagram of an electronic device according to the tenth embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0201] like Figure 11 As shown, the electronic device may include: a processor (processor) 1102 , a communication interface (Communications Interface) 1104 , a memory (memory) 1106 , and a communication bus 1108 .

[0202] in:

[0203] The processor 1102 , the communication interface 1104 , and the memory 1106 communicate with each other via a communication bus 1108 .

[0204] The communication interface 1104 is used to communicate with other electronic devices or servers.

[0205] The processor 1102 is used to execute the program 1110, and specifically can execute the above-mentioned language model fine-tuning method, or the relevant steps in the text classification method embodiment.

[0206] Specifically, the program 1110 may include program codes, which include computer operation instructions.

[0207] The processor 1102 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0208] The memory 1106 is used to store the program 1110. The memory 1106 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0209] Program 1110 can be specifically used to enable the processor 1102 to perform the following operations: obtain an input word vector, the input word vector including: a training sample word vector of a training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence; input the input word vector into a pre-trained language model to obtain a masked word prediction result and an adjacent sentence judgment result for the first text sentence and the second text sentence; obtain a first loss value based on the word prediction result and the true label word; obtain a second loss value based on the adjacent sentence judgment result and the true judgment result; train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model.

[0210] Alternatively, the program 1110 may be specifically configured to cause the processor 1102 to perform the following operations: receive a training sample sent by a client device;

[0211] An input word vector is generated based on the training sample, and the input word vector includes: a training sample word vector of the training sample, a first template word vector of the first template word, a mask, a second template word vector of the second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence; the input word vector is input into a pre-trained language model to obtain a word prediction result of the mask and an adjacent sentence judgment result for the first text sentence and the second text sentence; a first loss value is obtained based on the word prediction result and the true label word; a second loss value is obtained based on the adjacent sentence judgment result and the true judgment result; the pre-trained language model is trained according to the first loss value and the second loss value to obtain a trained language model, and the trained language model is returned to the client device.

[0212] Alternatively, program 1110 can be specifically used to enable processor 1102 to perform the following operations: obtain the target text to be classified; for each label word, obtain the prompt text vector corresponding to the label word, and input the prompt text vector into a pre-trained language model to obtain the word prediction result corresponding to the label word and the adjacent sentence judgment result; the prompt text vector includes: the target text word vector of the target text, a mask, a first template word vector, a second template word vector, and the label word vector of the label word; based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, obtain the confidence information of the label word, and the confidence information represents the possibility that the label word is the true label word of the target text; based on the confidence information of each label word, determine the category label of the target text; wherein, the pre-trained language model is obtained by any language model fine-tuning method in the aforementioned method embodiments one to embodiment three.

[0213] Alternatively, program 1110 can be specifically used to enable processor 1102 to perform the following operations: receive a target text to be classified sent by a client device; for each label word, obtain a prompt text vector corresponding to the label word, and input the prompt text vector into a pre-trained language model to obtain a word prediction result corresponding to the label word and an adjacent sentence judgment result; the prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a label word vector of the label word; based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, obtain confidence information of the label word, the confidence information characterizing the possibility that the label word is the true label word of the target text; based on the confidence information of each label word, determine the category label of the target text, and return the category label to the client device; wherein, the pre-trained language model is obtained by any language model fine-tuning method in the aforementioned method embodiments one to three.

[0214] The specific implementation of each step in program 1110 can be found in the above-mentioned embodiment of the language model fine-tuning method, or the corresponding descriptions of the corresponding steps and units in the embodiment of the text classification method, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the above-mentioned method embodiments, and will not be repeated here.

[0215] Through the electronic device of this embodiment, by adding prompt template words to the training samples, the training samples are constructed to include the mask to be predicted and the text data of two text sentences, and then the above text data is input into the pre-trained language model, so that the pre-trained language model simultaneously performs the word prediction (masked language modeling) task corresponding to the mask and the adjacent sentence judgment task, and the language model is re-trained (fine-tuned) according to the execution results of the above two tasks. That is to say, in the embodiment of the present application, by transforming the training samples, the training tasks in the language model fine-tuning stage are consistent with the training tasks in the pre-training stage, both of which include: masked language modeling tasks that tend to understand the semantics between word units within text sentences, and adjacent sentence judgment tasks that tend to understand the semantics between adjacent text sentences. In this way, the semantic understanding ability between word units and the semantic understanding ability between adjacent text sentences learned by the language model in the pre-training stage can be fully utilized to effectively improve the performance of the language model that is finally trained.

[0216] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform operations corresponding to any language model fine-tuning method in the above-mentioned multiple method embodiments, or operations corresponding to the text classification method.

[0217] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0218] The above-described method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the language model fine-tuning method or the text classification method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the language model fine-tuning method or the text classification method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the language model fine-tuning method or the text classification method shown herein.

[0219] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0220] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A language model fine-tuning method, comprising: Obtain an input word vector, the input word vector including: a training sample word vector of a training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence; Inputting the input word vector into a pre-trained language model to obtain a word prediction result of the mask and an adjacent sentence judgment result for the first text sentence and the second text sentence; A first loss value is obtained based on the word prediction result and the true label word; a second loss value is obtained based on the adjacent sentence judgment result and the true judgment result; Training the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model; Wherein, obtaining the input word vector includes: performing a word embedding operation on a combined sample consisting of the first text sentence and the second text sentence to obtain the input word vector.

2. The method according to claim 1, wherein Performing a word embedding operation on a combined sample consisting of the first text sentence and the second text sentence to obtain the input word vector includes: Obtaining the training sample, a preset first template word, a preset second template word, and a plurality of preset label words; Performing word-gram combination based on the training sample, the first template word, and the mask to obtain the first text sentence, and performing word-gram combination based on the second template word and the single label word to obtain the second text sentence; the first text sentence and the second text sentence constitute the combined sample; Perform a word embedding operation on the combined sample to obtain an input word vector.

3. The method according to claim 2, wherein: The first text sentence is obtained by combining word units based on the training sample, the first template word, and the mask, and the second text sentence is obtained by combining word units based on the second template word and the single label word; The first text sentence and the second text sentence are used to form the combined sample, including: Selecting the real label words of the training sample and a preset number of non-real label words of the training sample from the multiple preset label words as target label words; Based on the training sample, the first template word and the mask, word gram combination is performed to obtain the first text sentence, and based on the second template word and the single target label word, word gram combination is performed to obtain the second text sentence; the first text sentence and the second text sentence constitute the combined sample.

4. A language model fine-tuning method, applied to a server device, comprising: receiving training samples sent by a client device; generating an input word vector based on the training sample, the input word vector including: a training sample word vector of the training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence; Inputting the input word vector into a pre-trained language model to obtain a word prediction result of the mask and an adjacent sentence judgment result for the first text sentence and the second text sentence; A first loss value is obtained based on the word prediction result and the true label word; a second loss value is obtained based on the adjacent sentence judgment result and the true judgment result; Training the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model, and returning the trained language model to the client device; Among them, generating the input word vector based on the training sample includes: performing a word embedding operation on a combined sample consisting of the first text sentence composed of the training sample, the first template word and the mask, and the second text sentence composed of the second template word and the single label word to obtain the input word vector.

5. A text classification method, comprising: Get the target text to be classified; For each label word, obtain the prompt text vector corresponding to the label word, and input the prompt text vector into the pre-trained language model to obtain the word prediction result corresponding to the label word and the adjacent sentence judgment result; The prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a label word vector of the label word; Based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, the confidence information of the label word is obtained, and the confidence information represents the possibility that the label word is the true label word of the target text; Determining the category label of the target text based on the confidence information of each label word; Wherein, the pre-trained language model is obtained by any method described in claims 1-4.

6. The method according to claim 5, wherein: The confidence information of the label word is obtained based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, including: Determining a first probability that the masked predicted word is the label word based on a word prediction result corresponding to the label word; Determining a second probability that a third text sentence and a fourth text sentence are adjacent sentences based on a result of determining adjacent sentences corresponding to the label word; wherein the third text sentence includes: the target text, a first template word corresponding to the first template word vector, and the mask; and the fourth text sentence includes: a second template word corresponding to the second template word vector and the label word; The first probability and the second probability are combined to obtain confidence information of the label word.

7. A text classification method, applied to a server device, comprising: receiving a target text to be classified sent by a client device; For each label word, obtain the prompt text vector corresponding to the label word, and input the prompt text vector into the pre-trained language model to obtain the word prediction result corresponding to the label word and the adjacent sentence judgment result; The prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a label word vector of the label word; Based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, the confidence information of the label word is obtained, and the confidence information represents the possibility that the label word is the true label word of the target text; Determining a category label of the target text based on confidence information of each label word, and returning the category label to the client device; Wherein, the pre-trained language model is obtained by any method described in claims 1-4.

8. A language model fine-tuning device, comprising: A first acquisition module is configured to acquire an input word vector, wherein the input word vector includes: a training sample word vector of a training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence; A first prediction module is configured to input the input word vector into a pre-trained language model to obtain a word prediction result of the mask and an adjacent sentence judgment result for the first text sentence and the second text sentence; A loss value obtaining module is configured to obtain a first loss value based on the word prediction result and the true label word; and obtain a second loss value based on the adjacent sentence judgment result and the true judgment result; A first training module is configured to train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model; Among them, the first acquisition module is used to obtain the input word vector by executing the following steps: performing a word embedding operation on a combined sample consisting of the first text sentence and the second text sentence to obtain the input word vector.

9. A language model fine-tuning device, applied to a server device, comprising: A training sample receiving module, configured to receive training samples sent by a client device; a generating module, configured to generate an input word vector based on the training sample, the input word vector comprising: a training sample word vector of the training sample, a first template word vector of a first template word, a mask, a second template word vector of a second template word, and a label word vector corresponding to a single label word; wherein the training sample, the first template word, and the mask constitute a first text sentence, and the second template word and the single label word constitute a second text sentence; A second prediction module is configured to input the input word vector into a pre-trained language model to obtain a word prediction result of the mask and an adjacent sentence judgment result for the first text sentence and the second text sentence; A loss value calculation module is configured to obtain a first loss value based on the word prediction result and the true label word; and obtain a second loss value based on the adjacent sentence judgment result and the true judgment result; A second training module is used to train the pre-trained language model according to the first loss value and the second loss value to obtain a trained language model; A model returning module, configured to return the trained language model to the client device; The generation module is used to generate an input word vector based on the training sample by executing the following steps: performing a word embedding operation on a combined sample consisting of the first text sentence composed of the training sample, the first template word and the mask, and the second text sentence composed of the second template word and the single label word to obtain the input word vector.

10. A text classification device comprising: The second acquisition module is used to obtain the target text to be classified; The third prediction module is used to obtain a prompt text vector corresponding to each label word, and input the prompt text vector into a pre-trained language model to obtain a word prediction result corresponding to the label word and an adjacent sentence judgment result; The prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a label word vector of the label word; A first confidence information obtaining module is configured to obtain confidence information of the label word based on a word prediction result corresponding to the label word and an adjacent sentence judgment result, wherein the confidence information represents the possibility that the label word is a true label word of the target text; A first category label determination module, configured to determine the category label of the target text based on confidence information of each label word; Wherein, the pre-trained language model is obtained by any method described in claims 1-4.

11. A text classification device, applied to a server device, comprising: A target text receiving module is used to receive the target text to be classified sent by the client device; A fourth prediction module is configured to obtain, for each label word, a prompt text vector corresponding to the label word, and input the prompt text vector into a pre-trained language model to obtain a word prediction result corresponding to the label word and an adjacent sentence judgment result; The prompt text vector includes: a target text word vector of the target text, a mask, a first template word vector, a second template word vector, and a label word vector of the label word; A second confidence information obtaining module is used to obtain confidence information of the label word based on the word prediction result corresponding to the label word and the adjacent sentence judgment result, wherein the confidence information represents the possibility that the label word is the true label word of the target text; A second category label determination module, configured to determine the category label of the target text based on the confidence information of each label word; A category label returning module, configured to return the category label to the client device; Wherein, the pre-trained language model is obtained by any method described in claims 1-4.

12. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, which enables the processor to perform operations corresponding to the language model fine-tuning method described in any one of claims 1-4, or to perform operations corresponding to the text classification method described in any one of claims 5-7.

13. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the language model fine-tuning method as described in any one of claims 1 to 4, or implements the text classification method as described in any one of claims 5 to 7.

14. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the language model fine-tuning method described in any one of claims 1 to 4, or to perform operations corresponding to the text classification method described in any one of claims 5 to 7.

Images