Text classification model training method and device, readable storage medium and terminal
By improving the text classification model structure and training method, combining text classification submodel and loss function optimization, the problem of poor training effect of text classification model in the existing technology is solved, and the accuracy and predictability of text classification results are improved.
Patent Information
- Application Number
- CN202310761818.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-26
- Publication Date
- 2024-12-27
AI Technical Summary
The training methods of existing text classification models rely on encoder-decoder-based models, resulting in inaccurate classification results and inadequate optimization of training effects and inference results.
By improving the model structure, adding text classification submodels, and introducing two-part loss functions for weighting operations, optimizing the model training process, combining original text and prompt text for training, improving the model's understanding and generation ability.
The optimization of the model training effect is achieved, the accuracy of the pre-trained text classification model is improved, and the predictability and accuracy of subsequent text classification results are reduced, and the generation of irrelevant texts is reduced.
Smart Images
Figure CN119202240A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text classification, and in particular to a training method and device for a text classification model, a readable storage medium, and a terminal. Background Art
[0002] Text classification is one of the more common task types in text processing technology. It is widely used in various fields, such as entity recognition, spam filtering, news classification, part-of-speech tagging, etc.
[0003] In the existing related technologies, when implementing text classification, a large number of sample texts are usually annotated first to obtain sample text data and their corresponding text category labels to construct a training data set; then a supervised training method is used to input the training data set into the model to be trained to obtain a pre-trained text classification model. In this way, after any text to be classified is input into the pre-trained text classification model, the text can be automatically classified.
[0004] However, due to deficiencies in model architecture and training methods (for example, the models used in the prior art are usually based on encoder-decoder models, which rely on supervised text classification methods and the classification results are not accurate enough), the pre-trained text classification models obtained by the prior methods are not optimized enough. This will result in insufficient accuracy of the text classification results obtained when they are subsequently applied to the text classification process. Summary of the invention
[0005] The purpose of the embodiment of the present invention is to provide a training method for a text classification model, which improves the training effect of the model by improving the model structure and the training method, thereby obtaining a more optimized text classification model.
[0006] To achieve the above-mentioned purpose, an embodiment of the present invention provides a training method for a text classification model, comprising the following steps: determining an initialization text classification model, wherein the initialization text classification model includes an encoding sub-model, a decoding sub-model and a text classification sub-model; determining a training data set, wherein the training data set includes multiple text data and a category label for each text data; using a target loss function, iteratively training one or more of the following items to be optimized to obtain a pre-trained text classification model: the encoding sub-model, the decoding sub-model and the text classification sub-model; wherein, in each round of iteration, the text data in the training data set is input into the encoding sub-model for encoding, and then the encoded data is input into the text classification sub-model for text classification processing, and the encoded data and the category label of the text data are input into the decoding sub-model for decoding.
[0007] Optionally, the target loss function is obtained by weighted operation of a preset first text classification loss function and a second text classification loss function; during the iterative training process, the first text classification loss function is used to iteratively optimize the parameters of the encoding sub-model and the text classification sub-model, and the second text classification loss function is used to iteratively optimize the parameters of the encoding sub-model and the decoding sub-model.
[0008] Optionally, the target loss function is obtained by weighted summing the first text classification loss function and the second text classification loss function.
[0009] The objective loss function is expressed by the following expression:
[0010] Loss = w1 × L classifier +w2×L generate ;
[0011] w1+w2=1;
[0012] Among them, Loss represents the target loss function, L classifier represents the first text classification loss function, L generate represents the second text classification loss function, w1 represents the weight of the first text classification loss function, and w2 represents the weight of the second text classification loss function.
[0013] Optionally, the weight of the first text classification loss function is the same as the weight of the second text classification loss function.
[0014] Optionally, the first text classification loss function is expressed by the following expression:
[0015]
[0016]
[0017] Among them, L classifier represents the first text classification loss function, N represents the total number of text data in the training data set, i represents the sequence number of the text data, L represents the total number of category labels corresponding to the N text data, j represents the sequence number of the category label, y ij represents the coefficient of the i-th text data. When the category label of the i-th text data is the j-th category label, y ij The value of y is 1, otherwise ij The value of p is 0, ij Indicates the probability that the current predicted label of the i-th text data is the j-th category label, It represents the first state matrix output by the text classification sub-model after the i-th text data is input into the encoding sub-model of the initialized text classification model, wherein the category label is the encoded label obtained by encoding the original category label.
[0018] Optionally, the second text classification loss function is expressed by the following expression:
[0019]
[0020]
[0021] Among them, L generate represents the second text classification loss function, W represents the total number of characters included in the original category label, w represents the character sequence number in the original category label, V represents the total number of characters included in the character set, v represents the character sequence number in the character set, represents the coefficient of the w-th character in the original category label, when the w-th character is the v-th character in the character set, The value of is 1, otherwise The value of p is 0, wv represents the probability that the wth character in the current predicted label is the vth character in the character set, It represents the second state matrix output by the decoding sub-model after the w-th character of the original category label is input into the decoding sub-model of the initialized text classification model.
[0022] Optionally, each piece of text data is obtained by concatenating the original text and the prompt text; the prompt text is used to indicate the training data set and the target task of the initialized text classification model.
[0023] Optionally, the text classification sub-model is selected from any one of the following: a network structure consisting of multiple linear layers, Dropout layers and activation functions; a model consisting of multiple self-attention mechanism modules; a model consisting of a single linear classification layer.
[0024] An embodiment of the present invention also provides a training device for a text classification model, comprising: an initialization model determination module, used to determine the initialization text classification model, the initialization text classification model includes an encoding sub-model, a decoding sub-model and a text classification sub-model; a training data set determination module, used to determine the training data set, the training data set includes multiple text data and a category label for each text data; a model training module, used to use a target loss function to iteratively train one or more of the following items to be optimized to obtain a pre-trained text classification model: the encoding sub-model, the decoding sub-model and the text classification sub-model; wherein, in each round of iteration, the text data in the training data set is input into the encoding sub-model for encoding, and then the encoded data is input into the text classification sub-model for text classification processing, and the encoded data and the category label of the text data are input into the decoding sub-model for decoding.
[0025] An embodiment of the present invention further provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the training method of the above-mentioned text classification model are executed.
[0026] An embodiment of the present invention further provides a terminal, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above-mentioned text classification model training method when running the computer program.
[0027] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:
[0028] In the prior art, a pre-trained text classification model is usually obtained by training based on an Encoder-Decoder model (e.g., Bart model, T5 model, etc.). This model relies on a supervised text classification method, and the training effect and model reasoning results are not optimized enough. In contrast, in an embodiment of the present invention, the model structure is improved and expanded, and a text classification sub-model is added on the basis of the Encoder-Decoder-based model architecture, so that the initialization text classification model used in this embodiment includes both an "Encoder+Decoder" structure and an "Encoder+text classification sub-model" structure, and can simultaneously perform supervised text classification and generative text classification. Thus, the optimization of the model training effect is achieved through the improvement of the model structure.
[0029] Further, in an embodiment of the invention, the loss function of the training process is also optimized. Specifically, by introducing two parts of loss functions (i.e., the first text classification loss function and the second text classification loss function), and performing weighted operations on the two parts of loss functions, the target loss function of the training process is obtained, and the two parts of loss functions are respectively used to optimize the parameters of different sub-model combinations in the initialized text classification model. Therefore, compared with the use of a single loss function for model training, this implementation scheme integrates two parts of loss, which can not only strengthen the understanding of the input text by the encoding sub-model part, but also strengthen the connection between the encoding sub-model input text data and the category label input by the decoding sub-model, so that the text classification results generated by the decoding sub-model part are closer to the real category of the text, and reduce the generation of new irrelevant texts, thereby helping to improve the refinement of model training and obtain a more optimized pre-trained text classification model. Further, based on a more optimized pre-trained text classification model, it is also helpful to improve the accuracy of the text classification results obtained by subsequent text classification using the model.
[0030] Furthermore, each piece of text data may be obtained by concatenating the original text and the prompt text; the prompt text is used to indicate the target task of the training data set and the initialized text classification model. In an embodiment of the present invention, prompt text is added on the basis of the original text and its category label to indicate the target task of the training data set and the initialized text classification model. Thus, the model can be trained as a guide for the target task indicated by the prompt text, so that the model training process is more targeted and the training effect meets expectations, which helps to avoid errors such as extracting keywords or generating summaries for the input text to be classified in the subsequent text classification process using the optimized text classification model obtained after training, thereby improving the predictability and accuracy of the text classification results. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 is a flowchart of a training method for a text classification model in an embodiment of the present invention;
[0032] Figure 2 is a structural diagram of an initialization text classification model in an embodiment of the present invention;
[0033] Figure 3 It is a structural schematic diagram of a training device for a text classification model in an embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and understandable, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0035] Reference Figure 1 , Figure 1 1 is a flowchart of a method for training a text classification model in an embodiment of the present invention. The method may include steps S11 to S13:
[0036] Step S11: Determine an initialization text classification model, wherein the initialization text classification model includes an encoding sub-model, a decoding sub-model and a text classification sub-model;
[0037] Step S12: determining a training data set, wherein the training data set includes a plurality of text data and a category label for each text data;
[0038] Step S13: adopting the target loss function, iteratively training one or more of the following items to be optimized to obtain a pre-trained text classification model: the encoding sub-model, the decoding sub-model and the text classification sub-model; wherein, in each round of iteration, the text data in the training data set is input into the encoding sub-model for encoding, and then the encoded data is input into the text classification sub-model for text classification processing, and the encoded data and the category label of the text data are input into the decoding sub-model for decoding.
[0039] In the specific implementation of step S11, the initialized text classification model can be a large model including a multi-layer sub-model structure obtained by constructing the encoding sub-model, the decoding sub-model and the text classification sub-model according to a preset hierarchical order.
[0040] Reference Figure 2 , Figure 2 It is a structural diagram of an initialization text classification model in an embodiment of the present invention.
[0041] The initialized text classification model may include an encoding sub-model 21, a decoding sub-model 22, and a text classification sub-model 23. The encoding sub-model 21 and the text classification sub-model 23 may constitute a first sub-model combination 213 (or referred to as a first sub-model module 213), and the encoding sub-model 21 and the decoding sub-model 22 may constitute a second sub-model combination 212 (or referred to as a second sub-model module 212).
[0042] As a non-limiting embodiment, the text classification sub-model 23 and the decoding sub-model 22 are respectively located in the latter layer of the encoding sub-model 21. During the model training process, the encoded data output by the encoding sub-model 21 is used as input data of the text classification sub-model 23 and combined with the category label of the text data as input data of the decoding sub-model 22.
[0043] Specifically, the encoding sub-model 21 can encode based on the input data to obtain encoded data (also referred to as encoded hidden state data). For example, the input text data (or binary data obtained by converting the text) can be encoded to obtain a matrix or vector containing several codes, wherein the matrix or vector obtained after encoding generally contains semantic information (or referred to as semantic features) contained in the input text data.
[0044] Specifically, the decoding sub-model 22 can decode based on the input data to obtain decoded data. For example, the text classification result can be output based on the matrix or vector output by the encoding sub-model 21 and the category label of the input text data.
[0045] Specifically, the text classification sub-model 23 can perform text classification based on the input data to obtain a text classification result. For example, the matrix or vector output by the encoding sub-model 21 can be processed to output a text classification result.
[0046] It should be pointed out that since both the decoding sub-model 22 and the text classification sub-model 23 can output text classification results, during the model inference process, one of them can be used as the final text classification result. For example, the output result of the decoding sub-model 22 can be used as the final text classification result.
[0047] Among them, the encoding sub-model 21 can adopt an existing model with data encoding function, for example, it can be a model based on an encoder (Encoder), including but not limited to: Bidirectional Encoder Representations from Transformer (BERT) model, A Robustly Optimized BERT (RoBERTa) model, etc.
[0048] The decoding sub-model 22 can adopt an existing model with data decoding function, for example, a decoder-based model, including but not limited to the second-generation generative pre-trained transformation model (Generative Pre-TrainedTransformer 2, referred to as GPT2), the third-generation generative pre-trained transformation model (Generative Pre-TrainedTransformer 3, referred to as GPT3), the fourth-generation generative pre-trained transformation model (Generative Pre-TrainedTransformer 4, referred to as GPT4), the large language model Bloom, etc.
[0049] In some embodiments, the text classification sub-model 23 may be a network structure consisting of multiple linear layers, Dropout layers (the Dropout layers may be disposed between at least a portion of the linear layers) and activation functions. By introducing multiple linear layers into the text classification sub-model 23, it is helpful to map the output data of the encoding sub-model 21 to a higher dimensional space, thereby improving the performance of the entire large model. Furthermore, by adding a Dropout layer, the overfitting problem during the training process can be prevented.
[0050] In other embodiments, the text classification sub-model 23 may be a model composed of multiple self-attention modules. By introducing multiple self-attention mechanisms into the text classification sub-model 23, it is possible to perform weighted summation on the output data of the encoding sub-model 21 to learn the important semantic information contained therein, thereby improving the accuracy of text classification.
[0051] In some other embodiments, the text classification sub-model 23 can directly adopt an existing classification network model with text classification function, or adopt a model composed of a single-layer linear classification layer. In this way, the construction process of the text classification sub-model can be omitted, and the construction efficiency of the entire large model can be improved.
[0052] It should be pointed out that, in this embodiment, the second sub-model combination 212 composed of the encoding sub-model 21 and the decoding sub-model 22 can adopt the existing encoder-decoder (Encoder-Decoder) based model, including but not limited to the bidirectional and auto-regressive transformation (Bidirectional and Auto-Regressive Transformers, referred to as Bart) model, T5 model, Chinese pre-trained unbalanced transformation (Chinese Pre-trained Unbalanced Transformer, referred to as CPT) model, etc.
[0053] Specifically, the initialized text classification model may further include a tokenizer 24, which includes a class for vectorizing text or converting text into a sequence (ie, decomposing a text stream into tokens, each token being generally a subsequence of characters in the text).
[0054] Continue to refer to Figure 1In the specific implementation of step S12, the category label of each text data can be manually annotated to improve the accuracy of data annotation as much as possible; or, the text data can be input into an existing text classification model to obtain the category label of the input text data to improve the efficiency of annotating the category label as much as possible. Different text data can correspond to the same category label.
[0055] In some embodiments, the category labels may be category labels corresponding to text data in different fields. For example, in the field of entertainment, the category labels may be selected from: music, movies, novels, operas, sketches, games, etc. For another example, in the field of teaching, the category labels may be selected from: Chinese, English, mathematics, history, geography, biology, chemistry, sports, etc.
[0056] In other embodiments, the category labels may also be category labels corresponding to text data of different branches in the same field. For example, in the field of games, text data of different game types (or game branches) may have their own corresponding category labels, which may be selected from: fighting games, makeup (dress-up) games, intellectual games, etc. For another example, in the field of novels, text data of novels from different countries may have their own corresponding category labels, which may be selected from: Chinese novels, British novels, American novels, French novels, etc.
[0057] Furthermore, each piece of text data may be obtained by concatenating an original text and a prompt text; the prompt text is used to indicate the training data set and the target task of the initialized text classification model.
[0058] Specifically, in the process of using the training data set to train the model, each text data (i.e., the original text and the prompt text) and its category label can be spliced, and the splicing result can be used as a single input data. Among them, the splicing order of the original text, the prompt text and the category label can be original text + prompt text + category label, or prompt text + original text + category label, or other appropriate splicing order. The embodiment of the present invention does not limit the splicing order of the three in each input data, as long as it can help the optimized text classification model after training to output the expected text classification result.
[0059] In a specific implementation, the prompt text may include prompt words, and there is a many-to-one or one-to-one relationship between the prompt words and the target task. For example, for the target task of "text classification", the corresponding prompt words may be selected from: "text classification", "category", "classification", "label", "text category", "text label".
[0060] In the embodiment of the present invention, the use of prompt words helps to more clearly and accurately indicate the target task of the training data set and the initialized text classification model, thereby ensuring the predictability of the model training effect.
[0061] In a specific embodiment, a certain input data in the training data set may be expressed as follows: "Have Zhang San and Li Si ever participated in a fight? What category is described in this text? Game". The original text is "Have Zhang San and Li Si ever participated in a fight?", the prompt text is "What category is described in this text?" (where the prompt word is "category"), and the category label is "game".
[0062] In another specific embodiment, during the splicing process, the order of the original text and the prompt text can be adjusted. For example, the input data can be expressed as follows: "What is the category of this text? Have Zhang San and Li Si participated in a fight? Game".
[0063] It should be pointed out that the names and contents contained in the above original text are only non-limiting examples. In specific implementations, the original text can be selected or set in combination with actual application scenarios.
[0064] Furthermore, the prompt words contained in the prompt text in each piece of input data, and the expression of the prompt text are not limited to the expression provided in the above embodiment, and the text design and language organization can be appropriately adjusted as long as they do not deviate from the target task that the training model of the training data set should achieve. For example, for the target task of text classification, the corresponding prompt text can also be expressed as "What is the label of this text?" (where the prompt word is "label"), "What is the text category of this paragraph" (where the prompt word is "text category"), "What is the text category of this paragraph" (where the prompt word is "text category"), and so on.
[0065] It is understandable that in practical applications, for some initialization models related to text processing, the target tasks (or functions) that can be achieved may be diverse, including but not limited to text classification, keyword extraction, text summary extraction, automatic question answering, etc. Specifically, the same initialized text processing model can be trained for keyword extraction, trained for text classification, trained for generating text summaries, and so on. Therefore, in the prior art, in the process of text classification using a trained text classification model, it may not be possible to output the expected and accurate text classification results. For example, after the text to be classified is input into a trained text classification model, the output result obtained is a summary or keyword of the text, thereby reducing the user experience.
[0066] To solve the above problems, in an embodiment of the present invention, a prompt text is added on the basis of the original text and its category label to indicate the target task of the training data set and the initialized text classification model. Thus, the model can be trained with the target task indicated by the prompt text as a guide, so that the model training process is more targeted and the training effect meets expectations, which helps to avoid errors such as extracting keywords or generating summaries for the input text to be classified in the subsequent text classification process using the optimized text classification model obtained after training, thereby improving the predictability and accuracy of the text classification results.
[0067] In the specific implementation of step S13, the pre-trained text classification model obtained by training with the target loss function and the training data set includes an encoding sub-model, a decoding sub-model and a text classification sub-model after parameter optimization.
[0068] Furthermore, the target loss function is obtained by weighted operation of a preset first text classification loss function and a second text classification loss function; in the iterative training process, the first text classification loss function is used to iteratively optimize the parameters of the encoding sub-model and the text classification sub-model (i.e., for optimizing Figure 2 The second text classification loss function is used to iteratively optimize the parameters of the encoding sub-model and the decoding sub-model (i.e., to optimize Figure 2 Parameters of the second sub-model combination 212 shown).
[0069] Furthermore, the target loss function is obtained by weighted summing the first text classification loss function and the second text classification loss function.
[0070] In a specific implementation, the objective loss function can be expressed by the following expression:
[0071] Loss = w1 × L classifier +w2×L generate ;
[0072] w1+w2=1;
[0073] Among them, Loss represents the target loss function, L classifier represents the first text classification loss function, L generate represents the second text classification loss function, w1 represents the weight of the first text classification loss function, and w2 represents the weight of the second text classification loss function.
[0074] In some non-limiting embodiments, the weight of the first text classification loss function and the weight of the second text classification loss function are the same.
[0075] Specifically, the first text classification loss function is expressed by the following expression:
[0076]
[0077]
[0078] Among them, L classifier represents the first text classification loss function, N represents the total number of text data in the training data set, i represents the sequence number of the text data, L represents the total number of category labels corresponding to the N text data, j represents the sequence number of the category label, y ij represents the coefficient of the i-th text data. When the category label of the i-th text data is the j-th category label, y ij The value of y is 1, otherwise ij The value of p is 0, ij Indicates the probability that the current predicted label of the i-th text data is the j-th category label, It represents the first state matrix output by the text classification sub-model after the i-th text data is input into the encoding sub-model of the initialized text classification model, wherein the category label is the encoded label obtained by encoding the original category label.
[0079] Specifically, the second text classification loss function can be expressed by the following expression:
[0080]
[0081]
[0082] Among them, L generate represents the second text classification loss function, W represents the total number of characters included in the original category label, w represents the character sequence number in the original category label, V represents the total number of characters included in the character set, v represents the character sequence number in the character set, represents the coefficient of the w-th character in the original category label, when the w-th character is the v-th character in the character set, The value of is 1, otherwise The value of p is 0, wv represents the probability that the wth character in the current predicted label is the vth character in the character set, It represents the second state matrix output by the decoding sub-model after the w-th character of the original category label is input into the decoding sub-model of the initialized text classification model.
[0083] In the prior art, a pre-trained text classification model is usually obtained by training based on an Encoder-Decoder model (for example, the Bart model, the T5 model, etc.). This model relies on a supervised text classification method, and the training effect and model inference results are not optimized. In contrast, this embodiment makes improvements and expansions in the model structure (see Figure 2 In the embodiment shown in the figure, a text classification sub-model is added on the basis of the model architecture based on Encoder-Decoder, so that the initialization text classification model adopted in this embodiment includes both the "Encoder+Decoder" structure and the "Encoder+text classification sub-model" structure, and can perform supervised text classification and generative text classification at the same time. Thus, the optimization of the model training effect is achieved through the improvement of the model structure.
[0084] Further, in an embodiment of the invention, the loss function of the training process is also optimized. Specifically, by introducing two parts of loss functions (i.e., the first text classification loss function and the second text classification loss function), and performing weighted operations on the two parts of loss functions, the target loss function of the training process is obtained, and the two parts of loss functions are respectively used to optimize the parameters of different sub-model combinations in the initialized text classification model. Therefore, compared with the use of a single loss function for model training, this implementation scheme integrates two parts of loss, which can not only strengthen the understanding of the input text by the encoding sub-model part, but also strengthen the connection between the encoding sub-model input text data and the category label input by the decoding sub-model, so that the text classification results generated by the decoding sub-model part are closer to the real category of the text, and reduce the generation of new irrelevant texts, thereby helping to improve the refinement of model training and obtain a more optimized pre-trained text classification model. Further, based on a more optimized pre-trained text classification model, it is also helpful to improve the accuracy of the text classification results obtained by subsequent text classification using the model.
[0085] Furthermore, after the pre-trained text classification model is obtained through training, the method may further include: inputting the text to be classified into the pre-trained text classification model to obtain a text classification result.
[0086] It is understandable that if each text data in the training data set used in the model training process is obtained by splicing the original text and the prompt text, then when the pre-trained text classification model is used for model reasoning or application, the text to be classified is also obtained by splicing the original text and the prompt text. In this way, the consistency of the data form of the model training process and the reasoning process can be maintained, and the accuracy of the text classification result can be improved.
[0087] Further, the process of performing inference / text classification using the pre-trained model may specifically include: encoding the to-be-input text using the encoding sub-model to obtain encoded data (which can also be referred to as encoded hidden state data); based on the encoded data, a preset decoding start symbol, and the decoding sub-model, performing at least one round of iteration to obtain the text classification result. It should be noted that each sub-model involved in the inference process is an optimized sub-model obtained after training.
[0088] Specifically, the iterative process includes: in the first round of iteration, inputting the encoded data and the decoding start symbol into the decoding sub-model to obtain the text category character t1 of the first round; in each subsequent round of iteration, inputting the encoded data, the decoding start symbol, and the concatenation result of the text category characters obtained in the previous rounds (encoded data + decoding start symbol + t1 + t2 + … t a ) into the decoding sub-model to obtain the text category character of the current round. If the text category character of the current round is not consistent with the preset decoding stop symbol, continue the next round of iteration until the text category character t n obtained in the current round is consistent with the decoding stop symbol; taking the concatenation result of the text category characters obtained in all rounds except the last round (t1 + t2 + … t n-1 ) as the text classification result.
[0089] Among them, the decoding start symbol and the decoding stop symbol can be preset according to the actual situation.
[0090] Combined with Figure 2 , the decoding start symbol can be represented by CLS, and the decoding end symbol can be represented by SEP. During the model inference process, in the first round of iteration, inputting the encoded data and the decoding start symbol CLS into the decoding sub-model 22 to obtain the text category character "You" of the first round; in the second round of iteration, inputting the concatenation result of the encoded data, the decoding start symbol CLS, and the character "You" (encoded data + CLS + You) into the decoding sub-model 22 to obtain the text category character "Xi" of the current round; in the third round of iteration, inputting the concatenation result of the encoded data, the decoding start symbol CLS, the character "You", and the character "Xi" (encoded data + CLS + You + Xi) into the decoding sub-model 22 to obtain the text category character "SEP" of the current round. This character is consistent with the decoding stop symbol, and at this time, the iteration stops, and the concatenation result of the text category characters obtained in all rounds except the last round ("YouXi") is taken as the text classification result.
[0091] Referring to Figure 3 , Figure 31 is a schematic diagram of a text classification model training device according to an embodiment of the present invention. The text classification model training device may include:
[0092] An initialization model determination module 31 is used to determine an initialization text classification model, wherein the initialization text classification model includes an encoding sub-model, a decoding sub-model and a text classification sub-model;
[0093] A training data set determination module 32, used to determine a training data set, wherein the training data set includes a plurality of text data and a category label for each text data;
[0094] A model training module 33 is used to iteratively train one or more of the following items to be optimized using a target loss function to obtain a pre-trained text classification model: the encoding sub-model, the decoding sub-model, and the text classification sub-model;
[0095] In each round of iteration, the text data in the training data set is input into the encoding sub-model for encoding, and then the encoded data is input into the text classification sub-model for text classification processing, and the encoded data and the category label of the text data are input into the decoding sub-model for decoding.
[0096] For the principles, specific implementation and beneficial effects of the training device for the text classification model, please refer to the previous article and Figure 1 The description of the training method of the text classification model shown is not repeated here.
[0097] The embodiment of the present invention further provides a readable storage medium, such as a computer readable storage medium, on which a computer program is stored, and the computer program is executed by a processor. Figure 1 The steps of the training method of the text classification model are shown. The computer-readable storage medium may include a non-volatile memory or a non-transitory memory, and may also include an optical disk, a mechanical hard disk, a solid-state hard disk, etc.
[0098] Specifically, in the embodiment of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0099] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM).
[0100] The embodiment of the present invention further provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the above-mentioned Figure 1The steps of the training method of the text classification model are shown. The terminal may include but is not limited to terminal devices such as mobile phones, computers, tablet computers, etc., and may also be a server, a cloud platform, etc.
[0101] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.
[0102] The "plurality" appearing in the embodiments of the present application refers to two or more.
[0103] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.
[0104] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.
[0105] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.
Claims
1. A training method for a text classification model, characterized in that: include: Determine an initialization text classification model, wherein the initialization text classification model includes an encoding sub-model, a decoding sub-model, and a text classification sub-model; Determine a training data set, wherein the training data set includes a plurality of text data and a category label for each text data; Using the target loss function, one or more of the following items to be optimized are iteratively trained to obtain a pre-trained text classification model: the encoding sub-model, the decoding sub-model and the text classification sub-model; wherein, in each round of iteration, the text data in the training data set is input into the encoding sub-model for encoding, and then the encoded data is input into the text classification sub-model for text classification processing, and the encoded data and the category label of the text data are input into the decoding sub-model for decoding.
2. The method according to claim 1, characterized in that The target loss function is obtained by performing a weighted operation on a preset first text classification loss function and a second text classification loss function; During the iterative training process, the first text classification loss function is used to iteratively optimize the parameters of the encoding sub-model and the text classification sub-model, and the second text classification loss function is used to iteratively optimize the parameters of the encoding sub-model and the decoding sub-model.
3. The method according to claim 2, characterized in that The target loss function is obtained by weighted summing the first text classification loss function and the second text classification loss function.
4. The method according to claim 3, characterized in that The objective loss function is expressed by the following expression: Loss=w1×L classifier +w2×L generate ; w1+w2=1; Among them, Loss represents the target loss function, L classifier represents the first text classification loss function, L generate represents the second text classification loss function, w1 represents the weight of the first text classification loss function, and w2 represents the weight of the second text classification loss function.
5. The method according to claim 3 or 4, characterized in that: The weight of the first text classification loss function is the same as the weight of the second text classification loss function.
6. The method according to claim 2, characterized in that The first text classification loss function is expressed by the following expression: Among them, L classifier represents the first text classification loss function, N represents the total number of text data in the training data set, i represents the sequence number of the text data, L represents the total number of category labels corresponding to the N text data, j represents the sequence number of the category label, y ij represents the coefficient of the i-th text data. When the category label of the i-th text data is the j-th category label, y ij The value of y is 1, otherwise ij The value of p is 0, ij Indicates the probability that the current predicted label of the i-th text data is the j-th category label, It represents the first state matrix output by the text classification sub-model after the i-th text data is input into the encoding sub-model of the initialized text classification model, wherein the category label is the encoded label obtained by encoding the original category label.
7. The method according to claim 2, characterized in that The second text classification loss function is expressed by the following expression: Among them, L generate represents the second text classification loss function, W represents the total number of characters included in the original category label, w represents the character sequence number in the original category label, V represents the total number of characters included in the character set, v represents the character sequence number in the character set, represents the coefficient of the w-th character in the original category label, when the w-th character is the v-th character in the character set, The value of is 1, otherwise The value of p is 0, wv represents the probability that the wth character in the current predicted label is the vth character in the character set, It represents the second state matrix output by the decoding sub-model after the w-th character of the original category label is input into the decoding sub-model of the initialized text classification model.
8. The method according to claim 1, characterized in that Each piece of text data is obtained by splicing the original text and the prompt text; The prompt text is used to indicate the training data set and the target task of the initialized text classification model.
9. The method according to claim 1, characterized in that: The text classification sub-model is selected from any of the following: The network structure consists of multiple linear layers, Dropout layers, and activation functions; A model consisting of multiple self-attention mechanism modules; A model consisting of a single linear classification layer.
10. A training device for a text classification model, characterized in that: include: An initialization model determination module is used to determine an initialization text classification model, wherein the initialization text classification model includes an encoding sub-model, a decoding sub-model and a text classification sub-model; A training data set determination module, used to determine a training data set, wherein the training data set includes a plurality of text data and a category label for each text data; A model training module, used to iteratively train one or more of the following items to be optimized using a target loss function to obtain a pre-trained text classification model: the encoding sub-model, the decoding sub-model, and the text classification sub-model; In each round of iteration, the text data in the training data set is input into the encoding sub-model for encoding, and then the encoded data is input into the text classification sub-model for text classification processing, and the encoded data and the category label of the text data are input into the decoding sub-model for decoding.
11. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for training a text classification model according to any one of claims 1 to 9 are executed.
12. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the processor performs the steps of the text classification model training method according to any one of claims 1 to 9.