Method and apparatus for training a text classification model based on training data volume

Abstract

This disclosure relates to the field of machine learning technology, and provides a method and apparatus for training a text classification model based on the amount of training data. The method includes: acquiring a training dataset; adding markers and marker masks to the training data of the training dataset using a Prompt Tuning algorithm; determining a first data size of the training dataset after the Prompt Tuning algorithm processing; and training the text classification model according to the first data size: when the first data size is less than a first preset size, freezing the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker, and optimizing the parameters of the embedding layer corresponding to the marker only based on the marker; when the first data size is not less than the first preset size, optimizing the parameters of all embedding layers in the text classification model based on the training data after adding markers and marker masks.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning technology, and in particular to a method and apparatus for training a text classification model based on the amount of training data. Background Technology

[0002] In recent years, with the development of machine learning technology, more and more large-scale models have been applied to the field of text processing. To ensure that these large-scale models meet requirements and improve training efficiency, the common practice is to further train pre-trained large-scale models. However, when further training a pre-trained large-scale model, the training results are entirely dependent on the size of the training dataset. A large training dataset yields better results, while a small dataset results in worse results. Furthermore, current technology lacks methods to address the issue of poor training results due to a small training dataset.

[0003] In realizing the present invention, the inventors discovered at least the following technical problem in the related technology: the inability to flexibly select model training methods based on the amount of data in the training dataset. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus, electronic device, and computer-readable storage medium for training a text classification model based on the amount of training data, in order to solve the problem in the prior art that the model training method cannot be flexibly selected according to the amount of training dataset.

[0005] A first aspect of this disclosure provides a method for training a text classification model based on the amount of training data, comprising: acquiring a training dataset; adding marker words and marker masks to the training data of the training dataset using a Prompt Tuning algorithm; determining a first data size of the training dataset after processing by the Prompt Tuning algorithm; and training the text classification model according to the first data size: when the first data size is less than a first preset size, freezing the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker words, and optimizing the parameters of the embedding layer corresponding to the marker words in the text classification model only based on the marker words; when the first data size is not less than the first preset size, optimizing the parameters of all embedding layers in the text classification model based on the training data after adding marker words and marker masks.

[0006] A second aspect of this disclosure provides an apparatus for training a text classification model based on the amount of training data, comprising: a processing module configured to acquire a training dataset and add marker words and marker masks to the training data of the training dataset using a Prompt Tuning algorithm; and a training module configured to determine a first data size of the training dataset after processing by the Prompt Tuning algorithm, and train the text classification model according to the first data size: when the first data size is less than a first preset size, freezing the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker word, and optimizing the parameters of the embedding layer corresponding to the marker word in the text classification model only based on the marker word; and when the first data size is not less than the first preset size, optimizing the parameters of all embedding layers in the text classification model based on the training data after adding marker words and marker masks.

[0007] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0008] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0009] The beneficial effects of this disclosure embodiment compared with the prior art are as follows: Because this disclosure embodiment obtains a training dataset, adds marker words and marker masks to the training data of the training dataset using the Prompt Tuning algorithm; determines the first data size of the training dataset after the Prompt Tuning algorithm, and trains the text classification model according to the first data size: when the first data size is less than a first preset size, freezes the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker word, and optimizes the parameters of the embedding layer corresponding to the marker word in the text classification model only based on the marker word; when the first data size is not less than the first preset size, optimizes the parameters of all embedding layers in the text classification model based on the training data after adding marker words and marker masks. Therefore, by adopting the above technical means, the problem in the prior art that the model training method cannot be flexibly selected according to the data size of the training dataset can be solved, thereby flexibly training the model according to the data size of the training dataset and improving the training effect. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart (I) illustrating a method for training a text classification model based on the amount of training data provided in this embodiment of the disclosure.

[0012] Figure 2 This is a flowchart (II) illustrating a method for training a text classification model based on the amount of training data provided in this embodiment of the disclosure.

[0013] Figure 3 This is a schematic diagram of the structure of an apparatus for training a text classification model based on the amount of training data provided in an embodiment of this disclosure;

[0014] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0015] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0016] Figure 1 This is a flowchart (I) illustrating a method for training a text classification model based on the amount of training data provided in this embodiment of the present disclosure. Figure 1 Methods for training text classification models based on the amount of training data can be executed by a computer or server, or by software on a computer or server. For example... Figure 1 As shown, the method for training a text classification model based on the amount of training data includes:

[0017] S101, Obtain the training dataset, and add label words and label masks to the training data of the training dataset using the Prompt Tuning algorithm;

[0018] S102, determine the initial size of the training dataset after processing by the Prompt Tuning algorithm, and train the text classification model based on the initial size of the data.

[0019] S103, when the first data volume is less than the first preset size, freeze the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker word, and optimize the parameters of the embedding layer corresponding to the marker word in the text classification model only based on the marker word.

[0020] S104, when the size of the first data volume is not less than the first preset size, optimize the parameters of all embedding layers in the text classification model based on the training data after adding the marker word and marker mask.

[0021] Let the training dataset before the Prompt Tuning algorithm be denoted as the first training dataset, and the training data in the first training dataset be denoted as the first training data. Let the training dataset after the Prompt Tuning algorithm be denoted as the second training dataset, and the training data in the second training dataset be denoted as the second training data. Then, the second training dataset includes multiple second training data, and the first training dataset includes multiple first training data. Each second training data corresponds to one first training data. Each second training data is based on its corresponding first training data with additional label words and multiple label masks.

[0022] The Prompt Tuning algorithm, also known as the P-Tuning algorithm, adds prompt patterns to training data during cue learning. This can be understood as adding prompt content to the training data. The labeled text is the cue content, and the labeled mask is used to improve the efficiency of cue learning training and also to output the category of the training data. The text classification model outputs the category of the training data at the positions of the labeled mask.

[0023] Text classification models are BERT, ALBERT, Roberta, or Debert models that are pre-trained based on masked language modeling tasks.

[0024] The text classification model can be trained based on the cross-entropy loss function. The text classification model includes multiple embedding layers, attention layers, and fully connected layers. Freezing the parameters of the embedding layers other than those corresponding to the tokens in the text classification model actually also freezes the attention layers that are connected to other embedding layers. Optimizing only the parameters of the embedding layer corresponding to the token in the text classification model also requires optimizing the parameters of the attention layers connected to the token embedding layer. It is important to note that the fully connected layers are not frozen regardless of whether the initial data size is less than a first preset size, because fully connected layers are used to output training results; without fully connected layers, training is impossible. Therefore, the parameters of the attention layers are optimized regardless of whether the initial data size is less than the first preset size.

[0025] According to the technical solution provided by the embodiments of the present disclosure, a training data set is obtained, and token words and token masks are added to the training data of the training data set through the Prompt Tuning algorithm; the first data volume size of the training data set processed by the Prompt Tuning algorithm is determined, and the text classification model is trained according to the first data volume size: when the first data volume size is less than the first preset size, the parameters of other embedding layers except the embedding layer corresponding to the token words in the text classification model are frozen, and only the parameters of the embedding layer corresponding to the token words in the text classification model are optimized according to the token words; when the first data volume size is not less than the first preset size, the parameters of all embedding layers in the text classification model are optimized according to the training data with token words and token masks added. Therefore, by adopting the above technical means, the problem in the prior art that the model training method cannot be flexibly selected according to the data volume of the training data set can be solved, and the model can be flexibly trained according to the data volume of the training data set, thereby improving the training effect.

[0026] Adding token words and token masks to the training data of the training data set through the Prompt Tuning algorithm includes: determining the first position for adding token words in the training data; adding token words at the first position of the training data through the Prompt Tuning algorithm, where the token words are from the vocabulary corresponding to the text classification model; determining the second position for adding token masks in the training data; adding token masks at the second position of the training data through the Prompt Tuning algorithm.

[0027] For example, the first position can be before the original training data, that is, inserting token words before the original training data, and the second position can be inserted in the middle of the first position: the original training data is "China Women's Volleyball Team Wins the Championship Again", and the training data after adding token words and token masks is "u1 u2 u3 [M] [M]u4 u5 u6 China Women's Volleyball Team Wins the Championship Again", [M] is the token mask, and "u1 u2 u3 u4 u5 u6" are six token words. Each text classification model has a corresponding vocabulary, and the token words are from the vocabulary corresponding to the text classification model.

[0028] After being represented by the text classification model, the two token masks become two mask vectors. These two mask vectors are connected to a new fully connected layer for classification to obtain the classification results of the two token masks. The number of categories of the classification results of the two token masks represents the number of words in the vocabulary, and the labels of the two token masks are "sports" and "physical education". During the training of the text classification model: according to the classification results and labels of the two token masks, the loss value is calculated using the cross-entropy loss function, and the model parameters of the text classification model are optimized according to the loss value.

[0029] This disclosure provides another method for adding marker words and marker masks to the training data of a training dataset using the Prompt Tuning algorithm, including: randomly adding marker words and marker masks at any position in the training data using the Prompt Tuning algorithm, wherein the marker words are derived from the vocabulary corresponding to the text classification model.

[0030] After adding markers and marker masks to the training data of the training dataset using the Prompt Tuning algorithm, the method further includes: determining the second data size of the training dataset before the Prompt Tuning algorithm processing, and training the text classification model based on the second data size: when the second data size is less than a second preset size, freezing the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker, and optimizing the parameters of the embedding layer corresponding to the marker only based on the marker; when the second data size is not less than the second preset size, optimizing the parameters of all embedding layers in the text classification model based on the training data after adding markers and marker masks.

[0031] This disclosure provides another approach to training a text classification model. Unlike the above, this disclosure trains the text classification model based on the second data size of the training dataset before the Prompt Tuning algorithm.

[0032] The text classification model is trained based on a first data size, including: when the first data size is less than a first preset size, freezing the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker word, and optimizing the parameters of the embedding layer corresponding to the marker word in the text classification model only based on the marker word; when the first data size is greater than or equal to the first preset size but less than a third preset size, dividing the training dataset processed by the Prompt Tuning algorithm into a first training dataset and a second training dataset, performing the first stage of training on the text classification model based on the first training dataset, and performing the second stage of training based on the second training dataset; when the first data size is greater than or equal to the third preset size, optimizing the parameters of all embedding layers in the text classification model based on the training data after adding marker words and marker masks.

[0033] This disclosure proposes a more detailed method for training a text classification model based on a first data size. When the first data size is greater than or equal to a first preset size, it is necessary to further determine whether the first data size is less than a third preset size. The method for training the text classification model when the first data size is greater than or equal to the third preset size is the same as the method for training the text classification model when the first data size is not less than the first preset size; when the first data size is less than the third preset size, the training dataset processed by the Prompt Tuning algorithm is divided into a first training dataset and a second training dataset, and the text classification model is trained in multiple stages based on the first training dataset and the second training dataset.

[0034] Optionally, the proportion of the first training dataset and the second training dataset in the training dataset is determined based on the difference between the first data size and the third preset size.

[0035] The greater the difference between the first data size and the third preset size, the greater the proportion of the first training dataset in the training dataset.

[0036] The text classification model is trained in the first stage using the first training dataset and in the second stage using the second training dataset. The training process includes: training the text classification model in the first stage using the first training dataset by freezing the parameters of all embedding layers in the text classification model except for the embedding layers corresponding to the labeled words in the first training dataset, and optimizing the parameters of the embedding layers corresponding to the labeled words in the first training dataset using only the labeled words in the first training dataset; and training the text classification model in the second stage using the second training dataset by optimizing the parameters of all embedding layers in the text classification model using the training data in the second training dataset.

[0037] The first stage of training involves freezing the parameters of all embedding layers in the text classification model except for the embedding layer corresponding to the token word, and optimizing the parameters of the embedding layer corresponding to the token word in the text classification model only based on the token words in the first training dataset; the second stage of training is consistent with conventional training in existing technologies.

[0038] Figure 2 This is a flowchart (II) illustrating a method for training a text classification model based on the amount of training data provided in this embodiment of the disclosure. Figure 2 As shown, it includes:

[0039] S201, determine whether the size of the first data volume is less than the first preset size;

[0040] S202, if the first data size is less than the first preset size, freeze the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker word, and optimize the parameters of the embedding layer corresponding to the marker word in the text classification model only based on the marker word.

[0041] S203, determine whether the size of the first data volume is less than the third preset size;

[0042] S204, the first data size is smaller than the third preset size, so the training dataset processed by the Prompt Tuning algorithm is divided into the first training dataset and the second training dataset.

[0043] S205, The first stage of training of the text classification model is carried out based on the first training dataset;

[0044] S206, Perform the second stage of training based on the second training dataset;

[0045] S207, when the size of the first data volume is greater than or equal to the third preset size, optimize the parameters of all embedding layers in the text classification model based on the training data after adding the marker word and marker mask.

[0046] According to the technical solution provided in this disclosure, it is determined whether the size of the first data volume is less than a first preset size; when the size of the first data volume is less than the first preset size, the parameters of other embedding layers in the text classification model, except for the embedding layer corresponding to the marker word, are frozen, and the parameters of the embedding layer corresponding to the marker word in the text classification model are optimized only based on the marker word; it is also determined whether the size of the first data volume is less than a third preset size; when the size of the first data volume is less than the third preset size, the training dataset processed by the Prompt Tuning algorithm is divided into a first training dataset and a second training dataset, and the text classification model is trained in the first stage based on the first training dataset and in the second stage based on the second training dataset; when the size of the first data volume is greater than or equal to the third preset size, the parameters of all embedding layers in the text classification model are optimized based on the training data after adding the marker word and the marker mask. Therefore, by adopting the above technical means, the problem in the prior art that the model training method cannot be flexibly selected according to the data volume of the training dataset can be solved, thereby flexibly training the model according to the data volume of the training dataset and improving the training effect.

[0047] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0048] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0049] Figure 3This is a schematic diagram of an apparatus for training a text classification model based on the amount of training data provided in an embodiment of this disclosure. Figure 3 As shown, the apparatus for training a text classification model based on the amount of training data includes:

[0050] Processing module 301 is configured to acquire the training dataset and add label words and label masks to the training data of the training dataset using the Prompt Tuning algorithm;

[0051] Training module 302 is configured to determine the first data size of the training dataset after the Prompt Tuning algorithm, and train the text classification model based on the first data size: when the first data size is less than the first preset size, the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker word are frozen, and the parameters of the embedding layer corresponding to the marker word in the text classification model are optimized only based on the marker word; when the first data size is not less than the first preset size, the parameters of all embedding layers in the text classification model are optimized based on the training data after adding the marker word and the marker mask.

[0052] The large-scale target model has been pre-trained, while the small-scale target model has not been pre-trained; or both the large-scale and small-scale target models have been pre-trained.

[0053] Let the training dataset before the Prompt Tuning algorithm be denoted as the first training dataset, and the training data in the first training dataset be denoted as the first training data. Let the training dataset after the Prompt Tuning algorithm be denoted as the second training dataset, and the training data in the second training dataset be denoted as the second training data. Then, the second training dataset includes multiple second training data, and the first training dataset includes multiple first training data. Each second training data corresponds to one first training data. Each second training data is based on its corresponding first training data with additional label words and multiple label masks.

[0054] The Prompt Tuning algorithm, also known as the P-Tuning algorithm, adds prompt patterns to training data during cue learning. This can be understood as adding prompt content to the training data. The labeled text is the cue content, and the labeled mask is used to improve the efficiency of cue learning training and also to output the category of the training data. The text classification model outputs the category of the training data at the positions of the labeled mask.

[0055] Text classification models are BERT, ALBERT, Roberta, or Debert models that are pre-trained based on masked language modeling tasks.

[0056] The text classification model can be trained based on the cross-entropy loss function. The text classification model includes multiple embedding layers, attention layers, and fully connected layers. Freezing the parameters of the embedding layers other than those corresponding to the tokens in the text classification model actually also freezes the attention layers that are connected to other embedding layers. Optimizing only the parameters of the embedding layer corresponding to the token in the text classification model also requires optimizing the parameters of the attention layers connected to the token embedding layer. It is important to note that the fully connected layers are not frozen regardless of whether the initial data size is less than a first preset size, because fully connected layers are used to output training results; without fully connected layers, training is impossible. Therefore, the parameters of the attention layers are optimized regardless of whether the initial data size is less than the first preset size.

[0057] According to the technical solution provided in this disclosure, a training dataset is obtained, and a marker word and marker mask are added to the training data of the training dataset using the Prompt Tuning algorithm. The size of the first data volume of the training dataset after processing by the Prompt Tuning algorithm is determined, and the text classification model is trained based on the first data volume. When the first data volume is less than a first preset size, the parameters of other embedding layers in the text classification model, except for the embedding layer corresponding to the marker word, are frozen, and the parameters of the embedding layer corresponding to the marker word in the text classification model are optimized only based on the marker word. When the first data volume is not less than the first preset size, the parameters of all embedding layers in the text classification model are optimized based on the training data after adding the marker word and marker mask. Therefore, by adopting the above technical means, the problem in the prior art that the model training method cannot be flexibly selected according to the data volume of the training dataset can be solved, thereby flexibly training the model according to the data volume of the training dataset and improving the training effect.

[0058] Optionally, the processing module 301 is further configured to determine a first position for adding a marker word to the training data; add a marker word to the first position in the training data using a Prompt Tuning algorithm, wherein the marker word is derived from the vocabulary corresponding to the text classification model; determine a second position for adding a marker mask to the training data; and add a marker mask to the second position in the training data using a Prompt Tuning algorithm.

[0059] For example, the first position can be before the original training data, that is, inserting marker words before the original training data. The second position can be inserting in the middle of the first position. The original training data is "Chinese Women's Volleyball Team Wins the Championship Again". The training data after adding marker words and marker masks is "u1 u2 u3 [M] [M]u4 u5 u6 Chinese Women's Volleyball Team Wins the Championship Again", where [M] is the marker mask, and "u1 u2 u3 u4 u5 u6" are six marker words. Each text classification model has a corresponding vocabulary, and the marker words are from the vocabulary corresponding to the text classification model.

[0060] After being represented by the text classification model, the two marker masks become two mask vectors. These two mask vectors are connected to a new fully connected layer for classification, and the classification results of the two marker masks are obtained. The number of categories in the classification results of the two marker masks represents the number of words in the vocabulary, and the labels of the two marker masks are "Sports" and "Physical Education". During the training of the text classification model: according to the classification results and labels of the two marker masks, the loss value is calculated using the cross-entropy loss function, and the model parameters of the text classification model are optimized according to the loss value.

[0061] Optionally, the processing module 301 is further configured to randomly add marker words and marker masks at any position of the training data through the Prompt Tuning algorithm, where the marker words are from the vocabulary corresponding to the text classification model.

[0062] Optionally, the training module 302 is further configured to determine the size of the second data volume of the training data set before being processed by the Prompt Tuning algorithm, and train the text classification model according to the size of the second data volume: when the size of the second data volume is less than the second preset size, freeze the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker words, and only optimize the parameters of the embedding layer corresponding to the marker words in the text classification model according to the marker words; when the size of the second data volume is not less than the second preset size, optimize the parameters of all embedding layers in the text classification model according to the training data after adding marker words and marker masks.

[0063] The embodiments of the present disclosure provide another idea for training a text classification model. Different from the above, the embodiments of the present disclosure train the text classification model according to the size of the second data volume of the training data set before being processed by the Prompt Tuning algorithm.

[0064] Optionally, the training module 302 is further configured to: when the first data size is less than a first preset size, freeze the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker word, and optimize the parameters of the embedding layer corresponding to the marker word in the text classification model only based on the marker word; when the first data size is greater than or equal to the first preset size but less than a third preset size, divide the training dataset processed by the Prompt Tuning algorithm into a first training dataset and a second training dataset, perform the first stage training of the text classification model based on the first training dataset, and perform the second stage training based on the second training dataset; when the first data size is greater than or equal to the third preset size, optimize the parameters of all embedding layers in the text classification model based on the training data after adding the marker word and marker mask.

[0065] This disclosure proposes a more detailed method for training a text classification model based on a first data size. When the first data size is greater than or equal to a first preset size, it is necessary to further determine whether the first data size is less than a third preset size. The method for training the text classification model when the first data size is greater than or equal to the third preset size is the same as the method for training the text classification model when the first data size is not less than the first preset size; when the first data size is less than the third preset size, the training dataset processed by the Prompt Tuning algorithm is divided into a first training dataset and a second training dataset, and the text classification model is trained in multiple stages based on the first training dataset and the second training dataset.

[0066] Optionally, the proportion of the first training dataset and the second training dataset in the training dataset is determined based on the difference between the first data size and the third preset size.

[0067] The greater the difference between the first data size and the third preset size, the greater the proportion of the first training dataset in the training dataset.

[0068] Optionally, the training module 302 is further configured to perform a first-stage training on the text classification model based on the first training dataset: freezing the parameters of other embedding layers in the text classification model except for the embedding layers corresponding to the labeled words in the first training dataset, and optimizing the parameters of the embedding layers corresponding to the labeled words in the first training dataset based only on the labeled words in the first training dataset; and performing a second-stage training on the second training dataset: optimizing the parameters of all embedding layers in the text classification model based on the training data in the second training dataset.

[0069] The first stage of training involves freezing the parameters of all embedding layers in the text classification model except for the embedding layer corresponding to the token word, and optimizing the parameters of the embedding layer corresponding to the token word in the text classification model only based on the token words in the first training dataset; the second stage of training is consistent with conventional training in existing technologies.

[0070] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0071] Figure 4 This is a schematic diagram of the electronic device 4 provided in an embodiment of this disclosure. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, it implements the steps in the various method embodiments described above. Alternatively, when the processor 401 executes the computer program 403, it implements the functions of each module / unit in the various device embodiments described above.

[0072] Electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 4 may include, but is not limited to, processor 401 and memory 402. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or different components.

[0073] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0074] The memory 402 can be an internal storage unit of the electronic device 4, such as a hard disk or RAM of the electronic device 4. The memory 402 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 4. The memory 402 can also include both internal and external storage units of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.

[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0076] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0077] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A method for training a text classification model based on the amount of training data, applied to the field of text classification, characterized in that, include: Obtain the training dataset, and add label words and label masks to the training data of the training dataset using the Prompt Tuning algorithm; Determine the initial size of the training dataset after processing by the Prompt Tuning algorithm, and train the text classification model based on the initial size of the dataset. When the first data volume is less than the first preset size, the parameters of other embedding layers in the text classification model, except for the embedding layer corresponding to the marker, are frozen, and the parameters of the embedding layer corresponding to the marker in the text classification model are optimized only based on the marker. When the size of the first data volume is not less than the first preset size, the parameters of all embedding layers in the text classification model are optimized based on the training data after adding the marker word and the marker mask. The text classification model is trained based on the first data volume, including: When the first data volume is less than the first preset size, the parameters of other embedding layers in the text classification model, except for the embedding layer corresponding to the marker, are frozen, and the parameters of the embedding layer corresponding to the marker in the text classification model are optimized only based on the marker. When the size of the first data volume is greater than or equal to the first preset size but less than the third preset size, the training dataset processed by the Prompt Tuning algorithm is divided into a first training dataset and a second training dataset. The text classification model is trained in the first stage based on the first training dataset: the parameters of all embedding layers in the text classification model except those corresponding to the tokens in the first training dataset are frozen, and the parameters of the embedding layers corresponding to the tokens in the first training dataset are optimized only based on the tokens in the first training dataset. The second stage of training is then performed based on the second training dataset: the parameters of all embedding layers in the text classification model are optimized based on the training data in the second training dataset. When the size of the first data volume is greater than or equal to the third preset size, the parameters of all embedding layers in the text classification model are optimized based on the training data after adding the marker word and the marker mask.

2. The method according to claim 1, characterized in that, The text classification model is a BERT model, ALBERT model, Roberta model, or Debert model pre-trained based on the masked language model task.

3. The method according to claim 1, characterized in that, The training data of the training dataset is augmented with tokens and a token mask using the Prompt Tuning algorithm, including: Determine the first position in the training data where the marker word is added, the first position being before the original training data; The Prompt Tuning algorithm is used to add the marker word to the first position of the training data, wherein the marker word comes from the vocabulary corresponding to the text classification model; Determine a second position in the training data where the marker mask is added, the second position being the middle of the first position; The Prompt Tuning algorithm is used to add the marker mask at the second position in the training data.

4. The method according to claim 1, characterized in that, The training data of the training dataset is augmented with tokens and a token mask using the Prompt Tuning algorithm, including: The Prompt Tuning algorithm randomly adds the marker word and the marker mask to any position in the training data, wherein the marker word comes from the vocabulary corresponding to the text classification model.

5. The method according to claim 1, characterized in that, After adding tokens and token masks to the training data of the training dataset using the Prompt Tuning algorithm, the method further includes: Determine the second data size of the training dataset before the Prompt Tuning algorithm processes the data, and train the text classification model based on the second data size: When the second data volume is less than the second preset size, the parameters of other embedding layers in the text classification model, except for the embedding layer corresponding to the marker, are frozen, and the parameters of the embedding layer corresponding to the marker in the text classification model are optimized only based on the marker. When the second data volume is not less than the second preset size, the parameters of all embedding layers in the text classification model are optimized based on the training data after adding the marker word and the marker mask.

6. An apparatus for training a text classification model based on the amount of training data, applied in the field of text classification, characterized in that, include: The processing module is configured to acquire the training dataset and add label words and label masks to the training data of the training dataset using the Prompt Tuning algorithm; The training module is configured to determine the first data size of the training dataset after processing by the Prompt Tuning algorithm, and train the text classification model based on the first data size: when the first data size is less than a first preset size, the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker are frozen, and the parameters of the embedding layer corresponding to the marker are optimized only based on the marker; when the first data size is not less than the first preset size, the parameters of all embedding layers in the text classification model are optimized based on the training data after adding the marker and the marker mask. The training module is specifically configured as follows: when the first data size is less than a first preset size, freeze the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker word, and optimize the parameters of the embedding layer corresponding to the marker word in the text classification model only based on the marker word; when the first data size is greater than or equal to the first preset size but less than a third preset size, divide the training dataset processed by the Prompt Tuning algorithm into a first training dataset and a second training dataset, and perform a first stage of training on the text classification model based on the first training dataset: freeze the parameters of other embedding layers in the text classification model except for the embedding layer corresponding to the marker word in the first training dataset, and optimize the parameters of the embedding layer corresponding to the marker word in the first training dataset in the text classification model only based on the marker word in the first training dataset; The second stage of training is performed based on the second training dataset: the parameters of all embedding layers in the text classification model are optimized based on the training data in the second training dataset; when the size of the first data is greater than or equal to the third preset size, the parameters of all embedding layers in the text classification model are optimized based on the training data after adding the marker word and the marker mask.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images