Small sample text classification method and system based on prompt learning virtual class representation
By using a virtual class representation method based on prompt learning, few-sample text classification is transformed into a cloze test task. By using a pre-trained language model and cosine similarity calculation of virtual class representation, the accuracy problem of few-sample text classification is solved, and effective classification is achieved on AG's News and DB-Pedia datasets.
Patent Information
- Application Number
- CN202310994477.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-08
- Publication Date
- 2025-12-26
- Estimated Expiration
- 2043-08-08
AI Technical Summary
Existing few-sample text classification methods generally perform poorly in few-sample scenarios, traditional fine-tuning methods are ineffective, and cue-based learning methods require a high degree of generalization of the dataset categories and cannot prove the effectiveness of the generalized word set.
A virtual class representation method based on prompting learning is adopted to transform the original data into a cloze test task. The hidden layer regression value of the mask position is obtained through a pre-trained language model, and random numbers are used as virtual class representations. The final classification result is calculated by cosine similarity.
It achieves good results on small sample datasets, reduces the loss of semantic information in hidden layer vectors, avoids textual descriptions of categories, and improves classification accuracy.
Smart Images
Figure CN117009525B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of text classification, and particularly relates to a small sample text classification method and system based on virtual class representation of prompt learning. BACKGROUND
[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.
[0003] With the rapid development and popularization of Internet technology, the amount of data on the network is growing exponentially, but most of the massive data on the network is unlabeled data. If the massive data is to be classified, it is obvious that manually labeling the massive data will consume a lot of financial and human resources, and even in some scenarios, it is unrealistic to label massive data. Therefore, it is extremely important and has great development prospects to design a small sample-based method for relatively accurate text classification and prediction.
[0004] At present, the existing small sample text classification task methods include the traditional "pre-training + fine-tuning" model optimization paradigm and the "pre-training + prompt learning" model optimization method.
[0005] The traditional fine-tuning-based method is to add a [CLS] token at the beginning of the original input, bring the data with [CLS] into the pre-trained language model, select the last layer hidden layer vector of the [CLS] position, and bring it into a randomly initialized classification layer to obtain the final model regression. Through the calculation of the regression value and a small amount of labeled class loss, the back propagation and model optimization are realized. A large number of experiments prove that this method is generally poor in the small sample scenario.
[0006] The prompt learning-based method is to integrate the prompt template with the original data, convert the input form of text classification into the input form of the cloze task, and then input the integrated data into the pre-trained language model. Finally, the MLM Head regression value of the [MASK] position obtained by the pre-trained language model is introduced into the mapping mechanism of the model to obtain the final model regression value. The mapping mechanism of this method sometimes needs to be very clear about the categories of the data set and can clearly summarize one or more words to express each class of data. The disadvantage of this is that it needs to understand the data very well and has a high summarization and language expression ability. Secondly, even if a summary word set is designed, it cannot be proved that it is the best summary word set for text classification of the data set. SUMMARY
[0007] To solve at least one technical problem in the background art, the present application provides a small sample text classification method and system based on prompt learning virtual class representation, which wraps the original data and introduces it into a pre-trained language model to obtain a regression value module of the pre-trained language model, a class semantic information module is obtained by training a virtual class representation through a small amount of labeled data, and finally a regression result module of a final model is obtained by calculating the regression values of module one and module two.
[0008] To achieve the above-mentioned purposes, the present application adopts the following technical solutions:
[0009] The first aspect of the present application provides a small sample text classification method based on prompt learning virtual class representation, comprising:
[0010] The original text sequence and the prompt template are integrated, the text classification task is converted into a cloze task, and an integrated input sequence with MASK is obtained;
[0011] Based on the integrated input sequence with MASK and the pre-trained language model, the regression value of the last layer of the hidden layer of the MASK position is obtained;
[0012] A random number of the same length is designed for each class as a virtual representation of the class, a random initialization embedding mechanism is initialized, and the virtual class representation is put into the random initialization embedding mechanism to obtain an initial representation of the class;
[0013] Multi-dimensional feature extraction is performed based on the initial representation of the class to obtain a class representation vector;
[0014] The cosine similarity of the regression value of the last layer of the hidden layer of the MASK position and each class representation vector is calculated, and the index of the maximum value of the calculated similarity vector is taken as the final classification prediction result.
[0015] The second aspect of the present application provides a small sample text classification system based on prompt learning virtual class representation, comprising:
[0016] The data integration module is configured to integrate the original text sequence and the prompt template, convert the text classification task into a cloze task, and obtain an integrated input sequence with MASK;
[0017] The regression value acquisition module is configured to obtain the regression value of the last layer of the hidden layer of the MASK position based on the integrated input sequence with MASK and the pre-trained language model;
[0018] a virtual representation module configured to: design a random number of the same length as each class as a virtual representation of the class, randomly initialize an embedding mechanism, and put the virtual class representation into the randomly initialized embedding mechanism to obtain an initial representation of the class;
[0019] a feature extraction module configured to: perform multi-dimensional feature extraction based on the initial representation of the class to obtain a class representation vector of each class;
[0020] a classification prediction module configured to: calculate the cosine similarity between the regression value of the last layer of the hidden layer of the MASK position and each class representation vector, and take the index of the maximum value of the calculated similarity vector as the final classification prediction result.
[0021] The third aspect of the present application provides a computer readable storage medium.
[0022] A computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps in the few-shot text classification method based on prompt learning of virtual class representation according to the first aspect.
[0023] The fourth aspect of the present application provides a computer device.
[0024] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps in the few-shot text classification method based on prompt learning of virtual class representation according to the first aspect when executing the program.
[0025] Compared with the prior art, the present application has the following advantages:
[0026] 1. The present application uses virtual class representation technology to obtain the effect that "even if a summary word set of a class cannot be well summarized or there is no definition of the class, and there is no actual textual representation of the class, the virtual class representation part parameters can be trained by supervised data. After the virtual class representation is trained by supervised data, the semantic information of the corresponding class will be obtained";
[0027] 2. The present application uses the similarity calculation technology to obtain the effect that "without the previous prompt learning based on MLMHead to realize the mapping from the last layer of the hidden layer to the word table and then from the word table to the class, because in the mapping from the word table to the class, the sum of the regression values of the words representing the class is used to represent the final probability that the class is the true value. This will lose a large amount of semantic information in the hidden layer vector. The present application directly uses the similarity between the last layer of the hidden layer vector and the virtual class representation vector to represent the probability that the class can be predicted, greatly reducing the semantic loss of the last layer of the hidden layer vector".
[0028] Advantages of the additional aspects of the application will be in part apparent from the descriptions below, in part will be apparent from the descriptions below, or will be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS
[0029] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated by reference in their entirety.
[0030] Figure 1 is the overall flow chart of the small sample text classification method based on prompt learning virtual class representation provided by the embodiment of the application;
[0031] Figure 2 is a detailed diagram of the virtual class representation acquisition semantic module provided by the embodiment of the application. DETAILED DESCRIPTION
[0032] The application will be further described below with reference to the drawings and embodiments.
[0033] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application pertains.
[0034] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.
[0035] For the technical problems mentioned in the background of the application: in the actual application scene, manual data labeling of massive data will consume a lot of manpower and financial resources, and in some scenes, it is impossible to label massive data. Secondly, sometimes it is impossible to summarize one or more words for each class of massive data, and even if such a summarized word set is obtained, it cannot be proved that the summarized word set is the optimal technical problem; the application proposes a small sample text classification method based on prompt learning using virtual class representation, which mainly includes three parts, which are respectively a pre-training language model regression value module obtained by packaging original data and introducing a pre-training language model, a virtual class representation training module for obtaining class semantic information through a small amount of labeled data, and finally a module for obtaining a final model regression result module by regression value of module one and module two. Experimental results show that good results are obtained on data sets AG's News and DB-Pedia, and the effectiveness of the proposed model has been proved by ablation experiments.
[0036] Embodiment one
[0037] As Figure 1 shown, the embodiment provides a small sample text classification method based on prompt learning virtual class representation, including the following steps:
[0038] Step 1: Select an existing or self-defined prompt template to integrate the original data into the input form of the mask language model task (one of the pre-training stage tasks of the pre-trained language model). The pre-trained language model can be stimulated to obtain the knowledge learned by the pre-trained language model in the pre-training stage.
[0039] For example, directly select a prompt template: a [MASK] news: <original input>
[0040] Step 2: Integrate the original input and the prompt template with [MASK] special token obtained in step 1 to convert the text classification problem into a cloze task.
[0041] x p =f prompt (x) (1)
[0042] Where x is the original input sequence, f prompt is the prompt template. x p represents the integrated input sequence containing the [MASK] special token.
[0043] Wherein, the process of converting the text classification task into a cloze task includes:
[0044] Replace the original input sequence with the <original input> placeholder, for example, for an original input "2022 Zhoushan Port annual cargo throughput exceeds 125 million tons, ranked first in the world for 14 consecutive years", integrate the prompt template described above to get a new input: "a [MASK] news: 2022 Zhoushan Port annual cargo throughput exceeds 125 million tons, ranked first in the world for 14 consecutive years".
[0045] Step 3: Process the integrated template data into token form using the tokenizer tool of the pre-trained language model.
[0046] Step 4: Input the token into the pre-trained language model and take the last layer hidden layer vector at the [MASK] position.
[0047] hidden=M(x p ) (2)
[0048] M represents the pre-trained language model, and hidden represents the regression value of the last layer hidden layer at the [MASK] position.
[0049] The regression value of the last hidden layer of the MASK position is obtained based on the integrated input sequence with MASK and the pre-trained language model, and specifically includes:
[0050] After introducing the new input sequence into the pre-trained language model, a max_sequence_length×hidden_size tensor is obtained, and the vector of the MASK position is selected, with a size of 1×hidden_size.
[0051] Step 5: Design a random number of the same length as a virtual representation of each class, randomly initialize a small embedding mechanism, and put the virtual class representation into the randomly initialized embedding mechanism to obtain the initial representation of the class.
[0052] classembedded i = Embedding(class i ) (3)
[0053] where Embedding represents a small embedding mechanism, class i represents a virtual class representation composed of random numbers, and classembedded i represents the class representation vector obtained by the embedding mechanism.
[0054] As Figure 2 shown in the details of the virtual class representation acquisition semantic module, it is assumed that each class is represented by 10 random numbers, and then an embedding mechanism of (10×class_num, 5) is constructed to obtain a tensor of (class_num, 10, 5) by introducing the virtual class representation of random numbers into the custom embedding mechanism.
[0055] Step 6: Use CNN to realize multi-dimensional feature extraction of the class representation vector obtained in step 5 to realize more accurate prediction, and use linear transformation to convert each class representation vector obtained by the convolution layer into a hidden layer vector size vector.
[0056] con = Convolution (class_embedded) (4)
[0057] pool = Pool (con) (5)
[0058] label = Linear (pool) (6)
[0059] where class_num represents the number of classes, and hidden_size represents the size of the pre-trained language model hidden layer.
[0060] The above technical solution has the advantages that no class representation word set needs to be defined, and there is no doubt about whether the defined word set is the best word set for achieving optimal prediction effect for the data set.
[0061] No manual textual representation of the classes of the data set is required, and only a randomized digital sequence needs to be defined for each class as a virtual class representation. Through optimization of the parameters of the self-defined embedding mechanism and convolutional model components, the capture of class semantic information is achieved, and the problem of not knowing how to represent the class in words is solved.
[0062] Step 7: Calculate the cosine similarity between the regression value obtained by the pre-trained language model in step 4 and each row (i.e., each class representation vector) of the regression value obtained in step 6.
[0063] outputs=cosine_similarity(label,hidden) (7)
[0064] represents the data input of a batch and the regression value obtained by training the language model.
[0065] The above technical solution has the advantages that some existing prompt learning methods need to use the MLM Head to realize the mapping of the last hidden layer to the word table, and then select the corresponding numerical value of the class representation word as the final class probability. This method loses a lot of semantic information of the hidden layer vector, and directly compares the vector containing class semantic information obtained through the virtual class representation with the hidden layer vector for similarity, which greatly reduces the semantic loss of the hidden layer vector.
[0066] Step 8: In the training phase of the model, use cross-entropy loss for backpropagation and model optimization.
[0067] L=cross_entropy(Labels,outputs) (8)
[0068] Step 9: In the inference phase, select the index of the maximum value of the similarity vector obtained in step 7 as the final prediction result.
[0069] pred=max(outputs) (9)
[0070] The size of pred is the batch size, which is the classification prediction result.
[0071] Experimental results show that good results are achieved on the data sets AG's News and DB-Pedia, and the effectiveness of the proposed model is proved by ablation experiments.
[0072] Embodiment Two
[0073] The embodiment provides a small sample text classification system based on prompt learning virtual class representation, including:
[0074] The data integration module is configured to integrate the original text sequence and the custom prompt template or the existing prompt template, convert the text classification task into a cloze task, and obtain an integrated input sequence with a MASK.
[0075] The regression value acquisition module is configured to obtain a regression value of a last layer hidden layer of a MASK position based on the integrated input sequence with the MASK and the pre-trained language model.
[0076] The virtual representation module is configured to design a random number with the same length as each class as a virtual representation of the class, randomly initialize an embedding mechanism, and put the virtual class representation into the randomly initialized embedding mechanism to obtain an initial representation of the class.
[0077] The feature extraction module is configured to perform multi-dimensional feature extraction based on the initial representation of the class to obtain a class representation vector.
[0078] The classification prediction module is configured to calculate a cosine similarity between the regression value of the last layer hidden layer of the MASK position and each class representation vector, and take an index of a maximum value of the calculated similarity vector as a final classification prediction result.
[0079] The feature extraction module is further configured to, when performing multi-dimensional feature extraction based on the initial representation of the class, further include: using linear transformation to convert each class representation vector obtained by the convolution layer into a hidden layer vector size vector.
[0080] Embodiment Three
[0081] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement steps in the small sample text classification method based on prompt learning virtual class representation.
[0082] Embodiment Four
[0083] The embodiment provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements steps in the small sample text classification method based on prompt learning virtual class representation when executing the program.
[0084] Those skilled in the art will appreciate that embodiments of the application can be readily used as a method, a system or a computer program product. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, magnetic disks and optical storage media) embodying computer program code.
[0085] The application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for carrying out each of the one or more functions specified in the flowchart illustrations and / or block diagrams.
[0086] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for carrying out each of the one or more functions specified in the flowchart illustrations and / or block diagrams.
[0087] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for carrying out each of the one or more functions specified in the flowchart illustrations and / or block diagrams.
[0088] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM) or the like.
[0089] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement and the like within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A small sample text classification method based on prompt learning virtual class representation, characterized in that, The method comprises the following steps: integrate the original text sequence and the prompt template, convert the text classification task into a cloze task, and obtain an integrated input sequence with a MASK; based on the integrated input sequence with the MASK and a pre-trained language model, obtain a regression value of the last layer hidden layer of the MASK position; design a random number of the same length as the virtual representation of each class, randomly initialize an embedding mechanism, and put the virtual class representation into the randomly initialized embedding mechanism to obtain the initial representation of the class; based on the initial representation of the class, perform multi-dimensional feature extraction to obtain a class representation vector; calculate the cosine similarity between the regression value of the last layer hidden layer of the MASK position and each class representation vector, and take the index of the maximum value of the obtained similarity vector as the final classification prediction result.
2. The few-shot text classification method based on prompt learning of virtual class representation according to claim 1, wherein, The prompt template is a prompt with a MASK special token.
3. The few-shot text classification method based on prompt learning of virtual class representation of claim 1, wherein, Before the integrated input sequence with the MASK is input, the tokenizer tool of the pre-trained language model is used to process it into token form.
4. The few-shot text classification method based on prompt learning of virtual class representation of claim 1, wherein, The prompt template is a self-defined prompt template or an existing prompt template.
5. The few-shot text classification method based on prompt learning of virtual class representation of claim 1, wherein, When performing multi-dimensional feature extraction based on the initial representation of the class, the method further comprises: using linear transformation to convert each class representation vector obtained by the convolution layer into a hidden layer vector size vector.
6. The few-shot text classification method based on prompt learning of virtual class representation of claim 1, wherein, In the training stage of the pre-trained language model, cross-entropy loss is used for back propagation and model optimization.
7. A small sample text classification system based on prompt learning virtual class representation, characterized in that, The method comprises the following steps: a data integration module configured to integrate the original text sequence and the prompt template, convert the text classification task into a cloze task, and obtain an integrated input sequence with a MASK; a regression value acquisition module configured to obtain a regression value of the last layer hidden layer of the MASK position based on the integrated input sequence with the MASK and a pre-trained language model; a virtual representation module configured to design a random number of the same length as the virtual representation of each class, randomly initialize an embedding mechanism, and put the virtual class representation into the randomly initialized embedding mechanism to obtain the initial representation of the class; a feature extraction module configured to perform multi-dimensional feature extraction based on the initial representation of the class to obtain a class representation vector; a classification prediction module configured to calculate the cosine similarity between the regression value of the last layer hidden layer of the MASK position and each class representation vector, and take the index of the maximum value of the obtained similarity vector as the final classification prediction result.
8. The few-shot text classification system based on prompt learning of claim 7, wherein, The feature extraction module is further configured to, when performing multi-dimensional feature extraction based on the initial representation of the class, further comprise: using linear transformation to convert each class representation vector obtained by the convolution layer into a hidden layer vector size vector.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the small sample text classification method based on prompt learning and virtual class representation according to any one of claims 1-6.
10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the small sample text classification method based on prompt learning and virtual class representation according to any one of claims 1-6.