Zero-sample text data classification method and system based on knowledge embedding prompt learning

Abstract

The invention discloses a zero-sample text data classification method and system based on knowledge embedding prompt learning. The method comprises the steps that text data are acquired and preprocessed; designing and selecting a prompt template corresponding to the classification task; performing natural language understanding by using a pre-training language model, and performing feature extraction of classification tasks through a prompt learning framework; and performing category similarity comparison by using the embedding space of the knowledge graph to obtain a classification result of the text data. According to the method, a pre-training language model based on Transform is used for natural language understanding, feature extraction of classification tasks is performed through a prompt learning framework, category similarity comparison is performed by using an embedding space of a knowledge graph, a zero-sample classification task of social media data is completed, data labeling is not needed, and prediction accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of text classification, and specifically relates to a zero-sample text data classification method and system based on knowledge embedding prompt learning, which is particularly suitable for social media text. Background Art

[0002] There are three common modes for existing text data classification, especially the classification of social media text data:

[0003] The first is based on traditional text representation and classification models such as bag-of-words, word embedding, Bayesian, K-nearest neighbor, etc. These methods ignore the semantic information between contexts and are difficult to truly understand social text data, resulting in relatively poor results.

[0004] The second is to use deep learning methods to achieve social media data classification tasks. Since deep neural networks usually require a large amount of labeled data, and the collected social media big data usually does not contain any labels, it requires a lot of manpower and training costs.

[0005] The third method is based on pre-trained language models and prompt learning. This method can use the knowledge gained from large corpora and the framework of prompt learning to understand social text data and perform zero-shot classification tasks. However, since the inherent connection between entities is not taken into account, there is still room for improvement in the authenticity of the generated content and the accuracy of classification.

[0006] For example, application number 2022100621887 discloses a fast image classification method based on prompt learning, which improves the performance of image classification tasks through prompt learning and image-text multimodal pre-training models. This method does not take into account the inherent connection between entities, and the authenticity of the generated content and the accuracy of classification still have room for improvement. Summary of the invention

[0007] In view of the above-mentioned technical problems, the purpose of the present invention is to provide a zero-shot text data classification method and system based on knowledge embedding prompt learning, which utilizes a Transformer-based pre-trained language model for natural language understanding, extracts features for classification tasks through a prompt learning framework, and uses the embedding space of the knowledge graph to perform category similarity comparison, thereby completing the zero-shot classification task of social media data without the need for unlabeled data and improving the prediction accuracy.

[0008] The technical solution of the present invention is:

[0009] A zero-shot text data classification method based on knowledge embedding prompt learning includes the following steps:

[0010] S01: Obtain text data and preprocess it;

[0011] S02: Design and select prompt templates corresponding to the classification task;

[0012] S03: Use pre-trained language models for natural language understanding and feature extraction for classification tasks through a prompt learning framework;

[0013] S04: Use the embedding space of the knowledge graph to compare category similarities and obtain the classification results of the text data.

[0014] In the preferred technical solution, in step S02

[0015] Add the prompt template p of the classification task before the original text input x, and transform it into the prompt enhanced input x p :

[0016] x p =p:x

[0017] Among them, the prompt template p contains at least one language model identifier [mask]

[0018] In the preferred technical solution, the step S03 specifically includes:

[0019] Use a pre-trained language model to get text input x p The predicted probability P of each word corresponding to the position of the [mask] identifier in :

[0020] logits=f([mask]=v|x p ,v∈V)

[0021] P = softmax(logits)

[0022] Among them, V refers to all the words covered by the pre-trained model, v is the element in V, logits is the original probability of the output word corresponding to the pre-trained model under the given model input, and softmax is the activation function.

[0023] In the preferred technical solution, the method of using the embedding space of the knowledge graph to compare category similarities in step S04 includes:

[0024] S41: Get the knowledge graph embedding, calculate the cosine similarity of all the words V covered by the pre-trained language model and all the classification label words Y in the knowledge graph embedding space k, and get the similarity matrix S:

[0025] S v,y = Cos(k v ,k y )

[0026] Among them, the classification label vocabulary Y is selected from all the vocabulary V covered by the model according to the actual task, and y is the element of Y;

[0027] S42: By fusing the similarity matrix S obtained by embedding the knowledge graph and the probability vector P obtained by the pre-trained language model, we get the weighted similarity matrix W:

[0028] W=(S°|S| °θ )°P

[0029] in, is the Hadamard product, and θ is an adjustable parameter.

[0030] In the preferred technical solution, the method for obtaining the classification result of the text data in step S04 includes:

[0031] Select the label with the highest probability among all the classification label vocabulary Y As the classification result of text data:

[0032]

[0033] Among them, argmax is a function that finds the parameter (set) of a function.

[0034] The present invention also discloses a computer storage medium on which a computer program is stored. When the computer program is executed, the zero-sample text data classification method based on knowledge embedding prompt learning is implemented.

[0035] The present invention further discloses a zero-sample text data classification system based on knowledge embedding prompt learning, comprising:

[0036] Acquisition module, acquires text data and performs preprocessing;

[0037] A classification task prompt template selection module is used to design and select prompt templates corresponding to the classification task;

[0038] Pre-trained language model feature extraction module, which uses pre-trained language models for natural language understanding and extracts features for classification tasks through a prompt learning framework;

[0039] The knowledge graph embedding and classification module uses the embedding space of the knowledge graph to compare category similarities and obtain the classification results of text data.

[0040] In the preferred technical solution, the processing of the pre-trained language model feature extraction module specifically includes:

[0041] Use a pre-trained language model to get text input x p The predicted probability P of each word corresponding to the position of the [mask] identifier in :

[0042] logits=f([mask]=v|x p ,v∈V)

[0043] P = softmax(logits)

[0044] Among them, V refers to all the words covered by the pre-trained model, v is the element in V, logits is the original probability of the output word corresponding to the pre-trained model under the given model input, and softmax is the activation function.

[0045] In a preferred technical solution, the knowledge graph embedding and classification module includes a knowledge embedded label mapping module and a classification module. The method for comparing category similarities using the embedding space of the knowledge graph in the knowledge embedded label mapping module includes:

[0046] S41: Get the knowledge graph embedding, calculate the cosine similarity of all the words V covered by the pre-trained language model and all the classification label words Y in the knowledge graph embedding space k, and get the similarity matrix S:

[0047] S v,y = Cos(k v ,k y )

[0048] Among them, the classification label vocabulary Y is selected from all the vocabulary V covered by the model according to the actual task, and y is the element of Y;

[0049] S42: By fusing the similarity matrix S obtained by embedding the knowledge graph and the probability vector P obtained by the pre-trained language model, we get the weighted similarity matrix W:

[0050]

[0051] in, is the Hadamard product, and θ is an adjustable parameter.

[0052] In the preferred technical solution, the knowledge graph embedding and classification module includes a classification module, and the method for obtaining the classification result of text data in the classification module includes:

[0053] Select the label with the highest probability among all the classification label vocabulary Y As the classification result of text data:

[0054]

[0055] Among them, argmax is a function that finds the parameter (set) of a function.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. No training samples are required, which avoids the high cost of manually labeling data and training models.

[0058] 2. Use pre-trained language models to understand the meaning of text and implement zero-shot social media data classification tasks through a prompt learning framework. The model can be applied to various classification tasks without fine-tuning and can classify itself according to user input categories.

[0059] 3. The model introduces external knowledge graph information, further strengthening the model's ability to understand the relationship between entities during the prediction stage and improving the model's prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0061] Figure 1 This is a flow chart of the zero-sample text data classification method based on knowledge embedding prompt learning in this embodiment;

[0062] Figure 2 This is a principle block diagram of a zero-shot text data classification system based on knowledge embedding prompt learning in this embodiment;

[0063] Figure 3 This is a workflow diagram of the zero-shot text data classification system based on knowledge embedding prompt learning in this embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0065] Principle of the invention:

[0066] A zero-shot social data classification method based on knowledge-embedded prompt learning. This method uses Transformer-based pre-trained language models (such as BERT, RoBERTa, T5, etc.) for natural language understanding, extracts features for classification tasks through the framework of prompt learning, and uses the embedding space of the knowledge graph to compare category similarities to complete the zero-shot classification task of social media data. This method can effectively use unlabeled data, save the cost of manually labeled data and training models, and improve prediction accuracy.

[0067] Example

[0068] like Figure 1 As shown, a zero-shot text data classification method based on knowledge embedding prompt learning includes the following steps:

[0069] S01: Obtain text data and preprocess it;

[0070] S02: Design and select prompt templates corresponding to the classification task;

[0071] S03: Use pre-trained language models for natural language understanding and feature extraction for classification tasks through a prompt learning framework;

[0072] S04: Use the embedding space of the knowledge graph to compare category similarities and obtain the classification results of the text data.

[0073] Specifically, in step S01, social media data is collected through an API provided by a social media platform (such as Weibo, Twitter, etc.).

[0074] Through data preprocessing, irrelevant information such as symbols, emoticons, web links, etc. contained in social text data is filtered out.

[0075] In a preferred embodiment, in step S02

[0076] Add the prompt template p of the classification task before the original text input x, and transform it into the prompt enhanced input x p :

[0077] x p =p:x

[0078] Among them, the prompt template p contains at least one language model identifier [mask]

[0079] In a preferred embodiment, step S03 specifically includes:

[0080] Use a pre-trained language model to get text input x p The predicted probability P of each word corresponding to the position of the [mask] identifier in :

[0081] logits=f([mask]=v|x p ,v∈V)

[0082] P = softmax(logits)

[0083] Among them, V refers to all the words covered by the pre-trained model, v is the element in V, logits is the original probability of the output word corresponding to the pre-trained model under the given model input, and softmax is the activation function.

[0084] In a preferred embodiment, the method of using the embedding space of the knowledge graph to compare category similarities in step S04 includes:

[0085] S41: Get the knowledge graph embedding, calculate the cosine similarity of all the words V covered by the pre-trained language model and all the classification label words Y in the knowledge graph embedding space k, and get the similarity matrix S:

[0086] S v,y = Cos(k v ,k y )

[0087] Among them, the classification label vocabulary Y is selected from all the vocabulary V covered by the model according to the actual task, and y is the element of Y;

[0088] S42: By fusing the similarity matrix S obtained by embedding the knowledge graph and the probability vector P obtained by the pre-trained language model, we get the weighted similarity matrix W:

[0089] W=(S°|S| °θ )°P

[0090] in, is the Hadamard product, and θ is an adjustable parameter.

[0091] In a preferred embodiment, the method for obtaining the classification result of the text data in step S04 includes:

[0092] Select the label with the highest probability among all the classification label vocabulary Y As the classification result of text data:

[0093]

[0094] Among them, argmax is a function that finds the parameter (set) of a function.

[0095] In another embodiment, a computer storage medium stores a computer program thereon, and when the computer program is executed, the zero-shot text data classification method based on knowledge embedding prompt learning is implemented.

[0096] In another embodiment, if Figure 2 As shown, a zero-shot text data classification system based on knowledge embedding prompt learning includes:

[0097] An acquisition module 10 acquires text data and performs preprocessing;

[0098] A classification task prompt template selection module 20 designs and selects a prompt template corresponding to the classification task;

[0099] A pre-trained language model feature extraction module 30, which uses the pre-trained language model for natural language understanding and extracts features for classification tasks through a prompt learning framework;

[0100] The knowledge graph embedding and classification module 40 includes a knowledge embedding label mapping module and a classification module, which uses the embedding space of the knowledge graph to compare category similarities and obtain the classification results of the text data.

[0101] The following describes in detail the workflow of a zero-shot text data classification system based on knowledge embedding prompt learning using a preferred embodiment as an example. Figure 3 As shown, the following steps are included:

[0102] Step 1: Collect social media data through the API provided by social media platforms (such as Weibo, Twitter, etc.).

[0103] Step 2: Filter out irrelevant information such as symbols, emoticons, web links, etc. contained in social text data through data preprocessing.

[0104] Step 3: Design and select the prompt template corresponding to the classification task. Add the prompt template p of the classification task before the original text input x to transform it into the prompt enhanced input x p :

[0105] x p =p:x

[0106] The prompt template p contains at least one language model identifier [mask].

[0107] For example Figure 3 If the task is a news text classification task, we can add a prompt template "a [mask] news" to the original text input x and transform the input into x p :

[0108] x p =A[mask]news:x[1]

[0109] Step 4: Take social media text input x using a pre-trained language model (e.g. BERT, RoBERTa, T5, etc.) p The predicted probability P of each word corresponding to the position of the [mask] identifier in :

[0110] logits=f([mask]=v|x p ,v∈V) [2]

[0111] P = softmax(logits) [3]

[0112] Among them, V refers to all the words covered by the pre-trained model, v is the element in V, logits is the original probability of the output word corresponding to the pre-trained model under the given model input, and softmax is the activation function.

[0113] Step 5: Collect the knowledge graph embedding space k (such as ConceptNet Numberbatch) corresponding to the knowledge graph (such as ConceptNet), or obtain the knowledge graph embedding through retrofitting and other technical training. Calculate the cosine similarity of all the words V covered by the pre-trained language model and all the classification label words Y in the knowledge graph embedding space k to obtain the similarity matrix S:

[0114] S v,y = Cos(k v ,k y ),v∈V,y∈Y[4]

[0115] Among them, the classification label vocabulary Y can be selected from all the vocabulary V covered by the model according to the actual task.

[0116] Step 6: By fusing the similarity matrix S obtained by embedding the knowledge graph and the probability vector P obtained by the pre-trained language model, we get the weighted similarity matrix W:

[0117]

[0118] in, is the Hadamard product, and θ is an adjustable parameter (θ = 0, 1, 2, ...) to reduce the negative impact of irrelevant words in the prediction results of the masked language model (MLM).

[0119] Step 7: Select the label with the highest probability among all the classification label vocabulary Y As a result of social text data classification:

[0120]

[0121] We conducted experiments on four social media datasets (AG News, BBC News, IMDB, and Yelp Reviews) and compared them with the following three baseline models.

[0122] BERT+Prompt Learning (BERT+PL): A pre-trained BERT model is used to extract features and the model predicted words are mapped to labels in a one-to-one manner.

[0123] RoBERTa+Prompt Learning (RoBERTa+PL): The RoBERTa model is used instead of the BERT model to extract text features, and the prediction results are obtained through prompt learning.

[0124] KePL with GloVe: The proposed KePL model process is adopted, but the knowledge graph embedding is changed to GloVe word embedding as an ablation experiment for comparison.

[0125] Table I

[0126] Performance comparison of various models

[0127]

[0128] The experimental results of different models on four datasets are shown in the table above. It can be seen that compared with BERT+PL and RoBERTa+PL, the RoBERTa model can capture more comprehensive knowledge and produce better zero-shot text classification performance. Based on RoBERTa, the proposed KePL model outperforms other baselines in both average and best cases. Specifically, compared with RoBERTa+PL, our model improves the average results of AG News, BBC News, IMDB and Yelp Reviews by about 3.68%, 15.24%, 1.42% and 1.67%, respectively, indicating the effectiveness of our knowledge embedded language expression construction. After integrating the knowledge graph embedding, the proposed KePL model significantly outperforms the KePL model without knowledge graph embedding (KePL with GloVe). The average accuracy on the Yelp Review and IMDB datasets is improved by 23.37% and 14.92%, respectively, confirming the effectiveness of incorporating external knowledge into zero-shot text classification.

[0129] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.

Claims

1. A zero-shot text data classification method based on knowledge embedding prompt learning, characterized in that: The following steps are involved: S01: Obtain text data and preprocess it; S02: Design and select prompt templates corresponding to the classification task; S03: Use pre-trained language models for natural language understanding and feature extraction for classification tasks through a prompt learning framework; S04: Use the embedding space of the knowledge graph to compare category similarities and obtain the classification results of the text data.

2. The zero-shot text data classification method based on knowledge embedding prompt learning according to claim 1 is characterized in that: In step S02, the prompt template p of the classification task is added before the original text input x, and converted into the prompt enhanced input x p : x p =p:x The prompt template p contains at least one language model identifier [mask].

3. The zero-shot text data classification method based on knowledge embedding prompt learning according to claim 1 is characterized in that: The step S03 specifically includes: Use a pre-trained language model to get text input x p The predicted probability P of each word corresponding to the position of the [mask] identifier in : logits=f([mask]=v|x p ,v∈V) p = softmax(logits) Among them, V refers to all the words covered by the pre-trained model, v is the element in V, logits is the original probability of the output word corresponding to the pre-trained model under the given model input, and softmax is the activation function.

4. The zero-shot text data classification method based on knowledge embedding prompt learning according to claim 1 is characterized in that: The method for using the embedding space of the knowledge graph to compare category similarities in step S04 includes: S41: Get the knowledge graph embedding, calculate the cosine similarity of all the words V covered by the pre-trained language model and all the classification label words Y in the knowledge graph embedding space k, and get the similarity matrix S: S v，y =Cos(k v ,k y ) Among them, the classification label vocabulary Y is selected from all the vocabulary V covered by the model according to the actual task, and y is the element of Y; S42: By fusing the similarity matrix S obtained by embedding the knowledge graph and the probability vector P obtained by the pre-trained language model, we get the weighted similarity matrix W: in, is the Hadamard product, and θ is an adjustable parameter.

5. The zero-shot text data classification method based on knowledge embedding prompt learning according to claim 1 is characterized in that: The method for obtaining the classification result of text data in step S04 includes: Select the label with the highest probability among all the classification label vocabulary Y As the classification result of text data: Among them, argmax is a function that finds the parameter (set) of a function.

6. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the zero-sample text data classification method based on knowledge embedding prompt learning described in any one of claims 1 to 5 is implemented.

7. A zero-shot text data classification system based on knowledge embedding prompt learning, characterized in that: include: Acquisition module, acquires text data and performs preprocessing; A classification task prompt template selection module is used to design and select prompt templates corresponding to the classification task; Pre-trained language model feature extraction module, which uses pre-trained language models for natural language understanding and extracts features for classification tasks through a prompt learning framework; The knowledge graph embedding and classification module uses the embedding space of the knowledge graph to compare category similarities and obtain the classification results of text data.

8. The zero-shot text data classification system based on knowledge embedding prompt learning according to claim 7 is characterized in that: The processing of the pre-trained language model feature extraction module specifically includes: Use a pre-trained language model to get text input x p The predicted probability P of each word corresponding to the position of the [mask] identifier in : logits=f([mask]=v|x P ,v∈V) P = softmax(logits) Among them, V refers to all the words covered by the pre-trained model, v is the element in V, logits is the original probability of the output word corresponding to the pre-trained model under the given model input, and softmax is the activation function.

9. The zero-shot text data classification system based on knowledge embedding prompt learning according to claim 7, characterized in that: The knowledge graph embedding and classification module includes a knowledge embedded label mapping module and a classification module. The method of using the embedding space of the knowledge graph to compare category similarities in the knowledge embedded label mapping module includes: S41: Get the knowledge graph embedding, calculate the cosine similarity of all the words V covered by the pre-trained language model and all the classification label words Y in the knowledge graph embedding space k, and get the similarity matrix S: S v，y =Cos(k v ,k y ) Among them, the classification label vocabulary Y is selected from all the vocabulary V covered by the model according to the actual task, and y is the element of Y; S42: By fusing the similarity matrix S obtained by embedding the knowledge graph and the probability vector P obtained by the pre-trained language model, we get the weighted similarity matrix W: in, is the Hadamard product, and θ is an adjustable parameter.

10. The zero-shot text data classification system based on knowledge embedding prompt learning according to claim 7, characterized in that: The knowledge graph embedding and classification module includes a classification module, and the method for obtaining the classification result of text data in the classification module includes: Select the label with the highest probability among all the classification label vocabulary Y As the classification result of text data: Among them, argmax is a function that finds the parameter (set) of a function.

Images