Aspect Category Detection Method Based on Automatically Constructing Hint Learning Mapper
By obtaining the predicted probability of category words in the pre-trained language model vocabulary and designing an automatic filtering mechanism, the problem of lack of label data is solved, and category detection is achieved in efficient conditions under zero-sample and few-sample conditions, especially in semantic fuzzy categories.
Patent Information
- Application Number
- CN202311152334.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-08
- Publication Date
- 2025-08-26
- Estimated Expiration
- 2043-09-08
AI Technical Summary
The existing aspect category detection methods have deteriorated performance when label data is missing, especially in the zero-sample and small-sample conditions, and it is difficult to effectively utilize the prior knowledge of pre-trained language models. The existing Verbalizer methods require manual intervention or rely on external knowledge bases and cannot effectively deal with semantic fuzzy categories.
The prediction probability of the pre-trained language model vocabulary list is obtained through the semantic expansion of category words, an automatic filtering mechanism is designed to refine the indication matrix, automatically select prompt words, and use part of speech, category semantic correlation and category specificity to capture related prompt words and map them to category labels.
The effect of category detection in zero samples and few samples was improved. The experimental results performed excellently on multiple data sets, especially in semantic fuzzy categories, showing high accuracy and generalization ability.
Smart Images
Figure CN117194660B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an aspect category detection method based on automatically building a prompt learning mapper, and belongs to the field of aspect-level sentiment analysis. Background Art
[0002] Sentiment analysis of comments on social media platforms has been a hot topic in the field of Natural Language Processing (NLP) in recent years. Aspect Category Detection (ACD), a subtask of aspect-level sentiment analysis, aims to detect the categories contained in a comment from a predefined set of aspect categories. For example, the sentence "Anyway, the food itself is quite good" contains the aspect category "food"; the sentence "But the staff's service attitude is terrible" contains the aspect category "service." Existing excellent methods for solving ACD tasks mostly adopt the [pre-train, fine-tuning] paradigm, and their effectiveness depends heavily on the scale of labeled data. However, comments on social media update quickly, and aspect categories are not fixed. It is difficult to always provide sufficient labeled data for newly emerging categories. Therefore, when there are only a small number of labeled samples, the performance of the above methods will drop significantly.
[0003] To maximize the performance of pre-trained language models even in few-shot and zero-shot conditions, researchers, inspired by GTP-3 and LAMA, proposed using prompts to transform classification tasks into cloze tasks. This unifies downstream tasks and PLMs into the same model to maximize the use of the prior knowledge of the pre-trained language model. The Verbalizer, a key component of the prompt learning model, contains the mapping between category prompt words and final aspect categories.
[0004] Current verbalizers can be roughly categorized into three types: manual construction, search-based, and continuously learnable. These methods require either intensive manual effort or labeled data to obtain a robust set of category words, making them incapable of cost-effectively handling zero-shot tasks. Some researchers have also proposed searching for cue words from external knowledge bases, which greatly expands the scope of mapping. However, mature knowledge bases are not always readily available, and there are few external knowledge bases that achieve comparable performance on non-English datasets, making the models difficult to transfer to other domains. Furthermore, experiments have revealed that these methods have issues with semantically ambiguous categories, such as "miscellaneous." Manually constructed word sets contain only the category words themselves, resulting in insufficient coverage. Search-based methods fail to account for word confusion, where a word may appear in different category word sets. Furthermore, many words retrieved from external knowledge bases do not appear in the pre-trained language model vocabulary. This is because the word sets in the external knowledge base and the pre-trained language model are designed differently, resulting in a lack of overlap between the two.
[0005] To address these issues, we propose a novel hint-based approach for zero-shot and few-shot ACD. Our approach leverages semantic extensions of category labels as hints to automatically acquire an initial verbalizer. Furthermore, we introduce an automatic verbalizer filtering mechanism to select important and reliable hint words for each category without manual intervention. Summary of the Invention
[0006] The present invention provides an aspect category detection method based on automatically building a prompt learning mapper to solve the problem of aspect category detection when label data is missing. The present invention improves the aspect category detection effect in both zero-sample and few-sample cases.
[0007] The technical solution of the present invention is: an aspect category detection method based on automatically building a prompt learning mapper, and the specific steps of the method are as follows:
[0008] Step 1: Use the semantic extension sentence of the category word and the task-specific template to form a prompt to obtain the predicted probability of all tokens in the pre-trained language model vocabulary;
[0009] Step 2: By designing an automatic filtering mechanism, the most appropriate prompt words are searched in the vocabulary to form a prompt word set for each category of words;
[0010] Step 3: During the category prediction process, the mapper maps the prompt word to the final category label.
[0011] Furthermore, in the step 1, the semantic extension of the classifier is adopted, that is, the definition sentence of the classifier on the Wikipedia website and the task-specific semantic extension of the classifier. <mask>Token template splicing composition prompt Will Send it to the pre-trained language model and get all the tokens in the vocabulary V, that is, words to fill in each <mask>The probability of the position, and then obtain the probability distribution vector of the given category For different categories, construct a complete probability matrix C represents the number of categories, and D represents the size of the pre-trained language model vocabulary.
[0012] Furthermore, in the step 2, an automatic filtering mechanism is proposed to improve the mapper. The automatic filtering mechanism is as follows: for each probability p in the probability matrix P ji Set an indicator value b ji ; The indicator value represents the relevance of the token i to a specific category j. An indicator value of 1 means that the token is very important for the corresponding category, while an indicator value of 0 means the opposite. Initially, all indicator values are set to 1 to form an indicator matrix C represents the number of categories, and D represents the size of the pre-trained language model vocabulary. The indicator matrix is refined by three aspects: part of speech, category semantic relevance, and category specificity to obtain the final indicator matrix to better capture the prompt words related to the aspect category. Finally, the mapper's final prompt word set for each category consists of the corresponding tokens with an indicator value of 1 in the final indicator matrix under that category.
[0013] Furthermore, the specific steps of refining the indicator matrix by the three aspects of part of speech, category semantic relevance and category specificity to obtain the final indicator matrix are as follows:
[0014] Step 2.1. De-noise the vocabulary V, that is, filter out meaningless words through part-of-speech tagging. First, use the pos_tag method in the nltk package to define the token set matching nouns, verbs, and adjectives in the vocabulary V as {pos}, and then adjust the corresponding element values in the indicator matrix B. The indicator matrix B is obtained by formula 1. pos , B pos The calculation formula for the element value in the j-th row and i-th column in is as follows:
[0015]
[0016] Among them, b ji For each probability p in the probability matrix P ji An indicator value set, v i Represents the i-th word in the vocabulary;
[0017] Step 2.2: Further modify the indicator matrix B based on the semantic relevance of the categories pos The element value in gets the indicator matrix B sem , B sem The calculation formula for the element value in the j-th row and i-th column in is as follows:
[0018]
[0019] Where MAX_M(.) represents the Mth largest probability value in the vector, A vector representing the probability distribution of a given class.
[0020] Step 2.3: Adjust the matrix B based on category specificity pos The element values in get the updated indicator matrix B spe , B spe The calculation formula for the element value in the j-th row and i-th column in is as follows:
[0021]
[0022] Where α is the threshold, indicating that words above this threshold are category-specific;
[0023] Step 2.4. Finally, the refined indicator matrix B′ is:
[0024]
[0025] where ° represents the hadamard product of two matrices.
[0026] Furthermore, in the step 3, the comment sentence to be predicted x i and task-specific <mask>The token templates are concatenated to form natural language text and sent to the pre-trained language model to obtain the probability distribution vector Finally, the mapper maps it to the aspect category label; the mapping process is discussed in two cases; for the zero-shot scenario, it is assumed that all words have the same contribution to the prediction of the corresponding category, so the sentence x is calculated using the following formula i The class probability
[0027]
[0028] For the few-sample scenario, a weight parameter is set for each prompt word in the prompt word set and further training is performed. Then the sentence x i The category probability is
[0029]
[0030] in is the parameter vector to be trained, optimized using cross entropy loss, P is the probability matrix, and B′ is the final indicator matrix after refinement.
[0031] The beneficial effects of the present invention are:
[0032] This paper proposes a method for aspect category detection based on hint learning in the absence of labeled samples. By semantically expanding the category words, the method obtains the predicted probabilities of all tokens in the vocabulary of a pre-trained language model. A pointer mechanism is designed to search for hint words, ultimately improving the performance of aspect category detection in zero- and few-shot scenarios. Theoretical and technical validation is conducted on the Restaurant dataset from the International Semantic Evaluation Task (Semeval) and the Amazon dataset in three different domains (boots, keyboards, and televisions). Experimental results demonstrate the effectiveness of this method. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 The invention provides an aspect category detection model framework based on automatically building a prompt learning mapper;
[0034] Figure 2 Experiments on four datasets using different ablation models in this invention;
[0035] Among them, (a) represents the experimental results of different ablation models on the "restaurant" dataset, (b) represents the experimental results of different ablation models on the "boots" dataset, (c) represents the experimental results of different ablation models on the "keyboard" dataset, and (d) represents the experimental results of different ablation models on the "TV" dataset;
[0036] Figure 3 This is an experiment on the "Miscellaneous" category in the present invention; among them, (a) represents the macro F1 (%) result of the zero-sample experiment, and (b) represents the macro F1 (%) result of the few-sample experiment. DETAILED DESCRIPTION
[0037] Example 1: Figure 1-Figure 3 As shown, the aspect category detection method based on automatic construction of prompt learning mapper, the specific steps of the method are as follows:
[0038] Step 1: Use semantic extension of classifiers, i.e., the definition of classifiers on Wikipedia and task-specific semantic extensions. <mask>Token template splicing composition prompt For example: <x>This is about <mask>category., where <x>is a placeholder for the input sentence. Send it to the pre-trained language model and get all the tokens in the vocabulary V, that is, words to fill in each <mask>The probability of the position, and then obtain the probability distribution vector of the given category For different categories, construct a complete probability matrix C represents the number of categories, and D represents the size of the pre-trained language model vocabulary.
[0039] Step 2: Design an automatic filtering mechanism. The automatic filtering mechanism is as follows: for each probability p in the probability matrix P ji Set an indicator value b ji ; The indicator value represents the relevance of the token i to a specific category j. An indicator value of 1 means that the token is very important for the corresponding category, while an indicator value of 0 means the opposite. Initially, all indicator values are set to 1 to form an indicator matrix C represents the number of categories, and D represents the size of the pre-trained language model vocabulary. The indicator matrix is refined by three aspects: part of speech, category semantic relevance, and category specificity to obtain the final indicator matrix to better capture the prompt words related to the aspect category. Finally, the mapper's final prompt word set for each category consists of the corresponding tokens with an indicator value of 1 in the final indicator matrix under that category.
[0040] The specific steps of refining the indicator matrix by the three aspects of part of speech, category semantic relevance and category specificity to obtain the final indicator matrix are as follows:
[0041] Step 2.1. De-noise the vocabulary V, that is, filter out meaningless words through part-of-speech tagging. First, use the pos_tag method in the nltk package to define the token set matching nouns, verbs, and adjectives in the vocabulary V as {pos}, and then adjust the corresponding element values in the indicator matrix B. The indicator matrix B is obtained by formula 1. pos , B pos The calculation formula for the element value in the j-th row and i-th column in is as follows:
[0042]
[0043] Among them, b ji For each probability p in the probability matrix P ji An indicator value set, v i Represents the i-th word in the vocabulary;
[0044] Step 2.2: Further modify the indicator matrix B based on the semantic relevance of the categories pos The element value in gets the indicator matrix B sem , B sem The calculation formula for the element value in the j-th row and i-th column in is as follows:
[0045]
[0046] Where MAX_M(.) represents the Mth largest probability value in the vector, A vector representing the probability distribution of a given class.
[0047] Step 2.3: Adjust the matrix B based on category specificity pos The element values in get the updated indicator matrix B spe , B spe The calculation formula for the element value in the j-th row and i-th column in is as follows:
[0048]
[0049] Where α is the threshold, indicating that words above this threshold are category-specific;
[0050] Step 2.4. Finally, the refined indicator matrix B′ is:
[0051]
[0052] where ° represents the hadamard product of two matrices.
[0053] Step 3: The comment sentence to be predicted x i and task-specific <mask>The token templates are concatenated to form natural language text and sent to the pre-trained language model to obtain the probability distribution vector Finally, the mapper maps it to the aspect category label; the mapping process is discussed in two cases; for the zero-shot scenario, it is assumed that all words have the same contribution to the prediction of the corresponding category, so the sentence x is calculated using the following formula i The class probability
[0054]
[0055] For the few-sample scenario, a weight parameter is set for each prompt word in the prompt word set and further training is performed. Then the sentence x i The category probability is
[0056]
[0057] in is the parameter vector to be trained, optimized using cross entropy loss, P is the probability matrix, and B′ is the final indicator matrix after refinement.
[0058] In order to verify the effectiveness of the model proposed in this paper, the prompt learning model related to this invention was selected as the baseline model, which is mainly based on the prompt text classification model, as follows:
[0059] Fine-Tuning (FT): In contrast to other cued learning methods, traditional fine-tuning methods add a classification layer after the pre-trained language model, obtain the hidden vector of [CLS], input it into the classification layer and make predictions.
[0060] Manual prompts: The manually constructed verbalizer contains limited category-related words. Here, the category word itself is used to represent the only related word of the category.
[0061] WARP: This model uses continuous vectors instead of single words to represent categories, <mask>The output of the position also obtains its hidden vector, and the two are used to calculate the probability of belonging to different categories through dot product. In the experiment, the word embedding of the category word is used as the initialization of the category vector.
[0062] PETAL: This model uses labeled and unlabeled data to automatically search for category-related words from a pruned vocabulary of a pre-trained language model, and selects words that appear more frequently by maximizing the likelihood function.
[0063] Auto-L: The model first trains the data <mask>Initial candidate words are selected based on the conditional probability of , then the candidate space is further pruned based on the zero-shot performance. Finally, the best-performing token is searched for by re-ranking on the test set. The automatically generated templates in the model are fixed, and only the verbalizer is searched for.
[0064] KPT: This method extends the verbalizer with external knowledge and then refines the selected classifiers in various ways on the support set. It uses variants that include all refinement methods.
[0065] The method proposed in the present invention is experimented on the restaurant dataset on the International Semantic Evaluation Task (Semeval) and three different domains of the Amazon dataset (boots domain, keyboard domain, and TV domain).
[0066] The experimental environment of this invention is Python 3.7 under Windows system, and all prompt learning methods are implemented using the OpenPrompt toolkit. In the few-shot experiment, following the settings of most few-shot learning, the Nway K shot training mode is adopted. K data are randomly selected for each category, and the validation set is the same size as the training set, that is, |D dev |=|D train |= K. Set the evaluation metric to macro-F1.
[0067] Table 1 Experimental results of different models on four datasets
[0068]
[0069]
[0070] Table 1 contains all the experimental results of the four data sets, where AVG represents the average performance of each model on the four data sets. Restaurant, Boots, Keyboards, and TV represent three different fields (boots field, keyboard field, and TV field) of the restaurant data set and the Amazon data set, respectively. As shown in the table, the model of the present invention achieved the best results under all settings. Compared with the second-best model, in the zero-shot task (rows 2, 3, 4, and 5 in Table 1), the model of the present invention increased by 9.3%, 5.9%, 4.7%, and 9.9% on the four data sets, respectively, with a particularly significant increase. This shows that the prompt words searched from the pre-trained language model vocabulary using the method of the present invention can better represent category labels. Under different K values in the few-shot task (rows 11, 16, and 21 in Table 1), the average performance (AVG) of the model of the present invention compared to the second-best model increased by 2.2%, 2.8%, and 2.0%. This shows that introducing weights for each prompt word and further training is conducive to the optimization of the mapping process. With only a small amount of labeled data, the performance of the datasets in different fields has maintained a certain degree of growth, indicating that the model of the present invention has a certain degree of generalization.
[0071] In order to evaluate the impact of some designs in the model on the final effect, an ablation experiment was conducted. The effects of three aspects of the indicator filtering mechanism were tested on four datasets. The results are as follows: Figure 2 As shown in the figure, "w / o pos" means not using part-of-speech tagging to remove meaningless tokens; "w / o spe" means not using category-specific instructions for the search process; "w / o sem" means not using category-specific instructions.
[0072] Compared to the full model, the significant decrease in the experimental results of the three ablation models demonstrates that these three aspects of the indicator mechanism ensure that the most appropriate cue words are searched for each category, thereby ensuring model performance. Furthermore, it is notable that: 1) the "w / o pos" model performs the worst on all four datasets, and its performance growth rate remains slower than that of the other models. Observing the experimental process reveals that the cue word set selected from the undenoised vocabulary contains a large number of meaningless tokens, and these tokens have a high prediction probability when filling the {MASK} position, resulting in a decrease in verbalizer mapping performance. 2) The performance of the "w / o sem" and "w / o spe" models is similar, indicating that category specificity and category semantic similarity are equally important when searching for cue words. The combined constraints of these two ensure that each category word set not only has a sufficient number of cue words but also avoids mapping conflicts between different categories, facilitating subsequent mapping optimization.
[0073] In addition, we further quantitatively and qualitatively evaluated the performance of different models on the "miscellaneous" and "general" categories. The Amazon dataset contains the "general" category. For the sake of convenience, both labels are collectively referred to as "miscellaneous" below. Figure 3 The experimental results of each model on these two categories under zero-shot and few-shot conditions are shown. Table 2 shows the prompt words of these two categories obtained by different models.
[0074] Table 2 Prompt words for the "miscellaneous" category
[0075]
[0076] (The underlined word in the middle indicates that the token cannot be recognized by the pre-trained language model; the underlined word below indicates that the token appears in other categories)
[0077] In different settings, the proposed model shows excellent results, especially in zero-shot tasks, where the improvement effect is obvious. Compared with the suboptimal model on four datasets, the improvement is 14.1%, 10.6%, 6.9%, and 11.1%, respectively. Figure 2 As shown. Referring to the data in Table 2, it is speculated that since the sentences in the "miscellaneous" category have no obvious features and the range of semantic expression is relatively broad, Manual Prompt only uses category words as prompt words, which obviously cannot cover all the data in this category, and the result is not ideal. Although KPT has expanded the mapping range, the types of prompt words searched from the external knowledge base for this category are richer. According to the data in Table 2, most of these rich prompt words are uncommon and cannot be recognized by PLMs, resulting in less performance improvement in this category. Although the search-based model does not have these two problems, it ignores the confusion problem between categories, which can easily cause model misjudgment during the prediction process. After the model of the present invention focuses on solving the above problems, it can show good capabilities in semantically ambiguous categories such as "miscellaneous". In addition, from a semantic perspective, the prompt words obtained by the present invention have a high correlation with "miscellaneous", so they can more accurately represent categories.
[0078] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.< / mask> < / mask> < / mask> < / mask> < / x> < / mask> < / x> < / mask> < / mask> < / mask> < / mask>
Claims
1. Aspect category detection method based on automatically building a hint learning mapper, characterized by: The specific steps of the method are as follows: Step 1: Use the semantic extension sentence of the category word and the task-specific template to form a prompt to obtain the predicted probability of all tokens in the pre-trained language model vocabulary; Step 2: By designing an automatic filtering mechanism, the most appropriate prompt words are searched in the vocabulary to form a prompt word set for each category of words; Step 3: During the category prediction process, the mapper maps the prompt word to the final category label; In the step 2, an automatic filtering mechanism is proposed to improve the mapper. The automatic filtering mechanism is as follows: for each probability p in the probability matrix P ji Set an indicator value b ji ; The indicator value represents the relevance of the token i to a specific category j. An indicator value of 1 means that the token is very important for the corresponding category, while an indicator value of 0 means the opposite. Initially, all indicator values are set to 1 to form an indicator matrix C represents the number of categories, and D represents the size of the pre-trained language model vocabulary. The indicator matrix is refined by three aspects: part of speech, category semantic relevance, and category specificity to obtain the final indicator matrix to better capture the prompt words related to the aspect category. Finally, the mapper's final prompt word set for each category consists of the corresponding tokens with an indicator value of 1 in the final indicator matrix under that category. The specific steps of refining the indicator matrix by the three aspects of part of speech, category semantic relevance and category specificity to obtain the final indicator matrix are as follows: Step 2.
1. De-noise the vocabulary V, that is, filter out meaningless words through part-of-speech tagging. First, use the pos_tag method in the nltk package to define the token set matching nouns, verbs, and adjectives in the vocabulary V as {pos}, and then adjust the corresponding element values in the indicator matrix B. The indicator matrix B is obtained by formula 1. pos , B pos The calculation formula for the element value in the j-th row and i-th column in is as follows: Among them, b ji For each probability p in the probability matrix P ji An indicator value set, v i Represents the i-th word in the vocabulary; Step 2.2: Further modify the indicator matrix B based on the semantic relevance of the categories pos The element value in gets the indicator matrix B sem , B sem The calculation formula for the element value in the j-th row and i-th column in is as follows: Where MAX_M(.) represents the Mth largest probability value in the vector, represents the probability distribution vector for a given class; Step 2.3: Adjust the matrix B based on category specificity pos The element values in get the updated indicator matrix B spe , B spe The calculation formula for the element value in the j-th row and i-th column in is as follows: Where α is the threshold, indicating that words above this threshold are category-specific; Step 2.
4. Finally, the refined indicator matrix B′ is: where ° represents the hadamard product of two matrices.
2. The aspect category detection method based on automatic construction of prompt learning mappers according to claim 1, characterized in that: In the step 1, the semantic extension of the classifier is adopted, that is, the definition sentence of the classifier on the Wikipedia website and the task-specific <mask>Token template splicing composition prompt Will Send it to the pre-trained language model and get all the tokens in the vocabulary V, that is, words to fill in each <mask>The probability of the position, and then obtain the probability distribution vector of the given category For different categories, construct a complete probability matrix C represents the number of categories, and D represents the size of the pre-trained language model vocabulary.< / mask> < / mask> 3. The aspect category detection method based on automatic construction of prompt learning mappers according to claim 1, characterized in that: In the step 3, the comment sentence to be predicted x i and task-specific <mask>The token templates are concatenated to form natural language text and sent to the pre-trained language model to obtain the probability distribution vector Finally, the mapper maps it to the aspect category label; the mapping process is discussed in two cases; for the zero-shot scenario, it is assumed that all words have the same contribution to the prediction of the corresponding category, so the sentence x is calculated using the following formula i The class probability < / mask> For the few-sample scenario, a weight parameter is set for each prompt word in the prompt word set and further training is performed. Then the sentence x i The category probability is in is the parameter vector to be trained, optimized using cross entropy loss, P is the probability matrix, and B′ is the final indicator matrix after refinement.