Aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction
By using generative knowledge enhancement and implicit masking prediction, the prompt information and loss function of the large language model are optimized, which solves the misjudgment problem in implicit sentiment analysis and achieves more efficient sentiment analysis results.
Patent Information
- Application Number
- CN202511156093.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-18
- Publication Date
- 2025-11-07
AI Technical Summary
Existing technologies struggle to effectively handle implicit sentiment analysis. Pre-trained large language models are prone to misjudging objective, neutral comments as having emotional connotations, resulting in a high misjudgment rate.
We employ generative knowledge enhancement and implicit masking prediction to guide a large language model to focus on the latent semantic intent in the text. By combining word mappers and label smoothing cross-entropy loss, we optimize the performance of the sentiment analysis model.
It significantly improves the accuracy of the model in implicit sentiment analysis, reduces the false positive rate, enhances the automation of sentiment analysis, and reduces the need for manual annotation.
Smart Images
Figure CN120911476A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to an aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction, and belongs to the technical fields of Internet and artificial intelligence. BACKGROUND
[0002] In recent years, aspect-based sentiment analysis (ABSA) as a specialized branch of sentiment analysis (SA) has attracted extensive attention in the academic community. The goal of the present application is to develop a prediction model based on manually annotated sentences to determine the sentiment polarity of each specific aspect word. For example, given the sentence "My laptop with Windows 7 crashed and I did not want Windows 8.", the present application aims to automatically identify the sentiment polarity of the aspect terms "Windows 7" (negative) and "Windows 8" (negative).
[0003] Over the past decade, a large number of studies have been devoted to solving the ABSA task. Some early studies employ attention mechanisms to capture sentiment expressions directed at specific aspects. In addition, some methods construct dependency syntax trees for sentences to further connect aspect words with opinion words and use graph convolution networks to generate aspect word representations. Both of these methods are suitable for handling the ABSA task, as they aim to identify specific aspects, usually opinion words, in the input text to capture the corresponding sentiment expressions. Recently, many researchers have adopted the approach of combining pre-trained large language models (LLMs) with prompt engineering techniques to fully leverage their capabilities in the ABSA task, achieving good results.
[0004] However, the above methods pay less attention to implicit sentiment analysis (ISA) scenarios. ISA involves sentiment expressions that are only composed of factual descriptions, lack explicit opinion expressions, but can still convey clear emotional tendencies in the context. For example: "Having USB3 is why I bought this Mini.", which mainly explains the reason for buying the "Mini" model, and does not directly use language expressing emotions about USB3, but can be clearly interpreted as a positive emotion. In addition, implicit sentiment expressions are prevalent in aspect-level sentiment analysis. For example, in commonly used benchmark datasets, 27.47% of reviews in the restaurant dataset and 30.09% of reviews in the laptop dataset contain implicit sentiment. In addition, current powerful pre-trained large language models also have difficulty handling this task. When they process objective and neutral comments in the dataset, they tend to classify them as emotionally charged comments, resulting in a large number of misjudgments. Therefore, analyzing such implicit sentiment expressions is both crucial and challenging.
[0005] To solve the above technical problems, the present application proposes a novel aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction. First, the present application makes full use of the extensive knowledge acquisition capabilities of pre-trained large language models (LLM) and generates information that helps solve aspect-level sentiment analysis (ABSA) tasks through effective guidance, thereby making up for the lack of external knowledge acquisition of pre-trained language models.
[0006] Second, the present application designs a prompt mechanism specifically for guiding large language models to focus on the actual semantic intent in the text, and converts the sentiment analysis problem into a masked language modeling generation problem. Through prompt tuning technology based on pre-trained language models, the sentiment polarity of specific aspects in the review is generated. This method can encourage the model to focus on information not explicitly expressed in the sentence, thereby significantly improving the performance of the model in the implicit sentiment analysis (ISA) scenario. Third, to further improve the adaptability and generalization ability of the model, the present application designs a word mapper, which can automatically select the most suitable words to map to each sentiment category according to different datasets. Compared with the fixed word mapping method, this dynamic mechanism can effectively avoid the selection of inappropriate words by the model, thereby significantly improving the overall performance and stability of the model. SUMMARY
[0007] In order to solve the problems and deficiencies in the prior art, the present application proposes an aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction, which enhances the ability of large language models to acquire and utilize task-related knowledge, thereby significantly improving the overall performance of the model in implicit sentiment analysis (ISA) and explicit sentiment analysis (ESA). The present application optimizes and improves the construction method of prompt information, enabling the model to focus more on identifying potential real sentiment content in the text, thereby achieving significant performance improvement in implicit sentiment analysis. The present application also designs a word mapper to expand the selection range of label words using a knowledge graph, enabling the model to select the most suitable words to map to each sentiment category, thereby distinguishing the real sentiment labels on the aspect items.
[0008] In order to achieve the above-mentioned purpose, the technical scheme of the present application is as follows: an aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction, the method comprising the following steps:
[0009] Step 1: sentiment text normalization;
[0010] Step 2: generative knowledge enhancement architecture;
[0011] Step 3: implicit mask sentiment prediction paradigm;
[0012] Step 4: word mapper construction;
[0013] Step 5: model loss training and sentiment analysis.
[0014] As an improvement of the present application, step 1: sentiment text normalization, first, the crawler technology and data preprocessing technology are used to obtain a dataset containing aspect-level sentiment analysis, and the dataset is normalized to contain one sentence X and one specific aspect item w for each group of data. The purpose is to judge the sentiment of the aspect item in the sentence.
[0015] As an improvement of the present application, step 2: generating the enhanced knowledge architecture, the pre-trained large language model (LLM) will usually attach the corresponding explanation when providing the answer. These explanations contain a large amount of practical knowledge related to problem solving, effectively saving the step of knowledge retrieval or additional reasoning model required in traditional enhancement methods. In order to guide the large model to generate valuable explanatory knowledge related to sentiment classification tasks, the present application uses structured templates to prompt the LLM. Assuming there is a set of data, X1 = "this laptop has only 2 usb ports." w1 = "usb ports", the final prompt is: "Given the sentence X1, based on the sentence, is the description of w1 just a statement, or does it contain implicit opinion towards w1".
[0016] The present application guides the LLM to think from two aspects. The first aspect is to judge from the surface whether the description of w1 is an objective statement. The second aspect is to analyze the potential opinion or attitude towards w1 implied in X1 from a deeper level. The first aspect helps to effectively prevent the model from misjudging objective descriptions as subjective emotional expressions, thereby reducing errors caused by improper use of knowledge in the reasoning process of the second aspect. The second aspect guides the model to find the potential opinion for the mentioned target w1. As mentioned earlier, the corresponding emotional expression for a certain aspect is usually a viewpoint word that clearly expresses the emotional polarity. Therefore, when the large model extracts the hidden opinion, it will also return the information related to w1. The reasoning process of the large language model contains a large amount of useful knowledge, which is of great value for correct analysis of sentiment. The present application retains the reasoning part of the LLM answer as external knowledge k to avoid noise interference caused by the model giving wrong answers. In order to enhance the expressiveness and information quantity of the data set, the present application combines the knowledge obtained by the large language model with the constructed prompts, thereby realizing knowledge enhancement.
[0017] As an improvement of the present application, step 3: implicit mask sentiment prediction paradigm, uses the and data obtained in step 2 and the knowledge related to the sentiment classification task, and integrates it into the prompt by constructing a prompt. In addition, the model of the present application for processing the sentiment classification task is the BERT mask language model (MLM), and for this purpose, the classification task is converted into an MLM task. Specifically, specific markers [MASK] are added in the input prompt to guide the model to predict these positions, thereby completing the sentiment classification. The present application also fully considers the characteristics of implicit emotional expression when constructing the prompt template. A representative feature of implicit sentiment analysis is that the text does not directly express the target emotional word, and the true intention is often hidden behind the literal content. For implicit text, an effective prompt template should reflect the potential emotions embedded in the original text; while for non-implicit text, it should be consistent with the expressed intention. Therefore, the present application uses fact-related sentences to emphasize attention to hidden opinions in sentences, while not affecting the extraction of emotional information from non-implicit emotional texts. In addition, the present application uses "!" at multiple places to enhance the semantic expression effect. The present application simulates the tone of the speaker and designs five guiding prompts. T1: "X k As a matter of fact, w was actually [MASK]!";T2: "X k To tell you the truth, w was actually [MASK]!";T3: "X k To my surprise, w was actually [MASK]!";T4: "X k Believe it or not, w was actually [MASK]!" T5: "X k Can you believe it, w was actually [MASK]!". Phrases such as "Believe it or not" and "To my surprise" are used to dig up the true sentiment of aspect word w in sentence X. The present application places external knowledge k after X to ensure the clarity and orderliness of the prompt structure. The semantic representation of the [MASK] position represents the complete semantic information by capturing the global semantic structure of the sentence. First, put the sentence X, knowledge k, aspect item w into the prompt template T to obtain the input sequence T S . Then, input the sequence T S into the BERT model to obtain the hidden state H S of the last layer. Finally, extract the semantic representation of the [MASK] marker as the sentence-level semantic representation V S of the target word:
[0018] T S = tokenizer(S)
[0019] Hs = BERT(T S )
[0020]
[0021] where, denotes the hidden state corresponding to the [MASK] token. In addition, define where n is the length of the tokenized sequence T S and d is the hidden layer dimension of the BERT model.
[0022] As an improvement of the present application, step 4: Verbalizer construction, uses the sentiment cues obtained in step 3 to explore the words that can be chosen for the [MASK] position in the cue. The present application can obtain a new input T by merging the original input with the cue template. BERT will then predict the word for the [MASK] position in the new input. Predicting the masked word based on the context is not a single-choice task, that is, there is no unique correct answer. In fact, there are a variety of words in the rich vocabulary that can reasonably fill this context. Therefore, the construction of the Verbalizer is very crucial. The purpose of the Verbalizer construction is to identify the Label Words that should be placed in the masked position and to specify how these Label Words map to their corresponding labels. The implementation of this step can be divided into the following sub-steps:
[0023] Sub-step 4-1, find Label Words. The process of predicting the masked word based on the context is not a single-choice task; there is no unique correct answer, but there are many words that can reasonably fill the context. Therefore, the present application selects Related Words as an external knowledge base to expand the coverage of Label Words. Related Words is a knowledge graph aggregated from multiple resources These resources include word embeddings, knowledge graph ConceptNet, and knowledge graph WordNet, etc. However, if there are too many Label Words, it will interfere with the model's judgment. Therefore, the present application selects the top 10 most relevant words for each category word (such as positive, negative, neutral), and eliminates those words that do not meet the respective categories. The selected relevant words and category words are collectively referred to as Label Words, and these Label Words are organized into a Label Words vocabulary V. Subsequently, BERT can output a probability distribution on this finely constructed Label Words vocabulary.
[0024] Sub-step 4-2, static word mapper. After establishing the label words, the predicted probability of the label words needs to be mapped to the final label y. The present invention assumes that each label word in each class has the same influence on label prediction. Therefore, the present invention uses the average of the label word prediction scores to represent the probability of the class. The probability calculation formula of each label is:
[0025]
[0026] where, define: M is a pre-trained language model (BERT), and V y is the subset of the vocabulary V mapped to a specific label y. The model M can score the probability of each word v in the vocabulary filling the position of the [MASK] token, that is, give the conditional probability P M ([MASK]=v|x p ).
[0027] Sub-step 4-3, dynamic word mapper. The present invention also designs a dynamic word mapper, which can use more resources on the basis of the original static word mapper to obtain better results than the static word mapper. Pre-trained large language models (LLM) are prone to errors when judging neutral sentiment (neutral) and polar sentiment (positive, negative). However, if it is only asked to judge the polarity direction of the sentiment (i.e. positive or negative), its performance is very good. This shows that LLM can effectively exclude the least likely sentiment option in a three-class sentiment analysis task, thereby assisting the model to focus more accurately on the remaining candidate categories. Therefore, the present invention instructs the LLM to identify all possible sentiment categories among the given three options, and lets the BERT make a selection among the sentiment categories returned by the LLM. The likelihood score of each class is the same as that in the static word mapper, but the final class selection depends on the sentiment category returned by the LLM. The present invention designs the following prompt words to better utilize the capabilities of LLM, thereby assisting the pre-trained language model (BERT) in sentiment analysis. The prompt content is as follows: “Given the sentence X, what are all the possible sentiment polarities towards w? Finally, start a new line and ensure that the last line contains only possible labels (positive, neutral, or negative).”
[0028] As an improvement of the present application, step 5: model loss training and sentiment analysis, according to the design of the previous 4 steps, the present application uses label smoothing cross-entropy loss, aiming to optimize the classification performance of the model and the learning of the embedding space at the same time, thereby enhancing the model's ability to analyze sentiment. Label smoothing cross-entropy loss is a widely used loss function in classification tasks, which improves the traditional cross-entropy loss function by smoothing the labels. The traditional cross-entropy loss function assumes that the labels of the training samples are absolutely correct, which may lead to overfitting of the model to the training data and is not robust enough for predictions with uncertainty. When using label smoothing, the probability of the true label is not set to 100% certainty, but is replaced by a smaller certainty (such as 0.9), and the remaining probability mass is evenly distributed to other classes (the probability of each class increases (1-certainty) / (number of classes-1)). Specifically, the formula is as follows:
[0029]
[0030] wherein, w i is the class weight, p i is the model's prediction probability for each class, C is the total number of classes, and ∈ is the smoothing factor. By introducing the label smoothing technique, not only can the overfitting phenomenon be reduced, but also the accuracy and robustness of the model in aspect-level sentiment analysis tasks can be improved to some extent, especially in the face of label noise or ambiguous boundaries between classes.
[0031] The present application has significant practical value in the field of aspect-level sentiment analysis: in view of the industry pain points of difficult to capture implicit emotional expression and high cost of manual annotation, the present application automatically refines the explanatory knowledge in the large language model through the generative knowledge enhancement architecture, replacing traditional manual rules or external knowledge bases, thereby reducing the complexity of semantic analysis; innovatively reforming the MLM task of BERT by combining the implicit mask sentiment prediction paradigm, and explicitly modeling potential viewpoints through prompt templates, so that the model's recognition accuracy of implicit negative evaluations such as "few USB interfaces" is improved. This scheme can efficiently realize fine-grained decision support, accurately positioning user sentiment tendencies such as "mobile phone battery life" in e-commerce review analysis, social media public opinion monitoring, and other scenarios, helping product iteration; operational efficiency upgrade, automatically processing millions of comments, reducing the need for a large amount of manual annotation; implicit intent mining, identifying the true sentiment of complex expressions such as irony in "this packaging is really environmentally friendly", and avoiding the risk of misjudgment.
[0032] An electronic device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction when executing the program.
[0033] A storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction.
[0034] Compared with the prior art, the advantages of the present application are as follows:
[0035] (1) The present application introduces an aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction, which combines a pre-trained model with strong learning language information, focuses on mining complex hidden sentiment situations, and deeply explores the sentence content related to the aspect item to accurately understand the sentence semantics and determine the true sentiment.
[0036] (2) The present application introduces a knowledge enhancement mechanism based on a pre-trained large language model (LLM). At the same time, a prompt tuning module is designed and optimized to guide the model to focus on the potential emotional expression in the target sentence, and the word mapping component is improved to fully utilize the advantages of the model. The overall performance of sentiment analysis is significantly improved, especially in implicit sentiment analysis (ISA).
[0037] (3) The present application can effectively improve the aspect-level sentiment analysis capability of the model, optimize the detection effect, and significantly improve the practical value of various natural language processing related systems: by automatically generating explanatory knowledge and intelligent prompt templates, the dependence on manual annotation data and domain knowledge in traditional sentiment analysis is greatly reduced; in e-commerce comment analysis, social media monitoring and other practical scenarios, it can accurately identify the aspect-level sentiment tendency in complex expressions such as "battery endurance is short but the screen is good", providing fine-grained data support for enterprise product optimization and market decision-making; in intelligent customer service systems, it can analyze the implicit sentiment in user feedback in real time, improve the accuracy of service response, etc. The improvement of aspect-level sentiment analysis capability enhances the practical value of various text understanding systems, and exhibits significant cost-benefit advantages in actual business scenarios, with clear technical protection value and business application prospects. BRIEF DESCRIPTION OF DRAWINGS
[0038] Figure 1 The overall model diagram of the embodiment of the present application;
[0039] Figure 2 The method processing flowchart of the embodiment of the present application. DETAILED DESCRIPTION
[0040] In order to deepen the understanding and understanding of the present application, the following specific examples are further illustrated.
[0041] Embodiment 1: An aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction, which enhances the ability of large language models to acquire and utilize task-related knowledge, thereby significantly improving the overall performance of the model in implicit sentiment analysis (ISA) and explicit sentiment analysis (ESA). The present application optimizes and improves the construction method of the prompt information, so that the model can focus more on identifying the potential true sentiment content in the text, thereby achieving significant performance improvement in implicit sentiment analysis. The present application also designs a word mapper to expand the selection range of label words using a knowledge graph, so that the model can select the most appropriate words to map to each sentiment category, thereby distinguishing the true sentiment label on the aspect item.
[0042] The specific model is described in Figure 1 , and the detailed implementation steps are as follows:
[0043] Step 1: Normalize the sentiment text, first use the crawler technology and data preprocessing technology to obtain the data set containing aspect-level sentiment analysis, and normalize it to each group of data containing a sentence X and a specific aspect item w. The purpose is to judge the sentiment of the aspect item in the sentence.
[0044] Step 2: Generative knowledge enhancement architecture, pre-training large language model (LLM) usually provides answers along with corresponding explanations. These explanations contain a lot of practical knowledge related to problem solving, effectively saving the steps of knowledge retrieval or additional reasoning model required in traditional enhancement methods. In order to guide the large model to generate valuable explanatory knowledge related to sentiment classification tasks, the present application uses a structured template to design prompts for LLM. Assuming there is a set of data, X1 = "this laptop has only 2usb ports." w1 = "usb ports", the final prompt is: "Given the sentence X1, based on the sentence, is the description of w1 just a statement, or does it contain implicit opinion towards w1".
[0045] The present application guides the LLM to think from two aspects. The first aspect is to judge whether the statement about w1 is an objective statement from the surface. The second aspect is to analyze the potential view or attitude towards w1 implied in X1 from a deeper level. The first angle helps to effectively prevent the model from misjudging objective descriptions as subjective emotional expressions, thereby reducing errors caused by the use of inappropriate knowledge in the reasoning process of the second angle. The second angle guides the model to find out the potential view for the mentioned target w1. As mentioned earlier, the corresponding emotional expression for a certain aspect is usually a view word that explicitly expresses the emotional polarity. Therefore, when the large model extracts the hidden view, it will also return the information related to w1. The reasoning process of the large language model contains a lot of useful knowledge, which is of great value for correct analysis of emotions. The present application retains the reasoning part in the answer of the LLM as external knowledge k to avoid the noise interference caused by the model giving wrong answers. In order to enhance the expressiveness and information quantity of the data set, the present application combines the knowledge obtained by the large language model with the constructed prompts, thereby realizing knowledge enhancement.
[0046] Step 3: Implicit Masked Sentiment Prediction Paradigm, using the data obtained in step 2 and the knowledge related to the sentiment classification task, it is integrated into the prompt by constructing a prompt. In addition, the model of the present invention for processing the sentiment classification task is the BERT Masked Language Model (MLM), and for this purpose, the classification task is converted into an MLM task. Specifically, specific markers [MASK] are added in the input prompt to guide the model to predict these positions, thereby completing the sentiment classification. The present invention also fully considers the characteristics of implicit sentiment expression when constructing the prompt template. A representative feature of implicit sentiment analysis is that the text does not directly express the target sentiment word, and the true intention is often hidden behind the literal content. For implicit text, an effective prompt template should reflect the potential sentiment embedded in the original text; while for non-implicit text, it should be consistent with the expressed intention. Therefore, the present invention uses fact-related sentences to emphasize attention to hidden opinions in sentences while not affecting the extraction of sentiment information from non-implicit sentiment text. In addition, the present invention uses "!" at multiple places to enhance the semantic expression effect. The present invention imitates the tone of the speaker and designs five guiding prompts. T1: "X k As a matter of fact, w was actually [MASK]!" ; T2: "X k To tell you the truth, w was actually [MASK]!" ; T3: "X k To my surprise, w was actually [MASK]!" ; T4: "X k Believe it or not, w was actually [MASK]!" T5: "X k Can you believe it, w was actually [MASK]!" Phrases such as "Believe it or not" and "To my surprise" are used to dig up the true sentiment of aspect word w in sentence X. The present invention places external knowledge k after X to ensure the clarity and orderliness of the prompt structure. The semantic representation of the [MASK] position represents the complete semantic information by capturing the global semantic structure of the sentence. First, put the sentence X, the knowledge k, the aspect item w into the prompt template T to obtain the input sequence T S . Then, input the sequence T S into the BERT model to obtain the hidden state H S of the last layer. Finally, extract the semantic representation of the [MASK] marker as the sentence-level semantic representation V S of the target word:
[0047] T S = tokenizer(S)
[0048] H sBERT(T S )
[0049]
[0050] where, denotes the hidden state corresponding to the [MASK] token. In addition, define where n is the length of the tokenized sequence T S . d is the hidden layer dimension of the BERT model.
[0051] As an improvement of the present application, step 4: Verbalizer construction, uses the sentiment cues obtained in step 3 to explore the words that can be selected for the [MASK] position in the cue. The present application can obtain a new input T by merging the original input with the cue template. BERT will predict the word in the [MASK] position in the new input. Predicting the masked word based on the context is not a single-choice task, that is, there is no unique correct answer. In fact, there are a variety of words in the rich vocabulary that can reasonably fill this context. Therefore, the construction of the Verbalizer is very crucial. The purpose of the Verbalizer construction is to identify the Label Words that should be placed in the masked position and to specify how these Label Words map to their corresponding labels. The implementation of this step can be divided into the following sub-steps:
[0052] Sub-step 4-1, find Label Words. The process of predicting the masked word based on the context is not a single-choice task; there is no unique correct answer, but there are many words that can reasonably fill the context. Therefore, the present application selects Related Words as an external knowledge base to expand the coverage of Label Words. Related Words is a knowledge graph aggregated from multiple resources These resources include word embeddings, knowledge graph ConceptNet, and knowledge graph WordNet, etc. However, if there are too many Label Words, it will interfere with the model's judgment. Therefore, the present application selects the top 10 most relevant words for each category word (such as positive, negative, neutral), and eliminates those words that do not meet the respective categories. The selected relevant words and category words are collectively referred to as Label Words, and these Label Words are organized into a Label Words vocabulary V. Subsequently, BERT can output a probability distribution on this finely constructed Label Words vocabulary.
[0053] Sub-step 4-2, static word mapper. After establishing the label words, the predicted probability of the label words needs to be mapped to the final label y. The present invention assumes that each label word in each class has the same influence on label prediction. Therefore, the present invention uses the average of the label word prediction scores to represent the probability of the class. The probability calculation formula of each label is:
[0054]
[0055] where, define: M is a pre-trained language model (BERT), and V y is the subset of the vocabulary V mapped to a specific label y. The model M can score the probability of each word v in the vocabulary filling the position of the [MASK] token, that is, give the conditional probability P M ([MASK]=v|x p ).
[0056] Sub-step 4-3, dynamic word mapper. The present invention also designs a dynamic word mapper that can use more resources on the basis of the original static word mapper to obtain better results than the static word mapper. Pre-trained large language models (LLM) are prone to errors when judging neutral sentiment and polar sentiment (positive, negative). However, if it is only asked to judge the polarity direction of sentiment (i.e. positive or negative), its performance is very good. This shows that LLM can effectively exclude the least likely sentiment option in a three-class sentiment analysis task, thereby assisting the model to focus more accurately on the remaining candidate categories. Therefore, the present invention instructs the LLM to identify all possible sentiment categories among the given three options, and lets the BERT make a selection among the sentiment categories returned by the LLM. The likelihood score of each class is the same as in the static word mapper, but the final class selection depends on the sentiment category returned by the LLM. The present invention designs the following prompt words to better utilize the capabilities of LLM, thereby assisting the pre-trained language model (BERT) in sentiment analysis. The prompt content is as follows: “Given the sentence X, what are all the possible sentiment polarities towards w? Finally, start a new line and ensure that the last line contains only possible labels (positive, neutral, or negative).”
[0057] Step 5: Model loss training and sentiment analysis. According to the design of the previous four steps, the present application uses a label smoothing cross-entropy loss, aiming to optimize the classification performance of the model and the learning of the embedding space at the same time, thereby enhancing the model's ability to analyze sentiment. Label smoothing cross-entropy loss is a widely used loss function in classification tasks, which improves the traditional cross-entropy loss function by smoothing the labels. The traditional cross-entropy loss function assumes that the labels of the training samples are absolutely correct, which may lead to overfitting of the model to the training data and is not robust enough for predictions with uncertainty. When using label smoothing, the probability of the true label is not set to 100% certainty, but is replaced by a smaller certainty (such as 0.9), and the remaining probability mass is evenly distributed to other classes (the probability of each class increases (1-certainty) / (class number-1)). Specifically, the formula is as follows:
[0058]
[0059] wherein the definition, w i is the class weight, p i is the model's prediction probability for each class, C is the total number of classes, and ∈ is the smoothing factor. By introducing the label smoothing technique, not only can the overfitting phenomenon be reduced, but also the accuracy and robustness of the model in the aspect-level sentiment analysis task can be improved to some extent, especially in the face of label noise or ambiguous boundaries between classes.
[0060] Based on the same inventive concept, the aspect-level sentiment analysis method and device based on generative knowledge enhancement and implicit mask prediction of the present application include a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is loaded into the processor, it implements the aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction described above.
[0061] Those skilled in the art will appreciate that the embodiments described herein are intended to help the reader understand the principles of the present application and should be understood to be used only to illustrate the present application and not to limit the scope of the present application. After reading the present application, those skilled in the art will make various modifications to the present application, all of which fall within the scope of the present application as defined by the claims.
Claims
1. An aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction, characterized in that, The method comprises the following steps: Step 1: sentiment text normalization; Step 2: generative knowledge enhancement architecture; Step 3: implicit mask sentiment prediction paradigm; Step 4: word mapper construction; Step 5: model loss training and sentiment analysis.
2. The aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction according to claim 1, characterized in that, Step 1: sentiment text normalization, first, use crawler technology and data preprocessing technology to obtain a dataset containing aspect-level sentiment analysis, and normalize it to contain one sentence X and one specific aspect item w for each group of data.
3. The aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction according to claim 1, characterized in that, Step 2: generative knowledge enhancement architecture, use structured templates to design prompts for LLM, assuming there is a set of data, X1 = "this laptop has only 2usb ports." w1 = "usb ports", the final prompt is: "Given the sentence X1, based on the sentence, is the description of w1 just a statement, or does it contain implicit opinion towards w1".
4. The aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction according to claim 1, characterized in that, Step 3: Implicit Masking Sentiment Prediction Paradigm, as follows, will add specific markers [MASK] in the input prompt to guide the model to predict these positions, thereby completing sentiment classification, placing external knowledge k after X to ensure the clarity and orderliness of the prompt structure, the semantic representation of the [MASK] position represents the complete semantic information by capturing the global semantic structure of the sentence, first, put the sentence X, knowledge k, aspect item w into the prompt template T to obtain the input sequence T S , then input the sequence T S into the BERT model to obtain the last layer hidden state H S , finally extract the semantic representation of the [MASK] marker as the sentence-level semantic representation V S of the target word: T S = tokenizer(S) H s = BERT(T S ) wherein, represents the hidden state corresponding to the [MASK] token, defined as where n is the length of the tokenized sequence T S and d is the hidden layer dimension of the BERT model.
5. The aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction according to claim 1, characterized in that, Step 4: word mapper construction, by merging the original input and the prompt template, a new input T is obtained, BERT will predict the word at the [MASK] position in the new input, predicting the masked word according to the context is not a single selection task, including the following sub-steps: Sub-step 4-1, find the label word, The process of predicting masked words based on context is not a multiple-choice task; there is no one correct answer, and Related Words, a knowledge graph aggregated from multiple resources, is chosen as an external knowledge base to expand the coverage of label words These resources include word embeddings, the knowledge graph ConceptNet, and the knowledge graph WordNet. However, if there are too many label words, it may interfere with the model's judgment. For each category word, the top 10 most relevant words are selected, and those that do not meet the respective category are removed. The selected related words and category words are collectively referred to as label words. These label words are organized into a label word vocabulary V. Subsequently, BERT outputs a probability distribution on this finely constructed label word vocabulary. Sub-step 4-2, static word mapper, After establishing the label word, the prediction probability of the label word needs to be mapped to the final label y, assuming that each label word in each category has the same influence on label prediction, the average of the label word prediction scores is used to represent the probability of this category, the probability calculation formula of each label is: wherein, the definition: M is a pre-trained language model (BERT), denoted as V y is a subset of the vocabulary V mapped to a specific label y, the model M scores the probability of each word v in the vocabulary filling the position of the [MASK] token, that is, gives the conditional probability P M ([MASK] = v | x p ); Sub-step 4-3, dynamic word mapper, Design the following prompt words to better utilize the capabilities of LLM, thereby assisting the pre-trained language model (BERT) in sentiment analysis, the prompt content is as follows: "Given the sentence X, what are all the possible sentiment polarities towards w? Finally, start a new line and ensure that the last line contains only possible labels (positive, neutral, or negative)".
6. The aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction according to claim 1, characterized in that, Step 5: model loss training and sentiment analysis, as follows: where the definitions are, w i is the class weight, p i is the model's predicted probability for each class, C is the total number of classes, and ∈ is a smoothing factor.
7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: The processor implements the program to realize the aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction as claimed in any one of claims 1 to 6.
8. A storage medium having stored thereon a computer program, characterized in that: The computer program, when executed by a processor, implements the aspect-level sentiment analysis method based on generative knowledge enhancement and implicit mask prediction according to any one of claims 1-6.