A multi-level fusion aspect-category sentiment analysis method based on self-attention

Abstract

This invention discloses a multi-level fusion aspect category sentiment analysis method based on self-attention, belonging to the field of sentiment word analysis. The method mainly includes: acquiring sample sentences and several sets of associated binary tag information; combining Bi-LSTM and convolutional attention mechanisms to acquire sequence information and local importance information of the input text; acquiring the positional information of aspect category indicator words; relearning aspect information using a context-integrated self-attention mechanism; and then introducing a multi-level fusion module to fuse and learn the interactions between multi-level contextual features to achieve aspect category sentiment analysis. This method differentiates aspect representations in different environments and achieves good classification accuracy in multiple domains. This invention provides an effective new solution for aspect category sentiment analysis in online comments and has effective guiding significance for e-commerce platform services, product improvement, and consumer purchasing decisions.

Description

Technical Field

[0001] This invention belongs to the field of sentiment word analysis using self-attention mechanisms, and specifically relates to a multi-level fusion aspect category sentiment analysis method based on self-attention. Background Technology

[0002] Opinions and perspectives are central to almost all human activities and are key influencing factors in our behavior. With the rise of e-commerce platforms, online reviews have become a valuable resource, especially for popular products and services. Identifying and extracting insightful information from these online reviews helps producers and consumers make informed decisions. In other words, reviews are not only an important factor in e-commerce product sales but also, to some extent, an indicator of consumer willingness. However, due to the sheer volume of product reviews, making a simple decision about any product based on every single review is impractical. Therefore, in recent years, considering the efficiency and the need for processing large amounts of online review data, sentiment analysis has been explored across various fields. Aspect-based sentiment analysis (ABSA), as a fine-grained task in the field of sentiment analysis, has received widespread attention in recent years. It mainly comprises two sub-tasks: aspect item sentiment analysis (ATSA) and aspect category sentiment analysis (ACSA). While research on ATSA and ACSA has shown promise in recent years, this invention primarily focuses on the ACSA task.

[0003] The goal of ACSA is to categorize each sentiment based on aspect categories while simultaneously classifying sentiment polarity (typically positive, negative, or neutral). It can be seen that it involves two tasks: aspect category identification (ACD) and aspect sentiment classification (ASC). Unlike the ATSA task, aspect categories in the ACSA task may not appear directly in the sentence and need to be predefined. To identify the sentiment of aspect categories, researchers have dedicated years to developing appropriate contextual features for a given aspect target.

[0004] In the past few decades, early traditional machine learning methods attracted considerable research attention. However, these methods heavily relied on complex manual rules and feature engineering, easily overlooking valuable features. With the widespread application of deep learning technology, data-driven approaches began to address the feature loss problem of traditional methods. Extensive research based on Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM) networks, has been successfully applied to aspect-level sentiment analysis. CNNs can learn local features from temporal or spatial distributions, but they are insufficient in acquiring long-range dependencies and modeling sequence information. Compared to CNNs, RNNs can effectively model sequence information, but lack position invariance and sensitivity to local information. The aforementioned methods use different neural network structures to model the relationship between context and target, but considering all information in the input sequence is overly redundant and does not explicitly model the contribution of each word to identifying the sentiment of a specific target.

[0005] The emergence of attention mechanisms has sparked a surge of interest in natural language processing and computer vision because it can capture the influence of each word on a specific target by mapping semantic associations, and also adds a degree of interpretability to the model. Its variant, self-attention, has also proven effective in natural language processing and image recognition. Therefore, many researchers have begun to apply the advantages of self-attention to this task. Its aim is to obtain a latent representation that considers global context, which can solve the problem of insufficient long-term dependencies. In addition, some studies have constructed graph structures based on dependency trees to implement ACSA, but the limitation is that it requires finding aspect terms related to aspect categories. Some texts have non-standardized issues, and aspect categories are often not explicitly described in the text, making it difficult to explicitly utilize the relationship between context words and aspect categories through language rules (e.g., dependency parsers). Self-attention mechanisms can learn the internal structural features of sentences, identify differences in sentence structure, and achieve better learning results. However, self-attention considers all elements through a weighted averaging operation, which disperses the attention distribution, and has certain limitations in ACSA. This is because it may lead to incorrect aspect category and sentiment matching, resulting in inaccurate results. Therefore, a self-attention network that considers contextual semantics has been proposed and applied to ABSA.

[0006] While previous research has yielded significant improvements, several challenges remain in most practical applications due to the following reasons: First, one challenge of the ACSA task is that aspect categories may not appear in the text. These models utilize given aspects to guide sentence encoding but neglect the fact that the meaning of aspect representations can vary across different contexts. This makes them susceptible to interference from strongly sentimental words related to other aspect categories, resulting in insufficient interaction between context and aspect information and poor performance. Second, most methods ignore the impact of the positional information of important words in a sentence on sentiment polarity, or simply concatenate the target with sentence features to obtain attention weights. However, the resulting positional vectors may be affected by the varying sentiments expressed by words for different aspect representations in different contexts, thus impacting model performance. Finally, the ACSA task focuses more on sectors such as catering and electronics, but has not yet explored the value of online reviews for fresh produce. Fresh grapes are one of the world's most popular fruit varieties, highly appreciated by consumers for their rich nutritional value and sensory characteristics. Therefore, further research is needed to better understand consumer expectations for fresh grapes on e-commerce platforms to improve the quality of merchants and products. Summary of the Invention

[0007] The purpose of this invention is to provide a multi-level fusion aspect-category sentiment analysis method based on self-attention, characterized in that the multi-level fusion aspect-category sentiment analysis method based on self-attention includes:

[0008] S1: Obtain the sample statement and several sets of binary tag information associated with the sample statement;

[0009] S2: A combination of Bi-LSTM and convolutional attention mechanisms is used to obtain sequence information and local importance information of the input text;

[0010] S3: Obtain the location information of aspect category indicator words;

[0011] S4: It incorporates a context-aware self-attention mechanism to relearn aspect information and introduces the fusion of multi-level information to integrate the interaction between multi-level context features, thereby achieving aspect category sentiment analysis. It utilizes both word-level and sentence-level features to extract text features. To address the noise problem that is inevitably introduced when considering context, a gating mechanism is introduced to filter noise information, thereby more accurately representing the acquisition of context-based aspect categories and effectively obtaining the location information of important words.

[0012] In step S1, the binary tag information includes the aspect category and sentiment polarity associated with the sample statement; the initial sentence sequence of the sample statement is obtained, and the binary tag sequence of the sample statement is constructed based on the initial sentence sequence of the sample statement and the binary tag information.

[0013] The sample statement obtained in step S1 includes several words.

[0014] Step S2 obtains the sequence information and local importance information of the input text, which are phrases and word groups.

[0015] The step S3, obtaining the location information of aspect category indicator words, is used to design an aspect category detection method to obtain important words related to aspect categories as aspect category indicator words, thereby obtaining the location information of the words;

[0016] The fusion of multi-level information introduced in step S4 is used to analyze the sentiment of comment text categories, thereby achieving accurate classification of the sentiment distribution of online comments.

[0017] The initial sentence sequence of the sample statements obtained in step S1 specifically includes:

[0018] The sentence sequence S is mapped through an embedding matrix to obtain the embedding vector sequence E = {x1, x2, ..., x...} n},in d e The dimension of word embedding is given by a predefined aspect category a. i ∈A, aspect embeddings can be obtained through mapping. Where d a It is the aspect-embedded vector dimension;

[0019] Embed the words E = {x1, x2, ..., x} n The input is fed into a multi-layer Bi-LSTM, and the output of the (l-1)th layer is fed into the next layer to obtain the hidden context representation. At each time step t, the current hidden state is... for:

[0020]

[0021]

[0022]

[0023] in l∈L represents the number of current Bi-LSTM layers, d s / 2 represents the dimension of a hidden state in a Bi-LSTM layer, and the output of this layer is a sequence of hidden representations.

[0024] In the formula and This represents the forward and backward propagation of the LSTM.

[0025] The Bi-LSTM cannot focus on local features, so it further utilizes a convolutional attention mechanism to extract high-dimensional semantics, representing the relationships between words. It contains a multi-head structure, performing attention operations in different subspaces, but can only consider one head information. On the one hand, the convolutional attention mechanism differs from traditional convolutional neural networks in that it performs feature mapping through filters after convolutional operations on high-dimensional semantics, thus avoiding limitations on long-distance dependency performance due to the size of the convolutional kernel. On the other hand, unlike self-attention mechanisms, it does not utilize information between different parts of the input vector, but is based on the internal information of the filter, preserving both sequence information and invariance to the input. In such a convolutional filtering space, the more important local features are closer to the corresponding filter, while the less relevant parts are farther away. Therefore, it can capture more important local features based on contextual information.

[0026] First, the convolutional attention mechanism accepts the output H of the Bi-LSTM as input, and then the CNN convolutional filter F = [f1, f2, ... f m Apply an n-ary convolution operation to H, where c is the size of the convolution kernel; the feature mapping matrix M∈R can be obtained using the following formula. m×n :

[0027] M = H * F (4)

[0028] In equation (4), * represents the convolution operation; after applying all filters to the output representation, the specific values ​​in the feature mapping matrix have the following meanings:

[0029] m ij =ψ(f i T ·cat(h j h j+1 , ..., h j+c-1 (5)

[0030] Where `cat` represents the concatenation operation, `ψ` represents the non-linear activation function, and `b` is the bias value; the convolutional attention mechanism can be viewed as an information filter that filters out irrelevant semantic information while retaining key information relevant to the context; the output process of the convolutional attention vector is as follows:

[0031] O M =F·M=F·(H*F) (6)

[0032] In formula (6) The module represents the attention weights of the convolution operation between the query (H) and the key (F). This module can effectively extract valuable information from the convolutional filter, which directly affects the hidden representation and captures important local features. The fusion mechanism proposed next does not use any pooling operation after the convolutional layer to maintain the input structure, which can avoid the loss of the input sequence order caused by pooling operation.

[0033] The self-attention mechanism in step S4 relearns aspect information by using an aspect category prediction method constructed with the self-attention mechanism of the scenario to obtain important words related to the aspect category as aspect category indicator words, and the position of the word is used as important position information. The specific steps include:

[0034] The input layer H is first transformed into the candidate query state Q. k Candidate key state K k Sum V k Each contains k heads; d s Mapping 1D space to d k 3D space, where d k =d s / k represents the dimension of each head; similar to multi-head self-attention models, it allows the model to learn semantic information from different representation subspaces, resulting in context-oriented attention output. Each head can be defined as:

[0035]

[0036] Step 4 introduces a gating mechanism to filter noise information, enabling more accurate representation of context-based aspect categories and effective acquisition of key word location information. The introduced context vector C transforms H into an information-based query layer and key layer. The improved {Q} k K k The following is an expression:

[0037]

[0038] in It is a context matrix. For trainable parameters, It is a weight vector, which can avoid due to and The problem of small gradients caused by excessively large values ​​is specifically addressed by assigning a gated unit to learn the context weight vector:

[0039]

[0040] in These are trainable parameters, and σ(·) represents the sigmoid function. As can be seen, the gated unit can quantize and preserve the contextual representation of each head. Then, the outputs of multiple attention models are concatenated to obtain the final output representation O′.

[0041] Introducing contextual information inevitably introduces noise; therefore, a gating mechanism is introduced. This gating mechanism accepts the output of the self-attention mechanism and the context matrix as input, and includes a non-linear activation function and a gate, defined as follows:

[0042] O = F i ⊙O′ (10)

[0043] F = σ f (W f ·O′+U f ·C+b f (11)

[0044] In equations (10) and (11), C represents the context matrix, which is obtained by... k The result is obtained by concatenating the hidden vector at each location with the hidden vector and then passing it to a feedforward linear layer with residual connections. ⊙ represents the Hadamard product, σ f It is the sigmoid function;

[0045] For simplicity, the entire process described above can be expressed as follows:

[0046] O = CGAtt(H, C) (12)

[0047] In Equation (12), CGAtt represents the general term for the above-mentioned self-attention mechanism that includes context; in aspect category sentiment analysis, positional information can focus on aspect information that appears in the sentence, but even if the aspect category does not exist in the text, positional information still plays an important role; if there are words in the sentence that represent aspect information, they can be regarded as aspect indicator words, and their positions can be regarded as the positions of the corresponding aspect categories; by identifying aspect categories, the word weights that have an important influence on aspect categories are obtained, and the positional information of the word is obtained.

[0048] The method for obtaining word weights that have a significant impact on aspect categories through aspect category recognition is as follows:

[0049] Word embeddings are input into an LSTM layer, and the hidden state at each time step t is output to H. t middle:

[0050] H t =LSTM(h t-1 x t (13)

[0051] Based on the fusion mechanism, attention is guided by hidden context to generate an aspect-specific hidden interaction state for aspect r, as shown in the following formula.

[0052]

[0053] For aspect r, the probability of recognizing a given aspect for each word is calculated using an attention mechanism.

[0054]

[0055]

[0056] in These are trainable parameters, making it easy to pay attention to the weights α. r ∈R n It can be represented as M r With μ r Normalized similarity;

[0057] Finally, the fused sentence representation is used as a weighted sum of the contextual hidden states for aspect detection:

[0058]

[0059]

[0060] The specific method for obtaining the location information of the word is as follows:

[0061] Generally, based on reading intuition, emotion words that are closer to a specific target are more likely to express that emotion; however, since the specific target may not appear in the sentence, aspectual attention weight α is used. i To calculate the local representation of words, we design a matrix P to represent the relative distance between words in the text:

[0062]

[0063] Where n represents the text length, i, j∈{1,2,…,n}, and |ij| represents the relative distance between the i-th character and the j-th character;

[0064] In terms of application, attention weights are used to obtain the position vector:

[0065] p l =Pα r (20)

[0066] For each aspect r, the word closest to the aspect introductory word is in p. l Words in the middle position have higher weights than other words. Applying the softmax function to the position vector yields the weight vector:

[0067]

[0068] Finally, the hidden output for positional attention of aspect category r is obtained as follows:

[0069]

[0070] Step 4 introduces the fusion of multi-level information to learn the interaction between multi-level contextual features, thereby achieving aspect category sentiment analysis. The specific method is as follows:

[0071] Aspect categories are closely related to the semantics of sentences, and it is necessary to extract the deep semantic relationships between them. Therefore, a self-attention mechanism is used to process three different inputs. First, the aspect embedding is input into the first CGATT module, and the output of the positional attention layer is used as the environment vector to construct a new aspect embedding based on contextual feature representation. The second and third CGATT modules both use the output of the first CGATT module as the environment vector. The difference is that the second one utilizes and retrieves useful relevant information from the hidden state of the positional attention layer, while the third one directly uses the word vector matrix of the original sentence as input. Then, the outputs of the two CGATT modules are concatenated to obtain the final sentence representation.

[0072] First, the hidden state is used as the global context vector, and then reconstructed. The aspect indicated:

[0073]

[0074] in It contains rich contextual information for operations on context representation vectors; after obtaining the contextual semantic aspect embedding, global information and aspect information are fused from the hidden space and word vector space respectively; for aspect r, it is calculated as follows:

[0075]

[0076]

[0077] The new interactive representation vector C is given by c′ r and c″ r They are formed by connecting them:

[0078] C r =[c′ r ,c″ r ], r=1,2,·…,C (26)

[0079] The above method links a given aspect with contextual features, integrates the relationship between the aspect and contextual words into the representation vector, and combines the interaction vector C...r The input is fed into two fully connected layers containing ReLU activation functions. The weights provided by aspect recognition, combined with the output of the fully connected layers, are then fed into a softmax layer to predict sentiment polarity as the final output. The specific process is as follows:

[0080]

[0081] y r =softmax(p r α r (28)

[0082] in For training parameters;

[0083] For the aspect recognition task, since each prediction is a binary classification problem, the loss function is defined as:

[0084]

[0085] For aspect-based sentiment classification tasks, cross-entropy with L2 regularization is used as the loss function:

[0086]

[0087] Finally, the aspect recognition task and aspect category sentiment classification task are combined into a two-task model, and the parameters in the model are trained using the combined loss function:

[0088]

[0089] Where γ is the weight of aspect category sentiment classification loss, λ represents the L2 regularization parameter, and θ represents all parameters of the model; in addition, a dropout strategy is added to avoid overfitting; finally, a multi-level fusion aspect category sentiment analysis prediction model based on self-attention is obtained.

[0090] The beneficial effects of this invention are as follows:

[0091] 1. This invention further utilizes Bi-LSTM by performing convolution operations on it and then applying feature mapping to the convolutional filters. This preserves sequence information while maintaining invariance to the input. It addresses the problems of LSTM networks being unable to focus on local features, and CNNs being unable to retain sequence information due to the limitation of convolutional kernel size.

[0092] 2. The aspect category recognition method proposed in this invention assigns higher weights to important words, thereby obtaining corresponding location information. This solves the problem that most methods ignore the influence of the location information of important words in a sentence on sentiment polarity, or simply concatenate the target with sentence features to obtain attention weights, thus affecting classification performance.

[0093] 3. Because aspect categories may not appear in the text, these models use given aspects to guide sentence encoding, but they ignore the fact that the meaning of aspect representations may differ in different contexts. This makes them susceptible to interference from strongly sentimental words related to other aspect categories, resulting in insufficient interaction between context and aspect information and poor performance. The multi-level fusion module proposed in this invention obtains different contextual feature representations from multiple subspaces, simultaneously considering multi-level semantic features. Furthermore, considering the impact of noise information, a gating mechanism is added to filter information, improving the interaction between context and aspect categories. Attached Figure Description

[0094] Figure 1 This is a schematic diagram of the overall process of a multi-level fusion aspect-based sentiment analysis method based on self-attention;

[0095] Figure 2 This is a structural diagram of a self-attention-based multi-level fusion aspect-category sentiment analysis model. Detailed Implementation

[0096] This invention provides a multi-level fusion aspect category sentiment analysis method based on self-attention. The invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0097] like Figure 1 The diagram shows the overall workflow of a self-attention-based, multi-level fusion aspect-category sentiment analysis method, which includes the following steps:

[0098] S1: Obtain the sample statement and several sets of binary tag information associated with the sample statement;

[0099] S2: A combination of Bi-LSTM and convolutional attention mechanisms is used to obtain sequence information and local importance information of the input text;

[0100] S3: Obtain the location information of aspect category indicator words;

[0101] S4: It incorporates a context-aware self-attention mechanism to relearn aspect information and introduces the fusion of multi-level information to integrate the interaction between multi-level context features, thereby achieving aspect category sentiment analysis. It utilizes both word-level and sentence-level features to extract text features. To address the noise problem that is inevitably introduced when considering context, a gating mechanism is introduced to filter noise information, thereby more accurately representing the acquisition of context-based aspect categories and effectively obtaining the location information of important words.

[0102] In step S1, the binary tag information includes the aspect category and sentiment polarity associated with the sample statement; the initial sentence sequence of the sample statement is obtained, and the binary tag sequence of the sample statement is constructed based on the initial sentence sequence of the sample statement and the binary tag information.

[0103] The sample statement obtained in step S1 includes several words.

[0104] Step S2 obtains the sequence information and local importance information of the input text, which are phrases and word groups.

[0105] The step S3, obtaining the location information of aspect category indicator words, is used to design an aspect category detection method to obtain important words related to aspect categories as aspect category indicator words, thereby obtaining the location information of the words;

[0106] The fusion of multi-level information introduced in step S4 is used to analyze the sentiment of comment text categories, thereby achieving accurate classification of the sentiment distribution of online comments.

[0107] The initial sentence sequence of the sample statements obtained in step S1 specifically includes:

[0108] The sentence sequence S is mapped through an embedding matrix to obtain the embedding vector sequence E = {x1, x2, ..., x...} n},in d e The dimension of word embedding is given by a predefined aspect category a. i ∈A, aspect embeddings can be obtained through mapping. Where d a It is the aspect-embedded vector dimension;

[0109] Embed the words E = {x1, x2, ..., x} n The input is fed into a multi-layer Bi-LSTM, and the output of the (l-1)th layer is fed into the next layer to obtain the hidden context representation. At each time step t, the current hidden state is... for:

[0110]

[0111]

[0112]

[0113] in l∈L represents the number of current Bi-LSTM layers, d s / 2 represents the dimension of a hidden state in a Bi-LSTM layer, and the output of this layer is a sequence of hidden representations.

[0114] In the formula and This represents the forward and backward propagation of the LSTM.

[0115] The Bi-LSTM cannot focus on local features, so it further utilizes a convolutional attention mechanism to extract high-dimensional semantics, representing the relationships between words. It contains a multi-head structure, performing attention operations in different subspaces, but can only consider one head information. On the one hand, the convolutional attention mechanism differs from traditional convolutional neural networks in that it performs feature mapping through filters after convolutional operations on high-dimensional semantics, thus avoiding limitations on long-distance dependency performance due to the size of the convolutional kernel. On the other hand, unlike self-attention mechanisms, it does not utilize information between different parts of the input vector, but is based on the internal information of the filter, preserving both sequence information and invariance to the input. In such a convolutional filtering space, the more important local features are closer to the corresponding filter, while the less relevant parts are farther away. Therefore, it can capture more important local features based on contextual information.

[0116] First, the convolutional attention mechanism accepts the output H of the Bi-LSTM as input, and then the CNN convolutional filter F = [f1, f2, ... f m Apply an n-ary convolution operation to H, where c is the size of the convolution kernel; the feature mapping matrix M∈R can be obtained using the following formula. m×n :

[0117] M = H * F (4)

[0118] In equation (4), * represents the convolution operation; after applying all filters to the output representation, the specific values ​​in the feature mapping matrix have the following meanings:

[0119] m ij ψ(f i T ·cat(h j h j+1 , ..., h j+c-1 (5)

[0120] Where `cat` represents the concatenation operation, `ψ` represents the non-linear activation function, and `b` is the bias value; the convolutional attention mechanism can be viewed as an information filter that filters out irrelevant semantic information while retaining key information relevant to the context; the output process of the convolutional attention vector is as follows:

[0121] O M =F·M=F·(H*F) (6)

[0122] In formula (6) The module represents the attention weights of the convolution operation between the query (H) and the key (F). This module can effectively extract valuable information from the convolutional filter, which directly affects the hidden representation and captures important local features. The fusion mechanism proposed next does not use any pooling operation after the convolutional layer to maintain the input structure, which can avoid the loss of the input sequence order caused by pooling operation.

[0123] The self-attention mechanism in step S4 relearns aspect information by using an aspect category prediction method constructed with the self-attention mechanism of the scenario to obtain important words related to the aspect category as aspect category indicator words, and the position of the word is used as important position information. The specific steps include:

[0124] The input layer H is first transformed into the candidate query state Q. k Candidate key state K k Sum V k Each contains k heads; d s Mapping 1D space to d k 3D space, where d k =d s / k represents the dimension of each head; similar to multi-head self-attention models, it allows the model to learn semantic information from different representation subspaces, resulting in context-oriented attention output. Each head can be defined as:

[0125]

[0126] Step 4 introduces a gating mechanism to filter noise information, enabling more accurate representation of context-based aspect categories and effective acquisition of key word location information. The introduced context vector C transforms H into an information-based query layer and key layer. The improved {Q} k K k The following is an expression:

[0127]

[0128] in It is a context matrix. For trainable parameters, It is a weight vector, which can avoid due to and The problem of small gradients caused by excessively large values ​​is specifically addressed by assigning a gated unit to learn the context weight vector:

[0129]

[0130] in These are trainable parameters, and σ(·) represents the sigmoid function. As can be seen, the gated unit can quantize and preserve the contextual representation of each head. Then, the outputs of multiple attention models are concatenated to obtain the final output representation O′.

[0131] Introducing contextual information inevitably introduces noise; therefore, a gating mechanism is introduced. This gating mechanism accepts the output of the self-attention mechanism and the context matrix as input, and includes a non-linear activation function and a gate, defined as follows:

[0132] O = F i ⊙O′ (10)

[0133] F = σ f (W f ·O′+U f ·C+b f (II)

[0134] In equations (10) and (11), C represents the context matrix, which is obtained by using C k The result is obtained by concatenating the hidden vector at each location with the hidden vector and then passing it to a feedforward linear layer with residual connections. ⊙ represents the Hadamard product, σ f It is the sigmoid function;

[0135] For simplicity, the entire process described above can be expressed as follows:

[0136] O = CGAtt(H, C) (12)

[0137] In Equation (12), CGAtt represents the general term for the above-mentioned self-attention mechanism that includes context; in aspect category sentiment analysis, positional information can focus on aspect information that appears in the sentence, but even if the aspect category does not exist in the text, positional information still plays an important role; if there are words in the sentence that represent aspect information, they can be regarded as aspect indicator words, and their positions can be regarded as the positions of the corresponding aspect categories; by identifying aspect categories, the word weights that have an important influence on aspect categories are obtained, and the positional information of the word is obtained.

[0138] The method for obtaining word weights that have a significant impact on aspect categories through aspect category recognition is as follows:

[0139] Word embeddings are input into an LSTM layer, and the hidden state at each time step t is output to H. t middle:

[0140] H t =LSTM(h t-1 x t (13)

[0141] Based on the fusion mechanism, attention is guided by hidden context to generate an aspect-specific hidden interaction state for aspect r, as shown in the following formula.

[0142]

[0143] For aspect r, the probability of recognizing a given aspect for each word is calculated using an attention mechanism.

[0144]

[0145]

[0146] in These are trainable parameters, making it easy to pay attention to the weights a. r ∈R n It can be represented as M r With μ r Normalized similarity;

[0147] Finally, the fused sentence representation is used as a weighted sum of the contextual hidden states for aspect detection:

[0148]

[0149]

[0150] The specific method for obtaining the location information of the word is as follows:

[0151] Generally, based on reading intuition, emotion words that are closer to a specific target are more likely to express that emotion; however, since the specific target may not appear in the sentence, aspectual attention weight α is used. i To calculate the local representation of words, we design a matrix P to represent the relative distance between words in the text:

[0152]

[0153] Where n represents the text length, i, j∈{1,2,…,n}, and |ij| represents the relative distance between the i-th character and the j-th character;

[0154] In terms of application, attention weights are used to obtain the position vector:

[0155] p l =Pα r (20)

[0156] For each aspect r, the word closest to the aspect introductory word is in p. l Words in the middle position have higher weights than other words. Applying the softmax function to the position vector yields the weight vector:

[0157]

[0158] Finally, the hidden output for positional attention of aspect category r is obtained as follows:

[0159]

[0160] Step 4 introduces the fusion of multi-level information to learn the interaction between multi-level contextual features, thereby achieving aspect category sentiment analysis. The specific method is as follows:

[0161] Aspect categories are closely related to the semantics of sentences, and it is necessary to extract the deep semantic relationships between them. Therefore, a self-attention mechanism is used to process three different inputs. First, the aspect embedding is input into the first CGATT module, and the output of the positional attention layer is used as the environment vector to construct a new aspect embedding based on contextual feature representation. The second and third CGATT modules both use the output of the first CGATT module as the environment vector. The difference is that the second one utilizes and retrieves useful relevant information from the hidden state of the positional attention layer, while the third one directly uses the word vector matrix of the original sentence as input. Then, the outputs of the two CGATT modules are concatenated to obtain the final sentence representation.

[0162] First, the hidden state is used as the global context vector, and then reconstructed. The aspect indicated:

[0163]

[0164] in It contains rich contextual information for operations on context representation vectors; after obtaining the contextual semantic aspect embedding, global information and aspect information are fused from the hidden space and word vector space respectively; for aspect r, it is calculated as follows:

[0165]

[0166]

[0167] The new interactive representation vector C is given by c′ r and c″ r They are formed by connecting them:

[0168] C r =[c′ r ,c″ r ], r = 1, 2, ..., C (26)

[0169] The above method links a given aspect with contextual features, integrates the relationship between the aspect and contextual words into the representation vector, and combines the interaction vector C...r The input is fed into two fully connected layers containing ReLU activation functions. The weights provided by aspect recognition, combined with the output of the fully connected layers, are then fed into a softmax layer to predict sentiment polarity as the final output. The specific process is as follows:

[0170]

[0171] y r =softmax(p r α r (28)

[0172] in For training parameters;

[0173] For the aspect recognition task, since each prediction is a binary classification problem, the loss function is defined as:

[0174]

[0175] For aspect-based sentiment classification tasks, cross-entropy with L2 regularization is used as the loss function:

[0176]

[0177] Finally, the aspect recognition task and aspect category sentiment classification task are combined into a two-task model, and the parameters in the model are trained using the combined loss function:

[0178]

[0179] Where γ represents the weights of the aspect category sentiment classification loss, λ represents the L2 regularization parameter, and θ represents all model parameters; in addition, a dropout strategy is added to avoid overfitting; the final result is as follows Figure 2 The model shown is a multi-level fusion aspect-based sentiment analysis prediction model based on self-attention.

Claims

1. A multi-level fusion aspect-category sentiment analysis method based on self-attention, characterized in that, The self-attention-based multi-level fusion aspect-category sentiment analysis method includes: Step S1: Obtain the sample statement and several sets of binary tag information associated with the sample statement; Step S2: Use a combination of Bi-LSTM and convolutional attention mechanisms to obtain sequence information and local importance information of the input text; Step S3: Obtain the location information of aspect category indicator words; Step S4: The self-attention mechanism with context is used to relearn aspect information, and multi-level information is introduced to fuse the interaction between multi-level context features to achieve aspect category sentiment analysis; text features are extracted using both word-level and sentence-level methods; to solve the noise problem that is inevitably introduced when considering context, a gating mechanism is introduced to filter noise information, so as to more accurately represent the acquisition of context-based aspect categories and effectively obtain the position information of important words. Introducing contextual information inevitably introduces noise; therefore, a gating mechanism is introduced. This gating mechanism accepts the output of the self-attention mechanism and the context matrix as input, and includes a non-linear activation function and a gate, defined as follows: （1） （2） In formulas (1) and (2) Representing the context matrix, by... The result is obtained by concatenating the hidden vector at each location with the hidden vector and then passing it to a feedforward linear layer with residual connections. , , This represents the Hadamard product. It is the sigmoid function; For simplicity, the process is described as follows: （3） Aspect categories are closely related to the semantics of sentences, and it is necessary to extract the deep semantic relationships between them. Therefore, a self-attention mechanism is used to process three different inputs. First, the aspect embedding is input into the first CGATT module, and the output of the positional attention layer is used as the environment vector to construct a new aspect embedding based on contextual feature representation. The second and third CGATT modules both use the output of the first CGATT module as the environment vector. The difference is that the second one utilizes and retrieves useful relevant information from the hidden state of the positional attention layer, while the third one directly uses the word vector matrix of the original sentence as input. Then, the outputs of the two CGATT modules are concatenated to obtain the final sentence representation.

2. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, In step S1, the binary tag information includes the aspect category and sentiment polarity associated with the sample statement; the initial sentence sequence of the sample statement is obtained, and the binary tag sequence of the sample statement is constructed based on the initial sentence sequence of the sample statement and the binary tag information.

3. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, The sample statement obtained in step S1 includes several words.

4. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, Step S2 obtains the sequence information and local importance information of the input text, which are phrases and word groups.

5. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, Step S3, obtaining the location information of aspect category indicator words, is used to design an aspect category detection method to obtain important words related to aspect categories as aspect category indicator words, thereby obtaining the location information of the words.

6. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, The fusion of multi-level information introduced in step S4 is used to analyze the sentiment of comment text categories, thereby achieving accurate classification of the sentiment distribution of online comments.

7. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, The initial sentence sequence of the sample statements obtained in step S1 specifically includes: The sentence sequence S is mapped using an embedding matrix to obtain an embedding vector sequence. ,in , The dimension representing the word embedding, given a predefined aspect category. Aspect embedding can be obtained through mapping. ,in It is the aspect-embedded vector dimension; Embed words The input is fed into a multi-layer Bi-LSTM, and the output of the (l-1)th layer is fed into the next layer to obtain the hidden context representation. At each time step t, the current hidden state is... for: （4） （5） （6） in , , , This indicates the current number of Bi-LSTM layers. This represents the dimension of a hidden state in a Bi-LSTM layer, whose output is a sequence of hidden representations. In the formula and This represents the forward and backward propagation of the LSTM.

8. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, This paper utilizes convolutional attention to extract high-dimensional semantics, compensating for the limitation of Bi-LSTM in capturing local features. On one hand, convolutional attention maps features through filters, unrestricted by the size of the convolutional kernel. On the other hand, unlike self-attention mechanisms, it does not utilize information between different parts of the input vector, but rather relies on the internal information of the filter, preserving both sequence information and invariance to the input. Therefore, it can both maintain contextual information and capture more important local features. First, the convolutional attention mechanism accepts the output H of the Bi-LSTM as input, and then a CNN convolutional filter... Apply an n-ary convolution operation to H, where Where c is the size of the convolution kernel; the feature mapping matrix can be obtained using the following formula. : （7） In equation (7) This represents a convolution operation; after applying all filters to the output representation, the specific values ​​in the feature map matrix have the following meanings: （8） in This indicates a splicing operation. Represents a non-linear activation function. The bias value is used; the convolutional attention mechanism can be viewed as an information filter that filters out irrelevant semantic information while retaining key information relevant to the context; the output process of the convolutional attention vector is as follows: （9） In formula (9) The attention weights represent the convolutional operation between the query (H) and the key (F). The convolutional attention mechanism can effectively extract valuable information from the convolutional filter, which directly affects the hidden representation and captures important local features. The fusion mechanism proposed next does not use any pooling operation after the convolutional layer to maintain the input structure, which can avoid the loss of the input sequence order caused by the pooling operation.

9. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, The self-attention mechanism in step S4 relearns aspect information by using an aspect category prediction method constructed with the self-attention mechanism of the scenario to obtain important words related to the aspect category as aspect category indicators, and the position of the aspect category indicator is used as important position information. The specific steps include: The input layer H is first transformed into a candidate query state. Candidate key state Sum Each contains k heads; dimensional space mapping to 3D space, in which This represents the dimension of each head; similar to multi-head self-attention models, it allows the model to learn semantic information from different representational subspaces, with context-oriented attention output. Each head can be defined as: （10）。 10. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, Step 4 introduces a gating mechanism to filter noise information, thereby more accurately representing the acquisition of context-based aspect categories and effectively obtaining the location information of important words. The introduced context vector C transforms H into an information-based query layer and key layer. The following is an expression: （11） in It is a context matrix. For trainable parameters, It is a weight vector, which can avoid due to and The problem of small gradients caused by excessively large values ​​is specifically addressed by assigning a gated unit to learn the context weight vector: （12） in These are trainable parameters. The sigmoid function is used to represent the contextual representation of each head. As can be seen, the gating unit quantizes and preserves the contextual representation of each head. Then, the outputs of multiple attention models are concatenated to obtain the final output representation. ; In Equation (3), CGAtt represents the collective term for the above-mentioned self-attention mechanism that includes context. In aspect category sentiment analysis, location information can focus on aspect information that appears in the sentence. However, even if the aspect category does not exist in the text, location information still plays an important role. If there are words in the sentence that represent aspect information, they can be regarded as aspect indicator words, and their positions can be regarded as the positions of the corresponding aspect categories. By identifying aspect categories, the word weights that have an important influence on aspect categories are obtained, and the location information of the word is obtained.

11. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 10, characterized in that, The method for obtaining word weights that have a significant impact on aspect categories through aspect category recognition is as follows: Word embeddings are input into an LSTM layer, and the hidden state at each time step t is output to... middle: （13） Based on the fusion mechanism, attention is guided by hidden context to generate an aspect-specific hidden interaction state for aspect r, as shown in the following formula. （14） For aspect r, the probability of recognizing a given aspect for each word is calculated using an attention mechanism. （15） （16） in , , These are trainable parameters, making it easy to pay attention to the weights. It can be represented as and Normalized similarity; Finally, the fused sentence representation is used as a weighted sum of the contextual hidden states for aspect detection: （17） （18）。 12. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 10, characterized in that, The specific method for obtaining the location information of the word is as follows: Generally, based on reading intuition, emotion words that are closer to a specific target are more likely to express that emotion; however, since the specific target may not appear in the sentence, aspectual attention weighting is used. To compute the local representation of a word, design a matrix. Indicates the relative distance between words in the text: （19） Where n represents the text length. , This represents the relative distance between the i-th character and the j-th character; In terms of application, attention weights are used to obtain the position vector: （20） For each aspect r, the word closest to the aspect conjunction is in Words in the middle have higher weight values ​​than other words. Applying the sqrt function to the position vector yields the weight vector: （21） Finally, the hidden output for positional attention of aspect category r is obtained as follows: （22）。 13. The multi-level fusion aspect-category sentiment analysis method based on self-attention according to claim 1, characterized in that, Step 4 introduces the fusion of multi-level information to learn the interaction between multi-level contextual features, thereby achieving aspect category sentiment analysis. The specific method is as follows: First, the hidden state is used as the global context vector, and then reconstructed. The aspect indicated: （23） in It contains rich contextual information for operations on context representation vectors; after obtaining the contextual semantic aspect embedding, global information and aspect information are fused from the hidden space and word vector space respectively; for aspect r, it is calculated as follows: （24） （25） The new interactive representation vector C is composed of and By connecting them, we can form: （26） The above method links a given aspect with contextual features, integrates the relationship between the aspect and contextual words into the representation vector, and combines the interaction vector... The input is fed into two fully connected layers containing ReLU activation functions. The weights provided by aspect recognition, combined with the output of the fully connected layers, are then fed into a softmax layer to predict sentiment polarity as the final output. The specific process is as follows: （27） （28） in For training parameters; For the aspect recognition task, since each prediction is a binary classification problem, the loss function is defined as: （29） For aspect-based sentiment classification tasks, cross-entropy with L2 regularization is used as the loss function: （30） Finally, the aspect recognition task and aspect category sentiment classification task are combined into a two-task model, and the parameters in the model are trained using the combined loss function: （31） in The weights of the sentiment classification loss for each aspect category. This represents the L2 regularization parameter. This represents all the parameters of the model; in addition, a dropout strategy was added to avoid overfitting; finally, a multi-level fusion aspect-based sentiment analysis prediction model based on self-attention was obtained.

Images