A Feature Enhancement Method for Multimodal Aspect-Level Sentiment Analysis

Abstract

This invention relates to the fields of natural language processing and sentiment analysis, specifically a feature enhancement method for multimodal aspect-level sentiment analysis. The method includes: extracting feature information of aspect words; extracting effective information from image regions; matching aspect words using a trilinear similarity algorithm; fusing the feature representation with the highest matching degree with the aspect words using an attention mechanism; obtaining preliminary feature information of the text; then obtaining part-of-speech related information of the text words; constructing a syntactic dependency graph pointing to aspect words; adding a multi-head cross-attention mechanism to weight text features according to aspect words; performing normalization processing on the original image; extracting image features using ResNet-152; inputting aspect words and image modal feature representations into the multi-head cross-attention mechanism to distinguish the relevance of image information; and for image and text modal feature representations containing aspect words, adding the sentiment information related to aspect words in the image to the final feature representation using a concatenation weighting method.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and sentiment analysis, and in particular to a feature enhancement method for multimodal aspect-level sentiment analysis. Background Technology

[0002] For early plain text statements, researchers focused on aspect-based sentiment analysis (ABSA) tasks, such as... Figure 1 As shown, current ABSA models have moved away from the previous approach of predicting the overall sentiment trend of text, and have yielded many results in judging the sentiment polarity of specific aspect words in the text. In recent years, as the data type of online comments has shifted from pure text to image-text modality, researchers have turned their attention to multimodal aspect-based sentiment analysis (MABSA). Figure 2 As shown, MABSA can analyze the sentiment polarity of words related to a specific aspect from both textual and image perspectives. The challenge of existing MABSA models lies in how to acquire feature information focused on aspect-specific words.

[0003] Most MABSA models, when representing text modalities, only use LSTM or BERT to extract semantic relationships in sentences, but ignore the syntactic information interaction between words. This makes the extracted text features unable to fully represent the information contained in the text, resulting in the neglect of key information related to aspect words when modeling the relationship between text and aspect words and images.

[0004] Compared to ABSA, another challenge of the MABSA task lies in how to process image visual features to ensure semantic consistency with aspect words. For the image visual feature representation extracted by the pre-trained model, it is necessary to model its interaction with aspect words and text. Traditionally, attention mechanisms are used to acquire image-text modalities pointing to specific aspect words. For example, attention mechanisms can be used to fuse aspect words and image-text modalities separately, embedding aspect word information into the final feature representation of the image-text modal. However, the above work only roughly combines aspect words and images, without considering noise in regions of the image unrelated to aspect words. Subsequently, a distance-based correlation algorithm is used to obtain the correlation between aspect words and local image regions using the obtained local semantic similarity, and the reliability of local semantic similarity is obtained using global semantic similarity to measure the contribution level of local image regions to the final sentiment judgment, thus acquiring image content related to aspect words. Following this, a new learning architecture is proposed, jointly performing coarse-grained classification and fine-grained alignment of the correlation between local image regions and aspect words, assigning weight coefficients to local image regions based on given aspect words to filter irrelevant image noise. However, the heterogeneity inherent in the image-text modal itself leads to semantic differences between the two. To address this, two approaches were taken: First, non-autoregressive text generation was used to generate image titles, which were then matched with aspect words. Simultaneously, facial features were extracted separately as emotional cues to enhance the emotional relevance of the titles. Second, an external knowledge base was constructed to mine the local regions with the highest matching degree to aspect words and their corresponding emotional information, mitigating the impact of structural differences between the image and text modalities. However, the semantic alignment of images with specific aspect words did not consider the importance of the image to the aspect word vectors, making it difficult to accurately match the local image regions with the highest relevance to the aspect words. Furthermore, the emotional information present in the image was not well integrated into the final feature representation, resulting in insufficiently rich emotional content in the feature vectors used for sentiment prediction. Summary of the Invention

[0005] To overcome the problems of semantic alignment between images and specific aspect words and the lack of syntactic dependencies in text feature representation, this invention provides a feature enhancement method for multimodal aspect-level sentiment analysis.

[0006] This invention provides a feature enhancement method for multimodal aspect-level sentiment analysis, comprising the following steps:

[0007] S1, aspect word encoding;

[0008] For aspect terms, the BERT model is used to initially encode the aspect terms, obtain the feature information of the aspect terms, and obtain the aspect term feature vector H. aThe semantic alignment module converts image data into multiple pairs of adjectives and nouns. The nouns contain information related to the aspect words, while the adjectives point to the emotional information contained in the corresponding nouns. The semantic alignment module determines which noun is the best choice for matching the image with the aspect words, and uses a trilinear similarity algorithm to calculate the semantic similarity between the aspect words and the nouns, identifying the noun with the highest similarity. As supplementary information to aspect term representation, the nouns calculated by the semantic alignment module are... With aspect word feature vector H a The fusion process yields the final aspect term representation H'. a ;

[0009] S2, Text Modality Coding;

[0010] For text S, the BERT model is used to initially encode the text, obtain the text's feature information, and obtain the text feature vector. To obtain long-distance dependencies in text and capture the true textual information, where n is the number of words in text S; H S The syntactic relations H between aspect terms and all words in text S are obtained through the syntactic knowledge enhancement module. r This draws attention to all words related to aspects; secondly, H r With H S The improved text representation H' is obtained by splicing and fusion. S The improved text representation H' S The final aspect term H' a In the common input multi-head cross-attention mechanism, the aspect-word-guided text feature representation H is obtained. S-a ;

[0011] S3, Image Visual Encoding;

[0012] The original image is resized and normalized to standardize its format. Then, a ResNet-152 network is used to perform H-feature analysis on the image. V Extraction; then the final aspect term representation H' a With image features H V By using a multi-head cross-attention mechanism for fusion, we obtain the aspect-word-guided image feature representation H. V-a ;

[0013] S4, Multimodal Fusion;

[0014] Text features guided by aspect terms are represented by H S-a Image feature representation H guided by aspect terms V-a After fusion and concatenation, a graph-text modal feature vector H containing aspect word information is obtained. cSimultaneously, the adjectives in the adjective-noun pairs are added to the H using the sentiment enhancement module. c In this way, the final emotional inclination is more directed towards the aspect words rather than the entire sentence, resulting in H. c Secondly, by fully integrating all the sentiment information about aspect words through a self-attention mechanism, a vector representation H for sentiment discrimination is finally obtained. c ".

[0015] The technical solution provided by this invention has the following technical effects compared with the prior art:

[0016] (1) To address the semantic alignment problem between images and specific aspect words, this invention uses text to represent the local image region information related to aspect words, and then constructs a semantic alignment module to narrow the semantic gap between images and text, enrich the feature representation of aspect words, and more effectively guide the image and text modal feature representation to focus on the vicinity of aspect words. At the same time, the sentiment enhancement module is used to give more sentiment information pointing to aspect words to the final feature representation.

[0017] (2) To address the problem of lack of syntactic dependencies in text feature representation, this invention designs a syntactic knowledge enhancement module. By constructing a syntactic dependency graph with aspect words as the root node, it obtains syntactic knowledge focused on aspect words, solves the problem of lack of syntactic information in the text modal feature extraction process, and enhances the learning of text modal feature information representation. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 Here is a framework diagram of the ABSA model in the background technology;

[0021] Figure 2 Here is a framework diagram of the MABSA model in the background technology;

[0022] Figure 3 This is a flowchart illustrating a feature enhancement method for multimodal aspect-level sentiment analysis according to a certain embodiment of the present invention;

[0023] Figure 4 This is an overall framework diagram of a feature enhancement method for multimodal aspect-level sentiment analysis according to a certain embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the semantic alignment module of a feature enhancement method for multimodal aspect-level sentiment analysis according to a certain embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of the syntactic knowledge enhancement module of a feature enhancement method for multimodal aspect-level sentiment analysis according to a certain embodiment of the present invention. Detailed Implementation

[0026] To better understand the above-mentioned objectives, features, and advantages of the present invention, the solutions of the present invention will be further described below. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0027] Many specific details are set forth in the following description in order to provide a full understanding of the invention, but the invention may also be practiced in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the invention, and not all embodiments.

[0028] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] In one embodiment, such as Figure 1 As shown, a feature enhancement method for multimodal aspect-level sentiment analysis is disclosed, including the following steps:

[0030] S1, aspect word encoding;

[0031] For aspect terms, the BERT model is used to initially encode the aspect terms, obtain the feature information of the aspect terms, and obtain the aspect term feature vector H. a The dataset consists of image-text pairs, so the contribution of image representation to aspect terms needs to be considered. The Semantic Alignment Module (SAM) converts image data into text, extracting highly relevant content from the image and reducing the semantic structural differences between the image visuals and aspect terms. The SAM transforms the image data into five adjective-noun pairs. Nouns contain information related to aspect terms, while adjectives indicate the emotional information contained in the corresponding nouns. The SAM determines which noun is the best choice for matching the image with the aspect terms and uses a trilinear similarity algorithm to calculate the semantic similarity between aspect terms and nouns, identifying the noun with the highest similarity. As supplementary information to aspect term representation, the nouns calculated by the semantic alignment module are... With aspect word feature vector H aThe fusion process yields the final aspect term representation H'. a The BERT model introduces a masked language model to perform unsupervised pre-training on the input data, in order to fully understand the contextual meaning of relevant information on the left and right sides of the text.

[0032] S2, Text Modality Coding;

[0033] For text S, the BERT model is used to initially encode the text, obtain the text's feature information, and obtain the text feature vector. To obtain long-distance dependencies in text and capture the true textual information, where n is the number of words in text S; H S The syntactic knowledge enhancement module (SKEM) is used to obtain the syntactic relations H between aspect terms and all words in the text S. r This draws attention to all words related to aspects; secondly, H r With H S The improved text representation H' is obtained by splicing and fusion. S The improved text representation H' S The final aspect term H' a In the common input multi-head cross-attention mechanism, information focused on the vicinity of aspect words in the text modality is obtained, resulting in a text feature representation H guided by aspect words and possessing deep semantic and syntactic knowledge. S-a ;

[0034] S3, Image Visual Encoding;

[0035] The original image is resized and normalized to standardize its format. Then, a ResNet-152 network is used to perform H-feature analysis on the image. V Extraction; then the final aspect term representation H' a With image features H V By using a multi-head cross-attention mechanism for fusion, we obtain the aspect-word-guided image feature representation H. V-a ;

[0036] S4, Multimodal Fusion;

[0037] The text modality utilizes syntactic knowledge to concentrate the sentiment information of the text modality about a given aspect word, while the sentiment information of the image modality is hidden in the extracted adjectives. In the aspect word encoding stage, nouns are used as references for the final representation of aspect words to enhance the aspect word representation. The corresponding adjectives are then fused into the final feature representation through the sentiment enhancement module in the multimodal fusion stage to enrich the sentiment information in the features.

[0038] Text features guided by aspect terms are represented by HS-a Image feature representation H guided by aspect terms V-a After fusion and concatenation, a graph-text modal feature vector H containing aspect word information is obtained. c Simultaneously, the EmotionalEnhance Module (EEM) adds the adjectives from the noun-adjective pairs to the H... c In this way, the final emotional inclination is more directed towards the aspect words rather than the entire sentence, resulting in H. c Secondly, by fully integrating all the sentiment information about aspect words through a self-attention mechanism, a vector representation H for sentiment discrimination is finally obtained. c ".

[0039] Based on the above embodiments, in a preferred embodiment, the semantic alignment module's processing procedure in step S1 is as follows:

[0040]

[0041] In formulas (1) and (2), The result includes the i-th noun. and b a All are learnable weight parameters, α m It is H a and The highest similarity score obtained using the trilinear similarity algorithm. For α m The corresponding noun, H' N It is a weighted noun representation that represents the matching of image information with aspect terms;

[0042] To enrich the information content of the nouns, the semantic alignment module calculates the nouns using formula (3). With aspect word feature vector H a To perform the fusion, formula (3) is:

[0043] H' a =H a +ω n H' N (3)

[0044] In formula (3), ω n It is an adjustable parameter used to dynamically change the final aspect term representation H' a This allows aspect-specific features to not only include textual features but also better reflect the important information represented by the image.

[0045] Based on the above embodiments, in a preferred embodiment, in step S2, the syntactic knowledge enhancement module discovers the syntactic structure of a sentence through dependency parsing. Dependency parsing is a task of generating a dependency graph to represent the grammatical structure. The relationships between words in a sentence are represented using directed edges and labels. Assume there are n nodes in the dependency graph, each node representing a word in the sentence, and the set of neighboring nodes of node i is represented by N. i It means that r ij The directed edge between node i and node j contains the syntactic relations between words. Traditional graph attention networks construct their root nodes using syntactic knowledge, which may result in the loss of syntactic information related to aspect words. Furthermore, encoding the entire dependency tree is prone to introducing task-irrelevant noise. The syntactic knowledge enhancement module, based on the basic syntactic dependency graph, reconstructs root nodes and constructs virtual syntactic relations to obtain the syntactic relations between all words and aspect words in the text context. This allows it to notice all words related to aspect words, thus enhancing the quality of text feature representation.

[0046] First, the Biaffine parse tree is used to learn the syntactic relationships between words in the sentence, and the root node of the obtained syntactic dependency tree is placed on the aspect word to obtain the improved dependency graph. Then, the dependency relationships between nodes are used to iteratively update the representation of each node.

[0047] Taking a single dependency graph attention hidden layer as an example, the first step is to aggregate the representations of all neighboring nodes using multi-head attention, as shown in formulas (4) and (5):

[0048]

[0049] In formulas (4) and (5), h α Let K be the weight coefficient between node i and node j. Since the node encoder used is BERT, the value of the number of attention heads K is 12. Represents the vector from x1 to x k splicing, W represents the normalized attention coefficient for the k-th attention head. k Given the transformation matrix as input, att(·) is a dot product operation;

[0050] Secondly, because the relational graph attention network incorporates the calculation of the importance of edge pairs between nodes, it integrates syntactic relation information into the feature representation of nodes, thereby better realizing node aggregation and updating, and utilizing r ij We measure the weight of each directed edge to node i in the improved dependency graph, and aggregate the relation feature representations obtained from all neighboring nodes by summing and concatenating them. Taking the attention layer of a single dependency graph as an example:

[0051]

[0052] In formulas (6), (7), and (8), μ represents the number of relation headers. In this implementation, a total of 5 relation headers are used. W is the attention weight assigned to the relationship between nodes, calculated using a linear transformation and the ReLU activation function, and then normalized. μ h is the transformation matrix for vector dimension transformation. β The features with syntactic relations are finally aggregated from the hidden layer; the obtained h α and h β By directly splicing them together, the generated h r The parse data contains syntactic information for both nodes and directed edges:

[0053] h r =h α ||h β (9)

[0054] In formula (8), || represents the concatenation operation; considering the complexity of the syntactic structure, the outputs of two graph attention layers are summed as the final output H of the syntactic knowledge enhancement module. r .

[0055] Based on the above embodiments, in a preferred embodiment, in step S2, to ensure that the final obtained text feature representation contains deep semantic information in both semantic and syntactic aspects, H... r With H S The improved text representation H' is obtained by splicing. S :

[0056] H' S =H r ||H S (10)

[0057] The self-attention mechanism encodes the input vector by using different matrix coefficients to obtain three different representations of the input vector: query vector Q, key vector K, and value vector V, thereby obtaining the degree of influence of each dimension within the input vector.

[0058]

[0059] In formula (9), d k It is the vector dimension of Q, and f(·) calculates the weight result of a single attention layer;

[0060] Multi-head cross-attention mechanism focuses on the importance allocation of relationships between different modalities or sequences. By calculating the similarity between the Q vector of one modality A and the K and V vectors of another modality B, it dynamically adjusts the attention of each modality to other modalities, thereby achieving comprehensive utilization of information.

[0061] The multi-head cross-attention mechanism first calculates the value of a single cross-attention layer to obtain the correlation coefficients between different parts of the input, and then uses multiple attention heads to focus on the influence of different dimensions between the input vectors, enriching the semantic features:

[0062]

[0063] In formula (10), These are the query and key transformation matrices for the m-th head, respectively. The output of C-att(·) is the result of the single-layer cross-attention mechanism.

[0064] Then H' S With H' a In the common input multi-head cross-attention mechanism, the aspect-word-guided text feature representation H S-a The formula is:

[0065] H S-a =Mul(H' a ,H' S (13).

[0066] Based on the above embodiments, in a preferred embodiment, in step S3, ResNet uses residual modules and residual connections to construct a deep residual network. This solves the gradient vanishing problem that occurs during the training of deep networks while preserving the original features, thus improving the robustness of the model in learning deep neural networks. The original image is segmented into multiple regions using a ResNet-152 network, and hidden features are mined through multiple convolutional layers. The output of the last convolutional layer is used as the final image representation vector. Each region uses a 2048-dimensional vector. Let j be the label value of the selected region; a linear function is used to project visual features into the same space as text features:

[0067]

[0068] In formula (12), W V ∈R d×2048 Here, d is a learnable parameter, and d is the dimension of the text feature vector.

[0069] H' a With H V Fusion is performed using a multi-head cross-attention mechanism, H' aAs a query-value input attention mechanism, it captures aspect-word-guided image feature representations H. V-a Reduce the impact of irrelevant noise on image feature representation:

[0070] H V-a =Mul(H' a H V (15).

[0071] 6. The feature enhancement method for multimodal aspect-level sentiment analysis according to claim 1, characterized in that, in step S4, after obtaining semantic vectors for adjectives using BERT, the nouns are identified using the one-to-one correspondence between adjectives and nouns. Corresponding adjectives This information is then used as sentiment information of the target region in the image to assist in the final sentiment prediction.

[0072]

[0073] In formula (14), ANP(·) refers to the operation of finding the adjective using the noun;

[0074] H c H' c and H c The calculation formulas for " are as follows:

[0075] H c =H S-a ||H V-a (17)

[0076]

[0077] H' c '=f (H' c ,H' c ,H' c (19)

[0078] In formulas (17), (18), and (19), ω Adj It is an adjustable parameter.

[0079] 7. A feature enhancement method for multimodal aspect-level sentiment analysis according to claim 5, characterized in that, in step S4, in order to learn the nouns with the highest relevance, a loss function L is constructed using mean squared error. m By minimizing the mean square error, the parameters are adjusted to improve the final prediction performance.

[0080]

[0081] In formula (20), M is the total number of samples;

[0082] Real Labels y i The final sentiment score p is calculated by inputting it into Softmax. i Using the cross-entropy loss function L p Calculate the error value of the predicted label; the final loss function L s Combined with L m With L p :

[0083] L p =-∑y i logp i (twenty one)

[0084] L s =(1-ω)L p +ωL m (twenty two)

[0085] In formulas (21) and (22), ω is an adjustable parameter.

[0086] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the present invention. Although detailed descriptions have been provided with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments, and they should all be covered within the protection scope of the claims.

Claims

1. A feature enhancement method for multimodal aspect-level sentiment analysis, characterized in that, Includes the following steps: S1, aspect word encoding; For aspect terms, the BERT model is used to initially encode the aspect terms, obtain the feature information of the aspect terms, and obtain the aspect term feature vector H. a The semantic alignment module converts image data into multiple pairs of adjectives and nouns. The nouns contain information related to the aspect words, while the adjectives point to the emotional information contained in the corresponding nouns. The semantic alignment module determines which noun is the best choice for matching the image with the aspect words, and uses a trilinear similarity algorithm to calculate the semantic similarity between the aspect words and the nouns, identifying the noun with the highest similarity. As supplementary information to aspect term representation, the nouns calculated by the semantic alignment module are... With aspect word feature vector H a The fusion process yields the final aspect term representation H'. a ; S2, Text Modality Coding; For text S, the BERT model is used to initially encode the text, obtain the text's feature information, and obtain the text feature vector. To obtain long-distance dependencies in text and capture the true textual information, where n is the number of words in text S; H S The syntactic relations H between aspect terms and all words in text S are obtained through the syntactic knowledge enhancement module. r This draws attention to all words related to aspects; secondly, H r With H S The improved text representation H' is obtained by splicing and fusion. S The improved text representation H' S The final aspect term H' a In the common input multi-head cross-attention mechanism, the aspect-word-guided text feature representation H is obtained. S-a ; S3, Image Visual Encoding; The original image is resized and normalized to standardize its format. Then, a ResNet-152 network is used to perform H-feature analysis on the image. V Extraction; then the final aspect term representation H' a With image features H V By using a multi-head cross-attention mechanism for fusion, we obtain the aspect-word-guided image feature representation H. V-a ; S4, Multimodal fusion; Text features guided by aspect terms are represented by H S-a Image feature representation H guided by aspect terms V-a After fusion and concatenation, a graph-text modal feature vector H containing aspect word information is obtained. c Simultaneously, the adjectives in the adjective-noun pairs are added to the H using the sentiment enhancement module. c In this way, the final emotional inclination is more directed towards the aspect words rather than the entire sentence, resulting in H. c Secondly, by fully integrating all the sentiment information about aspect words through a self-attention mechanism, a vector representation H for sentiment discrimination is finally obtained. c ".

2. The feature enhancement method for multimodal aspect-level sentiment analysis according to claim 1, characterized in that, In step S1, the semantic alignment module processes the following steps: In formulas (1) and (2), The result includes the i-th noun. and b a All are learnable weight parameters, α m It is H a and The highest similarity score obtained using the trilinear similarity algorithm. For α m The corresponding noun, H' N It is a weighted noun representation that represents the matching of image information with aspect terms; The semantic alignment module calculates the nouns using formula (3). With aspect word feature vector H a To perform the fusion, formula (3) is: H' a =H a +ω n H' N (3) In formula (3), ω n It is an adjustable parameter used to dynamically change the final aspect term representation H' a .

3. The feature enhancement method for multimodal aspect-level sentiment analysis according to claim 2, characterized in that, In step S2, the syntactic knowledge enhancement module discovers the syntactic structure of a sentence through dependency parsing. Dependency parsing is a task that generates a dependency graph to represent the grammatical structure. The relationships between words in a sentence are represented using directed edges and labels. Suppose there are n nodes in the dependency graph, each node represents a word in a sentence, and the set of neighboring nodes of node i is denoted by N. i It means that r ij This represents the directed edge between node i and node j, which contains the syntactic relations between words; First, the Biaffine parse tree is used to learn the syntactic relationships between words in the sentence, and the root node of the obtained syntactic dependency tree is placed on the aspect word to obtain the improved dependency graph. Then, the dependency relationships between nodes are used to iteratively update the representation of each node. Taking a single dependency graph attention hidden layer as an example, the first step is to aggregate the representations of all neighboring nodes using multi-head attention, as shown in formulas (4) and (5): In formulas (4) and (5), h α Let K be the weight coefficient between node i and node j. Since the node encoder used is BERT, the value of the number of attention heads K is 12. Represents the vector from x1 to x k splicing, W represents the normalized attention coefficient for the k-th attention head. k Given the transformation matrix as input, att(·) is a dot product operation; Secondly, syntactic relation information is integrated into the feature representation of nodes, utilizing r ij We measure the weight of each directed edge to node i in the improved dependency graph, and aggregate the relation feature representations obtained from all neighboring nodes by summing and concatenating them. Taking the attention layer of a single dependency graph as an example: In formulas (6), (7), and (8), μ is the number of relation headers. W is the attention weight assigned to the relationship between nodes, calculated using a linear transformation and the ReLU activation function, and then normalized. μ h is the transformation matrix for vector dimension transformation. β The features with syntactic relations are finally aggregated from the hidden layer; the obtained h α and h β By directly splicing them together, the generated h r The parse data contains syntactic information for both nodes and directed edges: h r =h α ||h β (9) In formula (8), || represents the concatenation operation; considering the complexity of the syntactic structure, the outputs of multiple graph attention layers are summed to serve as the final output H of the syntactic knowledge enhancement module. r .

4. The feature enhancement method for multimodal aspect-level sentiment analysis according to claim 3, characterized in that, In step S2, H r With H S The improved text representation H' is obtained by splicing. S : H' S =H r ||H S (10) The self-attention mechanism encodes the input vector by using different matrix coefficients to obtain three different representations of the input vector: query vector Q, key vector K, and value vector V, thereby obtaining the degree of influence of each dimension within the input vector. In formula (9), d k It is the vector dimension of Q, and f(·) calculates the weight result of a single attention layer; Multi-head cross-attention mechanism focuses on the importance allocation of relationships between different modalities or sequences. By calculating the similarity between the Q vector of one modality A and the K and V vectors of another modality B, it dynamically adjusts the attention of each modality to other modalities, thereby achieving comprehensive utilization of information. The multi-head cross-attention mechanism first calculates the value of a single cross-attention layer to obtain the correlation coefficients between different parts of the input, and then uses multiple attention heads to focus on the influence of different dimensions between the input vectors, enriching the semantic features: In formula (10), These are the query and key transformation matrices for the m-th head, respectively. The output of C-att(·) is the result of the single-layer cross-attention mechanism. Then H' S With H' a In the common input multi-head cross-attention mechanism, the aspect-word-guided text feature representation H S-a The formula is: H S-a =Mul(H' a ,H' S ) (13)。 5. A feature enhancement method for multimodal aspect-level sentiment analysis according to claim 1, characterized in that, In step S3, the original image is segmented into multiple regions using a ResNet-152 network, and hidden features are mined through multiple convolutional layers. The output of the last convolutional layer is used as the final image representation vector; each region is represented by a 2048-dimensional vector. Let j be the label value of the selected region; a linear function is used to project visual features into the same space as text features: In formula (12), W V ∈R d×2048 Here, d is a learnable parameter, and d is the dimension of the text feature vector. H' a With H V Fusion is performed using a multi-head cross-attention mechanism, H' a As a query-value input attention mechanism, it captures aspect-word-guided image feature representations H. V-a : H V-a =Mul(H' a ,H V ) (15)。 6. A feature enhancement method for multimodal aspect-level sentiment analysis according to claim 1, characterized in that, In step S4, after obtaining semantic vectors for the adjectives using BERT, the nouns are identified by utilizing the one-to-one correspondence between adjectives and nouns. Corresponding adjectives This information is then used as sentiment information of the target region in the image to assist in the final sentiment prediction. In formula (14), ANP(·) refers to the operation of finding the adjective using the noun; H c H' c and H c The calculation formulas for " are as follows: H c =H S-a ||H V-a (17) H' c '=f(H' c ,H' c ,H' c ) (19) In formulas (17), (18), and (19), ω Adj It is an adjustable parameter.

7. A feature enhancement method for multimodal aspect-level sentiment analysis according to claim 5, characterized in that, In step S4, in order to learn the nouns with the highest relevance, a loss function L is constructed using mean squared error. m By minimizing the mean square error, the parameters are adjusted to improve the final prediction performance. In formula (20), M is the total number of samples; Real Labels y i The final sentiment score p is calculated by inputting it into Softmax. i Using the cross-entropy loss function L p Calculate the error value of the predicted label; the final loss function L s Combined with L m With L p : L p =-∑y i logp i (21) L s =(1-ω)L p +ωL m (22) In formulas (21) and (22), ω is an adjustable parameter.

Images