Aspect-level sentiment analysis method and device based on cross-modal syntax-visual graph convolutional network

Abstract

The application discloses an aspect-level sentiment analysis method and device based on a cross-modal syntax-visual graph convolutional network, containing a cross-modal graph structure construction module and a type-sensitive graph convolutional network for updating features between modes. Feature vector representations of pictures and texts are obtained through a pre-training model, a graph structure representation of the text is obtained through a syntax analysis method, a new dependency relationship is constructed to integrate the picture feature vector into the graph to form a new integrated cross-modal graph structure, a type-sensitive graph convolutional network is used to update the feature vector corresponding to the aspect word, and a full-connection neural network is used for final aspect-level sentiment detection of the group of comments. The pre-training model is used to obtain basic feature representations containing prior knowledge, syntax analysis and construction of the new cross-modal graph structure are used for fine-grained division and combination of the cross-modal comments, a graph network is used for fine-grained fusion, and finally, sentiment classification detection is completed.

Description

Technical Field

[0001] This invention relates to the field of computer perception, specifically to an aspect-level sentiment analysis method and device based on a cross-modal grammar-visual graph convolutional network. Background Technology

[0002] In today's information-saturated world, sentiment analysis has become a powerful tool for understanding and processing user-generated content. With technological advancements, sentiment analysis has evolved from simple text-based methods to more complex aspect-level and multimodal sentiment analysis. Aspect-level sentiment analysis focuses on identifying and analyzing the emotional tendencies of specific aspects or attributes within text, providing a nuanced approach to emotional understanding. Multimodal sentiment analysis, on the other hand, combines multiple data modalities such as text, images, and audio, aiming to comprehensively capture and analyze emotional information. This not only improves the accuracy of sentiment analysis but also broadens the avenues for understanding complex human emotional expressions.

[0003] In practical applications, text and images, as the most common carriers of information, frequently appear together, especially in social media, news reports, and online content. This combination of text and images directly conveys information and emotions through language, while simultaneously using visual elements to enhance or supplement the textual content, providing a more intuitive and emotional expression. For example, social media users often share personal experiences and emotions through a combination of text and images. This cross-modal information presentation makes the joint analysis of text and images particularly important, as it provides a more comprehensive and profound understanding than a single modality.

[0004] While significant progress has been made in unimodal sentiment analysis and visual information analysis, research still faces numerous challenges in multimodal sentiment analysis across different data types and analytical techniques. Current methods largely focus on information fusion at the sentence and image levels, which fails to fully explore finer-grained information such as the syntactic structure of text and local regions in images, thus limiting the in-depth understanding and analysis of complex emotional information. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies in the field of multimodal aspect-level sentiment analysis by proposing a multimodal aspect-level sentiment analysis method and device based on grammar-visual graph networks.

[0006] The aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks designed in this invention includes the following steps:

[0007] Step 1: Construct a comment dataset. Each comment in this dataset includes an image, text, aspect words, and the sentiment category corresponding to those aspect words. Divide the dataset into disjoint training and test sets.

[0008] Step 2: Construct a deep learning model, using the set of images and text in the training set as input data for the deep learning model; the set of sentiment categories corresponding to the aspect words is used to calculate the loss function of the deep learning model. The specific process is as follows:

[0009] The text and images are converted into word vectors containing prior knowledge, and then sequentially concatenated to form a feature matrix to construct the initial features;

[0010] The text is converted into a parsing tree, the feature matrix after re-segmentation is updated, the feature vector of the image is added to the parsing tree and directly connected to the aspect words; the final cross-modal graph structure is obtained.

[0011] By using a multi-layer type-sensitive graph roll machine network to update the cross-modal graph structure, a new feature vector incorporating neighbor node information is obtained; thus, the final feature vector representation of aspect words is obtained.

[0012] A fully connected neural network is used to predict the category probability of the fused and updated aspect word feature vector, and the more likely type is output as the detection result.

[0013] Step 3: Train the deep learning model by inputting the comment images, text, aspect terms, and corresponding sentiment categories from the dataset into the model, calculating the loss function between the predicted results and the true labels, and optimizing the model; then use the trained model for image-text aspect-level sentiment analysis.

[0014] Furthermore, the emotional categories corresponding to the aspect words in step 1 are divided into negative, neutral, and positive, and represented by different numbers.

[0015] Furthermore, the comment images described in step 2 are randomly cropped and resized before being used as input data for the model. This enhances the model's generalization ability and improves its learning performance on images. The resizing is to accommodate the subsequent ResNet input format.

[0016] Furthermore, the text S in the comment described in step 2 n It can be viewed as a sequence of words joined together, represented as:

[0017] Sequence n =[word n,1 ,…,word n,i ,…,word n,s ]

[0018] Among them, Sequence n The word represents the sequence of words formed by concatenating the words in the nth comment text of the dataset. n,i(i∈[1,s]) represents the nth word sequence. n The i-th word;

[0019] The image, converted from a ResNet pre-trained model, contains word vectors with prior knowledge as follows:

[0020] ResNet(Image) = {r j |r j ∈R 2048 ,j=1,2,…,49}

[0021] H image =W v ResNet(Image)

[0022] Where, r j W represents the feature vector representation of the j-th sub-region in the image. v ∈R d×2048 It is used to perform linear transformations of vectors, completing the dimensional conversion from visual space to text space;

[0023] H image =[r1,r2,…,r 49 ],r i ∈R d

[0024] Where, r i This represents the feature vector representation of a sub-region of the image after spatial transformation, which has the same dimension as the text vector, and has a dimension of d.

[0025] The text, converted by the Bert pre-trained model, contains word vectors with prior knowledge as follows:

[0026] Bert(Sequence) = {w j |w j ∈R d}

[0027] H text =[w1,w2,…,w k ],w i ∈R d

[0028] Among them, w i This represents the feature vector of the i-th word after word segmentation, with dimension d;

[0029] The initial feature matrix set of the comments is H, which serves as the subsequent input data for the model and is used for model training.

[0030] H = [H] text H image ]

[0031] Furthermore, step 2 involves converting the text into a parsing tree and using Biaffineparser to obtain the dependency matrix E. R ; where E R It is a two-dimensional matrix, where non-zero elements represent node pairs that have dependencies;

[0032] The set of all possible dependencies of the text is represented by R. t Add image-aspect-word cross-modal relationships as a new set of dependencies, and randomly generate initial feature vectors of dependencies through the embedding layer:

[0033] R emb =Embed(R) t ∪{e i-t})

[0034] The defined cross-modal graph structure is as follows:

[0035] G = (V, E)

[0036] V = [w1, w2, ..., w k ,r1,r2,…,r 49 ]

[0037] The nodes that make up the graph are composed of phrase nodes obtained from grammatical parsing and sub-image nodes after image segmentation.

[0038] The newly generated edge set E is based on the dependency matrix E R The connection relationship between image nodes and corresponding aspect term nodes is updated. The specific update method can be described as follows:

[0039] For each edge connecting an image node and an aspect term node, set a new dependency e. i-t ;

[0040] A is E R The corresponding adjoint matrix has a value of 1 for positions with edges and 0 for all other positions.

[0041] Furthermore, the type-sensitive graph convolutional network described in step 2, specifically, for each connection between two nodes x... i and x j The edges will utilize the feature representations of these two nodes in the previous layer of the neural network to represent h. i and h j And concatenate the dependency relationships between them to calculate the weights:

[0042]

[0043]

[0044]

[0045] The calculation method for updating node representations using type-sensitive weights is as follows:

[0046] For node i, first merge the feature vectors of the neighboring nodes and the feature vectors of the dependency relationship between the two nodes:

[0047]

[0048] Then, the feature vector representation of node i is updated using the neighboring nodes and their corresponding weights:

[0049]

[0050] Where W and b are trainable parameters of the current network layer, and σ corresponds to the ReLU activation function;

[0051] The final feature vector representation is the feature vector h corresponding to the aspect words after being updated by the graph convolutional network. aspect Summing this sum with the global representation feature vector initially generated by the pre-trained model:

[0052] h output =h aspect +h cls .

[0053] Furthermore, in step 2, a fully connected neural network is used to predict the probability of the feature vectors corresponding to the aspect words. The calculation formula is as follows:

[0054] p(y|h output = softmax(W) out ·h output +b out )

[0055] Wherein, p(y|h output W represents the probability of predicting each sentiment category given the aspect term Aspect; out and b out These are the weights and biases of the output layer; h output It is the output feature vector of the last graph neural network;

[0056] The softmax function is defined as follows:

[0057]

[0058] Among them, z i It is the i-th element of the linear transformation of the output layer, and the denominator is the sum of the exponents of this transformation over all possible categories, ensuring that the sum of the probabilities of all output categories is 1;

[0059] In this way, fully connected neural networks can be based on the feature vector h of aspect words. output For each possible sentiment category, a prediction probability is output; finally, the category with the highest prediction probability is selected as the predicted sentiment category.

[0060] Furthermore, the loss function described in step 3 is defined as follows:

[0061]

[0062] Where C is the total number of categories, y c It is the element corresponding to class c in the one-hot encoding of the true label. If the sample belongs to class c, then y c =1; otherwise, y c =0; p c It is the probability that the model predicts that the sample belongs to class c;

[0063] The model optimization uses the AdamW optimizer, which introduces a weight decay mechanism on top of Adam. This mechanism uses regularization to limit the growth of model weights and improves the model's generalization ability when the amount of comment dataset is limited.

[0064] Based on the same inventive concept, this solution also designs an electronic device, including:

[0065] One or more processors;

[0066] Storage device for storing one or more programs;

[0067] When one or more programs are executed by the one or more processors, the one or more processors implement an aspect-level sentiment analysis method based on a cross-modal grammar-visual graph convolutional network.

[0068] Based on the same inventive concept, this solution also designs a computer-readable medium storing a computer program that, when executed by a processor, implements an aspect-level sentiment analysis method based on a cross-modal grammar-visual graph convolutional network.

[0069] The advantages of this invention are:

[0070] The introduction of text syntactic analysis: Existing cross-modal methods, in order to achieve inter-modal alignment and interaction, typically treat text as sequential data, neglecting the internal structure of the text itself. This method introduces text syntactic analysis, transforming the text sequence into a parse tree, explicitly revealing the structured connections and dependencies between words. This enhances the model's understanding of the complex relationships within the text and enables the model to more accurately identify and resolve subtle semantic differences within the text. This deep text understanding provides a more solid foundation for inter-modal alignment and interaction, allowing cross-modal analysis to go beyond superficial sequence matching and delve into the core of the text structure to capture more nuanced and complex semantic information.

[0071] An innovative cross-modal integration strategy is proposed: by combining in-depth grammatical analysis of text with key regions of images, an integrated graph structure called a "grammar-visual graph" is designed. In this structure, key regions of images are no longer isolated elements, but rather serve as new nodes, closely connected to the grammatical analysis tree of the text, forming a unified, dynamically interactive network. This integration approach not only strengthens the dependency between text and images but also endows images with a completely new role—becoming part of interpreting the text structure. Furthermore, the application of the "grammar-visual graph" provides an unprecedented framework for understanding the complex interactions between text and images. It enables cross-modal analysis to transcend the limitations of traditional sequential or single-modal approaches, capturing and understanding the subtle connections and mutual influences between the two modalities more accurately through this deeply integrated structure. This method not only increases the model's ability to process cross-modal content but also provides a powerful tool for further exploring the interaction between text and images.

[0072] Modality-Sensitive Graph Convolutional Networks: An innovative modality-sensitive graph convolutional network was designed and implemented, specifically designed to handle complex graph structures that integrate text and image features. The unique feature of this network is that it considers modality types (text or image) and dependency types within the text as different categories of edges in the graph, cleverly utilizing this information to adjust the weight update strategy during feature updates. This design not only enhances the network's understanding of the intrinsic connections between multimodal data but also dynamically adjusts weights based on the relative importance of different modalities in a specific context, thereby achieving higher accuracy and effectiveness in sentiment analysis. By introducing modality-sensitivity, this graph convolutional network can capture the complex interactions and dependencies between multimodal data more meticulously and accurately. It can identify and strengthen modal features crucial for sentiment analysis while flexibly adjusting the processing of secondary information, ensuring the comprehensiveness and accuracy of the analysis. Attached Figure Description

[0073] Figure 1This is a schematic diagram of the flowchart structure of the aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional network in this invention. Detailed Implementation

[0074] To better understand the above technical solutions, the specific implementation methods and devices for aspect-level sentiment analysis based on cross-modal grammar-visual graph convolutional networks provided by this invention will be described in detail below with reference to the accompanying drawings:

[0075] Example 1

[0076] This invention designs a multimodal aspect sentiment analysis method based on a grammar-visual graph network. The method includes a cross-modal graph structure construction module and a type-sensitive graph convolutional network to update inter-modal features. Feature vector representations of images and text are obtained through a pre-trained model. A graph structure representation of the text is obtained through grammar analysis. New dependencies are constructed to incorporate image feature vectors into the graph, forming a new integrated cross-modal graph structure. The feature vectors corresponding to aspect words are updated via a type-sensitive graph convolutional network, and a fully connected neural network is used to perform final aspect-level sentiment detection on the set of comments.

[0077] To achieve the above objectives, the specific design includes the following steps:

[0078] Step 1: Construct a comment dataset. In this invention, a Twitter comment dataset is used. Each comment contains the comment text, the attached image, the sentiment aspect to be analyzed, and a numerical label. The numerical label for negative sentiment is 0, the numerical label for neutral sentiment is 1, and the numerical label for positive sentiment is 2. The Twitter comment data is divided into disjoint training and test sets.

[0079] The text statement corresponding to a Twitter comment is a Sequence. n The corresponding image is Image. n The corresponding aspect term for sentiment analysis is Aspect. n The sentiment numerical tag corresponding to the aspect term is L. n The Twitter comment dataset is set as D, specifically defined as:

[0080] D = {(Sequence n Image n Aspect n ,L n )}, n∈[1,N]

[0081] Sequence n ={word1,…,word i ,…,word s}

[0082] Aspect n ={word p ,word p+1 ,…,word q}

[0083] Where D represents the Twitter comment dataset, N represents the number of samples in the Twitter comment dataset, and Sequence n Image represents the text portion of the nth comment sample. n Aspect represents the image portion of the nth comment sample. n L represents the aspect term portion in the nth comment sample. n This represents the sentiment label corresponding to the aspect term in the nth comment sample; and Aspect n It is a Sequence n It contains a continuous subset of text;

[0084] Step 1 involves constructing the test set and training set from the Twitter comment dataset as follows:

[0085] From the N smart contract file entities in the Twitter comment dataset, P are randomly selected as the training set, using D. train This means that the remaining DP comment data samples in D are used as the test set, and D is used as the test set. test express.

[0086] Step 2: Construct a deep learning model. The set of comment images and texts in the training set, after random cropping and resizing, serves as the input data for the deep learning model. The set of sentiment digit tags for the aspect terms contained in the comment texts in the training set is used to calculate the loss function of the deep learning model. The sentiment digit tag set is L. lab The deep learning model processing procedure is as follows:

[0087] The text and images in the comments are converted into word vectors containing prior knowledge using a BERT pre-trained model and its dictionary, and a ResNet pre-trained model, respectively. These vectors are then concatenated sequentially to form a feature matrix; initial features are then constructed. The specific process is as follows:

[0088] Text S in the comment n It can be viewed as a sequence of words joined together, represented as:

[0089] Sequence n =[word n,1 ,…,word n,i ,…,word n,s ]

[0090] Among them, Sequence nThis represents the word sequence formed by concatenating the words in the nth comment text within a Twitter comment dataset. n,i (i∈[1,s]) represents the nth word sequence. n The i-th word;

[0091] The image, converted from a ResNet pre-trained model, contains word vectors with prior knowledge as follows:

[0092] ResNet(Image) = {r j |r j ∈R 2048 ,j=1,2,…,49}

[0093] H image =W v ResNet(Image)

[0094] Where, r j W represents the feature vector representation of the j-th sub-region in the image. v ∈R d×2048 It is used to perform linear transformations of vectors, completing the dimensional conversion from visual space to text space;

[0095] H image =[r1,r2,…,r 49 ],r i ∈R d

[0096] Where, r i This represents the feature vector representation of a sub-region of the image after spatial transformation, which has the same dimension as the text vector, and has a dimension of d.

[0097] The text, converted by the Bert pre-trained model, contains word vectors with prior knowledge as follows:

[0098] Bert(Sequence) = {w j |w j ∈R d}

[0099] H text =[w1,w2,…,w k ],w i ∈R d

[0100] Among them, w i This represents the feature vector of the i-th word after word segmentation, with dimension d;

[0101] The initial feature matrix set H of the Twitter comments is used as the subsequent input data for model training.

[0102] H = [H] text H image ].

[0103] The text is converted into a parsing tree, the feature matrix after re-segmentation is updated, and the feature vectors of multiple regions of the image are added to the parsing tree with new dependencies, directly connected to aspect words; the final cross-modal graph structure is obtained, as detailed below:

[0104] Convert the text into a parsing tree and use Biaffineparser to obtain the dependency matrix E. R ; where E R It is a two-dimensional matrix, where non-zero elements represent node pairs that have dependencies;

[0105] The set of all possible dependencies of a text is represented by R. t Add image-aspect-word cross-modal relationships as a new set of dependencies, and randomly generate initial feature vectors of dependencies through the embedding layer:

[0106] R emb =Embed(R) t ∪{e i-t})

[0107] The defined cross-modal graph structure is as follows:

[0108] G = (V, E)

[0109] V = [w1, w2, ..., w k ,r1,r2,…,r 49 ]

[0110] The nodes that make up the graph are composed of phrase nodes obtained from grammatical parsing and sub-image nodes after image segmentation.

[0111] The newly generated edge set E is based on the dependency matrix E R The connection relationship between image nodes and corresponding aspect term nodes is updated. The specific update method can be described as follows:

[0112] For each edge connecting an image node and an aspect term node, set a new dependency e. i-t ;

[0113] A is E R The corresponding adjoint matrix has a value of 1 for each edge and 0 for all other positions.

[0114] Type-sensitive graph convolutional networks, specifically, for each connection between two nodes x i and x j The edges will utilize the feature representations of these two nodes in the previous layer of the neural network to represent h.i and h j And concatenate the dependency relationships between them to calculate the weights:

[0115]

[0116]

[0117]

[0118] A multi-layer type-sensitive scroll machine network is used to update the cross-modal graph structure, obtaining a new feature vector that incorporates neighbor node information; the final feature vector representation of aspect words is obtained; the calculation method for updating node representations using type-sensitive weights is as follows:

[0119] For node i, first merge the feature vectors of the neighboring nodes and the feature vectors of the dependency relationship between the two nodes:

[0120]

[0121] Then, the feature vector representation of node i is updated using the neighboring nodes and their corresponding weights:

[0122]

[0123] Where W and b are trainable parameters of the current network layer, and σ corresponds to the ReLU activation function;

[0124] The final feature vector representation is the feature vector h corresponding to the aspect words after being updated by the graph convolutional network. aspect Summing this sum with the global representation feature vector initially generated by the pre-trained model:

[0125] h output =h aspect +h cls .

[0126] A fully connected neural network is used to predict the probability (negative / neutral / positive) of the fused and updated aspect word feature vector. The probability with the higher probability is output as the detection result. The calculation formula is as follows:

[0127] p(y|h output = softmax(W) out ·h output +b out )

[0128] Wherein, p(y|h output W represents the probability of predicting each sentiment category given the aspect term Aspect; out and b out These are the weights and biases of the output layer; h outputIt is the output feature vector of the last graph neural network;

[0129] The softmax function is defined as follows:

[0130]

[0131] Among them, z i It is the i-th element of the linear transformation of the output layer, and the denominator is the sum of the exponents of this transformation over all possible categories, ensuring that the sum of the probabilities of all output categories is 1;

[0132] In this way, fully connected neural networks can be based on the feature vector h of aspect words. output For each possible sentiment category, a prediction probability is output; finally, the category with the highest prediction probability is selected as the predicted sentiment category.

[0133] Step 3: Train the deep learning model. Input the comment image, text, aspect terms, and corresponding numerical labels into the model, calculate the loss function between the predicted results and the true labels, and optimize the model. The process in Step 2 automatically yields the model's detection results for aspect terms in the current image and text. The loss function is defined as follows:

[0134]

[0135] Where C is the total number of categories, y c It is the element corresponding to class c in the one-hot encoding of the true label. If the sample belongs to class c, then y c =1; otherwise, y c =0; p c It is the probability that the model predicts that the sample belongs to class c;

[0136] The optimizer used for model optimization is AdamW, which introduces a weight decay mechanism on top of Adam. This mechanism can limit the growth of model weights through regularization, thereby improving the model's generalization ability when the amount of comment dataset is limited.

[0137] Example 2

[0138] Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in Embodiment 1.

[0139] Since the device described in Embodiment 3 of this invention is the same electronic device used in implementing the aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.

[0140] Example 3

[0141] Based on the same inventive concept, the present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Embodiment 1.

[0142] Since the device described in Embodiment 4 of this invention is a computer-readable medium used to implement the aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.

[0143] It should be understood that the above description of the preferred embodiments is quite detailed and should not be construed as a limitation on the scope of protection of this application. Those skilled in the art, under the guidance of this invention, may make substitutions or modifications without departing from the scope of protection of the claims of this application, all of which fall within the scope of protection of this application. The scope of protection claimed in this application shall be determined by the appended claims.

Claims

1. An aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks, characterized in that, Includes the following steps: Step 1: Construct a comment dataset. Each comment in this dataset includes an image, text, aspect words, and the sentiment category corresponding to those aspect words. Divide the dataset into disjoint training and test sets. Step 2: Construct a deep learning model, using the set of images and text in the training set as input data for the deep learning model; the set of sentiment categories corresponding to the aspect words is used to calculate the loss function of the deep learning model. The specific process is as follows: The text and images are converted into word vectors containing prior knowledge, and then sequentially concatenated to form a feature matrix to construct the initial features; The text is converted into a parsing tree, the feature matrix after re-segmentation is updated, the feature vector of the image is added to the parsing tree and directly connected to the aspect words; the final cross-modal graph structure is obtained. By using a multi-layer type-sensitive graph convolutional network to update the cross-modal graph structure, a new feature vector incorporating neighbor node information is obtained; thus, the final feature vector representation of aspect words is obtained. A fully connected neural network is used to predict the category probability of the fused and updated aspect word feature vector, and the more likely type is output as the detection result. Step 3: Train the deep learning model by inputting the comment images, texts, aspect words, and corresponding sentiment categories from the dataset into the model, calculating the loss function between the predicted results and the true labels, and optimizing the model.

2. The aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks according to claim 1, characterized in that: In step 1, the emotional categories corresponding to the aspect words are divided into negative, neutral, and positive, and are represented by different numbers.

3. The aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks according to claim 1, characterized in that: The comment images mentioned in step 2 are randomly cropped and resized, and used as input data for the model.

4. The aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks according to claim 1, characterized in that: The text S in the comment described in step 2 n It can be viewed as a sequence of words joined together, represented as: Sequence n =[word n,1 ,…,word n,i ,…,word n,s ] Among them, Sequence n The word represents the sequence of words formed by concatenating the words in the nth comment text of the dataset. n,i (i∈[1,s]) represents the nth word sequence. n The i-th word; The image, converted from a ResNet pre-trained model, contains word vectors with prior knowledge as follows: ResNet(Image)={r j |r j ∈R 2048 ,j=1,2,…,49} H image =W v ResNet(Image) Where, r j W represents the feature vector representation of the j-th sub-region in the image. v ∈R d×2048 It is used to perform linear transformations of vectors, completing the dimensional conversion from visual space to text space; H image =[r1,r2,…,r 49 ],r i ∈R d Where, r i This represents the feature vector representation of a sub-region of the image after spatial transformation, which has the same dimension as the text vector, and has a dimension of d. The text, converted by the Bert pre-trained model, contains word vectors with prior knowledge as follows: Bert(Sequence)={w j |w j ∈R d } H text =[w1,w2,…,w k ],w i ∈R d Among them, w i This represents the feature vector of the i-th word after word segmentation, with dimension d; The initial feature matrix set of the comments is H, which serves as the subsequent input data for the model and is used for model training. H=[H text ,H image ]。 5. The aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks according to claim 4, characterized in that: Step 2 involves converting the text into a parsing tree and using the Biaffine parser to obtain the dependency matrix E. R ; where E R It is a two-dimensional matrix, where non-zero elements represent node pairs that have dependencies; The set of all possible dependencies of the text is represented by R. t Add image-aspect-word cross-modal relationships as a new set of dependencies, and randomly generate initial feature vectors of dependencies through the embedding layer: R emb =Embed(R t ∪{e i-t }) The defined cross-modal graph structure is as follows: G = (V, E) V=[w1,w2,…,w k ,r1,r2,…,r 49 ] The nodes that make up the graph are composed of phrase nodes obtained from grammatical parsing and sub-image nodes after image segmentation. The newly generated edge set E is based on the dependency matrix E R The connection relationship between image nodes and corresponding aspect term nodes is updated. The specific update method can be described as follows: For each edge connecting an image node and an aspect term node, set a new dependency e. i-t ; A is E R The corresponding adjoint matrix has a value of 1 for positions with edges and 0 for all other positions.

6. The aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks according to claim 5, characterized in that: The type-sensitive graph convolutional network described in step 2, specifically, for each connection between two nodes x i and x j The edges will utilize the feature representations of these two nodes in the previous layer of the neural network to represent h. i and h j And concatenate the dependency relationships between them to calculate the weights: The calculation method for updating node representations using type-sensitive weights is as follows: For node i, first merge the feature vectors of the neighboring nodes and the feature vectors of the dependency relationship between the two nodes: Then, the feature vector representation of node i is updated using the neighboring nodes and their corresponding weights: Where W and b are trainable parameters of the current network layer, and σ corresponds to the ReLU activation function; The final feature vector representation is the feature vector h corresponding to the aspect words after being updated by the graph convolutional network. aspect Summing this sum with the global representation feature vector initially generated by the pre-trained model: h output =h aspect +h cls 。 7. The aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks according to claim 1, characterized in that: Step 2 involves using a fully connected neural network to predict the probability of the feature vectors corresponding to the aspect words. The calculation formula is as follows: p(y|h output )=softmax(W out ·h output +b out ) Wherein, p(y|h output W represents the probability of predicting each sentiment category given the aspect term Aspect; out and b out These are the weights and biases of the output layer; h output It is the output feature vector of the last graph neural network; The softmax function is defined as follows: Among them, z i It is the i-th element of the linear transformation of the output layer, and the denominator is the sum of the exponents of this transformation over all possible categories, ensuring that the sum of the probabilities of all output categories is 1; In this way, fully connected neural networks can be based on the feature vector h of aspect words. output For each possible sentiment category, a prediction probability is output; finally, the category with the highest prediction probability is selected as the predicted sentiment category.

8. The aspect-level sentiment analysis method based on cross-modal grammar-visual graph convolutional networks according to claim 1, characterized in that: The loss function described in step 3 is defined as follows: Where C is the total number of categories, y c It is the element corresponding to class c in the one-hot encoding of the true label. If the sample belongs to class c, then y c =1; otherwise, y c =0; p c It is the probability that the model predicts that the sample belongs to class c; The model optimization uses the AdamW optimizer, which introduces a weight decay mechanism on top of Adam. This mechanism uses regularization to limit the growth of model weights and improves the model's generalization ability when the amount of comment dataset is limited.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images