Multi-modal sentiment analysis method and system based on attention and graph enhanced text

Abstract

The invention provides a multi-modal sentiment analysis method and system based on attention and graph enhanced text, and belongs to the technical field of multi-modal sentiment analysis. The method comprises the steps of obtaining text features, image features and voice features in video data and performing preprocessing; an adaptive cross-modal interaction module is adopted to calculate attention weights between the text features and the image features and between the text features and the voice features, and enhanced text features are obtained; inputting the enhanced text features into a hierarchical multi-modal graph fusion network, and constructing an adjacent matrix by using a self-attention mechanism; inputting the adjacent matrix and the enhanced text features into a jump connection graph convolutional network to obtain a final feature matrix; and combining the feature matrix with the adjacent matrix, and obtaining a prediction result of sentiment analysis by using an encoder and a classifier. The problem of deficiency in context alignment among different modes is solved, and emotion polarity prediction is performed more comprehensively and accurately.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal sentiment analysis, and in particular relates to a multimodal sentiment analysis method and system based on attention and graph-enhanced text. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Humans understand and experience the world through the perception of multimodal signals, and emotional expression also presents multi-level and three-dimensional characteristics. In addition to language content, humans also convey emotions and attitudes through intonation, volume, facial expressions, gestures, etc. In recent years, the rapid rise of short video platforms has led to an explosive growth in the number of short videos based on human activities, accompanied by the generation of a large amount of multimodal data. This phenomenon highlights the importance of research in the field of multimodal sentiment analysis and has aroused widespread attention to the use of information from multiple modalities for emotional state recognition.

[0004] Most of the existing multimodal sentiment analysis methods are dominated by text information, and supplement the deficiencies of text information with the assistance of visual and auditory information, thereby improving the comprehensiveness and accuracy of sentiment prediction. However, the existing methods still face the following problems: First, there are obvious deficiencies in the context alignment between different modalities. Due to the locality and inconsistency of semantic expression in modalities such as text, audio and vision, for example, intonation changes in audio or facial expressions in vision may only be related to part of the semantics in the text. Existing methods mostly use simple fusion strategies such as weighting or splicing, such as LMF, TFN, etc., but fail to effectively capture the fine-grained associations between modalities, resulting in the problem of context misalignment. Secondly, the existing methods are insufficient in capturing the complex dependencies between different modalities. Many methods fail to explicitly model the global and local dependencies between modalities, resulting in the neglect of potential semantic interactions between modalities in the fusion process of modal information, which in turn affects the accuracy of sentiment prediction. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a multimodal sentiment analysis model method and system based on attention and graph-enhanced text, adopts a multimodal sentiment analysis model based on attention and graph-enhanced text (MAG-TE), introduces discrete sequence and cross-modal attention mechanism to dynamically align and integrate image and speech features through an adaptive cross-modal interaction module, adopts a hierarchical multimodal graph fusion network, explicitly models the local and global dependencies between modalities through a graph structure, and uses a skip-connected graph convolutional network to capture the multi-level interaction of multimodal sentiment features, thereby significantly improving the performance of multimodal sentiment prediction.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of the present invention provides a multimodal sentiment analysis method based on attention and graph-enhanced text; Multimodal sentiment analysis methods based on attention and graph-enhanced text, including: Obtain text features, image features and speech features in video data and perform preprocessing; Input the preprocessed text features, image features, and speech features into the trained multimodal sentiment analysis model to obtain the prediction results of sentiment analysis; The trained multimodal sentiment analysis model uses an adaptive cross-modal interaction module to calculate the attention weights between text features, image features, and speech features to obtain enhanced text features. The enhanced text features are input into the hierarchical multimodal graph fusion network, and the adjacency matrix is ​​constructed using the self-attention mechanism; the adjacency matrix and the enhanced text features are input into the skip connection graph convolutional network to obtain the final feature matrix; The feature matrix and the adjacency matrix are combined, and the prediction results of the sentiment analysis are obtained using an encoder and a classifier.

[0007] As a further technical solution, the pretreatment process includes: Process text features, use a pre-trained language model to segment and embed text, and generate word embedding and position embedding; Extracting image information of facial expressions, action units and postures from video frames; performing dimensionality reduction operations on the image information to obtain image features; Speech emotion information is extracted from the speech frame, and the speech emotion information is subjected to dimensionality reduction operation to obtain speech features.

[0008] As a further technical solution, an adaptive cross-modal interaction module is used to calculate the attention weights between text features, image features, and speech features. The process of obtaining enhanced text features is as follows: An adaptive cross-modal interaction module is used to map the image features and speech features into a unified index sequence; The index vector is processed using the cross-modal attention mechanism, and the attention weights between text features, image features, and speech features are calculated to obtain enhanced text features.

[0009] As a further technical solution, the process of processing the index sequence using the cross-modal attention mechanism, calculating the attention weights between the text features, the image features, and the speech features, and obtaining the enhanced text features is as follows: The discrete index sequence is mapped into a continuous high-dimensional vector representation through the embedding layer; Take the text feature as the query vector, the image feature and the speech feature as the key vector and the value vector, and obtain the attention weight matrix between the text feature, the image feature and the speech feature; Based on the attention weight matrix, the enhanced non-language information is extracted from the image features and speech features to generate non-language embedding information corresponding to the text features; The non-language embedding information and the text features are fused to obtain enhanced text features.

[0010] As a further technical solution, the adjacency matrix constructed using the self-attention mechanism is:

[0011] In the formula, is the adjacency matrix representing the association between nodes; yes The matrix transpose of represents the similarity between word pairs in text features; is the scaling factor, used to stabilize the calculation and prevent the dot product from being too large; is applied to each row of the matrix operation, indicating the strength of the relationship between each pair of words.

[0012] As a further technical solution, the process of inputting the adjacency matrix and the enhanced text features into the skip connection graph convolutional network to obtain the final feature matrix is: The adjacency matrix and the enhanced text features are used as the initial input of the skip connection graph convolutional network, where the enhanced text features are used as the initial feature matrix; The initial feature matrix is ​​weighted and aggregated using the adjacency matrix. The aggregated feature matrix represents the weighted average features of each node and its neighbors. A skip connection is used to add the input feature matrix to the transformed feature matrix, and the ReLU activation function is used to obtain the final feature matrix.

[0013] As a further technical solution, the feature matrix and the adjacency matrix are combined, and the process of obtaining the prediction result of sentiment analysis using the encoder and the classifier is as follows: use The function concatenates the final feature matrix and the adjacency matrix to obtain the fused features; Adding the fused features to the original text embedding to obtain text enhanced feature representation; The text enhanced feature representation is input into the trained Transformer encoder and classifier to obtain the prediction result of sentiment analysis.

[0014] A second aspect of the present invention provides a multimodal sentiment analysis system based on attention and graph-enhanced text.

[0015] A multimodal sentiment analysis system based on attention and graph-augmented text, including: The multimodal feature acquisition module is configured to: acquire text features, image features and speech features in the video data and perform preprocessing; The text feature enhancement module is configured to: use an adaptive cross-modal interaction module to calculate the attention weights between text features, image features, and speech features to obtain enhanced text features; The final feature matrix acquisition module is configured to: input the enhanced text features into the hierarchical multimodal graph fusion network, and construct the adjacency matrix using the self-attention mechanism; input the adjacency matrix and the enhanced text features into the skip connection graph convolutional network to obtain the final feature matrix; The sentiment analysis prediction module is configured to: combine the final feature matrix and the adjacency matrix, and use an encoder and a classifier to obtain a prediction result of the sentiment analysis.

[0016] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the multimodal sentiment analysis method based on attention and graph-enhanced text as described in the first aspect of the present invention.

[0017] The fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the multimodal sentiment analysis method based on attention and graph-enhanced text as described in the first aspect of the present invention are implemented.

[0018] One or more of the above technical solutions have the following beneficial effects: The present invention uses an adaptive cross-modal interaction module to transform the features of visual and audio modalities into discrete index sequence representations, alleviating the formal differences between different modalities. Subsequently, based on the cross-modal attention mechanism, the emotional signals related to the text semantics can be dynamically extracted, thereby enhancing the understanding of the emotional expression of the text. It can effectively ignore irrelevant interference information, such as meaningless smiles in vision or background noise in audio.

[0019] The hierarchical multimodal graph fusion network explicitly models the dependencies between modalities by constructing a multimodal adjacency matrix. The fine-grained dependencies between modalities are captured through local convolution operations, and the efficient propagation of cross-layer information is ensured through the jump connection mechanism, avoiding the information loss in traditional deep graph convolutional networks. Finally, the hierarchical multimodal graph fusion network generates a unified multimodal sentiment representation by hierarchically fusing local and global features, thereby achieving more comprehensive and accurate sentiment polarity prediction.

[0020] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0022] Figure 1 This is a flow chart of the method of the first embodiment.

[0023] Figure 2 It is a system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0025] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0026] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0027] The present invention maps the original visual and auditory features into a unified index sequence through an adaptive cross-modal interaction module to reduce the heterogeneity between modalities, and then uses a cross-modal attention mechanism to calculate the attention weights between visual and auditory embeddings and text embeddings; uses a hierarchical multimodal graph fusion network to construct an adjacency matrix between multimodal features, and uses a skip connection graph convolutional neural network to capture the complex dependencies between different modalities, thereby achieving deep fusion of multimodal features. Finally, the fused multimodal features are combined with the enhanced text features and input into the pre-trained Transformer encoder and classifier to predict sentiment polarity.

[0028] Embodiment 1 This embodiment discloses a multimodal sentiment analysis method based on attention and graph-enhanced text; like Figure 1 As shown in the figure, the multimodal sentiment analysis method based on attention and graph-enhanced text includes: Step S1, obtaining text features, image features and speech features in video data and performing preprocessing; Multimodal sentiment analysis is to detect the emotion of video clips using multimodal signals. For a video clip X, it consists of three parts: text sequence (t), acoustic sequence (a) and visual sequence (v), which can be expressed as , n∈{t, a, v}. Among them, represents the sequence length of mode n, denote the i-th word, audio frame, and visual frame, respectively.

[0029] For text sequences We use the tokenizer of a pre-trained language model (such as SentiLARE) to extract sentiment-related features such as sentiment polarity and part of speech to enhance the sentiment expression ability of text representation. By segmenting the text into corresponding word index sequences .in, is the length of the text sequence.

[0030] Next, the pre-trained language model is used to generate word embeddings, position embeddings, and sentiment-related feature embeddings (such as sentiment polarity and part of speech):

[0031] In the formula, represents word embedding, represents position embedding, Represents sentiment polarity embedding.

[0032] For acoustic sequences and visual sequences, Facet and COVAREP are used to extract original features respectively. The original acoustic and visual feature sequences are represented as and ,in are the number of frames of acoustic and visual sequences, respectively. and They are acoustic and visual feature dimensions, respectively. Specifically, for visual sequences, Facet is used to extract image information such as facial expressions, action units, and postures from video frames. The image information is subjected to dimensionality reduction operations to obtain image features, which facilitates subsequent modeling and fusion.

[0033] For speech features, the COVAREP tool is used to extract relevant emotional information from the audio signal, such as Mel Frequency Cepstral Coefficient (MFCC), pitch, speech segmentation, and other speech emotional information. The speech emotional information is subjected to dimensionality reduction operations to obtain speech features.

[0034] By extracting and reducing the features of text, image and speech, the generated multimodal features lay the foundation for subsequent modal alignment and fusion.

[0035] Step S2, the multimodal sentiment analysis model based on attention and graph-enhanced text uses an adaptive cross-modal interaction module to calculate the attention weights between text features, image features, and speech features to obtain enhanced text features; In multimodal sentiment analysis tasks, information from different modalities always has time and content misalignment problems, which leads to deviations in the model's understanding of the overall sentiment. Therefore, in order to solve the misalignment problem between asynchronous modalities, the adaptive cross-modal interaction module (ACIM) is used to unify the representation of visual and auditory features into an index sequence, and the cross-modal attention mechanism is used to explicitly model the information interaction between text features, image features, and speech features, effectively capturing the complementary information between text and non-text modalities and enhancing text representation.

[0036] Specifically, text features are usually represented by an index sequence obtained from a vocabulary, while image features and speech features are represented by a real vector sequence.

[0037] Step S21, first obtain the image frames and speech frames in the video data, and construct a feature set in Represents image features or auditory features, Represents the number of frames, Represents the feature dimension of each frame, Represents image features; Represents speech features; k-means clustering is used to divide image frames and speech frames into The clusters are as follows:

[0038] in The cluster centers representing modality n. Through these cluster centers, visual and auditory vocabularies are constructed, and image and speech features are lexicalized through cluster centers, so that the index sequence matches the discrete vocabulary form of the text sequence, providing a consistent representation for the subsequent modality fusion and attention mechanism calculation.

[0039] Step S22, converting the feature sequence into an index. Specifically, given a feature sequence , which needs to be converted into the corresponding index sequence For the i-th frame , whose index is calculated as follows:

[0040] In the formula, is the i-th frame of mode n The corresponding cluster center index; To find the distance The nearest cluster center The index j of To calculate the Euclidean distance between the feature and the cluster center; finally get the index sequence as a representation of the modality n.

[0041] The above methods can better reduce the difference between modalities, and mapping video and voice features into discrete indexes can effectively reduce the difference between modalities and improve fusion efficiency. This mapping enables non-language features to be better integrated with text features, improving the effect of multimodal sentiment analysis.

[0042] Step S23: The index vector output in the above step is Input to the model's Embedding layer:

[0043] in is the output of the embedding layer, is the embedding dimension, is the embedding layer function. The main function of the embedding layer is to map discrete indexes into continuous high-dimensional vector representations. Using high-dimensional vector representations as input to the model can provide richer and more learnable feature representations.

[0044] Step S24, taking the text feature as the query vector (Q), the image feature and the speech feature as the key vector (K) and the value vector (V), obtains the attention weight matrix between the text feature and the image feature and the speech feature, as shown in the following formula:

[0045]

[0046]

[0047] in is the learnable parameter matrix, are the dimensions of the query, key, and value vectors respectively.

[0048] Step S25, by acquiring the attention weights and correlations between the text features, image features, and speech features, an attention weight matrix of each word in the text features for each video frame or speech frame is generated:

[0049] in is the attention weight matrix, which represents the attention distribution between text features, image features and speech features.

[0050] Step S26, based on the acquired attention weight matrix, extract enhanced non-language information from the image features and speech features to generate non-language embedding information corresponding to the text:

[0051] These enhanced non-verbal embeddings It can be regarded as the emotional information of the image and speech selected by the text features. After obtaining the enhanced embedding from the image features and speech features, it is fused through a gating mechanism (Gate) to combine the text, image and speech information:

[0052] Where ';' represents a series operation, Gate(;) is a gating mechanism consisting of a fully connected layer. It is the enhanced information extracted from image features that is consistent with text features; Enhanced information extracted from speech features and aligned with text features; used to fuse information from two modalities.

[0053] Step S3, inputting the enhanced text features into the hierarchical multimodal graph fusion network, and constructing an adjacency matrix using the self-attention mechanism; inputting the adjacency matrix and the enhanced text features into the skip connection graph convolutional network to obtain the final feature matrix; Step S31: The text features enhanced by ACIM are As the input of the Hierarchical Multimodal Graph Fusion Network (HMGFN), the adjacency matrix A is constructed using the self-attention mechanism:

[0054] in, is the adjacency matrix representing the association between nodes; yes The dot product calculates the similarity between word pairs in the text features, that is, the similarity between each word pair. is a scaling factor used to stabilize the calculation and prevent the dot product from being too large. is applied to each row of the matrix Operation to ensure that the sum of the elements in each row is 1, that is, to obtain a probability distribution that represents the strength of the relationship between each pair of words.

[0055] Step S32, use the skip connection graph convolutional network (SGCN) to perform multi-level information aggregation. The graph convolutional network of each layer combines the node features with the adjacency matrix to capture the dependencies between modalities. The adjacency matrix A and the enhanced text features obtained by ACIM are combined. As the initial input of SGCN, let , then for the Layer SGCN:

[0056] in is the feature matrix of the i-th layer; It is The learnable weight matrix of the layer; ReLU is the activation function; the skip connection term Used to preserve the original feature information. is the length of the text sequence; d is the feature dimension of each text word embedding.

[0057] The key step of SGCN is to aggregate the neighbor information of each node. Adjacency matrix A is used to transform the input feature matrix The weighted aggregation is performed, and the aggregated feature matrix represents the weighted average features of each node and its neighbors. The input feature matrix is ​​connected through the jump connection Adding it to the transformed feature matrix can retain the input features and avoid information loss.

[0058] Step S33, nonlinear transformation is introduced by applying the ReLU activation function to obtain the final feature matrix S, so as to ensure that the model can learn more complex nonlinear relationships, thereby increasing the representation ability of the model.

[0059] Step S4, combining the final feature matrix and the adjacency matrix, and using an encoder and a classifier to obtain a prediction result of sentiment analysis.

[0060] The final feature matrix S can capture the complex dependencies between different modalities and has structured global information; while the adjacency matrix A obtained through ACIM aligns the temporal and content information of different modalities and captures the local correlation between text and non-text modalities.

[0061] Step S41, using The function concatenates the final feature matrix and the adjacency matrix to obtain the fused features;

[0062] Where concat(·) is the concatenation function.

[0063] Then use the Linear layer to perform a linear transformation to transform the H dimension:

[0064] in Represents a linear transformation operation.

[0065] Step S42, adding the fused features to the initial text embedding T to obtain text enhanced feature representation;

[0066] In the formula, Enhance feature representation for text. By summing up text embeddings, it ensures that the comprehensive information of all modalities can be optimally represented in downstream tasks.

[0067] Step S43, inputting the text enhanced feature representation into the trained Transformer encoder and classifier to obtain the prediction result of sentiment analysis.

[0068] Furthermore, in this embodiment, experiments are performed on two public data sets, CMU-MOSI and CMU-MOSEI, to demonstrate the method proposed in the present invention.

[0069] The CMU-MOSI dataset is a public dataset commonly used in multimodal sentiment analysis and emotion recognition. It contains information in three modalities: text, vision, and audio. It consists of 93 personal videos from YouTube, which are subjectively annotated into a total of 2199 annotated segments, each of which is a part of the video. Each segment has an emotional intensity annotation in the range of [-3, 3]. Among them, -3 represents strong negative emotion, 0 represents neutral, and 3 represents strong positive emotion. We divide the dataset into three parts: training set (1284 segments), validation set (229 segments), and test set (686 segments).

[0070] The CMU-MOSEI dataset is an extended version of MOSI, which makes it suitable for larger-scale multimodal sentiment analysis tasks. It also comes from YouTube videos, covering a variety of topics such as movie reviews and speeches. It contains 3229 videos, which are decomposed into 23454 labeled segments after subjective annotation of sentiment intensity. Each of its labeled segments not only has a sentiment intensity label of [-3, 3] and also adds binary classification labels of 6 basic emotions (happy, sad, angry, fearful, disgusted, and surprised).

[0071] In order to evaluate the performance of the model proposed in this invention, two evaluation tasks, binary classification and regression, are constructed. The binary classification task is evaluated using binary classification accuracy (Acc-2) and weighted F1 score (F1-score). There are two classification methods: negative / non-negative and negative / positive. For the regression task, the model performance is evaluated by mean absolute error (MAE) and Pearson correlation (Corr), as shown in Table 1.

[0072] Table 1. Performance comparison and evaluation indicators of MAG-TE with 19 baselines on CMU-MOSI and CMU-MOSEI datasets. An upward arrow indicates that the higher the indicator, the better, and a downward arrow indicates the opposite.

[0073]

[0074] Experimental results show that the MAG-TE model outperforms the compared models on four benchmark datasets, which verifies the effectiveness of the proposed model in the MSA task.

[0075] As can be seen from the table, the Transformer-based models (such as MAG, Self-MM, MISA, CENet, etc.) are generally better than traditional multimodal methods (such as TFN, LMF, MFN, MulT) in various evaluation indicators on the two datasets. This trend shows that the Transformer architecture can better capture the interactive relationship of multimodal information and fully explore the dependencies between modalities through its global attention mechanism. Among all Transformer-based models, the MAG-TE model proposed in this application shows significant advantages in various indicators, which further proves the effectiveness of the multimodal enhancement module (MTE) in aligning and fusing multimodal information.

[0076] Compared with the current mainstream multimodal alignment-based methods (such as MAG, Self-MM, and CENet), the present invention has achieved leading results in the two core indicators, MAE and Corr. On the CMU-MOSI dataset, MAG-TE's MAE reached 0.581, which was 2.5% and 18.5% higher than the best performing CENet (0.596) and Self-MM (0.713), respectively. At the same time, on the CMU-MOSEI dataset, MAG-TE's MAE was further reduced to 0.516, significantly better than other models. This shows that MAG-TE effectively solves the problems of modal asynchrony and context misalignment through the adaptive cross-modal interaction module (ACIM), greatly improving the prediction accuracy. In addition, in terms of the Corr indicator reflecting emotional correlation, MAG-TE reached the highest value of 0.868 on the CMU-MOSI dataset, and the Corr on the CMU-MOSEI dataset also reached 0.850, which fully verified the significant ability of MAG-TE in modeling multimodal correlation.

[0077] In the comparison based on multimodal fusion methods, the MAG-TE method also achieved better performance in classification indicators such as Acc-2 and F1. Specifically, on the CMU-MOSI dataset, MAG-TE's Acc-2 is 87.76% and F1 is 87.69%, surpassing CENet (Acc-2 is 86.74%, F1 is 86.69%) and MAG (Acc-2 is 84.20%, F1 is 84.14%). Similarly, on the CMU-MOSEI dataset, MAG-TE's Acc-2 and F1 indicators also achieved comprehensive surpassing. This shows that MAG-TE effectively improves the accuracy and robustness of sentiment classification by explicitly modeling the global and local dependencies between modalities through the hierarchical multimodal graph fusion network (HMGFN).

[0078] Embodiment 2 This embodiment discloses a multimodal sentiment analysis system based on attention and graph-enhanced text; like Figure 2 As shown in the figure, the multimodal sentiment analysis system based on attention and graph-enhanced text includes: The multimodal feature acquisition module is configured to: acquire text features, image features and speech features in the video data and perform preprocessing; The text feature enhancement module is configured to: use an adaptive cross-modal interaction module to calculate the attention weights between text features, image features, and speech features to obtain enhanced text features; The final feature matrix acquisition module is configured to: input the enhanced text features into the hierarchical multimodal graph fusion network, and construct the adjacency matrix using the self-attention mechanism; input the adjacency matrix and the enhanced text features into the skip connection graph convolutional network to obtain the final feature matrix; The sentiment analysis prediction module is configured to: combine the final feature matrix and the adjacency matrix, and use an encoder and a classifier to obtain a prediction result of the sentiment analysis.

[0079] Embodiment 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0080] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the multimodal sentiment analysis method based on attention and graph-enhanced text as described in Example 1.

[0081] Embodiment 4 The purpose of this embodiment is to provide an electronic device.

[0082] An electronic device comprises a memory, a processor and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the multimodal sentiment analysis method based on attention and graph-enhanced text as described in Example 1 are implemented.

[0083] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0084] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0085] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A multimodal sentiment analysis method based on attention and graph-enhanced text, characterized in that: include: Obtain text features, image features and speech features in video data and perform preprocessing; Input the preprocessed text features, image features, and speech features into the trained multimodal sentiment analysis model to obtain the prediction results of sentiment analysis; The trained multimodal sentiment analysis model uses an adaptive cross-modal interaction module to calculate the attention weights between text features, image features, and speech features to obtain enhanced text features. The enhanced text features are input into the hierarchical multimodal graph fusion network, and the adjacency matrix is ​​constructed using the self-attention mechanism; the adjacency matrix and the enhanced text features are input into the skip connection graph convolutional network to obtain the final feature matrix; The feature matrix and the adjacency matrix are combined, and the prediction results of the sentiment analysis are obtained using an encoder and a classifier.

2. The multimodal sentiment analysis method based on attention and graph-enhanced text as claimed in claim 1, characterized in that The pre-treatment process includes: Process text features, use a pre-trained language model to segment and embed text, and generate word embedding and position embedding; Extracting image information of facial expressions, action units and postures from video frames; performing dimensionality reduction operations on the image information to obtain image features; Speech emotion information is extracted from the speech frame, and the speech emotion information is subjected to dimensionality reduction operation to obtain speech features.

3. The multimodal sentiment analysis method based on attention and graph-enhanced text as claimed in claim 1, characterized in that The adaptive cross-modal interaction module is used to calculate the attention weights between text features, image features, and speech features. The process of obtaining enhanced text features is as follows: An adaptive cross-modal interaction module is used to map the image features and speech features into a unified index sequence; The index vector is processed using the cross-modal attention mechanism, and the attention weights between text features, image features, and speech features are calculated to obtain enhanced text features.

4. The multimodal sentiment analysis method based on attention and graph-enhanced text as claimed in claim 3, characterized in that The process of using the cross-modal attention mechanism to process the index vector, calculate the attention weights between the text features, the image features, and the speech features, and obtain the enhanced text features is as follows: The discrete index sequence is mapped into a continuous high-dimensional vector representation through the embedding layer; Take the text feature as the query vector, the image feature and the speech feature as the key vector and the value vector, and obtain the attention weight matrix between the text feature, the image feature and the speech feature; Based on the attention weight matrix, the enhanced non-language information is extracted from the image features and speech features to generate non-language embedding information corresponding to the text features; The non-language embedding information and the text features are fused to obtain enhanced text features.

5. The multimodal sentiment analysis method based on attention and graph-enhanced text as claimed in claim 1, characterized in that The adjacency matrix constructed using the self-attention mechanism is: In the formula, is the adjacency matrix representing the association between nodes; yes The matrix transpose of represents the similarity between word pairs in text features; is the scaling factor, used to stabilize the calculation and prevent the dot product from being too large; is applied to each row of the matrix operation, indicating the strength of the relationship between each pair of words.

6. The multimodal sentiment analysis method based on attention and graph-enhanced text as claimed in claim 1, characterized in that: The process of inputting the adjacency matrix and the enhanced text features into the skip connection graph convolutional network to obtain the final feature matrix is ​​as follows: The adjacency matrix and the enhanced text features are used as the initial input of the skip connection graph convolutional network, where the enhanced text features are used as the initial feature matrix; The initial feature matrix is ​​weighted and aggregated using the adjacency matrix. The aggregated feature matrix represents the weighted average features of each node and its neighbors. A skip connection is used to add the input feature matrix to the transformed feature matrix, and the ReLU activation function is used to obtain the final feature matrix.

7. The multimodal sentiment analysis method based on attention and graph-enhanced text as claimed in claim 1, characterized in that The process of combining the feature matrix and the adjacency matrix and using the encoder and the classifier to obtain the prediction result of the sentiment analysis is as follows: use The function concatenates the final feature matrix and the adjacency matrix to obtain the fused features; Adding the fused features to the original text embedding to obtain text enhanced feature representation; The text-enhanced feature representation is input into the trained Transformer encoder and classifier to obtain the prediction result of sentiment analysis.

8. A multimodal sentiment analysis system based on attention and graph-augmented text, characterized by: include: The multimodal feature acquisition module is configured to: acquire text features, image features and speech features in the video data and perform preprocessing; The text feature enhancement module is configured to: use an adaptive cross-modal interaction module to calculate the attention weights between text features, image features, and speech features to obtain enhanced text features; final The feature matrix acquisition module is configured to: input the enhanced text features into the hierarchical multimodal graph fusion network and construct an adjacency matrix using the self-attention mechanism; input the adjacency matrix and the enhanced text features into the skip connection graph convolutional network to obtain the final feature matrix; The sentiment analysis prediction module is configured to: combine the final feature matrix and the adjacency matrix, and use an encoder and a classifier to obtain a prediction result of the sentiment analysis.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the multimodal sentiment analysis method based on attention and graph-enhanced text are implemented as described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the multimodal sentiment analysis method based on attention and graph-enhanced text are implemented as described in any one of claims 1-7.

Images