Multi-modal sentiment analysis method and system for graph structure optimization and representation separation

Abstract

The invention discloses a multi-modal sentiment analysis method and system for graph structure optimization and representation separation, and relates to the technical field of multi-modal sentiment analysis, and the method comprises the steps: processing original video data to obtain multi-modal representation carrying time sequence information; carrying out inter-sample data enhancement on the multi-modal representation; performing context interaction processing on the multi-modal representation after data enhancement between the samples to obtain context interaction enhanced multi-modal representation; performing emotion representation decoupling processing on the context interaction enhanced multi-modal representation to obtain a multi-modal representation containing emotion information; and performing emotion prediction according to the multi-modal representation containing the emotion information to obtain a final emotion prediction result. According to the method, the characterization learning in the mode, the isomerism of the multi-mode characterization and the emotion information are fully mined, and the accuracy and the robustness of the multi-mode emotion analysis are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal sentiment analysis, and more specifically to a multimodal sentiment analysis method and system for graph structure optimization and representation separation. Background Art

[0002] As a key technology in the field of artificial intelligence, sentiment analysis can improve machine intelligence perception capabilities, improve the quality of human-computer interaction, and promote the iterative upgrade of wearable smart devices, digital home smart terminal devices, and intelligent perception and control devices. It is of great significance. Nowadays, the information provided by single-modal data can no longer meet the needs of intelligent systems to perceive and understand the world. Multimodal cognitive computing, which aims to simulate human cognition to efficiently perceive and comprehensively understand multimodal inputs, has become the key to realizing general artificial intelligence. Sentiment analysis has also gradually evolved from the initial single-modal analysis to multimodal sentiment analysis.

[0003] The core of multimodal sentiment analysis is to establish a reliable mapping between multimodal data and sentiment polarity. This task focuses on learning people's emotional information from multimodal sequences such as text, vision, and audio, and mining emotional associations and complex relationships between modalities. Related technologies have attempted to integrate the emotional information of heterogeneous data by building complex models based on deep learning, and promote information interaction between multimodal data. Although these methods have achieved certain results, there are still many challenges and technical bottlenecks.

[0004] First, how to learn effective representations has become a major difficulty. Many existing methods use the attention mechanism to learn modal representations. This method does not explicitly model the complexity relationship within the modalities, resulting in the model being unable to fully utilize the complete representation information within each modality. Secondly, there is heterogeneity between different modal data, resulting in differences in the representation distribution of emotional information. This difference directly affects the extraction of emotional information during modal interaction, and thus affects the multimodal fusion results. In addition, another major challenge is how to fully mine emotionally relevant information while suppressing the interference of irrelevant information. In the research process, existing methods consider removing inter-modal noise information more, while ignoring the weak emotional related information contained in the modal after interaction, which may lead to a decrease in the accuracy and generalization ability of the model. In summary, the main problems facing current multimodal sentiment analysis technology are intra-modal representation learning, heterogeneity of multimodal representations, and insufficient mining of emotional information. These problems have seriously restricted the development and application of multimodal sentiment analysis technology.

[0005] Therefore, how to propose a multimodal sentiment analysis method and system with graph structure optimization and representation separation, improve the extraction effect of intra-modal and inter-modal correlation information, mine sentiment-related information after modal interaction, and improve the accuracy and robustness of multimodal sentiment analysis is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0006] In view of this, the present invention provides a multimodal sentiment analysis method and system with graph structure optimization and representation separation, improves the extraction effect of intra-modal and inter-modal correlation information, mines sentiment-related information after modal interaction, and improves the accuracy and robustness of multimodal sentiment analysis. In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] A multimodal sentiment analysis method with graph structure optimization and representation separation, comprising:

[0008] Processing the original video data to obtain a multimodal representation carrying time series information;

[0009] Perform inter-sample data augmentation on multimodal representations;

[0010] The multimodal representation after data enhancement between samples is subjected to context interaction processing to obtain a context interaction enhanced multimodal representation;

[0011] Decouple the context interaction enhanced multimodal representation from the emotional representation to obtain a multimodal representation containing emotional information.

[0012] Emotion prediction is performed based on the multimodal representation containing emotional information to obtain the final emotion prediction result.

[0013] Optionally, the multimodal representation includes: text modality representation, audio modality representation and visual modality representation.

[0014] Optionally, obtaining a multimodal representation carrying time series information includes:

[0015] Divide a piece of original video data into multiple short video clips;

[0016] Extracting utterance-level representations of textual modality, utterance-level representations of audio modality, and utterance-level representations of visual modality from multiple short video clips;

[0017] The quantum long short-term memory network is used to analyze the time series correlation information between the discourse-level representation of the text modality, the discourse-level representation of the audio modality, the discourse-level representation of the visual modality and the representation of other video clips, and the correlation information is fused with the discourse-level representation to obtain the representation of each modality.

[0018] Optionally, the performing inter-sample data enhancement on the multimodal representation includes:

[0019] The visual modality representation is input into the graph convolutional network, the adjacency matrix is ​​calculated through the correlation between nodes, and then the graph adjacency matrix is ​​optimized using constraints to obtain the visual modality representation after data enhancement between samples.

[0020] The text modal representation and audio modal representation are input into the graph attention network, and the feature expression between low-correlation nodes is constrained by similarity calculation, and a relationship network between samples is constructed to obtain the text modal representation after data enhancement between samples and the audio modal representation after data enhancement between samples.

[0021] Optionally, constructing a relationship network between samples includes:

[0022] Use the features of two samples to calculate the cosine similarity to obtain the correlation between samples;

[0023] The absolute difference between the labels is calculated using the true labels of the two samples to obtain the sentiment difference between the samples.

[0024] Both similarity threshold and sentiment threshold are network parameters;

[0025] By comparing the difference between the correlation between samples and the similarity threshold, as well as the difference between the emotion difference between samples and the emotion threshold, a relationship network between samples is established.

[0026] Optionally, obtaining the context-interaction-enhanced multimodal representation includes:

[0027] The text modality representation enhanced by inter-sample data and the audio modality representation enhanced by inter-sample data are jointly input into the quantum-inspired cross-modal attention network to obtain the context-interaction-enhanced audio modality representation;

[0028] The text modality representation enhanced by inter-sample data and the visual modality representation enhanced by inter-sample data are jointly input into the quantum-inspired cross-modal attention network to obtain the context-interaction-enhanced visual modality representation;

[0029] The audio modal representation after inter-sample data enhancement and the visual modal representation after inter-sample data enhancement are concatenated with the text representation after inter-sample data enhancement and input into the positive and negative correlation attention network to obtain the context interaction enhanced text modal representation.

[0030] Optionally, obtaining a multimodal representation containing emotional information includes:

[0031] The context-interaction-enhanced text modality representation and the context-interaction-enhanced audio modality representation are jointly input into the sentiment feature fusion network to obtain text-audio modality information;

[0032] The context-interaction-enhanced text modality representation and the context-interaction-enhanced visual modality representation are jointly input into the sentiment feature fusion network to obtain text-visual modality information.

[0033] Merge text-audio modal information and text-visual modal information to obtain strong emotion-related information;

[0034] The context interaction enhanced text modality representation, the context interaction enhanced audio modality representation, and the context interaction enhanced visual modality representation are respectively input into independent encoders with the strong emotion related information to obtain the text emotion weak related information, the audio emotion weak related information, and the visual emotion weak related information;

[0035] The strong emotion-related information is weightedly summed with the text emotion-weakly related information, the audio emotion-weakly related information and the visual emotion-weakly related information to obtain the representations of each modality containing emotion information.

[0036] Optionally, obtaining the final emotion prediction result includes:

[0037] The text modality representation containing emotional information, the audio modality representation containing emotional information, and the visual modality representation containing emotional information are concatenated to form a fusion vector;

[0038] The fusion vector is input into the first part of the prediction network, and the weights of each modality feature are learned through the neural network to obtain the weight fusion vector;

[0039] The weighted fusion vector is input into the second part of the prediction network to gradually reduce the dimension of the modal information and finally obtain the emotion prediction result of the video data.

[0040] Optionally, a multimodal sentiment analysis system with graph structure optimization and representation separation, comprising:

[0041] Preprocessing module: used to process the original video data to obtain a multimodal representation carrying time series information;

[0042] Inter-sample data enhancement module: used to enhance the inter-sample data of multimodal representation;

[0043] Context interaction processing module: used to perform context interaction processing on the multimodal representation after data enhancement between samples to obtain context interaction enhanced multimodal representation;

[0044] Emotional representation decoupling processing module: used to decouple the context interaction enhanced multimodal representation from the emotional representation to obtain a multimodal representation containing emotional information;

[0045] Emotion prediction module: used to perform emotion prediction based on multimodal representations containing emotional information to obtain the final emotion prediction results.

[0046] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a multimodal sentiment analysis method and system for graph structure optimization and representation separation, which has the following beneficial effects:

[0047] The present invention discloses a multimodal sentiment analysis method and system with graph structure optimization and representation separation, including: processing original video data to obtain a multimodal representation carrying time series information; performing inter-sample data enhancement on the multimodal representation; performing context interaction processing on the multimodal representation after inter-sample data enhancement to obtain a context interaction enhanced multimodal representation; performing sentiment representation decoupling processing on the context interaction enhanced multimodal representation to obtain a multimodal representation containing sentiment information; performing sentiment prediction based on the multimodal representation containing sentiment information to obtain a final sentiment prediction result. The present invention fully exploits intra-modal representation learning, heterogeneity of multimodal representations, and sentiment information, thereby improving the accuracy and robustness of multimodal sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] FIG1( a ) is a schematic diagram of a multimodal sentiment analysis method for graph structure optimization and representation separation provided by the present invention;

[0050] FIG1( b ) is a framework diagram of a multimodal sentiment analysis method for graph structure optimization and representation separation provided by the present invention;

[0051] FIG2( a ) is a schematic diagram of a quantum LSTM provided by the present invention;

[0052] FIG2( b ) is a schematic diagram of a quantum circuit provided by the present invention;

[0053] FIG3( a ) is a schematic diagram of a quantum-inspired cross-modal attention network provided by the present invention;

[0054] FIG3( b ) is a schematic diagram of a positive and negative correlation attention network provided by the present invention;

[0055] Figure 4 A schematic diagram of a decoupler provided by the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The embodiment of the present invention discloses a multimodal sentiment analysis method for graph structure optimization and representation separation, including:

[0058] Processing the original video data to obtain a multimodal representation carrying time series information;

[0059] Perform inter-sample data augmentation on multimodal representations;

[0060] The multimodal representation after data enhancement between samples is subjected to context interaction processing to obtain a context interaction enhanced multimodal representation;

[0061] Decouple the context interaction enhanced multimodal representation from the emotional representation to obtain a multimodal representation containing emotional information.

[0062] Emotion prediction is performed based on the multimodal representation containing emotional information to obtain the final emotion prediction result.

[0063] In a specific implementation, a multimodal sentiment analysis method based on graph structure optimization and representation separation, as shown in Figure 1(a), the multimodal sentiment analysis method based on graph structure optimization and representation separation may include: a text representation extraction module 101, an audio representation extraction module 102, a visual representation extraction module 103, a representation enhancement module 104, a context interaction module 105, a sentiment representation decoupling module 106 and a sentiment prediction module 107.

[0064] The original video data is respectively input to the text representation extraction module 101, the audio representation extraction module 102 and the visual representation extraction module 103, and the corresponding text representation, audio representation and visual representation are obtained respectively after being processed by the text representation extraction module 101, the audio representation extraction module 102 and the visual representation extraction module 103. Specifically, the original video data can be divided into multiple short video clips, and then the text representation extraction module 101, the audio representation extraction module 102 and the visual representation extraction module 103 respectively obtain the discourse-level modal representation of the text, audio and visual modalities from one short video clip.

[0065] Input the obtained text representation, audio representation and visual representation to the representation enhancement module 104 to obtain the text representation, audio representation and visual representation after the inter-sample data enhancement;

[0066] Inputting the obtained text representation, audio representation and visual representation after the inter-sample data enhancement into the context interaction module 105 to obtain the text representation, audio representation and visual representation enhanced by the context interaction;

[0067] The context-interaction-enhanced text representation, audio representation, and visual representation are input into the emotion representation decoupling module 106 to obtain the text representation, audio representation, and visual representation containing emotion information.

[0068] The text representation, audio representation and visual representation containing emotional information are input into the emotion prediction module 107 to obtain the final multimodal emotion prediction result.

[0069] In a specific implementation, a multimodal sentiment analysis method for graph structure optimization and representation separation comprises the following specific steps:

[0070] Step 1: Input the original video data into the feature extraction module to obtain the modal representation carrying time series information.

[0071] As shown in Figure 1(b), a piece of original video data is divided into multiple short video clips based on each sentence, and then feature extraction technology is used to obtain the discourse-level modal representation of text, audio and visual modalities from a short video clip. Specifically, the GloVe model, COVAREP model and Facet model are used to encode the modal signals such as letters, acoustics and images in the video clips respectively, and finally the text modal features f of size 1*k are obtained. t , audio modal features f a and visual modality features f v The text modality features are input into the BERT model, and the audio and visual modality features are input into the quantum long short-term memory network model to mine the contextual information between different video clips, and then the modal representations F of the three modalities are obtained. t 、F a and F v , and their sizes are all 1*128.

[0072] Figure 2(a) shows the QLSTM network model structure. Each memory cell is also composed of three parts: forget gate, input gate, and output gate. The difference from the classic LSTM is that the QLSTM network model replaces the different weight parameter matrices in the classic LSTM with 5 newly constructed VQCs with different parameters. Each VQC implements different functions according to the position of the gate. VQC is a quantum circuit with adjustable parameters that can be iteratively optimized to extend the classic LSTM to the quantum field. Figure 2(b) shows the structure of VQC, which consists of three parts: encoding layer, variation layer, and measurement layer.

[0073] In a specific implementation, when performing quantum circuit processing, the coding layer is used to realize the conversion of classical data to quantum state. The present invention uses an angle coding method to process the original data. The specific process is implemented by the RX gate, which can be expressed as formula (1). By using the RX gate, the classical data can be effectively encoded into the rotation angle of the quantum bit, thereby taking advantage of quantum computing to perform complex quantum state transformation and processing. The total operation of the coding layer can be expressed as formula (2).

[0074]

[0075]

[0076] Among them, f i represents different classical data features, n represents the number of quantum bits, I is the unit matrix, is the tensor product, h t-1 represents the hidden state at time step t-1, h t represents the hidden state at time step t, C t-1 is the cell state at time step t-1, C t is the cell state at time step t, y t is the output at time step t, and X is the Pauli-X gate.

[0077] Specifically, the variational layer performs variational adjustment through parameterized quantum gates to optimize the quantum state. First, an RX gate is applied to each quantum bit to rotate the state of the quantum bit around the X-axis, and its parameter θ is trainable. This operation transforms the input state into a new quantum state. Next, a Hadamard (H) gate is applied to each quantum bit to transform the quantum bit state into a uniform superposition state, in which the state of each quantum bit is a superposition state of |0> and |1>. Finally, in order to generate an entangled state, a CNOT gate is applied to each pair of adjacent quantum bits to conditionally flip the state of the target quantum bit. The variational layer optimizes and entangles the quantum state through a series of trainable RX gates, H gates, and CNOT gates to form a complex quantum state adapted to a specific task. These operations not only change the amplitude of the quantum state, but also introduce entanglement relationships between quantum states, thereby providing rich information for subsequent measurements and calculations. The operation formula of the variational layer is shown in formula (3).

[0078]

[0079]

[0080] Among them, H i represents the application of H gate on the i-th quantum bit, θ i is the rotation angle of the i-th quantum bit, CNOT i，i+1represents the application of a controlled NOT gate on the i-th qubit and the i+1-th qubit, CNOT n-1，0 represents the application of a controlled NOT gate on the last qubit and the first qubit.

[0081] Specifically, in the measurement layer, as shown in formula (4), the quantum state after the encoding layer and the variation layer transformation is measured to extract classical information. In order to retain the information of the input data to the maximum extent, the present invention uses Pauli-Z basis measurement, and the measurement result reflects the projection of the quantum state in the Z-axis direction, thereby extracting useful features for further processing. The overall measurement result is shown in formula (5).

[0082] <Z i >=<ψ″|Z i |ψ″)(4);

[0083] Measurement = { <z0>, <Z1),...,<Z n-1 >} (5);

[0084] Wherein, i represents the i-th quantum bit, n represents the number of quantum bits, and n=8 in the present invention.

[0085] Step 2: Input the modal representation carrying time series information obtained in step 1 into the representation enhancement module, and use the graph neural network method to divide the processing process into two parts. In the first part, the visual modal representation is passed into the graph convolutional network to capture the local structure and boundary information of the image; in the second part, the text modality and audio modality are passed into the graph attention network, and the features of different samples are aggregated in units of samples to enhance the expressiveness of the samples; the nodes are constrained using cosine similarity and label difference, and a relationship network between samples is constructed to learn the modal representation between similar samples and obtain the modal representation after data enhancement between samples.

[0086] In a specific implementation, the first part includes, in order to create nodes that can realize information exchange between different samples of the same modality, when constructing a graph, all samples in a batch are put into an optimization pool, and these samples are constructed into a graph, with each sample as a node. If two nodes are close in feature embedding, they are assumed to be connected. The present invention uses a simple cross-node self-attention mechanism to determine the correlation between nodes, thereby defining the adjacency matrix A of the graph v . As shown in formula (6).

[0087]

[0088] Among them A v ∈R N×N represents the adjacency matrix, N v ∈R N×d represents the embedding vector of N sample nodes, represents the learnable parameters, f is set to the Relu activation function, I is the identity matrix, and d is the hidden layer dimension.

[0089] Specifically, as shown in formula (7), the graph structure optimization strategy optimizes the relationship between nodes by similarity calculation. The optimization strategy is: if the similarity between two nodes is greater than a certain threshold and the absolute value of the true label difference is less than a certain threshold, the two nodes are determined to be neighbors. Then, all nodes in the optimization pool are traversed, the edges between nodes that do not meet the requirements are disconnected, and the nodes with high similarity are connected, so as to dynamically adjust the adjacency matrix and reconstruct the graph.

[0090]

[0091]

[0092]

[0093] Among them, S represents the similarity between two nodes, and Y represents the absolute value of the difference between the true labels. is the feature representation of the i, jth sample, represents the true label of the i, jth sample, U represents the optimizer, is the updated adjacency matrix, θ v are optimizer parameters.

[0094] Specifically, as shown in formula (8), the GCN algorithm is used to enrich the feature representation of each sample, helping the model to understand and utilize the correlation between samples, ensuring that the node can accurately learn the features of all similar nodes instead of relying solely on a single sample information, thereby improving the potential correlation between sample features.

[0095]

[0096] Where D∈R N×N is the degree matrix, X v ∈R N×d Represents the sample features after GCN enhancement.

[0097] In a specific implementation, the second part includes, in the text graph and the audio graph, each sample is regarded as a node of the graph. The optimization strategy is: select K nodes that are most similar to the characteristic emotion expression of the own node as its own neighbor nodes, filter out other low-correlation nodes, build a relationship network between samples, and strengthen the characteristic expression of the nodes in the graph.

[0098] Specifically, the selection of neighbor nodes is also based on the difference between the similarity between samples and the true label to select K other samples that are highly similar to the sample. As shown in formula (9), according to the characteristics of GAT, for vertex i, the similarity coefficient e between its neighbor j and itself is calculated one by one ij First, we use the shared parameter W∈R d×d The linear mapping of the vertex increases the dimension of the vertex features, and then the vertex h i and h j The transformed features are concatenated, and finally, the concatenated high-dimensional features are mapped to a real number b, and then the attention coefficient α is obtained after softmax processing. ij The attention coefficient replaces the elements of the initial adjacency matrix to obtain the final adjacency matrix. Then the classic GAT algorithm is used to assign different weights to different neighbor nodes to obtain a richer node enhancement representation.

[0099]

[0100]

[0101]

[0102] Among them, m∈(t,a) represents different modes, b∈R 2d×1 Map the concatenated matrix to a real number, represents the similarity coefficient, X m ∈R 1×d is the sample feature enhanced by GAT, is the feature vector of the i-th sample, is the feature vector of the jth sample, and W is the linear layer.

[0103] The method of using cosine similarity and label difference to constrain node relationships is as follows: for each modal sample representation input into the graph neural network, the cosine similarity and the absolute difference of the true label are calculated pairwise, and thresholds are set for these two constraints. When the cosine similarity of the two samples is greater than the similarity threshold and the absolute difference of the true label is less than the label threshold, the relationship network between the samples is established.

[0104] Step 3: The modal representation enhanced by the inter-sample data is passed into the interactor, and the dominant role of the text modal representation after the inter-sample data enhancement is brought into play through the quantum-inspired cross-modal attention mechanism, so as to inject richer emotional information into the audio and visual modal representations after the inter-sample enhancement; considering the advantages of the audio and visual modal representations after the inter-sample enhancement in special contexts, the emotional information of the three modalities is balanced through positive and negative correlation attention; that is, the representations of each modality after interaction enhancement are obtained. Among them, Figure 3(a) and Figure 3(b) are schematic diagrams of a quantum-inspired cross-modal attention network and a schematic diagram of a positive and negative correlation attention network provided in an embodiment of the present invention.

[0105] In a specific implementation, as shown in formula (10), taking the text modality to enhance the visual modality as an example, the text and visual modalities are converted into query (Q), key (K) and value (V) through a linear layer, which will be used as the input of the attention module. The attention score is calculated by the inner product of Q and K, and then the weighted sum is obtained with V to obtain a new attention vector. In order to further improve the model's ability to capture data complexity, the model introduces a quantum layer after the attention calculation to enhance the expressiveness. The design of the quantum circuit is shown in Figure 2(b). Since multi-head attention allows the model to capture information at different levels of abstraction, the model uses multiple groups of W q , W k , W v Combine to learn the multi-dimensional representation of the input data, and finally concatenate the outputs of multiple heads to obtain a cross-modal representation Facilitates one modality to receive information from another modality.

[0106]

[0107] F att =Concat(head1,…,head h )W o ) (10);

[0108] Among them, F att is the sample feature vector after multi-head attention enhancement, is the Query, Key and Value vector corresponding to the i-th attention head, W o is a learnable weight parameter, W Q Used to convert data into a suitable dimension for quantum processing, head i is the number of attention heads, d k is the feature dimension, is the text vector of the i-th attention head, is the visual vector of the i-th attention head.

[0109] Furthermore, as shown in formula (11), in order to prevent the degradation problem in deep network training, the result after multi-head attention processing is residually connected with the initial information. At the same time, the feedforward layer is added to enhance the nonlinear modeling and expression capabilities of the model.

[0110]

[0111]

[0112] in, represents the non-text modality feature vector after text modality enhancement, FFN represents the feed-forward neural network layer, LN represents Layer Normalization, is a visual vector that carries text information.

[0113] In a specific implementation, as shown in formula (12), the audio and visual modalities are concatenated as the key (K) and value (V) of the attention mechanism, the text modality is regarded as the query (Q), and the cross-modal attention mechanism is used to capture the long-range non-language emotional information, and generate text-based non-language embedding. In particular, the present invention designs a dual-channel interaction method. In the first channel, a positively correlated single-head attention mechanism is used to promote the interaction of contextual information expressing similar emotions in different modalities. In the other channel, a negatively correlated attention mechanism is used to focus on the differences in emotional expression of different modalities, thereby obtaining a cross-modal enhanced representation with complete emotional semantics.

[0114] F va =Concat(X v , X a );

[0115]

[0116]

[0117]

[0118]

[0119] in, is a learnable parameter, p represents positive related attention, n represents negative related attention, is a scale parameter. During training, since the language mode and non-language mode are in two different feature spaces, the correlation between them is small, so the elements of the weight matrix will be small. In order to facilitate model learning, the weight coefficient is introduced in the process of calculating the weight coefficient. Scaling the matrix, is the single-head attention text query vector, For audio-visual keys, is the audio-visual value, is the audio-visual attention vector, is the transpose of the audio-visual key.

[0120] Furthermore, as shown in formula (13), the positively correlated attention result vector And the negatively correlated attention result vector Add and add time series information. Based on this, a dynamic weighting operation is implemented to combine the weighted fusion result with the original text vector. This process ensures that the text modality is effectively enhanced without losing text information.

[0121]

[0122]

[0123]

[0124] Among them, k va represents the size of the convolution kernel, β is the adaptive factor, ||X t ||2 and Respectively represent X t and The L2 norm of , μ is a learnable parameter, and δ is equal to 10e-6 to prevent the denominator from being zero.

[0125] Step 4: Rearrange and combine the interactively enhanced modal representations, extract emotional information from the two dimensions of features and time, and then splice them together to generate strong emotional related information; pass the interactively enhanced modal representations and the strong emotional related information into the encoder respectively to obtain weak emotional related information independent of each modality; dynamically add the strong emotional related information and the weak emotional related information to finally obtain the decoupled modal representation. Figure 4 As shown, a schematic diagram of a decoupler implementation is shown, and the steps of implementing the decoupler are:

[0126] As shown in formula (14), the text-visual modality and the text-audio modality are input into two independent feed-forward layers respectively, which promotes deep interactive fusion of different modalities in different dimensions.

[0127]

[0128]

[0129]

[0130] Among them, r∈(a, v), CMLP and TMLP represent the channel feedforward layer and the time feedforward layer respectively. and are the parameters of the two feed-forward layers, is the vector processed by the channel feed-forward layer, is the vector processed by the time feedforward layer, is the text feature after being processed by the graph attention network, It is the non-text modal feature after graph processing.

[0131] Furthermore, as shown in formula (15), the generated feature vectors are concatenated and transformed nonlinearly to generate gating factors, and then the representations of the two modal pairs are multiplied by their respective gating vectors to calculate the strong sentiment correlation vector F s .

[0132]

[0133] g a =Sigmoid(W ga [F ta ; F tv ]+b ga );

[0134] g v =Sigmoid(W gv [F ta ; F tv ]+b gv );

[0135] F s =g a ·(W ta F ta )+g v ·(W tv F tv )(15);

[0136] Among them, W ga and W gv is a learnable gating matrix, b ga and b gv is the corresponding bias scalar, W ta and W tv is a learnable parameter, is the vector of the text-audio bimodal modality after being processed by the channel and time feed-forward layer, is the vector of the text-visual bimodal modality after being processed by the channel and time feed-forward layer, g a is the audio control factor, g v It is a visual control factor.

[0137] Furthermore, as shown in formula (16), the strong sentiment correlation vector and the source representation of each modality are sent to independent encoders to generate the weak sentiment correlation vector F of each modality. w , further refine and supplement the representation of the modality. Finally, the strong emotion correlation and weak emotion correlation vectors are combined by dynamic weighting to obtain the single-modal emotion fusion vector F y .

[0138] F wc =E(F s , F c ,θ s ), c∈(a, t, v);

[0139] F yc =F s +αF wc (16);

[0140] Among them, E is the encoder, which is used to generate weak sentiment related information, θ s are encoder parameters and α is a hyperparameter.

[0141] Step 5: The decoupled modal representation is passed to the emotion prediction layer, which includes a two-part multimodal emotion prediction neural network. The first part receives the decoupled modal representations of each modality and learns the weights of each modal feature through the neural network to highlight the importance of different modalities. The second part consists of a three-layer neural network, which gradually reduces the dimensionality of the modal information and finally outputs the emotion prediction result.

[0142] Specifically, as shown in formula (17), the three-modal feature vectors obtained above are concatenated, the weight matrix att is calculated, and then the dot multiplication is performed with the concatenated vector as the result of the three-modal fusion. The final prediction After the linear layer, the predicted values ​​are distributed between -3 and 3.

[0143] h=Concat(F ya , F yt , F yv );

[0144] att = W2(RELU(W1*h));

[0145] h f = att*h;

[0146]

[0147] Among them, h is the fusion vector after the three-modal splicing, att is the attention weight matrix, and h f is the weight fusion vector, W1, W2, W out is a linear layer, b out is the bias term.

[0148] In a specific embodiment, Table (1) compares the results of the method provided by the embodiment of the present invention with various representative methods. The comparison is performed on the public dataset CMU-MOSEI in the field of multimodal sentiment analysis, and mainly tests the sentiment 2 classification performance of the model, that is, judging whether the sentiment category of the sample is "positive" or "negative". The test selects the accuracy of sentiment 2 classification, the accuracy of 7 classifications and the F1 score as evaluation indicators. The accuracy refers to the ratio of the number of correct sentiment classifications in the sample prediction results to the total number of samples. The larger the value, the better the prediction effect. The F1 score is the harmonic mean of the precision and the recall rate. The larger the value, the better the prediction effect. As shown in Table 1, compared with multiple classic models in the field of multimodal sentiment analysis, the embodiment of the present invention achieved better performance in the sentiment 2 classification test. Compared with the suboptimal MMML model, the 2 classification accuracy, 7 classification accuracy and F1 score were improved by 1.41% / 4.13%, 11.5% and 1.25% / 4.01% respectively.

[0149] Table 1

[0150]

[0151]

[0152] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0153] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal sentiment analysis method with graph structure optimization and representation separation, characterized in that: include: Processing the original video data to obtain a multimodal representation carrying time series information; Perform inter-sample data augmentation on multimodal representations; The multimodal representation after data enhancement between samples is subjected to context interaction processing to obtain a context interaction enhanced multimodal representation; Decouple the context interaction enhanced multimodal representation from the emotional representation to obtain a multimodal representation containing emotional information. Emotion prediction is performed based on the multimodal representation containing emotional information to obtain the final emotion prediction result.

2. The multimodal sentiment analysis method of graph structure optimization and representation separation according to claim 1, characterized in that: The multimodal representation includes: text modality representation, audio modality representation and visual modality representation.

3. The multimodal sentiment analysis method of graph structure optimization and representation separation according to claim 1 is characterized in that: The obtaining of a multimodal representation carrying time series information includes: Divide a piece of original video data into multiple short video clips; Extracting utterance-level representations of textual modality, utterance-level representations of audio modality, and utterance-level representations of visual modality from multiple short video clips; The quantum long short-term memory network is used to analyze the time series correlation information between the discourse-level representation of the text modality, the discourse-level representation of the audio modality, the discourse-level representation of the visual modality and the representation of other video clips, and the correlation information is fused with the discourse-level representation to obtain the representation of each modality.

4. The multimodal sentiment analysis method of graph structure optimization and representation separation according to claim 1, characterized in that: The inter-sample data enhancement of the multimodal representation includes: The visual modality representation is input into the graph convolutional network, the adjacency matrix is ​​calculated through the correlation between nodes, and then the graph adjacency matrix is ​​optimized using constraints to obtain the visual modality representation after data enhancement between samples. The text modal representation and audio modal representation are input into the graph attention network, and the feature expression between low-correlation nodes is constrained by similarity calculation, and a relationship network between samples is constructed to obtain the text modal representation after data enhancement between samples and the audio modal representation after data enhancement between samples.

5. The multimodal sentiment analysis method of graph structure optimization and representation separation according to claim 4 is characterized in that: The construction of the relationship network between samples includes: Use the features of two samples to calculate the cosine similarity to obtain the correlation between samples; The absolute difference between the labels is calculated using the true labels of the two samples to obtain the sentiment difference between the samples. Both similarity threshold and sentiment threshold are network parameters; By comparing the difference between the correlation between samples and the similarity threshold, as well as the difference between the emotion difference between samples and the emotion threshold, a relationship network between samples is established.

6. The multimodal sentiment analysis method of graph structure optimization and representation separation according to claim 1, characterized in that: The obtaining of the context interaction enhanced multimodal representation includes: The text modality representation enhanced by inter-sample data and the audio modality representation enhanced by inter-sample data are jointly input into the quantum-inspired cross-modal attention network to obtain the context-interaction-enhanced audio modality representation; The text modality representation enhanced by inter-sample data and the visual modality representation enhanced by inter-sample data are jointly input into the quantum-inspired cross-modal attention network to obtain the context-interaction-enhanced visual modality representation; The audio modal representation after inter-sample data enhancement and the visual modal representation after inter-sample data enhancement are concatenated with the text representation after inter-sample data enhancement and input into the positive and negative correlation attention network to obtain the context interaction enhanced text modal representation.

7. The multimodal sentiment analysis method of graph structure optimization and representation separation according to claim 1, characterized in that: The multimodal representation containing emotional information includes: The context-interaction-enhanced text modality representation and the context-interaction-enhanced audio modality representation are jointly input into the sentiment feature fusion network to obtain text-audio modality information; The context-interaction-enhanced text modality representation and the context-interaction-enhanced visual modality representation are jointly input into the sentiment feature fusion network to obtain text-visual modality information. Merge text-audio modal information and text-visual modal information to obtain strong emotion-related information; The context interaction enhanced text modality representation, the context interaction enhanced audio modality representation, and the context interaction enhanced visual modality representation are respectively input into independent encoders with the strong emotion related information to obtain the text emotion weak related information, the audio emotion weak related information, and the visual emotion weak related information; The strong emotion-related information is weightedly summed with the text emotion-weakly related information, the audio emotion-weakly related information and the visual emotion-weakly related information to obtain the representations of each modality containing emotion information.

8. The multimodal sentiment analysis method of graph structure optimization and representation separation according to claim 1, characterized in that: The final emotion prediction result obtained includes: The text modality representation containing emotional information, the audio modality representation containing emotional information, and the visual modality representation containing emotional information are concatenated to form a fusion vector; The fusion vector is input into the first part of the prediction network, and the weights of each modality feature are learned through the neural network to obtain the weight fusion vector; The weighted fusion vector is input into the second part of the prediction network to gradually reduce the dimension of the modal information and finally obtain the emotion prediction result of the video data.

9. A multimodal sentiment analysis system with graph structure optimization and representation separation, characterized in that: include: Preprocessing module: used to process the original video data to obtain a multimodal representation carrying time series information; Inter-sample data enhancement module: used to enhance the inter-sample data of multimodal representation; Context interaction processing module: used to perform context interaction processing on the multimodal representation after data enhancement between samples to obtain context interaction enhanced multimodal representation; Emotional representation decoupling processing module: used to decouple the context interaction enhanced multimodal representation from the emotional representation to obtain a multimodal representation containing emotional information; Emotion prediction module: used to perform emotion prediction based on multimodal representations containing emotional information to obtain the final emotion prediction results.

Images