An Unsupervised Multimodal Emotion Recognition Method Based on Attention Aggregation and Cross-Modal Graph Fusion

An unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion solves the problem of modeling long-distance dependencies in multimodal emotion recognition, thereby improving the accuracy and performance of emotion recognition.

CN119128616BActive Publication Date: 2026-11-17XIAN UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410980573.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2026-11-17
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

Existing multimodal emotion recognition methods cannot effectively handle long-distance spatial dependencies in tree or graph networks, and cannot fully capture syntactic dependencies in sentences, resulting in the model not being able to fully utilize multimodal data.

Method used

An unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion is proposed. A cross-modal graph network is constructed through an unsupervised encoder, an attention aggregation Transformer module, a cross-modal graph fusion module, and an unsupervised decoder. The self-attention mechanism and the graph Transformer module are used to model long-distance dependencies.

Benefits of technology

It enhances the model's expressive power, improves the accuracy of sentiment classification and prediction, reduces the sentiment semantic gap and feature redundancy, and improves the performance of multimodal sentiment recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119128616B_ABST
    Figure CN119128616B_ABST
Patent Text Reader

Abstract

This invention discloses an unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion. Specifically, the method comprises the following steps: Step 1, acquiring audio modal features, image modal features, and text modal features of video samples; Step 2, acquiring the corresponding audio vectors, image vectors, and text vectors for the audio modal features, image modal features, and text modal features; Step 3, constructing an unsupervised multimodal emotion recognition model; Step 4, inputting the audio vectors, image vectors, and text vectors into the unsupervised multimodal emotion recognition model for training, obtaining the model's optimal output as the final feature vector for prediction and classification; Step 5, using the trained unsupervised multimodal emotion recognition model for emotion recognition. This invention addresses the problem that existing technologies cannot handle long-distance spatial dependencies in tree or graph networks, are insufficient to capture syntactic dependencies in sentences, and thus prevent models from fully utilizing multimodal data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the technical field of multimodal emotion recognition methods, and relates to an unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion. Background Technology

[0002] With the rapid advancement of artificial intelligence and human-computer interaction technologies, numerous studies have focused on enhancing the intelligence of human-computer interaction, with emotion recognition becoming a key aspect. Constructing complex intelligent human-computer interaction models often involves multiple modalities; therefore, multimodal emotion recognition is an important research direction. Due to the diversity of human emotions, emotions can be conveyed through information from multiple modalities, such as natural language, facial expressions, gestures, and acoustic behavior, thereby comprehensively improving the accuracy of emotion recognition. Compared to single-modal emotion recognition, multimodal emotion recognition methods often achieve greater effectiveness by integrating information from multiple modalities.

[0003] Multimodal emotion recognition faces various challenges, one of which is how to achieve effective multimodal fusion. In the early stages of fusion, most methods utilize a concatenative approach to fuse multimodal information, combining features from different modalities into a single overall feature set. While this strategy of fusing different modalities through concatenation may help improve emotion recognition performance, it only demonstrates that fusing visual and audio features can improve performance; the mechanisms by which these two modalities contribute to the performance improvement of multimodal emotion recognition remain to be elucidated. It is well known that the visual modality contains a limited number of elements reflecting emotional changes, and the audio modality only characterizes emotional states through pitch, tempo, and volume, making it difficult to directly obtain its emotional semantic information. Compared to the visual and audio modalities, the text modality possesses richer and more direct emotional semantic information, inevitably leading to a certain degree of emotional semantic gap between these heterogeneous modalities. Furthermore, due to the differences in semantic features and complex relational dependencies between different modalities, the fusion is not sufficient. Existing methods can only model time-series relationships in the form of sequences, cannot handle long-distance spatial dependencies, and are even less able to capture syntactic dependencies in sentences, resulting in the model not being able to fully utilize multimodal data. Summary of the Invention

[0004] The purpose of this invention is to provide an unsupervised multimodal emotion recognition method that combines attention aggregation and cross-modal graph fusion. This method solves the problem in existing technologies that cannot handle long-distance spatial dependencies in tree or graph networks, are insufficient to capture syntactic dependencies in sentences, and thus prevent the model from fully utilizing multimodal data.

[0005] The technical solution adopted in this invention is an unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion, which is implemented according to the following steps:

[0006] Step 1, obtain the audio modal features I of the video samples. a Image modal features I v and text modal features I t ;

[0007] Step 2, Obtain audio modal features I a Image modal features I v and text modal features I t The corresponding audio vector F a Image vector F v Text vector F t ;

[0008] Step 3: Construct an unsupervised multimodal emotion recognition model;

[0009] Step 4, convert the audio vector F a Image vector F v Text vector F t The input is fed into an unsupervised multimodal emotion recognition model for training, and the model's optimal output, pred, is obtained. i As the final feature vector, it is used for prediction and classification;

[0010] Step 5: Perform emotion recognition using the trained unsupervised multimodal emotion recognition model.

[0011] The invention is further characterized in that:

[0012] Step 1 is as follows:

[0013] Multiple video samples are extracted from the video database to obtain the corresponding multimodal labels y for the video samples. m Audio modal features (I) were extracted from each video sample using a multimodal software development kit. a Image modal features I v and text modal features I t .

[0014] Step 2 is as follows:

[0015] Text modal features I t The pre-trained BERT model is used to encode sentences to obtain text vectors F. t Specifically:

[0016]

[0017] in, This represents the parameters of the pre-trained BERT model, where R represents the matrix space and d... t This represents the dimension of the text vector.

[0018] For audio modal features I a and image modal features I v Using an independent unidirectional long short-term memory network to learn in the time dimension, audio vectors F are obtained respectively. a Image vector F v Specifically:

[0019]

[0020] Specifically, an A-sLSTM network was used to extract sequence features from the audio modality using a long short-term memory network, while a V-sLSTM network was used to extract sequence features from the visual modality using a long short-term memory network. and F represents the parameters of the hidden layers of the A-sLSTM and V-sLSTM networks, respectively. a and F v Let represent the hidden layer output sequences of the audio modality and the visual modality, respectively, i.e., the feature representations of a single modality, and let represent the audio vector and the image vector, respectively. R represents the matrix space, and d a and d v These represent the dimensions of the audio vector and the image vector, respectively.

[0021] The unsupervised multimodal emotion recognition model in step 3 includes an unsupervised encoder module, an attention aggregation Transformer module, a cross-modal graph fusion module, and an unsupervised decoder module connected in sequence.

[0022] The unsupervised encoder module includes three parallel encoders: an audio encoder, a text encoder, and an image encoder. The three encoders have the same structure, each consisting of two convolutional layers with a stride of 2 connected in sequence, a self-attention module, a convolutional layer with a stride of 1, and a ReLU activation function layer. The output of the first convolutional layer is also directly connected to the input of the ReLU activation function layer through a residual network. The outputs of the ReLU activation function layers of the three encoders serve as the input to the attention aggregation Transformer module.

[0023] The attention aggregation Transformer module includes two parallel Transformer encoders. Each Transformer encoder includes six Transformer layers connected in sequence. Each Transformer layer includes a normalization layer, two multi-head self-attention mechanism modules, and a multilayer perceptron module connected in sequence. The input of the normalization layer of the first Transformer layer in each Transformer encoder is connected to the output of the unsupervised encoder module. The output of the multilayer perceptron module in the last Transformer layer in each Transformer encoder is connected to the cross-modal graph fusion module.

[0024] The cross-modal graph fusion module includes a cross-modal graph construction module and a graph Transformer module connected in sequence. The output of the attention aggregation Transformer module is used as the input of the cross-modal graph construction module, and the output of the cross-modal graph construction module is used as the input of the graph Transformer module. The graph Transformer module includes three linear layers, a scaling dot product attention mechanism layer, a multiplication operation processing layer, a normalization layer, and a ReLU activation function layer connected in sequence. The features of the central node are connected through a gated residual connection and the output is added to the ReLU activation function layer to finally obtain the optimal prediction result.

[0025] The unsupervised decoder module consists of three parallel decoders with identical structures: an audio decoder, a text decoder, and an image decoder. The three decoders use deconvolution operations with learnable parameters to achieve upsampling.

[0026] Step 4 specifically involves:

[0027] Step 4.1, convert the audio vector F a Image vector F v Text vector F t The inputs are respectively fed into the audio encoder, image encoder, and text encoder of the unsupervised encoder module to obtain the audio modality feature vector I. a' ={I a1 ,I a2 ,…,I an Text modal feature vector I t' ={I t1 ,I t2 ,…,I tn Image modal feature vector I v' ={I v1 ,I v2 ,…,I vn};

[0028] Step 4.2, convert the audio modal feature vector I a' Text modal feature vector I t' Image modal feature vector I v' The input attention aggregation Transformer module yields the text-to-audio aggregation enhanced feature vector F. t→a The text's visually enhanced feature vector F t→v ;

[0029] Step 4.3, F t→a F t F t→vThe cross-modal graph is constructed by inputting three feature vectors. in, E represents the nodes in the cross-modal graph, E represents the edges in the cross-modal graph, and the node features H = {h1, h2, ..., h} are extracted from the cross-modal graph. n},h1,h2,...,h n Represents n nodes;

[0030] Step 4.4, generate the cross-modal graph The input graph is processed by the Transformer module, which performs multimodal feature fusion to obtain the fused feature o. i ;

[0031] Step 4.5, o i The inputs are sent to the audio decoder, text decoder, and image decoder respectively to reconstruct the audio, text, and image modal features.

[0032] Step 4.6: During training, a loss function is used for optimization to minimize the reconstruction error. This means ensuring that the audio, text, and image modal features reconstructed by the audio decoder, text decoder, and image decoder are consistent with the original audio vector F input in Step 4.1. a Image vector F v Text vector F t Minimize error.

[0033] Step 4.2 specifically involves:

[0034] The audio modal feature vector I a' and text modal feature vector I t' They are grouped together, and blocks h are aggregated in the middle by sequence alignment. f The input attention aggregation Transformer module is used to obtain the text-to-audio aggregation enhanced feature vector F. t→a ; to transform the image modal feature vector I v' and text modal feature vector I t' They are grouped together, and blocks h are aggregated in the middle by sequence alignment. f The input attention aggregation Transformer module is used to obtain the visually enhanced feature vector F of the text pair in another Transformer encoder. t→v Specifically:

[0035] When inputting into the Transformer encoder, first process the audio modal feature vector I... a' Text modal feature vector I t' Image modal feature vector I v' Compress it to make it compatible with the aggregate block hf The sequence lengths are consistent, and then the audio modal feature vector I is... a' Aggregate block h f Text modal feature vector I t' The input is fed into a Transformer encoder, which then produces a cross-modal text-to-audio aggregation enhanced feature vector F. t→a ; to transform the image modal feature vector I v' Aggregate block h f Text modal feature vector I t' The input is fed into a Transformer encoder, which then produces a cross-modal text-pair visual aggregation enhancement feature vector F. t→v ;

[0036] in, That is, h f Depend on It consists of B vectors, where B is a random number.

[0037] In step 4.3, when constructing the cross-modal graph, for the interior of the modal feature vector, for F... t→a and F t→v Construct temporally adjacent connection edges based on temporal relationships, for F t The process involves constructing connection edges based on a syntactic dependency tree. For modal feature vectors, a feature similarity threshold is used to construct cross-modal edges between any two modal feature vectors. The higher the correlation between two nodes, the greater their similarity. Intra-modal relationships are constructed through internal edges of modal feature vectors, and relationships between different modalities are constructed through cross-modal edges between modal feature vectors, resulting in the final cross-modal graph. And extract the node features H = {h1, h2, ..., h} from the cross-modal graph. n The node features include a central node and neighboring nodes. Each node can serve as a central node, and the nodes connected to the central node are its neighboring nodes.

[0038] When constructing cross-modal edges between any two modal feature vectors using a feature similarity threshold, the similarity s between the two feature vectors is calculated by pre-setting a threshold ε. Edge connection is only possible when the similarity is greater than the threshold, i.e., s > ε. The similarity calculation formula is as follows:

[0039] s = CosSim(x,y) (9)

[0040] Here, x and y represent two arbitrary feature vectors.

[0041] Step 4.4 specifically involves:

[0042] The features of the central node are used as input and connected through a gated residual network to calculate the fractional output β of the original residual representation. i Simultaneously, the features of the central node are used as input and passed through a Linear layer to output a value vector v. c,a The features of the neighboring nodes corresponding to the features of the central node are used as inputs and passed through two other Linear layers to obtain the query vector q. c,i Key vector k c,j And each edge e from node j to node i ij The edge vector e in multi-head attention c,ij Then query vector q c,i Key vector k c,j Edge vector e c,ij The multi-head attention score α is obtained by outputting the scaled dot product attention mechanism layer. c,ij Then, the attention score α c,ij The tail node features are updated using attention-based multi-head message aggregation as input, and the output is a new vector representation of multi-head attention. Then, the attention score weights W will be used. f With the new vector representation The multiplication operation yields the output vector Z. i Finally, Z i The total output vector o, updated by the graph Transformer module, is obtained by summing the output vector after passing through the normalization layer, the ReLU activation function layer, and the residual network. i .

[0043] Neighboring node feature h i The query vector q of the mapping c,i Neighboring node features h j The key vector k of the mapping c,j , central node feature h a The mapped value vector v c,a The edge vector e from neighboring node j to node i c,ij and multi-head attention score α c,ij The specific calculation is performed according to the following formula:

[0044] q c,i =W c,q h i +b c,q (12)

[0045] k c,j =W c,k h j +b c,k (13)

[0046] e c,ij =W c,ee ij +b c,e (14)

[0047] v c,a =W c,v h a +b c,v (15)

[0048]

[0049] in, These represent the weight matrices of the neighboring node feature vectors after passing through the C-th head attention of two Linear layers, respectively; This represents the weight matrix of the feature vector of the center node after passing through the C-th head attention of another Linear layer. Let represent the weight matrix of the C-th head attention from node j to node i. These represent the bias vectors of the neighboring node feature vectors after passing through the C-th head attention of two Linear layers. h represents the bias vector of the central node feature vector after passing through the C-th head attention of the Linear layer. i ,h j Let e ​​represent the feature vectors of neighboring nodes i and j, respectively. i,j Let h represent the edge vector from neighboring node j to node i. a The eigenvector of the central node a The edge vector representing the C-th head attention. Let < > denote the edge vector of the C-th head attention neighborhood, and let < > denote the exponentially scaled dot product function. Representing the neighboring node feature h i The mapped query vector, Representing the neighboring node feature h j The key vector of the mapping, Let v represent the key vector of the C-th head attention neighborhood. c,a The central node feature h a The mapped value vector, Represents the neighborhood of each attention;

[0050] New vector representation in multi-head attention The specific calculation is performed according to the following formula:

[0051]

[0052] Here, || represents the concatenation operation, used to concatenate the vectors of the C heads, where each head vector represents its neighborhood. Attention weighting.

[0053]

[0054] Among them, the fraction β represented by the original residual is calculated by the gated residual network. i Specifically:

[0055]

[0056] in, This represents the updated original residual representation. These represent the learnable weight matrix and bias vector, respectively.

[0057] r i =W r h i +b r (10)

[0058] This represents the updated original residual representation. Let β represent the learned weight matrix and bias vector, respectively. i This represents the fraction of the original residual obtained from the calculation, with β... i Used to balance the original residual representation r i With the newly obtained vector representation The attention score weights are used to transform the multi-head representation vector back into d dimensions.

[0059] In step 4.6, the loss function is the L1 loss function and L... rec Sum of loss functions:

[0060] L all =L1+L rec (26)

[0061] The L1 loss function is:

[0062]

[0063] Where N represents the number of training samples, pred i y represents the prediction result for the i-th sample. i This represents the true value of the i-th sample;

[0064] L rec The loss function is:

[0065] L rec =L a +L v +L t (20)

[0066] Among them, L a L v L t These represent the losses for the audio modality, visual modality, and text modality, respectively.

[0067]

[0068]

[0069]

[0070] Among them, E a E v E t These represent audio encoder, image encoder, and text encoder, respectively; F HG F represents the attention-driven Transformer module and cross-modal graph fusion module in the codec. G Represents the cross-modal graph fusion module in the codec; D a D v D t These represent audio decoder, image decoder, and text decoder, respectively.

[0071] The goal of unsupervised learning is achieved by minimizing the reconstruction loss, as shown in Equation (24);

[0072]

[0073] The beneficial effects of this invention are:

[0074] This invention utilizes an unsupervised encoder-decoder module to extract multimodal heterogeneous features using multiple independent encoders and decoders. It integrates a self-attention mechanism into the encoder structure, filtering out a large amount of information unrelated to emotional semantics, and reduces the heterogeneity between output features through unsupervised training. An attention-driven Transformer module uses a strong textual modality to enhance and aggregate visual and audio modalities, thereby reducing the emotional semantic gap and feature redundancy. Nodes and edges are constructed in corresponding ways for different modalities, and cross-modal edges are built using feature similarity thresholds. Combined with a designed graph Transformer-based multimodal feature learning method, long-distance dependencies in the cross-modal graph network are modeled, enabling adaptive adjustment of dependencies and enhancing the model's expressive power. In summary, this invention's attention-driven feature enhancement and cross-modal graph fusion unsupervised multimodal emotion recognition method is reasonable and effective, improving the accuracy of emotion classification (Acc2, F1 score) and emotion prediction regression (MAE, Corr), demonstrating significant performance improvements in multimodal emotion recognition tasks. Attached Figure Description

[0075] Figure 1 This is a flowchart of the unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion of the present invention;

[0076] Figure 2 This is a schematic diagram of the encoder structure in the unsupervised multimodal emotion recognition model of this invention;

[0077] Figure 3 This is a schematic diagram of the attention aggregation Transformer module in the unsupervised multimodal emotion recognition model of this invention;

[0078] Figure 4 A schematic diagram illustrating the principle of cross-modal graph construction in the unsupervised multimodal emotion recognition model of this invention;

[0079] Figure 5 The diagram below shows the structure of the Transformer module in the unsupervised multimodal emotion recognition model of this invention. Detailed Implementation

[0080] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0081] Example 1

[0082] The present invention provides an unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion, the process of which is as follows: Figure 1 As shown, the specific steps are as follows:

[0083] Step 1, obtain the audio modal features I of the video samples. a Image modal features I v and text modal features I t Specifically, this involves extracting multiple video samples from a video database and obtaining the corresponding multimodal labels y for the video samples. m Audio modal features (I) were extracted from each video sample using a multimodal software development kit. a Image modal features I v and text modal features I t .

[0084] Step 2, Obtain audio modal features I a Image modal features I v and text modal features I t The corresponding audio vector F a Image vector F v Text vector F t Specifically:

[0085] Text modal features I t The pre-trained BERT model is used to encode sentences to obtain text vectors F. t Specifically:

[0086]

[0087] in, This represents the parameters of the pre-trained BERT model, where R represents the matrix space and d... t This represents the dimension of the text vector.

[0088] For audio modal features I a and image modal features I v Using an independent unidirectional long short-term memory network to learn in the time dimension, audio vectors F are obtained respectively. a Image vector F v Specifically:

[0089]

[0090] Specifically, an A-sLSTM network was used to extract sequence features from the audio modality using a long short-term memory network, while a V-sLSTM network was used to extract sequence features from the visual modality using a long short-term memory network. and F represents the parameters of the hidden layers of the A-sLSTM and V-sLSTM networks, respectively. a and F v Let represent the hidden layer output sequences of the audio modality and the visual modality, respectively, i.e., the feature representations of a single modality, and let represent the audio vector and the image vector, respectively. R represents the matrix space, and d a and d v These represent the dimensions of the audio vector and the image vector, respectively.

[0091] Step 3: Construct an unsupervised multimodal emotion recognition model;

[0092] The unsupervised multimodal emotion recognition model consists of an unsupervised encoder module, an attention aggregation Transformer module, a cross-modal graph fusion module, and an unsupervised decoder module connected in sequence.

[0093] The unsupervised encoder module includes three parallel encoders: an audio encoder, a text encoder, and an image encoder. All three encoders have the same structure, as shown below. Figure 2 As shown, each module includes two convolutional layers with a stride of 2 connected in sequence, a self-attention module, a convolutional layer with a stride of 1, and a ReLU activation function layer. The output of the first convolutional layer is also directly connected to the input of the ReLU activation function layer through a residual network direct connection edge. The outputs of the ReLU activation function layers of the three encoders serve as the inputs of the attention aggregation Transformer module.

[0094] like Figure 3As shown, the attention aggregation Transformer module includes two parallel Transformer encoders. Each Transformer encoder includes six Transformer layers connected in sequence. Each Transformer layer includes a normalization layer, two multi-head self-attention mechanism modules, and a multilayer perceptron module connected in sequence. The input of the normalization layer of the first Transformer layer in each Transformer encoder is connected to the output of the unsupervised encoder module. The output of the multilayer perceptron module in the last Transformer layer in each Transformer encoder is connected to the cross-modal graph fusion module.

[0095] The cross-modal graph fusion module includes a cross-modal graph construction module and a graph Transformer module connected in sequence. The output of the attention aggregation Transformer module serves as the input of the cross-modal graph construction module, and the output of the cross-modal graph construction module serves as the input of the graph Transformer module, such as... Figure 5 As shown, the Transformer module consists of three linear layers connected in sequence, a scaling dot product attention mechanism layer, a multiplication operation processing layer, a normalization layer, and a ReLU activation function layer. The features of the central node are connected through a gated residual connection and the output is added to the ReLU activation function layer to finally obtain the optimal prediction result.

[0096] The unsupervised decoder module includes three parallel decoders with identical structures: an audio decoder, a text decoder, and an image decoder. The three decoders use deconvolution operations with learnable parameters to achieve upsampling.

[0097] Step 4, convert the audio vector F a Image vector F v Text vector F t The input is fed into an unsupervised multimodal emotion recognition model for training, and the model's optimal output, pred, is obtained. i The final feature vector is used for prediction and classification; specifically:

[0098] Step 4.1, convert the audio vector F a Image vector F v Text vector F t The inputs are respectively fed into the audio encoder, image encoder, and text encoder of the unsupervised encoder module to obtain the audio modality feature vector I. a' ={I a1 ,I a2 ,…,I an Text modal feature vector I t' ={It1 ,I t2 ,…,I tn Image modal feature vector I v' ={I v1 ,I v2 ,…,I vn};

[0099] The formula for calculating the attention matrix output by the self-attention module in the audio encoder, image encoder, and text encoder is shown in formula (4):

[0100]

[0101] Where Q is the query vector, K is the key vector, V is the value vector, T is the transpose, and d is the key vector. k It is the dimension of the key vector K;

[0102] Step 4.2, convert the audio modal feature vector I a' Text modal feature vector I t' Image modal feature vector I v' The input attention aggregation Transformer module yields the text-to-audio aggregation enhanced feature vector F. t→a The text's visually enhanced feature vector F t→v Specifically:

[0103] The audio modal feature vector I a' and text modal feature vector I t' They are grouped together, and blocks h are aggregated in the middle by sequence alignment. f The input attention aggregation Transformer module is used to obtain the text-to-audio aggregation enhanced feature vector F. t→a ; to transform the image modal feature vector I v' and text modal feature vector I t' They are grouped together, and blocks h are aggregated in the middle by sequence alignment. f The input attention aggregation Transformer module is used to obtain the visually enhanced feature vector F of the text pair in another Transformer encoder. t→v Specifically:

[0104] When inputting into the Transformer encoder, first process the audio modal feature vector I... a' Text modal feature vector I t' Image modal feature vector I v' Compress it to make it compatible with the aggregate block h f The sequence lengths are consistent, and then the audio modal feature vector I is... a' Aggregate block hf Text modal feature vector I t' The input is fed into a Transformer encoder, which then produces a cross-modal text-to-audio aggregation enhanced feature vector F. t→a ; to transform the image modal feature vector I v' Aggregate block h f Text modal feature vector I t' The input is fed into a Transformer encoder, which then produces a cross-modal text-pair visual aggregation enhancement feature vector F. t→v ;

[0105] in, That is, h f Depend on It consists of B vectors, where B is a random number;

[0106] In any Transformer encoder, the feature vector input to the l-th Transformer layer... Updated after using the Transformer encoder The update process is shown in formulas (5)-(6).

[0107] For the l-th layer, any feature vector The input is used to drive the layer normalization (LN) module, multi-head self-attention (MSA) mechanism, and multilayer perceptron (MLP) module in the Transformer module through attention aggregation to update the Transformer layers, and the output is the feature vector of the next layer. The update process is shown in formulas (5)-(6):

[0108]

[0109] in, This represents the feature vector input to the l-th Transformer layer. The query vector, key vector, and value vector in the Transformer are all derived from... Linear projection; y l It is the feature vector output by the l-th Transformer layer;

[0110] For the l-th layer, the aggregation enhancement of the text modality to the audio modality is shown in equations (7)-(8):

[0111]

[0112] in, The text modal feature vector input to the l-th Transformer layer is... This is the text modal feature vector output by the (l+1)th Transformer layer. The aggregate block input to the l-th Transformer layer. The aggregate block is the input to the (l+1)th Transformer layer. The audio modal feature vector input to the l-th Transformer layer is... Let θ be the audio modal feature vector output by the l-th Transformer layer. a The input is the parameters of the l-th Transformer layer, θ. t The input is the parameters of the l-th Transformer layer, which is the text feature vector. This is the aggregate block output by the (l+1)th Transformer layer;

[0113] The principles of text modality aggregation enhancement on image modality and text modality aggregation enhancement on audio modality are the same.

[0114] Step 4.3, F t→a F t F t→v The cross-modal graph is constructed by inputting three feature vectors. in, E represents the nodes in the cross-modal graph, E represents the edges in the cross-modal graph, and the node features H = {h1, h2, ..., h} are extracted from the cross-modal graph. n},h1,h2,...,h n Represents n nodes; such as Figure 4 As shown, when constructing a cross-modal graph, for the interior of the modal feature vector, for F... t→a and F t→v Construct temporally adjacent connection edges based on temporal relationships, for F t The process involves constructing connection edges based on a syntactic dependency tree. For modal feature vectors, a feature similarity threshold is used to construct cross-modal edges between any two modal feature vectors. The higher the correlation between two nodes, the greater their similarity. Intra-modal relationships are constructed through internal edges of modal feature vectors, and relationships between different modalities are constructed through cross-modal edges between modal feature vectors, resulting in the final cross-modal graph. And extract the node features H = {h1, h2, ..., h} from the cross-modal graph. n The node features include a central node and neighboring nodes. Each node can serve as a central node, and the nodes connected to the central node are its neighboring nodes.

[0115] When constructing cross-modal edges between any two modal feature vectors using a feature similarity threshold, the similarity s between the two feature vectors is calculated by pre-setting a threshold ε. Edge connection is only possible when the similarity is greater than the threshold, i.e., s > ε. The similarity calculation formula is as follows:

[0116] s = CosSim(x,y) (9)

[0117] Where x and y represent two arbitrary feature vectors;

[0118] Step 4.4, generate the cross-modal graph The input graph is processed by the Transformer module, which performs multimodal feature fusion to obtain the fused feature o. i Specifically:

[0119] The features of the central node are used as input and connected through a gated residual network to calculate the fractional output β of the original residual representation. i Simultaneously, the features of the central node are used as input and passed through a Linear layer to output a value vector v. c,a The features of the neighboring nodes corresponding to the features of the central node are used as inputs and passed through two other Linear layers to obtain the query vector q. c,i Key vector k c,j And each edge e from node j to node i ij The edge vector e in multi-head attention c,ij Then query vector q c,i Key vector k c,j Edge vector e c,ij The multi-head attention score α is obtained by outputting the scaled dot product attention mechanism layer. c,ij Then, the attention score α c,ij The tail node features are updated using attention-based multi-head message aggregation as input, and the output is a new vector representation of multi-head attention. Then, the attention score weights W will be used. f With the new vector representation The multiplication operation yields the output vector Z. i Finally, Z i The total output vector o, updated by the graph Transformer module, is obtained by summing the output vector after passing through the normalization layer, the ReLU activation function layer, and the residual network. i .

[0120] Neighboring node feature h i The query vector q of the mapping c,i Neighboring node features h j The key vector k of the mapping c,j , central node feature h a The mapped value vector vc,a The edge vector e from neighboring node j to node i c,ij and multi-head attention score α c,ij The specific calculation is performed according to the following formula:

[0121] q c,i =W c,q h i +b c,q (12)

[0122] k c,j =W c,k h j +b c,k (13)

[0123] e c,ij =W c,e e ij +b c,e (14)

[0124] v c,a =W c,v h a +b c,v (15)

[0125]

[0126] in, These represent the weight matrices of the neighboring node feature vectors after passing through the C-th head attention of two Linear layers, respectively; This represents the weight matrix of the feature vector of the center node after passing through the C-th head attention of another Linear layer. Let represent the weight matrix of the C-th head attention from node j to node i. These represent the bias vectors of the neighboring node feature vectors after passing through the C-th head attention of two Linear layers. h represents the bias vector of the central node feature vector after passing through the C-th head attention of the Linear layer. i ,h j Let e ​​represent the feature vectors of neighboring nodes i and j, respectively. i,j Let h represent the edge vector from neighboring node j to node i. a The eigenvector of the central node a The edge vector representing the C-th head attention. Let < > denote the edge vector of the C-th head attention neighborhood, and let < > denote the exponentially scaled dot product function. Representing the neighboring node feature h i The mapped query vector, Representing the neighboring node feature h j The key vector of the mapping, Let v represent the key vector of the C-th head attention neighborhood. c,a The central node feature h a The mapped value vector, Represents the neighborhood of each attention;

[0127] New vector representation in multi-head attention The specific calculation is performed according to the following formula:

[0128]

[0129] Here, || represents the concatenation operation, used to concatenate the vectors of the C heads, where each head vector represents its neighborhood. Attention weighting.

[0130]

[0131] Among them, the fraction β represented by the original residual is calculated by the gated residual network. i Specifically:

[0132]

[0133] in, This represents the updated original residual representation. These represent the learnable weight matrix and bias vector, respectively.

[0134] r i =W r h i +b r (10)

[0135] This represents the updated original residual representation. Let β represent the learned weight matrix and bias vector, respectively. i This represents the fraction of the original residual obtained from the calculation, with β... i Used to balance the original residual representation r i With the newly obtained vector representation The attention score weights are used to transform the multi-head representation vector back into d dimensions.

[0136] Step 4.5, o i The inputs are sent to the audio decoder, text decoder, and image decoder respectively to reconstruct the audio, text, and image modal features.

[0137] Step 4.6: During training, a loss function is used for optimization to minimize the reconstruction error. This means ensuring that the audio, text, and image modal features reconstructed by the audio decoder, text decoder, and image decoder are consistent with the original audio vector F input in Step 4.1. a Image vector Fv Text vector F t Error minimization, where the loss function L all For L1 loss function and L rec Sum of loss functions:

[0138] L all =L1+L rec (26)

[0139] The L1 loss function is:

[0140]

[0141] Where N represents the number of training samples, pred i y represents the prediction result for the i-th sample. i This represents the true value of the i-th sample;

[0142] L rec The loss function is:

[0143] L rec =L a +L v +L t (20)

[0144] Among them, L a L v L t These represent the losses for the audio modality, visual modality, and text modality, respectively.

[0145]

[0146]

[0147]

[0148] Among them, E a E v E t These represent audio encoder, image encoder, and text encoder, respectively; F HG F represents the attention-driven Transformer module and cross-modal graph fusion module in the codec. G Represents the cross-modal graph fusion module in the codec; D a D v D t These represent audio decoder, image decoder, and text decoder, respectively.

[0149] The goal of unsupervised learning is achieved by minimizing the reconstruction loss, as shown in Equation (24);

[0150]

[0151] Step 5: Perform emotion recognition using the trained unsupervised multimodal emotion recognition model.

[0152] Example 2

[0153] Building upon Example 1, this invention trains the model using a video database comprising three multimodal sentiment analysis datasets: CMU-MOSEI, CMU-MOSEI, and CH-SIMS. These three datasets are used to train the model established in Example 1. The model is designed and implemented within the PyTorch deep learning framework. All experiments are conducted on a server configured with an Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz and a GeForce RTX 3080 GPU. The initial learning rate for the BERT model is set to 0.00005, while the learning rate for other models is also set to 0.0005. Gradient truncation is performed when the gradient value exceeds 0.8. The hyperparameter settings for these three datasets are shown in Table 1.

[0154] Table 1. Hyperparameter settings of the model

[0155] hyperparameters CMU-MOSI CMU-MOSEI CMU-SIMS Batchsize 32 64 32 Number of Epochs 30 30 30 A-LSTMDropoutRate 0.0 0.0 0.0 V-LSTM Dropout Rate 0.0 0.0 0.0 BERTDropoutRate 0.2 0.3 0.2 V-LSTM hidden layers 64 32 64 Number of hidden layers in A-LSTM 32 32 16 Optimizer Adam Adam Adam Number of layers in a graph network 2 2 2

[0156] To evaluate the performance of the proposed model, five evaluation metrics were used to measure its performance. Table 3 presents the experimental results of the model on the CMU-MOSI dataset. Regarding the data setup, this invention uses an unaligned data setup. Data marked with a dot (·) are experimental results obtained by reproducing the publicly available code in the original paper; data without a dot are directly from experimental data in relevant literature. Bold text indicates that the model's metric is optimal, and underlined text indicates that the metric is suboptimal. Works marked (B) all used BERT as the pre-trained model for the text modality. All works marked with the symbol used the attention aggregation-driven Transformer module, while those marked with the ⊙ symbol used the cross-modal graph fusion module.

[0157] In this embodiment, to verify the effectiveness of the present invention, the accuracy (Acc2), F1 score, mean absolute error loss (MAE), Pearson correlation coefficient (Corr), and seven-class accuracy (Acc7) were calculated for both the predicted sentiment category and the original sentiment category, and then compared with existing methods. The performance comparison results of different network models on the CMU-MOSI dataset are shown in Table 2.

[0158] Table 2

[0159]

[0160]

[0161] Table 2 shows that BERT significantly improves recognition accuracy compared to methods such as word embedding. This is attributed to the excellent dynamics of the BERT model, which can extract richer contextual information and effectively solve the problem that other methods can only learn word vector feature representations, highlighting its advantages in deep semantic extraction. Therefore, this invention selects BERT as the pre-training model for the text modality. The results in Table 2 show that the attention-driven Transformer module can fully utilize text modality information to help other modality information learn features, and its recognition performance is better than the method of directly fusing the three modalities. In addition, the results in Table 2 show that cross-modal graph fusion can not only consider the temporal correlation of the network, but also model the spatial correlation of the network.

[0162] Example 3

[0163] Building upon Example 2, for both the predicted and original sentiment categories, the accuracy (Acc2), F1 score, mean absolute error loss (MAE), Pearson correlation coefficient (Corr), and seven-category accuracy (Acc7) were calculated and then compared with existing methods. Table 3 shows the performance comparison results of different network models on the CMU-MOSEI dataset.

[0164] Table 3

[0165]

[0166]

[0167] Table 3 shows the experimental analysis conducted on the CMU-MOSEI dataset. The results indicate that the AAT-CGF model significantly outperforms other models in terms of data performance. Methods marked (B) in the table all used the BERT pre-trained model as the text feature learning method, thus exhibiting superior performance. The identification method not only extracts interaction information between multimodalities but also provides a good representation of information within a single modality. The AAT-CGF method proposed in this invention effectively represents single modalities while fully integrating multimodal features. Results show that its Acc2 and F1 scores achieve optimal performance, while its regression performance, although slightly lower than other models, remains suboptimal and significantly higher than the average. This indicates that our model's multi-task learning approach is more suitable for classification tasks, achieving higher recognition accuracy in classification.

[0168] Example 4

[0169] Building upon Example 2, for both the predicted and original sentiment categories, the accuracy (Acc2), F1 score, mean absolute error loss (MAE), Pearson correlation coefficient (Corr), and seven-category accuracy (Acc7) were calculated and then compared with existing methods. The performance comparison results of different network models on the CH-SIMS dataset are shown in Table 4.

[0170] Table 4

[0171] Model Acc2 F1 MAE Corr TFN

[2017] 79.86 80.16 0.428 0.605 MISA

[2022] 75.49 75.85 0.472 0.542 SaPIL

[2023] <![CDATA[ 81.25 ]]> <![CDATA[ 81.25 ]]> 0.423 <![CDATA[ 0.620 ]]> ConKI

[2023] 77.94 78.17 0.454 0.542 Humman-MM 80.74 80.78 <![CDATA[ 0.408 ]]> 0.647 SELF-MM

[2021] 80.74 80.78 0.419 0.616 AAT-CGF (This invention) 81.73 81.78 0.399 0.655

[0172] As shown in Table 4, experimental analysis was conducted on the Chinese sentiment analysis dataset CH-SIMS. The results show that the AAT-CGF model significantly outperforms other models in terms of data. This dataset uses an unaligned data setting. In Table 4, the model is compared with five existing multimodal sentiment recognition models based on unaligned data. The experimental results show that the model of this invention outperforms other baseline models in all four metrics. In the table, Humman-MM represents manually labeled single-modal tags, and the results show that the model of this invention achieves better recognition performance than manually labeled tags.

[0173] This invention leverages the advanced sentiment semantics of text features, employing attention aggregation blocks for cross-modal enhancement and compressing information from each modality to reduce the differences in sentiment semantics between modalities. Independent unsupervised encoders and decoders are used for different modalities, and a self-attention mechanism is incorporated into the encoder structure to filter out a large amount of information unrelated to sentiment semantics, driving the model to learn better sentiment representations. To model various dependencies between multimodalities, corresponding modal intra-modal connection edge construction methods are adopted for different modal data characteristics, and cross-modal connections are constructed based on feature similarity thresholds. A multimodal graph Transformer is used to model long-distance dependencies in the multimodal graph network. Attention operations are performed on nodes and edges to achieve adaptive adjustment of dependencies and aggregate and update node information.

Claims

1. An unsupervised multi-modal sentiment recognition method that aggregates attention and fuses cross-modal graphs, characterized in that, The specific steps are as follows: Step 1, obtaining audio modality features of the video sample , image modality features , and text modality features ; Step 2, obtain audio modality features , image modality features , and text modality features corresponding audio vector , image vector , text vector ; Step 3: Construct an unsupervised multimodal emotion recognition model. The unsupervised multimodal emotion recognition model in Step 3 includes an unsupervised encoder module, an attention aggregation Transformer module, a cross-modal graph fusion module, and an unsupervised decoder module connected in sequence. The unsupervised encoder module includes three parallel encoders: an audio encoder, a text encoder, and an image encoder. The three encoders have the same structure, each consisting of two convolutional layers with a stride of 2, a self-attention module, a convolutional layer with a stride of 1, and a ReLU activation function layer connected sequentially. The output of the first convolutional layer with a stride of 2 is directly connected to the input of the ReLU activation function layer via a residual network. The outputs of the ReLU activation function layers of the three encoders serve as the input to the attention aggregation Transformer module. The attention aggregation Transformer module includes two parallel Transformer encoders. Each Transformer encoder includes six Transformer layers connected in sequence. Each Transformer layer includes a normalization layer, two multi-head self-attention mechanism modules, and a multilayer perceptron module connected in sequence. The input of the normalization layer of the first Transformer layer in each Transformer encoder is connected to the output of the unsupervised encoder module. The output of the multilayer perceptron module in the last Transformer layer in each Transformer encoder is connected to the cross-modal graph fusion module. The cross-modal graph fusion module includes a cross-modal graph construction module and a graph Transformer module connected in sequence. The output of the attention aggregation Transformer module is used as the input of the cross-modal graph construction module, and the output of the cross-modal graph construction module is used as the input of the graph Transformer module. The graph Transformer module includes three linear layers, a scaling dot product attention mechanism layer, a multiplication operation processing layer, a normalization layer, and a ReLU activation function layer connected in sequence. The center node features are connected through a gated residual connection and the output is added to the ReLU activation function layer to finally obtain the optimal prediction result. The unsupervised decoder module includes three parallel and identical decoders: an audio decoder, a text decoder, and an image decoder. The three decoders use deconvolution operations with learnable parameters to achieve upsampling. Step 4, audio vector , image vector , text vector Input into an unsupervised multi-modal sentiment recognition model for training to obtain the optimal output of the model As the final feature vector for prediction and classification; Step 5: Perform emotion recognition using the trained unsupervised multimodal emotion recognition model.

2. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 1, characterized in that, Step 1 specifically involves: Multiple video samples are extracted from the video database to obtain corresponding multimodal labels for the video samples. Audio modal features were extracted from each video sample using a multimodal software development kit. Image modal features and text modal features .

3. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 1 or 2, characterized in that, Step 2 specifically involves: Text modal features The pre-trained BERT model is used to encode sentences to obtain text vectors. Specifically: (1) in, These represent the parameters of the pre-trained model BERT. Represents the matrix space. This represents the dimension of the text vector. For audio modal features and image modal features Using an independent unidirectional long short-term memory network to learn in the time dimension, audio vectors are obtained respectively. Image vectors Specifically: (2) (3) Among them, use The network uses a Long Short-Term Memory (LSTM) network to extract sequence features from audio modalities. The network uses a long short-term memory network to extract sequence features for visual modalities. and Represent and The parameters of the two hidden layers of the network, and These represent the hidden layer output sequences for the audio and visual modalities, respectively, i.e., the feature representations of a single modality, and represent the audio vector and image vector, respectively. Represents the matrix space. and These represent the dimensions of the audio vector and the image vector, respectively.

4. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 3, characterized in that, Step 4 specifically involves: Step 4.1, convert the audio vector Image vectors Text vectors The audio modality feature vectors are respectively input into the audio encoder, image encoder, and text encoder of the unsupervised encoder module. Text modal feature vectors Image modal feature vectors ; Step 4.2, convert the audio modal feature vector Text modal feature vectors Image modal feature vectors The input attention aggregation Transformer module yields a text-to-audio aggregation enhanced feature vector. Text-enhanced visual feature vectors ; Step 4.3, will , , The cross-modal graph is constructed by inputting three feature vectors. Where V represents a node in the cross-modal graph, Representing edges in a cross-modal graph and extracting node features from the cross-modal graph. , Represents n nodes; Step 4.4, generate the cross-modal graph The Transformer module inputs the graph and performs multimodal feature fusion to obtain fused features. ; Step 4.5, will The inputs are sent to the audio decoder, text decoder, and image decoder respectively to reconstruct the audio, text, and image modal features. Step 4.6: During training, a loss function is used for optimization to minimize the reconstruction error. This means ensuring that the audio, text, and image modal features reconstructed by the audio decoder, text decoder, and image decoder are consistent with the original audio vector input in Step 4.

1. Image vectors Text vectors Minimize error.

5. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 4, characterized in that, Step 4.2 specifically involves: Audio modality feature vector and text modal feature vectors They are grouped together and aggregated into blocks in the middle by sequence alignment. The input attention aggregation Transformer module is used to obtain a text-to-audio aggregation enhanced feature vector. ; to transform image modal feature vectors and text modal feature vectors They are grouped together and aggregated into blocks in the middle by sequence alignment. The input attention aggregation Transformer module is used to obtain the visually enhanced feature vector of the text pair in another Transformer encoder. Specifically: When inputting into the Transformer encoder, first process the audio modal feature vector. Text modal feature vectors Image modal feature vectors Compress it to make it compatible with the aggregate block. The sequence lengths are consistent, and then the audio modal feature vectors are... Aggregate block Text modal feature vectors The input is fed into a Transformer encoder, which then produces a cross-modal text-to-audio aggregation enhanced feature vector. ; to transform image modal feature vectors Aggregate block Text modal feature vectors The input is fed into a Transformer encoder, which then produces a cross-modal text-to-visual aggregation enhancement feature vector. ; in, That is to say Depend on common It consists of 1 vector, where B is a random number.

6. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 5, characterized in that, When constructing the cross-modal graph in step 4.3, for the interior of the modal feature vector, for... and Construct temporally adjacent connection edges based on temporal relationships, targeting... The process involves constructing connection edges based on a syntactic dependency tree. For modal feature vectors, a feature similarity threshold is used to construct cross-modal edges between any two modal feature vectors. The higher the correlation between two nodes, the greater their similarity. Intra-modal relationships are constructed through internal edges of modal feature vectors, and relationships between different modalities are constructed through cross-modal edges between modal feature vectors, resulting in the final cross-modal graph. And extract node features from cross-modal graphs. The node features include a central node and neighboring nodes. Each node can serve as a central node, and the nodes connected to the central node are the neighboring nodes. Specifically, when constructing cross-modal edges between any two modal feature vectors using a feature similarity threshold, the threshold is set in advance. Calculate the similarity s between two feature vectors. When the similarity is greater than a threshold, i.e. Only after these conditions are met can edges be connected. The similarity calculation formula is as follows: (9) in, and It represents two arbitrary eigenvectors.

7. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 6, characterized in that, Step 4.4 specifically involves: The features of the central node are used as input, connected by a gated residual network, to calculate the fractional output of the original residual representation. Simultaneously, the features of the central node are used as input and passed through a Linear layer to output a value vector. The query vector is obtained by taking the features of the neighboring nodes corresponding to the features of the central node as inputs and passing them through two other Linear layers. Key vector and from the node To the node Each edge Edge vectors in multi-head attention Then query vector Key vector Edge vectors The multi-head attention score is obtained by outputting the scaled dot product attention mechanism layer. Next, the attention score The tail node features are updated using attention-based multi-head message aggregation as input, and the output is a new vector representation of multi-head attention. Then, it will be weighted by the attention score. With the new vector representation Performing multiplication yields the output vector. Finally, The total output vector, updated by the graph Transformer module, is obtained by summing the output vector after passing through the normalization layer, the ReLU activation function layer, and the residual network. .

8. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 7, characterized in that, Neighboring node features Mapped query vector Neighboring node features Mapped key vector Central node characteristics Mapped value vector neighboring nodes To the node edge vectors and multi-head attention score The specific calculation is performed according to the following formula: (12) (13) (14) (15) (16) in, These represent the feature vectors of neighboring nodes after passing through two Linear layers. The weight matrix for head attention; This indicates that the feature vector of the central node has passed through another Linear layer. The weight matrix of head attention. Represents a node To the node No. The weight matrix of head attention. These represent the feature vectors of neighboring nodes after passing through two Linear layers. The bias vector for head attention. This indicates that the feature vector of the central node has passed through the Linear layer for the first time. The bias vector for head attention. Representing neighboring nodes eigenvectors, Indicates the neighboring nodes To the node edge vectors, Indicates the central node eigenvectors, Indicates the first The edge vector of head attention, Indicates the first The edge vectors of the head attention neighborhood. This represents the exponentially scaled dot product function. Representing the characteristics of neighboring nodes The mapped query vector, Representing the characteristics of neighboring nodes The key vector of the mapping, This represents the key vector of the C-th head attention neighborhood. Representing the characteristics of the central node The mapped value vector, Represents the neighborhood of each attention; New vector representation in multi-head attention The specific calculation is performed according to the following formula: (17) in, This indicates a concatenation operation, used to combine... The vectors of the heads are concatenated, and the vector of each head represents its neighborhood. Attention weighting; (18) (19) Among them, the fraction represented by the original residual is calculated by the gated residual network. Specifically: (11) (10) in, This represents the updated original residual representation. These represent the learned weight matrix and bias vector, respectively. This represents the fractional representation of the original residuals obtained from the calculation. Used to balance the original residual representation With the newly obtained vector representation , The attention score weights are used to transform the multi-head representation vector back into... dimension.

9. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 8, characterized in that, The loss function in step 4.6 is: loss function and Sum of loss functions: (26) in, The loss function is: (25) Where N represents the number of training samples. Represents the first Prediction results for each sample Represents the first The true value of each sample; The loss function is: (20) in, , , These represent the losses for the audio modality, visual modality, and text modality, respectively. (21) (22) (23) in, , , These represent audio encoder, image encoder, and text encoder, respectively. This represents the attention aggregation-driven Transformer module and the cross-modal graph fusion module in the codec. This represents the cross-modal graph fusion module in the codec; , , These represent audio decoder, image decoder, and text decoder, respectively. The goal of unsupervised learning is achieved by minimizing the reconstruction loss, as shown in Equation (24); (24)。

Citation Information

Patent Citations

  • Multi-mode depressive emotion recognition method and device

    CN115641543A

  • Multi-modal emotion recognition method combining time attention and single-modal label automatic generation

    CN118094458A