Unsupervised multi-modal emotion recognition method based on attention aggregation and cross-modal graph fusion

Abstract

The invention discloses an attention aggregation and cross-modal graph fused unsupervised multi-modal emotion recognition method. The method specifically comprises the steps of 1, acquiring audio modal features, image modal features and text modal features of a video sample; step 2, acquiring an audio vector, an image vector and a text vector corresponding to the audio modal feature, the image modal feature and the text modal feature; step 3, constructing an unsupervised multi-modal emotion recognition model; 4, inputting the audio vector, the image vector and the text vector into an unsupervised multi-modal emotion recognition model for training, and obtaining the optimal output of the model as a final feature vector for prediction and classification; and 5, performing emotion recognition by using the trained unsupervised multi-modal emotion recognition model. The problem that in the prior art, the long-distance spatial dependency relationship of a tree or graph network cannot be processed, the syntactic dependency relationship in sentences cannot be captured, and consequently the model cannot make full use of multi-modal data is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal emotion recognition methods, and relates to an unsupervised multimodal emotion recognition method combining attention aggregation and cross-modal graph fusion. Background Art

[0002] With the rapid progress of artificial intelligence and human-computer interaction technology, many studies have focused on enhancing the intelligence of human-computer interaction, among which emotion recognition has become the key to intelligent human-computer interaction. When building a complex intelligent human-computer interaction model, it is often necessary to involve multiple modalities, so multimodal emotion recognition is an important research direction. Due to the diversity of human emotions, emotions can be conveyed through information in multiple modalities such as natural language, facial expressions, gestures, acoustic behavior, etc., thereby comprehensively improving the accuracy of emotion recognition. Compared with single-modal emotion recognition, multimodal emotion recognition methods often make recognition more effective by integrating information from multiple modalities.

[0003] Multimodal emotion recognition faces various challenges, one of which is how to achieve effective multimodal fusion. In the early stages of fusion, most methods use a serial method to fuse multimodal information, combining features from different modalities into a single feature. Although the strategy of fusing different modal information in a serial manner may help improve the performance of emotion recognition, they only prove that fusing visual and audio features can improve the performance of emotion recognition, while the contribution mechanism of these two modal features to the improvement of multimodal emotion recognition performance remains to be elucidated. As we all know, the visual modality has limited elements that reflect emotional changes, and the audio modality only describes the emotional state through pitch, speed, volume, etc., and it is difficult to directly obtain its emotional semantic information. Compared with the visual modality and the audio modality, the emotional semantic information of the text modality is richer and more direct, which leads to a certain degree of emotional semantic gap between these heterogeneous modalities. In addition, due to the differences in semantic features and complex relational dependencies between different modalities, the fusion is not sufficient. Existing methods can only model time series relationships in sequence form, cannot handle long-distance spatial dependencies, and are not enough to capture syntactic dependencies in sentences, resulting in the model being unable to fully utilize multimodal data. Summary of the invention

[0004] The purpose of the present invention is to provide an unsupervised multimodal emotion recognition method of attention aggregation and cross-modal graph fusion, which solves the problem that the prior art cannot handle the long-distance spatial dependencies of tree or graph networks, is insufficient to capture the syntactic dependencies in sentences, and causes the model to be unable to fully utilize multimodal data.

[0005] The technical solution adopted by the present invention is an unsupervised multimodal emotion recognition method combining attention aggregation and cross-modal graph fusion, which is specifically implemented in the following steps:

[0006] Step 1: Get the audio modality feature I of the video sample a , Image modality features I v and text modality features I t ;

[0007] Step 2: Obtain audio modal feature I a , Image modality features I v and text modality features I t The corresponding audio vector F a , image vector F v , text vector F t ;

[0008] Step 3, construct an unsupervised multimodal emotion recognition model;

[0009] Step 4: transform the audio vector F a , image vector F v , text vector F t Input into the unsupervised multimodal emotion recognition model for training to obtain the optimal output pred of the model i As the final feature vector for prediction and classification;

[0010] Step 5: Use the trained unsupervised multimodal emotion recognition model to perform emotion recognition.

[0011] The present invention is also characterized in that:

[0012] Step 1 is as follows:

[0013] Extract multiple video samples from the video database and obtain the corresponding multimodal labels y for the video samples m , the audio modality features I are extracted from each video sample using the multimodal software development toolkit a , Image modality features I v and text modality features I t .

[0014] Step 2 is as follows:

[0015] Text modality features I t , use the pre-trained model BERT to encode the sentence to obtain the text vector F t , specifically:

[0016]

[0017] in, represents the parameters of the pre-trained model BERT, R represents the matrix space, and d t Represents the dimension corresponding to the text vector;

[0018] For audio modal feature I a and image modality features I v , use an independent unidirectional long short-term memory network to learn in the time dimension and obtain the audio vector F a , image vector F v , specifically:

[0019]

[0020] Among them, the A-sLSTM network is used to extract sequence features using the long short-term memory network for the audio modality, and the V-sLSTM network is used to extract sequence features using the long short-term memory network for the visual modality. and Represent the parameters of the hidden layers of the A-sLSTM and V-sLSTM networks, respectively. a and F v Represent the hidden layer output sequences of audio mode and visual mode, that is, the feature representation of single mode, respectively, representing audio vector and image vector, R represents matrix space, d a and d v Represent the dimensions corresponding to the audio vector and image vector respectively.

[0021] The unsupervised multimodal emotion recognition model in step 3 includes an unsupervised encoder module, an attention aggregation Transformer module, a cross-modal graph fusion module, and an unsupervised decoder module connected in sequence;

[0022] The unsupervised encoder module includes three parallel encoders: audio encoder, text encoder, and image encoder. The three encoders have the same structure, including two convolutional layers with a stride of 2, a self-attention module, a convolutional layer with a stride of 1, and a ReLU activation function layer. The output of the first convolutional layer is also directly connected to the input of the ReLU activation function layer through a residual network direct edge. The output of the ReLU activation function layer of the three encoders is used as the input of the attention aggregation Transformer module.

[0023] The attention aggregation Transformer module includes two parallel Transformer encoders, each of which includes six Transformer layers connected in sequence, each Transformer layer includes a normalization layer connected in sequence, two multi-head self-attention mechanism modules, and a multi-layer perceptron module. The input of the normalization layer of the first Transformer layer in each Transformer encoder is connected to the output of the unsupervised encoder module, and the output of the multi-layer perceptron module in the last Transformer layer in each Transformer encoder is connected to the cross-modal graph fusion module.

[0024] The cross-modal graph fusion module includes a cross-modal graph construction module and a graph Transformer module connected in sequence. The output of the attention aggregation Transformer module is used as the input of the cross-modal graph construction module, and the output of the cross-modal graph construction module is used as the input of the graph Transformer module. The graph Transformer module includes three linear layers connected in sequence, a scaled dot product attention mechanism layer, a multiplication operation processing, a normalization layer, and a ReLU activation function layer. The central node features are output through a gated residual connection and added to the ReLU activation function layer to finally obtain the optimal prediction result;

[0025] The unsupervised decoder module includes three parallel decoders with the same structure, namely, an audio decoder, a text decoder, and an image decoder. The three decoders use deconvolution operations with learnable parameters to achieve upsampling. The unsupervised decoder module uses deconvolution operations with learnable parameters to achieve upsampling.

[0026] Step 4 is as follows:

[0027] Step 4.1: transform the audio vector F a , image vector F v , text vector F t They are respectively input into the audio encoder, image encoder and text encoder of the unsupervised encoder module to obtain the audio modality feature vectors I a' = {I a1 ,I a2 ,…,I an}、Text modality feature vector I t' = {I t1 ,I t2 ,…,I tn}、Image modality feature vector I v' = {I v1 ,I v2 ,…,I vn};

[0028] Step 4.2: transform the audio modal feature vector I a' , text modality feature vector I t' , image modality feature vector I v' Input the attention aggregation Transformer module to obtain the feature vector F of the text-to-audio aggregation enhancement t→a , the feature vector F after the text is visually enhanced t→v ;

[0029] Step 4.3, F t→a , F t , F t→vThree feature vectors are input into the cross-modal graph construction module to construct the cross-modal graph in, represents the nodes in the cross-modal graph, E represents the edges in the cross-modal graph, and extracts the node features H = {h1,h2,...,h n},h1,h2,...,h n Represents n nodes;

[0030] Step 4.4: Transform the cross-modal graph The input graph Transformer module performs multimodal feature fusion to obtain the fusion feature o i ;

[0031] Step 4.5, o i are sent to the audio decoder, text decoder, and image decoder respectively to reconstruct the input of audio, text, and image modality features;

[0032] Step 4.6: Use the loss function optimization during training to minimize the reconstruction error, that is, to make the audio, text, and image modal features reconstructed by the audio decoder, text decoder, and image decoder consistent with the original audio vector F input in step 4.1. a , image vector F v , text vector F t Minimize errors.

[0033] Step 4.2 is as follows:

[0034] The audio modal feature vector I a' and the text modality feature vector I t' into a group, and aggregate blocks h in between by sequence alignment f , input into a Transformer encoder of the attention aggregation Transformer module, and obtain the feature vector F of the text-to-audio aggregation enhancement t→a ; The image modality feature vector I v' and the text modality feature vector I t' into a group, and aggregate blocks h in between by sequence alignment f , input into another Transformer encoder of the attention aggregation Transformer module to obtain the feature vector F after the text is visually enhanced t→v , specifically:

[0035] When inputting the Transformer encoder, the audio modal feature vector I is first a' , text modality feature vector I t' , image modality feature vector I v' Compress it so that it is consistent with the aggregate block hf The sequence length is consistent, and then the audio modal feature vector I a' , aggregation block h f , text modality feature vector I t' Input into a Transformer encoder, and obtain the feature vector F of cross-modal text-to-audio aggregation enhancement through the Transformer encoder t→a ; The image modality feature vector I v' , aggregation block h f , text modality feature vector I t' Input into a Transformer encoder, and obtain the feature vector F of cross-modal text-to-visual aggregation enhancement through the Transformer encoder t→v ;

[0036] in, That is h f Depend on It consists of B vectors in total, where B is a random number.

[0037] When constructing the cross-modal graph in step 4.3, for the modal eigenvector, for F t→a and F t→v According to the time relationship, the time-adjacent connecting edges are constructed. t , build connection edges according to the syntactic dependency tree; for modal feature vectors, use the feature similarity threshold to build cross-modal edges for any two modal feature vectors. The higher the degree of association between two nodes, the greater their similarity. The internal relationship of the modality is built through the internal edges of the modal feature vectors, and the relationship between different modalities is built through the cross-modal edges between the modal feature vectors. The final cross-modal graph is obtained. And extract the node features H = {h1,h2,...,h n}, the node features include central nodes and neighboring nodes. Each node can be used as a central node, and the nodes connected to the central node are neighboring nodes;

[0038] Among them, when constructing cross-modal edges for any two modal feature vectors using the feature similarity threshold, the threshold ε is set in advance to calculate the similarity s of the two feature vectors. When the similarity is greater than the threshold, that is, s>ε, the edge can be connected. The similarity calculation formula is as follows:

[0039] s=CosSim(x,y) (9)

[0040] Here, x and y represent two arbitrary eigenvectors.

[0041] Step 4.4 is as follows:

[0042] The central node features are taken as input and connected through the gated residual network to calculate the score output β of the original residual representation i At the same time, the central node feature is used as input and output through a Linear layer to obtain the value vector v c,a , the neighboring node features corresponding to the central node features are taken as input and output through the other two Linear layers to obtain the query vector q c,i , key vector k c,j and each edge e from node j to node i ij The edge vector e in the multi-head attention c,ij , and then the query vector q c,i , key vector k c,j , edge vector e c,ij The multi-head attention score α is obtained by scaling the dot product attention mechanism layer output c,ij , and then the attention score α c,ij As input, the attention-based multi-head message aggregation operation is used to update the tail node features, and the new vector representation of multi-head attention is obtained as output. Then, the attention score weight W f With the new vector representation Perform multiplication to get the output vector Z i Finally, Z i The total output vector o after updating by the graph Transformer module is obtained by adding the output vectors of the normalization layer, the ReLU activation function layer and the residual network. i .

[0043] Neighboring node feature h i The mapped query vector q c,i , neighboring node features h j The key vector k of the mapping c,j , central node feature h a The mapped value vector v c,a , the edge vector e from neighboring node j to node i c,ij , and the multi-head attention score α c,ij The specific calculation is based on the following formula:

[0044] q c,i =W c,q h i +b c,q (12)

[0045] k c,j =W c,k h j +b c,k (13)

[0046] e c,ij =W c,ee ij +b c,e (14)

[0047] v c,a =W c,v h a +b c,v (15)

[0048]

[0049] in, They represent the weight matrices of the neighboring node feature vectors after the C-th attention of the two Linear layers; Represents the weight matrix of the center node feature vector after the Cth head attention of another Linear layer, represents the weight matrix of the Cth head attention from node j to node i, They represent the bias vectors of the neighboring node feature vectors after the C-th attention of the two Linear layers, represents the bias vector of the central node feature vector after the Cth attention of the Linear layer, h i ,h j Represents the feature vectors of neighboring nodes i and j respectively, e i,j represents the edge vector from neighboring node j to node i, h a represents the feature vector of the central node a, represents the edge vector of the Cth head’s attention, represents the edge vector of the Cth head’s attention neighborhood, < > represents the exponentially scaled dot product function, Represents the neighboring node feature h i The mapped query vector, Represents the neighboring node feature h j the key vector of the mapping, represents the key vector of the Cth head’s attention neighborhood, v c,a Represents the central node feature h a The value vector of the mapping, represents the neighborhood of each head's attention;

[0050] New vector representation in multi-head attention The specific calculation is based on the following formula:

[0051]

[0052] Among them, || represents the concatenation operation, which is used to concatenate the vectors of C heads, and the vector of each head is represented as its neighborhood The attention weighting.

[0053]

[0054] Among them, the fraction β represented by the original residual is calculated by the gated residual network i Specifically:

[0055]

[0056] in, represents the updated original residual representation, Represent the learnable weight matrix and bias vector respectively;

[0057] r i =W r h i +b r (10)

[0058] represents the updated original residual representation, Represent the learned weight matrix and bias vector, β i represents the fraction of the calculated original residual, and β i Used to balance the original residual representation r i With the new vector representation Represents the attention score weight, which is used to transform the multi-head representation vector back into d dimensions.

[0059] The loss function in step 4.6 is L1 loss function and L rec The sum of loss functions:

[0060] L all =L1+L rec (26)

[0061] Among them, the L1 loss function is:

[0062]

[0063] Among them, N represents the number of training samples, pred i Represents the prediction result of the i-th sample, y i Represents the true value of the i-th sample;

[0064] L rec The loss function is:

[0065] L rec =L a +L v +L t (20)

[0066] Among them, L a , L v , L t Represent the losses of audio modality, visual modality, and text modality respectively;

[0067]

[0068]

[0069]

[0070] Among them, E a 、E v 、E t Respectively represent audio encoder, image encoder, and text encoder; F HG Represents the attention aggregation-driven Transformer module and the cross-modal graph fusion module between the encoder and decoder, F G represents the cross-modal graph fusion module between the encoder and decoder; D a , D v , D t Respectively represent audio decoder, image decoder, and text decoder;

[0071] By minimizing the reconstruction loss, the goal of unsupervised learning is achieved, as shown in formula (24);

[0072]

[0073] The beneficial effects of the present invention are:

[0074] The present invention uses an unsupervised codec module and multiple independent codecs to extract multimodal heterogeneous features, integrates the self-attention mechanism into the structure of the encoder, filters a large amount of information irrelevant to emotional semantics, and uses unsupervised training to reduce the heterogeneity between output features; uses a strong text modality to enhance the aggregation of visual and audio modalities through an attention aggregation-driven Transformer module, thereby reducing the emotional semantic gap and feature redundancy; constructs nodes and edges in a corresponding manner for different modalities, and constructs cross-modal edges through feature similarity thresholds, and then combines the designed graph Transformer-based multimodal feature learning method to model long-distance dependencies in cross-modal graph networks, realize adaptive adjustment of dependencies, and enhance the expressiveness of the model. In summary, the unsupervised multimodal emotion recognition method of the present invention that integrates feature enhancement driven by attention aggregation and cross-modal graph is reasonable and effective, and has improved the accuracy of emotion classification (Acc2, F1 score) and emotion prediction regression (MAE, Corr), that is, it can achieve a great performance improvement in multimodal emotion recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a flow chart of the unsupervised multimodal emotion recognition method of attention aggregation and cross-modal graph fusion of the present invention;

[0076] Figure 2 It is a schematic diagram of the structure of the encoder in the unsupervised multimodal emotion recognition model of the present invention;

[0077] Figure 3 It is a schematic diagram of the structure of the attention aggregation Transformer module in the unsupervised multimodal emotion recognition model of the present invention;

[0078] Figure 4 Schematic diagram of the principle of cross-modal graph construction in the unsupervised multimodal emotion recognition model of the present invention;

[0079] Figure 5 Schematic diagram of the structure of the Transformer module in the unsupervised multimodal emotion recognition model of the present invention. DETAILED DESCRIPTION

[0080] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0081] Example 1

[0082] The unsupervised multimodal emotion recognition method of the present invention based on attention aggregation and cross-modal graph fusion has the following process: Figure 1 As shown, the specific steps are as follows:

[0083] Step 1: Get the audio modality feature I of the video sample a , Image modality features I v and text modality features I t Specifically, extract multiple video samples from the video database and obtain the corresponding multimodal labels y for the video samples. m , the audio modality features I are extracted from each video sample using the multimodal software development toolkit a , Image modality features I v and text modality features I t .

[0084] Step 2: Obtain audio modal feature I a , Image modality features I v and text modality features I t The corresponding audio vector F a , image vector F v , text vector F t ; Specifically:

[0085] Text modality features I t , use the pre-trained model BERT to encode the sentence to obtain the text vector F t , specifically:

[0086]

[0087] in, represents the parameters of the pre-trained model BERT, R represents the matrix space, and d t Represents the dimension corresponding to the text vector;

[0088] For audio modal feature I a and image modality features I v , use an independent unidirectional long short-term memory network to learn in the time dimension and obtain the audio vector F a , image vector F v , specifically:

[0089]

[0090] Among them, the A-sLSTM network is used to extract sequence features using the long short-term memory network for the audio modality, and the V-sLSTM network is used to extract sequence features using the long short-term memory network for the visual modality. and Represent the parameters of the hidden layers of the A-sLSTM and V-sLSTM networks, respectively. a and F v Represent the hidden layer output sequences of audio mode and visual mode, that is, the feature representation of single mode, respectively, representing audio vector and image vector, R represents matrix space, d a and d v Represent the dimensions corresponding to the audio vector and image vector respectively;

[0091] Step 3, construct an unsupervised multimodal emotion recognition model;

[0092] The unsupervised multimodal emotion recognition model includes an unsupervised encoder module, an attention aggregation Transformer module, a cross-modal graph fusion module, and an unsupervised decoder module connected in sequence;

[0093] The unsupervised encoder module includes three parallel encoders: audio encoder, text encoder and image encoder. The three encoders have the same structure. Figure 2 As shown in the figure, they all include two convolutional layers with a stride of 2, a self-attention module, a convolutional layer with a stride of 1, and a ReLU activation function layer connected in sequence. The output of the first convolutional layer is also directly connected to the input of the ReLU activation function layer through a residual network direct edge; the output of the ReLU activation function layer of the three encoders is used as the input of the attention aggregation Transformer module;

[0094] like Figure 3As shown, the attention aggregation Transformer module includes two parallel Transformer encoders, each Transformer encoder includes six Transformer layers connected in sequence, each Transformer layer includes a normalization layer connected in sequence, two multi-head self-attention mechanism modules, and a multi-layer perceptron module, the input of the normalization layer of the first Transformer layer in each Transformer encoder is connected to the output of the unsupervised encoder module, and the output of the multi-layer perceptron module in the last Transformer layer in each Transformer encoder is connected to the cross-modal graph fusion module;

[0095] The cross-modal graph fusion module includes a cross-modal graph construction module and a graph Transformer module connected in sequence. The output of the attention aggregation Transformer module is used as the input of the cross-modal graph construction module, and the output of the cross-modal graph construction module is used as the input of the graph Transformer module, as shown in Figure 5 As shown in the figure, the Transformer module includes three linear layers connected in sequence, a scaled dot product attention mechanism layer, a multiplication operation processing, a normalization layer, and a ReLU activation function layer. The central node features are output through a gated residual connection and added to the ReLU activation function layer to finally obtain the optimal prediction result;

[0096] The unsupervised decoder module includes three parallel decoders with the same structure, namely, audio decoder, text decoder, and image decoder. The three decoders use deconvolution operations with learnable parameters to achieve upsampling. The unsupervised decoder module uses deconvolution operations with learnable parameters to achieve upsampling.

[0097] Step 4: transform the audio vector F a , image vector F v , text vector F t Input into the unsupervised multimodal emotion recognition model for training to obtain the optimal output pred of the model i As the final feature vector for prediction and classification; specifically:

[0098] Step 4.1: transform the audio vector F a , image vector F v , text vector F t They are respectively input into the audio encoder, image encoder and text encoder of the unsupervised encoder module to obtain the audio modality feature vectors I a' = {I a1 ,I a2 ,…,I an}、Text modality feature vector I t' = {It1 ,I t2 ,…,I tn}、Image modality feature vector I v' = {I v1 ,I v2 ,…,I vn};

[0099] The calculation formula of the attention matrix output by the self-attention module in the audio encoder, image encoder and text encoder is shown in formula (4):

[0100]

[0101] Where Q is the query vector, K is the key vector, V is the value vector, T is the transpose, and d k is the dimension of the key vector K;

[0102] Step 4.2: transform the audio modal feature vector I a' , text modality feature vector I t' , image modality feature vector I v' Input the attention aggregation Transformer module to obtain the feature vector F of the text-to-audio aggregation enhancement t→a , the feature vector F after the text is visually enhanced t→v ; Specifically:

[0103] The audio modal feature vector I a' and the text modality feature vector I t' into a group, and aggregate blocks h in between by sequence alignment f , input into a Transformer encoder of the attention aggregation Transformer module, and obtain the feature vector F of the text-to-audio aggregation enhancement t→a ; The image modality feature vector I v' and the text modality feature vector I t' into a group, and aggregate blocks h in between by sequence alignment f , input into another Transformer encoder of the attention aggregation Transformer module to obtain the feature vector F after the text is visually enhanced t→v , specifically:

[0104] When inputting the Transformer encoder, the audio modal feature vector I is first a' , text modality feature vector I t' , image modality feature vector I v' Compress it so that it is consistent with the aggregate block h f The sequence length is consistent, and then the audio modal feature vector I a' , aggregation block hf , text modality feature vector I t' Input into a Transformer encoder, and obtain the feature vector F of cross-modal text-to-audio aggregation enhancement through the Transformer encoder t→a ; The image modality feature vector I v' , aggregation block h f , text modality feature vector I t' Input into a Transformer encoder, and obtain the feature vector F of cross-modal text-to-visual aggregation enhancement through the Transformer encoder t→v ;

[0105] in, That is h f Depend on It consists of B vectors in total, where B is a random number;

[0106] In any Transformer encoder, the feature vector of the input of the lth Transformer layer is After the Transformer encoder is updated, The updating process is shown in formulas (5)-(6).

[0107] For the lth layer, any feature vector As input, the layer normalization (LN) module, multi-head self-attention mechanism (MSA), and multi-layer perceptron (MLP) module in the Transformer module are driven by attention aggregation to update the Transformer layer, and the feature vector of the next layer is output. The update process is shown in formulas (5)-(6):

[0108]

[0109] in, Represents the feature vector of the input of the lth Transformer layer. The query vector, key vector, and value vector in the Transformer are all composed of Linear projection; y l is the feature vector output by the lth Transformer layer;

[0110] For the lth layer, the aggregate enhancement of the text modality to the audio modality is shown in formulas (7)-(8):

[0111]

[0112] in, is the text modality feature vector input to the lth Transformer layer, is the text modality feature vector output by the l+1th Transformer layer, is the aggregation block of the input to the lth Transformer layer, is the aggregation block for the input of the l+1th Transformer layer, is the audio modality feature vector input to the lth Transformer layer, is the audio modality feature vector output by the lth Transformer layer, θ a is the parameter of the lth Transformer layer whose input is the audio feature vector, θ t is the parameter of the lth Transformer layer whose input is the text feature vector, Aggregation block for the output of the l+1th Transformer layer;

[0113] The principle of the aggregation enhancement of the text modality on the image modality is the same as the aggregation enhancement of the text modality on the audio modality.

[0114] Step 4.3, F t→a , F t , F t→v Three feature vectors are input into the cross-modal graph construction module to construct the cross-modal graph in, represents the nodes in the cross-modal graph, E represents the edges in the cross-modal graph, and extracts the node features H = {h1,h2,...,h n},h1,h2,...,h n represents n nodes; Figure 4 As shown in the figure, when constructing the cross-modal graph, for the modal eigenvector, for F t→a and F t→v According to the time relationship, the time-adjacent connecting edges are constructed. t , build connection edges according to the syntactic dependency tree; for modal feature vectors, use the feature similarity threshold to build cross-modal edges for any two modal feature vectors. The higher the degree of association between two nodes, the greater their similarity. The internal relationship of the modality is built through the internal edges of the modal feature vectors, and the relationship between different modalities is built through the cross-modal edges between the modal feature vectors. The final cross-modal graph is obtained. And extract the node features H = {h1,h2,...,h n}, the node features include central nodes and neighboring nodes. Each node can be used as a central node, and the nodes connected to the central node are neighboring nodes;

[0115] Among them, when constructing cross-modal edges for any two modal feature vectors using the feature similarity threshold, the threshold ε is set in advance to calculate the similarity s of the two feature vectors. When the similarity is greater than the threshold, that is, s>ε, the edge can be connected. The similarity calculation formula is as follows:

[0116] s=CosSim(x,y) (9)

[0117] Among them, x and y represent two arbitrary eigenvectors;

[0118] Step 4.4: Transform the cross-modal graph The input graph Transformer module performs multimodal feature fusion to obtain the fusion feature o i ; Specifically:

[0119] The central node features are taken as input and connected through the gated residual network to calculate the score output β of the original residual representation i At the same time, the central node feature is used as input and output through a Linear layer to obtain the value vector v c,a , the neighboring node features corresponding to the central node features are taken as input and output through the other two Linear layers to obtain the query vector q c,i , key vector k c,j and each edge e from node j to node i ij The edge vector e in the multi-head attention c,ij , and then the query vector q c,i , key vector k c,j , edge vector e c,ij The multi-head attention score α is obtained by scaling the dot product attention mechanism layer output c,ij , and then the attention score α c,ij As input, the attention-based multi-head message aggregation operation is used to update the tail node features, and the new vector representation of multi-head attention is obtained as output. Then, the attention score weight W f With the new vector representation Perform multiplication to get the output vector Z i Finally, Z i The total output vector o after updating by the graph Transformer module is obtained by adding the output vectors of the normalization layer, the ReLU activation function layer and the residual network. i .

[0120] Neighboring node feature h i The mapped query vector q c,i , neighboring node features h j The key vector k of the mapping c,j , central node feature h a The mapped value vector vc,a , the edge vector e from neighboring node j to node i c,ij , and the multi-head attention score α c,ij The specific calculation is based on the following formula:

[0121] q c,i =W c,q h i +b c,q (12)

[0122] k c,j =W c,k h j +b c,k (13)

[0123] e c,ij =W c,e e ij +b c,e (14)

[0124] v c,a =W c,v h a +b c,v (15)

[0125]

[0126] in, They represent the weight matrices of the neighboring node feature vectors after the C-th attention of the two Linear layers; Represents the weight matrix of the center node feature vector after the Cth head attention of another Linear layer, represents the weight matrix of the Cth head attention from node j to node i, They represent the bias vectors of the neighboring node feature vectors after the C-th attention of the two Linear layers, represents the bias vector of the central node feature vector after the Cth attention of the Linear layer, h i ,h j Represents the feature vectors of neighboring nodes i and j respectively, e i,j represents the edge vector from neighboring node j to node i, h a represents the feature vector of the central node a, represents the edge vector of the Cth head’s attention, represents the edge vector of the Cth head’s attention neighborhood, < > represents the exponentially scaled dot product function, Represents the neighboring node feature h i The mapped query vector, Represents the neighboring node feature h j the key vector of the mapping, represents the key vector of the Cth head’s attention neighborhood, v c,a Represents the central node feature h a The value vector of the mapping, represents the neighborhood of each head's attention;

[0127] New vector representation in multi-head attention The specific calculation is based on the following formula:

[0128]

[0129] Among them, || represents the concatenation operation, which is used to concatenate the vectors of C heads, and the vector of each head is represented as its neighborhood The attention weighting.

[0130]

[0131] Among them, the fraction β represented by the original residual is calculated by the gated residual network i Specifically:

[0132]

[0133] in, represents the updated original residual representation, Represent the learnable weight matrix and bias vector respectively;

[0134] r i =W r h i +b r (10)

[0135] represents the updated original residual representation, Represent the learned weight matrix and bias vector, β i represents the fraction of the calculated original residual, and β i Used to balance the original residual representation r i With the new vector representation represents the attention score weight, which is used to transform the multi-head representation vector back to d dimensions;

[0136] Step 4.5, o i are sent to the audio decoder, text decoder, and image decoder respectively to reconstruct the input of audio, text, and image modality features;

[0137] Step 4.6: Use the loss function optimization during training to minimize the reconstruction error, that is, to make the audio, text, and image modal features reconstructed by the audio decoder, text decoder, and image decoder consistent with the original audio vector F input in step 4.1. a , image vector Fv , text vector F t Error minimization, where the loss function L all is the L1 loss function and L rec The sum of loss functions:

[0138] L all =L1+L rec (26)

[0139] Among them, the L1 loss function is:

[0140]

[0141] Among them, N represents the number of training samples, pred i Represents the prediction result of the i-th sample, y i Represents the true value of the i-th sample;

[0142] L rec The loss function is:

[0143] L rec =L a +L v +L t (20)

[0144] Among them, L a , L v , L t Represent the losses of audio modality, visual modality, and text modality respectively;

[0145]

[0146]

[0147]

[0148] Among them, E a 、E v 、E t Respectively represent audio encoder, image encoder, and text encoder; F HG Represents the attention aggregation-driven Transformer module and the cross-modal graph fusion module between the encoder and decoder, F G represents the cross-modal graph fusion module between the encoder and decoder; D a , D v , D t Respectively represent audio decoder, image decoder, and text decoder;

[0149] By minimizing the reconstruction loss, the goal of unsupervised learning is achieved, as shown in formula (24);

[0150]

[0151] Step 5: Use the trained unsupervised multimodal emotion recognition model to perform emotion recognition.

[0152] Example 2

[0153] On the basis of Example 1, when the present invention performs model training, the video database includes three multimodal sentiment analysis data sets, namely CMU-MOSEI, CMU-MOSEI and CH-SIMS. These three data sets are used to train the model established in Example 1 of the present invention. The model of the present invention is designed and implemented under the PyTorch deep learning framework. All experiments are implemented on a server configured with Intel(R) Core(TM) i9-10900K CPU@3.70GHz and GeForce RTX 3080GPU, and 0.00005 is set as the initial learning rate of the BERT model, the learning rate of other models is set to 0.0005, and gradient truncation is performed when the gradient value exceeds 0.8. For these three data sets, the hyperparameter settings of the model are shown in Table 1.

[0154] Table 1 Hyperparameter settings of the model

[0155] Hyperparameters CMU-MOSI CMU-MOSEI CMU-SIMS Batchsize 32 64 32 Number of Epochs 30 30 30 A-LSTMDropoutRate 0.0 0.0 0.0 V-LSTMDropoutRate 0.0 0.0 0.0 BERTDropoutRate 0.2 0.3 0.2 Number of hidden layers in V-LSTM 64 32 64 Number of hidden layers of A-LSTM 32 32 16 Optimizer Adam Adam Adam Number of graph network layers 2 2 2

[0156] In order to evaluate the performance of the proposed model, five evaluation indicators are used to measure the model performance. Table 3 shows the experimental results of the model on the CMU-MOSI dataset. In terms of data setting, the present invention adopts an unaligned data setting. The data marked with · in the table are experimental results obtained by reproducing the original public code. The data without · are directly from the experimental data of relevant literature. The bold represents the optimal model indicator, and the underlined represents the suboptimal indicator. The works marked with (B) all use BERT as the pre-training model for text modality. The works marked with the symbol all use the attention aggregation driven Transformer module, and the works marked with the symbol ⊙ all use the cross-modal graph fusion module.

[0157] In this embodiment, in order to verify the effectiveness of the present invention, the two-category accuracy (Acc2), F1 score (F1 score), mean absolute error loss (MAE), Pearson correlation coefficient (Corr), and seven-category accuracy (Acc7) are calculated for the emotion categories predicted by the model and the original emotion categories, and then compared with the existing methods. The performance comparison results of different network models on the CMU-MOSI dataset are shown in Table 2:

[0158] Table 2

[0159]

[0160]

[0161] In Table 2, it can be observed that compared with methods such as word embedding, BERT can significantly improve the recognition accuracy. This is due to the excellent dynamics of the BERT model, which can extract richer contextual sentence information, effectively solving the problem that other methods can only learn word vector feature representations, highlighting its advantages in deep semantic extraction. Therefore, the present invention selects BERT as a pre-training model for text modality. The results in Table 2 show that the attention aggregation-driven Transformer module can make full use of text modal information to help the remaining modal information for feature learning, and the recognition performance is better than the method of directly fusing the three modalities. In addition, the results in Table 2 show that cross-modal graph fusion can not only consider the temporal correlation of the network, but also model the spatial correlation of the network.

[0162] Example 3

[0163] Based on Example 2, the two-category accuracy (Acc2), F1 score (F1 score), mean absolute error loss (MAE), Pearson correlation coefficient (Corr), and seven-category accuracy (Acc7) are calculated for the emotion categories predicted by the model and the original emotion categories, and then compared with the existing methods. The performance comparison results of different network models on the CMU-MOSEI dataset are shown in Table 3:

[0164] Table 3

[0165]

[0166]

[0167] As shown in Table 3, an experimental analysis of the CMU-MOSEI dataset was conducted, and the results show that the AAT-CGF model is significantly better than other models in terms of data. The methods marked with (B) in the table all use the BERT pre-training model as a text feature learning method, so their performance is relatively superior. The identification method not only extracts the interactive information between multiple modalities, but also has a good representation of the internal information of a single modality. The AAT-CGF method proposed in the present invention can effectively characterize a single modality while fully integrating multimodal features. The results show that its Acc2 and F1 scores achieve optimal performance, and the regression performance index is slightly lower than other models, but still at a suboptimal level and much higher than the average level, which shows that the multi-task learning method adopted by our model is more suitable for classification tasks and has a higher recognition accuracy in classification tasks.

[0168] Example 4

[0169] Based on Example 2, the two-category accuracy (Acc2), F1 score (F1 score), mean absolute error loss (MAE), Pearson correlation coefficient (Corr), and seven-category accuracy (Acc7) are calculated for the emotion categories predicted by the model and the original emotion categories, and then compared with the existing methods. The performance comparison results of different network models on the CH-SIMS dataset are shown in Table 4:

[0170] Table 4

[0171] Model Acc2 F1 MAE Corr TFN

[2017] 79.86 80.16 0.428 0.605 MISA

[2022] 75.49 75.85 0.472 0.542 SaPIL

[2023] <![CDATA[ 81.25 ]]> <![CDATA[ 81.25 ]]> 0.423 <![CDATA[ 0.620 ]]> ConKI

[2023] 77.94 78.17 0.454 0.542 Humman-MM 80.74 80.78 <![CDATA[ 0.408 ]]> 0.647 SELF-MM·

[2021] 80.74 80.78 0.419 0.616 AAT-CGF (present invention) 81.73 81.78 0.399 0.655

[0172] As shown in Table 4, an experimental analysis was conducted on the Chinese sentiment analysis dataset CH-SIMS, and the results showed that the AAT-CGF model was significantly better than other models in terms of data. This dataset uses a non-aligned data setting method. In Table 4, the model is compared with 5 existing multimodal sentiment recognition models based on non-aligned data. The experimental results show that the model of the present invention is better than other baseline models in all four indicators. In the table, Humman-MM represents the manually annotated unimodal label, and the results show that the model of the present invention achieves better recognition performance than manual annotation.

[0173] The present invention utilizes the advanced emotional semantic characteristics of text features, adopts attention aggregation blocks for cross-modal enhancement, and compresses the information of each modality, thereby reducing the difference in emotional semantic information between modalities; independent unsupervised codecs are used for different modalities, and the self-attention mechanism is added to the structure of the encoder to filter a large amount of information irrelevant to emotional semantics, driving the model to learn better emotional representation; in order to model multiple dependencies between multimodalities, the corresponding intra-modal connection edge construction method is adopted for different modal data characteristics, and then cross-modal connection edges are constructed according to the feature similarity threshold. Multimodal graph Transformer is used to model long-distance dependencies of multimodal graph networks, and adaptive adjustment of dependencies is achieved by performing attention operations on nodes and edges, and node information is aggregated and updated.

Claims

1. Unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion, It is characterized in that The specific steps are as follows: Step 1: Get the audio modality feature I of the video sample a , Image modality features I v and text modality features I t ; Step 2: Obtain audio modal feature I a , Image modality features I v and text modality features I t The corresponding audio vector F a , image vector F v , text vector F t ; Step 3, construct an unsupervised multimodal emotion recognition model; Step 4: transform the audio vector F a , image vector F v , text vector F t Input into the unsupervised multimodal emotion recognition model for training to obtain the optimal output pred of the model i As the final feature vector for prediction and classification; Step 5: Use the trained unsupervised multimodal emotion recognition model to perform emotion recognition.

2. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 1, It is characterized in that The step 1 is specifically as follows: Extract multiple video samples from the video database and obtain the corresponding multimodal labels y for the video samples m , the audio modality features I are extracted from each video sample using the multimodal software development toolkit a , Image modality features I v and text modality features I t .

3. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 1 or 2, It is characterized in that The step 2 is specifically as follows: Text modality features I t , use the pre-trained model BERT to encode the sentence to get the text vector F t , specifically: in, represents the parameters of the pre-trained model BERT, R represents the matrix space, and d t Represents the dimension corresponding to the text vector; For audio modal feature I a and image modality features I v , use an independent unidirectional long short-term memory network to learn in the time dimension and obtain the audio vector F a , image vector F v , specifically: Among them, the A-sLSTM network is used to extract sequence features using the long short-term memory network for the audio modality, and the V-sLSTM network is used to extract sequence features using the long short-term memory network for the visual modality. and Represent the parameters of the hidden layers of the A-sLSTM and V-sLSTM networks, respectively. a and F v Represent the hidden layer output sequences of audio mode and visual mode, that is, the feature representation of single mode, respectively, representing audio vector and image vector, R represents matrix space, d a and d v Represent the dimensions corresponding to the audio vector and image vector respectively.

4. The unsupervised multimodal emotion recognition method based on attention aggregation and cross-modal graph fusion according to claim 3, It is characterized in that The unsupervised multimodal emotion recognition model in step 3 includes an unsupervised encoder module, an attention aggregation Transformer module, a cross-modal graph fusion module, and an unsupervised decoder module connected in sequence; The unsupervised encoder module includes three parallel encoders, an audio encoder, a text encoder, and an image encoder. The three encoders have the same structure, and all include two sequentially connected convolutional layers with a step size of 2, a self-attention module, a convolutional layer with a step size of 1, and a ReLU activation function layer. The output of the first convolutional layer is also directly connected to the input of the ReLU activation function layer through a residual network direct edge; the outputs of the ReLU activation function layers of the three encoders are used as the input of the attention aggregation Transformer module; The attention aggregation Transformer module includes two parallel Transformer encoders, each of which includes six Transformer layers connected in sequence, each of which includes a normalization layer, two multi-head self-attention mechanism modules, and a multi-layer perceptron module connected in sequence, the input of the normalization layer of the first Transformer layer in each Transformer encoder is connected to the output of the unsupervised encoder module, and the output of the multi-layer perceptron module in the last Transformer layer in each Transformer encoder is connected to the cross-modal graph fusion module; The cross-modal graph fusion module includes a cross-modal graph construction module and a graph Transformer module connected in sequence, the output of the attention aggregation Transformer module is used as the input of the cross-modal graph construction module, and the output of the cross-modal graph construction module is used as the input of the graph Transformer module. The graph Transformer module includes three linear layers connected in sequence, a scaled dot product attention mechanism layer, a multiplication operation processing, a normalization layer, and a ReLU activation function layer. The central node features are output through a gated residual connection and added to the ReLU activation function layer to finally obtain the optimal prediction result; The unsupervised decoder module includes three parallel decoders with the same structure, namely an audio decoder, a text decoder, and an image decoder. The three decoders use deconvolution operations with learnable parameters to achieve upsampling. The unsupervised decoder module uses deconvolution operations with learnable parameters to achieve upsampling.

5. The unsupervised multimodal emotion recognition method of attention aggregation and cross-modal graph fusion according to claim 4, It is characterized in that The step 4 is specifically as follows: Step 4.1: transform the audio vector F a , image vector F v , text vector F t They are respectively input into the audio encoder, image encoder and text encoder of the unsupervised encoder module to obtain the audio modality feature vectors I a' = {I a1 ,I a2 ,…,I an }、Text modality feature vector I t' = {I t1 ,I t2 ,…,I tn }、Image modality feature vector I v' = {I v1 ,I v2 ,…,I vn }; Step 4.2: transform the audio modal feature vector I a' , text modality feature vector I t' , image modality feature vector I v' Input the attention aggregation Transformer module to obtain the feature vector F of the text-to-audio aggregation enhancement t→a , the feature vector F after the text is visually enhanced t→v ; Step 4.3, F t→a , F t , F t→v Three feature vectors are input into the cross-modal graph construction module to construct the cross-modal graph in, represents the nodes in the cross-modal graph, E represents the edges in the cross-modal graph, and extracts the node features H = {h 1 ,h 2 ,...,h n }, h 1 ,h 2 ,...,h n Represents n nodes; Step 4.4: Transform the cross-modal graph The input graph Transformer module performs multimodal feature fusion to obtain the fusion feature o i ; Step 4.5, o i are sent to the audio decoder, text decoder, and image decoder respectively to reconstruct the input of audio, text, and image modality features; Step 4.6: Use the loss function optimization during training to minimize the reconstruction error, that is, to make the audio, text, and image modal features reconstructed by the audio decoder, text decoder, and image decoder consistent with the original audio vector F input in step 4.

1. a , image vector F v , text vector F t Minimize errors.

6. The unsupervised multimodal emotion recognition method of attention aggregation and cross-modal graph fusion according to claim 5, It is characterized in that The step 4.2 is specifically as follows: The audio modal feature vector I a' and the text modality feature vector I t' into a group, and aggregate blocks h in between by sequence alignment f , input into a Transformer encoder of the attention aggregation Transformer module, and obtain the feature vector F of the text-to-audio aggregation enhancement t→a ; The image modality feature vector I v' and the text modality feature vector I t ' is a group, and blocks h are aggregated in between by sequence alignment f , input into another Transformer encoder of the attention aggregation Transformer module to obtain the feature vector F after the text is visually enhanced t→v , specifically: When inputting the Transformer encoder, the audio modal feature vector I is first a' , text modality feature vector I t' , image modality feature vector I v' Compress it so that it is consistent with the aggregate block h f The sequence length is consistent, and then the audio modal feature vector I a' , aggregate block h f , text modality feature vector I t' Input into a Transformer encoder, and obtain the feature vector F of cross-modal text-to-audio aggregation enhancement through the Transformer encoder t→a ; The image modality feature vector I v' , aggregate block h f , text modality feature vector I t 'Input into a Transformer encoder, and obtain the feature vector F of cross-modal text-to-visual aggregation enhancement through the Transformer encoder t→v ; in, That is h f Depend on It consists of B vectors in total, where B is a random number.

7. The unsupervised multimodal emotion recognition method of attention aggregation and cross-modal graph fusion according to claim 6, It is characterized in that When constructing the cross-modal graph in step 4.3, for the modal feature vector, for F t→a and F t→v According to the time relationship, the time-adjacent connecting edges are constructed. t , build connection edges according to the syntactic dependency tree; for modal feature vectors, use the feature similarity threshold to build cross-modal edges for any two modal feature vectors. The higher the degree of association between two nodes, the greater their similarity. The internal relationship of the modality is built through the internal edges of the modal feature vectors, and the relationship between different modalities is built through the cross-modal edges between the modal feature vectors. The final cross-modal graph G = {V, E} is obtained, and the node features H = {h 1 ,h 2 ,...,h n }, the node features include central nodes and neighboring nodes. Each node can be used as a central node, and the nodes connected to the central node are neighboring nodes; Among them, when constructing cross-modal edges for any two modal feature vectors using the feature similarity threshold, the threshold ε is set in advance to calculate the similarity s of the two feature vectors. When the similarity is greater than the threshold, that is, s>ε, the edge can be connected. The similarity calculation formula is as follows: s=CosSim(x,y) (9) Here, x and y represent two arbitrary eigenvectors.

8. The unsupervised multimodal emotion recognition method of attention aggregation and cross-modal graph fusion according to claim 7, It is characterized in that The step 4.4 is specifically as follows: The central node features are taken as input and connected through the gated residual network to calculate the score output β of the original residual representation i At the same time, the central node feature is used as input and output through a Linear layer to obtain the value vector v c,a , the neighboring node features corresponding to the central node features are taken as input and output through the other two Linear layers to obtain the query vector q c,i , key vector k c,j and each edge e from node j to node i ij The edge vector e in the multi-head attention c,ij , and then the query vector q c,i , key vector k c,j , edge vector e c,ij The multi-head attention score α is obtained by scaling the dot product attention mechanism layer output c,ij , and then the attention score α c,ij As input, the attention-based multi-head message aggregation operation is used to update the tail node features, and the new vector representation of multi-head attention is obtained as output. Then, the attention score weight W f With the new vector representation Perform multiplication to get the output vector Z i Finally, Z i The total output vector o after updating by the graph Transformer module is obtained by adding the output vectors of the normalization layer, the ReLU activation function layer and the residual network. i .

9. The unsupervised multimodal emotion recognition method of attention aggregation and cross-modal graph fusion according to claim 8, It is characterized in that Neighboring node feature h i The mapped query vector q c,i , neighboring node features h j The key vector k of the mapping c,j , central node feature h a The mapped value vector v c,a , the edge vector e from neighboring node j to node i c,ij , and the multi-head attention score α c,ij The specific calculation is based on the following formula: q c,i =W c,q h i +b c,q (12) k c,j =W c,k h j +b c,k (13) have been c,ij =W c,e have been ij +b c,e (14) v c,a =W c,v h a +b c,v (15) in, They represent the weight matrices of the neighboring node feature vectors after the C-th attention of the two Linear layers; Represents the weight matrix of the center node feature vector after the Cth head attention of another Linear layer, represents the weight matrix of the Cth head attention from node j to node i, They represent the bias vectors of the neighboring node feature vectors after the C-th attention of the two Linear layers, represents the bias vector of the central node feature vector after the Cth attention of the Linear layer, h i ,h j Represents the feature vectors of neighboring nodes i and j respectively, e i,j represents the edge vector from neighboring node j to node i, h a represents the feature vector of the central node a, represents the edge vector of the Cth head’s attention, represents the edge vector of the Cth head’s attention neighborhood, < > represents the exponentially scaled dot product function, Represents the neighboring node feature h i The mapped query vector, Represents the neighboring node feature h j the key vector of the mapping, represents the key vector of the Cth head’s attention neighborhood, v c,a Represents the central node feature h a The value vector of the mapping, represents the neighborhood of each head's attention; New vector representation in multi-head attention The specific calculation is based on the following formula: Among them, || represents the concatenation operation, which is used to concatenate the vectors of C heads, and the vector of each head is represented as its neighborhood Attention weighting; Among them, the fraction β represented by the original residual is calculated by the gated residual network i Specifically: in, represents the updated raw residual representation, Represent the learnable weight matrix and bias vector respectively; r i =W r h i +b r (10) represents the updated raw residual representation, Represent the learned weight matrix and bias vector, β i represents the fraction of the calculated original residual, and β i Used to balance the original residual representation r i With the new vector representation Represents the attention score weight, which is used to transform the multi-head representation vector back into d dimensions.

10. The unsupervised multimodal emotion recognition method of attention aggregation and cross-modal graph fusion according to claim 9, It is characterized in that The loss function in step 4.6 is L 1 Loss Function and L rec The sum of loss functions: L all =L 1 +L rec (26) Among them, L 1 The loss function is: Among them, N represents the number of training samples, pred i Represents the prediction result of the i-th sample, y i Represents the true value of the i-th sample; L rec The loss function is: L rec =L a +L v +L t (20) Among them, L a , L v , L t Represent the losses of audio modality, visual modality, and text modality respectively; Among them, E a 、E v 、E t Respectively represent audio encoder, image encoder, and text encoder; F HG Represents the attention aggregation-driven Transformer module and the cross-modal graph fusion module between the encoder and decoder, F G represents the cross-modal graph fusion module between the encoder and decoder; D a , D v , D t Respectively represent audio decoder, image decoder, and text decoder; By minimizing the reconstruction loss, the goal of unsupervised learning is achieved, as shown in formula (24);

Images