Video emotion recognition method based on cyclic interaction transformer and dimension cross fusion

Abstract

The application discloses a video emotion recognition method based on cyclic interaction Transformer and dimension cross fusion, solves the problems of inaccurate multi-modal feature expression, insufficient interaction and high sensitivity to time sequence dependence in the video emotion recognition process, and the technical solution points are as follows: a modal specific embedding layer is used, high-level abstract features are extracted for multi-modal information in the video, the abstract features are input into the proposed cyclic interaction Transformer network, a cyclic interaction attention mechanism is designed to promote sufficient interaction of multi-modal features, and a dimension cross fusion method is proposed, high-dimensional information is mapped into different subspaces along the length, width and channel dimensions through a factor decomposition method, and the video emotion recognition method based on the cyclic interaction Transformer and the dimension cross fusion has the advantages that the expression of key information and the robustness of the model are enhanced, effective information in the multi-modal can be fully utilized, and the recognition rate of the emotion recognition is improved.

Description

Technical Field

[0001] This invention relates to multimodal emotion computing technology, and in particular to a video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion. Background Technology

[0002] Affective computing, also known as subjective sentiment analysis, typically refers to the use of statistical knowledge and deep learning techniques to determine the opinion and attitude of an object in a scene towards a particular topic or event, and to judge their emotional state. Currently, numerous studies have conducted sentiment analysis on text, audio, and video modalities. However, sentiment analysis based solely on a single modality of data, such as text, often suffers from limitations such as incomplete data and susceptibility to noise contamination. Therefore, combining data from multiple modalities for effective representation and analysis has become a current trend.

[0003] Multimodal emotion computing tasks often involve raw data from multiple modalities, such as images, text, and audio. Since different modalities possess different statistical properties and are distributed across different feature spaces, numerous studies have designed various deep learning methods for representing single-modal features. For example, image data is typically processed by hierarchical spatial networks, while text data is encoded by sequence networks. However, the differences between high-level semantic concepts and low-level data lead to semantic gaps between intramodal embeddings. To bridge this gap, self-supervised embedding models have been introduced to represent data from different modalities. After pre-training on large amounts of unlabeled data, self-supervised embedding models exhibit strong generalization capabilities. However, in this process, the inconsistency in representations caused by long-distance dependencies is often overlooked, which is detrimental to the maintenance of intramodal semantic information.

[0004] The input to multimodal emotion computing models typically consists of multiple sequences that interact in a strictly temporal order. In sequence tasks, recurrent neural networks and attention mechanisms are often combined to capture dynamic interactions over time. However, this computational approach, which strictly aligns multimodal data according to time steps, can lead to earlier information being interfered with or even overwritten by later information. To address this issue, memory networks have been developed to enable key information retrieval by constructing a storage unit with a complex structure, but this still cannot solve the problem of the high sensitivity of multimodal information interaction to temporal dependencies.

[0005] Depending on the stage of multimodal information fusion, it can be divided into feature-level fusion and decision-level fusion. Feature-level fusion refers to combining multimodal features (text, images, audio, etc.) and then using them as input to classifiers and regressors. The advantage of feature-level fusion is that the correlation between various multimodal features in the early stages can potentially provide better task completion; however, the disadvantage is that it requires the features to be standardized. In decision-level fusion, the feature learning of each modality is still separate, but a fusion mechanism is added at the final prediction stage. This mechanism learns a weight matrix from the data labels, which determines which modality's representation information should occupy a larger proportion in each classification process. The advantage of decision-level fusion is that each modality can choose its most suitable feature extractor to extract feature representation vectors, but its drawback is that the information interaction between modalities is too superficial.

[0006] Current video multimodal emotion recognition tasks still suffer from problems such as inaccurate representation of multimodal information features, insufficient multimodal information interaction, and high sensitivity of multimodal information interaction to temporal dependencies. Summary of the Invention

[0007] The purpose of this invention is to provide a video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion, which can make full use of the effective information existing in multimodal data and improve the recognition rate of emotion recognition.

[0008] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0009] A video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion includes the following steps:

[0010] S1. Based on multimodal sentiment data in videos, a pre-trained self-supervised neural network model is used as a modality-specific embedding layer to extract features from image, audio, and text data respectively.

[0011] S2. Based on the embedding correction algorithm, the extracted multimodal feature vectors are corrected and transformed into a unified representation;

[0012] S3. Obtain cross-modal information of the corrected multimodal feature vector through a cyclic interactive attention mechanism, and introduce and construct a cyclic matrix;

[0013] S4. Based on the recurrent interactive attention mechanism, add residual connections and feedforward layers, encapsulate them into a recurrent interactive attention module, stack multiple recurrent interactive attention modules to build a recurrent interactive Transformer network, obtain cross-modal information, and complete multimodal feature interaction.

[0014] S5. Based on dimensional cross-fusion, the obtained interaction results are mapped to different subspaces along the length, width, and channel dimensions using factor decomposition to obtain the final fusion result, which is used for identification and prediction.

[0015] As a preferred option, the feature extractors used in step S1 are all pre-trained and obtained through transfer learning.

[0016] The feature extractor structure for image information is based on the self-supervised neural network model Fab-Net and pre-trained on a large-scale face dataset.

[0017] The feature extractor structure for audio information is based on the self-supervised neural network model Wav2Vec and pre-trained on a large-scale speech recognition dataset.

[0018] The text information feature extractor structure is based on the self-supervised neural network model RoBERTa and is pre-trained on a large-scale text dataset.

[0019] As a preferred embodiment, the embedding correction algorithm in step S2 is specifically as follows:

[0020] Input {X i} i∈A,V , representing an audio (A) or image (V) sequence;

[0021] Initialize the special notation CLS←[];

[0022] Segment by sentence, forming a sentence sequence Seq←[s1,s2,…,s m ];

[0023] Concatenate the special notation CLS with the original data sequence using Concate([CLS],Seq);

[0024] Encode the sequence positions Pos = [p0, p1, p2, ..., p n ];

[0025] Incorporating location information into data sequences

[0026] Iterate M times;

[0027] Obtain the query vector Q, where Q = W Q ·I seq , key vector K, K = W K ·I seq Value vector V, V = W V ·I seq W Q W K W V These are learnable weight parameters;

[0028] Calculate the similarity between query vector Q and key vector K.

[0029] Obtain the final modal information feature vector O seq

[0030]

[0031] Return O seq .

[0032] As a preferred embodiment, the method for constructing the cyclic matrix in step S3 is as follows:

[0033] For text sequence data O T The text sequence data is denoted as Where n is the sequence length, a cyclic matrix can be constructed:

[0034]

[0035] mat_T(.) is the method for constructing a cyclic matrix.

[0036] As a preferred embodiment, the cyclical interactive attention mechanism in step S3 is specifically as follows:

[0037] Implement text data and With audio data The interaction between them, where N(.) represents the length of the feature vector and d(.) represents the number of feature channels, takes the text modality as the target modality, and uses the recurrent interactive attention mechanism (CIT) to obtain cross-modal information from the audio modality as follows:

[0038]

[0039] in The query vector is represented by the text feature sequence O. T Calculated, These are learnable parameter weights; The key vector is represented by the audio feature sequence O. A Calculated, These are learnable parameter weights; The value vector is represented by the audio feature sequence O. A Calculated, These are learnable parameter weights; mat_T(Q) T ) i Represents the row vector of a cyclic matrix.

[0040] Preferably, using the text modality as the target modality, the complete process of acquiring cross-modal information from the audio modality through a recurrent interactive Transformer network can be represented as follows:

[0041]

[0042]

[0043]

[0044] MLP stands for Fully Connected Neural Network. This represents a multi-headed recurrent interactive attention mechanism in the i-th layer.

[0045] As a preferred option, the dimensional cross-fusion method in step S5 is as follows:

[0046] It contains three branches, each responsible for capturing the cross features and dependencies between the spatial dimension height H or width W of the input feature map and the channel dimension C;

[0047] In branch one, we project onto the plane formed by height H and channel C to explore the interaction between (H,C). First, we rotate μ counterclockwise by 90° along the H-axis to obtain the new feature tensor after rotation. Then, a comprehensive pooling operation is performed on the first dimension of μ1 to reduce it to obtain the tensor. After passing through a convolutional layer and a fully connected layer, the attention weight matrix is ​​obtained by using a sigmoid activation function. The value of each element is between 0 and 1;

[0048] In branch two, we project onto the plane formed by width W and height H to explore the interaction between (W,H). First, we rotate μ counterclockwise by 90° along the W axis to obtain the new feature tensor after rotation. Similar to branch ①, a comprehensive pooling operation is performed on the first dimension of μ2 to reduce it to obtain the tensor. The attention weight matrix is ​​obtained by passing through a convolutional layer and a fully connected layer, and then through a sigmoid activation function. The value of each element is between 0 and 1;

[0049] In branch three, a projection is made onto the plane formed by the width W and the channel C to explore the interaction between (W, C). This branch does not require rotation; instead, a comprehensive pooling is performed on the first dimension of μ to reduce it to obtain the tensor. The attention weight matrix is ​​obtained by passing through a convolutional layer and a fully connected layer, and then through a sigmoid activation function. The value of each element is between 0 and 1;

[0050] The results from the three branches are summed and averaged to obtain the final fusion result.

[0051] As a preferred method, the calculation of tensors is as follows:

[0052]

[0053] μ max =MaxPool 1d (μ), μ avg =AvgPool 1d (μ)

[0054] Where Concate(.) means concatenation, MaxPool 1d (.) denotes one-dimensional max pooling, AvgPool 1d (.) indicates one-dimensional average pooling;

[0055] The final fusion result is as follows:

[0056]

[0057] Where rotate(.) represents rotation, σ represents the sigmoid nonlinear activation function, and ψ1, ψ2, ψ3 represent three different two-dimensional convolutional layers.

[0058] In summary, the present invention has the following beneficial effects:

[0059] Using modality-specific embedding layers, high-level abstract features are extracted from multimodal information in videos, including images, audio, and text. These abstract features are then fed into a proposed recurrent interactive Transformer network. Based on the original Transformer neural network framework, this network employs a recurrent interactive attention mechanism to promote full interaction among multimodal features. By constructing a recurrent matrix, it explores as many combinations of multimodal feature interactions as possible, while also reducing the temporal sensitivity of multimodal interactions. A dimensional cross-fusion method is proposed, mapping high-dimensional information to different subspaces along the length, width, and channel dimensions using factorization. This addresses the redundancy problem of multimodal features in video sentiment analysis, enhances the expressive power of key information and the robustness of the model, fully utilizes the effective information existing in multimodal data, and improves the recognition rate of sentiment analysis. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the method.

[0061] Figure 2 This is a schematic diagram of the embedded correction algorithm in this invention;

[0062] Figure 3 This is a structural diagram of the cyclic interactive attention mechanism in this invention;

[0063] Figure 4This is a network structure diagram of the loop-interactive Transformer in this invention;

[0064] Figure 5 This is a schematic diagram of the dimensional cross-fusion structure in this invention;

[0065] Figure 6 This is the recognition confusion matrix diagram of the present invention on the emotion recognition dataset IEMOCAP. Detailed Implementation

[0066] The present invention will be further described in detail below with reference to the accompanying drawings.

[0067] According to one or more embodiments, a video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion is disclosed, such as... Figure 1 As shown, it includes the following steps:

[0068] S1. Based on multimodal emotion data in videos, a pre-trained self-supervised neural network model is used as a modality-specific embedding layer to extract features from image, audio, and text data respectively.

[0069] The feature extractors used for feature extraction are all pre-trained and acquired through transfer learning. Specifically:

[0070] The feature extractor structure for image information is based on the self-supervised neural network model Fab-Net, which is pre-trained on a large-scale face dataset. Specifically, the self-supervised neural network model Fab-Net in this example is pre-trained on the VoxCeleb dataset to learn face attributes as an auxiliary task, including key point detection, facial muscle movements, and expressions. Its embedding vector has a dimension of 256 and a maximum sequence length of 300.

[0071] The feature extractor structure for audio information is based on the self-supervised neural network model Wav2Vec, which is pre-trained on a large-scale speech recognition dataset. Specifically, the self-supervised neural network model Wav2Vec in this example is pre-trained on the librisspeech dataset. This network contains 3.35M parameters, has an embedding size of 512, and a maximum sequence length of 935.

[0072] The feature extractor structure for text information is based on the self-supervised neural network model RoBERTa, which is pre-trained on a large-scale text dataset. Specifically, the self-supervised neural network model RoBERTa in this example consists of a 24-layer Transformer encoder with 355M parameters. It is pre-trained on the English text datasets CC-NEWS, OPENWEBTEXT, and STORIES, with an embedding vector dimension of 1024 and a maximum sequence length of 512.

[0073] S2. Based on the embedding correction algorithm, the extracted multimodal feature vectors are corrected and transformed into a unified representation, which is beneficial to cross-modal interaction and fusion of multimodal information.

[0074] like Figure 2 As shown, this embedding correction algorithm introduces a special label CLS and uses a multi-head self-attention mechanism to uniformly encode the feature information inside the modality.

[0075] The embedding correction algorithm is specifically as follows:

[0076] Input {X i} i∈A,V , representing an audio (A) or image (V) sequence;

[0077] Initialize the special notation CLS←[];

[0078] Segment by sentence, forming a sentence sequence Seq←[s1,s2,…,s m ];

[0079] Concatenate the special notation CLS with the original data sequence using Concate([CLS],Seq);

[0080] Encode the sequence positions Pos = [p0, p1, p2, ..., p n ];

[0081] Incorporating location information into data sequences

[0082] Iterate M times;

[0083] Obtain the query vector Q, where Q = W Q ·I seq , Key vector K, K = W K ·I seq Value vector V, V = W V ·I seq W Q W K W V These are learnable weight parameters;

[0084] Calculate the similarity between query vector Q and key vector K.

[0085] Obtain the final modal information feature vector O seq

[0086]

[0087] Return O seq .

[0088] S3. Obtain cross-modal information of the corrected multimodal feature vector through a cyclic interactive attention mechanism, and introduce and construct a cyclic matrix. Through this mechanism, the corrected multimodal feature vector can enhance the expressive power of the current modality feature by leveraging cross-modal features, and reduce the dependence of multimodal features on time sequence by introducing and constructing a cyclic matrix.

[0089] in,

[0090] like Figure 3 As shown, the specific method for constructing a circular matrix is ​​as follows:

[0091] Taking text sequence data as an example, for text sequence data O T The text sequence data is denoted as Where n is the sequence length, a cyclic matrix can be constructed:

[0092]

[0093] mat_T(.) is the method for constructing a cyclic matrix.

[0094] The specific details of the cyclical interactive attention mechanism are as follows:

[0095] Taking text data and audio data as an example, how to implement text data and audio data... With audio data The interaction between them, where N(.) represents the length of the feature vector and d(.) represents the number of feature channels, takes the text modality as the target modality, and uses the recurrent interactive attention mechanism (CIT) to obtain cross-modal information from the audio modality as follows:

[0096]

[0097] in The query vector is represented by the text feature sequence O. T Calculated, These are learnable parameter weights; The key vector is represented by the audio feature sequence O. A Calculated, These are learnable parameter weights; The value vector is represented by the audio feature sequence O. A Calculated, These are learnable parameter weights; mat_T(Q) T ) i Represents the row vector of a cyclic matrix.

[0098] S4. Based on the recurrent interactive attention mechanism, residual connections and feedforward layers are added and encapsulated as recurrent interactive attention modules. Multiple recurrent interactive attention modules are stacked to construct a recurrent interactive Transformer network, which obtains cross-modal information and completes multimodal feature interaction. Each module abstracts high-level semantic features while preserving the low-level semantic information of the previous layer to the maximum extent. This fine-grained feature representation can effectively improve the model performance.

[0099] like Figure 4 As shown, the specific calculation method of the recurrent interactive Transformer network is as follows:

[0100] Using the text modality as the target modality, the complete process of acquiring cross-modal information from the audio modality through a recurrent interactive Transformer network can be represented as follows:

[0101]

[0102]

[0103]

[0104] MLP stands for Fully Connected Neural Network. This represents a multi-headed recurrent interactive attention mechanism in the i-th layer.

[0105] S5. Based on dimensional cross-fusion, the obtained interaction results are mapped to different subspaces along the length, width, and channel dimensions using factorization to obtain the final fusion result, which is then used for recognition and prediction. This solves the problem of multimodal feature redundancy in video sentiment analysis and helps to enhance the expressiveness of key information and the robustness of the model.

[0106] like Figure 5 As shown, a dimensional cross-fusion method is designed, which is specifically as follows:

[0107] It contains three branches, each responsible for capturing the cross features and dependencies between the spatial dimension height H or width W of the input feature map and the channel dimension C;

[0108] Figure 5 From left to right, specifically, when the input feature tensor is represented as In branch one, we project onto the plane formed by height H and channel C to explore the interaction between (H,C). First, we rotate μ counterclockwise by 90° along the H-axis to obtain the new feature tensor after rotation. Then, a comprehensive pooling operation is performed on the first dimension of μ1 to reduce it to obtain the tensor. After passing through a convolutional layer and a fully connected layer, the attention weight matrix is ​​obtained by using a sigmoid activation function. The value of each element is between 0 and 1;

[0109] In branch two, we project onto the plane formed by width W and height H to explore the interaction between (W,H). First, we rotate μ counterclockwise by 90° along the W axis to obtain the new feature tensor after rotation. Similar to branch ①, a comprehensive pooling operation is performed on the first dimension of μ2 to reduce it to obtain the tensor. The attention weight matrix is ​​obtained by passing through a convolutional layer and a fully connected layer, and then through a sigmoid activation function. The value of each element is between 0 and 1;

[0110] In branch three, a projection is made onto the plane formed by the width W and the channel C to explore the interaction between (W, C). This branch does not require rotation; instead, a comprehensive pooling is performed on the first dimension of μ to reduce it to obtain the tensor. The attention weight matrix is ​​obtained by passing through a convolutional layer and a fully connected layer, and then through a sigmoid activation function. The value of each element is between 0 and 1;

[0111] The results from the three branches are summed and averaged to obtain the final fusion result.

[0112] The calculation of tensors is specifically as follows:

[0113]

[0114] μ max =MaxPool 1d (μ), μ avg =AvgPool 1d (μ)

[0115] Where Concate(.) means concatenation, MaxPool 1d (.) denotes one-dimensional max pooling, AvgPool 1d (.) indicates one-dimensional average pooling;

[0116] The final fusion result is as follows:

[0117]

[0118] Where rotate(.) represents rotation, σ represents the sigmoid nonlinear activation function, and ψ1, ψ2, ψ3 represent three different two-dimensional convolutional layers.

[0119] To illustrate this clearly, here is an example:

[0120] This example uses the publicly available sentiment computing dataset IEMOCAP. This dataset contains five dialogues and ten male and female actors, with each dialogue performed by a fixed pair of actors. IEMOCAP contains approximately 12 hours of audiovisual data, including video, audio, transcribed text, and facial expressions. The dataset is labeled with seven categories: anger, happiness, sadness, neutrality, excitement, frustration, fear, and surprise.

[0121] Because the distribution of the dataset across categories is uneven, four of the most common labels were selected: Happy, Sad, Anger, and Neutral. The dataset was divided into training, validation, and test sets. The first four dialogues were used for training and validation, while the last dialogue was used for testing. Therefore, the two actors in the test set would not appear in the training and validation sets. This segmentation method also eliminates speaker-related interference when evaluating the sentiment analysis results. The statistical information is shown in Table 1.

[0122] IEMOCAP 4 2,717 789 938

[0123] Table 1

[0124] Based on the amount of data and the number of categories in the IEMOCAP dataset, the specific model parameters in this example are presented in Table 2:

[0125] Initial learning rate 3.00E-04 Learning rate strategy Adam Number of self-attention blocks 2 Number of cross-modal attention blocks 2 Self-attention count 4 Number of cross-modal attention heads 4 Dropout rate 0.1 Number of training rounds 20

[0126] Table 2

[0127] Table 3 compares the accuracy of multimodal emotion recognition models.

[0128] Table 3

[0129] As shown in Table 3, the model designed in this invention outperforms existing models in both accuracy (Acc) and F1-score. CTC represents connectionist temporal classification mechanism; combining CTC with the model enables end-to-end training in an unsupervised manner and allows the model to infer the alignment relationship between speech and text.

[0130] like Figure 6 The figure shows the accuracy-confusion matrix of the model involved in this invention on the IEMOCAP dataset. As can be seen from the figure, "Happy" has the highest recognition accuracy at 83.3%, while "Neutral" is the most difficult to recognize. Furthermore, "Angry" is most likely to be misidentified as "Happy," which we speculate may be because both of these expressions involve significant facial muscle changes.

[0131] The method proposed in this invention can fully utilize and effectively integrate various modal information in videos, thereby improving the accuracy of emotion recognition.

[0132] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion, characterized in that, It includes the following steps: S1. Based on multimodal sentiment data in videos, a pre-trained self-supervised neural network model is used as a modality-specific embedding layer to extract features from image, audio, and text data respectively. S2. Based on the embedding correction algorithm, the extracted multimodal feature vectors are corrected and transformed into a unified representation; S3. Obtain cross-modal information of the corrected multimodal feature vector through a cyclic interactive attention mechanism, and introduce and construct a cyclic matrix; S4. Based on the recurrent interactive attention mechanism, add residual connections and feedforward layers, encapsulate them into a recurrent interactive attention module, stack multiple recurrent interactive attention modules to build a recurrent interactive Transformer network, obtain cross-modal information, and complete multimodal feature interaction. S5. Based on dimensional cross-fusion, the obtained interaction results are mapped to different subspaces along the length, width, and channel dimensions using factor decomposition to obtain the final fusion result, which is used for identification and prediction. The specific method for constructing the circular matrix is ​​as follows: For text sequence data The text sequence data is denoted as Where n is the sequence length, a cyclic matrix can be constructed: in Methods for constructing circular matrices; The recurrent interactive attention mechanism is specifically as follows: Implement text data With audio data The interaction between them, where N(.) represents the length of the feature vector and d(.) represents the number of feature channels, takes the text modality as the target modality, and uses the recurrent interactive attention mechanism (CIT) to obtain cross-modal information from the audio modality as follows: in The query vector is represented by a sequence of text features. Calculated, These are learnable parameter weights; The key vector represents the sequence of audio features. Calculated, These are learnable parameter weights; Represents a value vector, composed of audio feature sequences. Calculated, These are learnable parameter weights; Represents the row vector of a cyclic matrix.

2. The video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion according to claim 1, characterized in that: The feature extractors used in step S1 are all pre-trained and obtained through transfer learning; The feature extractor structure for image information is based on the self-supervised neural network model Fab-Net and pre-trained on a large-scale face dataset. The feature extractor structure for audio information is based on the self-supervised neural network model Wav2Vec and pre-trained on a large-scale speech recognition dataset. The text information feature extractor structure is based on the self-supervised neural network model RoBERTa and is pre-trained on a large-scale text dataset.

3. The video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion according to claim 1, characterized in that, The embedding correction algorithm in step S2 is as follows: enter , representing an audio A or image V sequence; Initialize special tokens ; Segment by sentence to form a sentence sequence ; The special notation CLS is concatenated with the statement sequence Seq to obtain the input sequence Concate([CLS], Seq); Encoding sequence positions ; The location information is embedded into the input sequence to obtain = Concate([CLS], Seq); Iterate M times; Get query vector , Key vector K, Value vector V, V ,in These are learnable weight parameters; Calculate the similarity between query vector Q and key vector K. ; Obtain the final modal information feature vector. = return .

4. The video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion according to claim 1, characterized in that: Using the text modality as the target modality, the complete process of acquiring cross-modal information from the audio modality through a recurrent interactive Transformer network can be represented as follows: in This represents a fully connected neural network. This represents a multi-headed recurrent interactive attention mechanism in the i-th layer.

5. The video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion according to claim 1, characterized in that, The dimensional cross-fusion method in step S5 is as follows: It contains three branches, each responsible for capturing the cross features and dependencies between the spatial dimension height H or width W of the input feature map and the channel dimension C; In branch one, project onto the plane formed by height H and channel C, exploring the interaction between (H,C). First, Rotating 90° counterclockwise along the H-axis yields the new feature tensor represented as follows: Subsequently The first dimension is pooled to reduce the tensor. After passing through a convolutional layer and a fully connected layer, the attention weight matrix is ​​obtained by using a sigmoid activation function. Each element has a value between 0 and 1; In branch two, project onto the plane formed by width W and height H, exploring the interaction between (W,H). First, Rotating 90° counterclockwise along the W-axis, the resulting new feature tensor is represented as follows: ,right The first dimension is pooled to reduce the tensor. The attention weight matrix is ​​obtained by passing the signal through a convolutional layer and a fully connected layer, and then through a sigmoid activation function. Each element has a value between 0 and 1; In branch three, project onto the plane formed by width W and channel C, exploring the interaction between (W, C). This branch does not require rotation. The first dimension is pooled to reduce the tensor. The attention weight matrix is ​​obtained by passing the signal through a convolutional layer and a fully connected layer, and then through a sigmoid activation function. Each element has a value between 0 and 1; The results from the three branches are summed and averaged to obtain the final fusion result.

6. The video emotion recognition method based on cyclic interactive Transformer and dimensional cross-fusion according to claim 5, characterized in that: The calculation of tensors is specifically as follows: in Indicates splicing, This represents one-dimensional max pooling. This represents one-dimensional average pooling.

Images