A method and system for emotion recognition with dynamic rotation across sample modalities

Abstract

The present invention discloses a method and system for emotion recognition with dynamic rotation across sample modalities. First, multimodal data and emotion classification annotations are obtained, and then corresponding modal encoder preprocessing is performed to obtain corresponding preliminary modal feature information. Then, a gated residual self-attention mechanism is used to capture deeper modal attention features, and considering the richness of modal emotional semantic information, the unimodal independent emotional annotations are combined with them to jointly calculate the integrated modal score sequence, and then the modal rotation order is determined under the rotation cycle. Finally, a multimodal fusion method based on a cross-modal gated residual attention mechanism is used to perform fusion according to the rotation order. The present invention proposes the idea of ​​dynamically adjusting the fusion order of the modality itself based on the independent emotional annotations, focusing on the problem of emotional semantic information imbalance between modalities, exploring the impact of modal primary and secondary issues on the field of multimodal emotion detection, and breaking through the traditional primary and secondary model.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal sentiment analysis, and specifically relates to a sentiment recognition method and system with dynamic rotation across sample modalities. Background Art

[0002] In human social interactions, expressions between people often contain rich emotional values. The complexity of their recognition lies in the fact that they are the result of cross-expression in multiple ways (for example, facial and body movements, voice, and language). Therefore, multimodal fusion has always been the focus of people's attention. Unlike single-modal emotion recognition, which only mines the emotional clues contained in one modality, multimodal emotion recognition focuses more on the mining of isomorphism and heterogeneity between multiple modalities. It not only needs to consider the common characteristics between multiple modalities, but also needs to capture and analyze the heterogeneous characteristics of the modalities themselves in more detail. However, the emotional semantic information contained in multiple modalities themselves is not equal, that is, the emotional semantic information between modalities is unbalanced. This phenomenon can easily lead to semantic information overload of a certain modality. The modality with overloaded emotional semantic information is often given too much attention, which causes the order of fusion between modalities to generally be based on it. For specific cases, please refer to Figure 1 In contrast, this ignores the importance of the secondary modality, causing the overall fusion result to be overly dependent on the primary modality and thus biased. Mohit et al. have confirmed that the order of modal fusion has a significant impact on the results of sentiment analysis.

[0003] Most studies use a general modality fusion order that prioritizes text modalities and supplements them with non-text modalities. This order is a static, fixed fusion order that applies to the entire sentiment recognition framework. For example, the AMOA model proposed by Ziming Li et al. introduces global acoustic features into the field of multimodal sentiment analysis and proposes a modal integration order for text, audio, and video. MOsarcation et al. further studied the impact of modality injection order on sarcasm detection and discovered the optimal modality fusion order through experiments. This fusion order is essentially a fixed, static modality fusion order. Because it fails to account for the asymmetry of sentiment semantic information between different modalities at the cross-sample level, sentiment analysis accuracy is low.

[0004] To this end, the present invention proposes an emotion recognition method and system with dynamic rotation across sample modalities. By considering the impact of sequential interactions of different modalities across samples on the multimodal emotion analysis results, and combining the independent emotion annotations of the single modality itself, it aims to solve the capture of isomorphic heterogeneity between modalities from the level of modal fusion order, so that the system can more effectively detect more contradictory multimodal unified emotions.

[0005] Explanation of terms

[0006] Dynamic rotation: This method uses samples as units to determine the rotation order of each modality in different training batches. Because the direction of the dynamic alternation of the three modalities is similar to dynamic rotation, it is named dynamic rotation. Summary of the Invention

[0007] The purpose of the present invention is to address the neglect and shortcomings of the existing technology and propose a cross-sample modality dynamic rotation emotion recognition method to accurately grasp the interactive properties between each feature and capture deeper intra-modal homogeneous and heterogeneous characteristics, thereby improving the accuracy and effectiveness of emotion recognition. First, multimodal data and emotion classification annotations are obtained and modal features are extracted from them. Then, the corresponding features are processed into preliminary modal feature information. Then, from the perspective of the dynamic modal fusion order based on samples, considering the impact of the independent emotion annotation of the single modality itself on the final emotion analysis results, the gated residual attention mechanism is used to capture deeper single-modal independent emotion heterogeneous features. Finally, a solution and model are proposed, and good experimental results are achieved on two datasets for single-modal independent emotion annotations.

[0008] In a first aspect, the present invention provides a method for emotion recognition with dynamic rotation across sample modalities, the method comprising the following steps:

[0009] Step 1: Obtain multimodal data and corresponding sentiment classification annotations based on the video; the multimodal data includes text modality, audio modality, and visual modality; the sentiment classification annotations include multimodal unified sentiment classification annotations and independent sentiment classification annotations for the three modalities;

[0010] Step 2: Preprocess the multimodal data through the corresponding modal encoder to obtain the corresponding preliminary modal feature information; the preliminary modal feature information X m Including language embedding features, visual embedding features, and sound embedding features, m = l, v, a represent language embedding features, visual embedding features, and sound embedding features respectively;

[0011] Step 3: Process the preliminary modal feature information using the gated residual self-attention mechanism to obtain the modal self-attention feature; the modal self-attention feature Including language self-attention features Visual self-attention features Characteristics of auditory self-attention Specifically:

[0012] The preliminary modal feature information is processed by the gated residual self-attention mechanism, and a deeper modal self-attention feature is obtained after multiple iterations. and gated residual attention factor

[0013]

[0014] Among them, GatedResiAtten self (·) represents the gated residual self-attention mechanism; Initialize the nth iteration modal self-attention feature is the gated residual attention factor of the nth iteration, and N represents the maximum number of iterations.

[0015] Step 4: Based on the modal self-attention features obtained in step 3, calculate the multimodal fusion order of the three modalities. This fusion method is based on the gated residual self-attention mechanism proposed in step 3 with slight modifications; specifically:

[0016] S4-1. Extract the feature information corresponding to the single-modal independent sentiment annotation from the modal self-attention features and aggregate them into a modal score sequence

[0017]

[0018] in, Represents the unimodal independent sentiment classification annotation y corresponding to modality m i The i-th score sequence of ; B is the number of sample batches, ∑ represents the summation operator; Match(·) represents the matching operation; Softmax(·) is the activation function that performs the probability distribution calculation so that the value of each sequence position is between 0 and 1, and the sum of all positions is 1;

[0019] S4-2. Calculate the difference ratio ρ between two modes bc ;

[0020]

[0021] in, Represent the score sequences of two different modalities, b and c do not represent specific modal features, i represents the independent sentiment annotation of the corresponding sample; subscript j represents the corresponding relative position of the score sequence; ρ bc It is a general calculation of the difference ratio between two modes, specifically including ρ va ,ρ vl ,ρ al If ρ bc >1, it indicates a fractional sequence D represents the feature dimension;

[0022] S4-3. By comparing the score sequences of the three modes, the main mode P in the current modal rotation period is obtained;

[0023]

[0024] Wherein, Sort(·) indicates sorting the three difference ratios from large to small. If the first two selected difference ratios both contain a certain mode, then the mode is set as the main mode P; Both represent the identity after taking the mean of the sequence values;

[0025] S4-4. Based on the main modality P, the modal self-attention features of the remaining two modalities are used as the secondary modality I i ;

[0026] Minor Mode I i Perform similarity measurement calculations with P respectively, and find the modal feature with the closest similarity to P (that is, the degree of alignment of the two modalities) as the second modal feature;

[0027]

[0028] Among them, M PI Indicates the obtained main mode P and secondary mode I i Similarity measurement matrix, secondary mode I i is a general reference, i represents the labels of the remaining two modes; μ PI is the learnable threshold hyperparameter;

[0029] According to the main mode P and the second mode features, the three modes can be sorted to obtain the multimodal fusion order;

[0030] Step 5: Based on the multimodal fusion order of step 4, perform cross-modal fusion on the modal self-attention features;

[0031] Step 6: Input the multimodal fusion results into the classifier to obtain the emotion recognition results;

[0032] In a second aspect, the present invention provides an emotion recognition system, the system comprising:

[0033] The data acquisition module is responsible for acquiring multimodal data and sentiment classification annotations based on the video; the multimodal data includes text modality, audio modality, and visual modality; the sentiment classification annotations include multimodal sentiment unified classification annotations and three-modality independent sentiment classification annotations;

[0034] The feature extraction module is responsible for preprocessing the multimodal data to obtain the corresponding preliminary modal feature information; the modal feature information includes language embedding features, visual embedding features, and audio embedding features;

[0035] The modal dynamic fusion module is responsible for measuring the richness of emotional semantic information of each modality, obtaining the multimodal fusion order, and finally performing multimodal fusion according to the multimodal fusion order;

[0036] The recognition module is responsible for inputting the multimodal fusion results into the classifier to obtain the emotion recognition results;

[0037] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described above.

[0038] In a fourth aspect, the present invention provides a computing device comprising a memory and a processor, wherein the memory stores executable code and the processor implements the method when executing the executable code.

[0039] The beneficial effects of the present invention are as follows:

[0040] 1. This paper proposes a new gated residual attention mechanism that is adapted to dynamic rotation across sample modalities, thereby capturing the heterogeneous characteristics of modal attention under multiple layers of attention hierarchy while increasing the direct closeness of the hierarchy.

[0041] 2. This paper improves the inherent static modal fusion order and proposes for the first time a method of dynamic modal rotation based on samples, and has achieved good results on multiple datasets for single-modal independent sentiment annotation.

[0042] 3. This paper focuses on the imbalance of sentiment and semantic information between modalities, exploring and studying the impact of modal priority on multimodal sentiment detection. It breaks through the traditional priority model, which prioritizes the text modality over the non-text modality. It proposes a method that dynamically adjusts the fusion order based on independent sentiment annotations, making a significant contribution to the field of sentiment recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A diagram describing the problem proposed for this invention.

[0044] Figure 2 This is the overall structural diagram of the present invention (a multimodal rotation emotion feedback network based on gated residual attention).

[0045] Figure 3 This is the gated residual self-attention model diagram proposed by the present invention;

[0046] Figure 4 It is a flow chart of the score calculation diagram of the present invention (i.e., the score calculation part in the modal dynamic fusion score calculation module).

[0047] Figure 5 This is a flow chart of the rotation order similarity measurement diagram of the present invention (i.e., the rotation order matrix measurement part in the modal dynamic fusion score calculation module).

[0048] Figure 6 Flowchart of the gated residual attention fusion module of the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] The terms "including," "having," and any variations thereof, as used in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0051] The emotion recognition method of the present invention proposes the following ideas:

[0052] First, obtain multimodal data and sentiment classification annotations. Then, perform corresponding modal encoder preprocessing on different modal data, and project their time series feature dimensions into the same dimension to obtain the corresponding embedded feature information and modal feature information. Figure 3 Gated residual self-attention model diagram. The present invention finds that the use of an improved gated residual self-attention mechanism can effectively capture deeper modal attention features and is more suitable for the score calculation part of the modal order.

[0053] Based on the obtained modal attention feature, the present invention considers the richness of modal emotional semantic information and combines the single modal independent emotional annotation with it to calculate the integrated modal score sequence. The modal score queue will obtain different score results with the different characteristics of the sample itself in the modal rotation cycle, thereby obtaining different dynamic rotation orders. This part refers to the score calculation diagram Figure 4 Then, the modal attention feature with the richest emotional semantic information in the current rotation cycle is calculated based on the obtained modal score sequence, and its similarity is measured with the attention features of the remaining two modalities. You can refer to the rotation order similarity measurement diagram Figure 5 Finally, the modal rotation order determined in the current rotation cycle is obtained. Finally, a multimodal fusion method based on the cross-modal gated residual attention mechanism is used to fuse according to the rotation order, referring to the gated residual attention fusion module diagram. Figure 6 .

[0054] Based on this, the embodiment of the present invention provides an emotion recognition method that dynamically rotates across sample modalities, such as Figure 2 The specific steps are as follows:

[0055] Step 1: Obtain multimodal data and sentiment classification annotations based on the video; the multimodal data includes text modality, audio modality, and visual modality; the sentiment classification annotations include multimodal unified sentiment classification annotations and three modality independent sentiment classification annotations;

[0056] This paper selects three modalities as the main basis for emotion recognition, namely text modality, visual modality, and audio modality; and divides the independent emotion recognition annotations contained in the three modalities in more detail, that is, each modality itself contains a certain emotion classification label, reference Figure 1 .

[0057] Step 2: Preprocess the multimodal data through the corresponding modal encoder to obtain corresponding preliminary modal feature information; the preliminary modal feature information includes language embedding features, visual embedding features, and acoustic embedding features; since the fusion method adopted in the present invention belongs to the late fusion method, each single modal encoder requires a separate training process, that is, the results obtained by each encoder can be used as the classification result of the single modality for its independent emotional annotation.

[0058] For language embedding feature extraction, this paper uses the traditional natural language processing pre-trainer BERT (Bidirectional Encoder Representation from Transformers) to extract preliminary text features. During the pre-processing sequence, a traditional tokenization approach was used to pad the text sequence length to 50. Furthermore, for the individual text encoders, the optimizer used was AdamW.

[0059] Both visual and audio embedding feature extraction processes use the Transformer encoder for feature preprocessing. To better adapt these features to the Transformer encoder, a positional encoding training module was added. This module adds a [CLS] tag, similar to the Transformer, to help the Transformer encoder learn features at different positions in the sequences. Both encoders are trained using the Adam optimizer.

[0060] Step 3: Process the modal feature information using the gated residual self-attention mechanism to obtain the modal self-attention feature; the modal self-attention feature Including language self-attention features Visual self-attention features Characteristics of auditory self-attention like Figure 3 .

[0061] S3-1. After the modal feature information is transformed into a linear layer dimension, it is put into the gated residual self-attention mechanism to obtain the modal self-attention feature for the first time. and gated residual attention factor The gated residual attention mechanism of the present invention is different from the traditional attention mechanism. In the present invention, the effect of the attention factor is strengthened and the historical attention factor, namely the gated residual attention factor, is retained.

[0062]

[0063] Among them, GatedResiAtten self (·) represents the gated residual self-attention mechanism; is the n-th iteration modal self-attention feature, is the gated residual attention factor of the nth iteration, and N represents the maximum number of iterations.

[0064] Specific modality self-attention features The solution is as follows:

[0065]

[0066] in, is the gated residual attention factor of the nth iteration; the value matrix for The matrix space after linear layer projection; d is used for attention factor Perform fractional scaling.

[0067]

[0068] Update Gate Corresponding to the current attention factor Weight ratio operation; forget gate The weight ratio operation corresponding to the gated residual attention factor retained in the previous iteration. That is, if it is the first iteration, then No Gate forget Forget gate operation, at this time is a simple attention factor; otherwise, as described in the above formula, is the gated residual attention factor after mixing.

[0069] Attention Factor The specific formula is as follows:

[0070]

[0071] Among them, the bond matrix Query Matrix Both The matrix space after linear layer projection. Key matrix Value Matrix Query Matrix Initialization is defined as the preliminary modal feature information X m .

[0072] α shift The variable represents the proportion of α weight after the shift, which is used to adjust the two gating mechanisms for the current gated residual attention factor The following formula shows the effect of shift on α.

[0073]

[0074] The specific shift formula is as follows:

[0075]

[0076] Among them, Mean(·) is the mean function; tanh(·) is the activation function; Gate stable It represents the stabilization gate operation, which is actually a normalization layer used to stabilize the obtained value and prevent the value from exploding. As can be seen from the above two formulas, shift can well reflect the current attention factor and the gated residual factor of the previous layer , thereby affecting the original fixed weight α of 0.5.

[0077] Step 4: Calculate the multimodal fusion order of the three modalities based on the modal self-attention features obtained in step 3. Note that the modal rotation order in the present invention is not run only once, but is continuously and dynamically rotated within a certain rotation cycle in units of samples. Figure 4 、 5 :

[0078] S4-1. Extract the feature information corresponding to the single-modal independent sentiment annotation from the modal self-attention features and aggregate them into a modal score sequence

[0079]

[0080] in, Represents the unimodal independent sentiment classification annotation y corresponding to modality m iis the i-th score sequence; B is the number of sample batches, ∑ represents the summation operator; Match(·) represents the matching operation; Softmax(·) is the activation function that performs the probability distribution calculation so that the value of each sequence position is between 0 and 1, and the sum of all positions is 1.

[0081] The specific implementation of the matching operation is:

[0082] extract Single-modal independent sentiment classification annotation y i The corresponding sample space is recorded as because

[0083] is a two-dimensional tensor, so Batch processing samples (bashsize) are compressed to obtain a modal score sequence;

[0084] S4-2. Calculate the difference ratio ρ between each mode.

[0085]

[0086] in, Represent the score sequences of two different modalities, b and c do not represent specific modal features, i represents the independent sentiment annotation of the corresponding sample; subscript j represents the corresponding relative position of the score sequence; ρ bc It is a general calculation of the difference ratio between two modes, specifically including ρ va ,ρ vl ,ρ al If ρ bc >1, it indicates a fractional sequence D represents the feature dimension.

[0087] S4-3. By comparing the score sequences of the three modes, the main mode P in the current modal rotation cycle is obtained.

[0088]

[0089] Here, Sort(·) indicates sorting the three difference ratios from large to small. If the first two selected difference ratios both contain a certain mode, then the mode is set as the main mode P. Both represent the identity after taking the mean of the sequence values.

[0090] S4-4. Based on the main modality P, the modal self-attention features of the remaining two modalities are used as the secondary modality I i ;

[0091] Minor Mode I iPerform similarity measurement calculations with P respectively, and find the modal feature with the closest similarity to P (that is, the degree of alignment of the two modalities) as the second modal feature.

[0092]

[0093] Among them, M PI Indicates the obtained main mode P and secondary mode I i Similarity measurement matrix, secondary mode I i is a general reference, i represents the labels of the remaining two modes; μ PI is a learnable threshold hyperparameter.

[0094] According to the main modality P and the second modality characteristics, the three modalities can be sorted to obtain the multimodal fusion order.

[0095] Step 5: Based on the multimodal fusion order calculated in step 4, the modal self-attention features are cross-modally fused based on the gated residual attention mechanism;

[0096] Based on the gated residual self-attention mechanism in step 3, the input object is changed and the self-loop iteration part is removed.

[0097] The implementation process of cross-modal fusion based on the gated residual attention mechanism is as follows:

[0098]

[0099] in, Represents the modal self-attention features of the main modality, the second modality, and the third modality respectively, and its value is the corresponding GatedResiAtten cross (·) represents the gated residual attention mechanism, is the gated residual attention factor obtained by fusing the first modality with the second modality;

[0100] But the difference is that the input object of the gated residual self-attention mechanism and In the above formula, due to This is the first fusion, so there is no gated residual attention factor, where the superscript is always 0 to indicate loop iteration; These are the modal self-attention features of each modality after being processed by the gated residual self-attention mechanism. Specifically refers to three modes, but since the modal fusion is based on samples to perform dynamic modal sequence rotation, the present invention uses the main mode P and the secondary mode I i It is worth noting that the key matrix in the gated residual self-attention mechanism is Query Matrix Value Matrix Both The matrix space after linear layer projection, but in the above gated residual attention mechanism, the key matrix Value Matrix The input variable is changed to the attention matrix space of the secondary modality, and the query matrix The input is changed to the attention matrix space of the main modality. is the gated residual attention factor obtained by fusing the first modality with the second modality; Represents the final result of the three-modal fusion.

[0101] Step 6: Input the multimodal fusion results into the classifier to obtain the emotion recognition results.

[0102] The classifier uses a traditional perceptron structure, implemented by superimposing a linear layer (nn.linear) and an activation function (nn.GELU). The activation function is used to calculate the probability weight of the sample classification, and the linear layer is used to map the resulting feature space into the sample label space and integrate it into a numerical value.

[0103] In the present invention, the sentiment classification result of recognition does not use a simple loss function, but a specific objective function designed for this method, as follows:

[0104] L all =L pred +βL uni

[0105] Among them, L pred express The loss comparison between the predicted classification results and the actual classification results input into the classifier; L uni Comparison of the loss between the classification results obtained by putting the modal self-attention features of each modality obtained above into a single-modal independent sentiment classifier and the actual classification results; β is the weight ratio.

[0106] Finally, the proposed model was tested on two multimodal emotion-related and industry-recognized public datasets, and related ablation experiments were performed. The datasets and parameter settings are as follows:

[0107] In sarcasm recognition, the present invention uses the Chinese multimodal sentiment analysis datasets CH-SIMS and CH-SIMS2.0 to verify the effectiveness of the model of the present invention.

[0108] (1) CH-SIMS: Publicly released in 2020, it is a Chinese unimodal and multimodal sentiment classification dataset that contains 2,281 refined video clips in the wild with multimodal and independent unimodal annotations, allowing researchers to study the interaction between modalities or perform unimodal sentiment analysis using independent unimodal sentiment classification annotations.

[0109] (2) CH-SIMS2.0: This dataset is a supplementary and advanced version of CH-SIMS and was released in 2022. Based on CH-SIMS, this dataset focuses more on the collection of expressive audio and visual behaviors, and the diverse video scenes make it contain not only conventional emotion classification, but also other instances with ambiguous, ironic, and metaphorical text modalities.

[0110] Due to the specificity of single-modal independent sentiment classification annotation, this paper selects the above two datasets for testing. In terms of evaluation methods, this paper adopts common evaluation methods in multi-classification, namely, the accuracy of two-class classification acc-2, the F1 score of two-class classification, the accuracy of three-class classification acc-3, the accuracy of five-class classification acc-5, the mean absolute error (MAE), and the correlation coefficient (Corr).

[0111] The fusion method used in the model is late fusion based on the gated residual attention mechanism, that is, the three unimodal encoders require additional training parameters. For the audio encoder and visual encoder, the training batch size is 64, the learning rate is 0.0001, and the number of iterations is 300; for the text encoder, the training batch size is 64, the learning rate is 0.00001, and the number of iterations is 150. The model uses the Adamax optimizer as the target optimizer, with a training batch size of 32, a learning rate of 0.0001, and a number of iterations of 50.

[0112] Experimental results analysis:

[0113] The experimental evaluation results of the proposed method outperform those of similar models on various datasets, demonstrating the rationality and reliability of the proposed emotion recognition method with dynamic cross-modal rotation based on a gated residual attention mechanism. The average binary classification accuracy on the two datasets reached 84.03% and 83.17%, respectively; the three-class classification accuracy was 73.09% and 77.66%, respectively. This significantly outperforms other similar emotion recognition models, such as LF_DNNv2 (acc-2: 77.24%), AF_Self-MM (acc-2: 80.6%), BIMHA (acc-2: 82.71%, acc-3: 69.23%), MMML (acc-2: 82.93%, acc-3: 69.37%), ALMT (acc-2: 81.19%, acc-3: 68.93%), and ConFEDE (acc-2: 82.23%, acc-3: 70.15%).

Claims

1. A method for emotion recognition with dynamic rotation across sample modalities, characterized by The method comprises the following steps: Step 1: Obtain multimodal data and corresponding sentiment classification annotations based on the video; the multimodal data includes text modality, audio modality, and visual modality; the sentiment classification annotations include multimodal unified sentiment classification annotations and independent sentiment classification annotations for the three modalities; Step 2: Preprocess the multimodal data through the corresponding modal encoder to obtain the corresponding preliminary modal feature information X m , m=l,v,a represent language embedding features, visual embedding features, and audio embedding features respectively; Step 3: Preliminary modal feature information X m Perform gated residual self-attention mechanism processing to obtain modal self-attention features Step 4: Modal self-attention features obtained in step 3 Get the multimodal fusion order; specifically: S4-1. Extract the feature information corresponding to the single-modal independent sentiment annotation from the modal self-attention features and aggregate them into a modal score sequence in, Represents the unimodal independent sentiment classification annotation y corresponding to modality m i The i-th score sequence of ; B is the number of sample batches, ∑ represents the summation operator; Match(·) represents the matching operation; Softmax(·) is the activation function; S4-2. Calculate the difference ratio ρ between two modes bc ; in, They represent the score sequences of two different modes respectively, and j represents the corresponding relative position of the score sequence; S4-3. By comparing the difference ratios ρ of the three modes bc , get the main mode P in the current mode rotation period; Wherein, Sort(·) means that the first two difference ratios are taken after sorting the three difference ratio means in descending order, and the common mode in these two difference ratios is the main mode P; S4-4. Based on the main modality P, the modal self-attention features of the remaining two modalities are used as the secondary modality I i ; Minor Mode I i Calculate the similarity between each modality and the main modality P, and find the modal feature with the closest similarity to P as the second modal feature; According to the main mode P and the second mode, the three modes can be sorted to obtain the multimodal fusion order; Step 5: Based on the multimodal fusion order of step 4, the modal self-attention features Perform cross-modal fusion; Step 6: Input the cross-modal fusion result of step 5 into the classifier to obtain the emotion recognition result.

2. The method according to claim 1, characterized in that Step three is: The preliminary modal feature information X m After multiple iterations of gated residual self-attention mechanism processing, deeper modal self-attention features are obtained. and gated residual attention factor Among them, GatedResiAtten self (·) represents the gated residual self-attention mechanism; Initialize the nth iteration modal self-attention feature is the gated residual attention factor of the nth iteration, and N represents the maximum number of iterations.

3. The method according to claim 2, characterized in that In step 3, the gated residual self-attention mechanism is specifically: in, is the gated residual attention factor of the nth iteration, when initialized Value Matrix for The matrix space after linear layer projection; d is used for attention factor Perform fractional scaling; in, is the attention factor; the key matrix Query Matrix Both Matrix space after linear layer projection; initialize key matrix Value Matrix Query Matrix Defined as preliminary modal feature information X m ; α shift is the α weight ratio after shift.

4. The method according to claim 3, characterized in that In step three, Mean(·) is the mean function; tanh(·) is the activation function; Gate stable Indicates stable door operation.

5. The method according to claim 1, characterized in that Similarity M in step S4-4 PI The calculation is specifically: Among them, μ PI is a learnable threshold hyperparameter.

6. The method according to claim 1, characterized in that Step 5 specifically uses the gated residual attention mechanism to achieve cross-modal fusion; the implementation process is as follows: in, Represents the modal self-attention features of the main modality, the second modality, and the third modality respectively, and its value is the corresponding GatedResiAtten cross (·) represents the gated residual attention mechanism, is the gated residual attention factor obtained by fusing the primary modality with the secondary modality.

7. An emotion recognition system implementing the method according to any one of claims 1 to 6, characterized in that The system includes the following: The data acquisition module is responsible for acquiring multimodal data and sentiment classification annotations based on the video; the multimodal data includes text modality, audio modality, and visual modality; The sentiment classification annotations include unified multimodal sentiment classification annotations and independent sentiment classification annotations of the three modalities; The feature extraction module is responsible for preprocessing the multimodal data to obtain the corresponding preliminary modal feature information; the modal feature information includes language embedding features, visual embedding features, and audio embedding features; The modal dynamic fusion module is responsible for measuring the richness of emotional semantic information of each modality, obtaining the multimodal fusion order, and finally performing multimodal fusion according to the multimodal fusion order; The recognition module is responsible for inputting the multimodal fusion results into the classifier to obtain the emotion recognition results.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 6.

9. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented.

Images