Multi-modal emotion recognition method based on cross-modal homologous interaction

Abstract

The invention relates to an emotion recognition method based on cross-modal homologous interaction, and aims to solve the problems of single inter-modal feature interaction, loss of specific emotion features and the like in an existing multi-modal emotion recognition model. According to the method, multi-scale emotion features in voice, text and video modalities are extracted through a multi-scale encoder, and the aggregation of similar emotion features is enhanced by using a center loss function; a parameter-shared Transform encoder and a cross attention mechanism are adopted to extract the identity emotion features between the modals, and meanwhile, an orthogonal loss function is introduced to reserve the specific emotion features of all the modals; and finally, realizing deep interaction of identity and specificity characteristics through a double-gating fusion and enhanced global interaction module, and constructing emotion representation with higher discrimination capability. According to the method, the unweighted accuracy (UA) on IEMOCAP, RAVDESS and eNERFACE '05 data sets is averagely improved by 1.7%-5.1%, and the method can be widely applied to the fields of mental health monitoring, intelligent customer service, human-computer interaction and the like.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and emotional computing, and specifically relates to a multimodal emotion recognition method based on cross-modal same-sex interaction, which is particularly suitable for emotion analysis of multimodal data such as speech, text and video. Background Art

[0002] With the development of artificial intelligence technology, multimodal emotion recognition, by integrating multiple information sources such as speech, text, and vision, has shown important application value in areas such as mental health monitoring, intelligent customer service, and human-computer interaction. Traditional emotion recognition methods have the following technical problems:

[0003] 1. Loss of modality-specific features: Existing methods such as parameter-sharing encoding structures can extract common features across modalities, but they suppress the unique emotional expression features of each modality, such as intonation changes in speech and semantic emphasis in text.

[0004] 2. Simplification of cross-modal interaction: Mainstream methods over-rely on the fusion of common features and ignore the complementary value of specific features between different modalities, resulting in insufficient differentiation of emotional representation.

[0005] 3. Overfitting on small data: On limited-scale datasets such as IEMOCAP, complex models are prone to overfitting, resulting in decreased generalization performance.

[0006] Existing solutions using non-shared Transformer structures retain some modal characteristics but suffer from high feature redundancy. Other solutions using parameter-sharing structures mitigate overfitting but lack the ability to model multi-scale features. Therefore, an innovative approach that simultaneously addresses the problems of feature loss, single interactions, and overfitting is urgently needed. Summary of the Invention

[0007] This application provides a multimodal emotion recognition method based on cross-modal same-sex and different-sex interaction, which significantly improves the accuracy of emotion recognition and reduces the risk of overfitting through multi-scale feature extraction, same-sex and different-sex feature separation and multi-level fusion.

[0008] In a first aspect, the present application provides a multimodal emotion recognition method based on cross-modal same-sex and opposite-sex interaction, the method comprising the following steps:

[0009] Step 1: Use a multi-scale encoder to extract multi-scale emotional features of speech, text, and video modalities respectively;

[0010] Step 2: Optimize the features of each modality through the central loss function to enhance the aggregation of similar emotional features;

[0011] Step 3: Use the parameter-sharing Transformer encoder and cross-attention mechanism to extract the same sentiment features between modalities, while using the orthogonal loss function to retain the specific sentiment features of each modality;

[0012] Step 4: Through the homogeneity and heterogeneity feature fusion module, we can achieve deep interaction between identity and specificity features and build a more discriminative sentiment representation.

[0013] Step 5: Classify based on the fused emotional features and output the final emotion recognition results.

[0014] Furthermore, the step 1 specifically includes the following steps:

[0015] Step 1.1: Preprocess the speech, text, and video modalities separately to extract initial features;

[0016] Step 1.2: Use multi-scale dilated convolution to extract multi-scale features of each modality, where convolution kernels of different scales (1×3, 1×5, 1×7, 1×9) are used to extract time domain features respectively;

[0017] Step 1.3: Align the dimensions of features of different modalities through the fully connected layer, and use Tanh and Sigmoid activation functions to enhance emotional expression.

[0018] Furthermore, the step 2 specifically includes the following steps:

[0019] Step 2.1: Calculate the distance between each modal feature and the center of the emotion category to which it belongs, and optimize the feature distribution;

[0020] Step 2.2: By minimizing the center loss function, similar emotional features are clustered in the feature space.

[0021] Furthermore, the step 3 specifically includes the following steps:

[0022] Step 3.1: Use the parameter-sharing Transformer encoder to encode the features of each modality and extract long-sequence emotional features;

[0023] Step 3.2: Inter-modal feature interaction and alignment through parameter-sharing cross-attention mechanism;

[0024] Step 3.3: Use the orthogonal loss function to constrain the orthogonality of the eigenvectors of different modalities and reduce the redundant information between modalities.

[0025] Furthermore, the step 4 specifically includes the following steps:

[0026] Step 4.1: In the dual-gated fusion module, global sentiment information of identity and specificity features is extracted through global pooling and then concatenated and fused.

[0027] Step 4.2: In the enhanced global interaction module, the ReLU linear attention mechanism is used to perform local and multi-scale feature interactions to enhance emotion-related features.

[0028] Step 4.3: Concatenate the outputs of the dual-gated fusion module and the enhanced global interaction module to obtain the final sentiment representation.

[0029] Furthermore, the step 5 specifically includes the following steps:

[0030] Step 5.1: Input the fused sentiment features into the fully connected layer for classification;

[0031] Step 5.2: Use the Softmax function to output the final emotion category probability.

[0032] In a second aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described above is implemented.

[0033] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the method described above when the computer program is executed by a processor.

[0034] The above technical solution of this application has the following advantages:

[0035] The first aspect of this application provides a method for emotion recognition based on cross-modal same-sex interaction. Through a multi-scale feature extraction module and a same-sex feature fusion module, this method can more comprehensively mine emotional features from speech, text, and video modalities, significantly improving the accuracy of emotion recognition. Experiments on the IEMOCAP, RAVDESS, and eNTERFACE'05 datasets demonstrate that this method achieves an average improvement of 1.7%-5.1% in unweighted accuracy (UA) over existing state-of-the-art methods.

[0036] Effective extraction of inter-modal identity and specific features: This application uses a shared cross-modal encoder to extract common emotional features between modalities, while retaining the specific emotional information of each modality through an orthogonal loss function. This solves the problem of traditional methods over-relying on common features and ignoring modal uniqueness, making emotional representation more comprehensive and discriminative.

[0037] Effectively mitigate overfitting: This application uses a parameter-sharing structure and a central loss function to enhance the model's generalization capabilities and reduce the risk of overfitting when training on small datasets. Experimental results show that this method exhibits stable performance across different datasets and is suitable for multimodal emotion recognition tasks in real-world scenarios.

[0038] It can be understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 The overall architecture diagram of the model for the multimodal emotion recognition method based on cross-modal same-sex and opposite-sex interaction provided in this application;

[0041] Figure 2 The structure diagram of the multi-scale encoder provided by this application;

[0042] Figure 3 A diagram of the shared cross-modal encoder structure provided for this application;

[0043] Figure 4 Transformer structure diagram provided for this application;

[0044] Figure 5 Cross Attention calculation flow chart provided for this application;

[0045] Figure 6 Schematic diagram of the same-different feature fusion module provided in this application. DETAILED DESCRIPTION

[0046] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0047] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0048] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0049] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. "Multiple" means "two or more."

[0050] The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0051] In the field of multimodal emotion recognition, existing methods typically employ parameter-sharing encoding structures to extract common cross-modal features to mitigate overfitting caused by insufficient data. However, this single-scale interaction approach often overlooks the unique emotional characteristics of different modalities, resulting in insufficient discriminability in emotion representation and limiting model performance. Furthermore, traditional methods fail to fully exploit multi-scale features when fusing intermodal features, further impacting the accuracy of emotion recognition.

[0052] To address the above issues, this application proposes a multimodal emotion recognition method based on cross-modal homosexual interaction, which significantly improves the performance of emotion recognition through multi-scale feature extraction, separation of inter-modal identity and specificity features, and multi-level feature fusion. Figure 1 As shown, a multimodal emotion recognition method based on cross-modal same-sex and opposite-sex interaction of the present application includes the following steps:

[0053] Step 1: The three modal data of video, audio and text are preprocessed and input into the multi-scale encoder to output the multi-scale features of each modality.

[0054] Step 2: Input the multi-scale features of the three modalities of video, audio and text into the sameness and difference feature extraction module, and output the identity features and specificity features of the three modalities of video, audio and text.

[0055] Step 3: Input the identity features and specific features of the three modalities of video, audio and text into the same-different feature fusion module, and output the fused video, audio and text tri-modal features.

[0056] Step 4: Input the fused video, audio, and text trimodal features into the classifier to obtain the emotion recognition results, and optimize the model through the loss function.

[0057] The step 1 specifically includes the following steps:

[0058] Step 1.1: Use the HuBERT pre-trained model to extract speech and audio pre-trained features, use the BERT pre-trained model to extract text pre-trained features, and use the ViT pre-trained model to extract pre-trained features of each frame in the video, obtaining pre-trained features of the three modalities: video, audio, and text.

[0059] Step 1.2: If Figure 2 As shown in the figure, 1D dilated convolution with different convolution kernel sizes is used to extract multi-scale features from the video, audio and text pre-training features. According to formula (1), two groups of different scale features X are obtained through two parallel paths on the three modalities. k , where X is the pre-trained features of the three input modalities (speech / text / video), S k is the convolution kernel size (s k =3,5,7,9), and the expansion rate is 2.

[0060]

[0061] Step 1.3: Concatenate the two sets of different scale features (X1, X2, X3, X4) obtained in step 1.2 to obtain multi-scale features, and align the feature vectors of different modes through the fully connected layer to unify them to 384 dimensions to obtain two multi-scale features X L and X R .

[0062] Step 1.4: Get two multi-scale features X from step 1.3 L and X R , use Tanh and Sigmoid activation functions to L and X R After the operation, multiplication is performed to enhance the emotional expression and obtain the final multi-scale feature output F i .

[0063] The step 2 specifically includes the following steps:

[0064] Step 2.1: Output the multi-scale features F obtained in step 1.4 on the three modalities i According to formula (2), the central loss function optimizes the emotional representation of each modality, so that the same type of emotional features are more closely clustered in the feature space, and the emotional features of different types are further separated. i Feature Center The calculation is shown in formula (3).

[0065]

[0066] Step 2.2: Calculate the orthogonality between the modalities according to the orthogonal loss function of formula (4), so that the correlation between the feature vectors of different modalities is reduced, thereby better extracting the specific emotional information in the three modalities, where F′ a , F′ t , F′ v Represent the multi-scale features of audio, text and video optimized by the center loss function.

[0067]

[0068] Step 2.3: Concatenate the specific emotional information of the three modalities obtained in step 2.2 to obtain the inter-modal specific feature F d .

[0069] Step 2.4: If Figure 3 As shown, the features F′ of the three modalities of video, audio and text optimized by the central loss function in step 2.1 are v , F′ a , F′ t The input parameters share the Transformer encoder, and the Transformer structure is as follows Figure 4 As shown in Figure 2, parameter sharing includes the Transformer's multi-head attention mechanism and the weight sharing of the feedforward neural network. It uses the Transformer's powerful long-sequence feature modeling capability to capture the long-sequence emotional features F″ contained in videos, audio, and text. v 、F″ a 、F″ t .

[0070] Step 2.5: Use parameter sharing bidirectional cross-attention to the sentiment feature F″ obtained in step 2.3 v 、F″ a 、F″ t For alignment, the Cross Attention calculation process is as follows Figure 5As shown, the pairwise alignment features F between modalities are obtained according to formulas (5), (6), and (7) respectively. av 、F at 、F tv .

[0071]

[0072] Step 2.6: Align the features F between the two modes obtained in step 2.4 av 、F at 、F tv Splice and get the inter-modal identity feature F c .

[0073] The step 3 specifically includes the following steps:

[0074] Step 3.1: If Figure 6 As shown, the inter-modality specific feature F obtained in step 2 is d and intermodal identity feature F c They are sent to the channel pooling and time domain pooling modules respectively, and the global information representation of the inter-modal identity and specific features in the channel and time domain is obtained according to formulas (8) and (9).

[0075]

[0076] Step 3.2: Pool the global features on the channel obtained by channel pooling in step 3.1 and Splicing is performed, and the global features in the time domain obtained by time domain pooling are and Splice and get the fused channel global feature F C and the time domain global feature F T .

[0077] Step 3.3: The channel global feature F obtained in step 3.2 C After transposition, the global feature F in the time domain T Splicing, and then using the fully connected layer for global interaction to obtain the time-channel comprehensive global feature F G , in order to obtain more comprehensive global information of channels and time.

[0078] Step 3.4: According to formulas (10) and (11), the time-channel integrated global feature F obtained in step 3.3 is G , divided according to the channel dimension C and the time domain dimension T, where F G [:C] means take F G The first C channel dimensions of F G [C:T] means take F GFrom the Cth channel to the Tth time domain dimension. Then use the Sigmoid activation function to obtain the channel weight CW and time domain weight TW to strengthen the emotional features in the channel and time domain.

[0079] CW=Sigmoid(F G [:C])#Formula (10)

[0080] TW=Sigmoid(F G [C:T])#Formula (11)

[0081] Step 3.5: Inter-modality specific features F d and intermodal identity feature F c After splicing, they are multiplied with CW and TW respectively to strengthen the useful features in the spatial and temporal domains and suppress the useless features to obtain the final global fusion features.

[0082] Step 3.6: Set the identity feature F c and specific features F d The concatenation is performed on the channel dimension and the query matrix Q, key matrix K and value matrix V are generated through the fully connected layer.

[0083] Step 3.7: Use 3×3 and 5×5 small kernel convolutions to perform local token aggregation on Q, K, and V obtained in step 3.6 to achieve local information interaction between adjacent tokens and generate multi-scale token representations.

[0084] Step 3.8: According to formula (12), the ReLU linear attention mechanism is used to perform global feature fusion in the generated multi-scale tokens.

[0085]

[0086] Step 3.9: Concatenate the features obtained by fusing tokens at different scales in step 3.8 to obtain multi-scale fusion features.

[0087] Step 3.10: Concatenate the multi-scale fusion features obtained in step 3.9 with the global fusion features obtained in step 3.5 to obtain the final homogeneity fusion features.

[0088] The step 4 specifically includes the following steps:

[0089] Step 4.1: Input the same-different fusion features obtained in step 3 into the fully connected layer and output the sentiment classification probability through the Softmax function.

[0090] Step 4.2: According to formula (13), use the cross entropy loss function L c , center loss function L centerWith the orthogonal loss function L orth The loss function L composed of weighted components updates the model.

[0091] L=L c +αL center +βL orth #(13)

[0092] This application proposes a cross-modal emotion recognition method based on same-sex interaction. Through a multi-scale feature extraction module and a same-sex feature fusion module, it can more comprehensively mine emotional features from speech, text, and video modalities, significantly improving the accuracy of emotion recognition. Experiments on the IEMOCAP, RAVDESS, and eNTERFACE'05 datasets demonstrate that this method achieves an average improvement of 1.7%-5.1% in unweighted accuracy (UA) over existing state-of-the-art methods.

[0093] Effective extraction of inter-modal identity and specific features: This application uses a shared cross-modal encoder to extract common emotional features between modalities, while retaining the specific emotional information of each modality through an orthogonal loss function. This solves the problem of traditional methods over-relying on common features and ignoring modal uniqueness, making emotional representation more comprehensive and discriminative.

[0094] Effectively mitigate overfitting: This application uses a parameter-sharing structure and a central loss function to enhance the model's generalization capabilities and reduce the risk of overfitting when training on small datasets. Experimental results show that this method exhibits stable performance across different datasets and is suitable for multimodal emotion recognition tasks in real-world scenarios.

[0095] An embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described above is implemented.

[0096] In applications, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0097] In applications, the memory may be an internal storage unit of an electronic device in some embodiments, such as a hard disk or memory of the electronic device. In other embodiments, the memory may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (Smart Media Card, SMC), a secure digital (SD) card, a flash card (Flash Card), etc. The memory may also include both an internal storage unit of the electronic device and an external storage device. The memory is used to store an operating system, application programs, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program. The memory may also be used to temporarily store data that has been output or is to be output.

[0098] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described above is implemented.

[0099] The present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device that can carry the computer program code to an electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0100] Those skilled in the art will appreciate that the devices and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] In the embodiments provided herein, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interface, or the devices may be indirectly coupled or communicated in some manner, whether electrical, mechanical, or other.

[0102] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A multimodal emotion recognition method based on cross-modal same-sex and opposite-sex interaction, characterized by: The method comprises the following steps: Step 1: Use a multi-scale encoder to extract multi-scale emotional features of speech, text, and video modalities respectively; Step 2: Optimize the features of each modality through the central loss function to enhance the aggregation of similar emotional features; Step 3: Use the parameter-sharing Transformer encoder and cross-attention mechanism to extract the same sentiment features between modalities, while using the orthogonal loss function to retain the specific sentiment features of each modality; Step 4: Through the homogeneity and heterogeneity feature fusion module, we can achieve deep interaction between identity and specificity features and build a more discriminative sentiment representation. Step 5: Classify based on the fused emotional features and output the final emotion recognition result.

2. A multimodal emotion recognition method based on cross-modal same-sex and opposite-sex interaction according to claim 1, characterized in that: The step 1 specifically includes the following steps: Step 1.1: Preprocess the speech, text, and video modalities separately to extract initial features; Step 1.2: Use multi-scale dilated convolution to extract multi-scale features of each modality, where convolution kernels of different scales (1×3, 1×5, 1×7, 1×9) are used to extract time domain features respectively; Step 1.3: Align the dimensions of features of different modalities through the fully connected layer, and use Tanh and Sigmoid activation functions to enhance emotional expression.

3. A multimodal emotion recognition method based on cross-modal same-sex and opposite-sex interaction according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: Calculate the distance between each modal feature and the center of the emotion category to which it belongs, and optimize the feature distribution; Step 2.2: By minimizing the center loss function, similar emotional features are clustered in the feature space.

4. A multimodal emotion recognition method based on cross-modal same-sex and opposite-sex interaction according to claim 1, characterized in that: The step 3 specifically includes the following steps: Step 3.1: Use the parameter-sharing Transformer encoder to encode the features of each modality and extract long-sequence emotional features; Step 3.2: Inter-modal feature interaction and alignment through parameter-sharing cross-attention mechanism; Step 3.3: Use the orthogonal loss function to constrain the orthogonality of the eigenvectors of different modalities and reduce the redundant information between modalities.

5. The multimodal emotion recognition method based on cross-modal same-sex and opposite-sex interaction according to claim 1 is characterized in that: The step 4 specifically includes the following steps: Step 4.1: In the dual-gated fusion module, global pooling is used to extract the global sentiment information of identity and specificity features, and then concatenate and fuse them. Step 4.2: In the enhanced global interaction module, the ReLU linear attention mechanism is used to perform local and multi-scale feature interactions to enhance emotion-related features. Step 4.3: Concatenate the outputs of the dual-gated fusion module and the enhanced global interaction module to obtain the final sentiment representation.

6. A multimodal emotion recognition method based on cross-modal same-sex and opposite-sex interaction according to claim 1, characterized in that: The step 5 specifically includes the following steps: Step 5.1: Input the fused sentiment features into the fully connected layer for classification; Step 5.2: Use the Softmax function to output the final emotion category probability.

7. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images