Emotion recognition method and device based on contrast feature fusion, equipment and medium

Abstract

The application discloses a sentiment recognition method and device based on comparative feature fusion, equipment and medium. The method can include: respectively extracting speech single-modal features and expression single-modal features; extracting shared information of the speech single-modal features and the expression single-modal features through comparative learning, obtaining a comparative loss function; extracting unique information of the speech single-modal features and the expression single-modal features through orthogonal constraint, obtaining an orthogonal loss function; performing sentiment classification according to the shared information and the unique information, obtaining a total loss function; and obtaining final sentiment classification by minimizing the total loss function. The application effectively extracts and fuses the shared information and the unique information between the speech and expression modalities.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of emotion recognition, and more particularly, to an emotion recognition method, device, equipment and medium based on contrast feature fusion. BACKGROUND

[0002] Emotion recognition is a cross-disciplinary research field that combines computer science, psychology, sociology, etc., which uses computers to automatically recognize human emotional states. With the explosive growth of multimedia data and the rapid development of artificial intelligence, emotion recognition has been widely applied in intelligent driving, distance learning, smart home, health monitoring, tourism recommendation, intelligent robots, etc. However, due to the complexity of emotional expression and perception, emotion recognition is a highly challenging problem. In addition, humans usually express emotions through multiple modalities, such as speech, visual expressions, etc., and there are complex interactions between multiple modalities. In order to solve these problems, multi-modal emotion recognition is introduced to model the complex interactions between modalities.

[0003] In multi-modal emotion recognition, the fusion method is particularly important. The current popular fusion method is feature layer fusion, which first extracts the features of each modality, then fuses these features, and finally realizes the classification of emotions. Although this fusion method can extract shared information between modalities, it does not fully extract unique information between each modality, reducing the performance of emotion recognition. On the other hand, existing literature has confirmed that the unique information of each modality can supplement the shared information between modalities, and the fusion of unique information and shared information can provide a more comprehensive multi-modal representation.

[0004] Therefore, it is necessary to develop an emotion recognition method, device, equipment and medium based on contrast feature fusion.

[0005] The information disclosed in the background section of the present application is only intended to deepen the understanding of the general background of the present application and should not be considered as recognition or in any form as implying that this information constitutes prior art known to those skilled in the art. SUMMARY

[0006] The present application proposes an emotion recognition method, device, equipment and medium based on contrast feature fusion, which effectively extracts and fuses the shared information and unique information between speech and expression modalities.

[0007] In a first aspect, the present disclosure provides an emotion recognition method based on contrast feature fusion, comprising:

[0008] extracting speech single-modality features and expression single-modality features respectively;

[0009] extracting shared information of the speech single-modal feature and the expression single-modal feature through contrastive learning, obtaining a contrastive loss function;

[0010] extracting unique information of the speech single-modal feature and the expression single-modal feature through orthogonal constraint, obtaining an orthogonal loss function;

[0011] performing emotion classification according to the shared information and the unique information, obtaining a total loss function;

[0012] obtaining final emotion classification by minimizing the total loss function.

[0013] Preferably, extracting shared information of the speech single-modal feature and the expression single-modal feature through contrastive learning, obtaining a contrastive loss function comprises:

[0014] taking single-modal emotion features and fused multi-modal features of the same emotion category as positive samples, and taking others as negative samples, to obtain the shared information of the speech single-modal feature and the expression single-modal feature;

[0015] calculating the contrastive loss function according to the shared information and the speech single-modal feature and the expression single-modal feature.

[0016] Preferably, the contrastive loss function is:

[0017]

[0018] wherein, denotes a set of emotion categories, denotes an L1 norm of a vector, denotes modal shared information of an i-th emotion category, denotes a corresponding speech modal feature and an expression modal feature, respectively, denotes mutual information, , , , , denotes a temperature parameter, denotes a cosine similarity, wherein, and denote two vectors with equal lengths, respectively, denotes a dot product, denotes an L2 norm of a vector.

[0019] Preferably, extracting unique information of the speech single-modal feature and the expression single-modal feature through orthogonal constraint, obtaining an orthogonal loss function comprises:

[0020] calculating the modal unique information according to the modal shared information;

[0021] According to the modality-specific information, the orthogonal loss function is calculated.

[0022] Preferably, the modality-specific information is calculated by the following formula:

[0023]

[0024]

[0025] wherein, represents the specific information of the expression modality, represents the specific information of the speech modality.

[0026] Preferably, the orthogonal loss function is:

[0027]

[0028] wherein, represents the specific information of the expression modality, represents the specific information of the speech modality.

[0029] Preferably, the total loss function is:

[0030]

[0031] wherein, represents a hyperparameter, represents a cross-entropy loss function.

[0032] As a specific implementation manner of the embodiments of the present disclosure,

[0033] In a second aspect, the embodiments of the present disclosure further provide a sentiment recognition device based on contrast feature fusion, comprising:

[0034] A single-modal feature extraction module extracts speech single-modal features and expression single-modal features respectively.

[0035] A contrast feature fusion module extracts shared information of the speech single-modal features and the expression single-modal features through contrast learning, and obtains a contrast loss function.

[0036] The specific information of the speech single-modal features and the expression single-modal features is extracted through orthogonal constraint, and an orthogonal loss function is obtained.

[0037] A sentiment classification module performs sentiment classification according to the shared information and the specific information, obtains a total loss function, and obtains a final sentiment classification by minimizing the total loss function.

[0038] Preferably, the shared information between the speech monomodal features and the facial expression monomodal features is extracted through contrastive learning, and the contrastive loss function is obtained as follows:

[0039] Using single-modal emotion features and fused multimodal features of the same emotion category as positive samples and others as negative samples, the shared information between the speech single-modal features and the facial expression single-modal features is obtained;

[0040] The contrast loss function is calculated based on the shared information, the speech monomodal features, and the facial expression monomodal features.

[0041] Preferably, the contrast loss function is:

[0042]

[0043] in, A set representing emotion categories. Describes the L1 norm of a vector. Indicates the first Modal sharing information across emotion categories and These represent the corresponding speech modality features and facial expression modality features, respectively. Indicates mutual information, , , Indicates temperature parameter, Represents cosine similarity. ,in, and Let each represent two vectors of equal length. Represents the dot product. This represents the L2 norm of a vector.

[0044] Preferably, the unique information of the speech monomodal features and the facial expression monomodal features is extracted through orthogonal constraints to obtain the orthogonal loss function, which includes:

[0045] Based on the modality sharing information, the modality-specific information is calculated;

[0046] The orthogonal loss function is calculated based on the modality-specific information.

[0047] Preferably, the modality-specific information is calculated using the following formula:

[0048]

[0049]

[0050] in, Specific information indicating facial expression modality, represents the unique information of the speech modality.

[0051] Preferably, the orthogonal loss function is:

[0052]

[0053] wherein, represents the unique information of the expression modality, represents the unique information of the speech modality.

[0054] Preferably, the total loss function is:

[0055]

[0056] wherein, represents the hyperparameter, represents the cross-entropy loss function.

[0057] In a third aspect, the present disclosure also provides an electronic device, which comprises:

[0058] a memory storing executable instructions;

[0059] a processor running the executable instructions in the memory to implement the emotion recognition method based on contrast feature fusion.

[0060] In a fourth aspect, the present disclosure also provides a computer readable storage medium storing a computer program, which is executed by a processor to implement the emotion recognition method based on contrast feature fusion.

[0061] The beneficial effects are that:

[0062] The present application adopts contrast learning to extract shared information between speech and expression modalities, and adopts orthogonal constraints to extract unique information of speech and expression modalities, respectively. These extracted shared information and unique information are fused to improve the performance of emotion recognition.

[0063] The method and device of the present application have other characteristics and advantages, which will be apparent from or set forth in the accompanying drawings and the following detailed description, which together serve to explain certain principles of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0064] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout the figures, and in which: EXEMPLARY EMBODIMENTS OF THE INVENTION

[0065] Figure 1 A flow chart showing steps of a method of emotion recognition based on contrastive feature fusion according to an embodiment of the present application is shown.

[0066] Figure 2 A schematic diagram showing a method of emotion recognition based on contrastive feature fusion according to an embodiment of the present application is shown.

[0067] Figure 3 A recognition result of each emotion category according to an embodiment of the present application is shown.

[0068] Figure 4 A block diagram of an emotion recognition device based on contrastive feature fusion according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0069] Preferred embodiments of the present application will be described in more detail below. Although the following describes preferred embodiments of the present application, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0070] To facilitate understanding of the scheme and effects of the embodiments of the present application, four specific application examples are given below. Those skilled in the art should understand that the examples are only for the convenience of understanding the present application, and any specific details thereof are not intended to limit the present application in any way.

[0071] Example 1

[0072] Figure 1 A flow chart showing steps of a method of emotion recognition based on contrastive feature fusion according to an embodiment of the present application is shown.

[0073] As Figure 1 shown, the method of emotion recognition based on contrastive feature fusion includes: step 101, respectively extracting speech single-modal features and expression single-modal features; step 102, extracting shared information of the speech single-modal features and the expression single-modal features through contrastive learning, obtaining a contrastive loss function; step 103, extracting unique information of the speech single-modal features and the expression single-modal features through orthogonal constraint, obtaining an orthogonal loss function; step 104, performing emotion classification according to the shared information and the unique information, obtaining a total loss function; and step 105, obtaining a final emotion classification by minimizing the total loss function.

[0074] In one example, extracting shared information of the speech single-modal features and the expression single-modal features through contrastive learning, obtaining a contrastive loss function includes:

[0075] The single-modal sentiment feature of the same sentiment category and the fused multi-modal feature are taken as positive samples, and others are taken as negative samples, to obtain shared information of the speech single-modal feature and the expression single-modal feature;

[0076] According to the shared information and the speech single-modal feature and the expression single-modal feature, a contrast loss function is calculated.

[0077] In one example, the contrast loss function is:

[0078]

[0079] wherein, denotes a set of sentiment categories, denotes an L1 norm of a vector, denotes modal shared information of an i-th sentiment category, denotes a corresponding speech modal feature and an expression modal feature, denotes mutual information, , , denotes a temperature parameter, denotes a cosine similarity, wherein, and denote two vectors with equal lengths, denotes a dot product, denotes an L2 norm of a vector. In one example, the specific information of the speech single-modal feature and the expression single-modal feature is extracted by orthogonal constraint, and an orthogonal loss function is obtained, including:

[0080] According to the modal shared information, modal specific information is calculated;

[0081] According to the modal specific information, an orthogonal loss function is calculated.

[0082] In one example, the modal specific information is calculated by the following formula:

[0083]

[0084]

[0085]

[0086] wherein, denotes specific information of the expression modal, denotes specific information of the speech modal.

[0087] In one example, the orthogonal loss function is:

[0088] ​​

[0089] wherein, represents the unique information of the expression modality, represents the unique information of the speech modality.

[0090] In one example, the total loss function is:

[0091]

[0092] wherein, represents a hyperparameter, represents a cross-entropy loss function.

[0093] Specifically, speech-expression emotion recognition first needs to extract individual single-modality features. To this end, a speech single-modality feature extraction module and an expression single-modality feature extraction module are introduced to extract the features of the speech and expression modalities, respectively.

[0094] For the speech modality, the Mel-frequency Cepstral Coefficients (MFCC) feature based on the cepstrum conforms to human hearing habits and is a commonly used speech feature extraction method. First, 64 filter banks are used, and one frame is selected every 16 frames to extract MFCCs of size 39; then the first-order difference and the second-order difference of the speech MFCCs are calculated to obtain the first-order MFCC and the second-order MFCC of the speech; finally, the speech MFCC, the first-order MFCC, and the second-order MFCC are used as inputs, a Residual Network (ResNet) is used to extract the speech modality features. Therefore, the output speech features are represented as:

[0095]

[0096] wherein, represents the MFCC of the speech sample of the th emotion category, represents the output feature of the speech sample of the th emotion category.

[0097] For the expression modality, the Three-Dimensional Convolutional Neural Network (3D-CNN) is a commonly used visual feature extraction method because it can effectively extract spatial and temporal features at the same time. Therefore, the 3D-CNN is applied to extract the expression modality features, which are represented as:

[0098]

[0099] wherein, denotes the expression sample of the th emotion category, denotes the expression sample output feature of the th emotion category.

[0100] Next, the speech monomodal feature and the expression monomodal feature are fused by the contrastive feature fusion module to fully extract the shared information between the speech and expression modalities and the unique information of each modality.

[0101] The shared information between the speech and expression modalities is extracted by using the contrastive learning, and the unique information of the speech and expression modalities is extracted by using the orthogonal constraint.

[0102] The contrastive learning is a self-supervised learning framework that maximizes the common features between the samples of the same category, which has been proved to be effective in multi-view representation learning and multi-modal tasks. For this purpose, the shared information of the speech and expression modalities is extracted by using the contrastive learning. Specifically, the monomodal emotion features of the same emotion category and the fused multi-modal features are taken as positive samples, and others are taken as negative samples. The contrastive loss function generated thereby is:

[0103]

[0104] wherein, denotes the set of emotion categories. denotes the L1 norm of a vector. denotes the fused feature of the th emotion category, that is, the shared information between the speech and expression modalities, which is obtained by using three fully-connected (FC) layers to map the speech modality feature and the expression modality feature as inputs. denotes the mutual information, which is realized by an InfoNCE estimator. and are defined as:

[0105]

[0106]

[0107] wherein, denotes a temperature parameter, denotes the cosine similarity, which is defined as:

[0108]

[0109] where, and denote two vectors of equal length, denotes the dot product, denotes the L2 norm of a vector. By minimizing , the shared information between speech and expression modalities can be effectively extracted.

[0110] In addition to the shared information of speech and expression modalities, the unique information of speech and expression modalities is also very important for emotion recognition. According to the obtained single modality speech feature , single modality expression modality feature , the fused multi-modal speech expression feature , then the unique information of each modality can be obtained by subtracting the single modality speech feature and the single modality expression modality feature from the fused multi-modal speech expression feature :

[0111]

[0112]

[0113] where, denotes the unique information of expression modality, denotes the unique information of speech modality.

[0114] In order to better extract the unique information of speech and expression modalities, should not be encoded, and should not be encoded. Therefore, in order to achieve these, the orthogonal loss function is defined as:

[0115] By minimizing in the orthogonal loss function,

[0116] ensures that and do not encode each other, and ensures that and do not encode , respectively. According to the contrast loss function and the orthogonal loss function, the loss function of the contrast feature fusion is: ​​

[0117]

[0118] By optimizing the formula with respect to , it can guarantee that , , the shared information between the modalities of speech and facial expressions and the shared information of each modality are effectively extracted, which is very beneficial to emotion recognition.

[0119] In order to realize the classification of emotions, the modality shared information and the modality specific information , are input into . Among them, a multi-layer perceptron (MLP) with a three-layer structure is adopted, and the last layer of the MLP uses a Softmax activation function to output the emotion classification result. In addition, considering the contrast feature fusion and emotion classification comprehensively, the final loss function is defined as:

[0120]

[0121] Among them, is a hyperparameter that controls the effect of contrast learning and orthogonal constraint. is a classic cross-entropy loss function, which is widely used in classification tasks.

[0122] Figure 2 The principle diagram of the emotion recognition method based on contrast feature fusion according to one embodiment of the present application is shown.

[0123] As shown in Figure 2 , BAUM-1s, RML and eNTERFACE05 datasets are selected to evaluate the performance of the present method. Six emotion categories are recognized on the three datasets respectively, including anger, disgust, fear, happiness, sadness and surprise. PyTorch is utilized to realize the speech and expression single modality feature extraction module and , the contrast feature fusion module and the emotion classification module . For , it includes 16 convolutional layers, 5 max-pooling layers, and 3 FC layers. For , it includes 8 convolutional layers, 5 max-pooling layers and 3 FC layers. and the number of FC neurons in the last layer is 1024. For , the number of neurons of three FC layers is 2048-1024-1024. For the MLP, the number of neurons is 3072-1024-6. The learning rate is set to 0.001, the batch size is set to 32, and the regularization parameter is set to 0.3.

[0124] To better evaluate the performance of the method, Table 1 shows the ablation experiment results. The baseline method only utilizes the modality-shared information, and does not utilize the modality-specific information, i.e., the modality-specific information and is not input to for emotion classification. By comparing with the baseline method, the recognition rate of the method is increased by 6.89% on the BAUM-1s dataset, 5.65% on the RML dataset, and 3.92% on the eNTERFACE05 dataset.

[0125] Table 1

[0126]

[0127] Figure 3 shows the recognition results of each emotion category according to an embodiment of the application.

[0128] The recognition rate of each emotion category of the method on the BAUM-1s, RML, and eNTERFACE05 datasets is shown in Table 2. In addition, the method is compared with other methods, and the recognition rate is shown in Table 2. Among them, the hybrid deep fusion model directly adopts a deep belief network to fuse the speech and expression features, and the related feature fusion model first calculates the correlation between the speech and expression features using a canonical correlation analysis algorithm, and then performs fusion. As shown in Table 2, on the BAUM-1s, RML, and eNTERFACE05 datasets, the recognition rates of the method are 74.28%, 98.39%, and 99.26%, respectively, which are higher than those of the hybrid deep fusion model and the related feature fusion model. The reasons are as follows: the hybrid deep fusion model simply connects the features of the speech and expression modalities together without considering the complementarity between the speech and expression modalities, resulting in a lower recognition rate. Although the related feature fusion model considers the complementarity between the speech and expression modalities, it does not consider the modality-specific information of each modality, which reduces the performance of emotion recognition. In contrast, the method extracts the shared information between the speech and expression modalities using contrastive learning, and extracts the modality-specific information of the speech and expression modalities using orthogonal constraints, which effectively improves the performance of emotion recognition. Therefore, the recognition rate of the method is higher than that of other methods. Figure 3 Table 2

[0129]

[0130] ​​

[0131] Example 2

[0132] Figure 4 A block diagram of an emotion recognition device based on contrastive feature fusion according to an embodiment of the present invention is shown.

[0133] like Figure 4 As shown, the emotion recognition device based on contrast feature fusion includes:

[0134] The single-modal feature extraction module 201 extracts speech single-modal features and facial expression single-modal features respectively;

[0135] The contrastive feature fusion module 202 extracts shared information between speech monomodal features and facial expression monomodal features through contrastive learning, and obtains the contrastive loss function; and

[0136] The unique information of speech monomodal features and facial expression monomodal features is extracted by orthogonal constraints to obtain an orthogonal loss function;

[0137] The sentiment classification module 203 performs sentiment classification based on shared and unique information, obtains the total loss function, and obtains the final sentiment classification by minimizing the total loss function.

[0138] In one example, contrastive learning is used to extract shared information between speech monomodal features and facial expression monomodal features, and the contrastive loss function includes:

[0139] Using single-modal emotion features and fused multimodal features of the same emotion category as positive samples and others as negative samples, we can obtain the shared information between speech single-modal features and facial expression single-modal features.

[0140] Based on the shared information and the single-modal features of speech and facial expressions, a contrastive loss function is calculated.

[0141] In one example, the contrastive loss function is:

[0142]

[0143] in, A set representing emotion categories. Describes the L1 norm of a vector. Indicates the first Modal sharing information across emotion categories and These represent the corresponding speech modality features and facial expression modality features, respectively. Indicates mutual information, , , Indicates temperature parameter, Represents cosine similarity. wherein, and denote two vectors of equal length, denotes the dot product, denotes the L2 norm of a vector.

[0144] In one example, the specific information of the speech single-modal feature and the expression single-modal feature is extracted by orthogonal constraint, and the orthogonal loss function is obtained, including:

[0145] According to the modal shared information, the modal specific information is calculated;

[0146] According to the modal specific information, the orthogonal loss function is calculated.

[0147] In one example, the modal specific information is calculated by the following formula:

[0148]

[0149]

[0150] wherein, denotes the specific information of the expression modal, denotes the specific information of the speech modal.

[0151] In one example, the orthogonal loss function is:

[0152]

[0153] wherein, denotes the specific information of the expression modal, denotes the specific information of the speech modal.

[0154] In one example, the total loss function is:

[0155]

[0156] wherein, denotes a hyperparameter, denotes a cross-entropy loss function.

[0157] Example 3

[0158] The present disclosure provides an electronic device, including a memory storing executable instructions; a processor running the executable instructions in the memory to implement the above-mentioned emotion recognition method based on fusion of contrast features.

[0159] The electronic device according to the embodiments of the present disclosure includes a memory and a processor.

[0160] The memory is configured to store non-transitory computer readable instructions. Specifically, the memory can include one or more computer program products that can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read only memory (ROM), hard disk, flash memory, and / or the like.

[0161] The processor can be a central processing unit (CPU) or other form of processing unit that has data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is configured to execute the computer readable instructions stored in the memory.

[0162] Those skilled in the art will understand that, in order to solve the technical problem of how to obtain a good user experience effect, the embodiments can also include well-known structures such as a communication bus, an interface, and the like, which should also be included in the protection scope of the present disclosure.

[0163] Detailed descriptions of the embodiments can refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0164] Example 4

[0165] The embodiments of the present disclosure provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the emotion recognition method based on contrast feature fusion.

[0166] The computer readable storage medium according to the embodiments of the present disclosure has non-transitory computer readable instructions stored thereon. When the non-transitory computer readable instructions are run by a processor, all or part of the steps of the method of the embodiments of the present disclosure are executed.

[0167] The computer readable storage medium described above includes, but is not limited to, optical storage media (for example, CD-ROM and DVD), magneto-optical storage media (for example, MO), magnetic storage media (for example, magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (for example, memory card), and media with built-in ROM (for example, ROM cartridge).

[0168] Those skilled in the art will understand that the purpose of the above description of the embodiments of the present disclosure is only to exemplarily illustrate the beneficial effects of the embodiments of the present disclosure, and is not intended to limit the embodiments of the present disclosure to any examples given.

[0169] Having described various embodiments of the application, it is to be understood that the above description is meant to be illustrative only, and that many modifications and variations of the embodiments are possible without departing from the scope and spirit of the described embodiments. Many modifications and variations of the described embodiments are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the described embodiments can be practiced otherwise than as specifically described.

Claims

1. A sentiment recognition method based on contrastive feature fusion, characterized in that, include: Extract speech monomodal features and facial expression monomodal features separately; By extracting the shared information between the speech monomodal features and the facial expression monomodal features through contrastive learning, a contrastive loss function is obtained; The unique information of the speech monomodal features and the facial expression monomodal features is extracted by orthogonal constraints to obtain an orthogonal loss function; Based on the shared information and the unique information, sentiment classification is performed to obtain the total loss function; The final sentiment classification is obtained by minimizing the total loss function. The contrastive loss function is: in, A set representing emotion categories. Describes the L1 norm of a vector. Indicates the first Modal sharing information across emotion categories and These represent the corresponding speech modality features and facial expression modality features, respectively. Indicates mutual information, , , Indicates temperature parameter, Represents cosine similarity. ,in, and Let each represent two vectors of equal length. Represents the dot product. The L2 norm of a vector; The orthogonal loss function is: in, Specific information indicating facial expression modality, It represents the unique information of the speech modality.

2. The emotion recognition method based on contrast feature fusion according to claim 1, wherein, By extracting shared information between the speech monomodal features and the facial expression monomodal features through contrastive learning, the contrastive loss function is obtained, including: Using single-modal emotion features and fused multimodal features of the same emotion category as positive samples and others as negative samples, the shared information between the speech single-modal features and the facial expression single-modal features is obtained; The contrast loss function is calculated based on the shared information, the speech monomodal features, and the facial expression monomodal features.

3. The emotion recognition method based on contrastive feature fusion according to claim 1, wherein, The unique information of the speech monomodal features and the facial expression monomodal features is extracted through orthogonal constraints, and the orthogonal loss function is obtained, including: Based on modality sharing information, calculate the unique information of the speech modality and the unique information of the facial expression modality; The orthogonal loss function is calculated based on the unique information of the speech modality and the unique information of the facial expression modality.

4. The emotion recognition method based on contrast feature fusion according to claim 3, wherein, The specific information of the speech modality and the specific information of the facial expression modality are calculated using the following formula: in, Specific information indicating facial expression modality, It represents the unique information of the speech modality.

5. The emotion recognition method based on contrast feature fusion according to claim 1, wherein, The total loss function is: in, Indicates hyperparameters, This represents the cross-entropy loss function.

6. An emotion recognition device based on contrast feature fusion, characterized in that, include: The single-modal feature extraction module extracts speech single-modal features and facial expression single-modal features respectively; The contrastive feature fusion module extracts shared information between the speech monomodal features and the facial expression monomodal features through contrastive learning, and obtains a contrastive loss function; and The unique information of the speech monomodal features and the facial expression monomodal features is extracted by orthogonal constraints to obtain an orthogonal loss function; The sentiment classification module performs sentiment classification based on the shared information and the unique information, obtains a total loss function, and obtains the final sentiment classification by minimizing the total loss function. The contrastive loss function is: in, A set representing emotion categories. Describes the L1 norm of a vector. Indicates the first Modal sharing information across emotion categories and These represent the corresponding speech modality features and facial expression modality features, respectively. Indicates mutual information, , , Indicates temperature parameter, Represents cosine similarity. ,in, and Let each represent two vectors of equal length. Represents the dot product. The L2 norm of a vector; The orthogonal loss function is: in, Specific information indicating facial expression modality, It represents the unique information of the speech modality.

7. An electronic device, characterized in that, The electronic device includes: Memory, which stores executable instructions; A processor that executes the executable instructions in the memory to implement the emotion recognition method based on contrast feature fusion as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the emotion recognition method based on contrast feature fusion as described in any one of claims 1-5.

Images