A multimodal emotion recognition method and device

Abstract

The present invention discloses a multimodal emotion recognition method and device, wherein the method comprises obtaining a multimodal semantic vector, obtaining a modality-specific semantic vector and a common semantic vector by feature decoupling; performing feature enhancement and global distillation on the specific semantic vector and the common semantic vector; and outputting an emotion category according to the features of the specific semantic vector and the common semantic vector after distillation. The present invention can stably and effectively identify the corresponding character emotion type in an emotional video; and can ensure recognition efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically to a multimodal emotion recognition method and device. Background Art

[0002] With the rapid development of artificial intelligence technology, human-computer interaction technology has attracted more and more attention. As an important research field in human-computer interaction, automatic emotion recognition has broad application prospects. However, traditional single-modal emotion recognition methods often fail to accurately capture complex human emotions, so multimodal emotion recognition has become a research hotspot.

[0003] Multimodal emotion recognition uses physiological and behavioral responses (i.e., multimodal information) to identify and predict emotional states. For example, emotion recognition on multiple modalities such as speech, facial expressions, and body movements can improve recognition accuracy and robustness.

[0004] However, multimodal emotion recognition faces some challenges. First, labeling emotion data is a tedious and subjective task, and it is difficult to obtain a large amount of accurately labeled data. Second, there may be redundant information between different modalities, which increases the computational complexity and may affect the recognition performance. Third, it is necessary to extract effective emotion features from different modalities and fuse these features, but the choice of feature extraction methods and fusion strategies for different modalities has an important impact on the recognition performance. Finally, multimodal emotion recognition usually needs to be performed in real time, which places high demands on the real-time and robustness of the algorithm. Summary of the invention

[0005] In view of this, in order to solve the above problems, the present invention proposes a multimodal emotion recognition method and device, which can improve the accuracy and robustness of multimodal emotion recognition by combining information from multiple modalities and utilizing techniques such as feature decoupling and graph distillation.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] In one aspect, the present invention discloses a multimodal emotion recognition method, comprising:

[0008] Obtain multimodal semantic vectors, and obtain modality-specific semantic vectors and common semantic vectors through feature decoupling;

[0009] After feature enhancement of specific semantic vectors and common semantic vectors, global distillation is performed;

[0010] According to the distilled specific semantic vector features and common semantic vector features, the sentiment category is output.

[0011] Preferably, the specific semantic vector is optimized by a gravitational loss function before feature enhancement; the gravitational loss function is:

[0012]

[0013] In the formula, e is a natural constant, a is a hyperparameter, e is a natural constant, a is a hyperparameter.

[0014] F 1 is the loss function between specific semantic vectors of the same modality, F 2 is the loss function between different same-modality specific semantic vectors; and

[0015]

[0016] In the formula, represents the attraction between the i-th and j-th specific semantic vectors in modality h, represents the attraction between the i-th specific semantic vector in modality h and modality k, represents the attraction between the i-th specific semantic vector in modality h and modality z, represents the attraction between the i-th specific semantic vector in modality k and modality z; n is the total number of specific semantic vectors.

[0017] Preferably, the attraction between two specific semantic vectors is obtained by the following formula:

[0018]

[0019] In the formula, X i ,X j Represent the i-th and j-th specific semantic vectors respectively, I(X i ) represents the self-information of the i-th specific semantic vector, I(X j ) represents the self-information of the jth specific semantic vector, and the self-information calculation formula is:

[0020]

[0021] Where f(X0) represents the density function of a specific semantic vector X;

[0022]

[0023] In the formula, γ represents the variance.

[0024] Preferably, feature enhancement is performed on the specific semantic vector and the common semantic vector, including:

[0025] The specific semantic vector is enhanced by using the public semantic vector, and the public semantic vector is enhanced by using the specific semantic vector.

[0026] Among them, using the public semantic vector to enhance the specific semantic vector includes:

[0027] The common semantic vector X is extracted through the self-attention unit. C The features and each specific semantic vector Features;

[0028] The features of the common semantic vector are divided into C' With each specific semantic vector feature Input to the first cross-modal attention unit to obtain the corresponding enhanced specific semantic vector features

[0029] The first cross-modal attention unit includes a cross-modal attention head, a third normalization layer, a feed-forward network layer, and a fourth normalization layer connected in sequence; and the head of the first cross-modal attention unit is:

[0030]

[0031] In the formula, d m ,d c It means The vector dimension, X C 'Vector dimension, d k is a constant, usually the dimension of the word vector after embedding, which is 768 in this paper.

[0032] Enhance the public semantic vector with specific semantic vectors, including:

[0033] The self-attention unit is used to further enhance the specific semantic vector features. Perform feature extraction;

[0034] The results are respectively compared with the features X of the common semantic vector C 'Input into the second cross-modal attention unit to obtain the interactive features of multiple common features and specific features;

[0035] Perform weight learning on the multiple interactive features to obtain an enhanced common semantic vector feature X c* .

[0036] In the second cross-modal attention unit, a weight learning layer is provided after the cross-modal attention head, for performing weight learning on the multiple obtained results;

[0037] The weight learning process is:

[0038]

[0039] In the formula, β m is the attention score under m mode, y m It is expressed as the weight of mode m over all modes.

[0040] Furthermore, in the second cross-modal attention unit, the cross-modal attention head is defined as:

[0041]

[0042] Preferably, the weighted distillation loss of global distillation is:

[0043]

[0044] Among them, ω c→j represents the distillation strength of the edge from graph vertex c to graph vertex j, ∈ c→j It is represented as a distillation of graph vertex c to graph vertex j, which can be defined as the difference between their direct logits.

[0045] ω c→j =g([[f(X c ,θ 1 ),X c ],[f(X c ,θ 1 ),X j ]],θ 2 )

[0046] In the formula, [,] represents feature concatenation, g represents learnable parameters, and θ 2 represents the parameters of the fully connected (FC) layer, f is the FC layer for regressing logits, with parameters θ 1 .

[0047] As a preferred embodiment, the distillation loss is:

[0048] L dtl =||W⊙E||

[0049] Where W represents the edge weight matrix, and W jc =ω j→c , E represents the distillation matrix, and E jc =∈ j→c .

[0050] Preferably, outputting the sentiment category according to the distilled specific semantic vector and the common semantic vector comprises:

[0051] After the distilled specific semantic vector and the common semantic vector are concatenated, they are sequentially subjected to linear mapping and SoftMax, and the dimension with the largest value in the obtained result is the recognized emotion type. The formula is as follows:

[0052]

[0053] In the formula, y * It means that the dimension with the largest value after SoftMax is the corresponding emotion type, z j Represented as the j-th dimension value of the final vector.

[0054] On the other hand, a multimodal emotion recognition device is characterized by comprising:

[0055] The feature decoupling module is used to obtain multimodal semantic vectors, and obtain modality-specific semantic vectors and common semantic vectors through feature decoupling;

[0056] The semantic vector processing module is used to perform global distillation after feature enhancement of specific semantic vectors and common semantic vectors;

[0057] The multimodal emotion recognition module is used to output the emotion category based on the distilled specific semantic vector and the common semantic vector.

[0058] Preferably, the semantic vector processing module includes a feature enhancement unit and a global distillation unit.

[0059] The feature enhancement unit is used to enhance the specific semantic vector using the public semantic vector, and enhance the public semantic vector using the specific semantic vector;

[0060] The global distillation unit is used to perform global distillation on the enhanced semantic features.

[0061] Through the above technical solutions, it can be seen that the present invention discloses a multimodal emotion recognition method and device, which can more accurately identify the emotional state of a person by combining information from multiple modes and utilizing feature decoupling, feature optimization, feature enhancement and graph distillation technology; compared with the prior art,

[0062] The gravitational loss function in this application can better process redundant information between different modalities and reduce noise interference by optimizing the specific semantic vector relationship between the same modality and different modalities, thereby making the similarities and differences between vectors more obvious, and helping to improve the accuracy of emotion recognition;

[0063] The feature enhancement process of specific semantic vectors and common semantic vectors can strengthen the important features of each modality, which not only enables the model to learn more stable feature representations, but also better captures complex emotional information so as to have better adaptability when facing different environments and scenarios.

[0064] In summary, the emotion recognition method disclosed in the present invention can stably and effectively capture more complex human emotions and effectively improve the accuracy of emotion recognition.

[0065] At the same time, after sufficient training, it takes less time to test the actual situation, which greatly improves the efficiency of multimodal emotion recognition, so that the present invention has higher practical value in actual applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0067] Figure 1 A flow chart of the multimodal emotion recognition method provided by the present invention;

[0068] Figure 2 A schematic diagram of a multimodal emotion recognition network provided by the present invention;

[0069] Figure 3 A schematic diagram of a specific semantic vector feature enhancement process provided by the present invention;

[0070] Figure 4 A schematic diagram of the common semantic vector feature enhancement process provided by the present invention;

[0071] Figure 5 This is an example diagram of the process flow of the multimodal emotion recognition method provided by the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Embodiment 1

[0074] The embodiment of the present invention discloses a multimodal emotion recognition method, such as Figure 1 ,include:

[0075] S1. Obtain multimodal semantic vectors, and obtain modality-specific semantic vectors and common semantic vectors through feature decoupling;

[0076] S2, after feature enhancement of specific semantic vectors and common semantic vectors, global distillation is performed;

[0077] S3. Output the sentiment category based on the distilled specific semantic vector and public semantic vector.

[0078] In this embodiment, refer to Figure 2 , Figure 2 A schematic diagram of a multimodal emotion recognition network;

[0079] In step S1, obtaining a multimodal semantic vector includes:

[0080] S11, receiving an input multimodal emotional video, and extracting multimodal emotional features (image, text, audio) based on a deep learning neural network model;

[0081] The multimodal emotional video includes a video captured by a video capture device; and in this embodiment,

[0082] The text modality uses whisper in OpenAI to extract video text, and then uses BERT to convert the text into a 768-dimensional word vector. In this embodiment, BERT base is used, in which the model has 12 layers, the token dimension is 768 dimensions, the number of attention heads is 12, and the number of parameters is 110M.

[0083] The visual modality was encoded using the Openface toolkit, obtaining the presence of a total of 35 facial action units with an extracted dimension of 74;

[0084] The acoustic mode is obtained using the COVAREP toolkit to obtain 74-dimensional features.

[0085] Specifically, the video is captured by a video capture device, and then the captured video is received and divided into equal lengths of 1 minute. The captured video is separated from the audio and then input into the COVAREP toolkit to obtain 768-dimensional text features. The captured video is input into the FaceNet model to obtain a total of 35 facial action units with a feature dimension of 35*50. At the same time, the separated audio is input into the OpenAI-whisper model to obtain a text after speech recognition. The text is then vectorized by the BERT model.

[0086] S12: converting the multimodal features into multimodal semantic vectors;

[0087] First, the three modal features are first subjected to a layer of one-dimensional convolution to adjust the three modal features to the same dimension and obtain a multimodal semantic vector. The convolution kernel of the one-dimensional convolution is 5*5, the step size is 1, and the number of convolution kernels is 50.

[0088] Further, a modality-specific semantic vector and a common semantic vector are obtained by feature decoupling; in an optional embodiment, the method includes:

[0089] (1) The public encoder and three private encoders are used to obtain the decoupled features. The convolution kernel size of the public encoder is 1*1, the step size is 1, and the number of convolution kernels is 50. The convolution kernel size of the private encoder is 1*1, the step size is 1, and the number of convolution kernels is 50.

[0090] The decoupling formula is:

[0091]

[0092] in is the common semantic vector feature, is a specific semantic feature, E c is a public encoder, It is a private encoder. It is a multimodal feature.

[0093] (2) Use the decoder to autoregressively synthesize the decoupled features of the same mode to obtain the coupled features Among them, D s Represented as an encoder, the convolution kernel size is 1*1, the step size is 1, and the number of convolution kernels is 50.

[0094] (3) Use loss function to constrain the decoupled features.

[0095] The difference loss can be formulated as:

[0096]

[0097] The regression loss can be formulated as:

[0098]

[0099] Step S2: After feature enhancement of the specific semantic vector and the common semantic vector, global distillation is performed;

[0100] In this embodiment, the public semantic vector is used to enhance the specific semantic vector, and the specific semantic vector is used to enhance the public semantic vector.

[0101] S21. Use the public semantic vector to enhance the specific semantic vector. In this embodiment, the public semantic vector is obtained by concatenating the public semantic vectors of each modality, that is, in T C =T L +T V +T A ,T C ,d C Represents the length and dimension of feature x.

[0102] Further, if Figure 3 As shown, the enhancement steps include:

[0103] (1) Extract the features X of the common semantic vectors through the self-attention unit C 'With the characteristics of each specific semantic vector The self-attention unit includes a first normalization layer, a self-attention head, and a second normalization layer connected in sequence; and the output result of the first normalization layer is concatenated with the output result of the self-attention head and used as the input of the second normalization layer;

[0104] Specifically:

[0105] X C' =LN(SA mul (LN(X C )))

[0106]

[0107] in SA mul and LN represent the self-attention head and layer normalization operations. c =X C P q ,K c =X C P k and V c =X C P v . Among them, P q ,P k and P v is a learnable parameter. A single attention head can be expressed as: Where d represents Q c ,K c Dimension.

[0108] After the self-attention head, a second LN layer is used for normalization.

[0109] (2) The feature X of the public semantic vector C ' and the features of a specific semantic vector The input is sent to the cross-modal attention unit to strengthen the specific modality features by forming directional attention between the common features and the specific modality features through the cross-modal attention unit.

[0110] The cross-modal attention unit includes a cross-modal attention head, a third normalization layer, a feed-forward network layer, and a fourth normalization layer connected in sequence; and the output result of the cross-modal attention head is consistent with the specific semantic vector Features of a specific semantic vector After splicing, it is used as the input of the third normalization layer, and the output result of the third normalization layer is input into the feedforward network layer to obtain the enhanced features corresponding to the specific semantic vector

[0111] Similar to the self-attention mechanism, the cross-modal attention unit can be defined as: and and and A cross-modal attention head can be defined as:

[0112]

[0113] In the formula, d m ,d c It means The vector dimension, X C 'Vector dimension, d k is a constant, usually the dimension of the word vector after embedding, which is 768 in this paper.

[0114] Furthermore, the results Perform residual connection, and we have

[0115] then The enhanced features corresponding to the specific semantic vector are obtained through the residual connected feedforward network layer

[0116] The above steps are repeated until all enhanced specific semantic vectors are obtained.

[0117] S22, using specific semantic vectors to enhance the public semantic vector, such as Figure 4 As shown, including:

[0118] The features X of the common semantic vector are extracted through four self-attention units. C ' and the features of each enhanced specific semantic vector Right now

[0119] X C '=LN(SA mul (LN(X C )))

[0120]

[0121] Similarly, the common features are interacted with the specific features of each modality through cross-modal attention, that is, the feature X of the common semantic vector C 'Respectively with each enhanced specific semantic vector feature Input into the cross-modal attention unit to obtain the features after the common features interact with the specific features of each modality;

[0122] In this embodiment, the cross-modal attention unit structure is consistent with step S21, except that

[0123] The cross-modal attention head is:

[0124]

[0125] The cross-modal attention unit can be defined as: and and and

[0126] A weight learning layer is added after the cross-modal attention head to perform weight learning on multiple obtained results, so as to assign dynamic weights according to the importance of each modality enhanced representation, thereby obtaining an enhanced common semantic vector;

[0127] The process of weight learning is:

[0128]

[0129] In the formula, β m is the attention score under m mode, y m It is expressed as the weight of mode m over all modes.

[0130] in and are learnable parameters.

[0131] Further, the obtained results are connected with residuals, and we have X C”' =X C” +X C' +X C ;

[0132] Finally, we will c”' Through the feed-forward layer with residual connection, the final X is obtained c* .

[0133] S23. Input the enhanced common semantic features and specific semantic features into the global distillation graph for global distillation to reduce the distribution difference between the modality-specific semantic features and the common semantic features.

[0134] In this embodiment, v c Denotes nodes that represent shared sentiment features, with v j represents a node representing a modality-specific feature, where s∈{L,V,A} (L, V, A represent language, visual, and audio modalities, respectively). In addition, w c→jIndicates that from v c to v j distillation intensity; similarly, w j→c Indicates that from v j to v c The distillation intensity of c to v j For example, distillation is defined as the difference between their corresponding logits, ∈ c→j Let E represent the distillation matrix, where E cj =ω c→j and E jc =ω j→c The weighted distillation loss can be expressed as:

[0135]

[0136] Among them, ω c→j represents the edge distillation strength from graph vertex c to graph modal vertex j, ∈ c→j It is represented as a distillation of graph vertex c to graph modal vertex j, which can be defined as the difference between their direct logits.

[0137] To learn the distillation strength W, we utilize logits and representations to jointly learn the values ​​of graph edges. For the common semantic vector v c and the modality-specific semantic vector v j , which can be formally expressed as:

[0138] ω c→j =g([[f(X c ,θ 1 ),X c ],[f(X c ,θ 1 ),X j ]],θ 2 )

[0139] In the formula, [,] represents feature concatenation, g represents learnable parameters, and θ 2 represents a fully connected (FC) layer, f is a FC layer for regression logits, with parameter θ 1 .

[0140] The edge weight of the graph is W, where W cj / W jc =ω c→j / ω j→c , can be learned by repeatedly using the equation. The graph distillation loss can be expressed as:

[0141] L dtl =||W⊙E||

[0142] Where W represents the edge weight matrix, and W jc =ω j→c , E represents the distillation matrix, and E jc =∈ j→c ;

[0143] S3. Output the sentiment category based on the distilled specific semantic vector and public semantic vector.

[0144] The distilled specific semantic vector and the public semantic vector are concatenated into the final semantic vector, and the final semantic vector is linearly mapped to the dimension of the sentiment type. Then, after SoftMax, the dimension with the largest value is the corresponding sentiment type. The formula is as follows:

[0145]

[0146] In the formula, y * It means that the dimension with the largest value after SoftMax is the corresponding emotion type, z j Represented as the j-th dimension value of the final vector.

[0147] The present invention can stably and effectively identify the corresponding character emotion types in emotional videos; and after sufficient training, it takes less time to test actual conditions, greatly improving efficiency.

[0148] Embodiment 2

[0149] To further optimize the above technical solution, the specific semantic vector is optimized by the gravitational loss function before feature enhancement; the steps include:

[0150] (1) Calculate the gravitational force between mode-specific vectors.

[0151] For modality-specific semantic vectors between the same modality:

[0152]

[0153] where X represents the mode-specific vector, X i represents the i-th mode-specific vector, s represents the mode, s(h represents the h-mode, s∈{L,V,A}, is represented as a modality-specific vector with the i-th modality being k, F represents the modality cognitive gravity, It is expressed as the modal cognitive gravity of the modality-specific vector between the h-mode and the k-mode in the i-th sample.

[0154] For modality-specific semantic vectors between different modalities:

[0155]

[0156] Where F represents modal cognitive gravity, It is represented as the modality-specific vector of the modality-cognitive attraction between the h modalities in the ith sample.

[0157] (2) Calculate the self-information of the modality-specific semantic vector.

[0158] Self-information of modality-specific semantic vectors between different modalities:

[0159]

[0160] Among them I c (X) indicates the self-information of the mode-specific vector of the same mode, f c (X) represents the density function of the same mode.

[0161] Self-information of modality-specific semantic vectors between the same modality:

[0162]

[0163] where f p (x) represents the density functions of different modes.

[0164] (3) Calculate the density function of the modality-specific semantic vector.

[0165] Density function of modality-specific semantic vectors between the same modalities:

[0166]

[0167] Density function of modality-specific semantic vectors between different modalities:

[0168]

[0169] (4) Calculate the loss function of modal cognitive gravity.

[0170] The loss function for modality-specific semantic vectors between different modalities is:

[0171]

[0172] The loss function for modality-specific semantic vectors between the same modality is:

[0173]

[0174] L2=αF 2

[0175] The total modal cognitive gravity loss function:

[0176] L mcg =L1 +L 2

[0177] At this point, the multimodal emotion recognition method process is as follows Figure 5 As shown, including:

[0178] Receive an input multimodal emotional video and extract the multimodal emotional features (image, text, audio) based on a deep learning neural network model;

[0179] Converting the multimodal features into a multimodal semantic vector of the same dimension;

[0180] Inputting the multimodal semantic vector into a neural network decoupling module to obtain a modality-specific semantic vector and a modality-common semantic vector after feature decoupling;

[0181] Optimizing the modality-specific semantic vector using a modality-aware gravity loss function;

[0182] Inputting the modality-specific semantic vector and the modality-common semantic vector into a cross-modality alignment network, and inputting them into a global distillation graph after alignment;

[0183] The modality-specific semantic vector and the modality-common semantic vector are concatenated into a final semantic vector, and the final semantic vector is linearly mapped to the dimension of the emotion type. Then, after Softmax, the dimension with the largest value is the corresponding emotion type.

[0184] Embodiment 3

[0185] This embodiment discloses a multimodal emotion recognition device, comprising:

[0186] The feature decoupling module is used to obtain multimodal semantic vectors, and obtain modality-specific semantic vectors and common semantic vectors through feature decoupling;

[0187] The semantic vector processing module is used to perform global distillation after feature enhancement of specific semantic vectors and common semantic vectors;

[0188] The multimodal emotion recognition module is used to output the emotion category based on the distilled specific semantic vector and the common semantic vector.

[0189] Preferably, the semantic vector processing module includes a feature enhancement unit and a global distillation unit.

[0190] The feature enhancement unit is used to enhance the specific semantic vector using the public semantic vector, and enhance the public semantic vector using the specific semantic vector;

[0191] The global distillation unit is used to perform global distillation on the enhanced semantic features.

[0192] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0193] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal emotion recognition method, characterized in that: Obtain multimodal semantic vectors, and obtain modality-specific semantic vectors and common semantic vectors through feature decoupling; Perform feature enhancement and global distillation on specific semantic vectors and public semantic vectors in turn; Before feature enhancement, the specific semantic vector is optimized by the gravitational loss function; the gravitational loss function is: Where e is a natural constant, a is a hyperparameter, F1 is the loss function between specific semantic vectors of the same modality, and F2 is the loss function between specific semantic vectors of different modalities; and In the formula, represents the attraction between the i-th and j-th specific semantic vectors in modality h, represents the attraction between the i-th specific semantic vector in modality h and modality k, represents the attraction between the i-th specific semantic vector in modality h and modality z, represents the attraction between the i-th specific semantic vector in modality k and modality z; n is the total number of specific semantic vectors; According to the distilled specific semantic vector features and common semantic vector features, the sentiment category is output.

2. A multimodal emotion recognition method according to claim 1, characterized in that: Feature enhancement of specific semantic vectors and common semantic vectors, including: The specific semantic vector is enhanced by using the public semantic vector, and the public semantic vector is enhanced by using the specific semantic vector.

3. A multimodal emotion recognition method according to claim 2, characterized in that: Utilize public semantic vectors to enhance specific semantic vectors, including: The features of the common semantic vector and the features of each specific semantic vector are extracted through the self-attention unit; The features of the common semantic vector and the features of each specific semantic vector are input into the cross-modal attention unit respectively to obtain the corresponding enhanced specific semantic vector features.

4. A multimodal emotion recognition method according to claim 3, characterized in that: Enhance the public semantic vector with specific semantic vectors, including: The self-attention unit is used to further extract the enhanced specific semantic vector features; The obtained results are respectively input into the second cross-modal attention unit with the features of the public semantic vector to obtain the interactive features of multiple public features and specific features; Weight learning is performed on a plurality of the interactive features to obtain enhanced common semantic vector features.

5. A multimodal emotion recognition method according to claim 4, characterized in that: The process of weight learning is: In the formula, β m is the attention score under m mode, y m It is expressed as the weight of the m mode in all modes, U, and b m are learnable parameters, L, V, and A represent language, vision, and audio modalities, respectively.

6. A multimodal emotion recognition method according to claim 1, characterized in that: The weighted distillation loss for global distillation is: Among them, ω c→j represents the distillation strength of the edge from graph vertex c to graph vertex j, ∈ c→j Represented as the distillation from graph vertex c to graph vertex j; ω j→c represents the distillation strength of the edge from graph vertex j to graph vertex c, ∈ j→c Represented as the distillation from graph vertex j to graph vertex c; oh c→j =g([[f(X c ,θ1),X c ],[f(X c ,θ1),X j ]],θ2) Where [,] represents feature concatenation, g represents learnable parameters, θ1, θ2 represent the parameters of the fully connected layer, and f is the FC layer for regression logic.

7. A multimodal emotion recognition method according to claim 1, characterized in that: According to the distilled specific semantic vector and common semantic vector, the sentiment category is output. The steps include: After the distilled specific semantic vector and the common semantic vector are concatenated, they are sequentially subjected to linear mapping and SoftMax, and the dimension with the largest value in the obtained result is the recognized emotion type. The formula is as follows: In the formula, y * It means that the dimension with the largest value after SoftMax is the corresponding emotion type, z j Represented as the j-th dimension value of the final vector.

8. A multimodal emotion recognition device, characterized in that: include: The feature decoupling module is used to obtain multimodal semantic vectors, and obtain modality-specific semantic vectors and common semantic vectors through feature decoupling; The semantic vector processing module is used to perform global distillation after feature enhancement of specific semantic vectors and common semantic vectors; the specific semantic vector is optimized by the gravitational loss function before feature enhancement; the gravitational loss function is: Where e is a natural constant, a is a hyperparameter, F1 is the loss function between specific semantic vectors of the same modality, and F2 is the loss function between specific semantic vectors of different modalities; and In the formula, represents the attraction between the i-th and j-th specific semantic vectors in modality h, represents the attraction between the i-th specific semantic vector in modality h and modality k, represents the attraction between the i-th specific semantic vector in modality h and modality z, represents the attraction between the i-th specific semantic vector in modality k and modality z; n is the total number of specific semantic vectors; The multimodal emotion recognition module is used to output the emotion category based on the distilled specific semantic vector and the common semantic vector.

9. A multimodal emotion recognition device according to claim 8, characterized in that: The semantic vector processing module includes a feature enhancement unit and a global distillation unit. The feature enhancement unit is used to enhance the specific semantic vector using the public semantic vector, and enhance the public semantic vector using the specific semantic vector; The global distillation unit is used to perform global distillation on the enhanced semantic features.

Images