Multimodal emotion recognition method and apparatus

Abstract

The application provides a multimodal emotion recognition method and device, which can be used in the financial field or other fields. The method comprises the following steps: obtaining video data samples and actual emotion labels; obtaining single-modal feature data and emotion feature data of each modal according to the original data of the video data samples in multiple modes and single-modal emotion learning models of each modal; obtaining emotion prediction data according to a multimodal emotion recognition model and the original data of each modal; substituting the emotion feature data, the emotion prediction data and the actual emotion labels into an overall loss function to adjust the parameters of each model, completing the training of the model when the overall loss function value is the minimum, and the overall loss function comprises loss functions of the multimodal emotion recognition model and each single-modal emotion learning model; obtaining to-be-recognized video data, and determining an emotion recognition result according to the trained multimodal emotion recognition model. The application can improve the accuracy of multimodal emotion recognition.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a multimodal emotion recognition method and apparatus. Background Technology

[0002] In recent years, with the development of human-computer interaction technology, people's pursuit and expectations for machine intelligence have become increasingly urgent. How to make machines more human-like and better able to understand human emotions and needs has become a hot topic of concern for many researchers. Emotion recognition is an important research area that enables machines to perceive and understand human emotions. To make recognition more accurate, researchers are trying to fuse more emotional signals, and multimodal emotion recognition combining speech, text, facial expressions, and other modalities has become a focus of attention. Multimodal emotion recognition enables computers to better understand users' emotions and attentional expressions during human-computer interaction by analyzing user emotions through the processing of different physiological signals.

[0003] Despite some progress in multimodal emotion recognition, several challenges remain. One key issue is that the sampling frequencies and representations of different modalities are entirely different, and the fine-grained interactions between modalities are ignored, affecting the accuracy of emotion recognition. Summary of the Invention

[0004] To address at least one problem in the prior art, this application proposes a multimodal emotion recognition method and apparatus that can improve the accuracy of multimodal emotion recognition.

[0005] To address the aforementioned technical problems, this application provides the following technical solution:

[0006] Firstly, this application provides a multimodal emotion recognition method, including:

[0007] Obtain video data samples and their corresponding actual sentiment tags;

[0008] Based on the single-modal emotion learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality, the single-modal feature data and emotion feature data of each modality are obtained.

[0009] Based on the multimodal emotion recognition model and the raw data of each modality, emotion prediction data is obtained;

[0010] The emotional feature data of each modality, the emotional prediction data, and the actual emotional labels are substituted into the overall loss function, and the parameters of the multimodal emotion recognition model and each unimodal emotion learning model are adjusted. The training of the multimodal emotion recognition model is completed when the value of the overall loss function is minimized. The overall loss function includes the loss functions corresponding to the multimodal emotion recognition model and each unimodal emotion learning model.

[0011] Acquire the video data to be identified, and determine the emotion recognition result corresponding to the video data to be identified based on the trained multimodal emotion recognition model and the video data to be identified.

[0012] Further, the step of obtaining the unimodal feature data and sentiment feature data of each modality based on the unimodal sentiment learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality includes:

[0013] By applying a bidirectional gated recurrent neural network model, the original data of the video data samples in each modality are encoded to obtain the single-modal feature data of each modality.

[0014] Based on the single-modal feature data and attention function of each modality, the attention results of each modality are obtained;

[0015] The attention results of each modality are input into the fully connected layer of the single-modality emotion learning model corresponding to that modality. The output data of the fully connected layer is processed by the normalized exponential function to obtain the emotion feature data of that modality.

[0016] Furthermore, the process of obtaining sentiment prediction data based on the multimodal sentiment recognition model and the raw data of each modality includes:

[0017] Modal interaction processing is performed using the single-modal feature data of each modality to obtain feature data under time-level interaction and feature data under sentence-level interaction.

[0018] Sentiment inference processing is performed based on the feature data of the time-level interaction and the feature data of the sentence-level interaction to obtain sentiment prediction data.

[0019] Furthermore, the application of single-modal feature data of each modality for modal interaction processing yields feature data under time-level interaction and feature data under statement-level interaction, including:

[0020] By applying the single-modal feature data of each modality to perform temporal-level modal interaction processing, feature data under temporal-level interaction is obtained;

[0021] By applying the unimodal feature data of each modality to perform sentence-level modal interaction processing, feature data under sentence-level interaction is obtained.

[0022] Furthermore, the application of single-modal feature data of each modality for time-level modal interaction processing yields feature data under time-level interaction, including:

[0023] Based on the single-modal feature data of any two modalities, the first correlation matrix of the two modalities is obtained;

[0024] Based on the first correlation matrix and normalized exponential function of the two modes, the influence of one of the two modes on the other under time-level interaction is obtained;

[0025] Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data from one of the two modalities to the other under the time-level interaction.

[0026] Based on the attention function and the fusion feature data of one of the two modalities, determine the attention result from one of the two modalities to the other under the time-level interaction.

[0027] Tensor concatenation is performed on the attention results from one of the two modalities to the other under the time-level interaction to obtain the feature data under the time-level interaction.

[0028] Furthermore, the application of single-modal feature data from each modality to perform statement-level modal interaction processing yields feature data under statement-level interaction, including:

[0029] Based on the attention results of any two modalities, the second correlation matrix of the two modalities is obtained;

[0030] Based on the second correlation matrix and normalized exponential function of the two modalities, the influence of one of the two modalities on the other under the statement-level interaction is obtained;

[0031] Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data corresponding to one of the two modalities under the horizontal interaction of statements;

[0032] Tensor concatenation is performed on the fused feature data from one of the two modalities under the horizontal interaction of the statement to obtain the feature data under the horizontal interaction of the statement.

[0033] Furthermore, the overall loss function L overall as follows:

[0034]

[0035] in, This represents the loss function corresponding to the multimodal emotion recognition model. This represents the loss function of the unimodal emotion learning model corresponding to different modalities.

[0036] Secondly, this application provides a multimodal emotion recognition device, comprising:

[0037] The sample acquisition module is used to acquire video data samples and their corresponding actual sentiment tags;

[0038] The single-modal emotion learning module is used to obtain single-modal feature data and emotion feature data for each modality based on the single-modal emotion learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality.

[0039] The sentiment prediction module is used to obtain sentiment prediction data based on the multimodal sentiment recognition model and the raw data of each modality.

[0040] The training module is used to substitute the emotional feature data of each modality, the emotional prediction data, and the actual emotional labels into the overall loss function, adjust the parameters of the multimodal emotion recognition model and each unimodal emotion learning model, and complete the training of the multimodal emotion recognition model when the overall loss function value is minimized. The overall loss function includes the loss functions corresponding to the multimodal emotion recognition model and each unimodal emotion learning model.

[0041] The emotion recognition module is used to acquire video data to be recognized, and determine the emotion recognition result corresponding to the video data to be recognized based on the trained multimodal emotion recognition model and the video data to be recognized.

[0042] Furthermore, the single-modal emotion learning module includes:

[0043] The encoding module is used to apply a bidirectional gated recurrent neural network model to encode the original data of the video data samples in each modality to obtain the single-modal feature data of each modality.

[0044] The attention processing submodule is used to obtain the attention results for each modality based on the single-modal feature data and attention function of each modality;

[0045] The normalization processing submodule is used to input the attention results of each modality into the fully connected layer of the single-modality emotion learning model corresponding to that modality, and apply the normalization exponential function to process the output data of the fully connected layer to obtain the emotion feature data of that modality.

[0046] Furthermore, the sentiment prediction module includes:

[0047] The modal interaction submodule is used to apply the single modal feature data of each modality to perform modal interaction processing, and obtain feature data under time-level interaction and feature data under sentence-level interaction.

[0048] The sentiment reasoning submodule is used to perform sentiment reasoning processing based on the feature data under the time-level interaction and the feature data under the sentence-level interaction to obtain sentiment prediction data.

[0049] Furthermore, the modal interaction submodule includes:

[0050] The time-level modal interaction unit is used to perform time-level modal interaction processing on the single-modal feature data of each modality to obtain feature data under time-level interaction.

[0051] The statement-level modal interaction unit is used to perform statement-level modal interaction processing using single-modal feature data from each modality, thereby obtaining feature data under statement-level interaction.

[0052] Furthermore, the time-level modal interaction unit is specifically used for:

[0053] Based on the single-modal feature data of any two modalities, the first correlation matrix of the two modalities is obtained;

[0054] Based on the first correlation matrix and normalized exponential function of the two modes, the influence of one of the two modes on the other under time-level interaction is obtained;

[0055] Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data from one of the two modalities to the other under the time-level interaction.

[0056] Based on the attention function and the fusion feature data of one of the two modalities, determine the attention result from one of the two modalities to the other under the time-level interaction.

[0057] Tensor concatenation is performed on the attention results from one of the two modalities to the other under the time-level interaction to obtain the feature data under the time-level interaction.

[0058] Furthermore, the statement-level modal interaction unit is specifically used for:

[0059] Based on the attention results of any two modalities, the second correlation matrix of the two modalities is obtained;

[0060] Based on the second correlation matrix and normalized exponential function of the two modalities, the influence of one of the two modalities on the other under the statement-level interaction is obtained;

[0061] Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data corresponding to one of the two modalities under the horizontal interaction of statements;

[0062] Tensor concatenation is performed on the fused feature data from one of the two modalities under the horizontal interaction of the statement to obtain the feature data under the horizontal interaction of the statement.

[0063] Specifically, the overall loss function L overall It can be as follows:

[0064]

[0065] in, This represents the loss function corresponding to the multimodal emotion recognition model. This represents the loss function of the unimodal emotion learning model corresponding to different modalities.

[0066] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multimodal emotion recognition method.

[0067] Fourthly, this application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, implement the multimodal emotion recognition method.

[0068] As can be seen from the above technical solution, this application provides a multimodal emotion recognition method and apparatus. The method includes: acquiring video data samples and their corresponding actual emotion tags; obtaining single-modal feature data and emotion feature data for each modality based on the single-modal emotion learning models corresponding to multiple modalities and the original data of the video data samples in each modality; obtaining emotion prediction data based on the multimodal emotion recognition model and the original data of each modality; substituting the emotion feature data of each modality, the emotion prediction data, and the actual emotion tags into an overall loss function, adjusting the parameters of the multimodal emotion recognition model and each single-modal emotion learning model, and completing the training of the multimodal emotion recognition model when the overall loss function value is minimized, wherein the overall loss function includes the loss functions corresponding to the multimodal emotion recognition model and each single-modal emotion learning model; acquiring video data to be recognized, and based on the trained multimodal emotion recognition model and the video data to be recognized... Determining the emotion recognition result corresponding to the video data to be recognized can improve the accuracy of multimodal emotion recognition. Specifically, by performing sequence learning on image information and establishing a temporal sequence model for semantic information extraction, image data can be better integrated with speech and text data. A multi-stage interactive network can be formed by combining the sequence learning features of speech, text, and facial expressions, realizing the association of different modal information at different stages, thereby forming richer multimodal spatial information for emotion recognition. Facial expression information can be selectively added to text and speech features to prevent image information from becoming interference from other modal information, effectively improving the overall recognition performance of the model. Learning feature representations at multiple levels of different modalities is of great significance for the dynamic integration of facial expression information into text and speech, and can improve the discriminativeness of multiple modal feature representations, which is beneficial to improving the accuracy of emotion recognition. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a first flowchart illustrating the multimodal emotion recognition method in this application embodiment;

[0071] Figure 2 This is a second flowchart illustrating the multimodal emotion recognition method in this application embodiment;

[0072] Figure 3 This is a schematic diagram of the third process of the multimodal emotion recognition method in the embodiments of this application;

[0073] Figure 4 This is a framework diagram of the multimodal emotion recognition method in the application example of this application;

[0074] Figure 5 This is a schematic diagram of the structure of the multimodal emotion recognition device in the embodiments of this application;

[0075] Figure 6 This is a schematic diagram of the structure of the multimodal emotion recognition device in the application example of this application;

[0076] Figure 7 This is a schematic block diagram illustrating the system configuration of an electronic device according to an embodiment of this application. Detailed Implementation

[0077] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0078] In existing technologies, while speech and text, as sequential data, exhibit higher fit and adaptability in modal processing and fusion, image information differs significantly from these two forms, making fusion more difficult. Current multimodal models integrating facial expression recognition directly learn from pre-trained models on image data, abstracting a high-level semantic representation for direct interaction with other modalities. This approach ignores fine-grained interactions between modalities, causing facial expression information to lose its temporal dimension meaning, thus hindering effective emotion recognition with other modalities. How to extract effective feature parameters and apply appropriate models to express the correlation between these feature parameters and emotions, thereby improving the accuracy of emotion recognition, is a crucial problem that urgently needs to be solved.

[0079] Based on this, in order to better integrate image information with speech and text, this application proposes a multimodal emotion recognition method and apparatus, which can realize multimodal emotion recognition with fine-grained feature fusion, extract features of facial expressions at different levels, and progressively supplement them into the time-level and sentence-level features of speech and text.

[0080] It should be noted that the multimodal emotion recognition method and apparatus disclosed in this application can be used in the field of financial technology, as well as in any other field. The application areas of the multimodal emotion recognition method and apparatus disclosed in this application are not limited. The acquisition, storage, use, and processing of data in the technical solutions of each embodiment of this application all comply with relevant laws and regulations.

[0081] The following examples illustrate this in detail.

[0082] To improve the accuracy of multimodal emotion recognition, this embodiment provides a multimodal emotion recognition method whose execution subject is a multimodal emotion recognition device. This multimodal emotion recognition device includes, but is not limited to, a server, such as... Figure 1 As shown, this method specifically includes the following:

[0083] Step 100: Obtain video data samples and their corresponding actual sentiment tags.

[0084] Specifically, the video data samples can be in batches; a single video data sample can be a video segment; the actual emotion tags can include tags that reflect the true emotions of the video data samples, such as happiness, fear, sadness, and surprise.

[0085] Step 200: Based on the single-modal emotion learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality, obtain the single-modal feature data and emotion feature data of each modality.

[0086] Specifically, the modality can represent the source or form of information; the raw data of the multiple modalities can represent words, frames, and images.

[0087] Preferably, five frames containing human faces can be uniformly extracted from a single video data sample at equal time intervals, and each face can be extracted into a 112×112 image using the OpenFace face recognition model. As the raw image feature data, speech was separated from the video data samples. Frame segmentation was performed using a 25ms Hamming window and a 10ms frame shift. For each frame, a 26-dimensional log_Mel filter bank feature was calculated, followed by the calculation of its first and second derivatives to form a 78-dimensional acoustic feature set. As the raw speech feature data, N is the number of samples, and f is the number of frames. The transcribed text is then encoded into word vectors using a pre-trained BERT model, forming 768-dimensional text features. As the original speech feature data, d represents the number of words.

[0088] Step 300: Obtain sentiment prediction data based on the multimodal sentiment recognition model and the raw data of each modality.

[0089] Specifically, fine-grained interactions between modalities can be achieved by applying a multimodal emotion recognition model.

[0090] Step 400: Substitute the sentiment feature data of each modality, the sentiment prediction data, and the actual sentiment labels into the overall loss function, adjust the parameters of the multimodal sentiment recognition model and each unimodal sentiment learning model, and complete the training of the multimodal sentiment recognition model when the overall loss function value is minimized. The overall loss function includes the loss functions corresponding to the multimodal sentiment recognition model and each unimodal sentiment learning model.

[0091] Specifically, the overall loss function can be added to the model training process, and the gradient descent algorithm can be used for iteration until the maximum number of iterations is reached or the model converges.

[0092] Step 500: Obtain the video data to be identified, and determine the emotion recognition result corresponding to the video data to be identified based on the trained multimodal emotion recognition model and the video data to be identified.

[0093] Specifically, the emotion recognition result can be happiness, fear, sadness, and surprise, etc.

[0094] To further improve the reliability of single-modal emotion learning, such as Figure 2 As shown, in one embodiment of this application, step 200 includes:

[0095] Step 201: Apply a bidirectional gated recurrent neural network model to encode the original data of the video data samples in each modality to obtain the single-modal feature data of each modality.

[0096] Step 202: Based on the single-modal feature data and attention function of each modality, obtain the attention results of each modality.

[0097] Step 203: Input the attention results of each modality into the fully connected layer of the single-modality emotion learning model corresponding to that modality, and apply the normalized exponential function to process the output data of the fully connected layer to obtain the emotion feature data of that modality.

[0098] Specifically, for text modality s and speech modality t, the recurrent neural network-bidirectional gated recurrent neural network model can be as follows:

[0099]

[0100] in, This represents the j-th original data corresponding to the i-th video data sample in mode E. Let [·,·] represent the single-modal feature data corresponding to the j-th original data in modality E for the i-th video data sample, where [·,·] denotes the tensor concatenation operation (E∈M=s,t). The single-modal feature data corresponding to the j-th original data in modality E for a video data sample can be determined as the single-modal feature data corresponding to modality E. For example, if modality E is text, then... This represents the j-th word corresponding to the i-th video data sample.

[0101] Specifically, the attention function can be as follows:

[0102]

[0103] The specific process of A(·) is as follows:

[0104]

[0105]

[0106]

[0107] Where, θ E V T W and b represent parameters, and T represents the number of original data in the i-th video data sample. This represents the attention result corresponding to the i-th video data sample in modality E. It includes the weighted effect of each word / frame in the i-th video data sample.

[0108] Specifically, 2D convolution and 3D convolution can be performed on the raw data of the image modalities to obtain sequence features. and global features:

[0109]

[0110]

[0111] in, Let ζ represent the j-th image corresponding to the i-th video data sample, and let η represent parameters.

[0112] By applying sequence features and a bidirectional gated recurrent neural network model, single-modal feature data of the image modality is obtained:

[0113]

[0114] in, This represents the single-modal feature data corresponding to the j-th original data in the image modality for the i-th video data sample. The single-modal feature data corresponding to the j-th original data in the image modality can be determined as the single-modal feature data corresponding to the image modality.

[0115] Will Visual features U are obtained by feeding them into an attention mechanism for learning. v ′:

[0116]

[0117] Where, θ v Yes, the relevant parameters. (Will) and The results are stitched together as the attention learning outcome of the image modality. The result was obtained by feeding it into a fully connected layer for sentiment classification:

[0118]

[0119] in, This can represent the probability that the i-th video data sample is classified into the c-th class in modality F. The loss function of the unimodal sentiment learning model corresponding to modality F is... It can be as follows:

[0120]

[0121] Where C represents the total number of video data categories; y ic The i-th video data sample represents the actual sentiment label, i.e., the i-th video data sample belongs to the c-th class; N represents the total number of samples; MLP(·) represents a fully connected layer; softmax(·) represents a normalized exponential function; Let F represent the attention result of mode F (F∈M=s,t,v).

[0122] To achieve fine-grained level interaction between modalities and improve the accuracy of emotion recognition, such as Figure 3 As shown, in one embodiment of this application, step 300 includes:

[0123] Step 310: Apply the single-modal feature data of each modality to perform modal interaction processing to obtain feature data under time-level interaction and feature data under statement-level interaction.

[0124] Step 320: Perform sentiment inference processing based on the feature data under the time-level interaction and the feature data under the sentence-level interaction to obtain sentiment prediction data.

[0125] Specifically, feature data from time-level interactions and feature data from sentence-level interactions can be fused. The feature fusion result is then input into the fully connected layer of a multimodal emotion recognition model. The output data of the fully connected layer is processed using a normalized exponential function to obtain emotion prediction data P. ic As shown in the formula below:

[0126] U pi =[U tli U uli ]

[0127] The result was obtained by feeding it into a fully connected layer for sentiment classification:

[0128] P ic =softmax(MLP(U pi ))

[0129] The loss function corresponding to the multimodal emotion recognition model can be as follows:

[0130]

[0131] To achieve fine-grained level interaction between modalities and improve the accuracy of emotion recognition, in one embodiment of this application, step 310 includes:

[0132] Step 311: Apply the single-modal feature data of each modality to perform temporal-level modal interaction processing to obtain feature data under temporal-level interaction;

[0133] Step 312: Apply the single-modal feature data of each modality to perform sentence-level modal interaction processing to obtain feature data under sentence-level interaction.

[0134] To achieve temporal-level interaction between modalities and improve the accuracy of emotion recognition, in one embodiment of this application, step 311 includes:

[0135] Step 3111: Based on the single-modal feature data of any two modalities, obtain the first correlation matrix of the two modalities.

[0136] Step 3112: Based on the first correlation matrix and normalized exponential function of the two modes, obtain the influence of one of the two modes on the other under the time-level interaction.

[0137] Step 3113: Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data corresponding to one of the two modalities under the time-level interaction.

[0138] Step 3114: Based on the attention function and the fusion feature data of one of the two modalities, determine the attention result from one of the two modalities to the other under the time-level interaction.

[0139] Step 3115: Perform tensor concatenation operation based on the attention results from one of the two modalities to the other under the time-level interaction to obtain the feature data under the time-level interaction.

[0140] Specifically, the attention results corresponding to the transition from mode B to mode A under time-level interaction can be obtained according to the following formula.

[0141]

[0142]

[0143]

[0144]

[0145] in, This represents the single-mode feature data of mode A. Let s represent the single-modal feature data of modality B (A, B∈M=s,t,v and A≠B), where s represents the speech modality, t represents the text modality, and v represents the image modality. Let A(·) represent the first correlation matrix between modes A and B, and let θ represent the attention function. A←B Indicates parameters, This represents the influence of mode B on mode A under time-level interaction. This represents the fused feature data from mode B to mode A under time-level interaction. This represents the attention result from modality B to modality A under time-level interaction. Based on the above formula, we can obtain... and

[0146] Attention results from text modality to speech modality under time-level interaction Attention results from speech modality to text modality Attention results from image modality to speech modality And the attention results corresponding to the image modality to the speech modality Perform tensor concatenation to obtain the feature data U under the aforementioned time-level interaction. tli ,

[0147] To achieve sentence-level interaction between modalities and improve the accuracy of emotion recognition, in one embodiment of this application, step 312 includes:

[0148] Step 3121: Based on the attention results of any two modalities, obtain the second correlation matrix of the two modalities.

[0149] Step 3122: Based on the second correlation matrix and normalized exponential function of the two modalities, obtain the influence of one of the two modalities on the other under the statement-level interaction.

[0150] Step 3123: Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data corresponding to one of the two modalities under the statement level interaction.

[0151] Step 3124: Perform tensor concatenation operation on the fused feature data from one of the two modalities under the horizontal interaction of the statement to obtain the feature data under the horizontal interaction of the statement.

[0152] Specifically, the information fusion result corresponding to modality A to modality B under horizontal interaction of statements can be obtained according to the following formula:

[0153]

[0154]

[0155]

[0156] in, This represents the attention result for mode A. This represents the attention result for modality B (A, B∈M=s,t,v and A≠B). This indicates that, according to actual needs, The result after extending to one dimension This indicates that, according to actual needs, The result after extending to one dimension is as follows: s represents the speech modality, t represents the text modality, and v represents the image modality. This represents the second correlation matrix between mode A and mode B. This indicates the influence of modality B on modality A under horizontal interaction of statements. This represents the fusion feature data from modality B to modality A under horizontal interaction of statements. It can be Let be a matrix of elements. Based on the above formula, we can obtain... and

[0157] Based on the information fusion results corresponding to the text modality to the speech modality under horizontal sentence interaction, it is possible to determine the information fusion results. Information fusion results corresponding to speech modality to text modality Information fusion results corresponding to image modality to speech modality and image modality to text modality The corresponding information fusion results are subjected to tensor concatenation to obtain the feature data U under the horizontal interaction of the statement. uli ,

[0158] Specifically, the overall loss function L overall It can be as follows:

[0159]

[0160] in, This represents the loss function corresponding to the multimodal emotion recognition model. This represents the loss function of the unimodal emotion learning model corresponding to different modalities.

[0161] To further illustrate this solution, this application provides an application example of a multimodal emotion recognition method, as described in detail below:

[0162] Step 1: Data Preprocessing. Five frames containing human faces can be uniformly extracted from a single emotional video segment sample at equal time intervals. The OpenFace face recognition model is then used to extract each face into a 112×112 pixel image. Speech was separated from video and segmented using a 25ms Hamming window and a 10ms frame shift. For each frame, a 26-dimensional log_Mel filter bank feature was calculated, followed by the calculation of its first and second derivatives to form a 78-dimensional acoustic feature set. Where f is the number of frames. The transcribed text is encoded into word vectors using a pre-trained BERT model, forming 768-dimensional text features. d represents the number of words.

[0163] Step 2: Modal feature encoding. For text and speech data, a Bi-GRU recurrent neural network is used for encoding:

[0164]

[0165]

[0166] in, This represents the j-th word of the i-th sample. This represents the j-th frame of the i-th sample. and This is the representation in the new feature space after Bi-GRU learning, where hd is the number of hidden layer units in the GRU. [·,·] denotes the tensor concatenation operation. For image data, 2D and 3D convolutions are performed on 5 frames of face images respectively to obtain sequence features of a single face. and global features

[0167] S v =Cov2D(X v ;ζ)

[0168] U v =Cov3D(X v ;η)

[0169] Where Cov2D(·) and Cov3D(·) represent the operations of the ResNet18 network under 2D and 3D convolutions, respectively, and ζ and η are network-related parameters. Temporal learning is performed on single temporal face features using Bi-GRU.

[0170]

[0171] Step 3: Time-level interaction. Calculate the correlation matrix between two modes A and B (A, B ∈ M = {s, t, v} and A ≠ B):

[0172] R AB =(H A ) T (H B )

[0173] (a≠b and a, b∈{d,f,5}), then we can obtain the influence of features in mode A on features in mode B or the influence of features in mode B on features in mode A:

[0174]

[0175] in, The information from mode B is added to mode A using the following formula:

[0176] H A←B =W A←B H B +H A .

[0177] in, The time-level fusion feature H can be obtained through the above methods. s←t H s→t H s ←v and Ht←v By using the attention mechanism A(·), the corresponding sentence-level features can be obtained:

[0178]

[0179] Correspondingly, and Sentence-level features, based on time-level interactions, serve as the foundation for sentiment prediction.

[0180] Step 4: Sentence-level interaction. The temporal encoded features are fed into the attention function A(·), and the impact of each word / frame / image on the global information is calculated to obtain sentence-level text features. speech features and visual characteristics:

[0181] U t =A(H t ;θ t ),

[0182] U s =A(H s ;θ s ),

[0183] U v ′=A(H v ;θ v ),

[0184] θ t θ s and θ v These are the corresponding parameters. For statement-level interactions, the same relation matrix calculation method is used to first calculate the correlation of statement-level features in a single modality:

[0185] E AB =(U A ) T (U B ),

[0186] in, This allows us to determine the influence of features in modality A on features in modality B, or the influence of features in modality B on features in modality A, at the statement level.

[0187]

[0188] in, The information from mode B is added to mode A using the following formula:

[0189]

[0190] in and This serves as a feature under horizontal interaction of statements.

[0191] Step 5: Loss Function Construction. After feeding the unimodal sentence-level features into a fully connected layer, the classification loss is obtained by constraining them with sentiment labels.

[0192]

[0193]

[0194] C represents the total number of emotion categories, y i These are the sample labels. MLP(·) represents a fully connected layer, while softmax(·) is a normalized exponential function. For visual features, U... v and U v The combined features are used to identify the emotion of an expression.

[0195]

[0196] Sentiment prediction is performed by integrating features from both temporal and sentence-level interactions. The features from the temporal-level interaction are aggregated with those from the sentence-level interaction. p =[U sl U ul Perform sentiment prediction:

[0197] P = softmax(MLP(U) p ))

[0198] The corresponding loss function is:

[0199]

[0200] By combining the loss functions from the above process, the following overall loss function is obtained:

[0201]

[0202] L overall This is incorporated into the model training process, using the gradient descent algorithm iteratively until the maximum number of iterations is reached or the model converges. The overall framework diagram is as follows: Figure 4As shown. The method for obtaining sentiment prediction data during model training is the same as that for obtaining sentiment prediction data during sentiment recognition. When determining the sentiment recognition result corresponding to the video data to be recognized based on the trained multimodal sentiment recognition model and the video data to be recognized, the descriptions of steps 1 to 4 can be used to obtain the sentiment prediction data corresponding to the video data to be recognized. The sentiment prediction data is then input into the fully connected layer of the trained multimodal sentiment recognition model, and the output data of the fully connected layer is processed using a normalized exponential function to obtain the output result of the trained multimodal sentiment recognition model. Based on this output result, the sentiment recognition result corresponding to the video data to be recognized is determined.

[0203] In one example, the experimental results (Accuracy) on the multimodal public dataset IEMOCAP are shown in Table 1.

[0204] Table 1

[0205]

[0206]

[0207] Among them, 0.596, 0.673, 0.501, 0.748, 0.613, 0.711 and 0.774 represent the experimental results obtained by performing emotion recognition on video data in the audio modality, text modality, visual modality, combined audio and text modality, combined audio and visual modality, combined text and visual modality, and combined audio, text and visual modality, respectively.

[0208] From a software perspective, to improve the accuracy of multimodal emotion recognition, this application provides an embodiment of a multimodal emotion recognition device for implementing all or part of the multimodal emotion recognition method, see [link to embodiment]. Figure 5 The multimodal emotion recognition device specifically includes the following components:

[0209] The sample acquisition module 51 is used to acquire video data samples and their corresponding actual sentiment tags;

[0210] The single-modal emotion learning module 52 is used to obtain single-modal feature data and emotion feature data of each modality based on the single-modal emotion learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality.

[0211] The sentiment prediction module 53 is used to obtain sentiment prediction data based on the multimodal sentiment recognition model and the raw data of each modality.

[0212] Training module 54 is used to substitute the emotional feature data of each modality, the emotional prediction data, and the actual emotional labels into the overall loss function, adjust the parameters of the multimodal emotion recognition model and each unimodal emotion learning model, and complete the training of the multimodal emotion recognition model when the overall loss function value is minimized. The overall loss function includes the loss functions corresponding to the multimodal emotion recognition model and each unimodal emotion learning model.

[0213] The emotion recognition module 55 is used to acquire video data to be recognized and determine the emotion recognition result corresponding to the video data to be recognized based on the trained multimodal emotion recognition model and the video data to be recognized.

[0214] In one embodiment of this application, the single-modal emotion learning module includes:

[0215] The encoding module is used to apply a bidirectional gated recurrent neural network model to encode the original data of the video data samples in each modality to obtain the single-modal feature data of each modality.

[0216] The attention processing submodule is used to obtain the attention results for each modality based on the single-modal feature data and attention function of each modality;

[0217] The normalization processing submodule is used to input the attention results of each modality into the fully connected layer of the single-modality emotion learning model corresponding to that modality, and apply the normalization exponential function to process the output data of the fully connected layer to obtain the emotion feature data of that modality.

[0218] In one embodiment of this application, the sentiment prediction module includes:

[0219] The modal interaction submodule is used to apply the single modal feature data of each modality to perform modal interaction processing, and obtain feature data under time-level interaction and feature data under sentence-level interaction.

[0220] The sentiment reasoning submodule is used to perform sentiment reasoning processing based on the feature data under the time-level interaction and the feature data under the sentence-level interaction to obtain sentiment prediction data.

[0221] In one embodiment of this application, the modal interaction submodule includes:

[0222] The time-level modal interaction unit is used to perform time-level modal interaction processing on the single-modal feature data of each modality to obtain feature data under time-level interaction.

[0223] The statement-level modal interaction unit is used to perform statement-level modal interaction processing using single-modal feature data from each modality, thereby obtaining feature data under statement-level interaction.

[0224] In one embodiment of this application, the time-level modal interaction unit is specifically used for:

[0225] Based on the single-modal feature data of any two modalities, the first correlation matrix of the two modalities is obtained;

[0226] Based on the first correlation matrix and normalized exponential function of the two modes, the influence of one of the two modes on the other under time-level interaction is obtained;

[0227] Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data from one of the two modalities to the other under the time-level interaction.

[0228] Based on the attention function and the fusion feature data of one of the two modalities, determine the attention result from one of the two modalities to the other under the time-level interaction.

[0229] Tensor concatenation is performed on the attention results from one of the two modalities to the other under the time-level interaction to obtain the feature data under the time-level interaction.

[0230] In one embodiment of this application, the statement-based modal interaction unit is specifically used for:

[0231] Based on the attention results of any two modalities, the second correlation matrix of the two modalities is obtained;

[0232] Based on the second correlation matrix and normalized exponential function of the two modalities, the influence of one of the two modalities on the other under the statement-level interaction is obtained;

[0233] Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data corresponding to one of the two modalities under the horizontal interaction of statements;

[0234] Tensor concatenation is performed on the fused feature data from one of the two modalities under the horizontal interaction of the statement to obtain the feature data under the horizontal interaction of the statement.

[0235] Specifically, the overall loss function L overall It can be as follows:

[0236]

[0237] in, This represents the loss function corresponding to the multimodal emotion recognition model. This represents the loss function of the unimodal emotion learning model corresponding to different modalities.

[0238] The embodiments of the multimodal emotion recognition device provided in this specification can be used to execute the processing flow of the embodiments of the multimodal emotion recognition method described above. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the multimodal emotion recognition method described above.

[0239] To further illustrate this plan, such as Figure 6 As shown, this application provides an application example of a multimodal emotion recognition device, specifically including:

[0240] The data preprocessing module is used to separate text, audio, and video data from the video, and obtain acoustic features through relevant feature extraction. Text features and image information Where N is the number of samples, f is the number of frames, 78 and 768 are the feature encoding dimensions, and 5 is the number of frames extracted from a sample video. The data preprocessing module performs the same function as the sample acquisition module described above.

[0241] The single-modal emotion learning module encodes text and speech data using a bidirectional gated recurrent unit (Bi-GRU).

[0242]

[0243]

[0244] in This represents the j-th word / frame of the i-th sample. and This is the representation in the new feature space after Bi-GRU learning, where hd is the number of hidden units in the GRU. [·,·] represents the tensor concatenation operation. The above features are fed into the attention function A(·) to calculate the influence of each word / frame on the global information, obtaining the sentence-level text features. and speech features

[0245] U t =A(H t ;θ t ),

[0246] U s =A(H s ;θ s ),

[0247] θ t and θ s These are the corresponding parameters. The specific process of A(·) is as follows: for the t-th word / frame level h of the input... t ∈H, now perform the mapping:

[0248] c t =V T tanh(Wh t +b),

[0249] V T W and b are the corresponding parameters. Calculate the weight of each word / frame in all the data:

[0250]

[0251] Where T∈{f,d}. Weighted features at the statement level. for:

[0252]

[0253] The value in 'u' represents the weighted sum of each word / frame in a sentence. After being fed into a fully connected layer, a classification loss is obtained using sentiment labels as constraints.

[0254]

[0255]

[0256] C represents the total number of emotion categories, y i These are the sample labels. MLP(·) represents a fully connected layer, while softmax(·) is a normalized exponential function.

[0257] For the image data, 2D convolution kernels and 3D convolution are applied to 5 frames of face images respectively to obtain sequence features of a single face. and global features

[0258] S v =Cov2D(X v ;ζ)

[0259] U v =Cov3D(X v ;η)

[0260] Where Cov2D(·) and Cov3D(·) represent the operations of the ResNet18 network under 2D and 3D convolutions, respectively, and ζ and η are network-related parameters. Temporal learning is performed on single temporal face features using Bi-GRU.

[0261]

[0262] H v The visual features U learned by the temporal model are fed into the attention mechanism for learning. v ′:

[0263] U v ′=A(H v ;θ v )

[0264] θ v Yes, that's the relevant parameter. Therefore, U v ′ and U v The combined features are used to identify the emotion of an expression.

[0265]

[0266] The modal interaction module includes temporal-level interaction and sentence-level interaction. Temporal-level interaction occurs when the modal data contains time-series information, primarily manifested in pairwise interactions between text and speech features, and the unilateral influence of facial expression features on speech and text. Details are as follows.

[0267] First, calculate the correlation matrix between two modes A and B (A, B ∈ M = {s, t, v} and A ≠ B):

[0268] R AB =(H A ) T (H B )

[0269] (a≠b and a, b∈{d,f,5}) Then we can obtain the influence of features in mode A on features in mode B or the influence of features in mode B on features in mode A:

[0270]

[0271] in The information from mode B is added to mode A using the following formula:

[0272] H A←B =W A←B H B +H A

[0273] in The time-level fusion feature H can be obtained through the above methods. s←t H t←s H s←v and H t←v By utilizing attention mechanisms, we can obtain corresponding sentence-level features:

[0274]

[0275] Correspondingly, and Sentence-level features, based on time-level interactions, serve as the foundation for sentiment prediction. For statement-level interactions, the same approach is used to first calculate the correlation of statement-level features in a single modality:

[0276] E AB =(U A ) T (U B )

[0277] in This allows us to determine the influence of features in modality A on features in modality B, or the influence of features in modality B on features in modality A, at the statement level.

[0278]

[0279] in The information from mode B is added to mode A using the following formula:

[0280]

[0281] in and This serves as a feature under horizontal interaction of statements.

[0282] The sentiment inference module integrates features from time-level and sentence-level interactions for sentiment prediction. It aggregates features from time-level interactions with features from sentence-level interactions. p =[U sl U ul Perform sentiment prediction:

[0283] P = softmax(MLP(U) p ))

[0284] The corresponding loss function is:

[0285]

[0286] Combining the loss functions in the above process, we obtain the following overall loss function:

[0287]

[0288] L overall By incorporating gradient descent into the model training process and iterating until the maximum number of iterations is reached or the model converges, the trained multimodal emotion recognition model can be applied for emotion recognition. The functionality achieved by combining the modal interaction module and the emotion reasoning module is equivalent to the functionality achieved by combining the emotion prediction module and the training module.

[0289] From a hardware perspective, in order to improve the accuracy of multimodal emotion recognition, this application provides an embodiment of an electronic device for implementing all or part of the multimodal emotion recognition method. The electronic device specifically includes the following components:

[0290] The device comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the multimodal emotion recognition device and related devices such as user terminals; the electronic device can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the electronic device can be implemented with reference to the embodiments for implementing the multimodal emotion recognition method and the embodiments for implementing the multimodal emotion recognition device, the content of which is incorporated herein, and repeated parts will not be described again.

[0291] Figure 7 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 7 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 7 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0292] In one or more embodiments of this application, multimodal emotion recognition functionality can be integrated into a central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0293] Step 100: Obtain video data samples and their corresponding actual sentiment tags;

[0294] Step 200: Based on the single-modal emotion learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality, obtain the single-modal feature data and emotion feature data of each modality;

[0295] Step 300: Obtain sentiment prediction data based on the multimodal sentiment recognition model and the raw data of each modality;

[0296] Step 400: Substitute the sentiment feature data of each modality, the sentiment prediction data, and the actual sentiment labels into the overall loss function, adjust the parameters of the multimodal sentiment recognition model and each unimodal sentiment learning model, and complete the training of the multimodal sentiment recognition model when the overall loss function value is minimized. The overall loss function includes the loss functions corresponding to the multimodal sentiment recognition model and each unimodal sentiment learning model.

[0297] Step 500: Obtain the video data to be identified, and determine the emotion recognition result corresponding to the video data to be identified based on the trained multimodal emotion recognition model and the video data to be identified.

[0298] As can be seen from the above description, the electronic device provided by the embodiments of this application can improve the accuracy of multimodal emotion recognition.

[0299] In another embodiment, the multimodal emotion recognition device can be configured separately from the central processing unit 9100. For example, the multimodal emotion recognition device can be configured as a chip connected to the central processing unit 9100, and the multimodal emotion recognition function can be realized through the control of the central processing unit.

[0300] like Figure 7 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 7 All components shown; in addition, the electronic device 9600 may also include Figure 7 For components not shown, please refer to existing technologies.

[0301] like Figure 7 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0302] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0303] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0304] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0305] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0306] The communication module 9110 is a transmitter / receiver 9110 that transmits and receives signals via the antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0307] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored sound via the speaker 9131.

[0308] As can be seen from the above description, the electronic device provided in the embodiments of this application can improve the accuracy of multimodal emotion recognition.

[0309] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the multimodal emotion recognition method in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the multimodal emotion recognition method in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0310] Step 100: Obtain video data samples and their corresponding actual sentiment tags;

[0311] Step 200: Based on the single-modal emotion learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality, obtain the single-modal feature data and emotion feature data of each modality;

[0312] Step 300: Obtain sentiment prediction data based on the multimodal sentiment recognition model and the raw data of each modality;

[0313] Step 400: Substitute the sentiment feature data of each modality, the sentiment prediction data, and the actual sentiment labels into the overall loss function, adjust the parameters of the multimodal sentiment recognition model and each unimodal sentiment learning model, and complete the training of the multimodal sentiment recognition model when the overall loss function value is minimized. The overall loss function includes the loss functions corresponding to the multimodal sentiment recognition model and each unimodal sentiment learning model.

[0314] Step 500: Obtain the video data to be identified, and determine the emotion recognition result corresponding to the video data to be identified based on the trained multimodal emotion recognition model and the video data to be identified.

[0315] As can be seen from the above description, the computer-readable storage medium provided in the embodiments of this application can improve the accuracy of multimodal emotion recognition.

[0316] The various embodiments of the methods described in this application are presented in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on explaining the differences from other embodiments. Relevant details can be found in the descriptions of the method embodiments.

[0317] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0318] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0319] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0320] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0321] This application uses specific embodiments to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal emotion recognition method, characterized in that, include: Obtain video data samples and their corresponding actual sentiment tags; Based on the single-modal emotion learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality, the single-modal feature data and emotion feature data of each modality are obtained. Modal interaction processing is performed using the single-modal feature data of each modality to obtain feature data under time-level interaction and feature data under sentence-level interaction. The feature data under the time-level interaction and the feature data under the sentence-level interaction are fused. The feature fusion result is input into the fully connected layer of the multimodal emotion recognition model. The output data of the fully connected layer is processed by the normalized exponential function to obtain the emotion prediction data. The emotional feature data of each modality, the emotional prediction data, and the actual emotional labels are substituted into the overall loss function, and the parameters of the multimodal emotion recognition model and each unimodal emotion learning model are adjusted. The training of the multimodal emotion recognition model is completed when the value of the overall loss function is minimized. The overall loss function includes the loss functions corresponding to the multimodal emotion recognition model and each unimodal emotion learning model. Acquire the video data to be identified, and determine the emotion recognition result corresponding to the video data to be identified based on the trained multimodal emotion recognition model and the video data to be identified. Wherein, the overall loss function L overall as follows: in, This represents the loss function corresponding to the multimodal emotion recognition model. This represents the loss function of the unimodal emotion learning model corresponding to different modalities.

2. The multimodal emotion recognition method according to claim 1, characterized in that, The step of obtaining unimodal feature data and sentiment feature data for each modality based on the unimodal sentiment learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality includes: By applying a bidirectional gated recurrent neural network model, the original data of the video data samples in each modality are encoded to obtain the single-modal feature data of each modality. Based on the single-modal feature data and attention function of each modality, the attention results of each modality are obtained; The attention results for each modality are input into the fully connected layer of the corresponding single-modality emotion learning model. The output data of the fully connected layer is processed by a normalized exponential function to obtain the emotion feature data for that modality.

3. The multimodal emotion recognition method according to claim 1, characterized in that, The application of single-modal feature data from each modality for modal interaction processing yields feature data for time-level interaction and feature data for statement-level interaction, including: By applying the single-modal feature data of each modality to perform temporal-level modal interaction processing, feature data under temporal-level interaction is obtained; By applying the unimodal feature data of each modality to perform sentence-level modal interaction processing, feature data under sentence-level interaction is obtained.

4. The multimodal emotion recognition method according to claim 3, characterized in that, The application of single-modal feature data of each modality is used for time-level modal interaction processing to obtain feature data under time-level interaction, including: Based on the single-modal feature data of any two modalities, the first correlation matrix of the two modalities is obtained; Based on the first correlation matrix and normalized exponential function of the two modes, the influence of one of the two modes on the other under time-level interaction is obtained; Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data from one of the two modalities to the other under the time-level interaction. Based on the attention function and the fusion feature data of one of the two modalities, determine the attention result from one of the two modalities to the other under the time-level interaction. Tensor concatenation is performed on the attention results from one of the two modalities to the other under the time-level interaction to obtain the feature data under the time-level interaction.

5. The multimodal emotion recognition method according to claim 3, characterized in that, The application of single-modal feature data from each modality to perform statement-level modal interaction processing yields feature data under statement-level interaction, including: Based on the attention results of any two modalities, the second correlation matrix of the two modalities is obtained; Based on the second correlation matrix and normalized exponential function of the two modalities, the influence of one of the two modalities on the other under the statement-level interaction is obtained; Based on the single-modal feature data of the two modalities and the influence of one of the two modalities on the other, determine the fusion feature data corresponding to one of the two modalities under the horizontal interaction of statements; Tensor concatenation is performed on the fused feature data from one of the two modalities under the horizontal interaction of the statement to obtain the feature data under the horizontal interaction of the statement.

6. A multimodal emotion recognition device, characterized in that, include: The sample acquisition module is used to acquire video data samples and their corresponding actual sentiment tags; The single-modal emotion learning module is used to obtain single-modal feature data and emotion feature data for each modality based on the single-modal emotion learning models corresponding to each of the multiple modalities and the original data of the video data samples in each modality. The sentiment prediction module is used to obtain sentiment prediction data based on the multimodal sentiment recognition model and the raw data of each modality. The training module is used to substitute the emotional feature data of each modality, the emotional prediction data, and the actual emotional labels into the overall loss function, adjust the parameters of the multimodal emotion recognition model and each unimodal emotion learning model, and complete the training of the multimodal emotion recognition model when the value of the overall loss function is minimized. The overall loss function includes the loss functions corresponding to the multimodal emotion recognition model and each unimodal emotion learning model. The emotion recognition module is used to acquire video data to be recognized, and determine the emotion recognition result corresponding to the video data to be recognized based on the trained multimodal emotion recognition model and the video data to be recognized. The sentiment prediction module includes: The modal interaction submodule is used to apply the single modal feature data of each modality to perform modal interaction processing, and obtain feature data under time-level interaction and feature data under sentence-level interaction. The sentiment reasoning submodule is used to fuse the feature data under the time-level interaction and the feature data under the sentence-level interaction, input the feature fusion result into the fully connected layer of the multimodal sentiment recognition model, and apply the normalized exponential function to process the output data of the fully connected layer to obtain sentiment prediction data. Wherein, the overall loss function L overall as follows: in, This represents the loss function corresponding to the multimodal emotion recognition model. This represents the loss function of the unimodal emotion learning model corresponding to different modalities.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimodal emotion recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instruction is executed, it implements the multimodal emotion recognition method according to any one of claims 1 to 5.

Images