Multi-source information emotion recognition method based on common attention

Abstract

The invention discloses a multi-source information emotion recognition method based on common attention, and the method comprises the steps: 1, carrying out the preprocessing of video data, separating an audio stream from a video stream, and obtaining a video frame sequence xv and audio data xa; step 2, manually extracting an MFCC acoustic feature xm from the audio data xa as a part of input; 3, based on the input of the video frame sequence xv, the audio data xa and the MFCC acoustic feature xm, modeling and coding are carried out on the video frame sequence xv, the audio data xa and the MFCC acoustic feature xm respectively, and feature representations of deeper levels are obtained; and step 4, classifying the data processed in the step 3 through a full connection layer and a Softmax function to obtain an emotion recognition classification result. According to the method, the acoustic features and the semantic features are extracted from the voice, the global features are concerned in the video, the attention weights are generated by the acoustic and visual features to act on the semantic features, and the emotion recognition effect is better by using the cooperative relationship between different information sources.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a multi-source information emotion recognition method based on common attention. Background Art

[0002] Human emotions play a crucial role in social interactions, decision-making, and mental health. Therefore, accurately identifying and understanding human emotions is crucial for improving the human-computer interaction experience, enhancing the emotional intelligence of intelligent systems, and advancing mental health research. In recent years, with the advancement of technologies such as artificial intelligence, big data, and machine learning, emotion recognition and affective computing have become important research areas in the field of artificial intelligence. By enabling computers to identify emotions in multimodal data such as sound, images, and text, human-computer interactions become more intelligent and humane. Furthermore, emotion recognition has widespread applications in fields such as medicine, education, and social networking, significantly enhancing user experience and service quality, and providing valuable insights for the progress and development of human society.

[0003] The essence of emotion recognition is to use different mathematical methods to find the mapping relationship between various emotion-related feature data and internal emotional states, and convert them into computer algorithms to assist in judging people's emotional states.

[0004] With the development of technologies such as artificial intelligence, natural language processing, and computer vision, the field of emotion recognition has made significant progress, especially the integrated application of multimodal and multi-source information, which has brought new opportunities and challenges to emotion recognition. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-source information emotion recognition method based on common attention to solve the problems in the prior art of insufficient data generalization and neglect of global emotion characteristics.

[0006] The technical solution adopted in the present invention is:

[0007] A multi-source information emotion recognition method based on joint attention, comprising:

[0008] Step 1: Preprocess the video data, separate the audio stream and the video stream, and align the isochronous slices in the time dimension to obtain the video frame sequence x v and audio data x a ;

[0009] Step 2: From audio data x a Extract MFCC acoustic features x m ;

[0010] Step 3: Use the 3D convolutional network to model and encode the video data to obtain the video frame feature code x vf, use the Wav2Vec 2.0 model to model and encode the audio data to obtain the audio semantic feature encoding x af , use the bidirectional long short-term memory network to model and encode the MFCC acoustic features to obtain the MFCC acoustic feature encoding x mf , combining fully connected layers and using a common attention mechanism to fuse the three encodings;

[0011] Step 4: Classify the data processed in step 3 through the fully connected layer and Softmax function to obtain the emotion recognition classification results.

[0012] Optionally, the step three specifically includes:

[0013] Step 3.1: Use the pre-trained I3D model to model and encode the video frame sequence. The output of the average pooling layer is converted into a vector of size D×1 through the Flatten operation and the fully connected layer, which is used as the encoding representation x of the video frame sequence. vf , where D is the final feature unified dimension;

[0014] Step 3.2, use the bidirectional long short-term memory network to transform the MFCC acoustic features x m As input for further feature encoding, the output is a two-dimensional matrix of shape [t, s0×n], where t is the length of the time dimension of the input MFCC feature, s0 and n are the hidden layer size and layer number parameters of the Bi-LSTM model; after the Flatten operation and the fully connected layers, the output dimension is reduced to a D×1 vector, which is used as the encoding representation x of the MFCC feature. mf ;

[0015] Step 3.3: Use the pre-trained wav2vec 2.0 model to model and encode the original audio data, and use the final hidden state parameters of the model as the encoding representation x of the audio semantics. af , whose shape is [l,s1], s1 is the hidden layer dimension of the model, which is the internal parameter of the model. The relationship between l and the input is expressed as:

[0016]

[0017] where len(x a ) is the input sequence x a sr is the audio sampling rate.

[0018] Optionally, the step 4 specifically includes: encoding the video frame feature x through the common attention module vf , audio semantic feature encoding x af and MFCC acoustic feature encoding x mfThe prediction results are compared with the actual prediction labels, and the error value is calculated through the CrossEntropyLoss loss function and recorded in real time to feed back to the model for supervised training.

[0019] Optionally, the step 4 specifically includes:

[0020] Step 4.1, video frame feature encoding x vf and MFCC acoustic feature encoding x mf After the splicing operation, it is used as the input sequence of the joint attention module and the weight matrix x is generated through the linear embedding layer. att , whose shape is [1,l]; the process can be expressed as

[0021] Step 4.2, audio semantic feature encoding x af As another input sequence of the joint attention module, it is combined with the weight matrix x att Multiply to get the weighted code x af-att , whose shape is [1,s1]; the process can be expressed as x af-att =x att ·x af ;

[0022] Step 4.3, use the fully connected layer to weight the encoding x af-att Perform dimension unification and process to obtain a vector x′ of length D af-att ; x vf 、x mf and x′ af-att After splicing, it is used as the final output feature representation x; the process can be expressed as:

[0023] x′ af-att =σ(W·x af-att +b);

[0024]

[0025] The weight matrix Bias vector b∈R D , σ represents a nonlinear activation function.

[0026] Optionally, the CrossEntropyLoss loss function is expressed as:

[0027]

[0028] Among them, yi Indicates the true label, the value is 0 or 1; is the probability predicted by the model. The built-in Softmax layer of the CrossEntropyLoss loss function converts the discrete type of emotion recognition classification task output value into a probability distribution ranging from [0, 1] and summing to 1. C represents the number of emotion categories.

[0029] Optionally, the step 2 is as follows: each audio data x a The two-dimensional MFCC acoustic feature x is represented as t×d m , where t is the time dimension size and d is the dimension size of feature representation.

[0030] Optionally, Dropout regularization is used in steps 3 and 4 to prevent overfitting. The specific process is as follows: during the training phase, neuron outputs are randomly masked with probability p so that the network does not overly rely on the connections between specific neurons, thereby reducing model complexity and suppressing overfitting.

[0031] More specific plans include:

[0032] Step 1: Separate the audio stream and video stream and perform isochronous slicing processing;

[0033] Step 2, extract MFCC acoustic features;

[0034] Step 3: construct modeling and encoding modules for the three information sources of video frame sequence, MFCC acoustic features and audio data respectively;

[0035] Step 4: Construct a fusion module based on the joint attention mechanism; the encoding module and the fusion module together constitute a multi-source information emotion recognition model based on joint attention;

[0036] In step 5, the video frame sequence, MFCC acoustic features, and audio data of each piece of data are input into the emotion recognition model for supervised training to obtain the classification results.

[0037] Optionally, the step three is specifically as follows: based on the video frame sequence x v , audio data x a and MFCC features x m The input is modeled and encoded respectively to obtain the feature representation of each deeper level;

[0038] Use the pre-trained I3D model to model and encode the video frame sequence. The encoding representation of the video frame sequence is x vf ;

[0039] Use the bidirectional long short-term memory network to take the MFCC features as input for further feature encoding, and obtain the encoding representation x of the MFCC featuresmf ;

[0040] The pre-trained wav2vec 2.0 model is used to model and encode the raw audio data, and the final hidden state parameters of the model are used as the encoding representation x of the audio semantics. af .

[0041] Optionally, the step 4 is specifically as follows: fusing the codes obtained from different information sources through the common attention module, including the video frame feature code x vf , audio semantic feature encoding x af and MFCC acoustic feature encoding x mf ; At the same time, pay attention to the relevant information between each coding sequence.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] Video data contains both visual and audio information. With the prevalence of short videos online, video data is often the most readily available general data format. In contrast, emotion recognition methods based on multi-source information leverage both the visual and audio information of raw video data to achieve emotion recognition, thus offering a wider range of applications. They simultaneously extract acoustic and semantic features from speech and focus on global features in videos. Acoustic and visual features generate attention weights that act on semantic features, leveraging the synergy between different information sources to achieve better emotion recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the present invention but do not constitute a limitation of the present invention. In the accompanying drawings:

[0045] Figure 1 Schematic diagram of the process of the multi-source information emotion recognition method based on common attention of the present invention. DETAILED DESCRIPTION

[0046] It should be noted that, unless otherwise specified, all models and algorithms in the present invention adopt models and algorithms known in the prior art.

[0047] Emotional expression is an extremely complex process, typically integrating facial expressions, body movements, verbal content, voice intonation, and other aspects of emotional expression. Existing emotion recognition technologies mostly rely on facial expressions and textual content, which often use specialized data that doesn't conform to general data characteristics. Specific issues include: 1. The data is collected in a specific format for specific scenarios; 2. The data received by the model requires extensive preprocessing, sometimes even requiring manual intervention; and 3. The focus on local features neglects other global features.

[0048] The overall technical concept of the present invention is as follows: first, the video data expressing emotions is preprocessed, the audio and video data are separated, and isochronous slices are aligned in the time dimension. Subsequently, the Mel-frequency cepstral coefficient (MFCC) acoustic features are extracted from the audio data. Then, the video data is modeled and encoded using a three-dimensional convolutional network, the audio data is modeled and encoded using a Wav2Vec 2.0 model, and the MFCC acoustic features are modeled and encoded using a bidirectional long short-term memory network. The three encodings are fused by combining a fully connected layer and using a common attention mechanism to improve the accuracy of classification. In order to improve the performance and generalization ability of the model, Dropout regularization is used to prevent overfitting. Finally, the processed data is classified by a fully connected layer and a Softmax function to obtain more accurate emotion recognition classification results.

[0049] In accordance with the above technical solution, the specific implementation method of the present invention is given below. It should be noted that the present invention is not limited to the following specific implementation method, and all equivalent changes made on the basis of the technical solution of this application fall within the scope of protection of the present invention.

[0050] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Part 1:

[0052] Combine Figure 1 , describing the complete steps of the method of the present invention:

[0053] Step 1: Separate the audio stream and video stream and perform isochronous slicing; pre-process the video data to separate the audio stream and video stream. In order to ensure that the audio and video data are aligned in the time dimension, the data is sliced ​​in an isochronous manner. Specifically, the continuous audio and video streams are sliced ​​according to a fixed time window to generate a series of short-term data segments of equal length. Since the sampling rates of audio and video may be different, some redundant frames are discarded or copied and filled during the slicing process through mathematical calculations. This processing method can effectively eliminate the problem of audio and video data being out of sync due to different sampling rates, ensure the consistency of multimodal data in the time dimension, and thus provide a reliable basis for subsequent feature extraction and fusion. Finally, the video frame sequence x is obtained. v and audio data x a .

[0054] Step 2: Extract MFCC acoustic features. To improve the synergy between multi-source information, manually extract MFCC (Mel Frequency Cepstral Coefficents) acoustic features from the audio data as part of the input. The MFCC features calculated through pre-emphasis, framing, windowing, fast Fourier transform (FFT), logarithmic transform, discrete cosine transform, and dynamic feature calculation can simulate the human ear's perception of sound and reflect the speaker's voice and intonation, thereby representing the effective information in the speech signal. Finally, each audio is represented as a t×d two-dimensional MFCC feature x m , where t is the time dimension and d is the dimension of feature representation.

[0055] Step 3: construct modeling and coding modules for three information sources: video frame sequence, MFCC acoustic features, and audio data; based on the video frame sequence x v , audio data x a and MFCC features x m The input of is modeled and encoded respectively to obtain deeper feature representations of each, reduce the coupling between multi-source information, and further increase the synergistic effect of the model on different information sources. This step specifically includes the following steps:

[0056] Step 3.1: Use the pre-trained I3D (Inflated 3D ConvNet) model to model and encode the video frame sequence. The I3D model can simultaneously process the spatial and temporal features in the video data. Its key idea is to "inflate" the 2D convolution kernel and add the temporal dimension in addition to the spatial dimension, so that the model can capture the dynamic timing information of the video and thus efficiently extract spatiotemporal features from the video. The output of the average pooling layer is flattened and fully connected to reduce the dimension into a vector of size D×1, which is used as the encoding representation x of the video frame sequence. vf , where D is the final unified feature dimension (the same below).

[0057] In step 3.2, a bidirectional long short-term memory network (Bi-LSTM) is used to take the MFCC features as input for further feature encoding. Bi-LSTM is able to fully capture contextual information when processing sequence data. The MFCC features are encoded by Bi-LSTM and output a two-dimensional matrix of shape [t, s0×n], where t is the length of the time dimension of the input MFCC features, and s0 and n are the hidden layer size and number of layers of the Bi-LSTM model. After the Flatten operation and the fully connected layers, the output dimension is reduced to a D×1 vector, which is used as the encoding representation x of the MFCC features. mf .

[0058] Step 3.3: Use the pre-trained wav2vec 2.0 model to model and encode the original audio data. The wav2vec2.0 model is a self-supervised training model for processing speech signals. It can efficiently extract semantic information from speech signals and use it to describe emotional states. The final hidden state parameters of the model are used as the output code x af , whose shape is [l,s1], s1 is the hidden layer dimension of the model, which is the internal parameter of the model. The relationship between l and the input is expressed as:

[0059]

[0060] where len(x a ) is the input sequence x a sr is the audio sampling rate.

[0061] Step 4: Construct a fusion module based on the common attention mechanism; the encoding module and the fusion module together form a multi-source information emotion recognition model based on common attention; the encodings obtained from different information sources are fused through the common attention (Co-Attention) module, including x vf 、x af and x mf They represent video frame feature coding, audio semantic feature coding, and MFCC acoustic feature coding respectively; while focusing on the relevant information between each coding sequence, thereby improving the model's ability to understand and process data. This step has the following process:

[0062] Step 4.1, video frame feature encoding x vf and MFCC acoustic feature encoding x mf After the splicing operation, it is used as the input sequence of the joint attention module and the weight matrix x is generated through the linear embedding layer. att , whose shape is [1,l]. The process can be expressed as

[0063] Step 4.2, audio semantic feature encoding x af As another input sequence of the joint attention module, it is combined with the weight matrix x att Multiply to get the weighted code x af-att , whose shape is [1,s1]. The process can be expressed as x af-att =x att ·x af .

[0064] Step 4.3, use the fully connected layer to weight the encoding x af-att Perform dimension unification and nonlinear activation function to enable the model to have the ability to represent complex relationships, and process the vector x′ with a length of D af-att ; x vf、x mf and x′ af-att After concatenation, the final output feature representation x is obtained. The process can be expressed as:

[0065] x′ af-att =σ(W·x af-att +b);

[0066]

[0067] The weight matrix Bias vector b∈R D , σ represents a nonlinear activation function.

[0068] In step 5, the video frame sequence, MFCC acoustic features, and audio data for each piece of data are fed into the emotion recognition model for supervised training to obtain the classification results. x is fed into the fully connected layer and the Softmax layer to obtain a C×1 classification result, where C represents the number of emotion categories. The output prediction result is compared with the actual predicted label. The error value is calculated using the CrossEntropyLoss loss function and recorded and fed back to the model in real time for supervised training. Through multiple rounds of iteration and optimization, the final classification result is closer to the actual label.

[0069] The objective function is expressed as:

[0070]

[0071] Among them, y i Indicates the true label, the value is 0 or 1; is the probability predicted by the model. The built-in Softmax layer of the CrossEntropyLoss loss function converts the discrete type of emotion recognition classification task output value into a probability distribution ranging from [0, 1] and summing to 1. C represents the number of emotion categories.

[0072] Part II:

[0073] The theoretical feasibility of this method is mainly reflected in the following three aspects: the data isochronous slicing process ensures the consistency of multimodal data in the temporal dimension, providing a reliable foundation for subsequent feature extraction and fusion; the differential processing of multi-source features and the full utilization of the complementarity of multi-source features enhance the ability to represent emotions; the interactive calculation of the joint attention mechanism can dynamically capture nonlinear cross-modal correlations, making up for the static defects of traditional fusion methods. These theoretical advantages jointly guarantee the efficiency and robustness of this method in emotion recognition tasks, providing a solid theoretical foundation for practical applications. Specifically:

[0074] 1. In order to ensure that the audio and video data are aligned in the time dimension, this method uses isochronous slicing to segment the data. Specifically, the continuous audio and video streams are sliced ​​according to fixed time windows (such as 1 second or 2 seconds) to generate a series of short-term data segments of equal length. This slicing method can not only provide time-aligned input data for the model, but also support real-time prediction application scenarios, because short-term sliced ​​data can be directly input into the model for fast inference. During the slicing process, in order to solve the possibility of inconsistent data frame numbers due to differences in the sampling rates of audio and video, this method adjusts the data through mathematical calculations: for modalities with a large number of frames, uniform sampling is used to discard some redundant frames; for modalities with a small number of frames, missing frames are supplemented by replication.

[0075] 2. This method fully utilizes the complementarity of multi-source features. By fusing MFCC, Wav2Vec 2.0, and I3D feature coding, it can capture acoustic details while extracting semantic information and spatiotemporal dynamic features, significantly improving the comprehensiveness and discriminative ability of emotion representation. Specifically:

[0076] (1) Mel-frequency cepstral coefficients (MFCCs) are a classic acoustic feature that can effectively characterize the spectral characteristics of speech signals, especially the acoustic details related to emotions (such as pitch and speaking rate). Modeling MFCC features using a bidirectional long short-term memory (BiLSTM) network can capture the temporal dependencies in speech signals, thereby enhancing the robustness of emotion recognition.

[0077] (2) Wav2Vec 2.0 is a speech representation model based on self-supervised learning that can extract rich semantic information from raw speech signals. Compared with MFCC, Wav2Vec 2.0 is better at capturing high-level semantic features in speech (such as emotional tendency and tone intensity), thus compensating for the shortcomings of MFCC in semantic representation.

[0078] (3) The expanded three-dimensional convolutional network (I3D) can extract rich spatiotemporal features from video data and effectively capture visual information related to emotions such as facial expressions and body movements. Compared with traditional 2D convolutional networks, I3D can not only model static features in the spatial dimension, but also capture dynamic changes in the temporal dimension, thereby more accurately reflecting the continuous evolution of emotional states. I3D uses the pre-trained parameters of the 2D convolutional network for initialization. This strategy enables the model to have stronger feature extraction capabilities in the early stages of training, thereby accelerating model convergence. At the same time, since the pre-trained model has been learned on a large amount of data, its parameters have better generalization and robustness, further improving the performance of I3D in emotion recognition tasks.

[0079] 3. Through the interactive calculation of the joint attention mechanism, it is possible to effectively capture the nonlinear correlation between different modalities and make up for the static defects of traditional fusion methods. Traditional multimodal fusion methods (such as feature splicing or weighted averaging) usually assume that the interaction between modalities is fixed and cannot dynamically adapt to the modal dependencies in different scenarios. The joint attention mechanism can dynamically allocate attention weights for different modalities, thereby better modeling cross-modal correlations. This design not only enhances the expressive power of the model, but also effectively captures the complex dependencies between modalities. For example, in emotion recognition tasks, changes in tone in audio may be highly correlated with facial expressions in videos. The joint attention mechanism can automatically identify this correlation and assign a higher weight in feature fusion, thereby improving the accuracy of emotion discrimination.

[0080] Part III:

[0081] To verify the feasibility and effectiveness of this invention, the MSIER-CA classification model was evaluated using weighted average accuracy and unweighted average accuracy. For a four-class classification task, there are 16 different combinations of predicted labels and true labels, forming a confusion matrix. This confusion matrix allows the calculation of two metrics: weighted accuracy (WA) and unweighted accuracy (UA).

[0082] The experiments used 10-fold cross-validation to verify the classification results. Random seeding is used when randomly shuffling the training data. To ensure the validity and repeatability of the experimental results, two rounds of experiments were conducted with different random seeds. Weighted accuracy (WA) and unweighted accuracy (UA) were calculated for each experimental result. Finally, the average weighted accuracy and average unweighted accuracy of the two rounds of 20 experiments were calculated as the final evaluation criteria for the experiment.

[0083] Table 1 shows the experimental results of the impact of different feature combinations on model performance on the IEMOCAP dataset. The first group shows the emotion recognition results using single feature encoding input; the second group shows the results of combining MFCC features and I3D features with wav2vec features in different combinations; and the third group shows the experimental results of different combinations of information source encoding features after the joint attention module.

[0084] Table 1 Impact of different feature combination inputs on model performance on the IEMOCAP dataset

[0085] MFCC I3D wav2vec Joint attention WA (%) UA (%) √ — — — 57.80 58.12 — √ — — 55.34 55.38 — — √ — 64.03 65.67 √ — √ — 64.62 65.93 — √ √ — 64.17 66.24 √ √ √ — 67.25 67.83 √ — √ √ 67.52 68.65 — √ √ √ 65.94 67.47 √ √ √ √ 72.91 73.63

[0086] Experimental results show that, in the first set of experiments, semantic features extracted and encoded from audio information using the Wav2Vec 2.0 model performed better than other features. In the second set of experiments, combining multiple source information encodings leveraged complementarity, improving the model's overall performance. In the third set of experiments, the interactive computational approach of joint attention captured the complex relationships between information sources, further optimizing the model's overall performance. Compared to the second set of experiments, the joint attention mechanism improved weighted accuracy by approximately 2 to 5 percentage points and unweighted accuracy by approximately 1 to 5 percentage points, significantly outperforming the direct concatenation approach.

[0087] Part 5:

[0088] Take the IEMOCAP dataset as an example:

[0089] Step 1: Data preprocessing. The video sampling rate is 30 FPS and the audio sampling rate is 16000 Hz. The time window size for isochronous slicing is 3 seconds, that is, the audio and video data are cut into 3-second slices respectively, and the missing part is filled with the last frame data. Since the audio sampling frequency is higher than the video, in order to achieve the effect of aligning the audio and video in the time dimension, some audio sampling points need to be discarded. The resulting video frame sequence x v Length is 90, audio data x a The length is 48000.

[0090] Step 2: Calculate the Mel-frequency cepstral coefficients. The frame length is 400, the frame shift is 160, and the number of MFCC coefficients is 40, i.e., d = 40. According to the calculation rules, the output length t′ = (48000-400) / 160+1≈299 can be calculated. To facilitate subsequent calculations, t = 300 is taken, and the remaining part is filled with the last frame of data. Each audio is represented as a 300×40 two-dimensional MFCC feature x m .

[0091] Step 3: Model and encode the input data. When unifying the dimensions of the encodings obtained from different source data, take D = 128. Specifically:

[0092] Step 3.1: The input shape of the I3D model is [C, T, H, W]. C represents the color channel size of the video frame image. When using RGB channels, C = 3. T represents the time dimension, i.e., the input sequence x v The length of the vector is 90; H and W represent the height and width of each frame image, and H = W = 224. The output of I3D is Flattened, and then the Dropout regularization layer with a parameter of 0.5 and the fully connected layer are used to reduce the dimension to a vector of size 128 × 1. Finally, the ReLU activation function is used to obtain the encoded representation of the video frame sequence xvf .

[0093] In step 3.2, the hidden layer dimension of Bi-LSTM is s0 = 64, the number of stacked layers is n = 2, and the output layer is processed with a Dropout regularization layer with a parameter of 0.5. The MFCC features are encoded by Bi-LSTM and the output is a two-dimensional matrix with a shape of [300, 128]. After the Flatten operation, the fully connected layer reduces the dimension to a vector of size 128 × 1. Finally, the ReLU activation function is used to obtain the encoded representation x of the MFCC features. mf .

[0094] Step 3.3: Input the original audio into the pre-trained wav2vec 2.0 model. The hidden layer dimension s1 = 768, and the output length l = 149 is calculated based on the length of the input sequence. The final hidden state parameters of the model are used as the output code x af , whose shape is [149,768].

[0095] Step 4, encoding fusion based on common attention. By encoding x vf with x mf After the concatenation operation, the attention weight matrix x generated by the linear embedding layer is att The shape is [1,149]. Encoding x af With the weight matrix x att Multiply to get the weighted code x af-att The shape is [1,768]. After the full connection layer and the Dropout regularization layer with a parameter of 0.1, a vector x′ with a length of 128 is obtained. af-att ; x vf 、x mf and x′ af-att After concatenation, the output feature representation x with a length of 384 is obtained.

[0096] Step 5: Classification layer. The number of emotion categories C = 4, that is, the feature representation x passes through the fully connected layer and the softmax layer to obtain a 4×1 classification result.

[0097] The above are only specific embodiments of the present invention, but the present invention is not limited to the above embodiments. Without departing from the design concept and principle of the present invention, various modifications, improvements and changes of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-source information emotion recognition method based on common attention, characterized in that: include: Step 1: Preprocess the video data, separate the audio stream and the video stream, and align the isochronous slices in the time dimension to obtain the video frame sequence x v and audio data x a ; Step 2: From audio data x a Extract MFCC acoustic features x m ; Step 3: Use the 3D convolutional network to model and encode the video data to obtain the video frame feature code x vf , use the Wav2Vec 2.0 model to model and encode the audio data to obtain the audio semantic feature encoding x af , use the bidirectional long short-term memory network to model and encode the MFCC acoustic features to obtain the MFCC acoustic feature encoding x mf , combining fully connected layers and using a common attention mechanism to fuse the three encodings; Step 4: Classify the data processed in step 3 through the fully connected layer and Softmax function to obtain the emotion recognition classification results.

2. The multi-source information emotion recognition method based on common attention according to claim 1 is characterized in that The step three specifically includes: Step 3.1: Use the pre-trained I3D model to model and encode the video frame sequence. The output of the average pooling layer is converted into a vector of size D×1 through the Flatten operation and the fully connected layer, which is used as the encoding representation x of the video frame sequence. vf , where D is the final feature unified dimension; Step 3.2, use the bidirectional long short-term memory network to transform the MFCC acoustic features x m As input for further feature encoding, the output is a two-dimensional matrix of shape [t, s0×n], where t is the length of the time dimension of the input MFCC feature, s0 and n are the hidden layer size and layer number parameters of the Bi-LSTM model; after the Flatten operation and the fully connected layers, the output dimension is reduced to a D×1 vector, which is used as the encoding representation x of the MFCC feature. mf ; Step 3.3: Use the pre-trained wav2vec 2.0 model to model and encode the original audio data, and use the final hidden state parameters of the model as the encoding representation x of the audio semantics. af , whose shape is [l,s1], s1 is the hidden layer dimension of the model, which is the internal parameter of the model. The relationship between l and the input is expressed as: where len(x a ) is the input sequence x a sr is the audio sampling rate.

3. The multi-source information emotion recognition method based on common attention according to claim 1 or 2, characterized in that: The step 4 specifically includes: encoding the video frame feature x through the common attention module vf , audio semantic feature encoding x af and MFCC acoustic feature encoding x mf The prediction results are compared with the actual prediction labels, and the error value is calculated through the CrossEntropyLoss loss function and recorded in real time to feed back to the model for supervised training.

4. The method for emotion recognition based on multi-source information based on common attention according to claim 3, characterized in that: The step 4 specifically includes: Step 4.1, video frame feature encoding x vf and MFCC acoustic feature encoding x mf After the splicing operation, it is used as the input sequence of the joint attention module and the weight matrix x is generated through the linear embedding layer. att , whose shape is [1,l]; the process can be expressed as Step 4.2, audio semantic feature encoding x af As another input sequence of the joint attention module, it is combined with the weight matrix x att Multiply to get the weighted code x af-att , whose shape is [1,s1]; the process can be expressed as x af-att =x att ·x af ; Step 4.3, use the fully connected layer to weight the encoding x af-att Perform dimension unification and process to obtain a vector x′ of length D af-att ; x vf 、x mf and x′ af-att After splicing, it is used as the final output feature representation x; the process can be expressed as: x′ af-att =σ(W·x af-att +b); The weight matrix Bias vector b∈R D , σ represents a nonlinear activation function.

5. The method for emotion recognition based on multi-source information based on common attention according to claim 3, characterized in that: The CrossEntropyLoss loss function is expressed as: Among them, y i Indicates the true label, the value is 0 or 1; is the probability predicted by the model. The built-in Softmax layer of the CrossEntropyLoss loss function converts the discrete type of emotion recognition classification task output value into a probability distribution ranging from [0, 1] and summing to 1. C represents the number of emotion categories.

6. The method for emotion recognition based on multi-source information based on common attention according to claim 1 or 2, characterized in that: The specific step 2 is: each audio data x a The two-dimensional MFCC acoustic feature x is represented as t×d m , where t is the time dimension size and d is the dimension size of feature representation.

7. The method for emotion recognition based on multi-source information based on common attention according to claim 1 or 2, characterized in that: Dropout regularization is used in the processing of steps 3 and 4 to prevent overfitting. The specific process is: during the training phase, the neuron output is randomly masked with probability p, so that the network does not rely too much on the connection between specific neurons, thereby reducing the model complexity and suppressing overfitting.

8. The method for emotion recognition based on multi-source information based on common attention according to claim 1 or 2, characterized in that: Specifically include: Step 1: Separate the audio stream and video stream and perform isochronous slicing processing; Step 2, extract MFCC acoustic features; Step 3: construct modeling and encoding modules for the three information sources of video frame sequence, MFCC acoustic features and audio data respectively; Step 4: Construct a fusion module based on the joint attention mechanism; the encoding module and the fusion module together constitute a multi-source information emotion recognition model based on joint attention; In step 5, the video frame sequence, MFCC acoustic features, and audio data of each piece of data are input into the emotion recognition model for supervised training to obtain the classification results.

9. The method for emotion recognition based on multi-source information based on common attention according to claim 8, characterized in that: The step three is specifically as follows: based on the video frame sequence x v , audio data x a and MFCC features x m The input is modeled and encoded respectively to obtain the feature representation of each deeper level; Use the pre-trained I3D model to model and encode the video frame sequence. The encoding representation of the video frame sequence is x vf ; Use the bidirectional long short-term memory network to take the MFCC features as input for further feature encoding, and obtain the encoding representation x of the MFCC features mf ; The pre-trained wav2vec 2.0 model is used to model and encode the raw audio data, and the final hidden state parameters of the model are used as the encoding representation x of the audio semantics. af .

10. The method for emotion recognition based on multi-source information based on common attention according to claim 8, characterized in that: The step 4 is specifically as follows: the codes obtained from different information sources are fused through the common attention module, including the video frame feature code x vf , audio semantic feature encoding x af and MFCC acoustic feature encoding x mf ; At the same time, pay attention to the relevant information between each coding sequence.

Images