A method for constructing a wide-temporal emotion recognition system based on multi-modal and multi-branch fusion

By introducing attention mechanism and multi-branch fusion network into the multi-modal emotion recognition system, the problems of cross-modal imbalance and data loss are solved, and efficient emotion recognition in complex environments is achieved, especially through dynamic weight allocation and The contextual description of a wide time range significantly improves the recognition effect.

CN118070208BActive Publication Date: 2024-11-05BEIJING INST OF TECH ZHUHAI CAMPUS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311573511.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2023-11-20
Filing Date
2023-11-23
Publication Date
2024-11-05
Estimated Expiration
2043-11-23

AI Technical Summary

Technical Problem

When multimodal emotion recognition systems face cross-modal imbalance problems and data loss in real environments, the recognition effect is reduced, and it is difficult for the existing technology to effectively integrate information from different modes.

Method used

A multimodal multi-branch fusion network based on attention mechanism was designed, and dynamically allocated through feature extraction models such as ResNet18+ECA+FSFM, ResNeXt+CA, VGGish and BERT, combined with global average pooling and one-dimensional convolution kernel. Attention weights perform decision-level multimodal fusion to deal with modal missing situations.

Benefits of technology

It has achieved the improvement of emotion recognition effect in complex environments, especially in the multimodal fusion process, the more branches are input, the better the emotion recognition effect. The wide time range of situation description significantly improves the experimental results, proving that multimodal The robustness and effectiveness of state fusion networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118070208B_ABST
    Figure CN118070208B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method for constructing a wide-time-range emotion recognition system with multimodal multi-branch fusion; a multimodal multi-branch fusion network with strong robustness is designed based on the idea of ​​attention mechanism, which realizes multimodal fusion at the decision-making level, and dynamically allocates the weight of each branch according to the contribution of the recognition results of different branches to the final prediction. A multimodal multi-branch fusion network based on attention mechanism is designed for the multimodal emotion recognition system. In addition, the context information of the wide time range is integrated into the multimodal emotion recognition system, and multimodal emotion recognition under the wide time range is realized. Experiments have proved that the network can effectively realize multimodal multi-branch fusion, and in the process of multimodal fusion, the more branches are input, the better the effect of emotion recognition, and the context description greatly improves the experimental results. Experiments have proved that the multimodal fusion network can improve the recognition effect of the model in complex environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of facial capture technology, and in particular to a method for constructing a multi-modal, multi-branch fusion emotion recognition system with a wide time range. Background Art

[0002] Multimodal sentiment analysis methods can learn sentiment features from different perspectives through the complementarity of multiple modal information, thus achieving better recognition results than single-modal methods. However, in real environments, different ways of expressing emotions may show different levels of sentiment intensity, and human emotional expression is sometimes subject to subjective inhibition, resulting in the low contribution of sentiment features of certain modalities to sentiment analysis. It is necessary to analyze their emotional state from the perspective of other modalities. Therefore, multimodal sentiment recognition faces the problem of cross-modal imbalance. In addition, when data is missing in a certain modality, the recognition effect of the model will also be affected. For the above reasons, it is necessary to study how to integrate information from different modalities in an effective way. Summary of the invention

[0003] In order to solve the above problems, the present application provides a method for constructing a multimodal, multi-branch fusion emotion recognition system with a wide time range. Based on the idea of ​​attention mechanism, a multimodal, multi-branch fusion network with strong robustness is designed. The network realizes multimodal fusion at the decision-making level and improves the recognition effect of the model in complex environments.

[0004] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0005] A method for constructing a multi-modal, multi-branch fusion emotion recognition system with a wide time range is provided, which is characterized by comprising the following steps:

[0006] 1) The modal information including facial expressions, video global context information, audio and text information are trained separately through the ResNet18+ECA+FSFM feature extraction model, ResNeXt+CA+multi-scale feature fusion feature extraction model, VGGish feature extraction model, and Bert feature extraction model to obtain their respective classification results;

[0007] 2) Use global average pooling to transform the feature representation of each branch into a one-dimensional real number, and obtain a vector S with a dimension of 1×C. Each real number has the global receptive field of its corresponding input feature. The calculation formula is as follows:

[0008]

[0009] Among them, sj represents the one-dimensional real number corresponding to the classification result on the jth channel, The i-th element representing the feature on the j-th channel is added and averaged to obtain the result;

[0010] 3) The size of the one-dimensional convolution kernel k is dynamically calculated based on the input value of C. The calculation formula is as follows:

[0011]

[0012]

[0013] Among them, k is the convolution kernel size, C is the number of input channels of the module, γ and b are parameters defined for nonlinear mapping, which are set to 2 and 1 respectively. In the formula, |t odd The meaning of is to take the odd number closest to the absolute value of t;

[0014] 4) Use a k×1 convolution kernel to perform a convolution operation on the vector obtained in the previous step to obtain a vector of the same size of 1×C. The convolution kernel operation is used to model the relationship between local channels. The sigmoid function is used to convert the value of the vector to between 0 and 1 to obtain the attention weight corresponding to each channel. The calculation process is as follows:

[0015] Z=σ(f 1D,k (S)

[0016] Where σ represents the sigmoid function, f 1D,k (·) represents a one-dimensional convolution operation with a kernel size of k×1;

[0017] 5) Multiply the obtained attention weights by the previous input channel by channel to get the classification result representation of the redistributed weights. The calculation formula is as follows:

[0018]

[0019] After converting it into a one-dimensional vector, it is sent into the classifier to obtain the final emotion recognition result.

[0020] Furthermore, the facial expression classification result is P f The classification results of and context information are P f and P g , audio mode is P v , the results of fact description and situation context description in text mode are P t and P c After obtaining the classification results of each modality, they are concatenated along the feature dimension to obtain H = D × C-dimensional features as the input of the multimodal fusion module, where C represents the number of input classification results and D represents the dimension of the input feature, which is 28.

[0021] Furthermore, the loss function is MultiLabelSoftMarginLoss, and its operation is basically the same as BCEWithLogitsLoss, except that there is no weight parameter for each category. It still integrates the Sigmoid activation function with the BCELoss operation to make the data more stable. Its calculation expression is as follows:

[0022]

[0023] Where C is the number of sentiment categories, x is the predicted score of the sample, n is the nth element of the predicted score, and y is the true label.

[0024] Furthermore, in the multimodal fusion process, all data and annotations are used, and if there is missing data, a zero vector is used instead.

[0025] Furthermore, the initial learning rate is set to 0.001, the batch size is 16, and a total of 50 epochs are trained.

[0026] In the technical solution provided in the embodiment of the present application, a multimodal multi-branch fusion network with strong robustness is designed based on the idea of ​​the attention mechanism. The network realizes multimodal fusion at the decision-making level and dynamically allocates the weight of each branch according to the contribution of the recognition results of different branches to the final prediction. A multimodal multi-branch fusion network based on the attention mechanism is designed for the multimodal emotion recognition system. In addition, the contextual information of a wide time range is integrated into the multimodal emotion recognition system to realize multimodal emotion recognition under a wide time range. Experiments have proved that the network can effectively realize multimodal multi-branch fusion, and in the process of multimodal fusion, the more branches are input, the better the effect of emotion recognition. The context description greatly improves the experimental results, proving that the multimodal fusion network can improve the recognition effect of the model in a complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] The methods, systems and / or programs in the accompanying drawings will be further described according to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, wherein example numbers represent similar mechanisms in the various views of the accompanying drawings.

[0029] Figure 1It is a schematic diagram of decision-level fusion provided in an embodiment of the present application;

[0030] Figure 2 It is a schematic diagram of the VGGish network structure provided in an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of the BERT structure provided in an embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of the ResNet18 structure provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the overall architecture of multi-modal multi-branch emotion recognition provided by the application embodiment;

[0034] Figure 6 Schematic diagram of a multimodal fusion module based on an attention mechanism provided in an embodiment of the application; DETAILED DESCRIPTION

[0035] In order to better understand the above technical scheme, the technical scheme of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0036] In the following detailed description, numerous specific details are set forth by way of example in order to provide a comprehensive understanding of the relevant guidance. However, it will be apparent to those skilled in the art that the present application may be practiced without these details. In other cases, well-known methods, procedures, systems, compositions and / or circuits have been described at a relatively high level, without detail, in order to avoid unnecessarily obscuring aspects of the present application.

[0037] Flowcharts are used in the present application to illustrate the execution process performed by the system according to the embodiment of the present application. It should be clearly understood that the execution process of the flowchart may not be performed in order. On the contrary, these execution processes may be performed in reverse order or simultaneously. In addition, at least one other execution process may be added to the flowchart. One or more execution processes may be deleted from the flowchart.

[0038] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0039] (1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0040] (2) Based on is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0041] Multimodal multi-branch fusion model based on attention mechanism

[0042] According to the different levels of multimodal fusion, it can be divided into data-level fusion, feature-level fusion and decision-level fusion. Among them, data-level fusion is to fuse the original data of different modalities to form a new fused data as the input of the model. Currently commonly used fusion methods include numerical processing and parameter estimation. This fusion method can retain the information contained in each modality to the maximum extent, but it also has some disadvantages. Due to the differences in data storage formats and spatial dimensions of different modalities, it is cumbersome and difficult to effectively integrate different forms of data into fused data. Although the complete data information is retained, it will also lead to information redundancy.

[0043] Feature fusion also occurs at an earlier stage and occurs more frequently in multimodal fusion tasks. After extracting features from the input of each modality, the features of each modality are simply concatenated or bitwise added and then fed into the classifier. This method usually results in a feature vector with a higher dimension. If the dimension is too high, the model performance will degrade. In addition, there is information redundancy, and features can be selectively learned by introducing an attention mechanism. Although the fusion of feature layers can improve the recognition effect of the model, the differences in features of different modalities are not taken into account.

[0044] Decision-level fusion occurs after the features of each modality are classified, so it is also called late fusion. Figure 1 This is a schematic diagram of decision-level fusion. Data from different modalities are input into different branches, and the model training is completed independently. Then the classification results are fused according to a certain strategy to obtain the final emotion category prediction. This method requires the design of appropriate feature extraction networks for different types of modalities. When data from a certain modality is missing, decision-level fusion can complete the prediction of the emotion category through the classification results of the remaining modalities. The main issue to be considered in the fusion process is the importance of different branches to the emotion recognition results and how to fuse them.

[0045] Common fusion methods include product rule, maximum rule, average rule, enumeration weight, etc. Compared with feature-level fusion, decision-level fusion method has stronger robustness and takes into account the differences between different modalities. However, decision-level fusion ignores the effect of the interaction between low-level features of different modalities and cannot learn the connection between features of different modalities.

[0046] The multimodal emotion recognition method of this application integrates many types of information, namely facial expressions, global context information of the video, audio and text description, where the text information includes two parts: the factual description in the video and the context description related to the target person's experience. The time span of each modal information is large, and the correlation between different modal data is not high, so the multimodal fusion task is chosen at the decision layer.

[0047] First, by using VGGish, ResNet18+ECA+feature selection fusion, and Bert feature extraction models, different types of facial expressions, video global context information, audio, and text description modal signals are trained separately to obtain their respective classification results. The classification extraction process of each feature extraction model is described as follows.

[0048] Audio emotion recognition based on VGGish

[0049] Audio emotion recognition uses VGGish as the feature extraction network. The VGGis network is a pre-trained model, and the dataset used is the large-scale dataset AudioSet of manually annotated audio events released by Google. This dataset consists of more than 2 million YouTube audio clips with a duration of 10 seconds. A total of 623 audio event classes are annotated, covering most audio events that occur in real-world recordings. It is a relatively comprehensive category set. The VGGish model has learned most of the various sound categories in real life by pre-training on this dataset. This model is used as the feature extraction network for audio signals in the SVCEmotion dataset. The specific structure of the network is as follows: Figure 2 As shown, the module consists of four convolutional and maximum pooling layers and three fully connected layers.

[0050] The input of VGGish is the Mel spectrogram of audio, so the audio signal needs to be converted first. Call the FFmpeg tool to extract the audio from the video, and then resample it to 16kHz mono audio. Next, use a Hann window with a window length of 25ms to intercept the audio segment, and implement short-time Fourier transform on the audio segment with a frame shift of 10ms to capture the frequency domain characteristics of the speech signal. While converting it from the time domain signal to the frequency domain signal, it can ensure that time-related information is not lost. The final result is a spectrogram. By using the Mel scale filter bank, the spectrum is converted into a Mel spectrogram. Each frame has a duration of 10ms and contains 64 Mel frequency bands. The Mel spectrogram of each frame is combined without overlap, and each 0.96s duration is a group. Therefore, the size of each group of input Mel spectrograms is. The converted Mel spectrograms are used as the input of VGGish, and each group of inputs is extracted to obtain a 128-dimensional feature representation. Since the duration of each sample is greater than 0.96s, each sample can obtain multiple feature representations after feature extraction. During the training process, a set of feature representations of the dimensions of the audio sample is randomly selected each time and sent to two fully connected layers to obtain a 28-dimensional classification representation. During the verification process, the mean of all feature representations of each sample is taken as the feature representation of the video level and sent to the classifier to obtain the recognition result.

[0051] Text Sentiment Recognition Based on BERT

[0052] The BERT model uses two large corpora, BooksCorpus and English Wikipedia, to complete two unsupervised pre-training tasks to obtain strong text semantic representation capabilities. The first task is to train the language model using a masked method, randomly masking the tokens of the input sentence at a certain ratio, predicting the masked tokens, and obtaining a deep bidirectional language feature representation. The second task is to determine whether the two input texts are continuous, and train the bidirectional language model to learn the association between longer text sequences. The advantage of this model is that when processing different downstream tasks, the model architecture is unified, and only the output layer of the network needs to be added and fine-tuned for different tasks.

[0053] The BERT model has two model sizes. This article uses the BERTbase model. The network has 12 layers of encoders, the hidden layer size is 768, and the number of multi-head self-attention modules in each layer is 12. The overall scale is larger than the Transformer model. The specific structure is as follows: Figure 3As shown in Figure 1. The input of the model consists of three parts. First, the input sentence is tokenized and each word is converted into a corresponding 768-dimensional word embedding ID, that is, a token for each word. Then, two special tags [CLS] and [SEP] are inserted at the beginning and end of each input sequence, respectively. [CLS] is a tag for classification tasks, and the output state corresponding to its position integrates the semantic information of the entire sentence. The function of the [SEP] tag is to distinguish the input sentence pairs. Since the self-attention mechanism of the Transformer ignores the position of the word in the sentence, position encoding (position embedding) is performed for each word. The dimension of the encoding is the same as the word vector. Finally, segment embedding is added to each token to distinguish the two sentences in the sentence pair. If only one sentence is input, the segment embedding values ​​of all tokens are the same. The three parts of the embedding of each token are added by position to obtain the input vector of BERT. In order to unify the length of the input sequence, the remaining token positions of shorter sentences will be filled with zeros. This paper sets the maximum length of the sequence to 128.

[0054] In the text annotation content, the dataset constructed in this paper not only describes the events happening in the video, but also describes the events that the target subject experienced before the video content, that is, it introduces situational context information. Therefore, sentiment recognition in text modality contains two types of data, namely, factual description and situational description. By combining the objective content description of the video with the situational context information, we can obtain more abundant sentiment analysis clues with a wider time range, which is more in line with the human thinking logic. First, we learn the sentiment feature representation from these two text descriptions respectively, and then directly splice the contents of the two texts so that the model can extract features from the spliced ​​text, and study the effectiveness of situational description for sentiment analysis.

[0055] Since computers cannot directly recognize text, in order to implement text sentiment analysis, it is necessary to convert text into word vectors that machines can recognize. The data set in this article is annotated in Chinese, so it is necessary to use translation software to translate all the annotations into English before it can be used. The introduction of Transformer has made the self-attention mechanism widely used in the field of natural language processing, changing the previous application of attention mechanism in time series models. The BERT pre-trained model is a bidirectional encoder representation model based on the Transformer Encoder structure. It can combine the left and right contexts in each layer and obtain the correlation between words within the sentence, which can more accurately represent the semantics of the sentence. This section uses the BERT pre-trained model to extract features and classify sentiment of text data.

[0056] The text information includes two parts: the factual description and the situational description of each video. In order to verify the auxiliary role of situational information in a wide time range for emotion recognition, the two text descriptions of each sample are spliced ​​as the third text description. All three text information use BERTbase as the feature extraction network for text emotion classification. In the process of collecting data sets, facial expressions are not the only criterion for judging whether a video clip is selected as a data set. In some scenes in film and television dramas, faces cannot be extracted through automatic detection algorithms, and there is no dialogue in some shots. Therefore, some data may be missing in some modes. Only valid data is retained for learning during the training process. In the multimodal fusion stage, in order to obtain a robust decision fusion module, data with missing modes will not be eliminated, and the missing modal data will be replaced with a zero matrix.

[0057] Multimodal multi-branch fusion model based on attention mechanism

[0058] According to the different levels of multimodal fusion, it can be divided into data-level fusion, feature-level fusion and decision-level fusion. Data-level fusion is to fuse the original data of different modalities to form a new fusion data as the input of the model. Feature fusion also occurs at an earlier stage and appears more frequently in the task of multimodal fusion. After extracting features from the input of each modality, the features of each modality are simply concatenated or bitwise added and then sent to the classifier. Decision-level fusion occurs after the features of each modality are classified, so it is also called late fusion. Figure 5.3 is a schematic diagram of decision-level fusion. Data of different modalities are input into different branches respectively, and the model training is completed independently. Then the classification results are fused according to a certain strategy to obtain the final emotion category prediction. This method requires the design of appropriate feature extraction networks for different types of modalities. When data of a certain modality is missing, decision-level fusion can complete the prediction of emotion categories through the classification results of the remaining modalities. The main issue to be considered in the fusion process is the importance of different branches to the emotion recognition results and how to fuse them. Commonly used fusion methods for decision-level fusion include product rule, maximum rule, average rule, enumeration weight, etc. Compared with feature-level fusion, decision-level fusion method has stronger robustness and takes into account the differences between different modalities.

[0059] The multimodal emotion recognition method integrates many types of information, namely facial expressions, global context information of the video, audio and text description. The text information includes two parts: the factual description in the video and the context description related to the target person's experience. The time span of each modal information is large, and the correlation between different modal data is not high. Therefore, this paper chooses to perform multimodal fusion tasks at the decision layer. First, by using the feature extraction model designed in the previous article, different modalities are trained separately to obtain their respective classification results. Since facial expressions are not the only criterion for judging whether a video clip is selected as a dataset in the process of collecting data sets, faces cannot be extracted by automatic detection algorithms in some scenes in film and television dramas, and there is no dialogue in some shots. Therefore, some data will be missing in some modalities. Only valid data is retained for learning during the training process. In the multimodal fusion stage, in order to obtain a robust decision fusion module, the data with modal missing will not be eliminated, and the missing modal data will be replaced by a zero matrix.

[0060] Although the currently commonly used decision-level fusion methods take into account the contribution of each modality during fusion, they cannot be targeted at dynamic changes in different data. Once the parameters of the fusion rules or the weights of each branch are determined, they cannot be changed. Although the categories of emotions possessed by humans are basically the same, different people express their emotions in different ways, resulting in changes in the contribution of each modality in different samples. In order to address the problem of different importance of each modality in different samples, this application introduces an attention mechanism to adaptively assign weights to the classification results of each modality, and obtains the overall architecture of multimodal emotion recognition as follows: Figure 5 shown.

[0061] The visual part consists of two branches, where the classification results of facial expressions and situational information are P and f and P g , audio mode is P v , the results of fact description and situation context description in text mode are P t After obtaining the classification results of each modality, they are concatenated along the feature dimension to obtain H = D × C-dimensional features as the input of the multimodal fusion module, where C represents the number of input classification results and D represents the dimension of the input feature, which is 28, because the dataset has 28 categories of emotional states. Inspired by the ECANet module structure, the channel attention mechanism is used in this module. The specific structure is as follows: Figure 6 shown.

[0062] First, use global average pooling to transform the feature representation of each branch into a one-dimensional real number, and obtain a vector S with a dimension of 1×C. Each real number has the global receptive field of its corresponding input feature. The calculation formula is as follows:

[0063]

[0064] Among them, s j Represents the one-dimensional real number corresponding to the classification result on the j-th channel, The i-th element representing the feature on the j-th channel is added and averaged to get the result.

[0065] Then calculate the size of the one-dimensional convolution kernel k, the calculation formula is as follows:

[0066]

[0067]

[0068] According to the input value of C, the size of the one-dimensional convolution kernel k is dynamically calculated. The vector obtained in the previous step is convolved with the k×1 convolution kernel to obtain a vector of the same size of 1×C. The relationship between local channels is modeled through the operation of the convolution kernel. The sigmoid function is used to convert the value of the vector to between 0 and 1, and the attention weight corresponding to each channel is obtained. The output calculation process of this part is as follows:

[0069] Z=σ(f 1D,k (S)) (5-2)

[0070] Where σ represents the sigmoid function, f 1D,k (·) represents a one-dimensional convolution operation with a convolution kernel size of k×1. The obtained attention weight is multiplied by the previous input channel by channel to obtain the classification result representation of the redistributed weight. The calculation formula is as follows:

[0071]

[0072] After converting it into a one-dimensional vector, it is sent into the classifier to obtain the final emotion recognition result.

[0073] The loss function used in this application is MultiLabelSoftMarginLoss, which operates in the same way as BCEWithLogitsLoss, except that there is no weight parameter for each category. It still integrates the Sigmoid activation function with the BCELoss operation to make the data more stable. The calculation expression is as follows:

[0074]

[0075] Where C is the number of sentiment categories, x is the predicted score of the sample, n is the nth element of the predicted score, and y is the true label.

[0076] This section first introduces the experimental parameter settings of multimodal multi-branch fusion in this chapter, and then analyzes the effectiveness of the multimodal fusion network based on the attention mechanism through multimodal fusion experiments, as well as the role of each modality in the multimodal emotion recognition network in emotion recognition. Finally, it verifies the auxiliary role of situational description in a wide time range in emotion recognition.

[0077] Experimental parameters

[0078] Before conducting the experiment of this application, each modality was trained independently, and the audio modality was trained using global emotion labels, that is, labels obtained by synthesizing the emotional state of the target person in the video. In the multimodal fusion process, in order to enable the model to learn to dynamically assign weights to each branch, all data and annotations are used. If data is missing, it is replaced by a zero vector. The initial learning rate is set to 0.001, the batch size is 16, and a total of 50 epochs are trained.

[0079] Multi-modal multi-branch fusion experimental results and analysis

[0080] This paper studies multimodal emotion recognition based on a wide time range for data in three modalities: video, audio, and text. It includes five branches, namely facial expressions, global scenes in the video, audio, objective descriptions, and descriptions of situational context information. These five branches are used for ablation experiments, and the five branches are gradually added to the multimodal fusion network to compare the recognition effects under different circumstances. B2 represents the fusion of facial expression and situational emotion recognition branches, B3 represents the fusion of facial expression, scene, and audio branches, B4 represents the factual description of the two branches of video, audio, and text modalities, and B5 represents the fusion of all branches. Table 1 shows the experimental results of the multimodal emotion recognition network when different branches are fused.

[0081] Table 1 Experimental results of multi-modal multi-branch fusion network

[0082]

[0083] By continuously increasing the number of fused branches, the performance of the multimodal multi-branch fusion network gradually improves. After the facial expression and scene branches are fused, the M value reaches 42.62, which is 2.03 higher than the effect. This shows that some object or character scene information in the image has a certain auxiliary effect on emotion recognition. Since there may be multiple characters communicating in the image, the target character and the surrounding characters may share emotions. Therefore, the expression of the non-target character has a certain impact on the final classification result. After the visual modality is fused with the audio modality and factual description, the recognition effect is steadily improved. It can be proved that in the dataset constructed in this paper, audio data can improve the recognition effect of multimodal emotion recognition. It can be seen from the results of B4 that the factual description of the video also helps the model to infer the emotional state of the target character to a certain extent. When all five branches are fused, the M value reaches 49.96, which is 9.37 higher than the effect of facial expression recognition. Compared with other modalities, the situational context information in the form of text description greatly improves the performance of the model, proving that the situational context information in a wide time range has important research significance for emotion recognition. In addition, this experiment proves that in a more complex environment, the use of multimodal emotion recognition methods can effectively improve the recognition effect of the model, overcoming the problems of incomplete single-modal information and low recognition rate. The experimental results also verify that the multimodal fusion network based on the attention mechanism designed in this chapter can effectively integrate feature representations from multiple modalities.

[0084] In addition, the multimodal multi-branch fusion method is compared with the other two decision-layer fusion methods. Table 2 shows the experimental results of the fusion of all branches and the comparison with other decision-layer fusion methods.

[0085] Table 2 Experimental results of multi-modal multi-branch fusion network

[0086]

[0087] B5 (Con) in the first row of the table represents cascade fusion, which directly cascades and then performs sentiment classification. B5 (Wei) in the second row represents a weight-based fusion method, which performs weighted fusion by assigning appropriate weights to the recognition results of each branch. The best weight distribution results were selected for comparison, where the weights of face, scene, audio, fact description, and situation description are 0.2, 0.15, 0.2, 0.15, and 0.3, respectively. The weighted fusion method has a fixed weight distribution for each branch. B5 (Att) represents the multimodal fusion method based on the attention mechanism of this application. It can be seen from the table that the effect of the multimodal multi-branch fusion method is better than the other two decision fusion methods. Due to the introduction of the attention mechanism, the model can dynamically adjust the weights according to the contribution of each branch in the current sample when making decisions on each sample to achieve the best fusion ratio. Therefore, on the data set constructed in this article, the use of the multimodal multi-branch fusion method based on the attention mechanism can enable the multimodal emotion recognition system to obtain more accurate classification results.

[0088] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0089] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0090] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0091] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0093] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0095] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0096] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks.

[0097] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for constructing a multi-modal, multi-branch fusion emotion recognition system with a wide time range, characterized by The steps include: 1) The modal information including facial expressions, video global context information, audio and text information are trained separately through the ResNet18+ECA+FSFM feature extraction model, ResNeXt+CA+multi-scale feature fusion feature extraction model, VGGish feature extraction model, and Bert feature extraction model to obtain their respective classification results; 2) Use global average pooling to transform the feature representation of each branch into a one-dimensional real number, and obtain a vector S with a dimension of 1×C. Each real number has the global receptive field of its corresponding input feature. The calculation formula is as follows: Among them, s j Represents the one-dimensional real number corresponding to the classification result on the j-th channel, The i-th element representing the feature on the j-th channel is added and averaged to obtain the result; 3) The size of the one-dimensional convolution kernel k is dynamically calculated based on the input value of C. The calculation formula is as follows: C=φ(k)=2 (γ*k-b) Among them, k is the convolution kernel size, C is the number of input channels of the module, γ and b are parameters defined for nonlinear mapping, which are set to 2 and 1 respectively. In the formula, |t| odd The meaning of is to take the odd number closest to the absolute value of t; 4) Use a k×1 convolution kernel to perform a convolution operation on the vector obtained in the previous step to obtain a vector of the same size of 1×C. The convolution kernel operation is used to model the relationship between local channels. The sigmoid function is used to convert the value of the vector to between 0 and 1 to obtain the attention weight corresponding to each channel. The calculation process is as follows: Z=σ(f 1D,k (S)) Where σ represents the sigmoid function, f 1D,k (·) represents a one-dimensional convolution operation with a kernel size of k×1; 5) Multiply the obtained attention weights by the previous input channel by channel to get the classification result representation of the redistributed weights. The calculation formula is as follows: After converting it into a one-dimensional vector, it is sent into the classifier to obtain the final emotion recognition result.

2. The method for constructing a multi-modal, multi-branch fusion emotion recognition system with a wide time range according to claim 1, characterized in that: The audio input VGGish feature extraction model training includes calling the FFmpeg tool to extract the audio from the video and resampling it to 16kHz mono audio; using a Hann window with a window length of 25ms to intercept the audio segment, and implementing short-time Fourier transform on the audio segment with a frame shift of 10ms to obtain a spectrogram. Using a Mel-scale filter bank, log(mel-spectrum+0.01) is calculated to convert the spectrum into a Mel spectrogram. Each frame has a duration of 10ms and contains 64 Mel frequency bands. The Mel spectrograms of each frame are combined without overlapping, and each group is a group of 0.96s in duration, so that each group of input size is 96×64 Mel spectrograms; the converted Mel spectrograms are used as the input of the VGGish pre-training model. Each group of input is subjected to feature extraction to obtain a 128-dimensional feature representation. During the training process, a group of feature representations of the dimension of the audio sample is randomly selected each time and sent to two fully connected layers to obtain a 28-dimensional classification representation H. v ; During the verification process, the mean of all feature representations of each sample is taken as the feature representation at the video level and sent to the classifier to obtain the recognition result.

3. The method for constructing a multi-modal, multi-branch fusion emotion recognition system with a wide time range according to claim 1, characterized in that: The Bert feature extraction model uses the BERTbase model as the feature extraction network for text sentiment classification. The text information includes two parts: fact description and context description of each video. In order to verify the auxiliary role of context information in a wide time range in emotion recognition, the two text descriptions of each sample are spliced ​​as the third text description.

4. The method for constructing a multi-modal, multi-branch fusion emotion recognition system with a wide time range according to claim 1, characterized in that: The facial expression classification results and the situation information classification results are P f and P g , audio mode is P v , the results of fact description and situation context description in text mode are P t and P c After obtaining the classification results of each modality, they are concatenated along the feature dimension to obtain H=D×C-dimensional features as the input of the multimodal fusion module, where C represents the number of input classification results and D represents the dimension of the input feature, which is 28.

5. The method for constructing a multi-modal, multi-branch fusion emotion recognition system with a wide time range according to claim 1, characterized in that: The loss function is MultiLabelSoftMarginLoss, which integrates the Sigmoid activation function with the BCELoss operation. The calculation expression is as follows: Where C is the number of sentiment categories, x is the predicted score of the sample, n is the nth element of the predicted score, and y is the true label.

6. The method for constructing a multi-modal, multi-branch fusion emotion recognition system with a wide time range according to claim 1, characterized in that: In the multimodal fusion process, all data and annotations are used. If there is missing data, a zero vector is used instead.

7. The method for constructing a multi-modal, multi-branch fusion emotion recognition system with a wide time range according to claim 1, characterized in that: The initial learning rate was set to 0.001, the batch size was set to 16, and a total of 50 epochs were trained.