Emotion recognition method and system based on multimodal feature and hierarchical feature fusion

An emotion recognition method that integrates multimodal and hierarchical features utilizes visual, audio, and textual features and designs a hierarchical feature fusion strategy. This addresses the problem of insufficient multimodal fusion in existing technologies, improves the accuracy and robustness of emotion recognition, and is applicable to scenarios such as autonomous driving, school security, and hospital monitoring.

CN118709094BActive Publication Date: 2024-11-15UNIV OF SCI & TECH BEIJING
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411193418.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-15
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

Existing emotion recognition methods do not fully utilize multimodal features, especially the unreasonable weight allocation between different modalities, which leads to insufficient emotion recognition performance. Furthermore, the simple concatenation of multimodal fusion methods fails to effectively mine complementary information.

Method used

An emotion recognition method that integrates multimodal and hierarchical features is adopted. Visual, audio and text features are extracted through a multimodal pre-trained model, and a hierarchical feature fusion strategy is designed to adaptively assign weights to different features, highlight important features and suppress weak features. The ST-GCN network is used to extract spatiotemporal human pose features, the CLIP model is used to extract scene emotion word association features, and the Transformer model is combined for feature fusion.

Benefits of technology

It improves the robustness and generalization of emotion recognition systems, enabling more accurate identification of the emotion categories of people in videos, and is applicable to fields such as autonomous driving, school security, and hospital monitoring.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118709094B_ABST
    Figure CN118709094B_ABST
Patent Text Reader

Abstract

The present invention provides an emotion recognition method and system based on multimodal features and hierarchical features fusion, and relates to the field of artificial intelligence technology. The method comprises: collecting an emotion video and converting it into an image sequence containing multiple images; detecting a face in the image sequence, extracting the audio and text in the emotion video, and obtaining visual modal features, audio modal features, and text modal features; wherein the visual modal features include: visual face attribute features, visual facial action features, spatiotemporal human posture features, and scene emotion word association features, the audio modal features include multi-source audio features, and the text modal features include text features based on CLIP; obtaining multimodal fusion features based on a hierarchical feature fusion strategy; inputting the multimodal fusion features into a multi-layer perceptron prediction model to obtain an emotion recognition category. The present invention can improve the robustness and generalization of the emotion recognition system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotion recognition method and system based on the fusion of multimodal features and hierarchical features. Background Art

[0002] In recent years, multimodal emotion recognition has become an active research area with important applications in various fields, such as human-computer interaction, intelligent healthcare, and robotics. Emotion is a complex psychophysiological process associated with many external and internal activities. Human emotional expressions include not only physiological responses such as skin temperature and heart rate, but also natural language, facial expressions, motion gestures, and acoustic behaviors. Different modalities can provide rich information to understand human emotions and intentions. Benefiting from the application of excellent deep learning technology in processing various signals, researchers have developed various models to extract information from emotion-related multimodal sequences, such as convolutional neural networks, recurrent neural networks, transformers, and their variants.

[0003] At present, although there are many multimodal methods involving video, audio, and text for emotion recognition, most of the existing work focuses on learning the appearance knowledge of the visual modality based on the CNN architecture, and features such as dynamic geometric knowledge based on graph structure and facial attributes are not fully utilized. At the same time, existing emotion recognition methods ignore the role of multimodal correlation features and multi-source audio features in emotion recognition tasks. In addition, existing methods mostly simply combine multimodal fusion in series, and do not fully explore the complementary information of different modalities, especially how to achieve more effective fusion of multimodal features in a hierarchical fusion manner.

[0004] Research has found that an ideal system for automatically analyzing and identifying human emotional information should be multimodal, just like the human sensory system. However, most research work in affective computing does not combine different modalities into a system to analyze human emotional behavior, and different modal information is considered independent of each other. Existing emotion recognition systems have made few attempts to integrate spatiotemporal human posture features, visual face attribute features, and scene emotion word association features. In addition, in the real world, due to sensor noise, occlusion, etc., not every modality is equally reliable for a specific task. It is necessary to study more complex data-driven methods and feature fusion strategies based on attention mechanisms and hierarchical techniques, so that more important modal features are given greater weights, so as to more reliably combine multi-source features for emotion recognition. Summary of the invention

[0005] In order to solve the problem that the existing technology does not fully mine emotion features based on multimodality and that the weight distribution of influences between different modalities is unreasonable, the present invention provides an emotion recognition method and system based on multimodal features and hierarchical feature fusion, which makes full use of the multimodal pre-training model to extract multiple modal features, integrates spatiotemporal human posture features, visual facial action features, multi-source audio features and visual face attribute features, and extracts scene emotion word association features and text features through the multimodal pre-training model CLIP to capture multimodal features for emotion recognition. In response to the feature fusion problem, the present invention designs a hierarchical feature fusion strategy to fuse multimodal features, which can adaptively fuse and assign weights to different features hierarchically, highlight stronger features and suppress weaker features according to their importance, thereby improving the performance of the emotion recognition system.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] On the one hand, a method for emotion recognition based on fusion of multimodal features and hierarchical features is provided, the method comprising the following steps:

[0008] S1, collect emotional videos and convert them into image sequences containing multiple images;

[0009] S2, detecting faces in the image sequence, and extracting audio and text in the emotional video to obtain visual modality features, audio modality features, and text modality features;

[0010] The visual modality features include: visual face attribute features, visual facial action features, spatiotemporal human posture features, and scene emotion word association features; the audio modality features include multi-source audio features; and the text modality features include CLIP-based text features.

[0011] S3, fusing the visual modality feature, the audio modality feature and the text modality feature based on a hierarchical feature fusion strategy to obtain a multimodal fusion feature;

[0012] S4. Inputting the multimodal fusion features into a multi-layer perceptron prediction model for emotion recognition to obtain an emotion recognition category corresponding to the emotion video.

[0013] Optionally, in step S2, obtaining visual face attribute features specifically includes:

[0014] Input the panoramic image sequence into the FaceXFormer model to obtain the facial attribute features of each image;

[0015] The visual face attribute features are obtained by processing the time series information through a long short-term memory network (LSTM).

[0016] Optionally, in step S2, obtaining visual facial action features specifically includes:

[0017] Using a face detector to segment the portion of the image sequence containing the face, if no face is detected, using a panoramic image instead;

[0018] Based on the strategy of transfer learning, the image sequence is input into the pre-trained 3D-ResNet34 model to obtain the visual facial action features.

[0019] Optionally, in step S2, obtaining spatiotemporal human body posture features specifically includes:

[0020] The panoramic image sequence is input into MediaPipe to extract human key points, including 468 facial key points, 33 posture key points, and 21 hand key points for each of the left and right hands. For missing key points, 0 is used to fill them.

[0021] The extracted facial key points, posture key points, and hand key points are respectively input into the improved ST-GCN network, and then the obtained feature vectors are connected to form the spatiotemporal human posture features.

[0022] Optionally, in step S2, obtaining scene emotion word association features specifically includes:

[0023] Collect and organize words that describe emotional states to obtain a list of emotion description words;

[0024] The image sequence and the emotion description words in the emotion description word list are input into the CLIP pre-trained model, and the first image feature and the first text feature are obtained respectively through the image encoder and the text encoder. The vectors of the first image feature and the first text feature are normalized and matrix multiplied, and the similarity between the two is calculated as the scene emotion word association feature.

[0025] Optionally, in step S2, obtaining text features based on CLIP specifically includes:

[0026] Transcribing the emotional video to extract text from the emotional video;

[0027] Preprocess the extracted text;

[0028] Input the preprocessed text into the text encoder of the CLIP pre-trained model to obtain the second text feature;

[0029] The vectors of the second text feature and the first text feature are normalized and then matrix-multiplied to obtain the similarity between the two, which is used as the CLIP-based text feature.

[0030] Optionally, in step S2, obtaining multi-source audio features specifically includes:

[0031] Extracting audio from the emotional video and calculating a log-mel spectrogram;

[0032] The log-mel spectrogram is split into frames and input into the pre-trained VGGish CNN model to obtain the VGGish CNN audio feature vector;

[0033] Averaging multiple VGGish CNN audio feature vectors of each audio to obtain a first audio feature;

[0034] Extracting audio features related to emotion recognition in the emotion video using the Wav2Vec2-Large-Robus model fine-tuned on MSP-Podcast to obtain a second audio feature;

[0035] The first audio feature and the second audio feature are fused to obtain the multi-source audio feature.

[0036] Optionally, in step S3, the hierarchical feature fusion strategy specifically includes:

[0037] In the first layer, on the one hand, the multi-source audio features and the CLIP-based text features are fused to obtain non-visual features; on the other hand, the visual face attribute features, the visual facial action features and the spatiotemporal human posture features are fused to obtain visual features;

[0038] In the second layer, the non-visual features and the visual features are fused;

[0039] In the third layer, the features obtained in the second layer are fused with the scene emotion word associated features to finally obtain multimodal fusion features.

[0040] Optionally, in step S4, a multi-layer perceptron is used as a prediction model;

[0041] The first layer of the multilayer perceptron is the first fully connected layer, which linearly transforms the input multimodal fusion features into 32 dimensions; the second layer is a one-dimensional batch normalization layer, and the number of input and output channels is 32; the third layer is a ReLU activation function; the fourth layer is a random dropout layer, which discards 20% of neurons through regularization; the fifth layer is the second fully connected layer, which maps the feature dimension into 7 dimensions through linear transformation, corresponding to 7 classification results respectively; the sixth layer is a Sigmoid activation function, which converts the linear output to the [0,1] interval; the seventh layer is a label layer, and the label value is in the range of [0,1].

[0042] On the other hand, an emotion recognition system based on the fusion of multimodal features and hierarchical features is provided, for implementing any of the above methods, the system comprising:

[0043] A data acquisition module, used for acquiring emotional videos and converting them into image sequences containing multiple images;

[0044] A feature extraction module, used to detect faces in the image sequence, and extract audio and text in the emotional video to obtain visual modality features, audio modality features, and text modality features;

[0045] The visual modality features include: visual face attribute features, visual facial action features, spatiotemporal human posture features, and scene emotion word association features; the audio modality features include multi-source audio features; and the text modality features include CLIP-based text features.

[0046] A hierarchical feature fusion module, used for fusing the visual modality feature, the audio modality feature and the text modality feature based on a hierarchical feature fusion strategy to obtain a multimodal fusion feature;

[0047] The emotion recognition module is used to input the multimodal fusion features into a multi-layer perceptron prediction model for emotion recognition to obtain an emotion recognition category corresponding to the emotion video.

[0048] In another aspect, an electronic device is provided, the electronic device comprising:

[0049] processor;

[0050] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the above-mentioned emotion recognition method based on the fusion of multimodal features and hierarchical features are implemented.

[0051] On the other hand, a computer-readable storage medium is provided, in which a program code is stored. The program code can be called by a processor to execute the steps of the above-mentioned emotion recognition method based on the fusion of multimodal features and hierarchical features.

[0052] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0053] The present invention provides an emotion recognition method and system based on multimodal features and hierarchical features fusion, which utilizes three different main modal information, including visual modal features, audio modal features and text modal features, and can greatly improve the recognition performance of the automatic emotion recognition system. On the one hand, the present invention extracts spatiotemporal human posture features, visual face attribute features, visual facial action features, scene emotion word association features, CLIP-based text features, and multi-source audio features. Among them, the human body key points consider facial key points, posture key points and hand key points, and extract features through the ST-GCN spatiotemporal graph convolution network, which can more fully extract action features. The text feature uses the text encoder of CLIP pre-trained by image-text contrast learning to encode the transcribed text and personality descriptor respectively, and the similarity between the two is used as a new text feature. The audio model trained on various audios containing background noise is used to preliminarily extract audio features, and noise reduction processing is indirectly performed, and then added to the Transformer modality for fusion to obtain multi-source audio features. On the other hand, the present invention designs a hierarchical feature fusion strategy for the first time to fuse different multimodal features together. This hierarchical feature fusion strategy can process signals in a complementary way with different modal features, thus improving the robustness and generalization of the automatic emotion recognition system. The system can intelligently, quantitatively and conveniently identify the emotion categories of people in videos, and can be widely used in the analysis of people's emotional states in the fields of autonomous driving, school security, hospital monitoring, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 is a flow chart of an emotion recognition method based on multimodal features and hierarchical features fusion provided by an embodiment of the present invention;

[0056] Figure 2 It is a schematic diagram of the overall framework of the emotion recognition method based on the fusion of multimodal features and hierarchical features provided by an embodiment of the present invention;

[0057] Figure 3 is a schematic diagram of spatiotemporal human posture feature extraction provided by an embodiment of the present invention;

[0058] Figure 4 It is a schematic diagram of scene emotion word association features and text feature extraction based on CLIP provided by an embodiment of the present invention;

[0059] Figure 5is a schematic diagram of audio feature fusion provided by an embodiment of the present invention;

[0060] Figure 6 is a schematic diagram of hierarchical feature fusion provided by an embodiment of the present invention;

[0061] Figure 7 is a schematic diagram of an MLP model provided by an embodiment of the present invention;

[0062] Figure 8 is a schematic diagram of the structure of an emotion recognition system based on multimodal features and hierarchical features fusion provided by an embodiment of the present invention;

[0063] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the word "exemplary" is used to present concepts in a specific way.

[0066] All actions of acquiring signals, information or data in the present invention are performed in compliance with the corresponding data protection laws and policies of the country where the device is located and with the authorization of the corresponding device owner.

[0067] An embodiment of the present invention provides an emotion recognition method based on the fusion of multimodal features and hierarchical features. The method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 is a flow chart of an emotion recognition method based on multimodal features and hierarchical features fusion provided by an embodiment of the present invention, Figure 2 This is a schematic diagram of the overall framework of the method.

[0068] Specifically, the processing flow of the method may include the following steps:

[0069] S1. Collect emotional videos and convert them into image sequences containing multiple images.

[0070] In the embodiment of the present invention, an emotion video to be identified is obtained, an emotion video data set is constructed, and the video data is converted into an image sequence containing 15 images using the cross-platform computer vision library (Open Source Computer Vision Library, OpenCV).

[0071] S2. Detecting human faces in the image sequence, and extracting audio and text in the emotional video to obtain visual modality features, audio modality features, and text modality features.

[0072] Among them, the visual modality features include: visual face attribute features, visual facial action features, spatiotemporal human posture features and scene emotion word association features, the audio modality features include multi-source audio features, and the text modality features include CLIP-based text features.

[0073] As an optional implementation of the present invention, obtaining visual face attribute features specifically includes:

[0074] The panoramic image sequence is input into the FaceXFormer model to obtain the facial attribute features of each image; the time series information is processed through the long short-term memory network LSTM to obtain the visual facial attribute features.

[0075] FaceXFormer is an end-to-end unified transformer model for comprehensive facial analysis tasks. It can perform facial parsing, landmark detection, head pose estimation, attribute recognition, age / gender / race estimation, and landmark visibility prediction on the input image. It is pre-trained on the CelebA dataset containing more than 200,000 celebrity face images and can recognize 40 facial attributes, such as hair color, glasses, smile, beard, eyebrows, etc. The present invention uses the FaceXFormer model to try to use landmark visibility prediction, head pose estimation, attribute recognition, etc. for emotion recognition. Experiments show that these facial attribute features help improve the recognition effect. The panoramic image sequence is input into the FaceXFormer model to obtain the facial attribute features of each image. Each video consists of an image sequence containing 15 images. The present invention uses a long short-term memory network (LSTM) to process the time series information, selectively retains or forgets the time series information, and finally obtains the visual facial attribute features of each video.

[0076] As an optional implementation of the present invention, obtaining visual facial action features specifically includes:

[0077] A face detector is used to segment the part of the image sequence containing the face. If no face is detected, a panoramic image is used instead. Based on the transfer learning strategy, the image sequence is input into the pre-trained 3D-ResNet34 model to obtain visual facial action features.

[0078] The 3D-ResNet34 model is a 3D-ConvNet model with ResNet34 as the core. It contains 34 layers of deep network and residual connections are designed to overcome the gradient vanishing problem. The 3D-ResNet34 model uses three-dimensional convolution to capture richer spatiotemporal information and is more suitable for image sequences converted from videos. The image sequence containing 15 images is passed through a face detector to segment the part containing the face. If a video does not detect a face, a panoramic image is used instead. Since the model input requires 16 images, repeat the last image in the image sequence and input it into the pre-trained 3D-ResNet34 model.

[0079] Loading on Kinetics-700 、Moments in Time The 3D-ResNet34 model pre-trained on the dataset, according to the strategy of transfer learning, freezes the first few layers of the model, modifies the last classification layer, and retrains the remaining layers to obtain visual facial action features suitable for emotion recognition.

[0080] As an optional implementation of the present invention, obtaining spatiotemporal human body posture features specifically includes:

[0081] The panoramic image sequence is input into MediaPipe to extract human key points, including 468 facial key points, 33 posture key points and 21 hand key points for each hand. Missing key points are filled with 0; the extracted facial key points, posture key points and hand key points are respectively input into the improved ST-GCN network, and then the obtained feature vectors are connected to form spatiotemporal human posture features.

[0082] like Figure 3As shown, first extract the key points of the human body, including the whole body posture, left and right hands, and face. The present invention uses MediaPipe to extract the features of human gesture tracking, facial recognition, posture estimation, etc. in the emotional video data set. The input panoramic image sequence can obtain 468 facial key points, 33 posture key points (human skeleton points) and 21 hand key points for each left and right hand through MediaPipe. For the missing key points, use 0 to fill. Spatio-Temporal Graph Convolutional Network (ST-GCN) is a neural network for video action recognition. It combines spatial and temporal information to learn the representation of data, where the spatial information comes from the pixels of the video frame and the temporal information comes from the time series of the video frame. By alternating the use of graph convolutional network (GCN) layers and temporal convolutional network (TCN) layers in the ST-GCN unit, the time and space dimensions are transformed to form a graph convolutional network for processing spatiotemporal data. In the GCN layer, the integration of key point features within the frame is achieved through the coefficient matrix; in the TCN, the convolution operation is used to model and analyze the time series of each key point. The basic key point extraction method used by ST-GCN is OpenPose, and the present invention adapts it to the MediaPipe key point extraction. The present invention uses an ST-GCN network containing 8 ST-GCN units, thereby reducing the complexity of the network. The key point information of the whole body posture, face, left and right hands is respectively input into the improved ST-GCN network, and then the obtained feature vectors are connected to form a new spatiotemporal human posture feature.

[0083] As an optional implementation of the present invention, obtaining scene emotion word association features specifically includes:

[0084] Collect and organize words that describe emotional states to obtain a list of emotion description words; input the image sequence and the emotion description words in the emotion description word list into the CLIP pre-trained model, and obtain the first image feature and the first text feature through the image encoder and text encoder respectively, normalize the vectors of the first image feature and the first text feature, and then matrix multiply them, and calculate the similarity between the two as the scene emotion word association feature.

[0085] For the scene emotion word association features, the present invention uses CLIP to obtain the similarity between the panoramic image and the emotion description words. CLIP is a multimodal learning model. Its core concept is to perform pre-training through image-text comparison learning, associate images and texts, and thus realize the calculation of image-text feature similarity. It focuses on analyzing the similarity between a large number of image-text data pairs, thereby building a semantic bridge between images and texts. The text encoder used in the present invention is a Text Transformer model with a parameter volume of 63M, and the visual encoder is a Transformer-based visual self-attention model (Vision Transformer, ViT), ViT-L / 14@336px model. It has been pre-trained on the WIT (WebImageText) dataset, which is a large-scale image-text pair dataset collected from the Internet, containing about 400 million pairs of images and texts.

[0086] The present invention collects and compiles a list of emotion description words from the Internet and literature, wherein the words include words describing people's emotional states such as frightened, afraid, hateful, resentful, happy, and refreshing. The present invention utilizes the characteristics of CLIP, sends the emotion description words and the image of the emotion video into the CLIP pre-training model, and obtains the first image feature and the first text feature through the image encoder and the text encoder respectively, normalizes the vectors of the first image feature and the first text feature, and then matrix multiplies them, and calculates the similarity between them, which is used as the scene emotion word association feature.

[0087] As an optional implementation manner of the present invention, obtaining text features based on CLIP specifically includes:

[0088] The emotional video is transcribed to extract text from the emotional video; the extracted text is preprocessed; the preprocessed text is input into the text encoder of the CLIP pre-trained model to obtain a second text feature; the second text feature and the first text feature are normalized and matrix-multiplied to obtain the similarity between the two as the CLIP-based text feature.

[0089] For text features, the present invention calculates the similarity between the text transcribed from the emotional video data set and the above-mentioned emotional description words, and uses this as a text feature based on CLIP. The text encoder of CLIP is based on the Transformer architecture and uses the Text Transformer model. Since the pre-training model has requirements for the length of the input text, some of the transcribed texts of the emotional video data set exceed the limit, so the input text needs to be preprocessed. According to the grammar of English vocabulary, first remove some articles, pronouns, adverbs, etc. that have no meaning, such as "a", "The", "some", and "bit". If it still cannot meet the model input, continue to remove some conjunctions, prepositions that are not related to emotions, and uncommon nouns that only appear in a few videos, such as "to", "and", "of", "when", "at", etc. In this way, the transcribed text of the emotional video data set meets the input requirements of CLIP, and it is input into the text encoder of the CLIP pre-training model to obtain the second text feature and normalize it. As described above, after the emotional description words are input into the text encoder, the first text feature is obtained. The vectors of the second text feature and the first text feature are normalized and matrix multiplied to obtain the similarity between them, which is used as a text feature based on CLIP. The scene emotion word association features and the text feature extraction process based on CLIP are as follows Figure 4 shown.

[0090] As an optional implementation manner of the present invention, obtaining multi-source audio features specifically includes:

[0091] Extract audio from emotional videos and calculate log-mel spectrograms; segment the log-mel spectrograms into frames and input them into a pre-trained VGGish CNN model to obtain VGGish CNN audio feature vectors; average multiple VGGish CNN audio feature vectors of each audio to obtain a first audio feature; use the Wav2Vec2-Large-Robus model fine-tuned on MSP-Podcast to extract audio features related to emotion recognition in emotional videos to obtain a second audio feature; fuse the first audio feature and the second audio feature to obtain a multi-source audio feature.

[0092] In order to increase the robustness of audio modality feature extraction, the present invention first uses VGGish CNN and Wav2Vec2-Large-Robus to extract two audio features respectively, and then fuses the two audio features.

[0093] The VGGish CNN model is pre-trained on the AudioSet dataset, which contains 2 million 10-second YouTube video audios, covering more than 600 audio event classes. The VGGish CNN model is based on the VGG architecture and uses 2D convolutional layers and pooling layers to process the spectrogram representation of the audio to adapt to the input of the audio data. It can extract feature vectors from the audio waveform that capture the high-level semantic information of the audio. Before entering the VGGish model, the audio needs to be processed to obtain the Log-Mel Spectrogram features. The specific steps are: read audio through the Soundfile library; use the resampy library to resample the audio to a 16kHz mono signal; use a periodic Hann window with a window size of 25 ms and a window jump of 10 ms to perform a short-time Fourier transform to calculate the spectrogram; map the spectrogram to 64 mel boxes covering the range of 125-7500Hz; calculate a stable log-mel spectrogram by applying the formula log(mel spectrum + 0.01), where the offset 0.01 is used to avoid taking the logarithm of zero; split the log-mel spectrogram into frames to generate the input of the VGGish CNN model, use the pre-trained VGGish CNN model as a feature extractor, and the feature vector obtained has a direct relationship with the length of the audio. Some audio does not reach 15 seconds, and it is repeatedly unified to obtain the VGGish CNN audio feature vector. In addition, it has been verified that averaging the 15 features of each audio is better than splicing and extracting the timing information for emotion recognition. Therefore, the average feature of the 15 VGGish CNN audio feature vectors of each audio is taken as the VGGish CNN audio feature, which is recorded as the first audio feature.

[0094] Wav2Vec 2.0 is designed specifically for handling challenging audio environments, such as speech recognition in noisy backgrounds. It is pre-trained on multiple speech datasets, including Libri-Light (open source audiobooks from the LibriVox project; clean, read-out audio data), CommonVoice (audio data collected by crowdsourcing; read-out text snippets), Switchboard (telephone speech corpus; noisy phone data), and Fisher (conversational phone speech; noisy phone data). The Wav2Vec2-Large-Robus model works well for clean audio from open source streaming platforms as well as noisy phone sounds, meeting the requirements of audio scenarios for processing complex emotional video datasets. MSP-Podcast (Multi-Speaker Podcast) is a public multi-speaker podcast dataset that contains audio from multiple podcast programs, involving different speakers and background noise. At the same time, the source of the MSP-Podcast dataset is similar to that of the emotional video dataset, and the audio collection environment has a certain consistency. Both contain a small amount of background noise, which will cause interference and is not conducive to judging emotions. The Wav2Vec2-Large-Robus model has also been pre-trained on a large amount of noisy audio, so the present invention believes that the Wav2Vec2-Large-Robus model fine-tuned on MSP-Podcast (v1.7) can extract the audio features of the emotional video dataset and can also reduce the background noise well. The audio features related to emotion recognition are extracted using the Wav2Vec2-Large-Robus model fine-tuned on MSP-Podcast, which are recorded as the second audio features.

[0095] After obtaining two audio features, the present invention innovatively proposes to fuse the two audio features first before fusing all features, that is, to fuse the first audio feature and the second audio feature, so as to capture the interactive information within the same data modality and obtain multi-source audio features.

[0096] In the embodiment of the present invention, the audio feature fusion process is as follows: Figure 5As shown. First, the audio feature vector based on Wav2Vec2-Large-Robus is adjusted to 128 dimensions using MLP, the same as the audio feature vector based on VGGish CNN, to provide the same amount of information. Then they are directly connected, and then Linear linear transformation is used to convert the merged 256 dimensions into 128 dimensions. The ReLU (Rectified Linear Unit) activation function is applied, and a dropout layer is used for regularization to reduce overfitting. Then it passes through two multi-layer Transformer models containing attention mechanisms and residual connections. Specifically, each multi-layer Transformer model contains a multi-head attention layer, using four attention heads. Each single attention head actually outputs its own output. The features output by the four attention heads are spliced ​​together to become the output features of the multi-head attention. The audio features after dropout processing are saved twice for subsequent residual connections. Two linear transformation layers expand the feature dimension to 1024 dimensions, and then reduce it back to 128 dimensions, and intersperse the application layer normalization technology.

[0097] The multi-head attention mechanism can focus on different parts of the audio and extract valuable information from the audio features, including but not limited to pitch, rhythm, intensity, etc., which is crucial for understanding the audio content. Residual connections improve training stability and help gradients propagate more effectively in the network, especially in deep networks, which helps to speed up the convergence rate of the model, while effectively dealing with the common gradient vanishing problem in deep neural networks and ensuring direct information flow from input to output. By expanding the feature dimension to 1024 dimensions through linear transformation layers, the model can capture more complex feature representations; and then reducing it to 128 dimensions, which helps the model reduce computational complexity while retaining information, and can improve the learning ability of the model. Through layer normalization technology, the training process can be significantly accelerated and the efficiency of model training can be enhanced. Dropout and layer normalization can improve the robustness of the model, making it more stable when facing different audio inputs. The Transformer layer is repeated twice, so that it can learn deeper feature representations. The above mechanisms work together to improve the generalization ability of the model, so that it can also perform well on unseen data. After adding these mechanisms, the fused audio features perform better in emotion recognition than direct connection.

[0098] S3. Based on a hierarchical feature fusion strategy, the visual modality feature, the audio modality feature and the text modality feature are fused to obtain a multimodal fusion feature.

[0099] like Figure 6 As shown, the hierarchical feature fusion strategy specifically includes:

[0100] In the first layer, on the one hand, the multi-source audio features and the CLIP-based text features are fused to obtain non-visual features; on the other hand, the visual face attribute features, the visual facial action features and the spatiotemporal human posture features are fused to obtain visual features;

[0101] In the second layer, the non-visual features and the visual features are fused;

[0102] In the third layer, the features obtained in the second layer are fused with the scene emotion word associated features to finally obtain multimodal fusion features.

[0103] Specifically, after steps S1 and S2, 4 visual modal features, 1 text modal feature, 2 audio modal features and their fusion features are obtained. The present invention uses a mid-term fusion method to fuse multimodal features, fuses 7 features through a Transformer architecture, and then sends them to a multi-layer perceptron (MLP) prediction model. Following the idea of ​​first fusing audio features, the present invention proposes a hierarchical feature fusion strategy, which fuses all features into three levels to form multimodal fusion features. The improvement of the fusion method is based on Transformer.

[0104] The hierarchical feature fusion strategy is one of the innovative points of the present invention, which integrates a variety of multimodal feature fusion methods to achieve the effect of 1+1>2. In the first layer, the audio modality features and text modality features represent the low-level acoustic features and speech content of speech, respectively, both of which are related to speech, so they are chosen to be fused first, following the way of audio feature fusion, first directly connected, then linearly changed to 128 dimensions, and activation functions and dropout are added, and then passed through two multi-layer Transformer models containing self-attention mechanisms and residual connections, using eight attention heads, and the feature dimension is expanded to 2048 dimensions in the linear change layer, and then reduced back to 128 dimensions, and interspersed with application layer normalization technology, and the output features are recorded as non-visual features.

[0105] In the first layer, visual face attribute features, visual facial action features, and spatiotemporal human posture features all belong to visual modal features and are concentrated on the speaker, so we choose to merge them first. For the combination of these three features, we did not directly connect them, but innovatively improved trilinear pooling based on bilinear pooling. Three linear transformations were used, and then the results of these transformations were combined by outer product and summation. This method learns the interaction between different features, and through linear transformation and outer product, generates a feature representation that integrates all modal information and captures the complex relationship between different features. The same Transformer model is used later, and the output features are recorded as visual features.

[0106] In the second layer, the non-visual features and visual features obtained above are fused. Since they already contain rich features, after connecting them, they are not directly linearly transformed into 128 dimensions, but first pass through the activation function, dropout, and Transformer model before linearly transforming into 128 dimensions.

[0107] In the third layer, the features obtained in the second layer are fused with the scene emotion word related features containing the most information, and the same fusion method as the second layer is used to finally obtain a 128-dimensional multimodal fusion feature.

[0108] S4. Inputting the multimodal fusion features into a multi-layer perceptron prediction model for emotion recognition to obtain an emotion recognition category corresponding to the emotion video.

[0109] In the mid-term model-based fusion, it is necessary to equip the fusion features with an additional prediction head. In the embodiment of the present invention, a multi-layer perceptron (MLP) which is easy to operate in the experiment is used as the prediction model. The fused features are used as input, and the feature dimensions are mapped into 7 dimensions through a linear layer. The multi-layer perceptron model is as follows: Figure 7 shown.

[0110] The first layer of the multi-layer perceptron (MLP) is the first fully connected layer, which linearly transforms the input multimodal fusion features into 32 dimensions; the second layer is a one-dimensional batch normalization layer, and the number of input and output channels is 32; the third layer is a ReLU (Rectified Linear Unit) activation function; the fourth layer is a random dropout layer, which discards 20% of neurons through regularization; the fifth layer is the second fully connected layer, which maps the feature dimension into 7 dimensions through linear transformation, corresponding to 7 classification results respectively; the sixth layer is a Sigmoid activation function, which converts the linear output to the [0,1] interval; the seventh layer is a label layer, and the label value is in the range of [0,1].

[0111] In summary, the present invention utilizes three different main modal information, including visual modal features, audio modal features and text modal features, which can greatly improve the performance of automatic emotion recognition. On the one hand, the present invention extracts spatiotemporal human posture features, visual face attribute features, visual facial action features, scene emotion word association features, CLIP-based text features, and multi-source audio features. Among them, the human key points consider facial key points, posture key points and hand key points, and extract features through the ST-GCN spatiotemporal graph convolutional network, which can more fully extract action features. The text feature uses the text encoder of CLIP pre-trained by image-text contrast learning to encode the transcribed text and personality descriptor respectively, and the similarity between the two is the new text feature. The audio model trained on various audios containing background noise is used to preliminarily extract audio features, and the noise reduction process is performed indirectly, and then added to the Transformer modality for fusion to obtain multi-source audio features. On the other hand, the present invention designs a hierarchical feature fusion strategy for the first time to fuse different multimodal features together. This hierarchical feature fusion strategy can process signals in a complementary way with features from different modalities, thus improving the robustness and generalization of automatic emotion recognition.

[0112] Accordingly, an embodiment of the present invention further provides an emotion recognition system based on the fusion of multimodal features and hierarchical features. Figure 8 FIG. 1 is a structural block diagram of an emotion recognition system based on multimodal features and hierarchical features fusion according to an exemplary embodiment. Figure 8 As shown, the system includes:

[0113] The data acquisition module 201 is used to collect emotional videos and convert them into an image sequence containing multiple images;

[0114] A feature extraction module 202 is used to detect faces in the image sequence and extract audio and text in the emotional video to obtain visual modality features, audio modality features and text modality features;

[0115] The visual modality features include: visual face attribute features, visual facial action features, spatiotemporal human posture features, and scene emotion word association features; the audio modality features include multi-source audio features; and the text modality features include CLIP-based text features.

[0116] A hierarchical feature fusion module 203 is used to fuse the visual modality feature, the audio modality feature and the text modality feature based on a hierarchical feature fusion strategy to obtain a multimodal fusion feature;

[0117] The emotion recognition module 204 is used to input the multimodal fusion features into a multi-layer perceptron prediction model to perform emotion recognition and obtain an emotion recognition category corresponding to the emotion video.

[0118] For ease of explanation, Figure 8 Only the main components of the system are shown. The system of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0119] The present invention can improve the robustness and generalization of the automatic emotion recognition system. The system can intelligently, quantitatively and conveniently identify the emotion categories of people in videos, and can be widely used in the analysis of people's emotional states in the fields of autonomous driving, school security, hospital monitoring, etc.

[0120] In an exemplary embodiment, the present invention further provides an electronic device, the electronic device comprising:

[0121] processor;

[0122] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the above-mentioned emotion recognition method based on the fusion of multimodal features and hierarchical features are implemented.

[0123] Fig. 9 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the electronic device 300 may include a processor 3001 and a memory 3002. Optionally, the electronic device 300 may further include a transceiver 3003. The processor 3001, the memory 3002 and the transceiver 3003 may be connected, for example, via a communication bus. The memory 3002 stores computer-readable instructions, which, when executed by the processor 3001, implement the steps of the emotion recognition method based on the fusion of multimodal features and hierarchical features as described above.

[0124] In a specific implementation, as an embodiment, the processor 3001 may include one or more CPUs, such as Fig. 9 CPU0 and CPU1 are shown in FIG.

[0125] In a specific implementation, as an embodiment, the electronic device 300 may also include multiple processors, such as Fig. 9 3001 and processor 3004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0126] The memory 3002 is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor 3001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0127] The transceiver 3003 is used to communicate with a network device or a terminal device.

[0128] Optionally, the transceiver 3003 may include a receiver and a transmitter, wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0129] Optionally, the transceiver 3003 may be integrated with the processor 3001 , or may exist independently and be coupled to the processor 3001 via an interface circuit of the electronic device 300 , which is not specifically limited in the embodiment of the present invention.

[0130] It should be noted that Fig. 9 The structure of the electronic device 300 shown in the figure does not constitute a limitation on the electronic device. The actual electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects of the electronic device 300 can refer to the technical effects of the above-mentioned method embodiment, which will not be repeated here.

[0131] In an exemplary embodiment, the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is loaded and executed by a processor to implement the steps of the emotion recognition method based on multimodal features and hierarchical features fusion as described above. For example, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0132] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0133] References in the specification to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments", etc. indicate that the described embodiments may include a particular feature, structure, or characteristic, but not every embodiment may include the particular feature, structure, or characteristic. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the relevant art to implement such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0134] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0135] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0136] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0137] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of the system or unit, which can be electrical, mechanical or other forms.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0140] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0141] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An emotion recognition method based on the fusion of multimodal features and hierarchical features, It is characterized in that The following steps are involved: S1, collect emotional videos and convert them into image sequences containing multiple images; S2, detecting faces in the image sequence, and extracting audio and text in the emotional video to obtain visual modality features, audio modality features, and text modality features; The visual modality features include: visual face attribute features, visual facial action features, spatiotemporal human posture features, and scene emotion word association features; the audio modality features include multi-source audio features; and the text modality features include CLIP-based text features. Acquiring visual face attribute features specifically includes: inputting the panoramic image sequence into the FaceXFormer model to obtain the face attribute features of each image; processing the time series information through the long short-term memory network LSTM to obtain the visual face attribute features; Acquiring visual facial action features specifically includes: using a face detector to segment the portion of the image sequence containing a face, and if no face is detected, using a panoramic image instead; based on a transfer learning strategy, inputting the image sequence into a pre-trained 3D-ResNet34 model to obtain the visual facial action features; The acquisition of spatiotemporal human posture features specifically includes: inputting the panoramic image sequence into MediaPipe to extract human key points, including 468 facial key points, 33 posture key points and 21 hand key points for each of the left and right hands, and filling the missing key points with 0; inputting the extracted facial key points, posture key points and hand key points into the improved ST-GCN network respectively, and then connecting the obtained feature vectors to form the spatiotemporal human posture features; Acquiring scene emotion word associated features specifically includes: collecting and organizing words that describe emotional states to obtain an emotion description word list; inputting the image sequence and the emotion description words in the emotion description word list into a CLIP pre-trained model, and respectively obtaining a first image feature and a first text feature through an image encoder and a text encoder, and performing matrix multiplication of the vectors of the first image feature and the first text feature after normalization, and calculating the similarity between the two as the scene emotion word associated features; Acquiring the CLIP-based text feature specifically includes: transcribing the emotion video to extract text from the emotion video; preprocessing the extracted text; inputting the preprocessed text into a text encoder of a CLIP pre-trained model to obtain a second text feature; matrix-multiplying the vectors of the second text feature and the first text feature after normalization to obtain a similarity between the two as the CLIP-based text feature; Acquiring multi-source audio features specifically includes: extracting audio from the emotion video and calculating a log-mel spectrogram; dividing the log-mel spectrogram into frames, inputting the pre-trained VGGish CNN model, and obtaining a VGGish CNN audio feature vector; averaging multiple VGGish CNN audio feature vectors of each audio to obtain a first audio feature; using a Wav2Vec2-Large-Robus model fine-tuned on MSP-Podcast to extract audio features related to emotion recognition in the emotion video to obtain a second audio feature; fusing the first audio feature and the second audio feature to obtain the multi-source audio feature; S3, fusing the visual modality feature, the audio modality feature and the text modality feature based on a hierarchical feature fusion strategy to obtain a multimodal fusion feature; Among them, the hierarchical feature fusion strategy specifically includes: In the first layer, on the one hand, the multi-source audio features and the CLIP-based text features are fused to obtain non-visual features; on the other hand, the visual face attribute features, the visual facial action features and the spatiotemporal human posture features are fused to obtain visual features; In the second layer, the non-visual features and the visual features are fused; In the third layer, the features obtained in the second layer are fused with the scene emotion word associated features to finally obtain multimodal fusion features; S4. Inputting the multimodal fusion features into a multi-layer perceptron prediction model for emotion recognition to obtain an emotion recognition category corresponding to the emotion video.

2. The emotion recognition method based on the fusion of multimodal features and hierarchical features according to claim 1, It is characterized in that In step S4, a multi-layer perceptron is used as a prediction model; The first layer of the multilayer perceptron is the first fully connected layer, which linearly transforms the input multimodal fusion features into 32 dimensions; the second layer is a one-dimensional batch normalization layer, and the number of input and output channels is 32; the third layer is a ReLU activation function; the fourth layer is a random dropout layer, which discards 20% of neurons through regularization; the fifth layer is the second fully connected layer, which maps the feature dimension into 7 dimensions through linear transformation, corresponding to 7 classification results respectively; the sixth layer is a Sigmoid activation function, which converts the linear output to the [0,1] interval; the seventh layer is a label layer, and the label value is in the range of [0,1].

3. An emotion recognition system based on multimodal features and hierarchical features fusion, the system is used to implement the method as claimed in claim 1 or 2, It is characterized in that The system comprises: A data acquisition module, used for acquiring emotional videos and converting them into image sequences containing multiple images; A feature extraction module, used to detect faces in the image sequence, extract audio and text in the emotional video, and obtain visual modality features, audio modality features, and text modality features; The visual modality features include: visual face attribute features, visual facial action features, spatiotemporal human posture features, and scene emotion word association features; the audio modality features include multi-source audio features; and the text modality features include CLIP-based text features. A hierarchical feature fusion module, used for fusing the visual modality feature, the audio modality feature and the text modality feature based on a hierarchical feature fusion strategy to obtain a multimodal fusion feature; The emotion recognition module is used to input the multimodal fusion features into a multi-layer perceptron prediction model for emotion recognition to obtain an emotion recognition category corresponding to the emotion video.