Emotion recognition method based on visual language pre-training and multimodal collaborative fusion

Abstract

The present invention discloses an emotion recognition method based on visual language pre-training and multi-modal collaborative fusion, belonging to the technical field of multi-modal emotion recognition; the present invention firstly pre-trains a visual encoder using a large-scale visual-text pair data set through a visual language pre-training paradigm to improve its visual feature extraction capability; then extracts image frames, skeleton nodes, and audio from a video, and transmits them to encoders of each modality respectively to extract features of multiple modalities; fuses the features of the skeleton nodes and the features extracted from the video frames through a cross-attention mechanism to further enhance the visual features through information such as human motion posture; finally, fuses the visual features and the audio features into the query vector through a learnable query vector and a cross-attention mechanism; and uses the learned query vector features to perform emotion classification in the prediction stage.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal emotion recognition, and in particular to an emotion recognition method based on visual language pre-training and multimodal collaborative fusion. Background Art

[0002] With the rapid development of the Internet and digital media technologies, the generation and dissemination of multimedia data has become an important part of people's daily lives. As an important human-computer interaction technology, emotion recognition technology analyzes audio and video data through deep learning and automatically identifies the emotional information contained therein. It is widely used in social media analysis, intelligent customer service, mental health monitoring, intelligent surveillance and other fields. However, current emotion recognition technology faces many challenges, especially in how to effectively integrate multimodal information and improve recognition accuracy.

[0003] Although the continuous development of artificial intelligence has promoted significant progress in emotion recognition technology based on deep learning, previous technical methods usually rely on single-modal data types for analysis. For example, audio emotion recognition extracts voice features (such as pitch, volume, and speech speed) to determine the speaker's emotional state; video emotion recognition relies on visual information such as facial expressions, movements, the scene in which the character is located, and the interaction between the character and other objects. Although these single-modal emotion recognition methods can achieve certain results in certain specific scenarios, they often show great limitations when dealing with complex emotions and diverse scenarios.

[0004] In order to improve the accuracy and robustness of emotion recognition, multimodal emotion recognition technology attempts to make up for the shortcomings of single-modal methods by fusing multiple data types from different sources. Existing emotion recognition methods based on multimodal fusion usually only perform simple feature splicing and fusion of data from different modalities. For example, some studies extract audio and video features separately and then directly splice them, or perform weighted averaging of emotion recognition results of each modality at the decision level. However, these methods often lack sufficient synergy mechanisms in the fusion process, making it difficult to fully utilize the complementary information between different modalities.

[0005] In recent years, many methods for effectively fusing multimodal data have emerged in the field of deep learning. For example, the Cross-Attention Mechanism has been widely used in natural language processing and multimodal learning through multimodal cross-learning; by introducing learnable tokens or learnable query vectors for joint learning with different modal data, effective interaction and fusion between different modal information can also be achieved, thereby improving the performance of multimodal tasks. However, these methods have not yet been widely used in the field of emotion recognition.

[0006] In addition, due to the abstract nature of emotional information, how to effectively extract information related to emotions from complex videos or images is also a huge challenge. In recent years, the visual language pre-training paradigm has made significant progress in the field of computer vision, providing new ideas for improving the understanding of video information and the ability to recognize emotions. Through the pre-training method of comparative learning of visual and language features on a large-scale image-text pair dataset, it enables the visual model to learn rich visual feature representations and show strong generalization capabilities in multiple downstream tasks. At the same time, human skeleton key point data, as a structured data commonly used in posture estimation and behavior recognition, provides key information about human actions and postures, which is highly related to emotions and has not received enough attention in the field of emotion recognition. Through the effective fusion of the visual language pre-training paradigm and skeleton features, the performance of the model in emotion recognition can be further enhanced, providing more accurate technical support for in-depth understanding of complex emotional interactions.

[0007] Existing emotion recognition technologies still have many deficiencies in terms of single-modal and multi-modal fusion, and it is difficult to give full play to the advantages of multi-modal data. Therefore, how to effectively integrate multi-modal information such as vision, skeleton nodes and audio, and make full use of the powerful representation ability of visual language pre-training models to improve the performance of emotion recognition has become an important direction of current research.

[0008] In order to solve the above problems, the present invention proposes an emotion recognition method based on visual language pre-training and multimodal collaborative fusion. Summary of the invention

[0009] The purpose of the present invention is to provide an emotion recognition method based on visual language pre-training and multimodal collaborative fusion to solve the problems raised in the background technology:

[0010] 1) Existing technologies are limited to analyzing and processing data from a single modality, or can only simply fuse data features from multiple modalities, without fully utilizing the complementary information between data from different modalities. This has significant limitations when processing complex emotions and diverse scenarios.

[0011] 2) Due to the abstract nature of emotional information, it is difficult for existing visual emotion recognition methods to directly extract valuable visual features from videos or pictures. The extracted visual features will be interfered by other irrelevant objects such as the background, resulting in poor performance of the emotion recognition method and inability to be applied in practice.

[0012] In order to achieve the above object, the present invention adopts the following technical solutions:

[0013] The emotion recognition method based on visual language pre-training and multi-modal collaborative fusion includes the following steps:

[0014] S1. Using a large-scale public video-text pair dataset, according to the visual language pre-training paradigm, the video and the corresponding text of the dataset are input into the visual encoder and text encoder respectively to obtain visual and text features, and the cosine similarity between the two is calculated. The gradient is then back-propagated and the model parameters are updated through a symmetric cross-entropy loss function to obtain a pre-trained visual encoder.

[0015] S2, using the uniform sampling method to evenly divide all the frames in the video of the emotion recognition dataset into 8 groups, and then randomly extract a frame of image from each group, and then randomly crop it, and adjust the image size to 224×224;

[0016] S3, use the open source OpenPose tool to extract the skeleton nodes from the frame image in S2, and generate the corresponding heat map according to the two-dimensional coordinates and confidence information of the human skeleton nodes;

[0017] S4, extract the audio from the video of the emotion recognition dataset, generate a mel-spectrogram, and evenly divide it into 8 parts according to the time scale so that it corresponds to the time information of the visual data;

[0018] S5, divide the image into blocks of size 32×32, and then input it into the pre-trained visual encoder in S1 to extract visual features;

[0019] S6, inputting the skeleton node heat map obtained in S3 and the mel spectrogram data of the audio obtained in S4 into the corresponding ResNet encoders for encoding, extracting features, and aligning the skeleton and audio feature dimensions with the visual feature dimensions through linear layers and pooling layers;

[0020] S7. By using the cross-attention mechanism, the skeleton node features are used as queries and the visual image features are used as keys and values. The skeleton node features and the visual image features are fused to inject significant human motion posture information into the visual features to enhance the representation ability of the visual features.

[0021] S8, create a set of learnable query vectors and perform self-attention calculation on the vectors to enhance their features;

[0022] S9, using the cross attention mechanism, taking the learnable vector created in S8 as the query, taking the visual features and audio features as the key and value, and fusing them, injecting the visual and audio information into the learnable query vector, and then inputting the learnable query vector into the multi-layer perceptron to further enhance the features;

[0023] S10, averaging the learnable query vectors, and mapping the number of feature dimensions to the number of emotion categories through a linear layer;

[0024] S11. Calculate the binary cross entropy loss function for the predicted value and the actual sentiment label, and iteratively update the parameters of the entire model except the visual encoder through gradient backpropagation. The parameters of the visual encoder keep their pre-trained weights unchanged.

[0025] S12. After the training process is completed, the final output of the model is calculated by the Sigmoid function during the recognition process, and the emotion category with a calculated value greater than 0.5 is used as the final predicted emotion of the corresponding video.

[0026] Preferably, the large-scale video-text pair dataset in S1 is collected and obtained through the Internet, specifically: a long video obtained from the Internet is cut into short video clips containing character scenes, and the corresponding subtitle text is reorganized into short sentences to obtain video-text pairs;

[0027] Based on the above content, the S1 specifically includes:

[0028] Based on the visual language pre-training paradigm, the cosine similarity of visual features and text features is calculated. Let I be the visual feature matrix and T be the text feature matrix. Both are in the shape of N×d, where N represents the batch size and d represents the feature dimension. Let the cosine similarity matrix be S, and the specific calculation formula is:

[0029]

[0030] Among them, S ij represents the cosine similarity between the i-th video and the j-th text;

[0031] Calculate the cross entropy loss in two directions:

[0032] The formula for calculating the cross entropy loss from vision to text is:

[0033]

[0034] The formula for calculating the cross entropy loss from text to vision is:

[0035]

[0036] Calculate the sum of the above two cross entropy losses, recorded as the symmetric cross entropy loss L:

[0037] L=L I→T +L T→I

[0038] Minimizing the symmetric cross entropy loss L makes related videos and texts closer in the embedding space, realizes multimodal contrastive learning, and then obtains a pre-trained visual encoder. The visual encoder learned in this way can show strong generalization ability in various downstream tasks.

[0039] Preferably, the skeleton nodes in S3 include the coordinates of 18 key nodes of the human body and their estimated confidences, and the 18 nodes are respectively:

[0040] Serial number 1 2 3 4 5 6 7 8 9 node nose neck Right shoulder right elbow right wrist Left shoulder left elbow left wrist Right hip Serial number 10 11 12 13 14 15 16 17 18 node right knee Right ankle Left hip Left knee Left ankle Right eye Left Eye Right ear left ear

[0041] The S3 specifically includes the following contents:

[0042] The extracted skeleton node is recorded as v. For each skeleton node v, in each frame f, assume that the position of the skeleton node is (x f ,y f ) and the confidence score is s f , use a two-dimensional Gaussian function to generate a bone node heat map:

[0043]

[0044] Where σ represents the standard deviation of the Gaussian function;

[0045] Assuming there are F frames in total, the size of each frame is H×W, and each person has V bone nodes, then the final heat map dimensions of all frames are (F, V, H, W).

[0046] Preferably, the mel-spectrogram generation method in S4 includes the following contents:

[0047] Assume that the audio signal read is y(t), where t represents time; calculate the short-time Fourier transform of the audio signal and obtain the spectrum D:

[0048] D=STFT(y(t),hop_length,n_fft)

[0049] Among them, STFT represents short-time Fourier transform, hop_length represents the step length of window movement, and n_fft represents the number of FFT points;

[0050] Compute the magnitude spectrum:

[0051]

[0052] Where Re(D) and Im(D) represent the real part and imaginary part of D respectively;

[0053] Use a Mel filter to convert the linear spectrum to a Mel spectrum:

[0054] S=M·|D|

[0055] Where M represents the Mel filter matrix and S represents the Mel spectrum graph;

[0056] Convert the Mel spectrum to the logarithmic power Mel spectrum and normalize it to get the final Mel spectrum:

[0057] S log =20log 10 (max(S,ε))

[0058]

[0059] Among them, ε is a constant to prevent numerical problems in logarithmic calculations, and min_level_db represents the minimum value of the Mel spectrum;

[0060] Then divide it into 8 parts evenly according to the time scale to correspond to the time information of the visual data.

[0061] Preferably, the features extracted in S5 have wide adaptability and are conducive to improving the performance of emotion recognition. Therefore, the visual encoder is frozen during the training process to keep its parameters unchanged.

[0062] Preferably, the dimensions of the skeleton and audio data in S6 are small and the data pattern is relatively simple, so there is no need to use a Transformer model with a large number of parameters like a visual encoder, and no pre-training is required to achieve good performance. After extracting the features, the skeleton and audio feature dimensions are aligned with the visual feature dimensions through a linear layer and a pooling layer, and the dimension size is 512.

[0063] Preferably, the feature fusion described in S7 uses the skeleton node features as queries and the visual image features as keys and values, so that the rich human body information contained in the skeleton features can be injected into the visual features, so that the model can not only utilize the scene information in the video frame image, but also capture the human body's movement and posture information, thereby enhancing the emotion recognition performance in complex scenes.

[0064] Preferably, the S8 specifically includes the following contents:

[0065] Create 8 query vectors as a set of query vectors and initialize them with a normal distribution with a mean of 0 and a variance of 0.02. In each training, self-attention calculation is performed first, so that each query vector can consider the content of all other query vectors when fusing other modalities, thereby enhancing the representation ability of each query vector, capturing richer and more comprehensive contextual information, and making subsequent multimodal feature interactions more effective.

[0066] Preferably, the cross-attention mechanism described in S9 enables the learnable query vector to focus on the most relevant parts of the visual features and audio features, thereby extracting information that is helpful for emotion understanding and improving the flexibility and adaptability of the model.

[0067] Preferably, the emotion categories in S10 include the following:

[0068]

[0069]

[0070] Preferably, the calculation formula of the binary cross entropy loss function in S11 is:

[0071]

[0072] Among them, y represents the vector composed of the true sentiment labels of all samples in the batch, Represents a vector of predicted emotion probabilities for all samples in a batch, and N represents the number of samples in each batch of data. By calculating gradients and backpropagation, the binary cross entropy loss function is minimized, and the model parameters (except the visual encoder) are continuously updated to improve the performance of emotion recognition.

[0073] Preferably, the emotion prediction method in S12 is to map the prediction value corresponding to each emotion to the (0,1) interval through the Sigmoid function, and regard the emotion with a value greater than 0.5 as the emotion predicted by the sample, thereby realizing the emotion recognition function. The calculation formula of the Sigmoid function is:

[0074]

[0075] Among them, x represents the input feature and e represents the exponential operation.

[0076] Compared with the prior art, the present invention provides an emotion recognition method based on visual language pre-training and multi-modal collaborative fusion, which has the following beneficial effects:

[0077] (1) Different from the previous emotion recognition methods that directly extract visual features from original images or videos from scratch, the present invention obtains a visual encoder with excellent generalization ability through the visual language pre-training paradigm, and further strengthens the visual features through the fusion of skeleton features, highlighting the information contained therein, such as character postures and movements, which are highly related to emotions, and can greatly improve the emotion recognition performance.

[0078] (2) Different from the previous methods that only use single-modal data for emotion recognition, the present invention uses data in multiple modalities such as video frames, skeleton nodes and audio for collaborative training. By comprehensively utilizing the complementary information of these different types of data, the emotion recognition performance of the model is greatly improved, overcoming the limitations of traditional single-modal emotion recognition methods that perform poorly in complex emotions and diverse scenarios.

[0079] (3) Different from the previous method of multimodal emotion recognition by simple feature dimension splicing and fusion at the decision-making level, this invention innovatively adopts advanced mechanisms such as learnable query vectors and cross-attention, making full use of the complementary information between multiple modal data to achieve more effective fusion processing. The learnable vector can improve the flexibility and adaptability of the model, and the cross-attention mechanism further enhances the information interaction between different modal data, so that the model can show higher accuracy and robustness in complex and changing emotional scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a flow chart of using the emotion recognition method based on visual language pre-training and multi-modal collaborative fusion to perform emotion recognition in Example 1 of the present invention;

[0081] Figure 2 The emotion prediction results of different samples using the emotion recognition method based on visual language pre-training and multimodal collaborative fusion in Example 1 of the present invention;

[0082] Figure 3 Schematic diagram of different modal data used in Example 1 of the present invention. DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0084] The present invention proposes an emotion recognition method based on visual language pre-training and multimodal collaborative fusion, which strengthens the visual encoder through the visual language pre-training paradigm, and effectively fuses the data of multiple modes of video, skeleton nodes, and audio through the cross-attention mechanism and the learnable query vector, so as to improve the performance of the emotion recognition algorithm. The technical scheme adopted by the present invention is an audio-video multimodal fusion emotion recognition method based on visual language pre-training and learnable query vector. Different from the previous single-modal or simple multimodal fusion emotion recognition method, the present invention aims to break through the existing limitations by strengthening visual feature extraction and promoting multimodal effective fusion. Specifically, the present invention first pre-trains the visual encoder using a large-scale visual-text pair data set through the visual language pre-training paradigm to improve its visual feature extraction ability; then extracts image frames, skeleton nodes, and audio from the video, and respectively transmits them into the trained single-modal encoder to extract features of multiple modes; fuses the skeleton node features and the features extracted from the video frame through the cross-attention mechanism to further strengthen the visual features through information such as human action posture; finally, through the learnable query vector and the cross-attention mechanism, the visual features and audio features are fused into the query vector. The prediction phase uses the learned query vectors for sentiment classification.

[0085] This invention is funded by the National Natural Science Foundation of China - Human Micro-gesture Recognition and Sentiment Analysis Based on Self-supervised Learning 62171309. The classification performance and interpretability of the model interpretation method based on federated learning proposed in the present invention are described below in combination with specific drawings and specific examples, specifically including the following contents.

[0086] Embodiment 1:

[0087] This paper proposes an emotion recognition method based on visual language pre-training and multi-modal collaborative fusion, please refer to Figure 1 The overall process can be divided into three parts:

[0088] 1) Data preprocessing: Extract frame images and audio from the input video, extract skeleton nodes from the frame images and generate heat maps, and generate mel-spectrograms for audio data to facilitate feature extraction by subsequent deep learning models.

[0089] 2) Feature extraction module: First, a visual encoder with excellent generalization ability is obtained through the visual language pre-training paradigm, and the pre-trained visual encoder is used to extract features from the frame image. The ResNet encoder is used to extract features from the skeleton node heat map and Mel spectrogram to obtain skeleton features and audio features. The visual features are further enhanced by fusing the visual image features with the skeleton features, highlighting the information contained therein, such as the character posture and action, which is highly related to emotions.

[0090] 3) Multimodal fusion module: Using learnable query vectors and cross-attention modules, information from different modalities is injected into these vectors. After training, these learned vectors are used for emotion recognition.

[0091] For the details of each module above, the specific implementation steps are introduced in detail as follows:

[0092] Step 1: Use a large-scale video-text pair dataset to perform pre-training according to the visual language pre-training paradigm. This dataset is more than 3,000 TV series collected from the Internet, and these long videos are divided into short video clips containing character scenes. At the same time, the corresponding subtitle texts are reorganized into short sentences, and finally about one million video-text pairs are obtained. Both the visual encoder and the text encoder use the Transformer backbone network commonly used in the field. The cosine similarity calculation is shown in the following formula:

[0093]

[0094] Among them, u and v are two vectors, u·v represents the dot product of the vectors, and ||u|| and ||v|| represent the Euclidean norm of the vector (that is, the length of the vector).

[0095] Specifically, the visual language pre-training paradigm usually calculates the cosine similarity of visual features and text features. Let I be the visual feature matrix and T be the text feature matrix. Both are in the shape of N×d, where N is the batch size and d is the feature dimension. In this model, d is 512, and N can be adjusted as needed. Let the cosine similarity matrix be S, where S ij Represents the cosine similarity between the i-th video and the j-th text:

[0096]

[0097] Next, we calculate the cross entropy loss in both directions. The cross entropy loss from vision to text is:

[0098]

[0099] Cross entropy loss from text to vision:

[0100]

[0101] Finally, the symmetric cross entropy loss L is the sum of the losses in these two directions:

[0102] L=L I→T +L T→I

[0103] By minimizing this loss function, the visual-language pre-training paradigm can bring related videos and texts closer in the embedding space, thereby achieving efficient multimodal contrastive learning. The learned visual encoder can show strong generalization ability in various downstream tasks.

[0104] Step 2: Divide all frames of the video in the emotion recognition dataset into 8 groups evenly, then randomly select one frame from each group, perform uniform sampling, randomly crop, and resize the image to 224×224.

[0105] Step 3: For the subject in step 2, the two-dimensional coordinates of 18 key nodes of the human body and their estimated confidence levels are estimated. The 18 nodes are: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, and left ear.

[0106] For each skeleton key point v, in each frame f, assume that the position of the key node is (x f ,y f ) and the confidence score is s f , use a two-dimensional Gaussian function to generate a bone node heat map:

[0107]

[0108] Where σ is the standard deviation of the Gaussian function. Assuming there are F frames, the size of each frame is H×W, and each person has V bone nodes, then the dimensions of the heat map of all frames are (F, V, H, W).

[0109] Step 4: Extract audio from the video in the emotion recognition dataset and generate a mel-spectrogram. The generation method is as follows:

[0110] Assume that the audio signal read is y(t), where t represents time. Calculate the short-time Fourier transform (STFT) of the audio signal and obtain the spectrum D:

[0111] D=STFT(y(t),hop_length,n_fft)

[0112] Among them, STFT is short-time Fourier transform, hop_length is the step length of window movement, and n_fft is the number of points of FFT. In this method, the number of FFT points is 512 and the step length of window movement is 256.

[0113] Compute the magnitude spectrum:

[0114]

[0115] Where Re(D) and Im(D) represent the real part and imaginary part of D, respectively.

[0116] Then use a Mel filter to convert the linear spectrum to a Mel spectrum:

[0117] S=M·|D|

[0118] Where M is the Mel filter matrix and S is the Mel spectrogram.

[0119] The Mel spectrum is then converted to a logarithmic power Mel spectrum, which is then normalized to obtain the final Mel spectrum:

[0120] S log =20log 10 (max(S,ε))

[0121]

[0122] Among them, ε is a small constant to prevent numerical problems in logarithmic calculations, and min_level_db is the minimum value of the Mel spectrum.

[0123] Then divide it into 8 parts evenly according to the time scale to correspond to the time information of the visual data.

[0124] Step 5: Divide the frame image into 32×32 blocks and input them into the pre-trained visual Transformer encoder in step 1 to extract visual features. These features have wide adaptability and are conducive to improving the performance of emotion recognition. Therefore, the visual encoder is frozen during the training process to keep its parameters unchanged.

[0125] Step 6: Input the bone node heat map and the audio mel spectrogram data into the corresponding ResNet encoder for encoding and feature extraction. Since the dimensions of the original bone and audio data are small and the data pattern is relatively simple, there is no need to use a Transformer model with a large number of parameters like the visual encoder, and no pre-training is required to achieve good performance. After extracting the features, the bone and audio feature dimensions are aligned with the visual feature dimensions through linear layers and pooling layers, and the dimension size is 512.

[0126] Step 7: Fuse the skeleton features and visual image features through the cross-attention mechanism. Using the skeleton node features as queries and the visual image features as keys and values, the rich human information contained in the skeleton features can be injected into the visual features, so that the model can not only utilize the scene information in the video frame image, but also capture the human body's motion and posture information, thereby enhancing the emotion recognition performance in complex scenes.

[0127] Step 8: Create 8 query vectors as a set of query vectors and initialize them with a normal distribution with a mean of 0 and a variance of 0.02. In each training, self-attention calculation is performed first, so that each query vector can consider the content of all other query vectors when fusing other modalities, thereby enhancing the representation ability of each query vector, capturing richer and more comprehensive context information, and making subsequent multimodal feature interactions more effective.

[0128] Step 9: Through two layers of cross attention, the learnable vector is used as the query, and the visual features and audio features are used as the key and value respectively, so that the learnable query vector can focus on the most relevant parts of the visual features and audio features, thereby extracting information that is helpful for emotion understanding and improving the flexibility and adaptability of the model.

[0129] Step 10: Average the learned query vector set and map its dimension to the number of emotion categories through a linear layer. In this method, 26 emotion categories are selected, which are sufficient to cover a wide range of human emotional expressions, including: peace, love, self-esteem, expectation, commitment, confidence, happiness, pleasure, excitement, surprise, sympathy, confusion, indifference, fatigue, embarrassment, desire, disgust, disgust, annoyance, anger, sensitivity, sadness, anxiety, fear, pain, and suffering.

[0130] Step 11: Considering that there is not a single emotion category, there are often multiple emotion categories appearing at the same time. Therefore, this method chooses to use the binary cross entropy loss function and regards the emotion recognition problem as an emotion multi-label classification problem. The calculation formula of the loss function is as follows:

[0131]

[0132] Among them, y represents the vector composed of the true sentiment labels of all samples in the batch, Represents a vector of predicted emotion probabilities for all samples in a batch, and N represents the number of samples in each batch of data. By calculating gradients and backpropagation, the binary cross entropy loss function is minimized, and the model parameters (except the visual encoder) are continuously updated to improve the performance of emotion recognition.

[0133] Step 12: After the training process is completed, in the emotion recognition process, the predicted value corresponding to each emotion output by the model is mapped to the (0,1) interval through the Sigmoid function, and the emotion with a value greater than 0.5 is regarded as the emotion predicted by the sample, thereby realizing the emotion recognition function. The formula of the Sigmoid function is as follows:

[0134]

[0135] Among them, x represents the input feature and e represents the exponential operation.

[0136] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. Emotion recognition method based on visual language pre-training and multi-modal collaborative fusion, characterized in that: The following steps are involved: S1. Using a large-scale public video-text pair dataset, according to the visual language pre-training paradigm, the video and the corresponding text of the dataset are input into the visual encoder and text encoder respectively to obtain visual and text features, and the cosine similarity between the two is calculated. The gradient is then back-propagated and the model parameters are updated through a symmetric cross-entropy loss function to obtain a pre-trained visual encoder. S2, using the uniform sampling method to evenly divide all the frames in the video of the emotion recognition dataset into 8 groups, and then randomly extract a frame of image from each group, and then randomly crop it, and adjust the image size to 224×224; S3, use the open source OpenPose tool to extract the skeleton nodes from the frame image in S2, and generate the corresponding heat map according to the two-dimensional coordinates and confidence information of the human skeleton nodes; S4, extract the audio from the video of the emotion recognition dataset, generate a mel-spectrogram, and evenly divide it into 8 parts according to the time scale so that it corresponds to the time information of the visual data; S5, divide the image into 32×32 blocks, and then input it into the pre-trained visual encoder in S1 to extract visual features; S6, inputting the skeleton node heat map obtained in S3 and the mel spectrogram data of the audio obtained in S4 into the corresponding ResNet encoders for encoding, extracting features, and aligning the skeleton and audio feature dimensions with the visual feature dimensions through linear layers and pooling layers; S7. By using the cross-attention mechanism, the skeleton node features are used as queries and the visual image features are used as keys and values. The skeleton node features and the visual image features are fused to inject significant human motion posture information into the visual features to enhance the representation ability of the visual features. S8, create a set of learnable query vectors and perform self-attention calculation on the vectors to enhance their features; S9, using the cross attention mechanism, taking the learnable vector created in S8 as the query, taking the visual features and audio features as the key and value, and fusing them, injecting the visual and audio information into the learnable query vector, and then inputting the learnable query vector into the multi-layer perceptron to further enhance the features; S10, averaging the learnable query vectors, and mapping the number of feature dimensions to the number of emotion categories through a linear layer; S11. Calculate the binary cross entropy loss function for the predicted value and the actual sentiment label, and iteratively update the parameters of the entire model except the visual encoder through gradient backpropagation. The parameters of the visual encoder keep their pre-trained weights unchanged. S12. After the training process is completed, the final output of the model is calculated by the Sigmoid function during the recognition process, and the emotion category with a calculated value greater than 0.5 is used as the final predicted emotion of the corresponding video.

2. The emotion recognition method based on visual language pre-training and multi-modal collaborative fusion according to claim 1 is characterized in that: The large-scale video-text pair dataset described in S1 is collected and obtained through the Internet, specifically: the long video obtained from the Internet is cut into short video clips containing character scenes, and the corresponding subtitle text is reorganized into short sentences to obtain video-text pairs; Based on the above content, the S1 specifically includes: Based on the visual language pre-training paradigm, the cosine similarity of visual features and text features is calculated. Let I be the visual feature matrix and T be the text feature matrix. Both are in the shape of N×d, where N represents the batch size and d represents the feature dimension. Let the cosine similarity matrix be S, and the specific calculation formula is: Among them, S ij represents the cosine similarity between the i-th video and the j-th text; Calculate the cross entropy loss in two directions: The formula for calculating the cross entropy loss from vision to text is: The formula for calculating the cross entropy loss from text to vision is: Calculate the sum of the above two cross entropy losses, recorded as the symmetric cross entropy loss L: L=L I→T +L T→I Minimizing the symmetric cross entropy loss L makes related videos and texts closer in the embedding space, realizes multimodal contrastive learning, and then obtains the pre-trained visual encoder.

3. The emotion recognition method based on visual language pre-training and multi-modal collaborative fusion according to claim 1 is characterized in that: The S3 specifically includes the following contents: The extracted skeleton node is recorded as v. For each skeleton node v, in each frame f, assume that the position of the skeleton node is (x f ,y f ) and the confidence score is s f , use a two-dimensional Gaussian function to generate a bone node heat map: Where σ represents the standard deviation of the Gaussian function; Assuming there are F frames in total, the size of each frame is H×W, and each person has V bone nodes, then the final heat map dimensions of all frames are (F, V, H, W).

4. The emotion recognition method based on visual language pre-training and multi-modal collaborative fusion according to claim 1 is characterized in that: The method for generating a mel-spectrogram described in S4 includes the following contents: Assume that the audio signal read is y(t), where t represents time; calculate the short-time Fourier transform of the audio signal to obtain the spectrum D: D=STFT(y(t),hop_length,n_fft) Among them, STFT represents short-time Fourier transform, hop_length represents the step length of window movement, and n_fft represents the number of FFT points; Compute the magnitude spectrum: Where Re(D) and Im(D) represent the real part and imaginary part of D respectively; Use a Mel filter to convert the linear spectrum to a Mel spectrum: S=M·|D| Where M represents the Mel filter matrix and S represents the Mel spectrum graph; Convert the Mel spectrum to the logarithmic power Mel spectrum and normalize it to get the final Mel spectrum: S log =20log 10 (max(S,ε)) Among them, ε is a constant; min_level_db represents the minimum value of the Mel spectrum.

5. The emotion recognition method based on visual language pre-training and multi-modal collaborative fusion according to claim 1 is characterized in that: The calculation formula of the binary cross entropy loss function described in S11 is: Among them, y represents the vector composed of the true sentiment labels of all samples in the batch, Represents a vector of predicted sentiment probabilities for all samples in a batch, and N represents the number of samples in each batch of data.

6. The emotion recognition method based on visual language pre-training and multi-modal collaborative fusion according to claim 1 is characterized in that: The calculation formula of the Sigmoid function described in S12 is: Among them, x represents the input feature and e represents the exponential operation.

Images