A video emotion recognition method based on multimodal representation learning

Abstract

The present invention belongs to the technical fields of artificial intelligence and multimodal emotion recognition, and specifically relates to a video emotion recognition method based on multimodal representation learning, the method comprising: converting a video with a human face into three modal feature representations, and aligning the three modal features; automatically encoding the three features; using an automatic encoding and decoding network to fuse the three features after the automatic encoding representation to obtain a feature joint representation; using the joint features as data input of an improved neural network to obtain an emotion classification result; the present invention firstly performs representation learning on the multimodal information in the video; combining the improved classic convolutional neural network LeNet‑5s with transformer, which can extract the advantages of temporal and spatial information, thereby extracting the accuracy of emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and multimodal emotion recognition, and in particular relates to a video emotion recognition method based on multimodal representation learning. Background Art

[0002] Facial recognition (FR) technology is a biometric recognition technology that uses facial physiological features to identify people. With the development of artificial intelligence (AI) technology, especially deep learning (DL) technology and the accumulation of big facial data, great progress has been made in identity recognition based on facial feature information. Facial expression is a universal language that is universal regardless of national boundaries, race and gender, and everyone has a universal expression. Ekman et al. first defined six basic expressions through cross-cultural research: anger, fear, disgust, happiness, sadness and surprise, and then added the expression of "contempt". This pioneering work and intuitive definition are still very popular today. Facial expression recognition (FER) based on FR has become one of the current research hotspots. FER is a multidisciplinary cross-disciplinary research involving psychology, physiology, biometric recognition, affective computing, artificial psychological theory and computer vision, and has a wide range of applications in disease diagnosis, human-computer interaction, distance education and safe driving.

[0003] Humans understand the world through multiple sensory organs such as vision, hearing, taste, smell and touch. Each source or form of information can be called a modality. When the research problem includes multiple such forms, the research problem is described as multimodal. For example, the media of human emotional information include images, voice, video, text and other multimodalities. For the research of emotion recognition, traditional methods mainly focus on single modalities such as face emotion recognition, voice emotion recognition, text emotion recognition, and EEG emotion recognition. Although these traditional single-modal emotion recognitions have made a lot of progress in various fields, human emotions are expressed in various forms. If we simply judge a person's emotions from one modality and one form of expression, we cannot fully determine a person's emotions, and usually lose a lot of important data. Single-modal emotion recognition still has low recognition rate, poor robustness, and low success rate in actual scenarios. In addition, the single-modal information is insufficient and easily affected by various external factors, such as facial expressions are easily blocked and voice is easily disturbed by noise. In order for artificial intelligence to make progress in understanding human emotions, this multimodal signal needs to be interpreted together, so the research on emotion recognition has slowly shifted from unimodal emotion recognition to bimodal emotion recognition and multimodal emotion recognition. This is a vibrant multidisciplinary cross-field with great significance and extraordinary potential.

[0004] The basis and premise of multimodal emotion recognition is to learn how to represent and summarize multimodal data in a way that exploits the complementarity and redundancy of multiple modalities. Due to the heterogeneity of multimodal emotion data, representing raw data in a format that can be used by computational models has always been a major challenge in machine learning. Union and coordination are currently the mainstream multimodal representation methods. The present invention first performs representation learning on multimodal information in the video; combining the improved classic convolutional neural network LeNet-5s and transformer, it can extract the advantages of temporal and spatial information, thereby improving the accuracy of emotion recognition. Summary of the invention

[0005] In order to solve the problems existing in the above prior art, the present invention proposes a video emotion recognition method based on multimodal representation learning, the method comprising: obtaining video data to be recognized, preprocessing the obtained video data, inputting the preprocessed video data into a trained improved neural network model, and obtaining an emotion recognition result of the video data;

[0006] The process of training the improved neural network model includes:

[0007] S1: Obtain an original video dataset and divide the original video dataset into a training set and a test set;

[0008] S2: Convert the data in the training set into face image modality features, speech modality features, and text modality features;

[0009] S3: Use the auto-encoding network to learn and reduce the dimensionality of facial image modal features, speech modal features, and text modal features;

[0010] S4: Using an automatic encoding and decoding network to jointly represent the face image modality features, speech modality features and text modality features after dimensionality reduction, and obtain joint features;

[0011] S5: Input the processed joint multimodal features into the improved LeNets-5-transformer model to obtain the emotion classification results;

[0012] S6: Calculate the loss function of the model according to the emotion classification results, continuously adjust the parameters of the model, and complete the model training when the loss function is minimized.

[0013] Preferably, the process of converting the data in the training set includes:

[0014] Perform face detection on video frames, capture face areas, normalize the size and grayscale; detect and locate faces in video frames, crop detected faces to a fixed size and grayscale them; select fixed frames from each video as a group of video sequences to obtain face image modal features;

[0015] The speech signal of the video frame is converted into a spectrogram, and the speech signal is processed by framing; the time domain of the speech signal after framing is converted into a spectrogram in the frequency domain to obtain the speech modal features;

[0016] Extract the text information of the video frame, that is, perform speech recognition on the speech signal in the video to obtain text data, input the text data into the pre-trained Glove word embeddings model to obtain a multi-dimensional vector, and thus obtain the text modality features.

[0017] Preferably, the automatic encoding network is used to perform learning and dimensionality reduction processing on facial image modal features, speech modal features and text modal features, including: the automatic encoder encodes the input multimodal feature x to obtain a new feature y, and the original input feature x can be reconstructed through the new feature y; the encoding process for generating the new feature y is: y = f(Wx + b), and the decoding process is: x′ = f(W′x + b′); the model is trained using the loss function L = -logP(x|x′) that minimizes the negative log-likelihood, and the reduced-dimensional features are obtained through dimensionality reduction.

[0018] Preferably, the process of jointly representing the reduced-dimensional facial image modal features, speech modal features, and text modal features using an automatic encoding and decoding network includes: aligning facial, speech, and text features through a word-level alignment strategy; and concatenating multi-modal features to obtain joint features.

[0019] Preferably, the improved LeNets-5-transformer model includes a transformer module, a LeNet-5s model and two filters; the LeNet-5s model consists of four convolutional layers, four pooling layers and three fully connected layers; on the basis of the LeNet-5 network model, two convolutional layers with 3*3 filters and two pooling layers with 2*2 filters are added, and then an L2 regularization function is added after each fully connected layer; the transformer module is set after the first convolutional layer of the LeNet-5s model.

[0020] Preferably, the improved LeNets-5-transformer model is used to process the input data, including: obtaining spatial information through a convolutional neural network, inputting the joint multimodal features into the transformer through an encoder, and learning the temporal information through an attention mechanism; inputting the spatial information and the temporal information into a decoder, and inputting the decoded information into a classifier to obtain an emotion classification result.

[0021] Preferably, the loss function expression of the model is:

[0022]

[0023] Among them, y i ' is the actual expression category label; y i The expression probability predicted for sample i of the current model training.

[0024] To achieve the above objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it implements any of the above-mentioned video emotion recognition methods based on multimodal representation learning.

[0025] To achieve the above-mentioned objectives, the present invention also provides a video emotion recognition device based on multimodal representation learning, comprising a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory, and is used to execute the computer program stored in the memory, so that the video emotion recognition device based on multimodal representation learning performs any of the above-mentioned video emotion recognition methods based on multimodal representation learning.

[0026] The present invention has the following beneficial effects:

[0027] 1. In view of the content information of the video itself, the video with faces is converted into video frames and processed into three modal feature representations: face visual modality, speech is converted into spectrogram auditory modality and language modality, and the three modal features are aligned; the three features are further represented by automatic encoding; and the automatic encoding and decoding network is further used for fusion to ensure that a more robust and effective joint feature representation is learned;

[0028] 2. The joint features are used as data input for the improved neural network (LeNet-5s) for feature learning and model training. The model is trained using a classic convolutional neural network with fewer layers, which reduces the difficulty of training the model.

[0029] 3. The introduction of transformer in the LeNet-5s model has far greater feature extraction capabilities than LSTM and recurrent network (RNN), solves the gradient explosion problem, reduces the difficulty of training the model, and can extract the advantages of temporal and spatial information, thereby improving the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a flowchart of a video emotion recognition method based on multimodal representation learning. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] A video emotion recognition method based on multimodal representation learning, the method comprising: acquiring video data to be recognized, preprocessing the acquired video data, inputting the preprocessed video data into a trained improved neural network model, and obtaining emotion recognition results of the video data.

[0033] like Figure 1 As shown, the process of training the improved neural network model includes:

[0034] S1: Obtain an original video dataset and divide the original video dataset into a training set and a test set;

[0035] S2: Convert the data in the training set into face image modality features, speech modality features, and text modality features;

[0036] S3: Use the auto-encoding network to learn and reduce the dimensionality of facial image modal features, speech modal features, and text modal features;

[0037] S4: Using an automatic encoding and decoding network to jointly represent the face image modality features, speech modality features and text modality features after dimensionality reduction, and obtain joint features;

[0038] S5: Input the processed joint multimodal features into the improved LeNets-5-transformer model to obtain the emotion classification results;

[0039] S6: Calculate the loss function of the model based on the emotion classification results, continuously adjust the model parameters, and complete the model training when the loss function is minimized

[0040] In this example, the IEMOCAP dataset is used. The dataset has corresponding sound and image data, and the video in the data is converted into video frames.

[0041] In this embodiment, MTCNN is used to locate the face of the video frame, and the detected face is cropped to a fixed size. Here, the image is reduced to 48*48 by resize() and the changed image is grayed out; the corresponding voice data is preprocessed, framed, and converted into a two-dimensional voice signal by short-time Fourier transform to realize the spectrogram. Through fusion, the face data and voice data of the video are combined into a data set that needs to be trained.

[0042] Use an autoencoder network to jointly represent the three modalities: for a "word" in a given text, use a separate network to model the facial image and spectrogram features in a field to calculate the visual and auditory embeddings; calculate the weighted average. For the joint data of the three modalities, first set the default weight ratios, 0.6 for facial image features, 0.2 for speech, and 0.2 for text data, and calculate the weighted average. Later, through fine-tuning of the model, change the weight ratio of the model data to jointly represent the multi-modal features.

[0043] The process of transforming the data in the training set includes:

[0044] Perform face detection on video frames, capture face areas, normalize the size and grayscale; detect and locate faces in video frames, crop detected faces to a fixed size and grayscale them; select fixed frames from each video as a group of video sequences to obtain face image modal features;

[0045] The speech signal of the video frame is converted into a spectrogram, and the speech signal is processed by framing; the time domain of the speech signal after framing is converted into a spectrogram in the frequency domain to obtain the speech modal features;

[0046] Extract the text information of the video frame, that is, perform speech recognition on the speech signal in the video to obtain text data, input the text data into the pre-trained Glove word embeddings model to obtain a multi-dimensional vector, and thus obtain the text modality features.

[0047] First, the spoken text is used as the benchmark, and the face video and speech sequence are aligned with it. The text sequence is represented as L, where the i-th word is represented as L (i) ,i=1,2,…,n; the face video sequence is represented as: V (i) =[v1 (i) ,v2 (i) ,…,v tvi (i) ]; the spectrogram sequence is represented as: A (i) =[a1 (i) ,a2 (i) ,…,a tai (i) ].

[0048] The improved neural network model includes a transformer module, a LeNet-5s model and two filters; the LeNet-5s model consists of four convolutional layers, four pooling layers and three fully connected layers; on the basis of the LeNet-5 network model, two convolutional layers with 3*3 filters and two pooling layers with 2*2 filters are added, and then an L2 regularization function is added after each fully connected layer to avoid overfitting due to a large number of training parameters; the transformer module is set after the first convolutional layer of the LeNet-5s model to construct a LeNet-5s-transformer model; finally, a softmax classifier is used in the fully connected layer to classify emotions.

[0049] The improved LeNets-5-transformer model is used to process the input data, including: obtaining spatial information through a convolutional neural network, inputting the joint multimodal features into the transformer through the encoder, and learning the temporal information through the attention mechanism; inputting the spatial information and temporal information into the decoder, and inputting the decoded information into the classifier to obtain the emotion classification result.

[0050] When building a neural network, the input joint vector feature and the actual label of the vector are cross entropy, and the cross entropy is used as the loss function of the model. The softmax function is used for classification and the loss is calculated to judge the effect of the model. The loss function expression of the model is:

[0051]

[0052] Among them, y i ' is the actual expression category label; y i The expression probability predicted for sample i of the current model training.

[0053] In one embodiment of the present invention, the present invention also includes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned video emotion recognition methods based on multimodal representation learning.

[0054] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0055] A video emotion recognition device based on multimodal representation learning comprises a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the video emotion recognition device based on multimodal representation learning performs any of the above-mentioned video emotion recognition methods based on multimodal representation learning.

[0056] Specifically, the memory includes: ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk, etc., which can store program codes.

[0057] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0058] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A video emotion recognition method based on multimodal representation learning, characterized in that: include: Acquire video data to be recognized, preprocess the acquired video data, input the preprocessed video data into the trained improved neural network model, and obtain emotion recognition results of the video data; The process of training the improved neural network model includes: S1: Obtain an original video dataset and divide the original video dataset into a training set and a test set; S2: Convert the data in the training set into face image modality features, speech modality features, and text modality features; S3: Use the auto-encoding network to learn and reduce the dimensionality of facial image modal features, speech modal features, and text modal features; S4: Using an automatic encoding and decoding network to jointly represent the face image modality features, speech modality features and text modality features after dimensionality reduction, and obtain joint features; S5: The processed joint multimodal features are input into the improved LeNets-5-transformer model to obtain the emotion classification results; the improved LeNets-5-transformer model includes a transformer module, a LeNet-5s model and two filters; the LeNet-5s model consists of four convolutional layers, four pooling layers and three fully connected layers; on the basis of the LeNet-5 network model, two convolutional layers with 3*3 filters and two pooling layers with 2*2 filters are added, and then an L2 regularization function is added after each fully connected layer; the transformer module is set after the first convolutional layer of the LeNet-5s model; S6: Calculate the loss function of the model according to the emotion classification results, continuously adjust the parameters of the model, and complete the model training when the loss function is minimized.

2. The video emotion recognition method based on multimodal representation learning according to claim 1, characterized in that: The process of transforming the data in the training set includes: Perform face detection on video frames, capture face areas, normalize the size and grayscale; detect and locate faces in video frames, crop detected faces to a fixed size and grayscale them; select fixed frames from each video as a group of video sequences to obtain face image modal features; The speech signal of the video frame is converted into a spectrogram, and the speech signal is processed by framing; the time domain of the speech signal after framing is converted into a spectrogram in the frequency domain to obtain the speech modal features; Extract the text information of the video frame, that is, perform speech recognition on the speech signal in the video to obtain text data, input the text data into the pre-trained Glove word embeddings model to obtain a multi-dimensional vector, and thus obtain the text modality features.

3. The video emotion recognition method based on multimodal representation learning according to claim 1, characterized in that: The automatic encoding network is used to perform learning and dimensionality reduction processing on facial image modal features, speech modal features and text modal features, including: the automatic encoder encodes the input multimodal feature x to obtain a new feature y, and the original input feature x can be reconstructed through the new feature y; the encoding process of generating the new feature y is: y=f(Wx+b), and the decoding process is: x′=f(W'x+b′); the model is trained using the loss function L=-logP(x|x′) that minimizes the negative log-likelihood, and the reduced-dimensional features are obtained through dimensionality reduction.

4. The video emotion recognition method based on multimodal representation learning according to claim 1, characterized in that: The process of jointly representing the reduced-dimensional facial image modal features, speech modal features, and text modal features using an automatic encoding and decoding network includes: aligning facial, speech, and text features through a word-level alignment strategy; and concatenating multi-modal features to obtain joint features.

5. The video emotion recognition method based on multimodal representation learning according to claim 1, characterized in that: The improved LeNets-5-transformer model is used to process the input data, including: obtaining spatial information through a convolutional neural network, inputting the joint multimodal features into the transformer through the encoder, and learning the temporal information through the attention mechanism; inputting the spatial information and temporal information into the decoder, and inputting the decoded information into the classifier to obtain the emotion classification result.

6. The video emotion recognition method based on multimodal representation learning according to claim 1, characterized in that: The process of calculating the loss function of the model includes taking the cross entropy of the input joint vector feature and the actual label of the vector, and using the cross entropy as the loss function of the model; the loss function expression of the model is: Among them, y i ′ is the actual expression category label; y i The expression probability predicted for sample i of the current model training.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the video emotion recognition method based on multimodal representation learning according to any one of claims 1 to 6.

8. A video emotion recognition device based on multimodal representation learning, characterized in that: It includes a processor and a memory; the memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the video emotion recognition device based on multimodal representation learning executes any one of the video emotion recognition methods based on multimodal representation learning in claims 1 to 6.

Images