Model training method and device, face emotion recognition method and device, equipment and storage medium

Abstract

The invention provides a model training method and device, a face emotion recognition method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring an image sequence, a voice sequence and corresponding text information of a sample face video; emotion information is marked in the sample face video in advance, and the emotion information comprises at least two emotion categories and corresponding intensity information; performing feature extraction on the image sequence to obtain image features of the sample face video; performing feature extraction on the voice sequence to obtain voice features of the sample face video; performing feature extraction on the text information to obtain text features of the sample face video; and performing model training according to the image features, the voice features, the text features and the emotion information to obtain a face emotion recognition model. Compared with the prior art, the problems that emotion recognition has limitation and accuracy cannot be guaranteed are solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a model training, facial emotion recognition method, device, equipment and storage medium. Background Art

[0002] In today's society, human emotional behavior analysis is a rapidly developing field that has attracted much attention. This field is dedicated to analyzing and quantifying human emotions from various forms of data in order to explore human psychological states and behaviors. With the development of technologies such as artificial intelligence and machine learning, people have begun to focus on understanding and analyzing human emotional behavior through computer technology. The development of this technology has endowed machines with emotional intelligence, enabling them to better understand human emotional needs. At the same time, in the field of human-computer interaction (HCI), human emotional behavior analysis has also made significant contributions.

[0003] Sentiment analysis refers to the use of computers to automatically analyze and determine the emotions people want to express. In existing methods, emotions are mainly analyzed based on a modality in images or speech. Feature extraction algorithms are mainly based on image processing or mathematical statistics knowledge. By manually extracting and analyzing various features of images or other modal data, the required features are obtained to achieve recognition and analysis of data such as images, texts or speech. For example, in the field of images, commonly used traditional features include color features, texture features, edge features, shape features, etc.; in speech emotion analysis, traditional methods mainly analyze by extracting features such as rhythm and tone in audio.

[0004] However, when sentiment analysis relies solely on a single modal signal, only signals of a specific modality can be obtained, which has certain limitations in many cases, and the accuracy of emotion recognition cannot be guaranteed. Summary of the invention

[0005] The purpose of this application is to provide a model training, facial emotion recognition method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problem that emotion recognition in the prior art has limitations and its accuracy cannot be guaranteed.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for training a facial emotion recognition model, the method comprising:

[0008] Acquire an image sequence, a voice sequence and corresponding text information of a sample face video; the sample face video is pre-labeled with emotion information, and the emotion information includes: at least two emotion categories and corresponding intensity information;

[0009] Extracting features from the image sequence to obtain image features of the sample face video;

[0010] Extracting features from the speech sequence to obtain speech features of the sample face video;

[0011] Performing feature extraction on the text information to obtain text features of the sample face video;

[0012] Model training is performed based on the image features, the voice features, the text features and the emotion information to obtain a facial emotion recognition model.

[0013] In a second aspect, another embodiment of the present application provides a method for recognizing facial emotions, the method comprising:

[0014] Get the face video to be recognized;

[0015] A facial emotion recognition model is used to perform emotion recognition on the face video to be recognized, so as to obtain emotion information corresponding to the face video to be recognized; wherein the facial emotion recognition model is trained according to any one of the methods of the first aspect above.

[0016] In a third aspect, another embodiment of the present application provides a training device for a facial emotion recognition model, the device comprising: an acquisition module, a determination module and a training module, wherein:

[0017] The acquisition module is used to acquire an image sequence, a voice sequence and corresponding text information of a sample face video; the sample face video is pre-labeled with emotion information, and the emotion information includes: at least two emotion categories and corresponding intensity information;

[0018] The determination module is used to perform feature extraction on the image sequence to obtain image features of the sample face video; perform feature extraction on the voice sequence to obtain voice features of the sample face video; perform feature extraction on the text information to obtain text features of the sample face video;

[0019] The training module is used to perform model training based on the image features, the voice features, the text features and the emotion information to obtain a facial emotion recognition model.

[0020] In a fourth aspect, another embodiment of the present application provides a facial emotion recognition device, the device comprising: an acquisition module and a determination module, wherein:

[0021] The acquisition module is used to acquire the face video to be recognized;

[0022] The determination module is used to use a facial emotion recognition model to perform emotion recognition on the face video to be identified, and obtain emotion information corresponding to the face video to be identified; wherein the facial emotion recognition model is trained according to any one of the methods of the first aspect above.

[0023] In the fifth aspect, another embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the method described in any one of the first aspect or the second aspect above.

[0024] In the sixth aspect, another embodiment of the present application provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method described in the first aspect are executed.

[0025] The beneficial effect of the present application is: by adopting the training method of the facial emotion recognition model provided by the present application, after obtaining the image sequence, voice sequence and corresponding text information of the sample facial video, the image features, voice features and text features of the sample facial video are respectively extracted, and model training is performed based on the image features, voice features, text features and emotional information of the sample facial video to obtain the trained facial emotion recognition model; since the sample facial video is pre-labeled with at least two emotion categories and corresponding intensity information, the facial emotion recognition model trained in this way can not only recognize multiple emotions of the face during subsequent use, but also recognize the emotional intensity of each emotion, thereby providing a good data basis for subsequent facial emotion analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A flowchart of a method for training a facial emotion recognition model provided in one embodiment of the present application;

[0028] Figure 2A flowchart of a method for training a facial emotion recognition model provided by another embodiment of the present application;

[0029] Figure 3 A flowchart of a method for training a facial emotion recognition model provided by another embodiment of the present application;

[0030] Figure 4 A flowchart of a method for training a facial emotion recognition model provided by another embodiment of the present application;

[0031] Figure 5 A flowchart of a facial emotion recognition method provided in one embodiment of the present application;

[0032] Figure 6 A schematic diagram of the structure of a training device for a facial emotion recognition model provided in one embodiment of the present application;

[0033] Figure 7 A schematic diagram of the structure of a training device for a facial emotion recognition model provided by another embodiment of the present application;

[0034] Figure 8 A schematic diagram of the structure of a facial emotion recognition device provided in one embodiment of the present application;

[0035] Fig. 9 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0037] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0038] In addition, the flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of this application, and can also remove one or more operations from the flowchart.

[0039] The following is an explanation of a training method for a facial emotion recognition model provided in an embodiment of the present application in combination with a number of specific application examples. Figure 1 A flowchart of a method for training a facial emotion recognition model provided in one embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0040] S101: Obtain image sequences, voice sequences and corresponding text information of sample face videos.

[0041] The sample face video is pre-annotated with emotion information, and the emotion information includes: at least two emotion categories and corresponding intensity information; in some possible embodiments, the expression form of the emotion information corresponding to a sample face video A can be, for example, emotion categories: happiness, surprise, sadness, and the emotion intensities corresponding to the above three emotion categories are [1, 0.8, 0.1], that is, the facial emotions in the sample face video A include happiness, surprise and sadness, among which the emotion intensity of happiness is 1, the emotion intensity of surprise is 0.8, and the emotion intensity of sadness is 0.1; it should be understood that the above embodiment is only an exemplary description, and the content that may be included in the specific emotion category can be flexibly adjusted according to user needs, for example, it can also include anger, tension, rage, calmness, etc.; the number of emotion categories included can be two or three or more, and is not limited to the above embodiment.

[0042] For example, in an embodiment of the present application, before model training, emotional information of multiple sample facial videos will be annotated. In order to ensure the accuracy and consistency of the annotation, the emotional information of multiple facial videos can be annotated for a preset number of times to obtain the annotation results for each facial video, and multiple sample facial videos are generated based on the annotation results.

[0043] For example, if the preset number of times is 5 times, during the labeling process, different labelers are required to label the same face video with emotional information, and obtain the labeling results of the same face video by different labelers. If the consistency of the labeling results of the same face video by different labelers is greater than or equal to the preset threshold, the face video is considered to be a valid sample; for example, when the preset threshold is 80%, that is, if the labeling results of at least 4 out of 5 labelers are consistent, the face video is determined to be a valid sample, and multiple valid samples together constitute multiple sample face videos; it should be understood that the above embodiment is only an exemplary description, and the specific preset number of times and the preset threshold settings can be adjusted according to user needs, and are not limited to the above embodiment.

[0044] The multiple groups of sample data obtained in the above manner can reduce labeling errors and thus improve the quality of multiple sample face videos.

[0045] S102: Extract features from the image sequence to obtain image features of the sample face video.

[0046] In some possible embodiments, for example, feature extraction may be performed on an image sequence, and based on the order of images in the image sequence, a facial expression feature sequence corresponding to the image sequence may be obtained; or, in other possible embodiments, feature extraction may be performed on an image sequence to obtain a first temporal feature of the image sequence and facial expression features of each frame in the image sequence; and image features of a sample face video may be determined based on the facial expression features of each frame and the first temporal feature of the image sequence.

[0047] In some possible embodiments, the face pre-training model for feature extraction of image sequences may be, for example: a multi-task convolutional neural network (MTCNN), an unsupervised learning of deep face representations (FaceNet), a deep face model (Closing the Gap to Human-Level Performance in Face Verification, DeepFace), or a masked autoencoders (MAE), etc., wherein MTCNN is a multi-task convolutional neural network that can realize face detection and face recognition; FaceNet is a face recognition model based on a convolutional neural network that can extract the Euclidean distance of a face; DeepFace is a face recognition model based on deep learning that can extract a similarity score of a face. Any of the above models can be used as a preset face pre-training model to perform feature extraction on the image sequence to extract the face model features of each single frame image in the image sequence.

[0048] In some possible embodiments, the model for extracting the first temporal features of the image sequence can be, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a bidirectional gated regression module (BiGRU), a long short-term memory model (LSTM), or a gated recurrent unit (GRU) or other deep learning models to extract the temporal features of the image sequence.

[0049] For example, in an embodiment of the present application, the trained large-scale face pre-training model can be, for example, MAE, that is, the facial expression features of each single frame image in the image sequence are extracted by MAE. Then, the first temporal features of the image sequence are extracted based on BiGRU, and the image features of the sample face video are jointly determined based on the first temporal features and the facial expression features, and then the image features are reduced in dimension through a linear layer to obtain the image features of the processed sample face video.

[0050] S103: Extract features from the speech sequence to obtain speech features of the sample face video.

[0051] In some possible embodiments, for example, feature extraction may be performed on a speech sequence to obtain audio features and second time series features corresponding to the speech sequence; and speech features of a sample face video may be determined based on the audio features and second time series features of the speech sequence.

[0052] In some other possible embodiments, for example, the voice features of the sample face video may be determined based on text information and a voice sequence.

[0053] For example, in an embodiment of the present application, a pre-trained encoder can be used to extract audio features corresponding to a speech sequence, and a one-dimensional time series convolution and a natural language processing (transformer) model can be used to extract a second time series feature corresponding to the speech sequence; the speech features of the sample face video are jointly determined based on the audio features and the second time series features.

[0054] In some possible embodiments, for the above process of determining speech features, in order to further improve the accuracy of the extracted speech features, the present application adopts a self-supervised model based on joint training of speech sequences and text information to extract speech features from speech sequences.

[0055] S104: Extract features from the text information to obtain text features of the sample face video.

[0056] In some possible embodiments, segmentation words may be determined in the text information; and text features of the sample face video may be determined based on the segmentation words and context information of the segmentation words in the text information.

[0057] For example, in an embodiment of the present application, for example, the speech recognition tool Speech2Text based on transformer can be used to convert speech into text. Then, the large-scale text pre-training model DeBERTa ((Decoding-enhanced BERT with disentangled attention)) is used as an encoder to extract the sentence-breaking words in the text information, and then the one-dimensional temporal convolution is used to aggregate the contextual information of each sentence-breaking word in the text information to determine the text features of the sample face video.

[0058] In some possible embodiments, in order to improve the accuracy of text features, the present application also uses a transformer encoder to capture the timing features of text information, and further processes the above text features based on the timing features to obtain and output processed text features, wherein the processed text features are 64-dimensional text features; it should be understood that the above embodiments are only exemplary descriptions, and the selection of specific speech recognition tools can also be flexibly adjusted according to user needs, and is not limited to the above embodiments, for example, it can also be PaddleSpeech, DeepSpeech, whisper, etc.; the dimension of the text feature can also be 128 dimensions or 32 dimensions or any possible dimension and is not limited to the above embodiments.

[0059] S105: Perform model training based on image features, speech features, text features and emotion information to obtain a facial emotion recognition model.

[0060] In an embodiment of the present application, the method of performing model training may, for example, be: determining the predicted emotion information of a sample face video based on image features, speech features, and text features; determining a loss function based on the predicted emotion information and the emotion information of the sample face video; and performing model training based on the loss function and a preset loss threshold to obtain a facial emotion recognition model.

[0061] The facial emotion model trained by the above method can determine the facial emotions and emotion intensity in the video of the face to be identified by identifying the video of the face to be identified. This facial recognition can be applied to a variety of scenarios. For example, in the film, television, advertising and other industries, this technology can be used to evaluate the audience's emotional response to the content of the film, so as to better adjust the production and promotion strategies. It should be understood that the above embodiments are only exemplary descriptions, and the settings of specific application scenarios can be flexibly adjusted according to user needs, and are not limited to the above embodiments.

[0062] By adopting the training method of the facial emotion recognition model provided by the present application, after obtaining the image sequence, voice sequence and corresponding text information of the sample facial video, the image features, voice features and text features of the sample facial video are extracted respectively, and the model is trained based on the image features, voice features, text features and emotional information of the sample facial video to obtain the trained facial emotion recognition model. Since the sample facial video is pre-labeled with at least two emotion categories and corresponding intensity information, the facial emotion recognition model trained in this way can not only recognize multiple facial emotions during subsequent use, but also recognize the emotional intensity of each emotion, thereby providing a good data foundation for subsequent facial emotion analysis.

[0063] Optionally, based on the above embodiment, the embodiment of the present application may also provide a training method for a facial emotion recognition model. The implementation process of the above method is illustrated below with reference to the accompanying drawings. For example, in order to uniformly convert data of different lengths into data of fixed length, the embodiment of the present application also needs to obtain a target image sequence of a preset length in the image sequence before characterizing the image sequence. Figure 2 A flowchart of a method for training a facial emotion recognition model provided in another embodiment of the present application is shown in FIG. Figure 2 As shown, before S102, the method may further include:

[0064] S111: Divide the initial image sequence into sub-image sequences with a preset number of segments.

[0065] In some possible embodiments, in order to enhance the generalization of the network and be applicable to video data of various time lengths, the present application proposes a time series enhancement method, through which data of different lengths are uniformly converted into data of fixed length. First, if the length of the image sequence is T frames, the preset number of segments is K. The initial image sequence of length T frames is divided into K segments, each segment is a sub-image sequence, and an initial image sequence has a total of K sub-image sequences.

[0066] Such a segmentation method enables the image sequence to be split into K sub-image sequences when the number of frames of the initial image sequence is greater than K, regardless of the length of the initial image sequence.

[0067] S112: Randomly sample a frame of target image in each sub-image sequence.

[0068] This random sampling method based on the sub-image sequence can ensure the diversity of data, that is, a frame of image is randomly obtained in each sub-image sequence as the target image.

[0069] S113: Determine an image sequence of the sample face video according to the target image in each sub-image sequence.

[0070] The target images randomly obtained in each sub-image sequence are spliced, and the new sequence with a length of K after splicing is determined to be the image sequence of the sample face video. This setting makes the lengths of the corresponding image sequences of different sample face videos the same, that is, the initial image sequences of different lengths are converted into image sequences of fixed length, thereby enhancing the generalization of subsequent model training.

[0071] In the application process, even for the same facial expression, the speed (fast or slow) of the facial expression is different for different faces or different scenes. By constructing the image sequence of sample face videos in the above way, the final data can also have a certain degree of randomness and difference through the randomness of sampling. As a means of data enhancement, it can effectively increase the diversity of data, thereby improving the generalization of the model.

[0072] Optionally, based on the above embodiment, the embodiment of the present application may also provide a training method for a facial emotion recognition model, and the implementation process of the above method is illustrated below in conjunction with the accompanying drawings. For example, in some possible embodiments, in order to enhance the generalization of facial expression features and reduce the demand for labeled data for downstream tasks, the embodiment of the present application performs model pre-training based on multiple sample face images before S101 to obtain a face pre-training model. Figure 2 A flowchart of a method for training a facial emotion recognition model provided in another embodiment of the present application is shown in FIG. Figure 2 As shown, before S101, the method may further include:

[0073] S121: Divide each sample face image among the multiple sample face images into sub-blocks to obtain multiple sub-blocks of each sample face image.

[0074] In the embodiments of the present application, in order to enhance the generalization of facial expression features and reduce the demand for labeled data for downstream tasks, this solution first adopts a self-supervised learning method to perform representation learning based on a large number of unlabeled face images and train a Masked AutoEncoder (MAE). First, each face image in a plurality of sample face images will be divided into blocks. The block division method may be, for example, to divide each face image into blocks based on a preset number of blocks; or, to divide each face image into blocks according to the size of a preset sub-block. It should be understood that the above embodiments are only exemplary descriptions, and the specific method of dividing each face image into blocks can be flexibly adjusted according to user needs and is not limited to the above embodiments.

[0075] S122: Based on the multiple sub-blocks of the sample face image and a preset number of masks, perform random masking in the multiple sub-blocks.

[0076] For example, in an embodiment of the present application, for example, a random mask number or a random mask range value is preset. Taking the random mask range value as an example, first, a number is determined as the preset mask number within the random mask range value, and then, among multiple sub-blocks in the sample face image, a preset mask number of target sub-blocks are randomly determined, and the target sub-blocks are masked.

[0077] In some possible embodiments, the number of random masks can be generally determined according to the number of multiple sub-blocks in the sample face image, for example, it can be any percentage value such as 20%, 25%, 30%, 38% or 50^%, and is not limited to the above embodiments; the higher the percentage setting, the stronger the representativeness of the facial features subsequently determined by the face pre-training model, but too many masks may cause the subsequent sample face image to be unable to be reconstructed; therefore, in one embodiment of the present application, for example, the preset number of masks can be determined to be fifty percent of the number of multiple sub-blocks in the sample face image, that is, if the sample face image is divided into 16 sub-blocks, the preset number of masks is determined to be 8; it should be understood that the above embodiments are only exemplary illustrations, and the specific percentage settings can be flexibly adjusted according to user needs, for example.

[0078] S123: Obtain image features of the sample face image according to the masked sub-blocks.

[0079] The masked multiple target sub-blocks and the unmasked sub-blocks are input into the encoder together, and the encoder extracts the image features of the sample face image according to the input multiple target sub-blocks and the unmasked sub-blocks.

[0080] S124: Obtain a reconstructed face image based on the image features of the sample face image.

[0081] For example, the image features of the sample face image extracted by the encoder can be input into the decoder, so that the decoder reconstructs the sample face image according to the image features to obtain a reconstructed face image; as the face pre-training model is trained, the reconstructed face image will become closer and closer to the sample face image, that is, the similarity between the reconstructed face image and the sample face image will become higher and higher.

[0082] S125: Perform model pre-training based on the reconstructed face image and the sample face image to obtain a face pre-training model.

[0083] In some possible embodiments, the training method may be, for example, to train the face pre-training model according to the similarity between the reconstructed face image and the sample face image, and a preset similarity value, until the similarity between the reconstructed face image and the sample face image meets the preset similarity value.

[0084] Optionally, based on the above embodiment, the present application embodiment may also provide a method for training a facial emotion recognition model. The implementation process of the above method is illustrated below with reference to the accompanying drawings. Figure 4 A flowchart of a method for training a facial emotion recognition model provided in another embodiment of the present application is shown in FIG. Figure 4As shown, S105 may also include:

[0085] S131: Determine predicted emotion information of the sample face video based on image features, voice features, and text features.

[0086] The predicted emotion information includes multiple emotion categories and intensity information corresponding to each emotion category.

[0087] In the embodiment of the present application, in order to avoid the final extracted features and learned network from overfitting to a specific modality, the extracted image features, speech features and text features are fused. At the same time, in order to enhance the generalization of the model, two fully connected layers and an activation function are used to normalize the results of the intensity information of the predicted emotional information to 0-1, so as to obtain the intensity information of each emotional category in the final multi-emotion.

[0088] S132: Determine a loss function based on the predicted emotion information and the emotion information of the sample face video.

[0089] When inputting multimodal signals for emotion intensity estimation, the regression task uses the minimum mean square error (MSE) as the loss function. This regression loss trains the convolutional neural network with a larger learning rate at the beginning of training, and gradually reduces the learning rate as the network converges. The specific loss function L is determined MSE The method is:

[0090]

[0091] Where N is the batch size used during training (batch size refers to the number of training examples in each minibatch when training a neural network with backpropagation through an optimization algorithm based on stochastic gradient descent), and y i and They respectively represent the emotional information and predicted emotional information for the i-th sample face video.

[0092] S133: Perform model training according to the loss function and the preset loss threshold to obtain a facial emotion recognition model.

[0093] If the loss function is greater than a preset loss threshold, the model is iteratively trained until the loss function is less than or equal to the preset loss threshold, the model is determined to have converged, and the converged model is determined to be a facial emotion recognition model.

[0094] The training method of the facial emotion recognition model provided by the present application is adopted to obtain an action recognition method based on the perceptual coding of body movements in a continuous representation space. The model is pre-trained by multiple sample face images to obtain a face pre-training model, so that the generalization of facial expression features is effectively enhanced according to the face pre-training model, and more robust visual features are extracted. In addition, the present application increases the diversity of the initial image sequence by unifying the length of the initial image sequences of different lengths, and enhances the robustness of the subsequent extraction of temporal features, so that the method provided by the present application can adapt to initial image sequences of various lengths. In addition, the present application extracts the single-modal features of the three modalities of image features, voice features and text features of the sample face video, respectively, to prepare for subsequent multimodal fusion and prediction. By fusing multimodal information, the final result is prevented from being overfitted to a specific modal input. And a face emotion recognition model that can predict multiple emotional intensities based on multimodal signals is iteratively trained through a loss function and a preset loss threshold. This facial emotion recognition model can improve the stability and accuracy of emotion intensity estimation based on the highly generalized facial expression features and the time series features after time series enhancement, and has good application prospects.

[0095] Optionally, based on the above embodiment, the embodiment of the present application may also provide a facial emotion recognition method, and the implementation process of the above method is illustrated below in conjunction with the accompanying drawings. Figure 5 A flowchart of a facial emotion recognition method provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method may include:

[0096] S201: Obtain a video of a face to be recognized.

[0097] In some possible embodiments, the face video to be identified can be recorded, selected from a storage device, or directly crawled from the Internet. The specific method of obtaining the face video to be identified can be flexibly adjusted according to user needs and is not limited to the above embodiments.

[0098] S202: Using a facial emotion recognition model, perform emotion recognition on the face video to be recognized, and obtain emotion information corresponding to the face video to be recognized.

[0099] Among them, the facial emotion recognition model is based on the above Figure 1-Figure 4 trained by any method.

[0100] For example, in an embodiment of the present application, the emotion information corresponding to the face video to be identified may include: at least two emotion categories and corresponding intensity information.

[0101] The above-mentioned facial emotion recognition method is to use the above-mentioned Figure 1-Figure 4The method for facial emotion recognition using the trained facial emotion recognition model has the same beneficial effects as the above Figure 1-Figure 4 The beneficial effects brought by the method are the same, and this application will not repeat them here.

[0102] The following is an explanation of the training device for the facial emotion recognition model provided by the present application in conjunction with the accompanying drawings. The training device for the facial emotion recognition model can execute the above Figure 1-Figure 4 The specific implementation and beneficial effects of the training method of any facial emotion recognition model are as mentioned above and will not be repeated below.

[0103] Figure 6 A structural diagram of a training device for a facial emotion recognition model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device includes: an acquisition module 301, a determination module 302 and a training module 303, wherein:

[0104] The acquisition module 301 is used to acquire an image sequence, a voice sequence and corresponding text information of a sample face video; the sample face video is pre-labeled with emotion information, and the emotion information includes: at least two emotion categories and corresponding intensity information;

[0105] The determination module 302 is used to extract features from the image sequence to obtain image features of the sample face video; extract features from the voice sequence to obtain voice features of the sample face video; and extract features from the text information to obtain text features of the sample face video;

[0106] The training module 303 is used to perform model training based on image features, voice features, text features and emotion information to obtain a facial emotion recognition model.

[0107] Optionally, the acquisition module 301 is specifically configured to acquire an image sequence of a preset length in an initial image sequence of the sample face video.

[0108] Optionally, the determination module 302 is specifically used to divide the initial image sequence into sub-image sequences with a preset number of segments; randomly sample a frame of target image in each sub-image sequence; and determine the image sequence of the sample face video based on the target image in each sub-image sequence.

[0109] Optionally, the determination module 302 is specifically used to extract features from the image sequence to obtain a first temporal feature of the image sequence and a facial expression feature of each frame in the image sequence; and determine the image features of the sample face video based on the facial expression feature of each frame and the first temporal feature of the image sequence.

[0110] Optionally, the determination module 302 is specifically used to extract features from the speech sequence to obtain audio features and second time series features corresponding to the speech sequence; and determine the speech features of the sample face video based on the audio features and second time series features of the speech sequence.

[0111] Optionally, the determination module 302 is specifically configured to determine speech features of the sample face video according to the text information and the speech sequence.

[0112] Optionally, the determination module 302 is specifically configured to determine segmentation words in the text information; and determine text features of the sample face video according to the segmentation words and context information of the segmentation words in the text information.

[0113] Optionally, the training module 303 is specifically configured to perform model pre-training according to a plurality of sample face images to obtain a face pre-training model.

[0114] Optionally, based on the above embodiment, the present application embodiment may also provide a training device for a facial emotion recognition model, as shown in the following figure. Figure 6 The implementation process of the given device is described by way of example. Figure 7 A structural diagram of a training device for a facial emotion recognition model provided in another embodiment of the present application is shown in FIG. Figure 7 As shown, the device also includes: a mask module 304, wherein:

[0115] The determination module 302 is specifically used to divide each sample face image in the plurality of sample face images into sub-blocks to obtain a plurality of sub-blocks of each sample face image;

[0116] A masking module 304, configured to perform random masking in the plurality of sub-blocks based on the plurality of sub-blocks of the sample face image and a preset number of masks;

[0117] The determination module 302 is specifically used to obtain the image features of the sample face image according to the multiple masked sub-blocks; based on the image features of the sample face image, obtain the reconstructed face image;

[0118] The training module 303 is specifically used to perform model pre-training according to the reconstructed face image and the sample face image to obtain a face pre-training model.

[0119] Optionally, the determination module 302 is specifically used to determine the predicted emotion information of the sample face video according to the image features, the speech features and the text features; and determine the loss function according to the predicted emotion information and the emotion information of the sample face video;

[0120] The training module 303 is specifically used to perform model training according to the loss function and the preset loss threshold to obtain a facial emotion recognition model.

[0121] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0122] The following is an explanation of the facial emotion recognition device provided by the present application in conjunction with the accompanying drawings. The facial emotion recognition device can perform the above Figure 5 The facial emotion recognition method, its specific implementation and beneficial effects are as mentioned above and will not be repeated below.

[0123] Figure 8 A schematic diagram of the structure of a facial emotion recognition device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the device includes: an acquisition module 401 and a determination module 402, wherein:

[0124] The acquisition module 401 is used to acquire a video of a face to be recognized;

[0125] The determination module 402 is used to use a facial emotion recognition model to perform emotion recognition on the face video to be recognized, and obtain the emotion information corresponding to the face video to be recognized; wherein the facial emotion recognition model is based on the above Figure 1-Figure 4 trained by any method.

[0126] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0127] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0128] Fig. 9 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application, which electronic device can be integrated into a terminal device or a chip of the terminal device.

[0129] like Fig. 9 As shown, the electronic device includes: a processor 501 , a bus 502 and a storage medium 503 .

[0130] The processor 501 is used to store programs. The processor 501 calls the program stored in the storage medium 503 to execute the above Figure 1-Figure 5 The corresponding method embodiment has similar specific implementation methods and technical effects, which will not be described here.

[0131] Optionally, the present application also provides a program product, such as a storage medium, on which a computer program is stored, including a program that, when executed by a processor, executes an embodiment corresponding to the above method.

[0132] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0135] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

Claims

1. A training method for a facial emotion recognition model, characterized in that: The method comprises: Acquire an image sequence, a voice sequence and corresponding text information of a sample face video; the sample face video is pre-labeled with emotion information, and the emotion information includes: at least two emotion categories and corresponding intensity information; Extracting features from the image sequence to obtain image features of the sample face video; Extracting features from the speech sequence to obtain speech features of the sample face video; Performing feature extraction on the text information to obtain text features of the sample face video; Model training is performed based on the image features, the voice features, the text features and the emotion information to obtain a facial emotion recognition model.

2. The method according to claim 1, characterized in that Before extracting features from the image sequence to obtain image features of the sample face video, the method further includes: In the initial image sequence of the sample face video, an image sequence of a preset length is obtained.

3. The method according to claim 2, characterized in that The step of obtaining an image sequence of a preset length from an initial image sequence of a sample face video comprises: Dividing the initial image sequence into sub-image sequences with a preset number of segments; Randomly sampling a frame of target image in each of the sub-image sequences; According to the target image in each of the sub-image sequences, an image sequence of the sample face video is determined.

4. The method according to claim 1, characterized in that The extracting features of the image sequence to obtain image features of the sample face video includes: Performing feature extraction on the image sequence to obtain a first temporal feature of the image sequence and a facial expression feature of each frame of the image sequence; The image features of the sample face video are determined according to the facial expression features of each frame of the image and the first temporal features of the image sequence.

5. The method according to claim 1, characterized in that Extracting features from the speech sequence to obtain speech features of the sample face video includes: Perform feature extraction on the speech sequence to obtain audio features and second time series features corresponding to the speech sequence; The speech features of the sample face video are determined according to the audio features of the speech sequence and the second time series features.

6. The method according to claim 1, characterized in that The extracting features of the speech sequence to obtain speech features of the sample face video includes: The voice features of the sample face video are determined according to the text information and the voice sequence.

7. The method according to claim 1, characterized in that The extracting features of the text information to obtain text features of the sample face video includes: Determining sentence segmentation words in the text information; The text features of the sample face video are determined according to the segmentation words and context information of the segmentation words in the text information.

8. The method according to claim 1, characterized in that Before obtaining the image sequence, voice sequence and corresponding text information of the sample face video, the method further includes: Model pre-training is performed based on multiple sample face images to obtain a face pre-training model, wherein the face pre-training model is used to extract image features from an image sequence.

9. The method according to claim 8, characterized in that The method of performing model pre-training according to a plurality of sample face images to obtain a face pre-training model includes: Dividing each sample face image among the multiple sample face images into sub-blocks to obtain multiple sub-blocks of each sample face image; Based on the multiple sub-blocks of the sample face image and a preset number of masks, performing random masking in the multiple sub-blocks; Acquire image features of the sample face image according to the masked sub-blocks; Based on the image features of the sample face image, obtaining a reconstructed face image; Model pre-training is performed based on the reconstructed face image and the sample face image to obtain the face pre-training model.

10. The method according to claim 1, characterized in that The method of performing model training according to the image features, the voice features, the text features and the emotion information to obtain a facial emotion recognition model includes: Determining predicted emotion information of the sample face video according to the image features, the voice features, and the text features; Determining a loss function according to the predicted emotion information and the emotion information of the sample face video; Model training is performed according to the loss function and a preset loss threshold to obtain the facial emotion recognition model.

11. A method for recognizing facial emotions, characterized in that: The method comprises: Get the face video to be recognized; A facial emotion recognition model is used to perform emotion recognition on the face video to be recognized, so as to obtain emotion information corresponding to the face video to be recognized; wherein the facial emotion recognition model is trained according to any one of the methods of claims 1 to 10 above.

12. A training device for a facial emotion recognition model, characterized in that: The device comprises: an acquisition module, a determination module and a training module, wherein: The acquisition module is used to acquire an image sequence, a voice sequence and corresponding text information of a sample face video; the sample face video is pre-labeled with emotion information, and the emotion information includes: at least two emotion categories and corresponding intensity information; The determination module is used to perform feature extraction on the image sequence to obtain image features of the sample face video; perform feature extraction on the voice sequence to obtain voice features of the sample face video; perform feature extraction on the text information to obtain text features of the sample face video; The training module is used to perform model training based on the image features, the voice features, the text features and the emotion information to obtain a facial emotion recognition model.

13. A facial emotion recognition device, characterized in that: The device comprises: an acquisition module and a determination module, wherein: The acquisition module is used to acquire the face video to be recognized; The determination module is used to use a facial emotion recognition model to perform emotion recognition on the face video to be identified, and obtain emotion information corresponding to the face video to be identified; wherein the facial emotion recognition model is trained according to any one of the methods of claims 1-10 above.

14. An electronic device, characterized in that: The device includes: a processor, a storage medium and a bus, the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1 to 11.

15. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is executed.

Images