An emotion recognition method, device, storage medium and equipment

Abstract

The present application discloses an emotion recognition method, apparatus, storage medium and device. The method includes: first, obtaining a target video to be recognized, and separating an audio signal stream and an image signal stream therefrom; then, respectively inputting the audio signal stream and the image signal stream into a pre-constructed emotion recognition model to predict a first preliminary emotion recognition result and a second preliminary emotion recognition result corresponding to a target user in the target video; wherein, the emotion recognition model is obtained by alternately training using a sample audio signal stream and a sample image signal stream, and then training by means of a parameter update method of gradient fusion; then, determining a final emotion recognition result of the target user according to the first preliminary emotion recognition result and the second preliminary emotion recognition result. Since the present application first constructs an emotion recognition model by using a training method related to modalities and a parameter update method of gradient fusion, the accuracy of the recognition result can be improved when using this model for emotion recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an emotion recognition method, apparatus, storage medium, and device. Background Art

[0002] With the rapid development of artificial intelligence (AI), human-computer interaction (HCI) is increasingly common in our daily lives and work, bringing significant convenience. As a key branch of HCI, emotion recognition holds significant significance in fields such as healthcare and education, and has become a research hotspot across various fields.

[0003] Currently, existing emotion recognition methods typically employ multimodal approaches based on deep learning. However, these approaches utilize multi-task learning for model training. Emotional representation vectors abstracted from different modalities are merged through shared fusion. This training approach presupposes the complete sharing of abstract representation information across modalities. If the model network structure cannot aggregate a matching high-dimensional representation, then employing multi-task learning training methods will result in model bias and low recognition accuracy. Therefore, improving the accuracy of emotion recognition results is a pressing technical challenge. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to provide an emotion recognition method, apparatus, storage medium and device that can effectively improve the accuracy of emotion recognition results.

[0005] The present invention provides an emotion recognition method, including:

[0006] Acquire a target video to be identified, and separate an audio signal stream and an image signal stream from the target video;

[0007] Inputting the audio signal stream and the image signal stream into a pre-built emotion recognition model respectively to predict a first preliminary emotion recognition result and a second preliminary emotion recognition result corresponding to a target user in the target video; the emotion recognition model is trained by alternating training using sample audio signal streams and sample image signal streams, and then updating parameters using gradient fusion;

[0008] A final emotion recognition result corresponding to the target user is determined according to the first preliminary emotion recognition result and the second preliminary emotion recognition result.

[0009] In one possible implementation, the emotion recognition model is constructed as follows:

[0010] Obtain a first sample video, and separate a first sample audio signal stream and a first sample image signal stream from the first sample video;

[0011] Input the first sample audio signal stream into an initial emotion recognition model, calculate a first gradient of model parameter update by adjusting a first loss function, and train to obtain a first preliminary emotion recognition model;

[0012] Input the first sample image signal stream into the first preliminary emotion recognition model, calculate a second gradient of model parameter update by adjusting a second loss function, and train to obtain a second preliminary emotion recognition model;

[0013] Perform weighted summation calculation on the first gradient and the second gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain the trained emotion recognition model.

[0014] In a possible implementation manner, the step of inputting the first sample audio signal stream into an initial emotion recognition model, calculating a first gradient of model parameter update by adjusting a first loss function, and training to obtain a first preliminary emotion recognition model includes:

[0015] Slice the first sample audio signal stream into audio signal segments according to a preset fixed time window length, and pack all the audio signal segments into an audio training data block;

[0016] Select N audio signal segments from the audio training data block, and extract the acoustic features corresponding to each of the N audio signal segments; N is a positive integer greater than 0;

[0017] Input the acoustic features corresponding to each of the N audio signal segments into a multi-layer convolution module, a residual module, a pooling module, and a linear layer of the initial emotion recognition model for emotion prediction to obtain a first emotion prediction result of a first sample user in the first sample video;

[0018] Calculate a first loss function by comparing the first emotion prediction result of the first sample user with the true emotion of the first sample user;

[0019] According to the first loss function, calculate a first gradient of model parameter update through the error backpropagation algorithm, and train to obtain a first preliminary emotion recognition model.

[0020] In a possible implementation manner, the step of inputting the first sample image signal stream into the first preliminary emotion recognition model, calculating a second gradient of model parameter update by adjusting a second loss function, and training to obtain a second preliminary emotion recognition model includes:

[0021] Segment the first sample image signal stream into image signal segments according to the preset fixed time window length, and pack all the image signal segments into an image training data block;

[0022] Select N image signal segments corresponding to the N audio signal segments in time from the image training data block, and extract the image features corresponding to each of the N image signal segments;

[0023] Input the image features corresponding to each of the N image signal segments into the multi-layer convolution module, residual module, pooling module and linear layer of the first preliminary emotion recognition model for emotion prediction, and obtain the second emotion prediction result of the first sample user in the first sample video;

[0024] Calculate the second loss function by comparing the second emotion prediction result of the first sample user with the true emotion of the first sample user;

[0025] According to the second loss function, calculate the second gradient of the model parameter update through the error backpropagation algorithm, and train to obtain the second preliminary emotion recognition model.

[0026] In a possible implementation, both the first loss function and the second loss function are cross-entropy loss functions.

[0027] In a possible implementation, the emotion recognition model is constructed as follows:

[0028] Obtain a second sample video, and separate a second sample audio signal stream and a second sample image signal stream from the second sample video;

[0029] Input the second sample image signal stream into the initial emotion recognition model, calculate the third gradient of the model parameter update by adjusting the third loss function, and train to obtain the third preliminary emotion recognition model;

[0030] Input the second sample audio signal stream into the third preliminary emotion recognition model, calculate the fourth gradient of the model parameter update by adjusting the fourth loss function, and train to obtain the fourth preliminary emotion recognition model;

[0031] Perform a weighted sum calculation on the third gradient and the fourth gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain the trained emotion recognition model.

[0032] In a possible implementation, the initial emotion recognition model is a convolutional neural network CNN.

[0033] In a possible implementation, the method further includes:

[0034] Obtain a verification video, and separate a verification audio signal stream and a verification image signal stream from the verification video;

[0035] Input the verification audio signal stream and the verification image signal stream into the emotion recognition model respectively, and obtain a first emotion verification result and a second emotion verification result corresponding to the verified user in the verification video;

[0036] Determine a final emotion verification result corresponding to the verified user according to the first emotion verification result and the second emotion verification result;

[0037] When the final emotion verification result corresponding to the verified user is inconsistent with the true emotion, use the verification video as the first sample video or the second sample video again to update the emotion recognition model.

[0038] An embodiment of the present application further provides an emotion recognition device, including:

[0039] A first acquisition unit, configured to acquire a target video to be recognized, and separate an audio signal stream and an image signal stream from the target video;

[0040] A prediction unit, configured to input the audio signal stream and the image signal stream into a pre-constructed emotion recognition model respectively, and predict a first preliminary emotion recognition result and a second preliminary emotion recognition result corresponding to the target user in the target video; the emotion recognition model is obtained by alternately training with a sample audio signal stream and a sample image signal stream, and then training by a parameter update method of gradient fusion;

[0041] A first determination unit, configured to determine a final emotion recognition result corresponding to the target user according to the first preliminary emotion recognition result and the second preliminary emotion recognition result.

[0042] In a possible implementation manner, the device further includes:

[0043] A second acquisition unit, configured to acquire a first sample video, and separate a first sample audio signal stream and a first sample image signal stream from the first sample video;

[0044] A first training unit, configured to input the first sample audio signal stream into an initial emotion recognition model, calculate a first gradient of model parameter update by adjusting a first loss function, and train to obtain a first preliminary emotion recognition model;

[0045] A second training unit, configured to input the first sample image signal stream into the first preliminary emotion recognition model, calculate a second gradient of model parameter update by adjusting a second loss function, and train to obtain a second preliminary emotion recognition model;

[0046] A first update unit, configured to perform weighted summation calculation on the first gradient and the second gradient, and update parameters of the initial emotion recognition model by using the obtained calculation result, so as to obtain the trained emotion recognition model.

[0047] In a possible implementation manner, the first training unit includes:

[0048] A first segmentation sub-unit, configured to segment the first sample audio signal stream into audio signal segments according to a preset fixed time window length, and pack all the audio signal segments into an audio training data block;

[0049] A first extraction sub-unit, configured to select N audio signal segments from the audio training data block, and extract acoustic features corresponding to the N audio signal segments respectively; N is a positive integer greater than 0;

[0050] A first prediction sub-unit, configured to input the acoustic features corresponding to the N audio signal segments respectively into a multi-layer convolution module, a residual module, a pooling module and a linear layer of the initial emotion recognition model for emotion prediction, so as to obtain a first emotion prediction result of a first sample user in the first sample video;

[0051] A first calculation sub-unit, configured to calculate a first loss function by comparing the first emotion prediction result of the first sample user with the true emotion of the first sample user;

[0052] A first training sub-unit, configured to calculate a first gradient of model parameter update according to the first loss function by using an error backpropagation algorithm, and train to obtain a first preliminary emotion recognition model.

[0053] In a possible implementation manner, the second training unit includes:

[0054] A second segmentation sub-unit, configured to segment the first sample image signal stream into image signal segments according to the preset fixed time window length, and pack all the image signal segments into an image training data block;

[0055] A second extraction sub-unit, configured to select N image signal segments corresponding to the N audio signal segments in time from the image training data block, and extract image features corresponding to the N image signal segments respectively;

[0056] A second prediction sub-unit, configured to input the image features corresponding to the N image signal segments respectively into the multi-layer convolution module, the residual module, the pooling module and the linear layer of the first preliminary emotion recognition model for emotion prediction, so as to obtain a second emotion prediction result of the first sample user in the first sample video;

[0057] A second computing subunit, configured to calculate a second loss function by comparing the second emotion prediction result of the first sample user with the true emotion of the first sample user;

[0058] A second training subunit, configured to calculate a second gradient for updating model parameters through the error backpropagation algorithm according to the second loss function, and train to obtain a second preliminary emotion recognition model.

[0059] In a possible implementation, both the first loss function and the second loss function are cross-entropy loss functions.

[0060] In a possible implementation, the apparatus further includes:

[0061] A third acquisition unit, configured to acquire a second sample video, and separate a second sample audio signal stream and a second sample image signal stream from the second sample video;

[0062] A third training unit, configured to input the second sample image signal stream into the initial emotion recognition model, calculate a third gradient for updating model parameters by adjusting a third loss function, and train to obtain a third preliminary emotion recognition model;

[0063] A fourth training unit, configured to input the second sample audio signal stream into the third preliminary emotion recognition model, calculate a fourth gradient for updating model parameters by adjusting a fourth loss function, and train to obtain a fourth preliminary emotion recognition model;

[0064] A second update unit, configured to perform a weighted sum calculation on the third gradient and the fourth gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain the trained emotion recognition model.

[0065] In a possible implementation, the initial emotion recognition model is a convolutional neural network CNN.

[0066] In a possible implementation, the apparatus further includes:

[0067] A third acquisition unit, configured to acquire a verification video, and separate a verification audio signal stream and a verification image signal stream from the verification video;

[0068] An input unit, configured to respectively input the verification audio signal stream and the verification image signal stream into the emotion recognition model to obtain a first emotion verification result and a second emotion verification result corresponding to a verification user in the verification video;

[0069] A second determination unit, configured to determine a final emotion verification result corresponding to the verification user according to the first emotion verification result and the second emotion verification result;

[0070] A third update unit, configured to, when the final emotion verification result corresponding to the verified user is inconsistent with the true emotion, use the verified video as the first sample video or the second sample video again to update the emotion recognition model.

[0071] An embodiment of the present application further provides an emotion recognition device, including: a processor, a memory, and a system bus;

[0072] The processor and the memory are connected through the system bus;

[0073] The memory is configured to store one or more programs, and the one or more programs include instructions that, when executed by the processor, cause the processor to execute any implementation manner of the above emotion recognition method.

[0074] An embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored, and when the instructions are run on a terminal device, the terminal device is caused to execute any implementation manner of the above emotion recognition method.

[0075] An embodiment of the present application further provides a computer program product, and when the computer program product runs on a terminal device, the terminal device is caused to execute any implementation manner of the above emotion recognition method.

[0076] An emotion recognition method, device, storage medium, and device provided by an embodiment of the present application first obtain a target video to be recognized, and separate an audio signal stream and an image signal stream from the target video, and then input the audio signal stream and the image signal stream into a pre-constructed emotion recognition model respectively to predict a first preliminary emotion recognition result and a second preliminary emotion recognition result corresponding to a target user in the target video; wherein, the emotion recognition model is obtained by alternately training using a sample audio signal stream and a sample image signal stream, and then training by a parameter update method of gradient fusion; then, a final emotion recognition result corresponding to the target user can be determined according to the first preliminary emotion recognition result and the second preliminary emotion recognition result. It can be seen that since the present application first alternately trains using a sample audio signal stream and a sample image signal stream, and then constructs an emotion recognition model by a parameter update method of gradient fusion, the recognition effect of the model is effectively improved, so that when using the emotion recognition model to recognize the emotion of the target user in the target video, the accuracy of the recognition result can be effectively improved. Description of the Drawings

[0077] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0078] Figure 1 It is a schematic flowchart of an emotion recognition method provided by an embodiment of the present application;

[0079] Figure 2 It is a schematic diagram of the process of constructing an emotion recognition model provided by an embodiment of the present application;

[0080] Figure 3 It is a schematic diagram of the composition of an emotion recognition device provided by an embodiment of the present application. Detailed implementation manners

[0081] Emotion is an internal subjective experience and is the physiological and psychological reaction of people to external factors or self-stimuli. Emotion recognition is of great significance in fields such as medical treatment and education. As an important branch in human-computer interaction, emotion recognition has become a research hotspot in current various fields.

[0082] Traditional emotion recognition methods usually focus on single-modal fields such as speech, facial expressions, electroencephalogram signals, and text. Among them, facial expression recognition, as an important development direction in the field of emotion recognition, has made great progress. Most traditional facial expression recognition methods use manually designed features or shallow learning features. The recognition process mainly includes image acquisition, preprocessing, feature extraction, and classification. The preprocessing part usually includes techniques such as data augmentation, face recognition, and normalization. And speech emotion recognition has also developed rapidly with the continuous enrichment of datasets. As the most direct means of people's communication, speech contains rich emotion information, and the changes in people's emotions can be reflected through speech features. Speech emotion recognition is exactly to convert the input speech signal containing emotion information into readable physical features, extract the speech features related to emotion expression, then construct an emotion recognition classifier for testing and training, and finally output the emotion recognition classification result.

[0083] However, the disadvantage of single-modal emotion recognition methods is that the recognition basis is relatively single and the recognition accuracy is relatively low. Therefore, in most cases, multi-modal emotion recognition systems are far more accurate than the corresponding optimal single-modal emotion recognition systems and are also more widely applied. This is because by simultaneously using multi-modal information such as speech and facial expressions, the complementary relationship of the same emotion among different modalities can be fully exploited, the generalization ability of the emotion recognition system can be improved, and thus a relatively high emotion recognition accuracy can be achieved.

[0084] In recent years, the most commonly used emotion recognition method is a multimodal emotion recognition approach based on deep learning. Deep learning methods, among other things, do not require extensive domain expertise or a definition of the physical meaning of the extracted feature parameters. Instead, they primarily extract features and identify corresponding emotion categories through the design of a neural network architecture. This emotion category recognition approach is an end-to-end recognition method, establishing a mathematical model and algorithm for the neural network architecture and training the connection weight parameters, enabling the network to perform data-based pattern recognition, function mapping, and other functions. Utilizing training samples labeled with emotion category information and continuous model training, the model can deeply mine the subtle emotional essence contained in speech and images. Deep learning methods for emotion recognition do not require the development of sophisticated feature parameter extraction methods, reducing the reliance on prior knowledge and lowering the research threshold. Therefore, they have become a leading technical solution in the field of emotion recognition.

[0085] Specifically, deep learning-based emotion recognition methods typically employ a multi-task learning network structure, feeding input signals from multiple modalities into corresponding neural networks, such as time-delay neural networks (TDNNs) and recurrent neural networks (RNNs). Through multi-layer information connections and high-level information sharing, the multimodal input signals are abstractly fused together to uniformly predict the corresponding emotion category. However, the model training method of this recognition method presupposes the complete sharing of abstract representation information between different modalities. If the model network structure cannot aggregate a matching high-dimensional information representation, fails to consider prior information between modalities, and fails to utilize complementary information between models, then the use of multi-task learning training methods will result in model bias, leading to inaccurate recognition results. Therefore, improving the accuracy of emotion recognition results is a pressing technical issue.

[0086] To address the above deficiencies, the present application provides an emotion recognition method. First, a target video to be recognized is obtained, and an audio signal stream and an image signal stream are separated from the target video. Then, the audio signal stream and the image signal stream are respectively input into a pre-constructed emotion recognition model to predict the first preliminary emotion recognition result and the second preliminary emotion recognition result corresponding to the target user in the target video. Among them, the emotion recognition model is trained by alternately training with sample audio signal streams and sample image signal streams and then updating the parameters through gradient fusion. Next, the final emotion recognition result corresponding to the target user can be determined according to the first preliminary emotion recognition result and the second preliminary emotion recognition result. It can be seen that since the present application first alternately trains with sample audio signal streams and sample image signal streams and then constructs an emotion recognition model through the parameter update method of gradient fusion, the recognition effect of the model is effectively improved. Therefore, when using this emotion recognition model to recognize the emotion of the target user in the target video, the accuracy of the recognition result can be effectively improved.

[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0088] First Embodiment

[0089] See Figure 1 , which is a schematic flowchart of an emotion recognition method provided in this embodiment. The method includes the following steps:

[0090] S101: Obtain a target video to be recognized, and separate an audio signal stream and an image signal stream from the target video.

[0091] In this embodiment, any video that needs to perform user emotion recognition is defined as the target video. And the user who needs to perform emotion recognition included in the target video is defined as the target user. It should be noted that this embodiment does not limit the color type of the target video. For example, the target video can be a color video or a grayscale video, etc. And this embodiment does not limit the format type of the target video. For example, the target video can be in video formats such as mp4 or wmv (Windows Media Video). And the present application also does not limit the scene type of the target video. For example, the target video can be a film and television video clip, or it can also be a short video in the teaching field, etc.

[0092] It can be understood that the target video can be obtained by means such as shooting with a camera according to actual needs. For example, the teaching video of a teacher captured by a camera installed in a classroom, or a segment of a video stream containing at least one user can be used as the target video.

[0093] Furthermore, after obtaining the target video, existing or future video stream separation methods can be used to perform separation processing on the modal signals of the target video. For example, the open-source computer program FFmpeg (Fast Forward Mpeg) can be used to perform separation processing on the modal signals of the target video, so as to separately extract the audio signal stream and the image signal stream contained in the target video, and use them to execute the subsequent steps S102 - S103 to achieve emotion recognition of the target user in the target video and obtain a more accurate recognition result.

[0094] S102: Input the audio signal stream and the image signal stream into a pre-constructed emotion recognition model respectively, and predict the first preliminary emotion recognition result and the second preliminary emotion recognition result corresponding to the target user in the target video; among them, the emotion recognition model is trained by alternately training with sample audio signal streams and sample image signal streams and then updating the parameters through gradient fusion.

[0095] In this embodiment, after obtaining the target video to be recognized through step S101 and separating the audio signal stream and the image signal stream from it, in order to accurately identify the emotion category of the target user in the target video, further, the audio signal stream and the image signal stream can be input into a pre-constructed emotion recognition model respectively, so as to predict the first preliminary emotion recognition result and the second preliminary emotion recognition result corresponding to the target user in the target video, and define them as s1 and s2 respectively, for use in the subsequent step S103.

[0096] Among them, it should be noted that in order to accurately identify the emotion category of the target user in the target video, an emotion recognition model is pre-constructed in this embodiment. This model is obtained by alternately training using sample audio signal streams and sample image signal streams respectively, and then training through a parameter update method of gradient fusion. In this way, it is possible to avoid the prerequisite of sharing high-dimensional abstract emotion representation vectors of audio and images in the existing multi-task learning training method, ensuring the training effect of each single-modal system in the multi-modal. And, considering the characteristic that the audio and image modal signals exist in pairs, the parameter update gradients for the audio training data and the image training data are calculated respectively, and the gradients calculated separately in the audio and image signal domains are fused at the parameter update gradient level, thereby avoiding the problem of cross-audio and image signal domains in gradient calculation, and further improving the generalization ability of the model. When using this emotion recognition model to identify the emotion of the target user in the target video subsequently, the accuracy of the recognition result can be effectively improved.

[0097] Next, this embodiment will introduce the construction process of the emotion recognition model, as Figure 2 shown, which shows a schematic diagram of the process of constructing an emotion recognition model provided in this embodiment. The process includes the following steps A1 - A4:

[0098] Step A1: Obtain a first sample video, and separate a first sample audio signal stream and a first sample image signal stream from the first sample video.

[0099] In this embodiment, in order to construct an emotion recognition model, a large amount of preparatory work needs to be done in advance. First, a large number of videos containing the speech and images of users speaking need to be collected. For example, it can be through microphone array sound pickup and camera shooting, etc. The sound pickup device can be a tablet computer, or intelligent hardware devices such as smart speakers, TVs, and air conditioners, and each piece of video data collected can be used as the first sample video.

[0100] Moreover, after obtaining the first sample video, it cannot be directly used to train and generate an emotion recognition model. Instead, a method similar to that in step S101 of separating the audio signal stream and the image signal stream from the target video is needed. For example, using the open-source computer program FFmpeg, replacing the target video with the first sample video, the first sample audio signal stream and the first sample image signal stream contained in each first sample video can be extracted to perform the subsequent step A2.

[0101] Step A2: Input the first sample audio signal stream into the initial emotion recognition model, calculate the first gradient of the model parameter update by adjusting the first loss function, and train to obtain a first preliminary emotion recognition model.

[0102] Among them, an optional implementation is that the initial emotion recognition model can be (but not limited to) a convolutional neural network (Convolutional Neural Networks, abbreviated as CNN) including multiple convolutional modules, residual modules, pooling modules, and a fully connected layer. The reason for preferably choosing CNN as the initial emotion recognition model is that modules such as the local receptive field, weight sharing, and pooling in CNN can greatly reduce the scale of the network structure and largely alleviate the overfitting problem that occurs due to the large scale of the network model. The convolutional neural network generated by randomly initializing the parameters is used to predict the emotion category of the input sample multimodal signals such as audio and images. The error between the predicted value and the labeled value of the true emotion category is used to drive the parameter update of the convolutional neural network. After several rounds of model parameter updates, model parameters with a relatively high emotion recognition accuracy are obtained, and the model training can be completed.

[0103] Specifically, in this embodiment, after obtaining multiple first sample videos through step A1 and separating the first sample audio signal stream and the first sample image signal stream from each first sample video, in order to improve the recognition effect of the model, further, for each first sample audio signal stream separated from the first sample video, first, it can be segmented into each audio signal segment according to a preset fixed time window length (the specific value is not limited and can be set according to the actual situation and empirical values. For example, it can be set to 5 seconds). If the duration of the first sample audio signal stream is insufficient, it can be copied multiple times and spliced, and then segmented according to the preset fixed time window length, and the obtained audio signal segments are packaged into an audio training data block corresponding to the first sample video and defined as D audio .

[0104] Then, a batch of N (N is a positive integer greater than 0) audio signal segments can be randomly selected from the audio training data block D audio For example Figure 2As shown, the acoustic features corresponding to each of these N audio signal segments are extracted. Among them, when extracting the acoustic features corresponding to each of these N audio signal segments, first, each audio signal segment needs to be framed to obtain a corresponding sequence of speech frames, and then the framed sequence of speech frames is pre-emphasized; then, the acoustic features of each speech frame are sequentially extracted. Here, the acoustic feature refers to the feature data used to characterize the acoustic information of the corresponding speech frame, and its dimension is defined as d. The specific content format and dimension value are not limited. A preferred implementation is that the acoustic features corresponding to each of these N audio signal segments can be, but are not limited to, Mel-scale Frequency Cepstral Coefficients (MFCC) features or filterbank features, etc.; the dimension d can be 64 dimensions, etc.

[0105] It should be noted that the embodiments of the present application do not limit the extraction method of the acoustic features corresponding to each of these N audio signal segments, nor the specific extraction process. An appropriate extraction method can be selected according to the actual situation, and corresponding feature extraction operations can be performed.

[0106] Next, the acoustic features corresponding to each of these N audio signal segments can be input into the multi-layer convolutional module, residual module, pooling module, and linear layer of the initial emotion recognition model (such as CNN) for emotion prediction, to determine the probability value corresponding to each preset emotion category of the first sample user in the first sample video, and it is defined as q audio,n , the greater the probability value, the higher the degree that the first sample user belongs to the corresponding preset emotion category (the emotion category corresponding to the probability value), and vice versa. Thus, the first emotion prediction result of the first sample user in the first sample video can be obtained.

[0107] For example, assume that the preset emotion categories are: "joy", "trust", "fear", "surprise", "sadness", "disgust", "anger", "expectation". The first emotion prediction result of the first sample user output by the model is an 8-dimensional emotion category prediction vector [0.14, 0.05, 0.03, 0.31, 0.05, 0.05, 0.32, 0.05]. Among them, each vector value represents the probability value corresponding to each preset emotion category. This probability value characterizes the degree to which the first sample user belongs to the corresponding emotion category. The greater the probability value, the higher the degree that the first sample user belongs to this emotion category, and vice versa, it indicates that the first sample user belongs to this emotion category to a lower degree.

[0108] Next, by comparing the first emotion prediction result of the first sample user with the true emotion category to which the first sample user belongs, the first loss function can be calculated. And based on this first loss function, through the Error Back Propagation (BP) algorithm, the first gradient of the model parameter update (here defined as Gradient1) is calculated, completing one update of the network parameters based on the first sample audio signal stream, and obtaining a Convolutional Neural Network (CNN) with updated parameters as the first preliminary emotion recognition model.

[0109] Among them, the content of the first loss function is not limited in this application and can be set according to actual situations and empirical values. A preferred implementation is that the first loss function can be set as the Cross Entropy (CE) loss function, and the specific calculation formula is as follows:

[0110]

[0111] Among them, represents the probability value corresponding to the true emotion category i of the first sample user; L audi o represents the first loss function.

[0112] Step A3: Input the first sample image signal stream into the first preliminary emotion recognition model, calculate the second gradient of the model parameter update by adjusting the second loss function, and train to obtain the second preliminary emotion recognition model.

[0113] In this embodiment, after obtaining multiple first sample videos through step A1 and separating the first sample audio signal stream and the first sample image signal stream from each first sample video, in order to improve the recognition effect of the model, further, for each first sample image signal stream separated from the first sample video, first, it can be segmented into each image signal segment according to a preset fixed time window length (the specific value is not limited and can be set according to actual situations and empirical values. For example, it can be set to 5 seconds, only need to ensure that it is consistent with the preset fixed time window length used when the first sample audio signal stream is segmented). If the duration of the first sample image signal stream is insufficient, it can be copied multiple times and spliced, and then segmented according to the preset fixed time window length, and the obtained image signal segments are packaged into an image training data block corresponding to the first sample video, and defined as D image .

[0114] Then, a batch of N image signal segments corresponding to the above N audio signal segments in time can be selected from the image training data block D image as Figure 2As shown, the image features corresponding to each of these N image signal segments are extracted. It should be noted that the embodiments of the present application do not limit the method for extracting the image features corresponding to each of these N image signal segments, nor the specific extraction process. An appropriate extraction method can be selected according to the actual situation, and corresponding feature extraction operations can be performed, which will not be elaborated here.

[0115] Next, the image features (such as feature maps or feature vectors) corresponding to each of these N image signal segments can be input into the multi-layer convolution module, residual module, pooling module, and linear layer of the first preliminary emotion recognition model for emotion prediction, to determine the probability value corresponding to each preset emotion category of the first sample user in the first sample video, and define it as The larger the probability value, the higher the degree that the first sample user belongs to the corresponding preset emotion category (the emotion category corresponding to the probability value), and vice versa. Thus, the second emotion prediction result of the first sample user in the first sample video can be obtained.

[0116] Furthermore, by comparing the second emotion prediction result of the first sample user with the true emotion category to which the first sample user belongs, a second loss function can be calculated. And according to this second loss function, through the error backpropagation (BP) algorithm, the second gradient for updating the model parameters (defined here as Gradient2) can be calculated, to complete one update of the network parameters based on the first sample image signal stream, and obtain a second preliminary emotion recognition model, as Figure 2 shown.

[0117] Among them, the content of the second loss function is not limited in the present application either, and it can be set according to the actual situation and empirical values. A preferred implementation is that the second loss function can be set as the cross-entropy (CE) loss function, and the specific calculation formula is as follows:

[0118]

[0119] Among them, represents the probability value corresponding to the true emotion category i of the first sample user; L image represents the second loss function.

[0120] Step A4: Perform a weighted sum calculation on the first gradient and the second gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model, to obtain a trained emotion recognition model.

[0121] In this embodiment, after obtaining the first gradient Gradient1 through step A2 and the second gradient Gradient2 through step A3, further, the first gradient Gradient1 and the second gradient Gradient2 can be weighted and summed to obtain the gradient for updating the global network parameters. The specific calculation formula is as follows:

[0122] Gradient global = α · Gradient1 + (1 - α) · Gradient2

[0123] Among them, Gradient global represents the gradient for updating the global network parameters; α represents the fusion coefficient, and its specific value is not limited and can be set according to the actual situation and empirical values. For example, α can be set to 0.5, etc.

[0124] On this basis, further, the obtained gradient Gradient for updating the global network parameters global can be used to update the parameters of the initial emotion recognition model (such as CNN) to obtain the emotion recognition model after being trained with this batch of audio signal segments and image signal segments.

[0125] And so on, audio signal segments and image signal segments corresponding in time can be selected batch by batch from the audio training data block D audio and the image training data block D image for stacked training of the model to complete the update of the model parameters (such as the parameters of the convolutional neural network CNN) of all batches of audio signal segments and image signal segments corresponding in time in the audio training data block D audio and the image training data block D image corresponding to the first sample video.

[0126] Similarly, when training the model, a first sample video can be extracted from the training data in sequence as the model input, and the emotion recognition result of the corresponding first sample user as the output. According to the above training process, multiple rounds (such as 10 rounds) of model training are carried out, and the emotion recognition results obtained in each round of training are compared with the corresponding manually annotated results, and the model parameters are updated according to the difference between the two until the preset conditions are met. For example, the values of each loss function are very small and basically unchanged, then the update of the model parameters is stopped, and the training of the emotion recognition model is completed to generate a trained emotion recognition model, as Figure 2 shown.

[0127] It should be noted that by performing the above steps A1 - A4, it is possible to achieve sequential alternating training of the model on the sample audio signal stream first and then the sample image signal stream. Similarly, another way to train the model is to perform sequential alternating training on the model with the sample image signal stream first and then the sample audio signal stream. The specific training process includes the following steps B1 - B4:

[0128] Step B1: Obtain a second sample video, and separate a second sample audio signal stream and a second sample image signal stream from the second sample video.

[0129] Step B2: Input the second sample image signal stream into the initial emotion recognition model. By adjusting the third loss function, calculate the third gradient of the model parameter update, and train to obtain a third preliminary emotion recognition model.

[0130] Step B3: Input the second sample audio signal stream into the third preliminary emotion recognition model. By adjusting the fourth loss function, calculate the fourth gradient of the model parameter update, and train to obtain a fourth preliminary emotion recognition model.

[0131] Step B4: Perform a weighted sum calculation on the third gradient and the fourth gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain a trained emotion recognition model.

[0132] It should be noted that the implementation process of steps B1 - B4 is similar to the above steps A1 - A4. Only need to replace the first sample video, the first sample audio signal stream, the first sample image signal stream, the first gradient, the first preliminary emotion recognition model, the second preliminary emotion recognition model, and the second gradient with the second sample video, the second sample audio signal stream, the second sample image signal stream, the third gradient, the third preliminary emotion recognition model, the fourth gradient, and the fourth preliminary emotion recognition model in sequence, then a trained emotion recognition model can be obtained. The specific implementation process will not be elaborated here.

[0133] On this basis, after training and generating an emotion recognition model according to the first sample video or the second sample video, further, the generated emotion recognition model can be verified using a verification video. The specific verification process can include the following steps (1) - (4):

[0134] Step (1): Obtain a verification video, and separate a verification audio signal stream and a verification image signal stream from the verification video.

[0135] In this embodiment, in order to verify the emotion recognition model, it is first necessary to obtain verification videos. For example, with the user's permission, 1000 videos containing the voices and images of different users speaking can be collected as verification videos. The verification videos refer to the video information that can be used to verify the emotion recognition model. After obtaining these verification videos and the emotion recognition labels corresponding to the verified users in each verification video, the verification audio signal stream and the verification image signal stream can be separated from these verification videos to continue with the subsequent step (2).

[0136] Step (2): Input the verification audio signal stream and the verification image signal stream into the emotion recognition model respectively to obtain the first emotion verification result and the second emotion verification result corresponding to the verified user in the verification video.

[0137] After obtaining the verification videos through step (1) and separating the verification audio signal stream and the verification image signal stream from them, further, the verification audio signal stream and the verification image signal stream (their respective features) can be input into the emotion recognition model respectively to obtain the first emotion verification result and the second emotion verification result corresponding to the verified user in the verification video, for use in the subsequent step (3).

[0138] Step (3): Determine the final emotion verification result corresponding to the verified user according to the first emotion verification result and the second emotion verification result.

[0139] By obtaining the first emotion verification result and the second emotion verification result corresponding to the verified user in the verification video through step (2), an average calculation can be performed on the first emotion verification result and the second emotion verification result to determine the final emotion verification result corresponding to the verified user according to the calculation result.

[0140] Step (4): When the final emotion verification result corresponding to the verified user is inconsistent with the true emotion, use the verification video as the first sample video or the second sample video again to update the emotion recognition model.

[0141] After obtaining the final emotion verification result corresponding to the verified user through step (3), if this final emotion verification result is inconsistent with the true emotion recognition result corresponding to the verified user (such as the emotion marking result manually marked), the verification video can be used as the first sample video or the second sample video again to update the parameters of the emotion recognition model.

[0142] Through the above embodiment, the verification video can be used to effectively verify the emotion recognition model. When the final emotion verification result of the verified user in the verification video is inconsistent with the true emotion recognition result (such as the emotion marking result manually marked), the emotion recognition model can be adjusted and updated in a timely manner, which helps to improve the recognition accuracy and precision of the emotion recognition model.

[0143] S103: Determine the final emotion recognition result corresponding to the target user according to the first preliminary emotion recognition result and the second preliminary emotion recognition result.

[0144] In this embodiment, after obtaining the first preliminary emotion recognition result and the second preliminary emotion recognition result corresponding to the target user in the target video through step S102, and defining them as s1 and s2 respectively, further data fusion processing can be performed on the first preliminary emotion recognition result and the second preliminary emotion recognition result, and according to the processing result, the final emotion recognition result corresponding to the target user can be determined.

[0145] Specifically, an optional implementation method is to perform an average calculation on the first emotion verification result s1 and the second emotion verification result s2 to obtain a calculation result s, that is Then, the emotion category corresponding to the highest dimension in the calculation result s, or the emotion category corresponding to the dimension exceeding a preset threshold (the specific value is not limited), is used as the final emotion recognition result corresponding to the target user, that is, as the emotion category to which the target user belongs.

[0146] For example: Suppose the preset emotion categories are: "Joy", "Trust", "Fear", "Surprise", "Sadness", "Disgust", "Anger", "Anticipation". The first emotion verification result s1 output by the model is [0.14, 0.05, 0.03, 0.31, 0.05, 0.05, 0.32, 0.05], and the second emotion verification result s2 is [0.14, 0.05, 0.02, 0.32, 0.05, 0.05, 0.32, 0.05]. Then, an average calculation is performed on the first emotion verification result s1 and the second emotion verification result s2 to obtain a calculation result s of [0.14, 0.05, 0.025, 0.315, 0.05, 0.05, 0.32, 0.05]. In this way, the emotion type "Surprise" corresponding to the highest probability value (0.315) can be selected as the emotion category to which the target user belongs.

[0147] In summary, for the emotion recognition method provided in this embodiment, first, a target video to be recognized is obtained, and an audio signal stream and an image signal stream are separated from the target video. Then, the audio signal stream and the image signal stream are respectively input into a pre-constructed emotion recognition model to predict a first preliminary emotion recognition result and a second preliminary emotion recognition result corresponding to the target user in the target video. Among them, the emotion recognition model is obtained by alternately training using a sample audio signal stream and a sample image signal stream and then training through a parameter update method of gradient fusion. Next, the final emotion recognition result corresponding to the target user can be determined according to the first preliminary emotion recognition result and the second preliminary emotion recognition result. It can be seen that since this application first alternately trains using a sample audio signal stream and a sample image signal stream and then constructs an emotion recognition model through a parameter update method of gradient fusion, the recognition effect of the model is effectively improved. Therefore, when using this emotion recognition model to recognize the emotion of the target user in the target video, the accuracy of the recognition result can be effectively improved.

[0148] Second Embodiment

[0149] This embodiment will introduce an emotion recognition device. For related content, please refer to the above method embodiment.

[0150] See Figure 3 , which is a schematic diagram of the composition of an emotion recognition device provided in this embodiment. The device 300 includes:

[0151] A first acquisition unit 301, configured to acquire a target video to be recognized and separate an audio signal stream and an image signal stream from the target video;

[0152] A prediction unit 302, configured to respectively input the audio signal stream and the image signal stream into a pre-constructed emotion recognition model to predict a first preliminary emotion recognition result and a second preliminary emotion recognition result corresponding to the target user in the target video. The emotion recognition model is obtained by alternately training using a sample audio signal stream and a sample image signal stream and then training through a parameter update method of gradient fusion;

[0153] A first determination unit 303, configured to determine the final emotion recognition result corresponding to the target user according to the first preliminary emotion recognition result and the second preliminary emotion recognition result.

[0154] In an implementation manner of this embodiment, the device further includes:

[0155] A second acquisition unit, configured to acquire a first sample video and separate a first sample audio signal stream and a first sample image signal stream from the first sample video;

[0156] The first training unit is configured to input the first sample audio signal stream into an initial emotion recognition model, calculate a first gradient of model parameter update by adjusting a first loss function, and train to obtain a first preliminary emotion recognition model;

[0157] The second training unit is configured to input the first sample image signal stream into the first preliminary emotion recognition model, calculate a second gradient of model parameter update by adjusting a second loss function, and train to obtain a second preliminary emotion recognition model;

[0158] The first update unit is configured to perform weighted summation calculation on the first gradient and the second gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain the trained emotion recognition model.

[0159] In an implementation manner of this embodiment, the first training unit includes:

[0160] The first segmentation subunit is configured to segment the first sample audio signal stream into audio signal segments according to a preset fixed time window length, and pack all the audio signal segments into an audio training data block;

[0161] The first extraction subunit is configured to select N audio signal segments from the audio training data block, and extract acoustic features corresponding to the N audio signal segments respectively; N is a positive integer greater than 0;

[0162] The first prediction subunit is configured to input the acoustic features corresponding to the N audio signal segments into a multi-layer convolution module, a residual module, a pooling module, and a linear layer of the initial emotion recognition model for emotion prediction, to obtain a first emotion prediction result of a first sample user in the first sample video;

[0163] The first calculation subunit is configured to calculate a first loss function by comparing the first emotion prediction result of the first sample user with the true emotion of the first sample user;

[0164] The first training subunit is configured to calculate a first gradient of model parameter update according to the first loss function through an error backpropagation algorithm, and train to obtain a first preliminary emotion recognition model.

[0165] In an implementation manner of this embodiment, the second training unit includes:

[0166] The second segmentation subunit is configured to segment the first sample image signal stream into image signal segments according to the preset fixed time window length, and pack all the image signal segments into an image training data block;

[0167] A second extraction subunit, configured to select N image signal segments corresponding to the N audio signal segments in time from the image training data block, and extract image features corresponding to the N image signal segments respectively;

[0168] A second prediction subunit, configured to input the image features corresponding to the N image signal segments into the multi-layer convolutional module, residual module, pooling module and linear layer of the first preliminary emotion recognition model for emotion prediction, to obtain a second emotion prediction result of the first sample user in the first sample video;

[0169] A second calculation subunit, configured to calculate a second loss function by comparing the second emotion prediction result of the first sample user with the true emotion of the first sample user;

[0170] A second training subunit, configured to calculate a second gradient of model parameter update according to the second loss function by using the error backpropagation algorithm, and train to obtain a second preliminary emotion recognition model.

[0171] In an implementation manner of this embodiment, both the first loss function and the second loss function are cross-entropy loss functions.

[0172] In an implementation manner of this embodiment, the apparatus further includes:

[0173] A third acquisition unit, configured to acquire a second sample video, and separate a second sample audio signal stream and a second sample image signal stream from the second sample video;

[0174] A third training unit, configured to input the second sample image signal stream into the initial emotion recognition model, calculate a third gradient of model parameter update by adjusting a third loss function, and train to obtain a third preliminary emotion recognition model;

[0175] A fourth training unit, configured to input the second sample audio signal stream into the third preliminary emotion recognition model, calculate a fourth gradient of model parameter update by adjusting a fourth loss function, and train to obtain a fourth preliminary emotion recognition model;

[0176] A second update unit, configured to perform weighted summation calculation on the third gradient and the fourth gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain the trained emotion recognition model.

[0177] In an implementation manner of this embodiment, the initial emotion recognition model is a convolutional neural network CNN.

[0178] In an implementation manner of this embodiment, the apparatus further includes:

[0179] A third acquisition unit, configured to acquire a verification video, and separate a verification audio signal stream and a verification image signal stream from the verification video;

[0180] An input unit, configured to respectively input the verification audio signal stream and the verification image signal stream into the emotion recognition model, to obtain a first emotion verification result and a second emotion verification result corresponding to a verified user in the verification video;

[0181] A second determination unit, configured to determine a final emotion verification result corresponding to the verified user according to the first emotion verification result and the second emotion verification result;

[0182] A third update unit, configured to, when the final emotion verification result corresponding to the verified user is inconsistent with the true emotion, re-use the verification video as the first sample video or the second sample video to update the emotion recognition model.

[0183] Further, an embodiment of the present application further provides an emotion recognition device, including: a processor, a memory, and a system bus;

[0184] The processor and the memory are connected through the system bus;

[0185] The memory is configured to store one or more programs, and the one or more programs include instructions, and when the instructions are executed by the processor, the processor is caused to execute any implementation method of the above emotion recognition method.

[0186] Further, an embodiment of the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions run on a terminal device, the terminal device is caused to execute any implementation method of the above emotion recognition method.

[0187] Further, an embodiment of the present application further provides a computer program product, and when the computer program product runs on a terminal device, the terminal device is caused to execute any implementation method of the above emotion recognition method.

[0188] Through the description of the above embodiments, those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present application.

[0189] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0190] It should also be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0191] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for emotion recognition, characterized in that, Including: Obtain a target video to be recognized, and separate an audio signal stream and an image signal stream from the target video; Input the audio signal stream and the image signal stream into a pre-constructed emotion recognition model respectively, and predict a first preliminary emotion recognition result and a second preliminary emotion recognition result corresponding to a target user in the target video; The emotion recognition model is obtained by alternately training using a sample audio signal stream and a sample image signal stream, and then training through a parameter update method of gradient fusion; Determine a final emotion recognition result corresponding to the target user according to the first preliminary emotion recognition result and the second preliminary emotion recognition result; The construction method of the emotion recognition model is as follows: Obtain a first sample video, and separate a first sample audio signal stream and a first sample image signal stream from the first sample video; Input the first sample audio signal stream into an initial emotion recognition model, adjust a first loss function, calculate a first gradient of model parameter update, and train to obtain a first preliminary emotion recognition model; Input the first sample image signal stream into the first preliminary emotion recognition model, adjust a second loss function, calculate a second gradient of model parameter update, and train to obtain a second preliminary emotion recognition model; Perform weighted summation calculation on the first gradient and the second gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain the trained emotion recognition model; Or, the construction method of the emotion recognition model is as follows: Obtain a second sample video, and separate a second sample audio signal stream and a second sample image signal stream from the second sample video; Input the second sample image signal stream into an initial emotion recognition model, adjust a third loss function, calculate a third gradient of model parameter update, and train to obtain a third preliminary emotion recognition model; Input the second sample audio signal stream into the third preliminary emotion recognition model, adjust a fourth loss function, calculate a fourth gradient of model parameter update, and train to obtain a fourth preliminary emotion recognition model; Perform weighted summation calculation on the third gradient and the fourth gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain the trained emotion recognition model.

2. The method according to claim 1, wherein The step of inputting the first sample audio signal stream into an initial emotion recognition model, adjusting a first loss function, calculating a first gradient of model parameter update, and training to obtain a first preliminary emotion recognition model includes: Segment the first sample audio signal stream into audio signal segments according to a preset fixed time window length, and pack all the audio signal segments into an audio training data block; Select N audio signal segments from the audio training data block, and extract acoustic features corresponding to each of the N audio signal segments; N is a positive integer greater than 0; Input the acoustic features corresponding to each of the N audio signal segments into a multi-layer convolution module, a residual module, a pooling module, and a linear layer of the initial emotion recognition model for emotion prediction to obtain a first emotion prediction result of a first sample user in the first sample video; Calculate the first loss function by comparing the first emotion prediction result of the first sample user with the true emotion of the first sample user; According to the first loss function, calculate the first gradient of the model parameter update through the error backpropagation algorithm, and train to obtain the first preliminary emotion recognition model.

3. The method according to claim 2, wherein The step of inputting the first sample image signal stream into the first preliminary emotion recognition model, adjusting the second loss function, calculating the second gradient of the model parameter update, and training to obtain the second preliminary emotion recognition model includes: Segment the first sample image signal stream according to the preset fixed time window length into image signal segments, and package all the image signal segments into an image training data block; Select N image signal segments corresponding to the N audio signal segments in time from the image training data block, and extract the image features corresponding to each of the N image signal segments; Input the image features corresponding to each of the N image signal segments into the multi-layer convolution module, residual module, pooling module and linear layer of the first preliminary emotion recognition model for emotion prediction, and obtain the second emotion prediction result of the first sample user in the first sample video; Calculate the second loss function by comparing the second emotion prediction result of the first sample user with the true emotion of the first sample user; According to the second loss function, calculate the second gradient of the model parameter update through the error backpropagation algorithm, and train to obtain the second preliminary emotion recognition model.

4. The method according to claim 1, characterized in that, Both the first loss function and the second loss function are cross-entropy loss functions.

5. The method according to claim 1, characterized in that The initial emotion recognition model is a convolutional neural network CNN.

6. The method according to claim 1, characterized in that, The method further includes: Obtain a verification video, and separate a verification audio signal stream and a verification image signal stream from the verification video; Input the verification audio signal stream and the verification image signal stream into the emotion recognition model respectively, and obtain the first emotion verification result and the second emotion verification result corresponding to the verification user in the verification video; Determine the final emotion verification result corresponding to the verification user according to the first emotion verification result and the second emotion verification result; When the final emotion verification result corresponding to the verification user is inconsistent with the true emotion, use the verification video as the first sample video or the second sample video again to update the emotion recognition model.

7. An emotion recognition device, characterized in that, including: A first acquisition unit for acquiring a target video to be recognized, and separating an audio signal stream and an image signal stream from the target video; A prediction unit for inputting the audio signal stream and the image signal stream into a pre-constructed emotion recognition model respectively, and predicting to obtain the first preliminary emotion recognition result and the second preliminary emotion recognition result corresponding to the target user in the target video; The emotion recognition model is obtained by alternately training with a sample audio signal stream and a sample image signal stream, and then training through a parameter update method of gradient fusion; A first determination unit for determining the final emotion recognition result corresponding to the target user according to the first preliminary emotion recognition result and the second preliminary emotion recognition result; The device further includes: A second acquisition unit, configured to acquire a first sample video and separate a first sample audio signal stream and a first sample image signal stream from the first sample video; A first training unit, configured to input the first sample audio signal stream into an initial emotion recognition model, calculate a first gradient of model parameter update by adjusting a first loss function, and train to obtain a first preliminary emotion recognition model; A second training unit, configured to input the first sample image signal stream into the first preliminary emotion recognition model, calculate a second gradient of model parameter update by adjusting a second loss function, and train to obtain a second preliminary emotion recognition model; A first update unit, configured to perform a weighted sum calculation on the first gradient and the second gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain the trained emotion recognition model; Alternatively, the apparatus further includes: A third acquisition unit, configured to acquire a second sample video and separate a second sample audio signal stream and a second sample image signal stream from the second sample video; A third training unit, configured to input the second sample image signal stream into an initial emotion recognition model, calculate a third gradient of model parameter update by adjusting a third loss function, and train to obtain a third preliminary emotion recognition model; A fourth training unit, configured to input the second sample audio signal stream into the third preliminary emotion recognition model, calculate a fourth gradient of model parameter update by adjusting a fourth loss function, and train to obtain a fourth preliminary emotion recognition model; A second update unit, configured to perform a weighted sum calculation on the third gradient and the fourth gradient, and use the obtained calculation result to update the parameters of the initial emotion recognition model to obtain the trained emotion recognition model.

8. An emotion recognition device, characterized in that, Comprising: A processor, a memory, and a system bus; The processor and the memory are connected through the system bus; The memory is configured to store one or more programs, and the one or more programs include instructions that, when executed by the processor, cause the processor to execute the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Instructions are stored in the computer-readable storage medium, and when the instructions are run on a terminal device, the terminal device is caused to execute the method according to any one of claims 1-6.

Images