Emotion recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides an emotion recognition method and device, electronic equipment and storage medium, wherein the method comprises: extracting emotion features of a video to be recognized; performing feature matching on preset emotion features of each emotion category and the emotion features of the video to be recognized; determining an emotion category to which the preset emotion features matched with the emotion features belong as an emotion category of the video to be recognized; the preset emotion features of each emotion category are adaptively extracted in an emotion recognition process of sample videos under each emotion category; and an extraction manner of the preset emotion features is consistent with an extraction manner of the emotion features of the video to be recognized. The method, device, electronic equipment and storage medium provided by the application are less likely to be constrained by a training set compared with a simple emotion recognition method based on a model, and can cope with complex and variable scenes and environments in a practical process after being separated from training samples, thereby ensuring the accuracy and reliability of emotion recognition.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and more particularly to an emotion recognition method, device, electronic device, and storage medium. Background Technology

[0002] Emotion recognition is the foundation of emotion understanding, a prerequisite for computers to understand human emotions, and an effective way for people to explore and understand intelligence.

[0003] In existing technologies, emotion recognition methods based on deep learning models often rely on general computer vision recognition techniques to perform tasks such as classifying seven basic facial expressions or regressing quantitative emotion scores from images or videos.

[0004] However, emotion recognition methods based on deep learning models do not take into account the characteristics of each emotion and cannot specifically identify facial expressions and actions that are highly related to emotions. Furthermore, emotion recognition methods based solely on models are easily constrained by the training set, and after being removed from the training samples, they struggle to cope with complex and ever-changing scenarios and environments in practice. Summary of the Invention

[0005] This invention provides an emotion recognition method, device, electronic device, and storage medium to address the shortcomings of existing emotion recognition methods based on deep learning models, which do not take into account the characteristics of each emotion and cannot specifically identify facial expressions and actions that are highly related to emotions.

[0006] This invention provides an emotion recognition method, comprising:

[0007] Extract emotional features from the video to be identified;

[0008] The preset emotional features of each emotional category are matched with the emotional features of the video to be identified. The emotional category to which the preset emotional feature that matches the emotional feature belongs is determined as the emotional category of the video to be identified.

[0009] The preset emotion features for each emotion category are adaptively extracted during the emotion recognition process of sample videos under each emotion category, and the extraction method of the preset emotion features is consistent with the extraction method of the emotion features of the video to be recognized.

[0010] According to an emotion recognition method provided by the present invention, the step of extracting emotion features from a video to be recognized includes:

[0011] Extract image features from each frame of the video to be identified;

[0012] Based on an autoencoder, emotional features are extracted from the image features of each frame in the video to be identified, thus obtaining the emotional features of the video to be identified.

[0013] The autoencoder is trained by combining sample videos under each emotion category with an emotion classifier.

[0014] According to an emotion recognition method provided by the present invention, the determination step of the autoencoder includes:

[0015] Based on the initial encoder, emotion features are extracted from the image features of each frame in the sample video to obtain the predicted emotion features of the sample video.

[0016] Based on the initial classifier, the predicted emotion features of the sample video are applied to perform emotion classification to obtain the predicted emotion category of the sample video.

[0017] Based on the emotion category to which the sample video belongs and the predicted emotion category of the sample video, the parameters of the initial encoder and the initial classifier are iterated, and the autoencoder is determined based on the initial encoder after parameter iteration.

[0018] According to an emotion recognition method provided by the present invention, the step of extracting image features of each frame of the video to be recognized includes:

[0019] Extract static features from each frame of the video to be identified;

[0020] Static feature comparison is performed on each frame image and the next frame image to obtain the dynamic features of each frame image;

[0021] Based on the static and dynamic features of each frame of images, the image features of each frame of images are determined.

[0022] According to an emotion recognition method provided by the present invention, the step of extracting static features of each frame of the video to be recognized includes:

[0023] Based on the facial key point information of each frame image, the facial features of each frame image are constructed;

[0024] Based on the key eye information of each frame of the image, the eye features of each frame of the image are constructed.

[0025] Based on the facial and eye features of each frame of the image, the static features of each frame of the image are determined.

[0026] According to an emotion recognition method provided by the present invention, after extracting the image features of each frame of the video to be recognized, the method further includes:

[0027] Based on the image features of each frame, the emotion category and emotion intensity corresponding to each frame are determined.

[0028] Based on the emotion category and emotion intensity corresponding to each frame of the image, real-time emotion change information of the video to be identified is generated.

[0029] According to an emotion recognition method provided by the present invention, determining the emotion category and emotion intensity corresponding to each frame image based on the image features of each frame image includes:

[0030] Based on the static features in the image features of each frame, the emotion category corresponding to each frame is determined.

[0031] Based on the dynamic features in the image features of each frame, the emotional intensity corresponding to each frame is determined.

[0032] The present invention also provides an emotion recognition device, comprising:

[0033] The extraction unit is used to extract the emotional features of the video to be identified;

[0034] The matching unit is used to perform feature matching between the preset emotional features of each emotional category and the emotional features of the video to be identified, and to determine the emotional category to which the preset emotional features that match the emotional features belong as the emotional category of the video to be identified.

[0035] The preset emotion features for each emotion category are adaptively extracted during the emotion recognition process of sample videos under each emotion category, and the extraction method of the preset emotion features is consistent with the extraction method of the emotion features of the video to be recognized.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the emotion recognition method as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the emotion recognition method as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the emotion recognition method as described above.

[0039] The emotion recognition method, device, electronic device, and storage medium provided by this invention have preset emotion features for each emotion category that are adaptively extracted during the emotion recognition process of sample videos under each emotion category. These features are strongly correlated with each emotion category. The emotion category of the video to be recognized is determined by feature matching between the preset emotion features of each emotion category and the emotion features of the video to be recognized. Compared with emotion recognition methods based solely on models, this process is less constrained by the training set. After being removed from the training samples, it can cope with complex and ever-changing scenarios and environments in practice, ensuring the accuracy and reliability of emotion recognition. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is one of the flowcharts illustrating the emotion recognition method provided by the present invention;

[0042] Figure 2 This is a flowchart illustrating step 110 in the emotion recognition method provided by the present invention;

[0043] Figure 3 This is a flowchart illustrating the steps for determining the self-encoder provided by the present invention;

[0044] Figure 4 This is a flowchart illustrating step 111 in the emotion recognition method provided by the present invention;

[0045] Figure 5 This is a schematic diagram of the process for acquiring dynamic features provided by the present invention;

[0046] Figure 6 This is a flowchart illustrating step 1111 in the emotion recognition method provided by the present invention;

[0047] Figure 7 This is a schematic diagram of the process for generating real-time emotion change information provided by the present invention;

[0048] Figure 8 This is a flowchart illustrating step 111-1 in the emotion recognition method provided by the present invention;

[0049] Figure 9 This is the second flowchart of the emotion recognition method provided by the present invention;

[0050] Figure 10 This is a schematic diagram of the structure of the emotion recognition device provided by the present invention;

[0051] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] Among the relevant technologies, there are mainly emotion recognition methods based on scale assessment and deep learning models. Among them, emotion recognition methods based on scale assessment are more sensitive to the subject of the assessment. The user's subjective judgment during the emotion assessment has a significant impact on the results, and there is a large gap between doctors of different levels. In contrast, when users conduct self-assessment, they cannot gain a deeper understanding by selecting fixed options throughout the process, resulting in poor emotion recognition performance.

[0054] Emotion recognition methods based on deep learning models require external devices to monitor user behavior data and involve long inference times through multiple model predictions, making them unsuitable for widespread adoption. Furthermore, the pre-defined rules require a large amount of prior knowledge, and deep learning models lack adaptive weights, resulting in poor emotion recognition performance.

[0055] To address this issue, the present invention provides an emotion recognition method. Figure 1 This is one of the flowcharts illustrating the emotion recognition process provided by the present invention, such as... Figure 1 As shown, the method includes:

[0056] Step 110: Extract the emotional features of the video to be identified.

[0057] Specifically, the video to be identified is the video for which emotion recognition needs to be performed. The video data here can be a pre-shot and stored video or a video stream acquired in real time. This embodiment of the invention does not specifically limit this.

[0058] After obtaining the video to be identified, the emotional features of the video can be extracted. Here, the emotional features of the video to be identified can be extracted based on the feature encoding model to extract the emotional features of each frame of the video to be identified. The feature encoding model here can be the Transformer model, or a cascaded multilayer convolutional neural network (CNN), or a combination of deep neural networks (DNN) and CNN, etc. The embodiments of the present invention do not make specific limitations on this.

[0059] Step 120: Perform feature matching between the preset emotion features of each emotion category and the emotion features of the video to be identified, and determine the emotion category to which the preset emotion feature that matches the emotion feature belongs as the emotion category of the video to be identified.

[0060] The preset emotion features for each emotion category are adaptively extracted during the emotion recognition process of sample videos under each emotion category, and the extraction method of the preset emotion features is consistent with the extraction method of the emotion features of the video to be recognized.

[0061] Specifically, after extracting the emotional features of the video to be identified, the preset emotional features of each emotional category can be matched with the emotional features of the video to be identified.

[0062] The preset emotional features for each emotion category here are adaptively extracted during the emotion recognition process of sample videos under each emotion category. The sample videos here can be pre-shot and stored videos, or real-time captured video streams. The emotion category here can be any one of happiness, anger, resentment, surprise, disgust, fear, and neutrality; taking depressive emotion as an example, the emotion category can be analyzing whether the test subject has depressive emotions, but this embodiment of the invention does not specifically limit this.

[0063] Understandably, adaptive extraction during emotion recognition of sample videos under each emotion category means that the acquisition of preset emotion features for each emotion category no longer relies on manual feature selection, but is entirely adaptively extracted by the model applied to emotion recognition based on actual emotion recognition needs. This results in a strong correlation between preset emotion features and various emotion categories, allowing them to handle complex and ever-changing scenarios and environments in practice, even without training samples. Taking depressive emotion as an example, using the HAMD-17 (Hamilton Depression Scale), the adaptively extracted preset depressive emotion features correspond to 17 sub-scenes. These adaptively extracted preset depressive emotion features can be stored in the corresponding locations in the constructed feature knowledge base.

[0064] Here, the method for extracting the preset emotion features is the same as the method for extracting the emotion features of the video to be identified. That is, the method for extracting the emotion features of the video to be identified is also adaptively extracted during the emotion recognition process. Therefore, the emotion features of the video to be identified and the preset emotion features belong to the same feature space and can be directly compared through feature matching.

[0065] Here, feature matching between the preset emotion features of each emotion category and the emotion features of the video to be identified can be performed using methods such as cosine similarity or Pearson correlation coefficient. This embodiment of the invention does not specifically limit the method.

[0066] Understandably, the cosine similarity between the preset emotional features of each emotion category and the emotional features of the video to be identified can reflect the matching situation between the preset emotional features of each emotion category and the emotional features of the video to be identified. The higher the cosine similarity between the preset emotional features of each emotion category and the emotional features of the video to be identified, the better the match between the preset emotional features of each emotion category and the emotional features of the video to be identified; the lower the cosine similarity between the preset emotional features of each emotion category and the emotional features of the video to be identified, the worse the match between the preset emotional features of each emotion category and the emotional features of the video to be identified.

[0067] After matching the preset emotional features of each emotional category with the emotional features of the video to be identified, the emotional category to which the preset emotional features that match the emotional features can be determined as the emotional category of the video to be identified.

[0068] The method provided in this invention uses preset emotion features for each emotion category, which are adaptively extracted during the emotion recognition process of sample videos under each emotion category. These features are strongly correlated with each emotion category. The emotion category of the video to be recognized is determined by feature matching between the preset emotion features of each emotion category and the emotion features of the video to be recognized. Compared with emotion recognition methods based solely on models, this process is less constrained by the training set. After being removed from the training samples, it can cope with complex and ever-changing scenarios and environments in practice, ensuring the accuracy and reliability of emotion recognition.

[0069] Based on any of the above embodiments, the emotion recognition method provided by the embodiments of the present invention can be applied to the identification and quantitative analysis of depressive emotions. Furthermore, the severity of depressive emotions obtained based on emotion recognition in the embodiments of the present invention can be applied to clinical diagnosis as a reference factor for doctors in diagnosing depression. In addition, the severity of depressive emotions obtained based on emotion recognition in the embodiments of the present invention can also be applied to medical record quality inspection, comparing the severity of depressive emotions obtained from automated analysis with the severity of depression diagnosed by doctors in the medical records, thereby verifying the quality of the medical records. The embodiments of the present invention do not specifically limit this application.

[0070] Based on the above embodiments, Figure 2 This is a flowchart illustrating step 110 of the emotion recognition method provided by the present invention, as follows: Figure 2 As shown, step 110 includes:

[0071] Step 111: Extract image features from each frame of the video to be identified;

[0072] Step 112: Based on the autoencoder, extract the emotion features from the image features of each frame in the video to be identified to obtain the emotion features of the video to be identified.

[0073] The autoencoder is trained by combining sample videos under each emotion category with an emotion classifier.

[0074] Specifically, in order to extract the emotional features of the video to be identified, it is necessary to extract the image features of each frame of the video to be identified before performing step 112, and obtain the autoencoder through the following steps:

[0075] Sample videos for each emotion category and each emotion category can be collected in advance. An emotion classifier can also be built in advance. The emotion classifier is used to perform emotion recognition on the image features of each frame in the video to be recognized extracted by the autoencoder.

[0076] In addition, an initial autoencoder can be constructed. Subsequently, the initial autoencoder can be trained in conjunction with an emotion classifier based on sample videos under each emotion category, and the trained initial autoencoder can be used as the autoencoder.

[0077] In this process, the initial autoencoder and the initial emotion classifier can be used as the initial classification model, which is the initial model used to train the autoencoder. Here, the initial autoencoder can be a CNN, DNN, or other structure, and the initial emotion classifier can be a softmax layer. This embodiment of the invention does not specifically limit these features.

[0078] After obtaining the initial classification model, which includes the initial autoencoder and the initial emotion classifier, the initial classification model can be trained using pre-collected sample videos for each emotion category and each emotion category:

[0079] First, sample videos for each emotion category are input into the initial autoencoder. The initial autoencoder extracts features from the sample videos for each emotion category, obtaining and outputting the initial emotion features for each emotion category. It can be understood that the initial autoencoder is the initial model before the initial classification model is trained. To distinguish it from the emotion features output by the initial classification model, the emotion features output by the initial autoencoder are referred to here as the initial emotion features.

[0080] Secondly, the initial emotion features are input into the initial emotion classifier, which then performs emotion recognition on the initial emotion features and outputs the emotion classification results.

[0081] Once the emotion classification result is obtained based on the initial classification model, it can be compared with the pre-collected emotion categories. The loss function value is calculated based on the degree of difference between the two. Then, the parameters of the initial classification model are iterated as a whole based on the loss function value. The initial classification model after parameter iteration is denoted as the emotion classification model.

[0082] Understandably, the greater the difference between the emotion classification result and the pre-collected emotion categories, the larger the loss function value; conversely, the smaller the difference between the emotion classification result and the pre-collected emotion categories, the smaller the loss function value.

[0083] The emotion classification model after parameter iteration has the same structure as the initial classification model. Therefore, the emotion classification model can be divided into two parts: the initial autoencoder after parameter iteration and the initial emotion classifier after parameter iteration. The initial autoencoder after parameter iteration can be directly used as the autoencoder.

[0084] That is, during the training process of the autoencoder, it learns the function of extracting emotional features from the video to be recognized, so as to extract emotional features that can be used for subsequent emotion recognition.

[0085] After training the autoencoder, the image features of each frame in the extracted video to be recognized can be used to extract emotional features, thus obtaining the emotional features of the video to be recognized.

[0086] Based on the above embodiments, Figure 3 This is a flowchart illustrating the steps for determining the self-encoder provided by the present invention, as shown below. Figure 3 As shown, the steps for determining the self-encoder include:

[0087] Step 310: Based on the initial encoder, extract emotion features from the image features of each frame in the sample video to obtain the predicted emotion features of the sample video.

[0088] Step 320: Based on the initial classifier, apply the predicted emotion features of the sample video to perform emotion classification, and obtain the predicted emotion category of the sample video.

[0089] Step 330: Based on the emotion category to which the sample video belongs and the predicted emotion category of the sample video, perform parameter iteration on the initial encoder and the initial classifier, and determine the autoencoder based on the initial encoder after parameter iteration.

[0090] Specifically, the initial encoder and the initial classifier are the initial models required to train the autoencoder. The model parameters of the initial encoder can be pre-set or obtained through initialization, and the model parameters of the initial classifier can be pre-set or obtained through initialization. This embodiment of the invention does not impose specific limitations on this.

[0091] Training the initial encoder and initial classifier requires image features from each frame of the sample video, predicted sentiment features of the sample video, and the sentiment category to which the sample video belongs. The predicted sentiment features of the sample video can be extracted from the image features of each frame of the sample video using the pre-trained initial encoder.

[0092] After obtaining the predicted emotion features of the sample video, the predicted emotion features of the sample video can be input into the initial classifier. The initial classifier will then apply the predicted emotion features of the sample video to classify the emotion and obtain the predicted emotion category of the sample video.

[0093] After obtaining the predicted sentiment category of the sample video, the predicted sentiment category of the sample video can be compared with the sentiment category of the pre-collected sample videos. The loss function value is calculated based on the degree of difference between the two, and the parameters of the initial encoder and the initial classifier are iterated based on the loss function value. The initial encoder after the parameter iteration is completed is called the autoencoder.

[0094] It can be understood that the greater the difference between the predicted sentiment category of the sample video and the sentiment category of the pre-collected sample videos, the larger the loss function value; conversely, the smaller the difference between the predicted sentiment category of the sample video and the sentiment category of the pre-collected sample videos, the smaller the loss function value.

[0095] The method provided in this embodiment of the invention treats the initial encoder and the initial classifier as a whole for parameter iteration, which can further improve the reliability of the autoencoder when performing subsequent emotion recognition and ensure the accuracy of emotion recognition.

[0096] Based on the above embodiments, Figure 4 This is a flowchart illustrating step 111 of the emotion recognition method provided by the present invention, as follows: Figure 4 As shown, step 111 includes:

[0097] Step 1111: Extract the static features of each frame of the video to be identified;

[0098] Step 1112: Perform static feature comparison based on each frame image and the next frame image of each frame image to obtain the dynamic features of each frame image;

[0099] Step 1113: Determine the image features of each frame image based on the static and dynamic features of each frame image.

[0100] Specifically, the image features of each frame can be divided into two levels: static features and dynamic features.

[0101] Static features can be extracted from each frame of the video to be identified. Here, static features refer to the global features of the face in each frame of the video to be identified, which may include facial features and eye features. Specifically, static features can be extracted from 5 frames per second, 10 frames per second, or 8 frames per second; this embodiment of the invention does not impose a specific limitation on this.

[0102] Figure 5 This is a schematic diagram of the process for acquiring dynamic features provided by the present invention, such as... Figure 5 As shown, after extracting the static features of each frame in the video to be recognized, static feature comparison can be performed based on each frame and the next frame. That is, the dynamic features of each frame can be obtained based on the residuals between the static features of each frame and the next frame. Here, dynamic features refer to the dynamic features of the face in each frame of the video to be recognized, which can be the activity level of the face's Action Unit (AU). For example, an emotion amplitude spectrum of the corresponding AU can be plotted based on the timeline. Similar to an electrocardiogram, observing the emotion amplitude spectrum allows for observation of changes in various parts of the user's face, which helps improve the efficiency of emotion recognition.

[0103] After obtaining the static and dynamic features of each frame of the image, the image features of each frame can be determined based on these features.

[0104] The method provided in this invention obtains image features for each frame that include both static and dynamic feature information, thereby improving the accuracy of subsequent emotion recognition.

[0105] Based on the above embodiments, Figure 6 This is a flowchart illustrating step 1111 of the emotion recognition method provided by the present invention, as follows: Figure 6 As shown, step 1111 includes:

[0106] Step 1111-1: Based on the facial key point information of each frame image, construct the facial features of each frame image;

[0107] Step 1111-2: Based on the eye key point information of each frame image, construct the eye features of each frame image;

[0108] Step 1111-3: Determine the static features of each frame image based on the facial and eye features of each frame image.

[0109] Specifically, facial features for each frame can be constructed based on facial key point information. For example, 64 3D facial key points can be extracted using the open-source face detection algorithm Openface, and then these 64 3D facial key point information can be used to construct facial features for each frame. The facial key point information here may include the degree of mouth opening and closing, the tension of the mouth, or the degree of mouth opening and closing, the tension of the mouth, and the tension of the eyebrows. This embodiment of the invention does not specifically limit this.

[0110] Alternatively, eye features for each frame can be constructed based on the eye keypoint information of each frame. For example, 32 facial 3D keypoints can be extracted using the open-source face detection algorithm Openface and optical flow method, and then the eye features for each frame can be constructed using the 32 facial 3D keypoints. The eye keypoint information here may include the eye opening and blinking frequency, or the eye opening and gaze direction distribution. This embodiment of the invention does not specifically limit these aspects. The gaze direction distribution here may include up, down, left, right, lower left, lower right, etc., and this embodiment of the invention does not specifically limit these aspects.

[0111] After obtaining facial and eye features, the static features of each frame can be determined based on the facial and eye features of each frame.

[0112] The method provided in this invention obtains static features of each frame image, including facial features and eye features, which improves the richness of static features of each frame image and is beneficial to improving the accuracy of subsequent emotion recognition.

[0113] Multimodal emotion recognition methods in related technologies employ a method of fusing information from multiple modalities, but can only output a quantitative emotion score, resulting in the loss of interpretability information from each modality.

[0114] To address the aforementioned issues, embodiments of the present invention perform interpretability analysis on the information of each modality.

[0115] Based on the above embodiments, Figure 7 This is a schematic diagram of the process for generating real-time emotion change information provided by the present invention, such as... Figure 7 As shown, after step 111, the following steps are also included:

[0116] Step 111-1: Based on the image features of each frame, determine the emotion category and emotion intensity corresponding to each frame;

[0117] Step 111-2: Based on the emotion category and emotion intensity corresponding to each frame image, generate real-time emotion change information of the video to be identified.

[0118] Specifically, after extracting the image features of each frame in the video to be identified, the emotion category and emotion intensity corresponding to each frame can be determined based on the static and dynamic features contained in the image features of each frame.

[0119] The emotion category here can be any one of happiness, anger, resentment, surprise, disgust, fear, and neutrality; taking depressive emotion as an example, the emotion category can be to analyze whether the test subject has depressive emotions, and the embodiments of the present invention do not specifically limit this.

[0120] Taking depressive mood as an example, the intensity of mood here can be used to determine the severity of the depressive mood of the test subject. The intensity of mood can be determined according to the HAMD scale formula. The intensity of mood can be divided into four levels of severity: none, mild, moderate and severe. This embodiment of the invention does not make specific limitations on this.

[0121] After obtaining the emotion category and intensity corresponding to each frame, these values ​​can be visualized to generate real-time emotion change information for the video to be identified. This real-time emotion change information reflects the changes in the emotion category and intensity corresponding to each frame.

[0122] The method provided in this invention generates real-time emotion change information of the video to be identified based on the emotion category and emotion intensity corresponding to each frame of the image. The real-time emotion change information can improve the convenience of emotion analysis.

[0123] Based on the above embodiments, Figure 8This is a flowchart illustrating step 111-1 of the emotion recognition method provided by the present invention, as shown below. Figure 8 As shown, step 111-1 includes:

[0124] Step 810: Based on the static features in the image features of each frame, determine the emotion category corresponding to each frame;

[0125] Step 820: Based on the dynamic features in the image features of each frame, determine the emotional intensity corresponding to each frame.

[0126] Specifically, after obtaining the static and dynamic features of each frame's image features, the emotion category corresponding to each frame can be determined based on the static features. For example, the image features of each frame can be input into a classification layer, which then outputs the emotion category corresponding to each frame. The classification layer here can be a Softmax layer or a Sigmoid layer; this embodiment of the invention does not specifically limit the type of layer.

[0127] The dynamic features in the image features of each frame can be input into the regression layer, and the regression layer outputs the emotion intensity corresponding to each frame. The regression layer here can be a CNN, a DNN, or a combination of CNN and DNN, etc. The embodiments of the present invention do not make specific limitations on this.

[0128] Based on any of the above embodiments Figure 9 This is the second flowchart illustrating the emotion recognition method provided by this invention, as shown below. Figure 9 As shown, the method includes:

[0129] The first step is to construct facial features for each frame based on the facial key point information of each frame; then, to construct eye features for each frame based on the eye key point information of each frame; finally, to determine the static features of each frame based on the facial and eye features of each frame.

[0130] The second step is to perform static feature comparison based on each frame of the image and the next frame of the image to obtain the dynamic features of each frame of the image; then, based on the static and dynamic features of each frame of the image, the image features of each frame of the image can be determined.

[0131] The third step is to extract emotional features from the image features of each frame in the video to be identified based on the autoencoder, thereby obtaining the emotional features of the video to be identified. The autoencoder is trained by combining sample videos under each emotion category with an emotion classifier.

[0132] The fourth step involves feature matching between the preset emotion features of each emotion category and the emotion features of the video to be identified. The emotion category to which the preset emotion features that match the emotion features is determined as the emotion category of the video to be identified. The preset emotion features for each emotion category are adaptively extracted during the emotion recognition process of sample videos under each emotion category, and the extraction method of the preset emotion features is consistent with the extraction method of the emotion features of the video to be identified.

[0133] The fifth step can also be to determine the emotion category and emotion intensity corresponding to each frame based on the image features of each frame; and generate real-time emotion change information of the video to be identified based on the emotion category and emotion intensity corresponding to each frame.

[0134] The emotion recognition device provided by the present invention is described below. The emotion recognition device described below and the emotion recognition method described above can be referred to in correspondence.

[0135] Based on any of the above embodiments, the present invention provides an emotion recognition device. Figure 10 This is a schematic diagram of the emotion recognition device provided by the present invention, as shown below. Figure 10 As shown, the device includes:

[0136] Extraction unit 1010 is used to extract the emotional features of the video to be identified;

[0137] The matching unit 1020 is used to perform feature matching between the preset emotion features of each emotion category and the emotion features of the video to be identified, and to determine the emotion category to which the preset emotion feature that matches the emotion feature belongs as the emotion category of the video to be identified.

[0138] The preset emotion features for each emotion category are adaptively extracted during the emotion recognition process of sample videos under each emotion category, and the extraction method of the preset emotion features is consistent with the extraction method of the emotion features of the video to be recognized.

[0139] The apparatus provided in this invention uses preset emotion features for each emotion category, which are adaptively extracted during the emotion recognition process of sample videos under each emotion category. These features are strongly correlated with each emotion category. The emotion category of the video to be recognized is determined by feature matching between the preset emotion features of each emotion category and the emotion features of the video to be recognized. Compared with emotion recognition methods based solely on models, this process is less constrained by the training set. After being removed from the training samples, it can cope with complex and ever-changing scenarios and environments in practice, ensuring the accuracy and reliability of emotion recognition.

[0140] Based on any of the above embodiments, the extraction unit specifically includes:

[0141] An image feature extraction unit is used to extract image features from each frame of the video to be identified;

[0142] An emotion feature determination unit is used to extract emotion features from the image features of each frame in the video to be identified based on an autoencoder, so as to obtain the emotion features of the video to be identified.

[0143] The autoencoder is trained by combining sample videos under each emotion category with an emotion classifier.

[0144] Based on any of the above embodiments, the step of determining the self-encoder includes:

[0145] Based on the initial encoder, emotion features are extracted from the image features of each frame in the sample video to obtain the predicted emotion features of the sample video.

[0146] Based on the initial classifier, the predicted emotion features of the sample video are applied to perform emotion classification to obtain the predicted emotion category of the sample video.

[0147] Based on the emotion category to which the sample video belongs and the predicted emotion category of the sample video, the parameters of the initial encoder and the initial classifier are iterated, and the autoencoder is determined based on the initial encoder after parameter iteration.

[0148] Based on any of the above embodiments, the image feature extraction unit is specifically used for:

[0149] A static feature extraction unit is used to extract static features from each frame of the video to be identified;

[0150] A dynamic feature extraction unit is used to perform static feature comparison based on each frame image and the next frame image of each frame image to obtain the dynamic features of each frame image;

[0151] An image feature extraction unit is used to determine the image features of each frame image based on the static and dynamic features of each frame image.

[0152] Based on any of the above embodiments, the extraction of static feature units is specifically used for:

[0153] A facial feature unit is constructed to construct facial features for each frame of the image based on the facial key point information of each frame.

[0154] An eye feature unit is constructed to construct the eye features of each frame image based on the eye key point information of each frame image;

[0155] A static feature unit is defined to determine the static features of each frame image based on the facial and eye features of each frame image.

[0156] Based on any of the above embodiments, after extracting the image feature units, the method further includes:

[0157] A category and intensity unit is defined to determine the emotion category and emotion intensity corresponding to each frame image based on the image features of each frame image.

[0158] A real-time emotion change information unit is generated, which is used to generate real-time emotion change information of the video to be identified based on the emotion category and emotion intensity corresponding to each frame image.

[0159] Based on any of the above embodiments, determining the category and strength unit is specifically used for:

[0160] Based on the static features in the image features of each frame, the emotion category corresponding to each frame is determined.

[0161] Based on the dynamic features in the image features of each frame, the emotional intensity corresponding to each frame is determined.

[0162] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11 As shown, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute an emotion recognition method, which includes: extracting emotion features from a video to be recognized; performing feature matching between preset emotion features of each emotion category and the emotion features of the video to be recognized; determining the emotion category to which the preset emotion features that match the emotion features belong as the emotion category of the video to be recognized; the preset emotion features of each emotion category are adaptively extracted during the emotion recognition process of sample videos under each emotion category, and the extraction method of the preset emotion features is consistent with the extraction method of the emotion features of the video to be recognized.

[0163] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the emotion recognition method provided by the above methods. The method includes: extracting emotion features of a video to be recognized; performing feature matching between preset emotion features of each emotion category and the emotion features of the video to be recognized; determining the emotion category to which the preset emotion features that match the emotion features belong as the emotion category of the video to be recognized; wherein the preset emotion features of each emotion category are adaptively extracted during the emotion recognition process of sample videos under each emotion category, and the extraction method of the preset emotion features is consistent with the extraction method of the emotion features of the video to be recognized.

[0165] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the emotion recognition method provided by the above methods. The method includes: extracting emotion features of a video to be recognized; performing feature matching between preset emotion features of each emotion category and the emotion features of the video to be recognized; determining the emotion category to which the preset emotion features that match the emotion features belong as the emotion category of the video to be recognized; wherein the preset emotion features of each emotion category are adaptively extracted during the emotion recognition process of sample videos under each emotion category, and the extraction method of the preset emotion features is consistent with the extraction method of the emotion features of the video to be recognized.

[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0167] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An emotion recognition method, characterized in that, include: Extract image features from each frame of the video to be identified; Based on an autoencoder, emotional features are extracted from the image features of each frame in the video to be identified to obtain the emotional features of the video to be identified; the autoencoder is trained by combining sample videos under each emotional category with an emotional classifier. The preset emotional features of each emotion category are matched with the emotional features of the video to be identified. The emotion category to which the preset emotional features that match the emotional features are determined as the emotion category of the video to be identified. The preset emotional features of each emotion category are adaptively extracted during the emotion recognition process of sample videos under each emotion category. The extraction method of the preset emotional features is consistent with the extraction method of the emotional features of the video to be identified. Based on the static features in the image features of each frame, the emotion category corresponding to each frame is determined. Based on the dynamic features in the image features of each frame, the emotional intensity corresponding to each frame is determined. Based on the emotion category and emotion intensity corresponding to each frame of the image, real-time emotion change information of the video to be identified is generated.

2. The emotion recognition method according to claim 1, characterized in that, The steps for determining the self-encoder include: Based on the initial encoder, emotion features are extracted from the image features of each frame in the sample video to obtain the predicted emotion features of the sample video. Based on the initial classifier, the predicted emotion features of the sample video are applied to perform emotion classification to obtain the predicted emotion category of the sample video. Based on the emotion category to which the sample video belongs and the predicted emotion category of the sample video, the parameters of the initial encoder and the initial classifier are iterated, and the autoencoder is determined based on the initial encoder after parameter iteration.

3. The emotion recognition method according to claim 1, characterized in that, The extraction of image features from each frame of the video to be identified includes: Extract static features from each frame of the video to be identified; Static feature comparison is performed on each frame image and the next frame image to obtain the dynamic features of each frame image; Based on the static and dynamic features of each frame of images, the image features of each frame of images are determined.

4. The emotion recognition method according to claim 3, characterized in that, The step of extracting static features from each frame of the video to be identified includes: Based on the facial key point information of each frame image, the facial features of each frame image are constructed; Based on the key eye information of each frame of the image, the eye features of each frame of the image are constructed. Based on the facial and eye features of each frame of the image, the static features of each frame of the image are determined.

5. An emotion recognition device, characterized in that, include: The extraction unit is used to extract image features from each frame of the video to be identified. The emotion feature extraction unit is used to extract emotion features from the image features of each frame in the video to be identified based on an autoencoder, so as to obtain the emotion features of the video to be identified; the autoencoder is trained by combining sample videos under each emotion category with an emotion classifier. The matching unit is used to perform feature matching between the preset emotion features of each emotion category and the emotion features of the video to be identified, and to determine the emotion category to which the preset emotion features that match the emotion features belong as the emotion category of the video to be identified; the preset emotion features of each emotion category are adaptively extracted during the emotion recognition process of the sample videos under each emotion category, and the extraction method of the preset emotion features is consistent with the extraction method of the emotion features of the video to be identified. The first determining unit is used to determine the emotion category corresponding to each frame image based on the static features in the image features of each frame image; The second determining unit is used to determine the emotional intensity corresponding to each frame image based on the dynamic features in the image features of each frame image; The generation unit is used to generate real-time emotion change information of the video to be identified based on the emotion category and emotion intensity corresponding to each frame image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the emotion recognition method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the emotion recognition method as described in any one of claims 1 to 4.

Images