Micro-expression-based emotion recognition method, device, equipment and medium

Abstract

This application relates to the field of digital medicine, specifically to a method, device, equipment and medium for emotion recognition based on micro-expressions. This method comprehensively considers the two factors that the climax frame in a micro-expression video frame sequence contains more facial information and that the optical flow information can reflect the subtle changes in the face. RGB The two types of climax frames, climax frames and optical flow climax frames, obtain more and more subtle facial features from the micro-expression video to be identified, thereby improving the accuracy of emotion recognition of micro-expressions. At the same time, this method RGB The climax frames and optical flow climax frames are enhanced to increase the number of micro-expression data samples. RGB The number of climax frames and optical flow climax frames increases the number of micro-expression videos in the micro-expression data sample set in disguise. The greatly increased number of samples can improve the accuracy of the training model and further improve the accuracy of emotion recognition of micro-expressions.

Description

TECHNICAL FIELD

[0001] The present application is suitable for the field of digital medical technology, and particularly relates to a micro-expression-based emotion recognition method and device, equipment and medium. BACKGROUND

[0002] Micro-expression (MES) is a very short, subtle and involuntary facial expression. Compared with ordinary expressions, micro-expression has the following characteristics:

[0003] (1) Short duration, usually only ;

[0004] (2) Low action intensity, difficult to detect;

[0005] (3) Produced in an unconscious state, usually difficult to conceal or disguise;

[0006] (4) Ordinary expressions can be analyzed in images, while micro-expression analysis usually requires video.

[0007] Since micro-expression is spontaneously produced in an unconscious state and is difficult to conceal or disguise, it is usually directly related to true emotions, so micro-expression is more reliable in emotion analysis. In the current medical field, recognizing the true emotions of patients based on micro-expression is of great help to the auxiliary diagnosis of patient diseases, health management, etc.

[0008] In the prior art, micro-expression emotion analysis initially relies on manual analysis. In this case, only those with strong micro-expression analysis capabilities and rich micro-expression analysis experience can analyze the micro-expression video to obtain a high-reliability emotion recognition result. However, it takes a lot of time and effort to cultivate such talents, and not only is it difficult to cultivate, but the micro-expression analysis efficiency is also low.

[0009] With the rise of artificial intelligence technology, machine learning, deep learning, etc. are gradually increasing in micro-expression analysis. Although the analysis efficiency of micro-expression has been improved to some extent, there are still the following difficulties in the current micro-expression emotion recognition work: (1) Micro-expression has a short duration and low action intensity, and facial features are difficult to extract, so it is difficult to capture enough and subtle facial information, resulting in low accuracy of micro-expression-based emotion recognition; (2) The existing micro-expression dataset is limited and the sample size is small, so it is difficult to train a robust deep learning model from scratch, and model training is difficult. Therefore, how to extract more and more subtle facial information from micro-expression under the condition of limited micro-expression dataset and small sample size, and improve the accuracy of micro-expression emotion recognition has become a problem to be solved. SUMMARY

[0010] Therefore, the embodiment of the present application provides a micro-expression-based emotion recognition method, device, equipment and medium to solve the problem of how to extract more subtle facial information from micro-expression under the condition of limited micro-expression dataset and small sample size, and improve the emotion recognition accuracy of micro-expression.

[0011] In a first aspect, the embodiment of the present application provides a micro-expression-based emotion recognition method, which comprises:

[0012] obtaining a micro-expression video to be recognized;

[0013] performing data preprocessing on the micro-expression video to be recognized, and extracting RGB climax frames and optical flow climax frames of the micro-expression video to be recognized;

[0014] inputting the RGB climax frames and optical flow climax frames of the micro-expression video to be recognized into a pre-trained micro-expression emotion recognition model, and outputting a real emotion corresponding to the micro-expression video to be recognized; wherein the pre-trained micro-expression emotion recognition model is obtained by the following steps:

[0015] performing data preprocessing on each micro-expression video in a micro-expression data sample set, and extracting RGB climax frames and optical flow climax frames of each micro-expression video;

[0016] performing data enhancement processing on the RGB climax frames and optical flow climax frames of each micro-expression video in the micro-expression data sample set, and increasing the number of grouped RGB climax frames and optical flow climax frames in the micro-expression data sample set;

[0017] constructing a micro-expression emotion recognition model with a residual neural network as a feature extraction network, inputting the grouped RGB climax frames and optical flow climax frames in the micro-expression data sample set into the micro-expression emotion recognition model, for each group of RGB climax frames and optical flow climax frames, extracting facial features in the RGB climax frames and facial features in the optical flow climax frames by the feature extraction network respectively, and concatenating the two, and classifying a real emotion corresponding to the group of RGB climax frames and optical flow climax frames according to the concatenated features, and the classification result is a real emotion recognition result corresponding to the group of RGB climax frames and optical flow climax frames;

[0018] for the real emotion recognition result corresponding to any group of RGB climax frames and optical flow climax frames, combining it with the group of RGBThe climax frame and the optical flow climax frame are compared with the corresponding emotion classification label; according to all comparison results, a preset loss calculation function is used to calculate a difference loss between a true emotion judgment result and the emotion classification label, until the difference loss is minimum, and the model training is completed.

[0019] In an embodiment, the method further comprises: RGB The data enhancement processing of the climax frame and the optical flow climax frame comprises: using two AC-GAN networks with the same structure to respectively perform data enhancement on the climax frame and the optical flow climax frame of each micro-expression video in the micro-expression data sample set. RGB The data enhancement processing of the climax frame and the optical flow climax frame comprises: using two AC-GAN networks with the same structure to respectively perform data enhancement on the climax frame and the optical flow climax frame of each micro-expression video in the micro-expression data sample set.

[0020] In an embodiment, the method further comprises: RGB The climax frame and the optical flow climax frame of the micro-expression video to be identified comprise:

[0021] The method further comprises:

[0022] The method further comprises:

[0023] The method further comprises:

[0024] The climax frame of the micro-expression time domain interpolation sequence is extracted as the climax frame of the micro-expression video to be identified, and a dense optical flow between a starting frame and the climax frame of the micro-expression time domain interpolation sequence is calculated to obtain the optical flow climax frame of the micro-expression video to be identified. RGB The climax frame of the micro-expression time domain interpolation sequence is extracted as the climax frame of the micro-expression video to be identified, and a dense optical flow between a starting frame and the climax frame of the micro-expression time domain interpolation sequence is calculated to obtain the optical flow climax frame of the micro-expression video to be identified.

[0025] In an embodiment, the method further comprises:

[0026] The method further comprises: Dlib The method further comprises: Opencv The method further comprises: performing face positioning on each image in the micro-expression video frame sequence using the visual library or the visual library to determine the facial key feature points of each image; and according to the facial key feature points of each image, cropping the face of each image, and saving the cropped face images as the micro-expression face image sequence in the order of the micro-expression video frame sequence.

[0027] In an embodiment, the method further comprises: TIM The method further comprises:

[0028] In one embodiment, calculating the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence to obtain the optical flow climax frame of the micro-expression video to be recognized includes:

[0029] First, the starting frame and climax frame of the micro-expression temporal interpolation sequence are extracted through the SMEConvNet network. Then, the dense optical flow between the starting frame and climax frame of the micro-expression temporal interpolation sequence is calculated through the Farneback algorithm to obtain the optical flow climax frame in the micro-expression video to be identified.

[0030] In a second aspect, an embodiment of the present application provides an emotion recognition device based on micro-expressions, the emotion recognition device based on micro-expressions comprising:

[0031] A data acquisition module is used to obtain the micro-expression video to be recognized;

[0032] The data preprocessing module is used to preprocess the micro-expression video to be identified and extract the micro-expression data of the micro-expression video to be identified. RGB Climax frame and optical flow climax frame;

[0033] The real emotion recognition module is used to identify the micro-expression video RGB The climax frame and the optical flow climax frame are input into a pre-trained micro-expression emotion recognition model, which outputs the true emotion corresponding to the micro-expression video to be recognized; wherein the pre-trained micro-expression emotion recognition model is obtained by the following steps:

[0034] Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame;

[0035] For each micro-expression video in the micro-expression data sample set RGB Data enhancement processing is performed on climax frames and optical flow climax frames to increase the number of micro-expression data samples and group them together. RGB The number of climax frames and optical flow climax frames;

[0036] Construct a micro-expression emotion recognition model with residual neural network as feature extraction network, and collect micro-expression data samples into groups. RGB The climax frames and optical flow climax frames are input into the micro-expression emotion recognition model. For each group RGB Climax frame and optical flow climax frame are extracted by feature extraction network respectively RGB The facial features in the climax frame and the facial features in the optical flow climax frame are concatenated, and the group is classified according to the concatenated features. RGB The true emotions corresponding to the climax frames and the optical flow climax frames are classified, and the classification result is the group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame;

[0037] for any group RGB the real emotion recognition result corresponding to the climax frame and the optical flow climax frame, and compare it with the emotion classification label of the group RGB The real emotion recognition result corresponding to the climax frame and the optical flow climax frame, and compare it with the emotion classification label of the group

[0038] In an embodiment, the data preprocessing module comprises:

[0039] a micro-expression video frame sequence generation unit configured to extract a video frame sequence from the micro-expression video to be recognized as a micro-expression video frame sequence;

[0040] a micro-expression face image sequence generation unit configured to crop the micro-expression video frame sequence to obtain a micro-expression face image sequence;

[0041] a micro-expression time domain interpolation sequence generation unit configured to perform time domain interpolation on the micro-expression face image sequence to obtain a micro-expression time domain interpolation sequence;

[0042] a climax frame generation unit configured to extract a climax frame of the micro-expression time domain interpolation sequence as a RGB climax frame of the micro-expression video to be recognized, and calculate a dense optical flow between the starting frame and the climax frame of the micro-expression time domain interpolation sequence to obtain an optical flow climax frame of the micro-expression video to be recognized.

[0043] In a third aspect, an embodiment of the present application provides a computer device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the micro-expression-based emotion recognition method according to the first aspect when executing the computer program.

[0044] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the micro-expression-based emotion recognition method according to the first aspect.

[0045] Compared with the prior art, the embodiment of the present application has the beneficial effects that the embodiment of the present application improves the accuracy of micro-expression emotion recognition from two aspects in the process of micro-expression emotion recognition. In the first aspect, the embodiment of the present application comprehensively considers that the climax frame in a micro-expression video frame sequence has the maximum facial motion intensity and more facial information, and the optical flow information can reflect the subtle changes of the face, and extracts the RGBThe two types of climax frames, climax frames and optical flow climax frames, as the basis for true emotion recognition, can obtain more and more subtle facial features from the micro-expression video to be recognized, thereby improving the emotion recognition accuracy of micro-expression; in a second aspect, in order to solve the problem that the existing micro-expression dataset is limited and the sample number is small, and the model training is difficult, the data enhancement technology is used to increase the number of micro-expression videos in the micro-expression dataset by performing data enhancement processing on the micro-expression video in the micro-expression dataset RGB The number of climax frames and optical flow climax frames is increased, and the number of micro-expression videos in the micro-expression dataset is increased, so that the sample number is greatly increased, thereby improving the accuracy of the trained model and further improving the emotion recognition accuracy of micro-expression. RGB The number of climax frames and optical flow climax frames is increased, and the number of micro-expression videos in the micro-expression dataset is increased, so that the sample number is greatly increased, thereby improving the accuracy of the trained model and further improving the emotion recognition accuracy of micro-expression. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Fig. 1 is an application environment schematic diagram of a micro-expression-based emotion recognition method provided by an embodiment of the present application;

[0048] Fig. 2 is a flowchart of a micro-expression-based emotion recognition method provided by an embodiment of the present application;

[0049] Fig. 3 is a structure schematic diagram of a micro-expression emotion recognition model provided by an embodiment of the present application;

[0050] Fig. 4 is a structure schematic diagram of a micro-expression-based emotion recognition device provided by an embodiment of the present application;

[0051] Fig. 5 is a structure schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0052] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0053] It should be understood that the word “comprise” or variations such as “comprises” or “comprising”, when used in this specification and in the accompanying claims, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0054] It should also be understood that the term “and / or” when used in this specification and in the following claims is to be interpreted as open language, not as limiting, such that when used in a list of two or more items, the list is considered to contain each item individually, and the combination of each item in the list.

[0055] As used in this specification and in the claims, the terms “if’ and “when” can be interpreted to mean “upon” or “in response to a determination” or “in response to a detection” depending on the context. Similarly, the phrase “if it is determined” or “if [a described condition or event] is detected” can be interpreted to mean “upon a determination” or “in response to a determination” or “upon detecting [the described condition or event]” or “in response to detecting [the described condition or event]”, depending on the context.

[0056] In addition, the terms “first”, “second”, “third”, etc. as used in the description of the specification and the appended claims are merely used for distinguishing between similar underlying features, and do not imply or suggest any relative importance.

[0057] The terms “a or an” as used herein mean “one or more” or “at least one.” The terms “including” and / or “containing” as used herein are used in the sense of “including, but not limited to.” The term “or” as used herein is used in the sense of “and / or.” The phrase “associated with,” as used herein, means to have a constructive relationship with.

[0058] Embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Wherein, artificial intelligence (AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results.

[0059] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0060] It should be understood that the size of the serial number of each step in the following embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0061] In order to illustrate the technical solutions of the present application, the following will be illustrated by specific embodiments.

[0062] The emotion recognition method based on micro-expression provided by the embodiment of the present application can be applied in the application environment such as Fig. 1 , wherein the client and the server communicate. The client includes but is not limited to palmtop computer, desktop computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cloud computer device, personal digital assistant (PDA), etc. The server can be realized by an independent server or a server cluster composed of multiple servers.

[0063] Referring to Fig. 2 , it is a flowchart of the emotion recognition method based on micro-expression provided by the embodiment two of the present application, which is applied to the server in Fig. 1 , the computer device corresponding to the server is connected to the corresponding database to obtain the corresponding micro-expression video in the database. The computer device can also be connected to the corresponding client, which is operated by the user. The user can provide the corresponding micro-expression video to the server through the client. As shown in Fig. 2 , the emotion recognition method based on micro-expression can include the following steps:

[0064] Step S201, obtaining the micro-expression video to be identified;

[0065] In the present application, the micro-expression video can refer to any micro-expression video that needs to be identified. The micro-expression video is reported by the user, specifically, the user can upload the micro-expression video through the client, so as to be used for subsequent step execution. Of course, the file uploaded by the user can be stored in the corresponding database, and the server executing the method of the present application can call the micro-expression video from the database.

[0066] Step S202, data preprocessing is performed on the micro-expression video to be identified to extract the RGB Apex frame and optical flow apex frame;

[0067] Before extracting the facial features contained in the micro-expression video, data preprocessing needs to be performed on the micro-expression video to obtain the truly useful part for subsequent facial feature extraction in the micro-expression video.

[0068] In this application, considering that a micro-expression video frame sequence includes an onset frame, an apex frame and an offset frame, and the apex frame is the frame with the largest facial movement intensity among all frames and has more facial information, therefore, the apex frame in the micro-expression video frame sequence to be identified is first extracted, and in order to further capture subtle facial changes, the application extracts two types of apex frames, i.e., the apex frame and the optical flow apex frame. RGB Apex frame and optical flow apex frame.

[0069] Specifically, in this application, data preprocessing is performed on the micro-expression video to be identified to extract the RGB Apex frame and optical flow apex frame include:

[0070] (1) Extracting a video frame sequence from the micro-expression video to be identified as a micro-expression video frame sequence;

[0071] Extracting a corresponding video frame sequence from a video is prior art, which will not be described here.

[0072] (2) Face cropping is performed on the micro-expression video frame sequence to obtain a micro-expression face image sequence;

[0073] In this application, first, face positioning is performed on each frame of image in the micro-expression video frame sequence by using a Dlib vision library to determine the facial key feature points of each frame of image; then, according to the facial key feature points of each frame of image, the face of each frame of image is cropped, and the cropped face image is saved as a micro-expression face image sequence in the order of the micro-expression video frame sequence. Among them, the image resolution of the cropped face image is saved as 128x128.

[0074] Optionally, other micro-expression databases in prior art can also be used to perform face positioning on each frame of image in the micro-expression video frame sequence to determine the facial key feature points of each frame of image, for example, OpencvThe visual library and the like; in addition, the resolution of the cropped face image can be increased or decreased according to actual needs, for example, can be increased to 256*256, or decreased to 64*64, and the like, wherein appropriately increasing the resolution of the cropped face image can make it have higher clarity, which is beneficial to subsequent extraction of facial features.

[0075] (3) Time domain interpolation is performed on the micro-expression face image sequence to obtain a micro-expression time domain interpolation sequence;

[0076] Considering that the micro-expression duration is short, it is desirable to have a method that can extend the micro-expression duration while maintaining the micro-expression features, which is beneficial to subsequent stable extraction of facial features. Therefore, the micro-expression face image sequence is interpolated in the time domain to increase the number of images contained in the micro-expression face image sequence, thereby extending the duration of the micro-expression.

[0077] Specifically, the present application adopts TIM The interpolation algorithm interpolates the micro-expression face image sequence into 16 frames to obtain a 16-frame micro-expression time domain interpolation sequence.

[0078] Optionally, in actual application, the number of frames of the micro-expression face image sequence subjected to time domain interpolation can be increased or decreased according to actual needs, for example, can be increased to 32 frames, or decreased to 8 frames, and the like, wherein appropriately increasing the number of frames of time domain interpolation can further extend the duration of the micro-expression, which is more beneficial to subsequent stable extraction of facial features. In addition, in actual application, other time domain image interpolation algorithms in the prior art can be used to replace TIM The interpolation algorithm.

[0079] (4) Extracting the climax frame of the micro-expression time domain interpolation sequence as the RGB Climax frame of the micro-expression video to be identified, and calculating the dense optical flow between the starting frame and the climax frame of the micro-expression time domain interpolation sequence to obtain the optical flow climax frame of the micro-expression video to be identified.

[0080] Specifically, in the present application, the starting frame and the climax frame of the micro-expression time domain interpolation sequence are first extracted by the SMEConvNet network, and the specific extraction process is the prior art, which will not be described here. Then, the dense optical flow between the starting frame and the climax frame of the micro-expression time domain interpolation sequence is calculated by the Farneback algorithm to obtain the optical flow climax frame in the micro-expression video to be identified. The optical flow climax frame can further capture subtle facial changes, which is beneficial to subsequent extraction of facial features.

[0081] Optionally, in actual application, other starting frame and climax frame extraction methods in the prior art can also be used instead of SMEConvNet network, and other dense optical flow calculation methods in the prior art can also be used instead of Farneback algorithm, such as Horn-Schunck algorithm.

[0082] In step S203, the RGB climax frame and the optical flow climax frame of the micro-expression video to be identified are input into the pre-trained micro-expression emotion recognition model, and the real emotion corresponding to the micro-expression video to be identified is output.

[0083] In this application, the input of the pre-trained micro-expression emotion recognition model is the RGB climax frame and the optical flow climax frame of the micro-expression video to be identified, and the output is the real emotion corresponding to the micro-expression video, that is, the micro-expression emotion recognition model of this application determines the real emotion corresponding to the micro-expression video according to the RGB climax frame and the optical flow climax frame of the micro-expression video to be identified.

[0084] Wherein, the micro-expression emotion recognition model is trained by using a micro-expression data sample set, the micro-expression data sample set is a micro-expression video data set stored in the server by the medical unit, and the real emotion corresponding to each micro-expression video in the set has been labeled by an expert. When labeling, the real emotion corresponding to the micro-expression is divided into five categories, namely happiness, surprise, sadness, disgust and fear.

[0085] Specifically, the pre-trained micro-expression emotion recognition model is obtained by the following steps:

[0086] (1) Data preprocessing is performed on each micro-expression video in the micro-expression data sample set, and the RGB climax frame and the optical flow climax frame of each micro-expression video are extracted;

[0087] For convenience of description, the RGB climax frame and the optical flow climax frame are denoted as and respectively.

[0088] Wherein, the extraction method of the and of each micro-expression video in the micro-expression data sample set is referred to step S202, which will not be repeated here.

[0089] (2) The and Carrying out data enhancement processing to increase the number of micro-expression data sample sets in groups and ;

[0090] Since the number of samples in the micro-expression data sample set is small, the application uses a double-flow AC-GAN network to carry out data enhancement processing on each micro-expression video and , that is, two AC-GAN networks with the same structure are used to carry out data enhancement on and of the same micro-expression video respectively.

[0091] The network structure diagram of the AC-GAN network is shown in Fig. 3 The AC-GAN network is a variant of the ordinary generative adversarial network, and the main idea of the ordinary generative adversarial network comes from the idea of zero-sum game. In a deep learning neural network, it is through the continuous game between the generator G and the discriminator D to make the generator G learn the distribution of the data. The purpose of the generator G is to generate an image as real as possible, while the discriminator D tries to distinguish between real images and generated false images. The AC-GAN network adds an auxiliary classifier on the basis of the ordinary generative adversarial network, and its characteristics are that each generated sample has a corresponding class label z in addition to noise c .

[0092] The input of the discriminator of the AC-GAN network is an image, and the output is the probability distribution of the image on the source and the probability distribution on the class label: and , then for the input data:

[0093]

[0094] In the formula, the first term is the expectation when the input image is true and the discrimination result is true; the second term is the expectation when the input image is false and the discrimination result is false.

[0095] For the input label:

[0096]

[0097] In the formula, the first term is the expectation when the label of the input true image is judged to belong to a certain class c ; the second term is the expectation when the label of the input false image is judged to belong to a certain class c .

[0098] Wherein, the goal of the generator is maximization, and the goal of the discriminator is maximization.

[0099] In summary, this application uses two AC-GAN networks with the same structure to generate the same micro-expression video. and Data enhancement can achieve the following: Before data enhancement, the micro-expression video only corresponds to one set of and ; After data enhancement, in the original set of micro-expression videos and Based on this, multiple groups of different and , and each group and They all have corresponding emotion classification labels, which is equivalent to generating multiple different micro-expression videos, and each micro-expression video has an emotion classification label, thereby increasing the number of micro-expression videos in the micro-expression data sample set in disguise. The number of samples has been greatly increased, which is conducive to improving the accuracy of the training model.

[0100] (3) Construct a micro-expression emotion recognition model using a residual neural network as a feature extraction network (the model structure is as follows Fig. 3 As shown in Figure 2), the micro-expression data samples are grouped into and Input micro-expression emotion recognition model, for each group and , extracted by the feature extraction network Facial features and The facial features in the cascade are concatenated, and the group is and The corresponding real emotions are classified, and the classification result is the group and The corresponding real emotion recognition results.

[0101] In this application, ResNet10 is selected as the feature extraction network, such as Fig. 3 As shown, for each group and , two ResNet10 networks with the same structure are used to extract Facial features and The facial features in the , the extracted features are recorded as and , and cascade the extracted features. The features after cascading are:

[0102]

[0103] According to the cascaded features, the group and The corresponding true emotion is classified, and the classification result is the true emotion recognition result of the micro-expression video. and The corresponding true emotion recognition result.

[0104] Optionally, in actual application, other residual neural networks can also be selected as the feature extraction network, such as ResNet18, ResNet50, etc.

[0105] (4) For any group of and The corresponding true emotion recognition result is compared with the emotion classification label corresponding to the group and According to all the comparison results, the difference loss between the true emotion recognition result and the emotion classification label is calculated using a preset loss calculation function until the difference loss is minimized, and the model training is completed.

[0106] In this application, the preset loss calculation function is a cross-entropy loss function.

[0107] Optionally, in actual application, the micro-expression data set for model training can also be an open-source micro-expression data set in the prior art, such as the CASME II micro-expression data set, etc. Similarly, the true emotion corresponding to each micro-expression video in the open-source micro-expression data set has also been labeled.

[0108] In the medical scene, relying on the server carrying the above method, when true emotion recognition of a patient is needed, the micro-expression video of the patient is collected by a camera device, and the micro-expression video is sent to the server through a client, etc. The server executes the above method to process the micro-expression video to obtain an emotion recognition result, and sends the emotion recognition result to the client, so that doctors, patients, etc. can observe the emotion recognition result.

[0109] In summary, the present application improves the emotion recognition accuracy of micro-expression from two aspects. First, the present application considers that the climax frame in a micro-expression video frame sequence is the frame with the largest facial movement intensity, has more facial information, and the optical flow information can reflect the subtle changes of the face. By extracting two types of climax frames, i.e. the climax frame and the optical flow climax frame, of the micro-expression video to be recognized as the basis for true emotion recognition, more subtle facial features can be obtained from the micro-expression video to be recognized, thereby improving the emotion recognition accuracy of micro-expression; second, in order to solve the problem of limited micro-expression data set and small sample size, which makes model training difficult, the present application uses data augmentation technology to obtain more subtle facial features from the micro-expression video to be recognized. RGB RGB ​The climax frames and optical flow climax frames are enhanced to increase the number of micro-expression data samples. RGB The number of climax frames and optical flow climax frames increases the number of micro-expression videos in the micro-expression data sample set in disguise. The greatly increased number of samples can improve the accuracy of the training model and further improve the accuracy of emotion recognition of micro-expressions.

[0110] Corresponding to the emotion recognition method based on micro-expressions in the above embodiment, Fig. 4 The structure diagram of the emotion recognition device based on micro-expression provided in the third embodiment of the present application is shown. The emotion recognition device based on micro-expression is applied to Fig. 1 The server in the embodiment of the present invention is connected to a corresponding database by a computer device corresponding to the server to obtain the corresponding micro-expression video data in the database. The above-mentioned computer device can also be connected to a corresponding client, which is operated by a user and can provide the corresponding micro-expression video data to the server through the client. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0111] See also Fig. 4 , the emotion recognition device based on micro-expression includes:

[0112] A data acquisition module is used to obtain the micro-expression video to be recognized;

[0113] The data preprocessing module is used to preprocess the micro-expression video to be identified and extract the micro-expression video to be identified. RGB Climax frame and optical flow climax frame;

[0114] The real emotion recognition module is used to identify the micro-expression video RGB The climax frame and the optical flow climax frame are input into a pre-trained micro-expression emotion recognition model, which outputs the true emotion corresponding to the micro-expression video to be recognized. The pre-trained micro-expression emotion recognition model is obtained by the following steps:

[0115] Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame;

[0116] For each micro-expression video in the micro-expression data sample set RGB Data enhancement processing is performed on climax frames and optical flow climax frames to increase the number of micro-expression data samples and group them together. RGB The number of climax frames and optical flow climax frames;

[0117] Construct a micro-expression emotion recognition model with residual neural network as feature extraction network, and collect micro-expression data samples into groups. RGB The climax frames and optical flow climax frames are input into the micro-expression emotion recognition model. For each groupRGB climax frame and the optical flow climax frame are extracted by the feature extraction network respectively RGB the facial features in the climax frame and the facial features in the optical flow climax frame are concatenated, and the group RGB the real emotions corresponding to the climax frame and the optical flow climax frame are classified, and the classification result is the group RGB the real emotion recognition result corresponding to the climax frame and the optical flow climax frame;

[0118] For any group RGB the real emotion recognition result corresponding to the climax frame and the optical flow climax frame is compared with the group RGB the emotion classification label corresponding to the climax frame and the optical flow climax frame; according to all the comparison results, the difference loss between the real emotion recognition result and the emotion classification label is calculated using a preset loss calculation function, until the difference loss is minimized, and the model training is completed.

[0119] Optionally, the data preprocessing module comprises:

[0120] a micro-expression video frame sequence generation unit, configured to extract a video frame sequence from the micro-expression video to be recognized as the micro-expression video frame sequence;

[0121] a micro-expression face image sequence generation unit, configured to perform face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence;

[0122] a micro-expression time domain interpolation sequence generation unit, configured to perform time domain interpolation on the micro-expression face image sequence to obtain a micro-expression time domain interpolation sequence;

[0123] a climax frame generation unit, configured to extract a climax frame of the micro-expression time domain interpolation sequence as the climax frame of the micro-expression video to be recognized, and calculate a dense optical flow between a starting frame and the climax frame of the micro-expression time domain interpolation sequence to obtain an optical flow climax frame of the micro-expression video to be recognized. RGB

[0124] It should be noted that the information interaction between the above modules, the execution process and the like, since based on the same concept as the method embodiments of the present application, the specific functions and the technical effects brought by them can be referred to the method embodiment part, and will not be repeated here.

[0125] Fig. 5 A structural schematic diagram of a computer device provided in Embodiment Four of the present application. As shown in the figure, the computer device of this embodiment comprises at least one processor (CPU) Fig. 5 Fig. 5 ​​The computer device can include, but is not limited to, a processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the processor implementing the steps of any of the above micro-expression-based emotion recognition method embodiments when executing the computer program.

[0126] The computer device can include, but is not limited to, a processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the processor implementing the steps of any of the above micro-expression-based emotion recognition method embodiments when executing the computer program. Fig. 5 The computer device is merely an example and does not constitute a limitation on the computer device, the computer device can include more or less components than shown, or combine certain components, or include different components, for example, the computer device can also include a network interface, a display screen, an input device, etc.

[0127] The processor can be a CPU, and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0128] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be a memory of the computer device, and the internal memory provides an environment for the operation of the operating system and the computer readable instructions in the readable storage medium. The readable storage medium can be a hard disk of the computer device, and in other embodiments, the readable storage medium can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory can include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of a computer program, etc. The memory can also be used to temporarily store data that has been output or will be output.

[0129] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. If the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above-mentioned method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, computer readable medium cannot be electric carrier signal and telecommunication signal.

[0130] The above-mentioned embodiment methods can also be completed by a computer program product, when the computer program product runs on the computer equipment, so that the computer equipment can execute the steps in the above-mentioned method embodiments.

[0131] In the above-mentioned embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0132] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0133] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely schematic. The division of the modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0134] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0135] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for emotion recognition based on micro-expressions, characterized in that: The emotion recognition method based on micro-expression includes: Obtain the micro-expression video to be recognized; Perform data preprocessing on the micro-expression video to be identified, and extract the micro-expression video to be identified. RGB Climax frame and optical flow climax frame; The micro-expression video to be identified RGB The climax frame and the optical flow climax frame are input into a pre-trained micro-expression emotion recognition model, which outputs the true emotion corresponding to the micro-expression video to be recognized; wherein the pre-trained micro-expression emotion recognition model is obtained by the following steps: Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame; For each micro-expression video in the micro-expression data sample set RGB Data enhancement processing is performed on climax frames and optical flow climax frames to increase the number of micro-expression data samples and group them together. RGB The number of climax frames and optical flow climax frames; Construct a micro-expression emotion recognition model with residual neural network as feature extraction network, and collect micro-expression data samples into groups. RGB The climax frames and optical flow climax frames are input into the micro-expression emotion recognition model. For each group RGB Climax frame and optical flow climax frame are extracted by feature extraction network respectively RGB The facial features in the climax frame and the facial features in the optical flow climax frame are concatenated, and the group is classified according to the concatenated features. RGB The true emotions corresponding to the climax frames and the optical flow climax frames are classified, and the classification result is the group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame; For any group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame are compared with the group RGB The emotion classification labels corresponding to the climax frame and the optical flow climax frame are compared; based on all the comparison results, the preset loss calculation function is used to calculate the difference loss between the true emotion discrimination result and the emotion classification label until the difference loss is minimized and the model training is completed.

2. The emotion recognition method based on micro-expressions according to claim 1, characterized in that: The micro-expression data sample set for each micro-expression video RGB The data enhancement processing of climax frames and optical flow climax frames includes: using two AC-GAN networks with the same structure to perform data enhancement on each micro-expression video in the micro-expression data sample set. RGB Data enhancement is performed on climax frames and optical flow climax frames.

3. The emotion recognition method based on micro-expressions according to claim 2, characterized in that: Perform data preprocessing on the micro-expression video to be identified, and extract the micro-expression video to be identified. RGB Climax frames and optical flow climax frames include: Extracting a video frame sequence from the micro-expression video to be identified as a micro-expression video frame sequence; Performing face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence; Performing time-domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time-domain interpolation sequence; Extract the climax frame of the micro-expression temporal interpolation sequence as the micro-expression video to be recognized RGB The dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence is calculated to obtain the optical flow climax frame of the micro-expression video to be recognized.

4. The emotion recognition method based on micro-expressions according to claim 3, characterized in that: Performing face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence includes: First use Dlib Visual Library or Opencv The visual library locates the face of each frame image in the micro-expression video frame sequence and determines the facial key feature points of each frame image; then, based on the facial key feature points of each frame image, the face of each frame image is cropped, and the cropped face images are saved as a micro-expression face image sequence in the order of the micro-expression video frame sequence.

5. The method for emotion recognition based on micro-expressions according to claim 4, characterized in that: use TIM The interpolation algorithm performs time domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time domain interpolation sequence.

6. The emotion recognition method based on micro-expressions according to claim 5, characterized in that: The step of calculating the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence to obtain the optical flow climax frame of the micro-expression video to be identified comprises: First, the starting frame and climax frame of the micro-expression temporal interpolation sequence are extracted through the SMEConvNet network. Then, the dense optical flow between the starting frame and climax frame of the micro-expression temporal interpolation sequence is calculated through the Farneback algorithm to obtain the optical flow climax frame in the micro-expression video to be identified.

7. An emotion recognition device based on micro-expressions, characterized in that: The micro-expression-based emotion recognition device comprises: A data acquisition module is used to obtain the micro-expression video to be recognized; The data preprocessing module is used to preprocess the micro-expression video to be identified and extract the micro-expression data of the micro-expression video to be identified. RGB Climax frame and optical flow climax frame; The real emotion recognition module is used to identify the micro-expression video RGB The climax frame and the optical flow climax frame are input into a pre-trained micro-expression emotion recognition model, which outputs the true emotion corresponding to the micro-expression video to be recognized; wherein the pre-trained micro-expression emotion recognition model is obtained by the following steps: Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame; For each micro-expression video in the micro-expression data sample set RGB Data enhancement processing is performed on climax frames and optical flow climax frames to increase the number of micro-expression data samples and group them together. RGB The number of climax frames and optical flow climax frames; Construct a micro-expression emotion recognition model with residual neural network as feature extraction network, and collect micro-expression data samples into groups. RGB The climax frames and optical flow climax frames are input into the micro-expression emotion recognition model. For each group RGB Climax frame and optical flow climax frame are extracted by feature extraction network respectively RGB The facial features in the climax frame and the facial features in the optical flow climax frame are concatenated, and the group is classified according to the concatenated features. RGB The true emotions corresponding to the climax frames and the optical flow climax frames are classified, and the classification result is the group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame; For any group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame are compared with the group RGB The emotion classification labels corresponding to the climax frame and the optical flow climax frame are compared; based on all the comparison results, the preset loss calculation function is used to calculate the difference loss between the true emotion discrimination result and the emotion classification label until the difference loss is minimized and the model training is completed.

8. The micro-expression-based emotion recognition device according to claim 7, characterized in that: The data preprocessing module includes: a micro-expression video frame sequence generating unit, configured to extract a video frame sequence from the micro-expression video to be identified as a micro-expression video frame sequence; a micro-expression face image sequence generating unit, configured to perform face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence; a micro-expression time-domain interpolation sequence generating unit, configured to perform time-domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time-domain interpolation sequence; A climax frame generation unit is used to extract the climax frame of the micro-expression temporal interpolation sequence as the climax frame of the micro-expression video to be recognized. RGB The dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence is calculated to obtain the optical flow climax frame of the micro-expression video to be recognized.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the micro-expression-based emotion recognition method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the micro-expression-based emotion recognition method according to any one of claims 1 to 6 is implemented.

Images