Micro-expression-based emotion recognition method and device, equipment and medium

Abstract

The invention relates to the field of digital medical treatment, in particular to an emotion recognition method and device based on micro expressions, equipment and a medium. According to the method, two factors that the climax frames in a micro-expression video frame sequence contain more facial information and the optical flow information can reflect the subtle change of the face are comprehensively considered, and more and finer facial features are obtained from a micro-expression video to be recognized by extracting the two types of climax frames, namely, the RGB climax frames and the optical flow climax frames, so that the recognition accuracy of the micro-expression video to be recognized is improved. The emotion recognition accuracy of the micro-expression is improved; meanwhile, data enhancement processing is carried out on the RGB climax frames and the optical flow climax frames of each micro-expression video in the micro-expression data sample set, the number of the grouped RGB climax frames and the optical flow climax frames in the micro-expression data sample set is increased, and therefore the number of the micro-expression videos in the micro-expression data sample set is increased in a disguised mode. The number of samples is greatly increased, the accuracy of the training model can be improved, and the emotion recognition accuracy of micro-expressions is further improved.

Description

Technical Field

[0001] The present application is applicable to the field of digital medical technology, and in particular, relates to a method, device, equipment and medium for emotion recognition based on micro-expressions. Background Art

[0002] Micro-expressions (MES) are very brief, subtle and unconscious facial expressions. Compared with ordinary expressions, micro-expressions have the following main characteristics: (1) Short duration, usually only ; (2) The action intensity is low and difficult to detect; (3) Occurs unconsciously and is usually difficult to conceal or disguise; (4) Ordinary expressions can be analyzed in images, while analysis of micro-expressions usually requires video.

[0003] Since micro-expressions are produced unconsciously and are difficult to conceal or disguise, and are usually directly related to real emotions, micro-expressions are more reliable in emotion analysis. Currently in the medical field, identifying patients' real emotions based on micro-expressions is of great help in auxiliary diagnosis of patients' diseases and health management.

[0004] In the existing technology, emotional analysis of micro-expressions was initially done manually. In this case, only people with strong micro-expression analysis capabilities and rich experience in micro-expression analysis can obtain highly reliable emotion recognition results from micro-expression videos. However, cultivating such talents requires a lot of time and energy. Not only is the training difficult, but the efficiency of micro-expression analysis is also low.

[0005] With the rise of artificial intelligence technology, the application of machine learning and deep learning in micro-expression analysis has gradually increased. Although the efficiency of micro-expression analysis has improved to a certain extent, the current micro-expression emotion recognition work still has the following difficulties: (1) Micro-expressions are short in duration and low in intensity, and facial features are difficult to extract. It is impossible to capture enough and subtle facial information, resulting in low accuracy of emotion recognition based on micro-expressions; (2) The existing micro-expression datasets are limited and the number of samples is small. It is impossible to train a robust deep learning model from scratch, and model training is difficult. Therefore, how to extract more and more subtle facial information from micro-expressions and improve the accuracy of emotion recognition based on micro-expressions when the micro-expression dataset is limited and the number of samples is small has become an urgent problem to be solved. Summary of the invention

[0006] In view of this, the embodiments of the present application provide a micro-expression-based emotion recognition method, apparatus, device and medium to solve the problem of how to extract more and more subtle facial information from micro-expressions and improve the accuracy of emotion recognition of micro-expressions when the micro-expression data set is limited and the number of samples is small.

[0007] In a first aspect, an embodiment of the present application provides an emotion recognition method based on micro-expressions, and the emotion recognition method based on micro-expressions includes: Obtain the micro-expression video to be recognized; Perform data preprocessing on the micro-expression video to be identified, and extract the micro-expression video to be identified. RGB Climax frame and optical flow climax frame; The micro-expression video to be identified RGB The climax frame and the optical flow climax frame are input into a pre-trained micro-expression emotion recognition model, and the real emotion corresponding to the micro-expression video to be recognized is output; wherein the pre-trained micro-expression emotion recognition model is obtained by the following steps: Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame; For each micro-expression video in the micro-expression data sample set RGB Data enhancement processing is performed on climax frames and optical flow climax frames to increase the number of micro-expression data samples and group them together RGB The number of climax frames and optical flow climax frames; Construct a micro-expression emotion recognition model with residual neural network as feature extraction network, and collect micro-expression data samples into groups. RGB The climax frames and optical flow climax frames are input into the micro-expression emotion recognition model. For each group RGB Climax frames and optical flow climax frames are extracted by the feature extraction network respectively RGB The facial features in the climax frame and the facial features in the optical flow climax frame are concatenated, and the group is classified according to the concatenated features. RGB The real emotions corresponding to the climax frames and the optical flow climax frames are classified, and the classification result is the group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame; For any group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame are compared with the group RGB The emotion classification labels corresponding to the climax frame and the optical flow climax frame are compared; based on all the comparison results, the preset loss calculation function is used to calculate the difference loss between the true emotion discrimination result and the emotion classification label until the difference loss is minimized and the model training is completed.

[0008] In one embodiment, the micro-expression data sample set for each micro-expression video RGB The data enhancement processing of climax frames and optical flow climax frames includes: using two AC-GAN networks with the same structure to respectively enhance the climax frames of each micro-expression video in the micro-expression data sample set. RGB Data enhancement is performed on climax frames and optical flow climax frames.

[0009] In one embodiment, data preprocessing is performed on the micro-expression video to be identified, and the micro-expression video to be identified is extracted. RGB Climax frames and optical flow climax frames include: Extracting a video frame sequence from the micro-expression video to be identified as a micro-expression video frame sequence; Performing face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence; Performing time-domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time-domain interpolation sequence; Extract the climax frame of the micro-expression temporal interpolation sequence as the micro-expression video to be identified RGB The climax frame is obtained by calculating the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence, and obtaining the optical flow climax frame of the micro-expression video to be recognized.

[0010] In one embodiment, performing face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence includes: First use Dlib Visual Library or Opencv The visual library locates the face of each frame image in the micro-expression video frame sequence and determines the facial key feature points of each frame image; then, based on the facial key feature points of each frame image, the face of each frame image is cropped, and the cropped face images are saved as a micro-expression face image sequence in the order of the micro-expression video frame sequence.

[0011] In one embodiment, using TIM The interpolation algorithm performs time domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time domain interpolation sequence.

[0012] In one embodiment, the step of calculating the dense optical flow between the start frame and the climax frame of the micro-expression temporal interpolation sequence to obtain the optical flow climax frame of the micro-expression video to be identified comprises: Firstly, the starting frame and climax frame of the micro-expression temporal interpolation sequence are extracted through the SMEConvNet network; then, the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence is calculated through the Farneback algorithm to obtain the optical flow climax frame in the micro-expression video to be identified.

[0013] In a second aspect, an embodiment of the present application provides an emotion recognition device based on micro-expressions, and the emotion recognition device based on micro-expressions includes: A data acquisition module, used to obtain micro-expression videos to be identified; A data preprocessing module is used to perform data preprocessing on the micro-expression video to be identified, and extract the micro-expression video to be identified. RGB Climax frame and optical flow climax frame; A real emotion recognition module is used to convert the micro-expression video to be recognized RGB The climax frame and the optical flow climax frame are input into a pre-trained micro-expression emotion recognition model, and the real emotion corresponding to the micro-expression video to be recognized is output; wherein the pre-trained micro-expression emotion recognition model is obtained by the following steps: Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame; For each micro-expression video in the micro-expression data sample set RGB Data enhancement processing is performed on climax frames and optical flow climax frames to increase the number of micro-expression data samples and group them together RGB The number of climax frames and optical flow climax frames; Construct a micro-expression emotion recognition model with residual neural network as feature extraction network, and collect micro-expression data samples into groups. RGB The climax frames and optical flow climax frames are input into the micro-expression emotion recognition model. For each group RGB Climax frames and optical flow climax frames are extracted by the feature extraction network respectively RGB The facial features in the climax frame and the facial features in the optical flow climax frame are concatenated, and the group is classified according to the concatenated features. RGB The real emotions corresponding to the climax frames and the optical flow climax frames are classified, and the classification result is the group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame; For any group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame are compared with the group RGB The emotion classification labels corresponding to the climax frame and the optical flow climax frame are compared; based on all the comparison results, the preset loss calculation function is used to calculate the difference loss between the true emotion discrimination result and the emotion classification label until the difference loss is minimized and the model training is completed.

[0014] In one embodiment, the data preprocessing module includes: A micro-expression video frame sequence generating unit, used for extracting a video frame sequence from the micro-expression video to be identified as a micro-expression video frame sequence; A micro-expression face image sequence generating unit, used for performing face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence; A micro-expression time-domain interpolation sequence generating unit, used for performing time-domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time-domain interpolation sequence; A climax frame generation unit is used to extract the climax frame of the micro-expression temporal interpolation sequence as the climax frame of the micro-expression video to be identified. RGB The climax frame is obtained by calculating the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence, and obtaining the optical flow climax frame of the micro-expression video to be recognized.

[0015] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the micro-expression-based emotion recognition method as described in the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the micro-expression-based emotion recognition method as described in the first aspect is implemented.

[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects: in the process of emotion recognition of micro-expressions, the present application improves the accuracy of emotion recognition of micro-expressions from two aspects. First, the present application comprehensively considers the two factors that the climax frame in a micro-expression video frame sequence is the frame with the largest facial movement intensity and has more facial information, and the optical flow information can reflect the subtle changes in the face. RGB The two types of climax frames, climax frames and optical flow climax frames, as the basis for real emotion recognition, can obtain more and more subtle facial features from the micro-expression video to be recognized, thereby improving the accuracy of emotion recognition of micro-expressions; secondly, in order to solve the problem that the existing micro-expression data set is limited and the number of samples is small, and the model training is difficult, this application uses data enhancement technology to perform each micro-expression video in the micro-expression data sample set. RGB The climax frames and optical flow climax frames are enhanced to increase the number of micro-expression data samples. RGB The number of climax frames and optical flow climax frames increases the number of micro-expression videos in the micro-expression data sample set in disguise. The number of samples is greatly increased, which can improve the accuracy of the training model and further improve the accuracy of emotion recognition of micro-expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 This is a schematic diagram of an application environment of an emotion recognition method based on micro-expressions provided in Example 1 of the present application; Figure 2 This is a flow chart of an emotion recognition method based on micro-expressions provided in Example 2 of the present application; Figure 3 It is a structural schematic diagram of a micro-expression emotion recognition model provided in Example 2 of the present application; Figure 4 It is a structural schematic diagram of an emotion recognition device based on micro-expressions provided in Example 3 of the present application; Figure 5 It is a structural diagram of a computer device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0020] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0021] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0023] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0024] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0026] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0027] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0028] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0029] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0030] The present application provides an emotion recognition method based on micro-expressions, which can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs), and other computer devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0031] See also Figure 2 , is a flow chart of a method for emotion recognition based on micro-expressions provided in Example 2 of the present application. The method for emotion recognition based on micro-expressions is applied to Figure 1 The server in the embodiment of the present invention is connected to a corresponding database by a computer device corresponding to the server to obtain the corresponding micro-expression video in the database. The above-mentioned computer device can also be connected to a corresponding client, which is operated by a user. The user can provide the corresponding micro-expression video to the server through the client. Figure 2 As shown, the emotion recognition method based on micro-expression may include the following steps: Step S201, obtaining a micro-expression video to be recognized; In the present application, a micro-expression video may refer to any micro-expression video that requires real emotion recognition, and the micro-expression video is reported by the user. Specifically, the user can upload the micro-expression video through the client for use in the execution of subsequent steps. Of course, the file uploaded by the user can be stored in a corresponding database, and the server that executes the method of the present application can call the micro-expression video from the database.

[0032] Step S202: preprocess the micro-expression video to be identified, extract the micro-expression video to be identified, RGB Climax frame and optical flow climax frame; Before extracting the facial features contained in the micro-expression video, it is necessary to perform data preprocessing on the micro-expression video, and obtain the part of the micro-expression video that is truly useful for the subsequent extraction of facial features through data preprocessing.

[0033] In this application, considering that a micro-expression video frame sequence includes an onset frame (Onset), a climax frame (Apex) and an end frame (Offset), and the climax frame is the frame with the largest facial motion intensity among all frames and has more facial information, therefore, this application first extracts the climax frame in the micro-expression video frame sequence to be identified, and in order to further capture subtle facial changes, this application extracts RGBThere are two types of climax frames: climax frames and optical flow climax frames.

[0034] Specifically, in this application, data preprocessing is performed on the micro-expression video to be identified, and the micro-expression video to be identified is extracted. RGB Climax frames and optical flow climax frames include: (1) Extracting a video frame sequence from the micro-expression video to be identified as a micro-expression video frame sequence; Extracting a corresponding video frame sequence from a video is a prior art and will not be described in detail here.

[0035] (2) Perform face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence; In this application, first use Dlib The visual library locates the face of each frame in the micro-expression video frame sequence and determines the key facial feature points of each frame. Then, based on the key facial feature points of each frame, the face of each frame is cropped and the cropped face images are saved as a micro-expression face image sequence in the order of the micro-expression video frame sequence. The image resolution of the cropped face images is saved as 128×128.

[0036] Optionally, other micro-expression databases in the prior art can also be used to locate the face of each frame in the micro-expression video frame sequence to determine the key facial feature points of each frame, for example, Opencv visual library, etc.; in addition, the resolution of the cropped face image can be increased or decreased according to actual needs. For example, it can be increased to 256×256, or decreased to 64×64, etc. Among them, appropriately improving the resolution of the cropped face image can make it have higher clarity, which is conducive to the subsequent extraction of facial features.

[0037] (3) Performing time domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time domain interpolation sequence; Taking into account the short duration of micro-expressions, it is hoped that there is a method that can prolong the duration of micro-expressions while maintaining the characteristics of micro-expressions, which is conducive to the stable extraction of subsequent facial features. Therefore, the present application increases the number of images contained in the micro-expression face image sequence by interpolating the micro-expression face image sequence in the time domain, thereby extending the duration of micro-expressions in disguise.

[0038] Specifically, this application adopts TIM The interpolation algorithm interpolates the micro-expression facial image sequence into 16 frames to obtain a 16-frame micro-expression time-domain interpolation sequence.

[0039] Optionally, in actual application, the number of frames for time domain interpolation of micro-expression facial image sequences can be increased or decreased according to actual needs, for example, it can be increased to 32 frames, or decreased to 8 frames, etc., wherein, appropriately increasing the number of frames for time domain interpolation can further extend the duration of micro-expressions, which is more conducive to the stable extraction of subsequent facial features. In addition, in actual application, other time domain image interpolation algorithms in the prior art can also be used to replace TIM Interpolation algorithm.

[0040] (4) Extract the climax frame of the micro-expression temporal interpolation sequence as the micro-expression video to be identified. RGB The climax frame is obtained by calculating the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence, and obtaining the optical flow climax frame of the micro-expression video to be recognized.

[0041] Specifically, in this application, the starting frame and climax frame of the micro-expression time domain interpolation sequence are first extracted through the SMEConvNet network. The specific extraction process is the existing technology and will not be repeated here; then the dense optical flow between the starting frame and the climax frame of the micro-expression time domain interpolation sequence is calculated by the Farneback algorithm to obtain the optical flow climax frame in the micro-expression video to be identified. The optical flow climax frame can further capture subtle facial changes, which is conducive to the subsequent extraction of facial features.

[0042] Optionally, in actual application, other starting frame and climax frame extraction methods in the prior art can be used to replace the SMEConvNet network, and other dense optical flow calculation methods in the prior art can be used to replace the Farneback algorithm, such as the Horn-Schunck algorithm.

[0043] Step S203: The micro-expression video to be identified RGB The climax frames and optical flow climax frames are input into a pre-trained micro-expression emotion recognition model, and the real emotion corresponding to the micro-expression video to be recognized is output.

[0044] In this application, for the pre-trained micro-expression emotion recognition model, its input is the micro-expression video that needs to be recognized for real emotions. RGB The climax frame and the optical flow climax frame are output as the real emotion corresponding to the micro-expression video. In other words, the micro-expression emotion recognition model of this application is based on the micro-expression video to be identified. RGB Climax frames and optical flow climax frames are used to determine the corresponding real emotions.

[0045] Among them, the micro-expression data sample set is used to train the micro-expression emotion recognition model. The micro-expression data sample set is a collection of micro-expression video data that has been stored in the server by medical units, and the real emotions corresponding to each micro-expression video in the set have been labeled by experts. During the labeling process, the real emotions corresponding to the micro-expressions are divided into five categories, namely happiness, surprise, sadness, disgust, and fear.

[0046] Specifically, the pre-trained micro-expression emotion recognition model is obtained through the following steps: (1) Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame; For ease of description, the following RGB The climax frame and the optical flow climax frame are respectively denoted as , .

[0047] Among them, each micro-expression video in the micro-expression data sample set and For the extraction method, refer to step S202 and will not be repeated here.

[0048] (2) For each micro-expression video in the micro-expression data sample set and Perform data enhancement processing to increase the number of micro-expression data samples and group them together and the number of Since the number of samples in the micro-expression data sample set is small, this application uses a two-stream AC-GAN network to generate the and To perform data enhancement processing, two AC-GAN networks with the same structure are used to perform data enhancement processing on the same micro-expression video. and Perform data augmentation.

[0049] Among them, the network structure diagram of AC-GAN network is as follows Figure 3 As shown in the figure, the AC-GAN network is a variant of the ordinary generative adversarial network. The main idea of ​​the ordinary generative adversarial network comes from the idea of ​​zero-sum game. In deep learning neural networks, the generator G and the discriminator D are constantly competing with each other, so that the generator G can learn the distribution of data. The purpose of the generator G is to generate an image that is as realistic as possible, while the discriminator D tries to distinguish between real images and generated fake images. The AC-GAN network adds an auxiliary classifier to the ordinary generative adversarial network. Its characteristic is that each generated sample has no noise.z There is also a corresponding category label c .

[0050] The input of the discriminator of the AC-GAN network is an image, and the output is the probability distribution of the image over the source and the probability distribution over the class label: and , then for the input data: Wherein, the first term is the expectation that the discrimination result is true when the input image is true; the second term is the expectation that the discrimination result is false when the input image is false.

[0051] For input tags: In the formula, the first term is when the label of the input true image is judged to belong to a certain category c The second term is when the label of the input fake image is judged to belong to a certain category c expectations.

[0052] The goal of the generator is The maximum value is the discriminator. maximum.

[0053] In summary, this application uses two AC-GAN networks with the same structure to respectively generate the and Data enhancement can achieve the following: Before data enhancement, the micro-expression video only corresponds to one set of and ; After data enhancement, in the original set of micro-expression videos and Based on this, multiple groups of different and , and each group and They all carry corresponding emotion classification labels, which is equivalent to generating multiple different micro-expression videos, and each micro-expression video has an emotion classification label, thereby increasing the number of micro-expression videos in the micro-expression data sample set in disguise. The number of samples has been greatly increased, which is conducive to improving the accuracy of the training model.

[0054] (3) Construct a micro-expression emotion recognition model using a residual neural network as a feature extraction network (the model structure is as follows Figure 3 As shown in Figure 2), the micro-expression data samples are grouped into and Input micro-expression emotion recognition model, for each group and , extracted by the feature extraction network The facial features and The facial features in the group are concatenated and the two are concatenated. and The corresponding real emotions are classified, and the classification result is the group and The corresponding real emotion recognition results.

[0055] In this application, ResNet10 is selected as the feature extraction network. Figure 3 As shown, for each group and , two ResNet10 networks with the same structure are used to extract The facial features and The facial features in the image are recorded as and , and cascade the extracted features. The features after cascading are: According to the cascaded features, and The corresponding real emotions are classified, and the classification result is the group and The corresponding real emotion recognition results.

[0056] Optionally, in actual applications, other residual neural networks can be used as feature extraction networks, such as ResNet18, ResNet50, etc.

[0057] (4) For any group and The corresponding real emotion recognition results are compared with the group and The corresponding emotion classification labels are compared; based on all the comparison results, the preset loss calculation function is used to calculate the difference loss between the true emotion discrimination result and the emotion classification label until the difference loss is minimized and the model training is completed.

[0058] In this application, the preset loss calculation function is the cross entropy loss function.

[0059] Optionally, in actual application, the micro-expression dataset for model training can also be an open source micro-expression dataset in the prior art, such as the CASME II micro-expression dataset. Similarly, the real emotion corresponding to each micro-expression video in the open source micro-expression dataset has also been labeled.

[0060] When used in a medical scenario, relying on the server that carries the above method, when it is necessary to perform real emotion recognition on the patient, the patient's micro-expression video is collected through a camera device, and the micro-expression video is sent to the server through the client, etc. The server executes the above method to process the micro-expression video to obtain an emotion recognition result, and sends the emotion recognition result to the client, so that doctors, patients, etc. can observe the emotion recognition result.

[0061] In summary, the present application improves the accuracy of emotion recognition of micro-expressions from two aspects. First, the present application comprehensively considers the two factors that the climax frame in a micro-expression video frame sequence is the frame with the largest facial movement intensity and has more facial information, and the optical flow information can reflect the subtle changes in the face. RGB The two types of climax frames, climax frames and optical flow climax frames, as the basis for real emotion recognition, can obtain more and more subtle facial features from the micro-expression video to be recognized, thereby improving the accuracy of emotion recognition of micro-expressions; secondly, in order to solve the problem that the existing micro-expression data set is limited and the number of samples is small, and the model training is difficult, this application uses data enhancement technology to perform each micro-expression video in the micro-expression data sample set. RGB The climax frames and optical flow climax frames are enhanced to increase the number of micro-expression data samples. RGB The number of climax frames and optical flow climax frames increases the number of micro-expression videos in the micro-expression data sample set in disguise. The number of samples is greatly increased, which can improve the accuracy of the training model and further improve the accuracy of emotion recognition of micro-expressions.

[0062] Corresponding to the emotion recognition method based on micro-expressions in the above embodiment, Figure 4 The structure block diagram of the emotion recognition device based on micro-expression provided in the third embodiment of the present application is shown. The emotion recognition device based on micro-expression is applied to Figure 1 The server in the embodiment of the present invention is connected to a corresponding database by a computer device corresponding to the server, so as to obtain the corresponding micro-expression video data in the database. The above-mentioned computer device can also be connected to a corresponding client, which is operated by a user, and the user can provide the corresponding micro-expression video data to the server through the client. For the convenience of explanation, only the part related to the embodiment of the present application is shown.

[0063] See also Figure 4 , the emotion recognition device based on micro-expression comprises: A data acquisition module, used to obtain micro-expression videos to be identified; The data preprocessing module is used to preprocess the micro-expression video to be identified and extract the micro-expression video to be identified. RGB Climax frame and optical flow climax frame; The real emotion recognition module is used to identify the micro-expression video RGB The climax frame and the optical flow climax frame are input into a pre-trained micro-expression emotion recognition model, and the real emotion corresponding to the micro-expression video to be recognized is output; wherein the pre-trained micro-expression emotion recognition model is obtained by the following steps: Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame; For each micro-expression video in the micro-expression data sample set RGB Data enhancement processing is performed on climax frames and optical flow climax frames to increase the number of micro-expression data samples and group them together RGB The number of climax frames and optical flow climax frames; Construct a micro-expression emotion recognition model with residual neural network as feature extraction network, and collect micro-expression data samples into groups. RGB The climax frames and optical flow climax frames are input into the micro-expression emotion recognition model. For each group RGB Climax frames and optical flow climax frames are extracted by the feature extraction network respectively RGB The facial features in the climax frame and the facial features in the optical flow climax frame are concatenated, and the group is classified according to the concatenated features. RGB The real emotions corresponding to the climax frames and the optical flow climax frames are classified, and the classification result is the group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame; For any group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame are compared with the group RGB The emotion classification labels corresponding to the climax frame and the optical flow climax frame are compared; based on all the comparison results, the preset loss calculation function is used to calculate the difference loss between the true emotion discrimination result and the emotion classification label until the difference loss is minimized and the model training is completed.

[0064] Optionally, the data preprocessing module includes: A micro-expression video frame sequence generating unit is used to extract a video frame sequence from the micro-expression video to be identified as a micro-expression video frame sequence; A micro-expression face image sequence generation unit is used to perform face cropping on a micro-expression video frame sequence to obtain a micro-expression face image sequence; A micro-expression time-domain interpolation sequence generation unit is used to perform time-domain interpolation on a micro-expression facial image sequence to obtain a micro-expression time-domain interpolation sequence; The climax frame generation unit is used to extract the climax frame of the micro-expression temporal interpolation sequence as the climax frame of the micro-expression video to be recognized. RGBThe climax frame is obtained by calculating the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence, and obtaining the optical flow climax frame of the micro-expression video to be recognized.

[0065] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0066] Figure 5 This is a schematic diagram of the structure of a computer device provided in Example 4 of the present application. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned micro-expression-based emotion recognition method embodiments are implemented.

[0067] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 5 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0068] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0069] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0070] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0071] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing the computer program product.

[0072] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0073] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0074] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0076] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for emotion recognition based on micro-expressions, characterized in that: The emotion recognition method based on micro-expression comprises: Obtain the micro-expression video to be recognized; Perform data preprocessing on the micro-expression video to be identified, and extract the micro-expression video to be identified. RGB Climax frame and optical flow climax frame; The micro-expression video to be identified RGB The climax frame and the optical flow climax frame are input into a pre-trained micro-expression emotion recognition model, and the real emotion corresponding to the micro-expression video to be recognized is output; wherein the pre-trained micro-expression emotion recognition model is obtained by the following steps: Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame; For each micro-expression video in the micro-expression data sample set RGB Data enhancement processing is performed on climax frames and optical flow climax frames to increase the number of micro-expression data samples and group them together RGB The number of climax frames and optical flow climax frames; Construct a micro-expression emotion recognition model with residual neural network as feature extraction network, and collect micro-expression data samples into groups. RGB The climax frames and optical flow climax frames are input into the micro-expression emotion recognition model. For each group RGB Climax frames and optical flow climax frames are extracted by the feature extraction network respectively RGB The facial features in the climax frame and the facial features in the optical flow climax frame are concatenated, and the group is classified according to the concatenated features. RGB The real emotions corresponding to the climax frames and the optical flow climax frames are classified, and the classification result is the group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame; For any group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame are compared with the group RGB The emotion classification labels corresponding to the climax frame and the optical flow climax frame are compared; based on all the comparison results, the preset loss calculation function is used to calculate the difference loss between the true emotion discrimination result and the emotion classification label until the difference loss is minimized and the model training is completed.

2. The method for emotion recognition based on micro-expressions according to claim 1, characterized in that: The micro-expression data sample set for each micro-expression video RGB The data enhancement processing of climax frames and optical flow climax frames includes: using two AC-GAN networks with the same structure to respectively enhance the climax frames of each micro-expression video in the micro-expression data sample set. RGB Data enhancement is performed on climax frames and optical flow climax frames.

3. The method for emotion recognition based on micro-expressions according to claim 2, characterized in that: Perform data preprocessing on the micro-expression video to be identified, and extract the micro-expression video to be identified. RGB Climax frames and optical flow climax frames include: Extracting a video frame sequence from the micro-expression video to be identified as a micro-expression video frame sequence; Performing face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence; Performing time-domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time-domain interpolation sequence; Extract the climax frame of the micro-expression temporal interpolation sequence as the micro-expression video to be identified RGB The climax frame is obtained by calculating the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence, and obtaining the optical flow climax frame of the micro-expression video to be recognized.

4. The method for emotion recognition based on micro-expressions according to claim 3, characterized in that: Performing face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence includes: First use Dlib Visual Library or Opencv The visual library locates the face of each frame image in the micro-expression video frame sequence and determines the facial key feature points of each frame image; then, based on the facial key feature points of each frame image, the face of each frame image is cropped, and the cropped face images are saved as a micro-expression face image sequence in the order of the micro-expression video frame sequence.

5. The method for emotion recognition based on micro-expressions according to claim 4, characterized in that: use TIM The interpolation algorithm performs time domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time domain interpolation sequence.

6. The method for emotion recognition based on micro-expressions according to claim 5, characterized in that: The step of calculating the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence to obtain the optical flow climax frame of the micro-expression video to be identified comprises: Firstly, the starting frame and climax frame of the micro-expression temporal interpolation sequence are extracted through the SMEConvNet network; then, the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence is calculated through the Farneback algorithm to obtain the optical flow climax frame in the micro-expression video to be identified.

7. An emotion recognition device based on micro-expressions, characterized in that: The micro-expression-based emotion recognition device comprises: A data acquisition module, used to obtain micro-expression videos to be identified; A data preprocessing module is used to perform data preprocessing on the micro-expression video to be identified, and extract the micro-expression video to be identified. RGB Climax frame and optical flow climax frame; A real emotion recognition module is used to convert the micro-expression video to be recognized RGB The climax frame and the optical flow climax frame are input into a pre-trained micro-expression emotion recognition model, and the real emotion corresponding to the micro-expression video to be recognized is output; wherein the pre-trained micro-expression emotion recognition model is obtained by the following steps: Perform data preprocessing on each micro-expression video in the micro-expression data sample set and extract the RGB Climax frame and optical flow climax frame; For each micro-expression video in the micro-expression data sample set RGB Data enhancement processing is performed on climax frames and optical flow climax frames to increase the number of micro-expression data samples and group them together RGB The number of climax frames and optical flow climax frames; Construct a micro-expression emotion recognition model with residual neural network as feature extraction network, and collect micro-expression data samples into groups. RGB The climax frames and optical flow climax frames are input into the micro-expression emotion recognition model. For each group RGB Climax frames and optical flow climax frames are extracted by the feature extraction network respectively RGB The facial features in the climax frame and the facial features in the optical flow climax frame are concatenated, and the group is classified according to the concatenated features. RGB The real emotions corresponding to the climax frames and the optical flow climax frames are classified, and the classification result is the group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame; For any group RGB The real emotion recognition results corresponding to the climax frame and the optical flow climax frame are compared with the group RGB The emotion classification labels corresponding to the climax frame and the optical flow climax frame are compared; based on all the comparison results, the preset loss calculation function is used to calculate the difference loss between the true emotion discrimination result and the emotion classification label until the difference loss is minimized and the model training is completed.

8. The micro-expression based emotion recognition device according to claim 7, characterized in that: The data preprocessing module comprises: A micro-expression video frame sequence generating unit, used for extracting a video frame sequence from the micro-expression video to be identified as a micro-expression video frame sequence; A micro-expression face image sequence generating unit, used for performing face cropping on the micro-expression video frame sequence to obtain a micro-expression face image sequence; A micro-expression time-domain interpolation sequence generating unit, used for performing time-domain interpolation on the micro-expression facial image sequence to obtain a micro-expression time-domain interpolation sequence; A climax frame generation unit is used to extract the climax frame of the micro-expression temporal interpolation sequence as the climax frame of the micro-expression video to be identified. RGB The climax frame is obtained by calculating the dense optical flow between the starting frame and the climax frame of the micro-expression temporal interpolation sequence, and obtaining the optical flow climax frame of the micro-expression video to be recognized.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the micro-expression-based emotion recognition method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the micro-expression based emotion recognition method as described in any one of claims 1 to 6 is implemented.

Images