Question and answer emotion evaluation method and device based on action unit au and micro-expression

Abstract

The application discloses a kind of based on action unit AU and micro-expression question and answer mood evaluation method and device, obtain the question and answer video of user to be detected, the first image about non-basic question and answer in question and answer video is input into face detection model and obtains first face image, first face image is input into the multi-task identification model based on AU and micro-expression, output multiple first AU scores and multiple first micro-expression scores of first face image. By the corresponding relationship between multiple AU, multiple micro-expression and multiple emotional parameters, multiple first AU scores and multiple first micro-expression scores are weighted to obtain multiple target emotional values corresponding to multiple emotional parameters. Compare multiple target emotional values with multiple basic emotional values corresponding to multiple emotional parameters to obtain comparison result. The question and answer mood of user to be detected is evaluated by comparison result. In this way, the analysis standard of question and answer mood is set, and the fairness and credibility of evaluating question and answer mood are improved.

Description

Technical Field

[0001] This application relates to the field of micro-expression recognition technology, and in particular to a question-and-answer emotion assessment method and apparatus based on action unit (AU) and micro-expressions. Background Technology

[0002] With the development of modern society, people are paying more and more attention to the research of Action Units (AUs) and micro-expressions. Micro-expressions are a kind of behavior that is directly reflected by a person's emotional response. Facial expressions can also be regarded as a combination of different facial motion units defined by a facial expression coding system. Facial action units and micro-expressions can be captured and analyzed by machines, and people are applying this technology to various scenarios.

[0003] During the question-and-answer process, video recordings of the user's responses are used. Machine analysis of the user's micro-expressions and facial movement units is then employed to determine whether the user understands the question, responds actively or passively. This method eliminates human interference and helps the questioner better understand the user's reaction to the question.

[0004] However, the method of using machine analysis of user activity (AU) and micro-expressions to judge the emotional state of users in question-and-answer sessions lacks a unified standard, cannot provide an effective reference for users, and has low reference value and poor fairness. Summary of the Invention

[0005] In view of this, embodiments of this application provide a question-and-answer emotion assessment method and apparatus based on action unit (AU) and micro-expression, aiming to improve the fairness of assessing question-and-answer emotions.

[0006] In a first aspect, embodiments of this application provide a question-and-answer emotion assessment method based on action units (AUs) and micro-expressions, the method comprising:

[0007] Obtain the question-and-answer video of the user to be tested;

[0008] The first face image is obtained by detecting the first image related to non-basic questions and answers in the question-and-answer video using a face detection model;

[0009] The first face image is identified by a multi-task recognition model based on AU and micro-expression, and multiple first AU scores and multiple first micro-expression scores of the first face image are obtained.

[0010] Based on the correspondence between multiple AUs, multiple micro-expressions and multiple emotion parameters, the scores of the multiple first AUs and the scores of the multiple first micro-expressions are weighted and calculated to obtain multiple target emotion values ​​corresponding to the multiple emotion parameters.

[0011] comparing the plurality of target emotion values and a plurality of basic emotion values corresponding to the plurality of emotion parameters, obtaining a comparison result, the plurality of basic emotion values being obtained by performing emotion analysis on a second image related to a basic question and answer in the question and answer video based on the plurality of emotion parameters;

[0012] evaluating the question and answer emotion of the to-be-detected user according to the comparison result.

[0013] Optionally, the obtaining step of the plurality of basic emotion values comprises:

[0014] detecting, by the face detection model, the second image related to the basic question and answer in the question and answer video to obtain a second face image;

[0015] recognizing, by the multi-task recognition model based on AU and micro-expression, the second face image to obtain a plurality of second AU scores and a plurality of second micro-expression scores of the second face image;

[0016] According to the correspondence, the plurality of second AU scores and the plurality of second micro-expression scores are weighted and calculated to obtain the plurality of basic emotion values.

[0017] Optionally, the plurality of emotion parameters comprises a positive emotion parameter and a negative emotion parameter.

[0018] Optionally, the detecting, by the face detection model, the first image related to the non-basic question and answer in the question and answer video to obtain the first face image comprises:

[0019] detecting, by the face detection model, the first image to obtain first position information of a first target face frame and second position information of a preset face key point in the first target face frame;

[0020] cropping the first image according to the first position information and the second position information to obtain the first face image.

[0021] Optionally, the training step of the face detection model comprises:

[0022] pre-training a first detection network according to a first training face image in a first training image set and first face frame annotation data of the first training face image to obtain a first pre-training model;

[0023] fine-tuning training the first pre-training model according to a second training face image in a second training image set and second face frame annotation data and face key point annotation data of the second training face image to obtain the face detection model.

[0024] Optionally, the fine-tuning training of the first pre-training model according to the second training face image in the second training image set and the second face frame annotation data and the face key point annotation data of the second training face image obtains the face detection model, and the fine-tuning training of the first pre-training model according to the second training face image in the second training image set and the second face frame annotation data and the face key point annotation data of the second training face image comprises:

[0025] detecting the second training face image by the first pre-training model to obtain face frame prediction data and face key point prediction data of the second training face image;

[0026] fine-tuning model parameters of the first pre-training model according to the second face frame annotation data, the face key point annotation data, the face frame prediction data, the face key point prediction data and the first preset loss function;

[0027] determining the first pre-training model after the fine-tuning training as the face detection model.

[0028] Optionally, the training step of the multi-task recognition model based on AU and micro-expression comprises:

[0029] pre-training a multi-task recognition network according to a third training face image in a third training image set and face feature annotation data of the third training face image to obtain a second pre-training model;

[0030] fine-tuning training of the second pre-training model according to a fourth training face image in a fourth training image set, AU annotation data of the fourth training face image, a fifth training face image in a fifth training image set and micro-expression annotation data of the fifth training face image obtains the multi-task recognition model based on AU and micro-expression.

[0031] Optionally, the fine-tuning training of the second pre-training model according to the fourth training face image in the fourth training image set, the AU annotation data of the fourth training face image, the fifth training face image in the fifth training image set and the micro-expression annotation data of the fifth training face image obtains the multi-task recognition model based on AU and micro-expression, and the fine-tuning training of the second pre-training model according to the fourth training face image in the fourth training image set, the AU annotation data of the fourth training face image, the fifth training face image in the fifth training image set and the micro-expression annotation data of the fifth training face image comprises:

[0032] extracting features of the fourth training face image and the fifth training face image by a feature extraction layer in the second pre-training model respectively to obtain first image features of the fourth training face image and second image features of the fifth training face image;

[0033] performing convolution processing on the first image features and the second image features by a convolution layer in the second pre-training model respectively to obtain first convolution features and second convolution features;

[0034] detecting the first convolutional feature through an AU detection layer in the second pre-training model to obtain AU prediction data of the fourth training face image;

[0035] According to the AU annotation data, the AU prediction data, and the second preset loss function, the model parameters of the feature extraction layer, the convolutional layer, and the AU detection layer in the second pre-training model are fine-tuned and trained.

[0036] detecting the second convolutional feature through a micro-expression detection layer in the second pre-training model to obtain micro-expression prediction data of the fifth training face image;

[0037] According to the micro-expression annotation data, the micro-expression prediction data, and the third preset loss function, the model parameters of the feature extraction layer, the convolutional layer, and the micro-expression detection layer in the second pre-training model are fine-tuned and trained.

[0038] The second pre-training model after fine-tuning and training is determined as the multi-task recognition model based on AU and micro-expression.

[0039] In a second aspect, an emotion evaluation device for question and answer based on action units (AUs) and micro-expressions is provided, and the device includes:

[0040] An acquisition module is configured to acquire a question and answer video of a user to be detected.

[0041] A first detection module is configured to detect a first image related to a non-basic question and answer in the question and answer video through a face detection model to obtain a first face image.

[0042] A first recognition module is configured to recognize the first face image through a multi-task recognition model based on AUs and micro-expressions to obtain a plurality of first AU scores and a plurality of first micro-expression scores of the first face image.

[0043] A first calculation module is configured to perform weighted calculation on the plurality of first AU scores and the plurality of first micro-expression scores according to a corresponding relationship between a plurality of AUs, a plurality of micro-expressions, and a plurality of emotion parameters to obtain a plurality of target emotion values corresponding to the plurality of emotion parameters.

[0044] A comparison module is configured to compare the plurality of target emotion values with a plurality of basic emotion values corresponding to the plurality of emotion parameters to obtain a comparison result, wherein the plurality of basic emotion values are obtained by performing emotion analysis based on the plurality of emotion parameters on a second image related to a basic question and answer in the question and answer video.

[0045] An evaluation module is configured to evaluate a question and answer emotion of the user to be detected according to the comparison result.

[0046] In a third aspect, an emotion evaluation device for question and answer based on action units (AUs) and micro expressions is provided, and the device comprises:

[0047] a memory for storing a computer program;

[0048] a processor for executing the computer program to enable the device to perform the emotion evaluation method for question and answer based on AUs and micro expressions according to the first aspect.

[0049] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, which, when executed by a processor, implements the emotion evaluation method for question and answer based on AUs and micro expressions according to the first aspect.

[0050] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages:

[0051] The embodiments of the present application provide an emotion evaluation method and device for question and answer based on AUs and micro expressions. First, a question and answer video of a user to be detected is obtained, a first image about a non-basic question and answer in the question and answer video is input into a face detection model for detection to obtain a first face image, and the first face image is input into a multi-task recognition model of AUs and micro expressions to output scores of a plurality of first AUs and scores of a plurality of first micro expressions. Then, a plurality of target emotion values corresponding to a plurality of emotion parameters are obtained by weighted calculation of the plurality of first AU scores and the plurality of first micro expression scores according to a corresponding relationship between the plurality of AUs, the plurality of micro expressions and the plurality of emotion parameters. Finally, a comparison result is obtained by comparing the plurality of target emotion values with a plurality of basic emotion values, and the plurality of basic emotion values are obtained by emotion analysis based on emotion parameters according to a second image about a basic question and answer in the question and answer video of the user. The question and answer emotion of the user to be detected is evaluated according to the comparison result.

[0052] As can be seen, on the basis of obtaining the plurality of AU scores and the plurality of micro expression scores of the first image about the non-basic question and answer through the multi-task recognition model of AUs and micro expressions, the plurality of target emotion values corresponding to the plurality of emotion parameters are further obtained by weighted calculation according to the corresponding relationship between the plurality of AUs, the plurality of micro expressions and the plurality of emotion parameters, and compared with the plurality of basic emotion values corresponding to the plurality of emotion parameters, so as to evaluate the question and answer emotion of the user to be detected. This way sets the analysis standard of the question and answer emotion, and improves the fairness and credibility of evaluating the question and answer emotion. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to make the technical solution in the embodiments or prior art clearer, the drawings needed in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0054] Figure 1 The system framework schematic diagram involved in an application scenario in the embodiments of the present application is shown in the figure.

[0055] Figure 2 The method flowchart of the question and answer emotion evaluation method based on action unit AU and micro-expression provided by the embodiments of the present application is shown in the figure.

[0056] Figure 3 The structure schematic diagram of the question and answer emotion evaluation device based on action unit AU and micro-expression provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0057] In order to make the technical solution in the embodiments or prior art clearer, the drawings needed in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0058] At present, in the question and answer process, the question and answer situation of the user to be detected is recorded by video, and the micro-expression and facial action unit of the user to be detected are analyzed by machine to judge whether the user to be detected understands the question, answers the question actively or negatively. By this way, the interference of human factors on the user to be detected is excluded, and the questioner is also helped to understand the reaction of the user to be detected to the question.

[0059] However, the method of analyzing the AU and micro-expression of the user to be detected by machine to judge the question and answer emotion of the user to be detected lacks unified standard, cannot provide an effective reference for the user to be detected, has low referenceability and poor fairness.

[0060] Based on this, in order to solve the above problems, in the embodiment of the present application, first, the question and answer video of the user to be detected is obtained, the first image about the non-basic question and answer in the question and answer video is input into the face detection model for detection, the first face image is obtained, the first face image is input into the multi-task recognition model of AU and micro-expression, and the scores of the plurality of first AUs and the scores of the plurality of first micro-expressions of the first face image are output. Then, through the corresponding relationship between the plurality of AUs, the plurality of micro-expressions and the plurality of emotional parameters, the plurality of first AU scores and the plurality of first micro-expression scores are weighted and calculated to obtain the plurality of target emotional values corresponding to the plurality of emotional parameters. Finally, the plurality of target emotional values are compared with the plurality of basic emotional values to obtain a comparison result, and the plurality of basic emotional values are obtained by emotional analysis based on emotional parameters according to the second image of the basic question and answer in the user question and answer video. The question and answer emotion of the user to be detected is evaluated through the comparison result.

[0061] It can be seen that, on the basis of obtaining the plurality of AU scores and the plurality of micro-expression scores of the first image about the non-basic question and answer through the multi-task recognition model of AU and micro-expression, the plurality of target emotional values corresponding to the plurality of emotional parameters are further weighted and calculated through the corresponding relationship between the plurality of AUs, the plurality of micro-expressions and the plurality of emotional parameters, and compared with the plurality of basic emotional values corresponding to the plurality of emotional parameters, so as to evaluate the question and answer emotion of the user to be detected. This way sets the analysis standard of the question and answer emotion, and improves the fairness and credibility of evaluating the question and answer emotion.

[0062] For example, one of the scenarios of the embodiment of the present application can be applied to the scenario as shown in the figure. Figure 1 The scenario includes a camera 101 and a server 102, wherein the camera 101 collects the question and answer video of the user to be detected and sends it to the server 102, and the server 102 realizes the question and answer emotion evaluation based on action unit AU and micro-expression by using the implementation manner provided by the embodiment of the present application.

[0063] First, in the above application scenario, although the action description of the implementation manner provided by the embodiment of the present application is executed by the server 102; however, the embodiment of the present application is not limited in the execution subject, as long as the action disclosed by the implementation manner provided by the embodiment of the present application is executed.

[0064] Secondly, the above scenario is only one scenario example provided by the embodiment of the present application, and the embodiment of the present application is not limited to this scenario.

[0065] The specific implementation manner of the question and answer emotion evaluation method and device based on action unit AU and micro-expression in the embodiment of the present application will be described in detail in combination with the drawings and embodiments.

[0066] Figure 2A method flowchart of a question and answer emotion evaluation method based on action unit AU and micro-expression is provided for the embodiments of the present application. In combination with Figure 2 As shown in the method flowchart, the question and answer emotion evaluation method based on action unit AU and micro-expression provided by the embodiments of the present application can include the following steps.

[0067] S201: Obtain a question and answer video of a user to be detected.

[0068] The question and answer video of the user to be detected can be a question and answer video during an interview or a class. Of course, the specific content of the question and answer video is not specifically limited in the present application, and does not affect the implementation of the embodiments of the present application.

[0069] S202: Detect a first image about a non-basic question and answer in the question and answer video by using a face detection model to obtain a first face image.

[0070] The face detection model is a model for detecting a face frame and preset face key points in the face frame from an image. The face frame and the preset face key points in the face frame can be used to obtain a face image from the image.

[0071] The embodiments of the present application do not specifically limit the process of obtaining the face image. In order to facilitate understanding, a possible implementation manner is described below. The first image about the non-basic question and answer in the question and answer video is input into the face detection model to output the face frame of the user to be detected, i.e., the first position information of the first target face frame, and output the preset face key points in the face frame of the user to be detected, i.e., the second position information of the preset face key points in the first target face frame. The preset face key points include a left eye center point, a right eye center point, a nose center point, a left corner of the mouth center point, and a right corner of the mouth center point.

[0072] According to the first position information of the first target face frame and the second position information of the preset face key points in the first target face frame, the first image can be aligned, and then cropped according to a certain margin to obtain the first face image. After cropping, standardization can also be performed to obtain the first face image. The standardization operation can be light compensation or grayscale transformation operation on the image. In a possible implementation, the process of obtaining the first face image specifically includes: detecting the first image about the non-basic question and answer in the question and answer video through the face detection model (mobilenetv2) to obtain the first position information [x1, y1, x2, y2] of the first target face frame and the second position information [px1, py1, px2, py2, px3, py3, px4, py4, px5, py5] of the preset face key points in the first target face frame. [x1, y1, x2, y2] and [px1, py1, px2, py2, px3, py3, px4, py4, px5, py5] are input to the image cropping module constructed by the cross-platform computer vision and machine learning software library (opencv) and the image library (numpy). The first image is cropped to obtain the first face image, and the size of the first face image is 112*112.

[0073] As an example, the question in the non-basic question and answer in the question and answer video can be "Is the recent work pressure very large?" or "Is the work intensity acceptable?" and the like.

[0074] S203: identifying the first face image through the multi-task recognition model based on AU and micro-expression to obtain a plurality of first AU scores and a plurality of first micro-expression scores of the first face image.

[0075] The multi-task recognition model based on AU and micro-expression is a model for AU recognition and micro-expression recognition of a face image. AU refers to the action unit of facial muscles, and micro-expression refers to the instantaneous appearance of facial expression, which can show the real emotions and moods of the user to be detected. The multi-task recognition model based on AU and micro-expression refers to the use of the same model to simultaneously train two AU recognition tasks and micro-expression recognition tasks from different data sources. The first face image is input into the multi-task recognition model based on AU and micro-expression to obtain a plurality of first AU scores and a plurality of first micro-expression scores of the first face image.

[0076] As an example, inputting the first face image into the multi-task recognition model based on AU and micro-expression can obtain the scores of 25 labeled AUs and the scores of 35 micro-expressions.

[0077] S204: According to the correspondence between the plurality of AUs, the plurality of micro-expressions and the plurality of emotion parameters, the plurality of first AU scores and the plurality of first micro-expression scores are weighted and calculated to obtain a plurality of target emotion values corresponding to the plurality of emotion parameters.

[0078] The plurality of emotion parameters include positive emotion parameters and negative emotion parameters, and the plurality of target emotion values corresponding to the plurality of emotion parameters are obtained by emotion analysis based on the plurality of emotion parameters on the plurality of first images related to non-basic questions and answers in the question and answer video of the user to be detected. Emotion analysis refers to the weighted calculation of the plurality of first AU scores and the plurality of first micro-expression scores according to the correspondence between the plurality of AUs, the plurality of micro-expressions and the plurality of emotion parameters.

[0079] As an example, according to the correspondence between 3 AUs, 1 micro-expression and 1 positive emotion parameter, the weighted calculation of 3 first AU scores and 1 first micro-expression score obtains 1 positive emotion parameter value as 1 target emotion value. Of course, the correspondence between the plurality of AUs, the plurality of micro-expressions and the plurality of emotion parameters is not specifically limited in the present application, and does not affect the implementation of the embodiments of the present application.

[0080] S205: Comparing the plurality of target emotion values and the plurality of basic emotion values corresponding to the plurality of emotion parameters obtains a comparison result, and the plurality of basic emotion values are obtained by emotion analysis based on the plurality of emotion parameters on the second images related to basic questions and answers in the question and answer video.

[0081] The plurality of basic emotion values are obtained by emotion analysis based on the plurality of emotion parameters on the plurality of second images related to basic questions and answers in the question and answer video.

[0082] As an example, the question related to the basic question and answer in the question and answer video can be "What is your name?" or "What is your age?" etc. Of course, the content of the question in the basic question and answer is not specifically limited in the present application, and does not affect the implementation of the embodiments of the present application.

[0083] S206: According to the comparison result, the question and answer emotion of the user to be detected is evaluated.

[0084] According to the comparison result of the plurality of target emotion values and the plurality of basic emotion values corresponding to the plurality of emotion parameters, the question and answer emotion of the user to be detected is evaluated. As an example, when one positive emotion value of the target emotion value is significantly higher than the positive emotion value corresponding to the basic emotion value, the question and answer emotion of the user to be detected is evaluated as positive.

[0085] The above is a question and answer emotion evaluation method based on action units (AUs) and micro expressions provided in the embodiments of the present application. First, a question and answer video of a user to be detected is obtained. A first image about a non-basic question and answer in the question and answer video is input into a face detection model for detection to obtain a first face image. The first face image is input into a multi-task recognition model of AUs and micro expressions to output scores of a plurality of first AUs and scores of a plurality of first micro expressions of the first face image. Then, the scores of the plurality of first AUs and the scores of the plurality of first micro expressions are weighted and calculated based on a corresponding relationship between the plurality of AUs, the plurality of micro expressions and a plurality of emotion parameters to obtain a plurality of target emotion values corresponding to the plurality of emotion parameters. Finally, the plurality of target emotion values are compared with a plurality of basic emotion values to obtain a comparison result. The plurality of basic emotion values are obtained by emotion analysis based on emotion parameters on a second image about a basic question and answer in the question and answer video of the user. The question and answer emotion of the user to be detected is evaluated based on the comparison result.

[0086] It can be seen that, based on the scores of the plurality of AUs and the scores of the plurality of micro expressions of the first image about the non-basic question and answer obtained by the multi-task recognition model of AUs and micro expressions, the plurality of target emotion values corresponding to the plurality of emotion parameters are further obtained by weighted calculation based on the corresponding relationship between the plurality of AUs, the plurality of micro expressions and the plurality of emotion parameters, and compared with the plurality of basic emotion values corresponding to the plurality of emotion parameters, so as to evaluate the question and answer emotion of the user to be detected. The analysis standard of the question and answer emotion is set, and the fairness and credibility of the evaluation of the question and answer emotion are improved.

[0087] In the embodiments of the present application, S202 can specifically include S2021-S2024 as follows:

[0088] S2021: pre-training a first detection network according to a first training face image in a first training image set and first face box annotation data of the first training face image to obtain a first pre-training model.

[0089] The first training image set can be a wide face open source face dataset, and the first training face image in the first training image set has first face box annotation data. The first pre-training model is obtained by pre-training the first detection network according to the first training face image and the first face box annotation data.

[0090] S2022: detecting a second training face image in a second training image set by the first pre-training model to obtain face box prediction data and face key point prediction data of the second training face image.

[0091] The second training image set can be a collection of second training face images with second face bounding box annotation data and face key point annotation data collected independently. Inputting these second training face images into the first pre-trained model yields predicted face bounding box data and predicted face key point data for the second training face images. In one possible implementation, a feature pyramid structured face detection network (RetinaFace) method is used, employing a face detection model (Mobilenetv2) as the backbone network with a width coefficient of 0.25. Multi-task learning is utilized to jointly train the tasks of annotating face bounding boxes and annotating face key points.

[0092] S2023: Based on the second face bounding box annotation data, face key point annotation data, face bounding box prediction data, face key point prediction data, and the first preset loss function, fine-tune the model parameters of the first pre-trained model.

[0093] The second face bounding box annotation data and face key point annotation data come from the second training image set. The face bounding box prediction data and face key point prediction data are obtained based on the output of the first pre-trained model. The model parameters of the first pre-trained model are fine-tuned based on the second face bounding box annotation data, face key point annotation data, face bounding box prediction data, face key point prediction data, and the first preset loss function.

[0094] S2024: The first pre-trained model that has completed fine-tuning training is identified as the face detection model.

[0095] The first pre-trained model, after fine-tuning and training, is used as the face detection model.

[0096] A first detection network is pre-trained using first training face images and their first bounding box annotations from a first training image set to obtain a first pre-trained model. The first pre-trained model is then fine-tuned using second training face images from a second training image set, along with their second bounding box annotations and facial landmark annotations, to obtain a face detection model. This method enables multi-task learning, allowing simultaneous detection of both annotated bounding boxes and annotated facial landmarks, thus improving both the model's detection performance and speed.

[0097] In this embodiment of the application, S203 may specifically include the following S2031-S2038:

[0098] S2031: The multi-task recognition network is pre-trained based on the third training face image and the face feature annotation data of the third training face image in the third training image set to obtain the second pre-trained model.

[0099] The third training image set can be an open source data set, the third training face image in the third training set has label data of facial features, and the multi-task recognition network is pre-trained through the third training face image and the label data of facial features to obtain the second pre-training model.

[0100] S2032: Feature extraction is performed on the fourth training face image in the fourth training image set and the fifth training face image in the fifth training image set through the feature extraction layer in the second pre-training model respectively to obtain first image features of the fourth training face image and second image features of the fifth training face image.

[0101] The fourth training image set can be a set of fourth training face images with 25 self-labeled AU label data and three kinds of publicly labeled AU label data with different numbers of labels. The three kinds of publicly labeled AU label data with different numbers of labels can be 12 AU label data, 24 AU label data and 36 AU label data. The fifth training image set can be a set of fifth training face images with 35 labeled micro-expression data.

[0102] Feature extraction is performed on the fourth training face image through the feature extraction layer in the second pre-training model to obtain first image features of the fourth training face image; and feature extraction is performed on the fifth training face image to obtain second image features of the fifth training face image.

[0103] S2033: Convolution is performed on the first image features and the second image features through the convolution layer in the second pre-training model respectively to obtain first convolution features and second convolution features.

[0104] Convolution is performed on the first image features of the fourth training face image through the convolution layer in the second pre-training model to obtain first convolution features; and convolution is performed on the second image features of the fifth training face image to obtain second convolution features.

[0105] S2034: AU prediction data of the fourth training face image is obtained by detecting the first convolution features through the AU detection layer in the second pre-training model.

[0106] The first convolution features of the fourth training face image are input into the AU detection layer in the second pre-training model to obtain the AU prediction data of the fourth training face image. When detecting, when the number of AU label data of different fourth training face images is different, detection is performed according to the corresponding number of labels, for example, when the fourth training face image with 12 AU label data is detected, 12 AU prediction data of the fourth training face image is output, not 25 AU prediction data.

[0107] S2035: Fine-tune the model parameters of the feature extraction layer, the convolutional layer, and the AU detection layer in the second pre-trained model according to the AU annotation data, the AU prediction data, and a second preset loss function.

[0108] The AU annotation data comes from the fourth training image set, and the AU prediction data is obtained by detecting the AU detection layer of the second pre-trained model. The model parameters of the feature extraction layer, the convolutional layer, and the AU detection layer in the second pre-trained model are fine-tuned according to the AU annotation data, the AU prediction data, and the second preset loss function.

[0109] S2036: Detect the second convolutional feature through the micro-expression detection layer in the second pre-trained model to obtain micro-expression prediction data of the fifth training face image.

[0110] The second convolutional feature of the fifth training face image is input into the micro-expression detection layer in the second pre-trained model to obtain micro-expression prediction data of the fifth training face image.

[0111] S2037: Fine-tune the model parameters of the feature extraction layer, the convolutional layer, and the micro-expression detection layer in the second pre-trained model according to the micro-expression annotation data, the micro-expression prediction data, and a third preset loss function.

[0112] The micro-expression annotation data comes from the fifth training image set, and the micro-expression prediction data is obtained by detecting the micro-expression detection layer of the second pre-trained model. The model parameters of the feature extraction layer, the convolutional layer, and the micro-expression detection layer in the second pre-trained model are fine-tuned according to the micro-expression annotation data, the micro-expression prediction data, and the third preset loss function.

[0113] S2038: Determine the second pre-trained model after fine-tuning as a multi-task recognition model based on AU and micro-expression.

[0114] The second pre-trained model after fine-tuning is used as a multi-task recognition model based on AU and micro-expression.

[0115] The multi-task recognition network is pre-trained according to the third training face image in the third training image set and the face feature annotation data of the third training face image to obtain a second pre-trained model. The second pre-trained model is fine-tuned according to the fourth training face image in the fourth training image set, the AU annotation data of the fourth training face image, the fifth training face image in the fifth training image set, and the micro-expression annotation data of the fifth training face image to obtain a multi-task recognition model based on AU and micro-expression. In this way, the multi-task recognition model based on AU and micro-expression obtained can perform multi-task learning, expand the number of annotated AUs and annotated micro-expressions, and thus improve the reliability and efficiency of emotion analysis.

[0116] In the embodiment of the present application, S205 can specifically include S2051-S2053 as follows:

[0117] S2051: detecting the second image about the basic question and answer in the question and answer video through the face detection model to obtain a second face image.

[0118] The second image about the basic question and answer in the question and answer video is input into the face detection model to output the first position information of the face frame of the to-be-detected user, that is, the first position information of the second target face frame, and output the preset face key points in the face frame of the to-be-detected user, that is, the second position information of the preset face key points in the second target face frame. According to the first position information of the second target face frame and the second position information of the preset face key points in the second target face frame, the second image can be aligned, and then cropped according to a certain margin to obtain the second face image. After cropping, the second face image can also be standardized.

[0119] S2052: identifying the second face image through the multi-task recognition model based on AU and micro-expression to obtain a plurality of second AU scores and a plurality of second micro-expression scores of the second face image.

[0120] The second face image is input into the multi-task recognition model based on AU and micro-expression to obtain a plurality of second AU scores and a plurality of second micro-expression scores of the second face image. As an example, the second face image is input into the multi-task recognition model based on AU and micro-expression to obtain scores of 25 labeled AUs and scores of 35 micro-expressions.

[0121] S2053: according to the corresponding relationship, weighting calculation is performed on the plurality of second AU scores and the plurality of second micro-expression scores to obtain a plurality of basic emotion values.

[0122] The plurality of basic emotion values corresponding to the plurality of emotion parameters refer to the second image about the basic question and answer in the question and answer video of the to-be-detected user, which is obtained by emotion analysis based on the plurality of emotion parameters.

[0123] As an example, according to the corresponding relationship between 3 AUs, 1 micro-expression and 1 positive emotion parameter, weighting calculation is performed on 3 second AU scores and 1 second micro-expression score to obtain 1 positive emotion parameter value as 1 basic emotion value.

[0124] On the basis of obtaining the plurality of AU scores and the plurality of micro-expression scores of the second image about the basic question and answer through the multi-task recognition model of the AU and the micro-expression, the plurality of basic emotion values corresponding to the plurality of emotion parameters are obtained through the corresponding relationship between the plurality of AUs, the plurality of micro-expressions and the plurality of emotion parameters, and weighted calculation is performed, and the plurality of target emotion values corresponding to the plurality of emotion parameters are compared, so as to evaluate the question and answer emotion of the user to be detected. In this way, the analysis standard of the question and answer emotion is set, and the fairness and credibility of evaluating the question and answer emotion are improved.

[0125] The above is some specific implementation manners of the question and answer emotion evaluation method based on the action unit AU and the micro-expression provided by the embodiments of the present application, based on which the corresponding device is also provided. The device provided by the embodiments of the present application will be introduced from the perspective of functional modularization.

[0126] Referring to Figure 3 The question and answer emotion evaluation device based on the action unit AU and the micro-expression 300 shown in the structure schematic diagram includes an acquisition module 301, a first detection module 302, a first identification module 303, a first calculation module 304, a comparison module 305 and an evaluation module 306.

[0127] The acquisition module 301 is configured to acquire a question and answer video of a user to be detected.

[0128] The first detection module 302 is configured to detect a first image about a non-basic question and answer in the question and answer video through a face detection model to obtain a first face image.

[0129] The first identification module 303 is configured to identify the first face image through a multi-task recognition model based on the action unit AU and the micro-expression to obtain a plurality of first AU scores and a plurality of first micro-expression scores of the first face image.

[0130] The first calculation module 304 is configured to perform weighted calculation on the plurality of first AU scores and the plurality of first micro-expression scores according to the corresponding relationship between the plurality of AUs, the plurality of micro-expressions and the plurality of emotion parameters, to obtain a plurality of target emotion values corresponding to the plurality of emotion parameters.

[0131] The comparison module 305 is configured to compare the plurality of target emotion values and a plurality of basic emotion values corresponding to the plurality of emotion parameters to obtain a comparison result, the plurality of basic emotion values being obtained by performing emotion analysis based on the plurality of emotion parameters on a second image about a basic question and answer in the question and answer video.

[0132] The evaluation module 306 is configured to evaluate the question and answer emotion of the user to be detected according to the comparison result.

[0133] In the embodiment of the present application, through the cooperation of the acquisition module 301, the first detection module 302, the first identification module 303, the first calculation module 304, the comparison module 305 and the evaluation module 306, the corresponding relationship between the multiple AUs, the multiple micro-expressions and the multiple emotional parameters is used to obtain the multiple target emotional values corresponding to the multiple emotional parameters through weighted calculation, and the multiple basic emotional values corresponding to the multiple emotional parameters are compared to evaluate the question and answer emotion of the to-be-detected user. In this way, the analysis standard of the question and answer emotion is set, and the fairness and credibility of evaluating the question and answer emotion are improved.

[0134] As an implementation form, the first detection module 302 can specifically include:

[0135] The second detection module is configured to detect the first image through the face detection model to obtain first position information of a first target face frame and second position information of preset face key points in the first target face frame.

[0136] The cropping module is configured to crop the first image according to the first position information and the second position information to obtain the first face image.

[0137] As an implementation form, the basic emotional value can be obtained through the following modules:

[0138] The third detection module is configured to detect a second image related to a basic question and answer in the question and answer video through the face detection model to obtain a second face image.

[0139] The second identification module is configured to identify the second face image through the multi-task identification model based on the AU and the micro-expression to obtain multiple second AU scores and multiple second micro-expression scores of the second face image.

[0140] The second calculation module is configured to perform weighted calculation on the multiple second AU scores and the multiple second micro-expression scores according to the corresponding relationship to obtain the multiple basic emotional values.

[0141] As an implementation form, the face detection model can be obtained through the following modules:

[0142] The first pre-training module is configured to pre-train the first detection network according to a first training face image in a first training image set and first face frame annotation data of the first training face image to obtain a first pre-training model.

[0143] The first fine-tuning training module is configured to fine-tune the first pre-training model according to a second training face image in a second training image set and second face frame annotation data and face key point annotation data of the second training face image to obtain the face detection model.

[0144] As an implementation form, the first fine-tuning training module can specifically include:

[0145] The fourth detection module is configured to detect the second training face image by using the first pre-training model to obtain face bounding box prediction data and face key point prediction data of the second training face image.

[0146] The second fine-tuning training module is configured to fine-tune the model parameters of the first pre-training model according to the second face bounding box annotation data, the face key point annotation data, the face bounding box prediction data, the face key point prediction data, and the first preset loss function.

[0147] The first determination module is configured to determine the first pre-training model after fine-tuning training as the face detection model.

[0148] As an implementation form, the multi-task recognition model based on AU and micro-expression can be obtained through the following modules:

[0149] The second pre-training module is configured to pre-train the multi-task recognition network according to the third training face image in the third training image set and the face feature annotation data of the third training face image to obtain a second pre-training model.

[0150] The third fine-tuning training module is configured to fine-tune the second pre-training model according to the fourth training face image in the fourth training image set, the AU annotation data of the fourth training face image, the fifth training face image in the fifth training image set, and the micro-expression annotation data of the fifth training face image to obtain the multi-task recognition model based on AU and micro-expression.

[0151] As an implementation form, the third fine-tuning training module can specifically include:

[0152] The feature extraction module is configured to extract features of the fourth training face image and the fifth training face image respectively by using the feature extraction layer in the second pre-training model to obtain first image features of the fourth training face image and second image features of the fifth training face image.

[0153] The convolution module is configured to perform convolution processing on the first image features and the second image features respectively by using the convolution layer in the second pre-training model to obtain first convolution features and second convolution features.

[0154] The AU detection module is configured to detect the first convolution features by using the AU detection layer in the second pre-training model to obtain the AU prediction data of the fourth training face image.

[0155] The first training module is configured to fine-tune the model parameters of the feature extraction layer, the convolutional layer and the AU detection layer in the second pre-trained model according to the AU annotation data, the AU prediction data and a second preset loss function.

[0156] The micro-expression detection module is configured to detect the second convolutional features through the micro-expression detection layer in the second pre-trained model to obtain micro-expression prediction data of the fifth training face image.

[0157] The second training module is configured to fine-tune the model parameters of the feature extraction layer, the convolutional layer and the micro-expression detection layer in the second pre-trained model according to the micro-expression annotation data, the micro-expression prediction data and a third preset loss function.

[0158] The second determination module is configured to determine the second pre-trained model after the fine-tuning as the multi-task recognition model based on the AU and the micro-expression.

[0159] Embodiments of the present application also provide a corresponding device and a computer storage medium for implementing the scheme provided by the embodiments of the present application.

[0160] The device comprises a memory and a processor, the memory is configured to store instructions or codes, and the processor is configured to execute the instructions or codes to enable the device to perform the method for evaluating the emotion of a question and answer based on the action unit (AU) and the micro-expression according to any of the embodiments of the present application.

[0161] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the method for evaluating the emotion of a question and answer based on the action unit (AU) and the micro-expression according to any of the embodiments of the present application.

[0162] The terms "first" and "second" in the names mentioned in the embodiments of the present application are only used for name identification, and do not represent the first and second in order.

[0163] From the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps of the above-mentioned embodiment methods can be implemented by means of software plus a general hardware platform. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a read-only memory (English: read-only memory, ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the method described in each embodiment or some parts of the embodiments of the present application.

[0164] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0165] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.

Claims

1. A question-and-answer emotion assessment method based on action units (AU) and micro-expressions, characterized in that, The method comprises: acquiring a question and answer video of a user to be detected; detecting a first image related to a non-basic question and answer in the question and answer video through a face detection model to obtain a first face image; identifying the first face image through a multi-task identification model based on AU and micro-expression to obtain a plurality of first AU scores and a plurality of first micro-expression scores of the first face image; weighting and calculating the plurality of first AU scores and the plurality of first micro-expression scores according to a corresponding relationship between a plurality of AUs, a plurality of micro-expressions and a plurality of emotional parameters to obtain a plurality of target emotional values corresponding to the plurality of emotional parameters; the plurality of emotional parameters comprise a positive emotional parameter and a negative emotional parameter; comparing the plurality of target emotional values and a plurality of basic emotional values corresponding to the plurality of emotional parameters to obtain a comparison result, the plurality of basic emotional values being obtained by performing emotional analysis on a second image related to a basic question and answer in the question and answer video based on the plurality of emotional parameters; evaluating the question and answer emotion of the user to be detected according to the comparison result; the acquiring step of the plurality of basic emotional values comprises: detecting the second image related to the basic question and answer in the question and answer video through the face detection model to obtain a second face image; identifying the second face image through the multi-task identification model based on AU and micro-expression to obtain a plurality of second AU scores and a plurality of second micro-expression scores of the second face image; weighting and calculating the plurality of second AU scores and the plurality of second micro-expression scores according to the corresponding relationship to obtain the plurality of basic emotional values.

2. The method of claim 1, wherein, The detecting a first image related to a non-basic question and answer in the question and answer video through a face detection model to obtain a first face image comprises: detecting the first image through the face detection model to obtain first position information of a first target face frame and second position information of a preset face key point in the first target face frame; cropping the first image according to the first position information and the second position information to obtain the first face image.

3. The method of claim 1, wherein, The training step of the face detection model comprises: pre-training a first detection network according to a first training face image in a first training image set and first face frame annotation data of the first training face image to obtain a first pre-training model; fine-tuning training the first pre-training model according to a second training face image in a second training image set and second face frame annotation data and face key point annotation data of the second training face image to obtain the face detection model.

4. The method of claim 3, wherein, The fine-tuning training the first pre-training model according to a second training face image in a second training image set and second face frame annotation data and face key point annotation data of the second training face image to obtain the face detection model comprises: detecting the second training face image through the first pre-training model to obtain face frame prediction data and face key point prediction data of the second training face image; According to the second face frame label data, the face key point label data, the face frame prediction data, the face key point prediction data, and a first preset loss function, the model parameters of the first pre-training model are fine-tuned and trained; The first pre-training model after fine-tuning and training is determined as the face detection model.

5. The method of claim 1, wherein, The training step of the multi-task recognition model based on AU and micro-expression includes: According to the third training face image in the third training image set and the face feature label data of the third training face image, a second pre-training model is obtained by pre-training a multi-task recognition network. According to the fourth training face image in the fourth training image set, the AU label data of the fourth training face image, the fifth training face image in the fifth training image set, and the micro-expression label data of the fifth training face image, the multi-task recognition model based on AU and micro-expression is obtained by fine-tuning and training the second pre-training model.

6. The method of claim 5, wherein, The fine-tuning and training of the second pre-training model according to the fourth training face image in the fourth training image set, the AU label data of the fourth training face image, the fifth training face image in the fifth training image set, and the micro-expression label data of the fifth training face image, to obtain the multi-task recognition model based on AU and micro-expression, includes: The fourth training face image and the fifth training face image are respectively extracted by the feature extraction layer in the second pre-training model, to obtain the first image feature of the fourth training face image and the second image feature of the fifth training face image; The first image feature and the second image feature are respectively processed by the convolution layer in the second pre-training model, to obtain the first convolution feature and the second convolution feature; The first convolution feature is detected by the AU detection layer in the second pre-training model, to obtain the AU prediction data of the fourth training face image; According to the AU label data, the AU prediction data, and a second preset loss function, the model parameters of the feature extraction layer, the convolution layer, and the AU detection layer in the second pre-training model are fine-tuned and trained; The second convolution feature is detected by the micro-expression detection layer in the second pre-training model, to obtain the micro-expression prediction data of the fifth training face image; According to the micro-expression label data, the micro-expression prediction data, and a third preset loss function, the model parameters of the feature extraction layer, the convolution layer, and the micro-expression detection layer in the second pre-training model are fine-tuned and trained; The second pre-training model after fine-tuning and training is determined as the multi-task recognition model based on AU and micro-expression.

7. An action unit (AU) and micro-expression based question answering sentiment assessment device, comprising: The device includes: An acquisition module is configured to acquire a question and answer video of a user to be detected; A first detection module is configured to detect a first image related to a non-basic question and answer in the question and answer video by using a face detection model to obtain a first face image; The first identification module is configured to identify the first face image by using the multi-task identification model based on the AU and the micro-expression, and obtain a plurality of first AU scores and a plurality of first micro-expression scores of the first face image. The first calculation module is configured to perform weighted calculation on the plurality of first AU scores and the plurality of first micro-expression scores according to a corresponding relationship between a plurality of AUs, a plurality of micro-expressions and a plurality of emotional parameters, and obtain a plurality of target emotional values corresponding to the plurality of emotional parameters; the plurality of emotional parameters include a positive emotional parameter and a negative emotional parameter. The comparison module is configured to compare the plurality of target emotional values with a plurality of base emotional values corresponding to the plurality of emotional parameters to obtain a comparison result, and the plurality of base emotional values are obtained by performing emotional analysis on a second image related to a base question and answer in the question and answer video based on the plurality of emotional parameters. The evaluation module is configured to evaluate the question and answer emotion of the to-be-detected user according to the comparison result. The base emotional values are obtained by the following modules: The third detection module is configured to detect a second face image in a second image related to a base question and answer in the question and answer video by using a face detection model. The second identification module is configured to identify the second face image by using the multi-task identification model based on the AU and the micro-expression, and obtain a plurality of second AU scores and a plurality of second micro-expression scores of the second face image. The second calculation module is configured to perform weighted calculation on the plurality of second AU scores and the plurality of second micro-expression scores according to the corresponding relationship, and obtain the plurality of base emotional values.

8. An action unit (AU) and micro-expression based question answering sentiment assessment device, comprising: The memory is configured to store a computer program. The processor is configured to implement the steps of the question and answer emotion evaluation method based on the action unit (AU) and the micro-expression according to any one of claims 1 to 6 when the computer program is executed. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the question and answer emotion evaluation method based on the action unit (AU) and the micro-expression according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, ​

Images