Facial expression and subjective emotion scale-based user emotion experience evaluation method

Abstract

The invention relates to a method for analyzing user emotion experience based on facial expressions and a subjective emotion scale. Specifically, the method comprises the following steps: firstly, capturing and recording a face image of a user in a product (or interaction) using process in real time by using a camera, and continuously detecting a human face in the face image in real time by using a RetinaFace method; then, for the human face detected in each frame, various expression intensities, including surprising, afraid, dislike, happy, sadness, vigor and neutral expressions, are calculated by using a ResmaskNet facial expression recognition method; and finally, combining objective facial expressions of the user and a subjective emotion scale to comprehensively analyze the emotion experience of the user in a product using or interaction process. The facial expression of the user is captured in real time through the camera, the cost is low, the experience of the user in the interaction process is not affected, and the method is suitable for evaluating the user experience in most interaction scenes.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence, pattern recognition and human-computer interaction, and specifically relates to technologies such as face detection and facial expression recognition. Background Art

[0002] How a product (such as an app, interface, game, etc.) stands out from the crowd and is chosen by users for a long time depends on many factors, such as product operability, whether the interface is good-looking, user emotions, etc. Understanding user emotions and the connection between them and experience is one of the main goals in product design. For hedonic products, the user's emotional response has an important impact on his / her attitude towards the product and the duration of continuous use. On the other hand, product experience research can help designers understand user needs and optimize product design to attract more users, which is conducive to improving product competitiveness and expanding the market. Therefore, it is particularly important to evaluate user emotional experience.

[0003] In order to measure the user's emotional experience, a large number of methods have been proposed. Among them, the traditional method is to obtain their emotional experience based on the user's self-report. This type of method can quickly and directly reflect the user's emotions, but its disadvantages are also obvious, that is, the subjects may unconsciously report false emotions, and the reports are not timely. With the rapid development of physiological signals in the field of emotional computing, some scholars use physiological signals to measure players' emotional reactions. Physiological signals are not controlled by humans and can reflect emotions in a real and real-time manner, but the collection of physiological signals requires the use of corresponding equipment, which may affect the comfort of the user experience. In addition, the signal processing process is also very complicated. These shortcomings have led to the limited application of user experience evaluation methods based on physiological signals in practice.

[0004] As another branch of affective computing, facial image-based expression recognition has a wide range of applications between people or between humans and machines. It is generally believed that the communication of emotions depends on facial expressions for 55%, voice tone for 38%, and verbal communication for 7%. Therefore, facial expressions can be used to identify a person's inner thoughts. For example, in the field of transportation, the driver's emotions are very important for safe driving, and the driver's emotions can be monitored during the operation of the car by real-time recognition of expressions. There are also studies on identifying personal pain levels, controlling wheelchairs, identifying students' concentration, and evaluating food experiences based on facial expressions. Although facial expressions have made progress in many fields, their application in the field of user experience is still relatively small. Compared with subjective scales and objective physiological signals, facial expressions have the advantages of good real-time performance, not easy to affect user experience, and the ability to detect the emotions of multiple people at the same time.

[0005] Therefore, in order to accurately evaluate the user's emotional experience, the present invention proposes a user emotional experience evaluation method based on facial expression and subjective emotion scale. By combining subjective scale emotion data and objective facial expression data, they can complement each other and achieve more accurate evaluation. Summary of the invention

[0006] The present invention is a method for evaluating user emotional experience based on facial expressions and subjective emotional scales. The method is implemented by the following technical scheme: first, a facial image I of a user in the process of using a product (or interacting) is captured in real time by a camera, and a face detection method is used to detect the face in I. Then, for the detected face, the intensity of various expressions is calculated by a facial expression recognition method. Finally, the user's facial expression and subjective emotional scale are combined to comprehensively analyze their emotional experience.

[0007] The first is face detection. The accuracy of face detection seriously affects the subsequent analysis. After comprehensive consideration, this method adopts the RetinaFace method and uses mobilenet as the backbone network of the model. It can achieve better detection accuracy while having a faster computing speed, which is conducive to ensuring real-time performance. The process includes: first, using MobilenetV1 to extract image features;

[0008] F = MobilenetV1 (img)

[0009] Next, the FPN (feature pyramid) structure is used to fuse the last three layers of the feature map extracted by MobilenetV1. Then the SSH module is used to enhance the receptive field to further enhance the feature extraction. SSH improves the detection of small faces by introducing contextual information in the feature map. Finally, the prediction results are obtained from the feature layer, including the face box regression prediction results, the classification prediction results (whether it contains a face), and the face key point regression prediction results.

[0010] The second is facial expression recognition. Traditional facial expression recognition uses descriptors such as hog and LBP to extract features of facial images, and then uses machine learning methods such as svm to classify them into different expression categories. These methods have complex calculation processes and limited accuracy in complex scenes. With the continuous development of computer vision, facial expression recognition based on deep learning has become the mainstream direction and can have higher recognition accuracy. This method uses the ResmaskNet network that has achieved SOTA accuracy on the widely used public datasets FER2013 and VEMO to recognize facial expressions. Specifically, after using Retinaface to crop the face image I1, it is first scaled to 224×224 size, and then the scaled image I0 is used as the input of the ResmaskNet model. The model contains 4 residual mask blocks, each of which contains a residual layer and a mask block. The residual layer is used to generate a coarse feature map, and the mask block contains a U-net structure with a contraction path and an expansion path, which can refine the feature map so that the network can focus on information related to expression. After these four residual mask blocks, we can get the probabilities of seven types of expressions (happy, sad, angry, surprised, scared, disgusted, and neutral). The following two formulas describe the model reasoning process.

[0011]

[0012] Y pred =Softmax(FC(F4))

[0013]

[0014] Where F i-1 is the feature map output by the previous layer in the model, where F0 = I1. i is the refined feature map of this layer. R and M represent the residual layer and mask block respectively. It is the element-wise product. The last feature map F4 passes through a fully connected and Softmax layer to output the probability of each expression.

[0015] Finally, the user's subjective feelings are evaluated using a subjective emotion scale. The product interaction difficulty and product liking scales are both 7-point Likert scales. Among them, the liking scale ranges from very dislike (-3) to very like (3), and the difficulty assessment scale ranges from very simple (1) to very difficult (7). Since the subjects are all Chinese, the present invention uses the simplified Chinese version of the PAD emotion scale to measure the user's subjective emotions. The scale consists of 12 groups of words, and each dimension contains 4 groups. The appendix provides details of the simplified scale. The following formula shows how to calculate the values ​​of each dimension.

[0016] P=(Q1-Q4+Q7-Q 10) / 16

[0017] A=(-Q2+Q5-Q8+Q 11 ) / 16

[0018] D=(Q3-Q6+Q9-Q 12 ) / 16

[0019] Where Q i (i=1,...,12) represents the score of the i-th question in the scale. i The range is (-4,4), and the value range of PAD is between -1.0 and 1.0.

[0020] The advantages of the present invention are: (1) By combining facial expressions and subjective scales, the user's emotional experience can be evaluated more accurately. (2) It has good real-time performance and can continuously capture and reflect the user's emotional changes. (3) It can simultaneously obtain the facial expressions of multiple users, which can save a lot of time for a large number of user experience evaluations. (4) It will not interfere with or affect the user experience. (5) It is low cost and only requires a laptop with a camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 For interactive interface

[0022] Figure 2 For the experimental process

[0023] Figure 3 Screenshot of facial expression recognition during user interaction

[0024] Figure 4 Comparison of scores of the PAD subjective emotion scale in various modes

[0025] Figure 5 Comparison of the intensity of facial expressions in each mode DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with embodiments:

[0027] Appropriate materials are very important for user emotional stimulation. In order to avoid the influence of the subjects' experience, the present invention did not select existing products or apps on the market, but designed and developed a shooting game with multiple difficulties as emotional experience stimulation material. Different game modes are set from three aspects, including the speed of the charging bar, the scale of the charging bar, and the size of the basket. Among them, the easy mode means that the charging bar speed is slow, there is a scale and it is a large basket; the general mode represents that the progress bar speed is medium, there is a scale and it is a small basket; in the difficult mode, the progress bar speed is the fastest, there is no scale and it is a small basket. Attached Figure 1(b)-(d) are screenshots of the game interface at different difficulties. The dotted line in the figure is the shooting direction indicator line, and there will be an animated prompt after the shot is hit. Each difficulty mode lasts for 40 seconds. During the experiment, after opening the game, the player randomly selects a game difficulty according to his or her preferences. The selection interface is shown in the attached figure. Figure 1 (a) As shown. After each game is completed, the game interface returns to this interface and another difficulty is selected. When one of the difficulties is selected last time, the button of the difficulty will turn gray and cannot be selected again. Then, the subjects fill in the PAD subjective emotion, liking level and difficulty scale in order, and rest for 1 minute to let the emotions return to stability to minimize the carryover effect, then select another difficulty and start the next game. It takes an average of about 6 to 10 minutes to complete the three-difficulty game. Only the player's facial data is recorded during the game.

[0028] The complete experimental process is as attached. Figure 2 As shown. After each subject arrived at the designated location at the agreed time, the experimenter first introduced the experimental content to them and told the players to keep their heads as still as possible because head movement may affect the accuracy of facial detection. Do not let the head move too much or cover the face. Then, the subjects began the pre-experiment. The pre-experiment is to give the subjects a general understanding of the content of the game. After completing the pre-experiment, the subjects began the formal experimental process. The game experimental scene is as follows Figure 2 shown.

[0029] The experimental results were evaluated using the statistical analysis software SPSS22.0. For facial expression data, the maximum intensity values ​​of different categories of expressions were used for analysis. Since facial expressions last for a short time, it may be more appropriate to use the maximum intensity value. All data were tested for normality using the Shapiro-Wilk test. If the normal distribution is not satisfied, the non-parametric Friedman test is used to analyze the difference; if the normal distribution is satisfied, the Manchly test is further used to test whether it obeys the spherical assumption. For data that do not satisfy the spherical assumption, the Greenhouse-Geisser method is used to correct it. One-way Repeated measurement ANOVAs are used to analyze the impact of different game difficulties on players' subjective and objective data. The post hoc test uses Bonferroni adjustment.

[0030] Table 1 shows the means and variances of users' difficulty and preference scores in different game modes. The Friedman test showed that there were significant differences in the means of difficulty scores in different modes (χ2(2)=38.235, p<0.001). Pairwise comparisons showed that the differences in scores between different modes were statistically significant: easy vs. normal (Z=-2.425, p<0.05), easy vs. difficult (Z=-5.760, p<0.001), and normal vs. difficult (Z=-3.335, p<0.01). The Friedman test showed that there were no significant differences in the means of preference scores in different modes (χ2(2)=3.32, p=0.19).

[0031] Table 2 shows the mean and variance of PAD of users in different modes. One-way repeated measures ANOVA shows that different game difficulties have a significant main effect on PAD value. Among them, P: F (1.696, 55.964) = 5.659, η 2 =0.146,P<0.01; A:F(1.695,55.939)=13.560,η 2 =0.1291,p<0.001; D:F(2,66)=5.193,η 2 =0.136, p<0.01. Bonferroni posterior test showed that the pleasure of the general and difficult groups was about 0.149 (p=0.018<0.05) and 0.219 (p=0.012<0.05) lower than that of the easy group, respectively. The arousal of the general and difficult groups was 0.162 (p=0.001<0.05) and 0.256 (p<0.001) higher than that of the easy group, respectively. The control of the general and difficult groups was about 0.162 (p=0.004<0.01) and 0.256 (p<0.001) higher than that of the easy group, respectively. In addition, there was no significant difference in PAD values ​​between the general and difficult groups.

[0032] Table 3 shows the means and variances of all expressions in the three modes. Friedman showed that angry, sad, and neutral expressions showed significant differences in different difficulty modes. Among them, the test statistic for anger was (χ2(2)=6.059, p<0.05), the test statistic for sadness was (χ2(2)=8.647, p<0.05), and the test statistic for neutral was (χ2(2)=8.882, p<0.05). In addition, pairwise comparisons showed that the intensity differences of these three expressions between easy and difficult were statistically significant, angry (Z=-2.425, p<0.05), sad (Z=-2.910, p<0.05), and neutral (Z=-2.789, p<0.05).

[0033] Table 1 Game difficulty and liking ratings for the three modes

[0034]

[0035] Table 2 Subjective PAD values ​​of three game modes

[0036]

[0037] Table 3 Facial expression intensity values ​​in three modes

[0038]

[0039]

Claims

1. A method for evaluating user emotional experience based on facial expression and subjective emotion scale, comprising the following steps: (I) Face detection based on RetinaFace: (1) Use MobilenetV1 as the backbone network to extract image features; (2) Use the FPN (feature pyramid) structure to perform feature fusion on the last three layers of the feature map extracted by MobilenetV1; (3) Use the SSH module to enhance the receptive field to further enhance feature extraction; (4) Obtain prediction results from the feature layer, including face frame regression prediction results, classification prediction results, and face key point regression prediction results; (II) Facial expression recognition based on ResmaskNet: (1) Scale the detected face image to 224×224; (2) The scaled face image is sent to ResmaskNet, which contains four residual modules, for feature extraction; (3) The fully connected and Softmax layers output the probabilities of each type of expression. (III) Subjective sentiment evaluation based on the simplified Chinese version of PAD The simplified Chinese version of the PAD emotional scale is used to evaluate users' subjective emotional scores, including pleasure P, arousal A, and control D. Each dimension contains 4 questions with a value range of -1 to 1. Step 1: Before calculating facial expressions, we need to perform face detection. This method uses the RetinaFace method and uses mobilenet as the backbone network of the model. It can achieve better detection accuracy while having a faster computing speed, which is conducive to ensuring real-time performance. The process includes: first use MobilenetV1 to extract image features; F = MobilenetV1 (img) Next, the FPN (feature pyramid) structure is used to fuse the last three layers of the feature map extracted by MobilenetV1. Then the SSH module is used to enhance the receptive field to further enhance the feature extraction. SSH improves the detection of small faces by introducing contextual information in the feature map. Finally, the prediction results are obtained from the feature layer, including the face box regression prediction results, the classification prediction results (whether it contains a face), and the face key point regression prediction results. Step 2: Facial expression recognition. This method uses the ResmaskNet network, which has achieved SOTA accuracy on the widely used public datasets FER2013 and VEMO, to recognize facial expressions. Specifically, after using Retinaface to crop the face image I1, it is first scaled to 224×224, and then the scaled image I0 is used as the input of the ResmaskNet model. The model contains 4 residual mask blocks, each of which contains a residual layer and a mask block. The residual layer is used to generate a coarse feature map, and the mask block contains a U-net structure with a contraction path and an expansion path, which can refine the feature map so that the network can focus on information related to the expression. After these 4 residual mask blocks, the probabilities of 7 types of expressions (happy, sad, angry, surprised, afraid, disgusted, and neutral) can be obtained. The following two formulas describe the process of model reasoning. Y pred =Softmax(FC(F4)) Where F i-1 is the feature map output by the previous layer in the model, where F0 = I1. i is the refined feature map of this layer. R and M represent the residual layer and mask block respectively. It is the element-wise product. The last feature map F4 passes through a fully connected and Softmax layer to output the probability of each expression. Step 3: Finally, the user's subjective feelings are evaluated using a subjective emotion scale. The product interaction difficulty and product liking scales are both 7-point Likert scales. Among them, the liking scale ranges from very dislike (-3) to very like (3), and the difficulty assessment scale ranges from very simple (1) to very difficult (7). Since the subjects are all Chinese, the present invention uses the simplified Chinese version of the PAD emotion scale to measure the user's subjective emotions. The scale consists of 12 groups of words, and each dimension contains 4 groups. The appendix provides details of the simplified scale. The following formula shows how to calculate the values ​​of each dimension. P=(Q1-Q4+Q7-Q 10 ) / 16 <h2 style=";text-align:left;direction:ltr">A=(-Q2+Q5-Q8+Q<h2 style=";text-align:left;direction:ltr"> 11 <h2 style=";text-align:left;direction:ltr"> ) / 16 D=(Q3-Q6+Q9-Q 12 ) / 16 Where Q i (i=1,...,12) represents the score of the i-th question in the scale. i The range is (-4,4), and the value range of PAD is between -1.0 and 1.0.

Images