Emotion estimation device, emotion estimation method and emotion estimation program

Abstract

To provide an emotion estimation device capable of improving accuracy of emotion estimation.SOLUTION: An emotion estimation device includes an estimated emotion value generation part 36 for calculating an emotion value and an arousal level of a subject by using a learned neural network for autonomic nerve information (e.g., a pulse rate) and behavioral information (e.g., a body movement amount and expression) of the subject. The emotion estimation device further includes an estimated emotion estimation part 37 for estimating the emotion of the subject on the basis of the emotion value and the arousal level of the subject calculated by the estimated emotion value generation part 36. The emotion estimation device can thus improve accuracy of emotion estimation.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an emotion estimation device, an emotion estimation method, and an emotion estimation program. [Background technology]

[0002] Due to the spread of the new coronavirus, problems have emerged in the efficiency of medical care. In response, online medical consultations have been proposed, but in online consultations, doctors can only understand the patient's condition through the patient's subjective explanation, which raises the possibility of doubts about whether an accurate diagnosis has been made. In addition, in face-to-face consultations, there is a problem that misdiagnosis may occur due to the patient's tension and sensory bias. In particular, protective equipment such as masks and face shields worn by doctors further increases the sense of distance and increases the barriers to communication. Furthermore, not only in the relationship between doctors and patients, but also in business situations such as customer service, customer service, and marketing, it is becoming increasingly important to improve communication by sensing the other person's emotions.

[0003] In order to solve such problems, the techniques described in Patent Documents 1 and 2 have been proposed. The invention described in Patent Document 1 is to create an expression map by learning the correspondence between expressions and emotions in advance, and to estimate emotions from a person's expression based on the expression map. The invention described in Patent Document 2 is to estimate a subject's brain waves from biological information other than brain waves, such as pulse waves, and to estimate the subject's emotions by comparing the brain waves with characteristic patterns of brain waves stored in correspondence with emotions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2008-146318 A [Patent Document 2] JP 2015-109964 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when estimating emotions from a subject's facial expression, as in the invention described in Patent Document 1, there are emotions that are difficult to estimate from facial expression alone (emotions that are difficult to express in facial expressions, or cases where the actual emotion is deliberately hidden by using a different facial expression), and there is a problem in that the accuracy of estimating emotions is low.

[0006] In addition, the invention described in Patent Document 2 infers emotions by comparing electroencephalograms estimated from biometric information with electroencephalogram patterns stored in association with emotions. However, since electroencephalograms infer emotions based on which parts of the brain are activated or deactivated, they can only estimate superficial phenomena, resulting in low accuracy in inferring emotions.

[0007] In view of the above problems, an object of the present invention is to provide an emotion estimation device, an emotion estimation method, and an emotion estimation program capable of improving the accuracy of emotion estimation. [Means for solving the problem]

[0008] The above object of the present invention can be achieved by the following means. Note that the parentheses indicate reference symbols of the embodiments described below, but the present invention is not limited thereto.

[0009] The emotion estimation device according to claim 1 includes a calculation means (e.g., estimated emotion value generation unit 36 ​​shown in FIG. 2) that calculates an emotion valence and an arousal level of a subject (e.g., subject HM shown in FIG. 1) by using a trained neural network for autonomic nerve information (e.g., pulse rate) and behavioral information (e.g., body movement amount, facial expression) of the subject; The present invention is characterized by comprising an estimation means (e.g., estimated emotion estimation unit 37 shown in FIG. 2) that estimates the emotion of the subject based on the emotional valence and the arousal level of the subject calculated by the calculation means.

[0010] The feeling estimation device according to claim 2 is the feeling estimation device according to claim 1, characterized in that the autonomic nerve information (e.g., pulse rate) and behavior information (e.g., body movement amount, facial expression) of the subject (e.g., subject HM shown in FIG. 1) are calculated based on a captured image of the subject captured by an imaging means (e.g., imaging device 2 shown in FIG. 1).

[0011] A feeling estimation method according to claim 3 is a feeling estimation method executed by a predetermined device (e.g., the feeling estimation device 3 shown in FIG. 1 ), A step of calculating the emotional valence and arousal level of a subject (e.g., subject HM shown in FIG. 1) using a trained neural network for autonomic nerve information (e.g., pulse rate) and behavioral information (e.g., body movement amount, facial expression) of the subject (e.g., subject HM shown in FIG. 1) (e.g., see step S5 shown in FIG. 3); and a step of estimating the emotion of the subject based on the calculated valence and arousal level of the subject (see, for example, step S6 shown in FIG. 3).

[0012] The emotion estimating program according to claim 4 includes the steps of: a first process (for example, estimated emotion value generation unit 36 ​​shown in FIG. 2) for calculating an emotional valence and an arousal level of a subject (for example, subject HM shown in FIG. 1) using a trained neural network for autonomic nerve information (for example, pulse rate) and behavioral information (for example, body movement amount, facial expression) of the subject; and executing a second process (e.g., estimated emotion estimation unit 37 shown in FIG. 2) of estimating the emotion of the subject based on the calculated emotional valence and arousal level of the subject. Effect of the Invention

[0013] Next, the effects of the present invention will be described with reference to the drawings. Note that the reference symbols in the parentheses are those of the embodiments described below, but the present invention is not limited thereto.

[0014] According to the invention of claims 1, 3, and 4, the valence and arousal level of a subject (e.g., subject HM shown in FIG. 1) are calculated using a trained neural network for autonomic nerve information (e.g., pulse rate) and behavioral information (e.g., body movement, facial expression) of the subject. Then, the subject's emotion is estimated based on the calculated valence and arousal level. This makes it possible to improve the accuracy of emotion estimation according to the present invention.

[0015] According to the invention of claim 2, the subject (for example, subject HM shown in FIG. 1) does not need to wear a specific device, which saves time and reduces the subject's sense of caution. [Brief description of the drawings]

[0016] [Figure 1] 1 is a schematic overall view of an emotion estimation system according to an embodiment of the present invention. FIG. [Diagram 2] FIG. 2 is a functional configuration diagram of the feeling estimation device according to the embodiment. [Diagram 3] FIG. 4 is a flowchart illustrating the processing content of the feeling estimation device according to the embodiment. [Figure 4] 1A is an image of a difference image obtained by taking the difference between the facial images of a subject, and FIG. 1B is a graph showing the integral value of the image time difference and the periodic change of the image time difference. [Diagram 5] FIG. 2 is a schematic diagram of a neural network according to the embodiment. [Figure 6] FIG. 1 is a diagram showing the Russell ring model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of an emotion estimation system using an emotion estimation device according to the present invention will be specifically described with reference to the drawings. In the following description, when directions such as up, down, left, and right are indicated, they refer to up, down, left, and right when viewed from the front of the illustration.

[0018] <Outline of emotion estimation system> The emotion estimation system according to the present embodiment can improve the accuracy of emotion estimation. Specifically, the emotion estimation system 1 shown in Fig. 1 detects autonomic nerve information (pulse rate, etc.) and behavioral information (body movement, facial expression, etc.) by analyzing a captured image of a subject's face, and estimates the subject's emotional state. Hereinafter, the contents of the emotion estimation system 1 will be described in detail with reference to the drawings.

[0019] As shown in Fig. 1, the emotion estimation system 1 includes an imaging device 2 and an emotion estimation device 3. Each component will be described in detail below.

[0020] <Explanation of imaging device> The imaging device 2 is a head-mounted MR device such as Hololens (registered trademark), Vision Pro (registered trademark), or Google Glass (registered trademark) as shown in FIG. 1. This head-mounted MR device is an extension of AR (Augmented Reality) technology, and is an added function of fusing the virtual world with the real world, quantifying a virtual object, and manipulating the virtual object in the form of a hologram. Thus, as shown in FIG. 1, such an imaging device 2 can be worn on the head of a user UM, and the subject HM can be visually recognized through a lens 2a. The imaging device 2 has a built-in camera and can capture the face of the subject HM. The facial image of the subject HM captured by the imaging device 2 can be output to the emotion estimation device 3 shown in FIG. 1.

[0021] <Explanation of emotion estimation device> The feeling estimation device 3 is composed of a smartphone, a PC (Personal Computer), etc. Specifically, as shown in FIG. 1, the feeling estimation device 3 is composed of a CPU 3a that executes and controls each function of the feeling estimation device 3, an input unit 3b that can input predetermined data to the feeling estimation device 3 from the outside using a mouse, a keyboard, a touch panel, etc., an output unit 3c that can output predetermined data to the outside of the feeling estimation device 3, a ROM 3d consisting of a writable flash ROM or the like that stores predetermined programs, etc., a RAM 3e that functions as a working area, a buffer memory, etc., and a display unit 3f consisting of an LCD (Liquid Crystal Display) or the like. Note that the functions and processing of the feeling estimation device 3 described later are realized by the CPU 3a reading out a program stored in the ROM 3d and executing this program.

[0022] <Explanation of the functional configuration of the emotion estimation device> Next, the functional configuration of the feeling estimation device 3 will be described with reference to Fig. 2. As shown in Fig. 2, the functional configuration of the feeling estimation device 3 includes a determination unit 30, a memory unit 31, a face image acquisition unit 32, a pulse rate detection unit 33, a body movement detection unit 34, a facial expression detection unit 35, an estimated emotion value generation unit 36, and an estimated emotion estimation unit 37. Each component will be described below.

[0023] The determination unit 30 exchanges information between the functional blocks based on the state held by the determination unit 30 or the state held by each functional block. Furthermore, the determination unit 30 also exchanges information with the imaging device 2.

[0024] The storage unit 31 stores data necessary for processing.

[0025] The face image acquiring unit 32 acquires a face image of the subject HM shown in Fig. 1. Specifically, when the face image of the subject HM is captured by the imaging device 2 worn on the head of the user UM shown in Fig. 1, the imaging device 2 outputs the captured face image of the subject HM to the feeling estimation device 3. In response to this, the face image acquiring unit 32 acquires the face image of the subject HM shown in Fig. 1.

[0026] The pulse rate detection unit 33 detects the pulse rate of the subject HM from the face image of the subject HM acquired by the face image acquisition unit 32.

[0027] The body movement detection section 34 detects the amount of body movement of the subject HM from the face image of the subject HM acquired by the face image acquisition section 32.

[0028] The facial expression detection unit 35 detects the facial expression of the subject HM from the facial image of the subject HM acquired by the facial image acquisition unit 32. Specifically, the facial expression detection unit 35 estimates the value of the facial expression emotional valence of the subject HM and the value of the facial expression arousal of the subject HM from the facial image of the subject HM acquired by the facial image acquisition unit 32 using the AffectNet model, which is a previous study (Ali Mollahosseini, Student Member, IEEE, Behzad Hasani, Student Member, IEEE, and Mohammad H. Mahoor, Senior Member, IEEE, “AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild”, IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, VOL. 10, NO. 1, JANUARY-MARCH 2019).

[0029] The estimated emotion value generating unit 36 ​​uses a trained neural network to estimate the emotion value of the subject HM. Specifically, this neural network is made up of a feedforward neural network having four input neurons and two output neurons, for example, as shown in FIG. 5. As shown in FIG. 5, this neural network has five layers, with the first layer made up of four neurons, the second layer made up of eight neurons, the third layer made up of six neurons, the fourth layer made up of four neurons, and the fifth layer made up of two neurons. Furthermore, this neural network uses a supervised backpropagation learning method to adjust the weights and biases of the network so as to minimize the error between the predicted output and the actual output, and in forward propagation, the input passes through the network to generate a predicted output, and in backward propagation, the error is backpropagated through the network to adjust the weights and biases.

[0030] For learning such a neural network, a dataset with labeled inputs and outputs is required. In this embodiment, an image dataset called EmoMadrid is used as specific learning sample data. In this image dataset, still images are used to evoke various emotions in subjects, and facial images at that time are taken and used as learning sample data. Then, the valence and arousal label answers of the emotions evoked by each image are prepared. The correct labels for these images are prepared as numerical values ​​of valence and arousal.

[0031] Thus, the emotion value of subject HM is estimated using such a trained neural network. Specifically, estimated emotion value generation section 36 uses the pulse rate of subject HM detected by pulse detection section 33, the amount of body movement of subject HM detected by body movement detection section 34, and the facial expression of subject HM detected by facial expression detection section 35 (facial expression valence value and facial expression arousal value) as explanatory variables. In this way, estimated emotion value generation section 36 can generate values ​​of the emotional valence of the estimated emotion of subject HM and the arousal level of the estimated emotion of subject HM as objective variables.

[0032] The estimated emotion estimation unit 37 estimates the emotion of the subject HM based on the valence of the estimated emotion of the subject HM generated by the estimated emotion value generation unit 36 ​​and the arousal level of the estimated emotion of the subject HM. Specifically, the emotion of the subject HM is estimated using a Russell circle model as shown in FIG. 6. As shown in FIG. 6, this Russell circle model has valence as the horizontal axis and arousal level as the vertical axis. Thus, the estimated emotion estimation unit 37 estimates the emotion of the subject HM by plotting the valence value of the estimated emotion of the subject HM generated by the estimated emotion value generation unit 36 ​​and the arousal level of the estimated emotion of the subject HM on such a Russell circle model. The valence is a numerical value that evaluates the subject's intrinsic attractiveness (good) or aversion (bad), and the arousal level is a value that indicates the degree of physical and cognitive arousal caused by the subject's emotion.

[0033] <An example of how to use the emotion estimation system> Next, a usage example of the feeling estimation system 1 shown in this embodiment will be described with reference to the flowchart shown in FIG.

[0034] First, a facial image of the subject HM is captured by the imaging device 2 worn on the head of the user UM shown in Fig. 1. As a result, the imaging device 2 outputs the captured facial image of the subject HM to the feeling estimation device 3. In response to this, the determination unit 30 shown in Fig. 2 receives the facial image of the subject HM captured by the imaging device 2, and outputs it to the facial image acquisition unit 32. As a result, the facial image acquisition unit 32 acquires the facial image of the subject HM captured by the imaging device 2 (step S1).

[0035] Next, the judgment unit 30 receives the face image of the subject HM captured by the imaging device 2, which is acquired by the face image acquisition unit 32 shown in FIG. 2, from the face image acquisition unit 32, and outputs it to the pulse detection unit 33 shown in FIG. 2. In response to this, the pulse detection unit 33 detects the pulse rate of the subject HM. Specifically, the pulse detection unit 33 cuts out a region of interest (ROI) (for example, the center of the forehead and the left and right cheeks of the subject HM) from the face image of the subject HM, erases the data of the red channel and the blue channel, and performs processing to leave only the data of the green channel. Then, the pulse detection unit 33 applies a median filter to smooth the data in the region of interest, and performs noise reduction processing. The image obtained at this time is a green image of the skin surface in the smoothed region of interest. For this obtained image, the pulse detection unit 33 flattens the histogram because there is little change in the green light reflected by hemoglobin. The histogram flattening is a method for adjusting the brightness and contrast of an image by adjusting the histogram of unevenly distributed colors to be uniform to [0, 255] (color resolution 8 bits). Thus, for the image with the histogram flattened in this way, the pulse detection unit 33 sums up the values ​​of the green channel of each pixel. This allows the amount of reflected light of hemoglobin in the relevant ROI region in the relevant frame to be obtained. Therefore, if the video frame rate of the face image of the subject HM acquired by the face image acquisition unit 32 is 30 fps, the pulse detection unit 33 can analyze 30 images per second and generate time series data of the reflected light of hemoglobin in each ROI region for one second. At this time, since the periodicity of the reflected light time series data of hemoglobin is the same as the periodic characteristic of the pulse, the pulse detection unit 33 can detect the pulse rate of the subject HM by calculating the frequency of the generated time series data for one second (step S2).

[0036] Next, the determination unit 30 receives the face image of the subject HM captured by the imaging device 2, which is acquired by the face image acquisition unit 32 shown in FIG. 2, from the face image acquisition unit 32, and outputs it to the body movement detection unit 34 shown in FIG. 2. In response to this, the body movement detection unit 34 detects the amount of body movement of the subject HM. Specifically, in humans, various emotions are aroused in the central system (brain) in response to stimuli from the outside world, and ultimately lead to an output in the form of behavior. In this way, the human body unconsciously moves due to emotions aroused by stimuli obtained from the outside world. For example, emotions such as trembling, shaking, and excitement are output as head movement. Therefore, the amount of body movement is easy to detect from a video. Therefore, the body movement detection unit 34 takes the difference between the face images of the subject HM. An image diagram of the difference image when this difference is taken is shown in FIG. 4(a). Then, the body movement detection unit 34 takes the integral value of the image time difference within a predetermined period of such difference images as the amount of body movement. Specifically, the body movement detection unit 34 analyzes the integral value of the time difference for each frame to generate time series data. When generating this time series data, the body movement detection unit 34 also quantifies the number of body movements by extracting periodic changes using DFT (Discrete Fourier Transform) analysis. That is, the body movement detection unit 34 quantifies the amount of body movement of the subject HM into the integral value of the image time difference and the periodic change of the image time difference. An example of this is the graph shown in FIG. 4(b). Thus, in this manner, the body movement detection unit 34 can detect the amount of body movement of the subject HM (step S3).

[0037] Next, the determination unit 30 receives the facial image of the subject HM captured by the imaging device 2, which is acquired by the facial image acquisition unit 32 shown in Fig. 2, from the facial image acquisition unit 32, and outputs it to the facial expression detection unit 35 shown in Fig. 2. In response to this, the facial expression detection unit 35 detects the facial expression of the subject HM by estimating the value of the facial emotion valence of the subject HM and the value of the facial arousal of the subject HM from the facial image of the subject HM using the AffectNet model described above (step S4).

[0038] Next, the judgment section 30 receives the pulse rate of subject HM, detected by pulse detection section 33, from pulse detection section 33, and outputs this to estimated emotion value generation section 36 shown in Fig. 2. Furthermore, the judgment section 30 receives the amount of body movement of subject HM, detected by body movement detection section 34, from body movement detection section 34, and outputs this to estimated emotion value generation section 36 shown in Fig. 2. Furthermore, the judgment section 30 receives the facial expression of subject HM, detected by facial expression detection section 35 (the value of the facial emotion valence of subject HM and the value of the facial awakening level of subject HM), from facial expression detection section 35, and outputs this to estimated emotion value generation section 36 shown in Fig. 2.

[0039] Thus, estimated emotion value generation section 36 receives this information and uses as explanatory variables the pulse rate of subject HM detected by pulse detection section 33, the amount of body movement of subject HM detected by body movement detection section 34, and the facial expression of subject HM detected by facial expression detection section 35 (emotional valence value and facial arousal value).In this way, estimated emotion value generation section 36 can generate values ​​of the emotional valence of the estimated emotion of subject HM and the arousal value of the estimated emotion of subject HM as objective variables (step S5).

[0040] Next, the judgment unit 30 receives the valence of the estimated emotion of the subject HM generated by the estimated emotion value generation unit 36 ​​and the arousal level of the estimated emotion of the subject HM from the estimated emotion value generation unit 36, and outputs it to the estimated emotion estimation unit 37 shown in FIG. 2. In response to this, the estimated emotion estimation unit 37 plots the valence value of the estimated emotion of the subject HM generated by the estimated emotion value generation unit 36 ​​and the arousal level of the estimated emotion of the subject HM in the Russell circle model shown in FIG. 6. For example, if the valence value of the estimated emotion of the subject HM is -1.47 and the arousal level of the estimated emotion of the subject HM is 1.39, it will be located in the upper left of the Russell circle model shown in FIG. 6. From this, the estimated emotion estimation unit 37 estimates that the emotion of the subject HM is in a state of "disgust". Also, for example, if the valence value of the estimated emotion of the subject HM is 1.08 and the arousal value of the estimated emotion of the subject HM is 0.32, it will be located in the upper right phenomenon of the Russell Circle Model shown in FIG. 6. From this, the estimated emotion estimation unit 37 will estimate that the emotion of the subject HM is in a state of "joy". Furthermore, for example, if the valence value of the estimated emotion of the subject HM is -0.46 and the arousal value of the estimated emotion of the subject HM is -0.21, it will be located near the boundary between the upper left phenomenon and the lower left phenomenon of the Russell Circle Model shown in FIG. 6. From this, the estimated emotion estimation unit 37 will estimate that the emotion of the subject HM is in a state of "unpleasant".

[0041] Thus, in this manner, the estimated feeling estimation unit 37 estimates the feeling of the subject HM (step S6). The feeling of the subject HM estimated by the estimated feeling estimation unit 37 is output to the imaging device 2 by the determination unit 30. As a result, the estimated feeling of the subject HM is displayed on the lens 2a of the imaging device 2. Therefore, the user UM shown in FIG. 1 can recognize the estimated feeling of the subject HM simply by observing the subject HM through the lens 2a.

[0042] Therefore, according to the present embodiment described above, the valence and arousal level of subject M are calculated using a trained neural network for the autonomic nerve information (in this embodiment, pulse rate is exemplified) and behavioral information (in this embodiment, the body movement amount of subject HM and facial expression of subject HM are exemplified). Then, based on the calculated valence and arousal level of subject HM, the emotion of subject HM is estimated. As a result, according to this embodiment, the accuracy of estimating emotions can be improved.

[0043] <Description of Modifications> It should be noted that the shapes and the like shown in this embodiment are merely examples, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. According to this embodiment, the pulse rate is exemplified as the autonomic nerve information, but the invention is not limited thereto, and the autonomic nerve information may be detected by detecting the expansion and contraction of the blood vessels of the subject HM. Furthermore, as the autonomic nerve information, a plurality of pieces of information may be used, or only a single piece of information may be used. However, since the accuracy is improved by using a plurality of pieces of information, it is preferable to use a plurality of pieces of information.

[0044] In addition, in this embodiment, the amount of body movement of the subject HM and the facial expression of the subject HM are exemplified as the behavior information, but the present invention is not limited thereto and may be any information. Furthermore, the behavior information may be a plurality of pieces of information or may be a single piece of information. However, since the accuracy is improved by using a plurality of pieces of information, it is preferable to use a plurality of pieces of information.

[0045] In addition, in this embodiment, a head-mounted MR device is exemplified as the imaging device 2, but the present invention is not limited to this, and imaging may simply be performed using a camera. Also, instead of using the imaging device 2, the subject HM may be asked to wear a specific device (e.g., a wearable device) to acquire autonomic nerve information and behavior information of the subject HM. However, imaging the subject HM is preferable, as this saves the trouble of having the subject HM wear a specific device and reduces the subject HM's wariness.

[0046] The feeling estimation device 3 exemplified in this embodiment may be provided as the feeling estimation device 3 as in this embodiment, or may be installed on the cloud. In addition, it may be provided as an application program and installed on an existing smartphone, PC, or the like so that it can be used as the feeling estimation device 3. [Industrial Applicability]

[0047] Incidentally, the emotion estimation system according to the present embodiment can be applied to various fields such as automatic driving of cars and medicine. [Explanation of symbols]

[0048] 1. Emotion estimation system 2 Imaging device (imaging means) 3 Emotion estimation device (prescribed device, computer) 36 Estimated emotion value generation unit (calculation means) 37 Estimated emotion estimation unit (estimation means) HM Subject

Claims

1. A calculation means for calculating the emotional valence and arousal level of the subject from the autonomic nerve information and behavioral information of the subject using a trained neural network; and an estimation means for estimating an emotion of the subject based on the emotional valence and the arousal level of the subject calculated by the calculation means.

2. The feeling estimation device according to claim 1 , wherein the autonomic nerve information and the behavior information of the subject are calculated based on a captured image of the subject captured by an imaging means.

3. In a method for estimating emotion, the method comprises: calculating the subject's valence and arousal level based on the subject's autonomic nerve information and behavioral information using a trained neural network; and estimating the emotion of the subject based on the calculated emotional valence and arousal level of the subject.

4. On the computer, A first process of calculating an emotional valence and an arousal level of a subject based on autonomic nerve information and behavioral information of the subject using a trained neural network; a second process of estimating an emotion of the subject based on the calculated valence and arousal level of the subject.

Images