Psychological state estimation device, psychological state estimation method, and program

Abstract

To provide a technology to estimate a psychological state easily and accurately.SOLUTION: A psychological state estimation apparatus includes: display means for displaying a predetermined image; imaging means for imaging the face of an observer viewing the predetermined image displayed by the display means; expression estimation means for estimating an expression of the observer from the image of the face imaged by the imaging means; and state estimation means for estimating a psychological state of the observer on the basis of a change in the expression of the observer viewing the predetermined image estimated by the expression estimation means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0006] , , , , ,

[0001] The present invention relates to a technique for estimating a mental state.

Background Art

[0002] In modern society where problems such as karoshi, accidents, and mental health problems caused by worker fatigue have become social issues, it is important to visualize and manage mental states such as fatigue and stress. In addition, there are work styles such as remote work that make it more difficult to grasp the mental state of workers than before, and there is a need for a technology that can easily estimate the mental state in various environments.

[0003] As a technique for estimating a mental state, for example, there is a method of using biological information such as heart rate and brain waves. However, in the method of using biological information, since a dedicated measuring device may be used in some cases, it is difficult to easily estimate the mental state at home or the like.

[0004] As another technique for estimating a mental state, there is a method of using a face image of a user. For example, in the technique disclosed in Patent Document 1, the degree of psychosomatic illness is determined from a face image of a user using a diagnostic matrix in which the knowledge of experts is digitized. In the technique disclosed in Patent Document 2, feature amounts related to the relative positions of face parts and the like are calculated from a face image of a user, and emotions are estimated.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the technology disclosed in Patent Document 1 does not take into account individual differences in how psychological states are expressed in facial expressions, which may result in low accuracy in estimating psychogenic illnesses. The technology disclosed in Patent Document 2 does not take into account whether the estimated emotion matches the emotion the user is actually feeling, which may result in low accuracy in estimating emotions.

[0007] This invention has been made in view of the above problems, and aims to easily and accurately estimate the psychological state of a user. [Means for solving the problem]

[0008] A first aspect of the present invention provides a psychological state estimation device comprising: a display means for displaying a predetermined image; an imaging means for capturing an image of the face of an observer viewing the predetermined image displayed by the display means; an expression estimation means for estimating the observer's expression from the image of the face captured by the imaging means; and a state estimation means for estimating the observer's psychological state based on the changes in the observer's expression as viewed by the expression estimation means.

[0009] The aforementioned predetermined image may be a still image or a moving image. Furthermore, the predetermined image should preferably be an image that induces facial expression imitation, such as an image of a person expressing a certain emotion. The facial image only needs to show the observer's (user's) face and may include the head, neck, upper body, etc. Facial expression estimation involves multiple expressions (e.g., neutral, joy, surprise, sadness, anger, etc.). The device may estimate a range of emotions, or it may estimate a single facial expression (for example, the percentage of joy). Similarly, it may estimate multiple psychological states or just one psychological state. The psychological state estimation device does not use specialized measuring instruments for estimating psychological states, but rather uses the facial image of the observer looking at a predetermined image, making it easy to estimate psychological states. Furthermore, the psychological state estimation device captures and uses the unconscious expression of emotions, specifically the change in facial expression when viewing a predetermined image, for estimation, allowing for accurate estimation of psychological states. Therefore, this configuration allows for easy and accurate estimation of the user's psychological state.

[0010] The state estimation means may analyze the correlation between the change in facial expression and the psychological state and estimate the psychological state of the observer. The change in facial expression may be a change over the entire period during which the facial image was acquired, or it may be a change over a specific period within that period. With this configuration, the psychological state of an observer viewing a predetermined image can be estimated from the change in their facial expression.

[0011] The state estimation means is best used to analyze the correlation between each observer. There are individual differences in the degree to which emotions are expressed in facial expressions. Therefore, this configuration allows for accurate estimation of psychological states while taking into account individual differences among observers.

[0012] The facial expression estimation means may calculate a facial expression score, which is a numerical representation of the facial expression. With this configuration, the psychological state can be estimated using the changes in the calculated facial expression score.

[0013] The state estimation means may estimate the psychological state based on the temporal change in the facial expression score over a predetermined period. The predetermined period may be, for example, the entire period over which the facial image was acquired, or a specific period such as the period before and after the predetermined image was displayed. The temporal change may be calculated, for example, from the waveform features shown in the time-series data of the facial expression score. With this configuration, it is possible to determine, for example, whether or not there is less change in facial expression compared to normal conditions, based on the temporal change in the facial expression score.

[0014] The state estimation means may estimate the psychological state based on the average value of the facial expression score over a period corresponding to the display period during which the predetermined image is displayed. The period corresponding to the display period is, for example, a period during which the observer's face image during facial imitation can be considered to have been acquired. The average value of the facial expression score may be the average value over periods corresponding to multiple display periods, or it may be the average value over a period corresponding to a single display period. With this configuration, it is possible to determine, for example, whether or not there is less facial expression change compared to normal times from the change in the average value of the facial expression score during facial imitation.

[0015] The imaging means captures images of the face during a period corresponding to the display period in which the predetermined image is displayed, and the face during a period corresponding to the non-display period in which the predetermined image is not displayed. The state estimation means estimates the psychological state based on the change in the facial expression score during the display period and the facial expression score during the non-display period. The change in the facial expression score during the display period and the facial expression score during the non-display period may be calculated using, for example, the mean value or the variance. With this configuration, it is possible to determine, for example, whether or not there is less change in facial expression compared to normal times, from the change in the facial expression score during the display period and the facial expression score during the non-display period.

[0016] The display means may display different images at predetermined intervals. For example, if the psychological state estimation device displays a predetermined image for 10 seconds, it may display a different image every 2 seconds. Also, if the predetermined image is to be displayed after a predetermined time has elapsed (for example, after 2 hours), the psychological state estimation device may display an image different from the one displayed during the previous display period. With this configuration, the observer may become accustomed to the displayed images, and the degree of facial expression imitation may change. This can help avoid a decrease in the accuracy of estimating psychological states, such as through transformation.

[0017] The predetermined image may include a positive image for inducing positive emotions in the observer and a negative image for inducing negative emotions in the observer. The positive image and the negative image may be the same image regardless of the observer, or may be different images according to the observer's preference. According to this configuration, it is possible to determine whether or not the degree of facial imitation for a specific expression has changed.

[0018] It is preferable to further include output means for outputting information indicating one or more emotions based on the mental state estimated by the state estimation means. The information indicating one or more emotions may be output in a manner that enables the mental state of the observer to be grasped. The output destination may be the device used by the observer, or may be a device different from the device used by the observer. According to this configuration, the observer or the observer's supervisor, etc. can know the estimation result of the observer's mental state.

[0019] A second aspect of the present invention provides a mental state estimation method characterized by including a display step of displaying a predetermined image, an imaging step of imaging the face of an observer who views the predetermined image displayed in the display step, an expression estimation step of estimating the expression of the observer from the image of the face imaged in the imaging step, and a state estimation step of estimating the mental state of the observer based on the change in the expression estimated in the expression estimation step.

[0020] A third aspect of the present invention provides a program for causing a computer to execute each step of the above-described mental state estimation method.

Effects of the Invention

[0021] According to the present invention, the mental state of a user can be easily and accurately estimated.

Brief Description of the Drawings

[0022] [Figure 1] FIG. 1 is a diagram showing a usage example of a state estimation device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing details of the configuration of the state estimation device. [Figure 3] FIG. 3 is a table showing an example of the facial expression estimation result. [Figure 4] FIG. 4 is a flowchart showing the estimation process of the mental state. [Figure 5] FIG. 5 is a diagram showing an example of changes in the facial expression score.

Mode for Carrying Out the Invention

[0023] <Application Example> First, an example of a scene to which the present invention is applied will be described. FIG. 1 is a diagram showing an example of the use of the state estimation device according to an embodiment of the present invention.

[0024] The state estimation device 1 is an electronic device (mental state estimation device) that estimates the mental state of a user (observer) 11. In FIG. 1, the user 11 is looking at an expression image 13 (a predetermined image), which is an image that induces facial expression imitation and is displayed on the display (display device) of the state estimation device 1. Note that facial expression imitation is a phenomenon in which, by looking at the facial expression of another person, one unconsciously and reflexively makes the same facial expression as that of the other person. By using facial expression imitation, for example, the mental state can be estimated without giving stress to the user 11 such as creating an expression for estimating the mental state. The expression image 13 includes, for example, an image for inducing positive emotions in the user and an image for inducing negative emotions. The state estimation device 1 estimates changes in the facial expression from the face image 12 obtained by imaging the face of the user 11 looking at the expression image 13, and estimates the mental state.

[0025] The functions and specifications of the client program for estimating the mental state are arbitrary. In this application example, a program (hereinafter referred to as "state estimation software") that outputs the estimation result of the mental state of the user is exemplified. First, the user 11 activates the state estimation software of the state estimation device 1. Then, the state estimation device 1 (specifically, the CPU operating according to the state estimation software) displays the expression image 13 on the display at a predetermined time.

[0026] The state estimation device 1 estimates the user's facial expression from the face image 12 when the user 11 is presented with the facial expression image 13. The state estimation device 1 estimates the psychological state by analyzing the correlation between individual changes in facial expression and psychological state. By analyzing the correlation between individual changes in facial expression and psychological state and estimating the psychological state, the state estimation device 1 can estimate the psychological state with higher accuracy by taking into account individual differences in how psychological state is expressed in facial expressions.

[0027] State estimation device 1 outputs the estimated result of the psychological state. State estimation device 1 may output the degree of a single emotion, such as "vitality level 90%", or it may output the degree of multiple emotions, such as "vitality level 70%, stress level 30%". State estimation device 1 may also output in two patterns, such as "normal / high stress" or "positive / negative". The estimation result may also be output to a device other than the one used by user 11, or to an external server. For example, by sending the estimation result to the supervisor's device, the supervisor can easily understand whether the subordinate's psychological state is good or bad.

[0028] (Configuration of the state estimation device) Next, with reference to Figure 2, a specific example of the configuration of the state estimation device 1 of this embodiment will be described.

[0029] The state estimation device 1 includes a display unit (display means) 20, an imaging unit (imaging means) 21, and a control unit 22. The control unit 22 includes an image storage unit 220, a timing storage unit 221, an expression estimation unit 222, an expression estimation dictionary 223, an expression estimation result storage unit 224, a feature quantity calculation unit 225, a feature quantity storage unit 226, a state estimation unit 227, a state estimation dictionary 228, and a state estimation result storage unit 229.

[0030] The display unit 20 displays a predetermined image (an image that induces facial expression imitation, such as a facial expression image) stored in the image storage unit 220 at a timing stored in the timing storage unit 221. The display unit 20 can be, for example, a liquid crystal display or an organic EL display.

[0031] The imaging unit 21 generates and outputs image data by photoelectric conversion. The imaging unit 21 is composed of an image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 21 captures the user's face image at the timing stored in the timing memory unit 221 and outputs the captured face image to the expression estimation unit 222. The imaging unit 21 captures the face image not only during the display period when the expression image is displayed (presented), but also during the non-display period when the expression image is not displayed.

[0032] The image storage unit 220 stores a predetermined image. The predetermined image stored in the image storage unit 220 may be an image acquired from outside the state estimation device 1 via an interface, or an image acquired by the imaging unit 21.

[0033] The timing memory unit 221 stores the display timing for displaying a predetermined image on the display unit 20 and the imaging timing for capturing the user's face image with the imaging unit 21.

[0034] The facial expression estimation unit 222 estimates the user's facial expression based on the facial image acquired by the imaging unit 21 and the facial expression estimation dictionary 223. The facial expression estimation unit 222 considers the differences in brightness of the parts that make up the face and Facial expressions are estimated using image features, which are characteristic quantities such as shape. Image features include, for example, Haar-like features obtained from local differences in brightness, and Hog features obtained from the distribution of local brightness gradients, but are not limited to these. The facial expression estimation unit 222 can estimate facial expressions using generally known techniques for determining facial expressions. The facial expression estimation unit 222 outputs the facial expression estimation result to the facial expression estimation result storage unit 224.

[0035] The facial expression estimation dictionary 223 is a dictionary that learns the correlation between image features and facial expressions using machine learning, etc. Machine learning includes, but is not limited to, cascade classifiers and CNNs (Convolutional Neural Networks).

[0036] In this embodiment, the facial expression estimation unit 222 calculates a facial expression score, which is a numerical representation of facial expressions, as a measure of how to express facial expressions. For example, the facial expression score is calculated from the proportion of "blank expression, joy, surprise, sadness, and anger" estimated by the facial expression estimation unit 222 from the acquired facial image. The facial expression estimation unit 222 may calculate the score for some of the "blank expression, joy, surprise, sadness, and anger" expressions, or it may include other expressions in its calculation.

[0037] Let's explain this in detail with reference to Figure 3. Figure 3 is a table showing an example of the facial expression estimation results. The facial expression estimation unit 222 calculates scores for positive expressions (joy, surprise) and negative expressions (anger, sadness) from each facial expression, excluding the neutral expression. The facial expression estimation unit 222 calculates the sum of positive expressions as positive numbers and negative expressions as negative numbers, and uses this sum as the facial expression score. For example, the score Sp for positive expressions, the score Sn for negative expressions, and the facial expression score Se at time 0 are calculated using the following equations 1 to 3. Sp=(70+13) / (70+13+7+5)×100=87.4 (Formula 1) Sn=(7+5) / (70+13+7+5)×100=12.6 (Formula 2) Se=Sp-Sn=87.4-12.6=74.7...(Formula 3)

[0038] Returning to the explanation of Figure 2, the facial expression estimation result storage unit 224 stores the facial expression estimation results output by the facial expression estimation unit 222. The facial expression estimation result storage unit 224 stores information indicating when and what kind of facial expression the user had. In addition to time information, the facial expression estimation result storage unit 224 may store only the facial expression score, or it may also store the proportion of each facial expression.

[0039] The feature calculation unit 225 calculates score features, which are features related to changes in the user's facial expressions. For example, the feature calculation unit 225 calculates score features from the amount of change in the facial expression score and outputs the calculated result to the feature storage unit 226. Details of the score features used by the feature calculation unit 225 will be described later. The feature storage unit 226 stores the score features output from the feature calculation unit 225.

[0040] The state estimation unit 227 analyzes the correlation between changes in facial expression and psychological state to estimate the user's psychological state. For example, the state estimation unit 227 may estimate the psychological state using results that have been previously learned for each individual (user). For example, the state estimation unit 227 estimates the user's psychological state using the state estimation dictionary 228. The state estimation dictionary 228 is a dictionary that has learned the correlation between psychological state and score features for each individual. When learning the correlation between psychological state and score features in advance, the state estimation dictionary 228 may define the current psychological state of the user from, for example, the user's answers to a questionnaire. The questionnaire may include multiple sets of questions, or it may accept answers to a single question. For example, the questions may be "Are you stressed?" and "Are you feeling energetic?" and the user may answer "yes / no". Alternatively, the question "What is your stress level today?" may be asked the user to input a numerical value for their stress level.

[0041] For example, the state estimation unit 227 estimates the psychological state without performing prior training on each individual. It is also possible that the state estimation unit 227 estimates the psychological state using a general-purpose dictionary created from experimental results from a large number of subjects (for example, a dictionary that has learned the correlation between psychological states defined from the responses of a large number of subjects to questionnaires and score features). Alternatively, the state estimation unit 227 may estimate the psychological state using rule-based inference. In rule-based inference, the state estimation unit 227 may estimate the psychological state based on rules created from general knowledge, such as the fact that a lack of facial expression changes indicates a high level of stress. Thus, the state estimation unit 227 may estimate the psychological state using results learned for each individual, or it may estimate the psychological state without learning for each individual.

[0042] The state estimation unit 227 analyzes the psychological state and score features for each individual, and can estimate the psychological state, taking into account individual differences in how psychological states are expressed in facial expressions. The state estimation unit 227 outputs the estimated psychological state to the state estimation result storage unit 229. The state estimation result storage unit 229 stores the estimated psychological state output by the state estimation unit 227.

[0043] The state estimation device 1 is comprised of a computer equipped with hardware resources such as a CPU (processor), memory, storage, and display device. Blocks 20-22 and 220-229 shown in Figure 2 are realized by the CPU loading programs (operating system, state estimation software, etc.) stored in storage into memory and executing those programs. However, the configuration of the state estimation device 1 is not limited to this. For example, some or all of the functions provided by the state estimation device 1 may be realized using dedicated hardware such as an ASIC or FPGA. Alternatively, some of the functions of the state estimation device 1 may be executed on a cloud server.

[0044] (Estimation process) Next, we will explain the psychological state estimation process flow with reference to Figure 4. Figure 4 is a flowchart of the psychological state estimation process.

[0045] In step S41, the state estimation device 1 displays an expression image on the display, which is an image that induces expression imitation. The expression images include positive images that induce positive emotions in the user (e.g., a smiling picture) and negative images that induce negative emotions in the user (e.g., a crying picture). It is desirable that the displayed images have some randomness. For example, if the image displayed as a positive image is the same every time, the user may become accustomed to it, which could reduce the accuracy of estimating the expression and psychological state. Therefore, it is desirable to control the state estimation device 1 to display a different expression image at predetermined intervals. For example, it is desirable to control the state estimation device 1 to display a positive image different from the positive image that was displayed last time. The expression images may be still images or moving images.

[0046] In step S42, the state estimation device 1 captures the face of the user who is viewing the facial expression image displayed in step S41, and acquires the face image.

[0047] In step S43, the state estimation device 1 estimates the user's facial expression from the face image acquired in step S42. For example, the state estimation device 1 estimates the scores (percentages) of "blank expression, joy, surprise, sadness, and anger" shown in the face image.

[0048] In step S44, the state estimation device 1 calculates an expression score from the scores of positive expressions (joy and surprise) and negative expressions (anger and sadness) estimated in step S43.

[0049] In step S45, the state estimation device 1 calculates a score feature, which is a feature of the change in the facial expression score. Here, the change in the facial expression score will be explained with reference to Figure 5.

[0050] Figure 5 shows examples of changes in facial expression scores. Graphs 501-504 are graphs where the horizontal axis is time and the vertical axis is the facial expression score. Graph 501 shows an example where the user's psychological state is normal. Graphs 502-504 show examples where the user's psychological state is high stress. Periods 511 and 512 correspond to the display period when positive facial expression images are displayed. Periods 521 and 522 correspond to the display period when negative facial expression images are displayed. Periods other than 511, 512, 521, and 522 correspond to the non-display period when no facial expression images are displayed.

[0051] For example, during high stress, it is expected that the activity of facial muscles will be suppressed and facial expressions will be less pronounced compared to normal times. Graph 502 shows an example where the change in facial expression score is less pronounced compared to Graph 501. Also, during high stress, it is expected that certain facial expressions will be amplified or suppressed compared to normal times. Graph 503 shows an example where positive facial expressions are suppressed compared to Graph 501 (facial expression scores are lower in periods 521 and 522). Furthermore, during high stress, it is expected that there will be a delay in the activity of facial muscles compared to normal times. Graph 504 shows an example where the facial expression score changes with a delay compared to Graph 501. Facial expression scores are expected to change in this way according to psychological state.

[0052] The state estimation device 1 calculates score features to evaluate such changes in the facial expression score. The score features are, for example, waveform patterns that show the temporal changes in the facial expression score over a predetermined period. For example, a model that handles time-series data, such as GBDT, may be used to capture the shape of the waveform itself as the score features. The predetermined period may be, for example, the waveform pattern over the entire period during which the face image was acquired. Alternatively, the predetermined period may be a specific period, such as the minute before and after the display of the facial expression image (for example, from one minute before period 511 to the first minute of period 511), or the display period and the non-display period of the displayed image (for example, from after period 511 to the period including period 521).

[0053] Alternatively, the score feature may be the average value of the facial expression scores during facial expression imitation (the period corresponding to the display period of the facial expression image). The average value may be the average value of the periods during which both positive and negative images are displayed, or the average value of either one of the periods. Furthermore, if there are multiple periods during which positive images are displayed, the average value of the combined period (for example, period 511 and period 512) may be used as the score feature.

[0054] Alternatively, the score feature may be the change in the facial expression score from the non-facial expression imitation period (the period corresponding to the period when the facial expression image is not displayed) to the facial expression imitation period. For example, in the example in Graph 501, the score feature may be the difference between the average facial expression score for periods other than 511, 512, 521, and 522 (non-facial expression imitation period) and the average facial expression score for 511 and 512 (facial expression imitation period).

[0055] Alternatively, the score feature could be the variance between the facial expression score during non-facial expression imitation and the facial expression score during facial expression imitation. For example, it could be the variance between the facial expression score during the period before the positive image is displayed and the period during which the positive image is displayed (e.g., period 511). Note that the score feature is not limited to these examples; any feature that can evaluate changes in the facial expression score is acceptable.

[0056] Returning to the explanation of Figure 4, in step S46, the state estimation device 1 estimates the user's psychological state from the score features calculated in step S45 using the state estimation unit 227. For example, the state estimation device 1 may estimate a high-stress state if the waveform changes calculated as score features are less pronounced than in normal conditions. Also, for example, the state estimation device 1 may estimate a high-stress state if the facial expression score when imitating the facial expression of the negative image calculated as score features is amplified compared to normal conditions. If present, it may be estimated as a high-stress state. The state estimation device 1 may estimate whether the state is "normal" or "high-stress," or it may estimate as "stress level n%." When estimating as "stress level n%," for example, it may be calculated by comparing the score features when the stress level is a predefined "m%" with the current score features and considering the degree to which changes in facial expression have become less pronounced or delayed.

[0057] For example, let's describe an example of the process when the state estimation unit 227 estimates the psychological state of user A using the state estimation dictionary 228. Here, we assume that the state estimation dictionary 228 is a pre-trained model that has been pre-trained using deep learning or the like on user A's tendencies. First, the state estimation unit 227 identifies (specifies) the user whose psychological state is to be estimated. The method of identifying the user can be any method that allows the state estimation device 1 to recognize "who the user is," for example, a method of personal identification from a facial image, or a method of having the user input their own ID. In the method of having the user input their own ID, the user may manually input their ID using a touch panel, or they may have an ID card (e.g., employee ID) read by a reader. Next, the state estimation unit 227 retrieves user A's dictionary (state estimation dictionary 228, pre-trained model). Next, the state estimation unit 227 inputs the data measured this time (e.g., score features) into user A's dictionary. Next, the state estimation unit 227 obtains the psychological state, which is the output of user A's dictionary.

[0058] Next, we will explain an example of the process when the state estimation unit 227 estimates the user's psychological state using a rule-based inference engine. In this case, for example, several dictionaries such as the following are prepared. Dictionary A is a dictionary that outputs "stress level (an indicator showing the likelihood of high stress)" from "the average value of the facial expression score during the display period of the positive image". Dictionary B is a dictionary that outputs "stress level" from "the time lag between the display period of the facial expression image and the change in the facial expression score". Dictionary C is a dictionary that outputs "stress level" from "the difference between the facial expression score during the display period of the positive image and the facial expression score during the display period of the negative image". The state estimation unit 227 may estimate the stress level (psychological state) using any one of dictionaries A to C. Alternatively, the state estimation unit 227 may integrate the stress levels calculated in each of dictionaries A to C (for example, the average or maximum value) and output the final stress level.

[0059] In step S47, the state estimation device 1 outputs the psychological state. The output destination may be the device the user is using, or it may be a different device from the one the user is using (such as an external server). The state estimation device 1 may output only the psychological state estimation result (e.g., stress level n%), or it may also output the facial expression estimation result (percentage of each facial expression, facial expression score, etc.).

[0060] In the example shown in Figure 5, positive and negative images are displayed alternately, but the method of displaying each image is not limited to this. For example, the positive image may be displayed for two consecutive periods, or the display period may be longer or shorter.

[0061] Furthermore, in the example of estimating psychological state using a pre-trained model, the input data was explained as score features. Depending on the design of the pre-trained model, it may also be possible to input time-series data (waveforms) of facial expression scores into the pre-trained model and obtain the psychological state as the output.

[0062] By using the state estimation software described above, users can easily and accurately estimate their psychological state.

[0063] <Other> The above embodiments are merely illustrative examples illustrating the configuration of the present invention. The present invention is not limited to the above-described specific forms, and various modifications are possible within the scope of its technical concept. For example, the above embodiments described an example in which a user's psychological state is estimated using state estimation software, but the applications of the present invention are not limited to this.

[0064] <Note> A display means (20) for displaying a predetermined image (13), An imaging means (21) captures the face of an observer (11) who is viewing the predetermined image (13) displayed by the display means, A facial expression estimation means (222) estimates the facial expression of the observer (11) from the facial image (12) captured by the imaging means (21), A state estimation means (227) estimates the psychological state of the observer (11) based on the changes in the observer's (11) facial expression as estimated by the facial expression estimation means (222) when the observer (11) views the predetermined image (13), and A psychological state estimation device (1) characterized by having the following. [Explanation of Symbols]

[0065] 1: State estimation device 11: User 12: Face image 13: Facial expression image

Claims

1. A display means for displaying a predetermined image, An imaging means for capturing images of the face of an observer viewing the predetermined image displayed by the display means, A facial expression estimation means for estimating the facial expression of the observer from the facial image captured by the imaging means, A state estimation means for estimating the psychological state of the observer based on the changes in the observer's facial expression while viewing the predetermined image, which are estimated by the facial expression estimation means. It has, A psychological state estimation device characterized in that the predetermined image includes a positive image for inducing positive emotions in the observer and a negative image for inducing negative emotions in the observer.

2. A display means for displaying a predetermined image, An imaging means for capturing images of the face of an observer viewing the predetermined image displayed by the display means, A facial expression estimation means for estimating the facial expression of the observer from the facial image captured by the imaging means, A state estimation means for estimating the psychological state of the observer based on the changes in the observer's facial expression while viewing the predetermined image, which are estimated by the facial expression estimation means. It has, The display means is characterized by displaying a different image at predetermined intervals, thereby providing a psychological state estimation device.

3. The state estimation means analyzes the correlation between the change in facial expression and the psychological state, and estimates the psychological state of the observer. A psychological state estimation device according to claim 1 or 2.

4. The state estimation means analyzes the correlation for each observer. The psychological state estimation device according to feature 3.

5. The facial expression estimation means calculates a facial expression score, which is a numerical representation of the facial expression. A psychological state estimation device according to any one of claims 1 to 4.

6. The state estimation means estimates the psychological state based on the temporal change in the facial expression score over a predetermined period. The psychological state estimation device according to feature 5.

7. The state estimation means estimates the psychological state based on the average value of the facial expression score for a period corresponding to the display period during which the predetermined image is displayed. The psychological state estimation device according to feature 5.

8. The imaging means captures images of the face during a period corresponding to the display period in which the predetermined image is displayed, and of the face during a period corresponding to the non-display period in which the predetermined image is not displayed. The state estimation means estimates the psychological state based on the change in the facial expression score during the period corresponding to the display period and the facial expression score during the period corresponding to the non-display period. The psychological state estimation device according to feature 5.

9. The system further includes an output means that outputs information indicating one or more emotions based on the psychological state estimated by the state estimation means. A psychological state estimation device according to any one of claims 1 to 8.

10. A display step in which a computer displays a predetermined image, The computer performs an imaging step in which it captures the face of an observer who is viewing the predetermined image displayed by the display step, A computer performs an expression estimation step in which it estimates the facial expression of the observer from the facial image captured in the imaging step, A state estimation step in which the computer estimates the psychological state of the observer based on the changes in facial expressions estimated in the facial expression estimation step. It has, A method for estimating a psychological state, characterized in that the predetermined image includes a positive image for inducing positive emotions in the observer and a negative image for inducing negative emotions in the observer.

11. A program for causing a computer to perform each step of the psychological state estimation method described in claim 10.

Images