Emotion determination device, emotion determination method, and emotion determination program

Abstract

This feeling determination device comprises: a detection unit which detects heartbeat information including the heart rate of a subject; a feeling determination unit which determines, on the basis of the heartbeat information, whether the feeling of the subject is a negative feeling or a positive feeling; a counting unit which counts, in a prescribed period, the number of heart rate variations in which the heart rate is transitioned from a state smaller than an average heart rate to a state equal to or greater than the average heart rate with an increase in a heart rate by at least a prescribed value; an emotion determination unit which can determine, on the basis of the number of the heart rate variations, a psychological state of the subject and use the determination result from the feeling determination unit for determining the psychological state; and an output unit which outputs the determination result from the emotion determination unit.

Description

Technical Field

[0001] The present invention relates to an emotion determination device, an emotion determination method, and an emotion determination program for determining the emotion of a target person.

Background Art

[0002] In a scene of caring for the elderly, it is very important for the caregiver to grasp what kind of emotions the elderly have towards the caregiving actions performed by the caregiver in order to determine the subsequent caregiving policy. However, due to aging, the elderly may have less expressive faces and may also have difficulty in conversation, so there is a problem that it is difficult to read emotions from the expressions, atmosphere, or conversation content of the elderly.

[0003] Also, in a scene of video conferencing or teleconferencing, it is very important for the speaker to grasp what kind of emotions the listener has when listening to the words spoken by the speaker in order to proceed with the conversation. However, in teleconferences, etc., the image showing the listener's expression is limited to the image captured by the camera, so there is a problem that it is difficult to read emotions from the listener's expression and atmosphere. Generally, as the name of a meeting conducted using a monitor via a communication line, depending on the scene of use, "teleconference", "Web conference", or other meeting names may be used, but in this specification, these are collectively referred to as "Web conference".

[0004] Patent Document 1 discloses a communication device including an emotion analysis means for analyzing the emotion of the other party during communication, a storage means for storing the emotion data analyzed by the emotion analysis means in correspondence with the other party who performed the emotion analysis, a notification means for performing a notification based on the emotion data stored in the storage means, and a control means for reading out the emotion data corresponding to the other party from the storage means and causing the notification means to perform a notification when the other party is selected.

[0005] Patent Document 2 discloses an image processing apparatus including an image data acquisition unit that acquires image data of a plurality of conference participants, a face image detection unit that detects face images of each conference participant from the image data acquired by the image data acquisition unit, an image composition unit that cuts out the detected face images and reconstructs them into one image, an emotion estimation unit that estimates the emotion of each participant based on the detected face images, and a display mode change unit that changes the display mode of the face images of each participant based on the estimated emotion.

[0006] In the invention described in Patent Document 1, since the voice of the call partner is analyzed to determine the emotion, the partner needs to speak, and it cannot be applied to subjects with little conversation or listeners in a meeting. Further, in the invention described in Patent Document 2, although the expression is estimated from the face images of the conference participants and the emotion is estimated based on the estimated expression, there is a problem that it is difficult to estimate the emotion of a participant with a lack of expression.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Patent Document 2

Summary of the Invention

[0008] An object of the present invention is to provide an emotion determination device, an emotion determination method, and an emotion determination program capable of detecting the emotion of a partner without contact.

[0009] An emotion determination device according to an embodiment of the present disclosure includes a detection unit that detects heartbeat information including the heartbeat of a subject, an emotion determination unit that determines whether the emotion of the subject is a negative emotion or a positive emotion based on the heartbeat information, a counting unit that counts the number of heartbeat fluctuations that transition from a state below the average heartbeat to a state above the average heartbeat with an increase in the heartbeat of a predetermined value or more within a predetermined period, an emotional state determination unit that determines the psychological state of the subject based on the number of heartbeat fluctuations and can use the determination result by the emotion determination unit for the determination of the psychological state, and an output unit that outputs the determination result of the emotional state determination unit.

[0010] The above-mentioned emotion determination device may further include a photographing unit that photographs the face of the subject, and the detection unit may detect the heartbeat information based on the change in the image data acquired by the photographing unit.

[0011] In the above-mentioned emotion determination device, the psychological state includes a first psychological state that can be determined based on the number of heartbeat fluctuations regardless of whether the emotion is positive or negative, and a second psychological state that can be determined based on either positive or negative emotion and the number of heartbeat fluctuations. The emotional state determination unit may use the determination result by the emotion determination unit at least when determining the second psychological state.

[0012] In the above-mentioned emotion determination device, the predetermined period is repeatedly set, and the emotional state determination unit may perform the determination for each predetermined period.

[0013] In the above-mentioned emotion determination device, the photographing unit can photograph the faces of a plurality of subjects. Further, it has a measurement site specifying means for specifying each face from a screen on which a plurality of people are displayed and specifying a measurement site for each specified face. The detection unit may acquire the heartbeat information based on the change in the image of the measurement site of each face.

[0014] In the above-mentioned emotion determination device, the subject is a learner taking a lecture, and the emotional state determination unit may determine whether it is an ideal psychological state in taking the lecture based on the number of heartbeat fluctuations and the determination result of the emotion determination unit.

[0015] In the above-described emotion determination device, a stimulus generation unit that generates a stimulus including at least one of a stimulus whose presence or absence can be sensorially recognized by vision or hearing, a stimulus that can understand the content of information given by vision or hearing, and an image or voice of a specific person, and the stimulus generation unit repeatedly generates the same type of stimulus in a plurality of predetermined periods sandwiching a rest period, and the emotion determination unit may determine the mental state of the subject at least during the plurality of predetermined periods.

[0016] In the above-described emotion determination device, negative emotion may be an emotion in which the subject feels at least one of brain fatigue, anxiety, and depression.

[0017] In the above-described emotion determination device, the mental state of the subject determined by the emotion determination unit may include any one of a stable state, a surprised state, a grateful state, and an indignant state.

[0018] In the above-described emotion determination device, when the number of heart rate fluctuations counted by the counting unit is 1, the emotion determination unit may determine that the mental state is a surprised state.

[0019] In the above-described emotion determination device, when the number of heart rate fluctuations counted by the counting unit is a plurality of times and the emotion of the subject determined by the emotion determination unit is a positive emotion, the emotion determination unit may determine that the mental state is a grateful state.

[0020] In the above-described emotion determination device, when the number of heart rate fluctuations counted by the counting unit is a plurality of times and the emotion of the subject determined by the emotion determination unit is a negative emotion, the emotion determination unit may determine that the mental state is an indignant state.

[0021] In the above-described emotion determination device, when the number of heart rate fluctuations counted by the counting unit is 0 and a state where the heart rate is less than the average heart rate is maintained for a predetermined period, the emotion determination unit may determine that the mental state is a stable state.

[0022] In the above-described emotion determination device, when the number of heart rate fluctuations counted by the counting unit is 0 and the state where the heart rate is equal to or higher than the average heart rate is maintained for a predetermined period, the emotion determination unit may determine that it is an emotion determination impossible state in which the mental state cannot be determined.

[0023] The emotion determination program according to an embodiment of the present disclosure causes a computer to execute: a step of detecting heart rate information including the heart rate of a subject; a step of determining whether the emotion of the subject is a negative emotion or a positive emotion based on the heart rate information; a step of counting the number of heart rate fluctuations that have transitioned from a state below the average heart rate to a state equal to or higher than the average heart rate with an increase in the heart rate of a predetermined value or more in a predetermined period; a step of determining the mental state of the subject based on the number of heart rate fluctuations and using the determination result for the determination of the mental state; and a step of outputting the determination result of the mental state.

[0024] In the emotion determination method according to an embodiment of the present disclosure, a detection unit detects heart rate information including the heart rate of a subject, an emotion determination unit determines whether the emotion of the subject is a negative emotion or a positive emotion based on the heart rate information, a counting unit counts the number of heart rate fluctuations that have transitioned from a state below the average heart rate to a state equal to or higher than the average heart rate with an increase in the heart rate of a predetermined value or more in a predetermined period, an emotion determination unit determines the mental state of the subject based on the number of heart rate fluctuations and uses the determination result by the emotion determination unit for the determination of the mental state, and an output unit outputs the determination result of the emotion determination unit.

[0025] According to the emotion determination device, emotion determination method, and emotion determination program according to an embodiment of the present disclosure, it is possible to detect the emotion of the other party without contact.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Embodiments for Carrying Out the Invention

[0027] Hereinafter, with reference to the drawings, an emotion determination device, an emotion determination method, and an emotion determination program according to the present invention will be described. However, it should be noted that the technical scope of the present invention is not limited to these embodiments, and extends to the invention described in the claims and its equivalents.

[0028] FIG. 1 is a perspective view showing a usage state of an emotion determination device 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the emotion determination device 1 includes an imaging unit 10 and an information terminal 5. In the illustrated example, the imaging unit 10 is a mobile terminal such as a smartphone, and the information terminal 5 is a notebook personal computer (PC) having a display unit 41. However, the present invention is not limited to such an example, and a tablet terminal or a digital camera etc. may be used as the imaging unit 10, and a tablet terminal, a desktop PC, or a dedicated processing device etc. may be used as the information terminal 5. The imaging unit 10 and the information terminal 5 may be integrated.

[0029] Figure 1 shows a state in which the imaging unit 10 is installed on a stand 90 that holds the imaging unit 10. As shown in Figure 1, the imaging unit 10 includes an image sensor 11 and a touch panel 19 with a display unit for setting the operation of the imaging unit 10.

[0030] Among the elderly in particular, there are people who have a fear of measurement, people who feel resistance to the measurement act itself such as wearing a sensor during measurement, and people who show negative emotions temporarily just by listening to the measurement explanation. In some cases, negative emotions cannot be measured properly. Therefore, the emotion determination device 1 uses the imaging unit 10 having the image sensor (camera) 11 to photograph the exposed part of the subject's skin (for example, parts such as the forehead or cheek of the face) so that the measurement itself does not cause stress. Then, the emotion determination device 1 automatically detects a pulse wave signal, which is the subject's heartbeat information, non - contact with the subject and while the subject is unconscious, by extracting the luminance change synchronized with the blood flow from the obtained image.

[0031] The image sensor 11 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type image sensor. The image sensor 11 automatically takes a plurality of consecutive images, for example, an image Gr of the measurement frame Sa on the forehead of the subject HK as shown in Figure 1, at regular intervals without the subject's operation. The imaging unit 10 has a function of automatically tracking the measurement frame Sa on the forehead of the subject HK by a built - in face recognition application program. Thereby, even if the subject HK moves around within the installation area of the imaging unit 10, it is possible to capture the pulse wave of the subject HK. As shown in Figure 1, the imaging unit 10 transmits the photographed image data of the subject HK to the information terminal 5 via radio waves RW by a built - in wireless communication function.

[0032] FIG. 2 is a block diagram of the emotion determination device 1. As shown in FIG. 1, the information terminal 5 of the emotion determination device 1 includes an emotion detection unit 20, a determination unit 30, a notification unit 40, and a timing unit 50. The emotion detection unit 20 includes a face recognition unit 21, a pulse wave extraction unit 22, an interval detection unit 23, a pulse wave memory 24, a chaos analysis unit 25, and a counting unit 26. The determination unit 30 includes an emotion determination unit 31 and an emotion determination unit 32. The notification unit 40 includes a display unit 41 and a transmission unit 42. Among these, the pulse wave memory 24 may be configured by a hard disk, a semiconductor memory, or the like, the display unit 41 may be configured by a liquid crystal display, and the timing unit 50 may be configured by a known clock circuit. Other elements are realized as software (program) by a computer in the information terminal 5 including a CPU, a ROM, a RAM, and the like.

[0033] The face recognition unit 21 analyzes the state of the face of the subject HK in the image Gr captured by the imaging element 11 using a contour detection algorithm or a feature point extraction algorithm, and specifies the skin-exposed part such as the forehead as the measurement part. The face recognition unit 21 outputs a time-series signal E1, which is data indicating the skin color at the measurement part, to the pulse wave extraction unit 22.

[0034] The pulse wave extraction unit 22 extracts the pulse wave signal of the subject HK from the time-series signal E1 and outputs the signal to the interval detection unit 23. Since capillaries are concentrated inside the measurement frame Sa on the forehead of the subject HK, the image Gr includes a luminance change component synchronized with the blood flow of the subject HK. In particular, since the pulse wave (blood flow change) is most reflected in the luminance change component of the green light of the image Gr, the pulse wave extraction unit 22 uses a band-pass filter that passes a frequency of approximately 0.5 to 3 [Hz] that a human pulse wave has to extract the pulse wave signal from the luminance change component of the green light of the time-series signal E1.

[0035] The imaging unit 10, the face recognition unit 21, and the pulse wave extraction unit 22 are an example of a detection unit that detects the heartbeat information of the subject. However, the functions of the detection unit do not necessarily have to be separated into the imaging unit 10 and the information terminal 5. For example, the functions of the face recognition unit 21 and the pulse wave extraction unit 22 may be provided in the imaging unit 10, or the imaging unit 10 may be included in the information terminal 5.

[0036] Figures 3(A) to (C) are graphs for explaining the principle of determining negative emotions from a pulse wave. Among these, Figure 3(A) shows an example of the waveform of the pulse wave signal PW, where the horizontal axis t represents time (milliseconds) and the vertical axis A represents the intensity of the amplitude of the pulse wave. As shown in Figure 3(A), the pulse wave signal PW is in the shape of a triangular wave reflecting the variation in blood flow volume due to the heartbeat of the heart, and the intervals between the peak points P1 to P(n + 1) indicating the strongest amplitude intensity of the pulse wave when the blood flow volume is the largest are defined as the pulse wave intervals d1 to dn.

[0037] The interval detection unit 23 detects the peak points P1 to P(n + 1) of the pulse wave signal PW of the subject HK, calculates the pulse wave intervals d1 to dn in units of milliseconds using the timing unit 50, and further generates time series data of the pulse wave intervals from the pulse wave intervals d1 to dn.

[0038] The pulse wave memory 24 stores the pulse wave intervals d1 to dn detected by the interval detection unit 23 as time series data of the pulse wave intervals.

[0039] Figure 3(B) is a graph showing an example of the fluctuation degree of the pulse wave interval. This graph is called a Lorenz plot, with the horizontal axis being the pulse wave interval dn and the vertical axis being the pulse wave interval dn - 1 (both in units of milliseconds), and the time series data of the pulse intervals are plotted on the coordinates (dn, dn - 1) for n = 1, 2, ···. Since it is known that the degree of dispersion of the dot R in the graph of Figure 3(B) reflects the degree of brain fatigue of the subject HK, if the data scatter plot of Figure 3(B) is displayed on the display unit 41, it is also possible to simply monitor the degree of brain fatigue of the subject HK during measurement.

[0040] The chaos analysis unit 25 calculates the maximum Lyapunov exponent λ by the following formula (1) using the time series data of the pulse wave intervals stored in the pulse wave memory 24, that is, the coordinates (dn, dn - 1) in the Lorenz plot of Figure 3(B).

Equation

[0041] A brief explanation of the maximum Lyapunov exponent is as follows. If the heart of mammals including humans operates at a constant cycle like a machine, specific parts will be prone to fatigue and breakage. In order to continue driving for a long time, it usually avoids fatigue through the fluctuations of a complex system. This is the role of the autonomic nervous system. When stress or mental damage is applied here, the autonomic nervous system will concentrate on dealing with the stressor, so the fluctuations of the complex system cannot occur. It is not simply about the presence or absence of fluctuations in the complex system (chaotic nature), but the degree of fluctuations in the complex system correlates with positive and negative emotions (autonomic nervous system). For example, if the fluctuations have a specific periodicity, the graph will be a circle or an ellipse, and in this case, there are no fluctuations in the complex system. Therefore, when the maximum Lyapunov exponent is positive, it means there are fluctuations in the complex system, indicating that the subject has positive emotions. On the other hand, when the maximum Lyapunov exponent is negative, it means there are no fluctuations in the complex system, indicating that the subject has negative emotions. In this way, by using the maximum Lyapunov exponent as an index, the degree of fluctuations in the complex system can be quantified, and it can be determined whether the subject has positive or negative emotions. This is the principle of emotion sensing using the maximum Lyapunov exponent.

[0042] Note that when performing emotion determination, instead of using the maximum Lyapunov exponent, the LF / HF method may be used. The LF / HF method is used to evaluate autonomic nerve activity from heart rate variability, and it is the ratio of the low-frequency component (LF: Low frequency) to the high-frequency component (HF: High frequency) (LF A method using (LF / HF) as a sympathetic nerve index. When performing frequency analysis on the fluctuations in heart rate or pulse wave interval, with LF being the power spectrum from 0.04 to 0.15 Hz and HF being the power spectrum from 0.15 to 0.4 Hz, if LF / HF is less than 2.0, it is determined as "positive emotion"; if it is 2.0 or more and less than 5.0, it is determined as "slightly negative emotion"; and if it is 5.0 or more, it can be determined as "negative emotion". However, while the maximum Lyapunov exponent can be analyzed based on the fluctuations in the pulse wave over 30 seconds, the LF / HF method requires a measurement time of about 3 minutes to accurately measure the low-frequency components of LF.

[0043] Figure 3(C) is a graph showing the relationship between the maximum Lyapunov exponent indicating the degree of fluctuation in the interval between heartbeats or pulse waves and negative emotion. This graph is obtained by conducting a questionnaire survey on 10 adult men and women, asking them about the degree of fatigue they feel and whether they feel that fatigue is a state of mental fatigue, anxiety, or depression, and measuring the maximum Lyapunov exponent λ of the pulse wave interval for the same subjects, and summarizing the relationship between the obtained response content and the value of λ. F0 corresponds to "no fatigue", F1 corresponds to "age-appropriate fatigue", F2 corresponds to "temporary fatigue", F3 corresponds to "chronic fatigue", and F4 corresponds to "negative emotion". The vertical axis of the graph is the maximum Lyapunov exponent λ.

[0044] From Figure 3(C), it can be seen that the maximum Lyapunov exponent λ has a small absolute value close to 0 in a state of mere fatigue, but becomes a large negative absolute value when there is negative emotion. Regarding the 10 adult men and women, considering the measurement variations, the threshold of the maximum Lyapunov exponent for determining whether negative emotion is felt can be set to about -0.6.

[0045] When the maximum Lyapunov exponent λ obtained by the chaos analysis unit 25 satisfies the following formula (2), the emotion determination unit 31 determines that negative emotion has occurred in the subject, and when λ does not satisfy formula (2), it determines that negative emotion has not occurred in the subject. λ ≦ λt ···(2) Here, the threshold value λt is −0.6, but other values ​​may be used depending on the characteristics required for the emotion determination device 1. The emotion determination unit 31 is an example of an emotion determination unit that determines, based on the maximum Lyapunov exponent, whether the emotion of the subject is a negative emotion in which at least one of brain fatigue, anxiety, and depression is felt, or whether the emotion of the subject is a positive emotion in which none of brain fatigue, anxiety, and depression is felt.

[0046] The counting unit 26 counts the number of times that the heart rate changes from a state below the average heart rate to a state above the average heart rate, accompanied by an increase in the heart rate of a predetermined value or more, during a predetermined period of time. Figs. 4(A) to (C) are graphs for explaining the principle of determining emotions from heart rate fluctuations during a predetermined period of time. The vertical axis indicates the heart rate, and the horizontal axis indicates time. ave is the average heart rate of subject HK If the average heart rate of the subject HK is unknown, an average value of 65 bpm may be used. ave The above area is called the "emotional area" and the heart rate is the average heart rate. beat number b ave The area below this is called the "rest area."

[0047] As shown in Figures 4(A) to 4(C), the heart rate varies with time. The counter 26 detects when the heart rate in a predetermined period transitions from the resting region to the emotional region, that is, when the average heart rate b ave Less than Average heart rate from state b ave The state transition occurs when the heart rate fluctuation width Δs is within a predetermined range. The number of heart rate fluctuations that are greater than the value is counted. For example, the predetermined period may be 30 [sec] and the predetermined value may be 10 [bpm]. The predetermined period of 30 [sec] corresponds to the minimum period that can maintain the accuracy of the Lyapunov analysis. However, the predetermined period and the predetermined value are not limited to these values, and can be appropriately set according to the degree of change in the subject's heart rate associated with changes in emotions.

[0048] FIG. 4(A) shows the heart rate over a given period of time, average heart rate b ave Here is an example of the following: In this case, the curve L1 representing the heart rate during a predetermined period is in the resting region, and the counter 26 counts the number of times that the heart rate has transitioned from a state where the heart rate is below the average heart rate to a state where the heart rate is equal to or higher than the average heart rate as 0.

[0049] FIG. 4B shows an example in which the heart rate fluctuates significantly only once. As shown in FIG. 4B, in the curve L2 representing the heart rate, the heart rate s1 is equal to the average heart rate b ave is less than s2 is the average heart rate b ave Furthermore, the difference between s1 and s2, Δs, is equal to or greater than a predetermined value. In this case, the counting unit 26 counts the number of heart rate fluctuations in a predetermined period in which the heart rate has transitioned from a state below the average heart rate to a state above the average heart rate accompanied by an increase in heart rate of a predetermined value or more as one.

[0050] FIG. 4C shows an example in which the heart rate fluctuates multiple times. As shown in FIG. 4C, in the curve L3 representing the heart rate, the heart rates s1, s3, and s5 are the average heart rate b ave Less than 100% and heart rate s2, s4, and s6 are average heart rates b ave That's all. Furthermore, the difference between s1 and s2, Δs1 , Δs2 which is the difference between s3 and s4, and Δs3 which is the difference between s5 and s6 are all equal to or greater than a predetermined value, the counting unit 26 counts the number of heart rate fluctuations in a predetermined period in which the heart rate has increased by a predetermined value or more and transitioned from a state below the average heart rate to a state above the average heart rate as three. In FIG. 4C, an example in which the heart rate fluctuation number is three is shown, but the present invention is not limited to such an example, and even if the heart rate fluctuation number is two or four or more, the counting unit 26 determines that the heart rate fluctuation number is multiple.

[0051] The emotion determination unit 32 determines the mental state of the subject HK based on the determination result by the emotion determination unit 31 and the number of fluctuations in the heart rate counted by the counting unit 26. In FIGS. 4(A) to 4(C), only the period of 30 [sec], which is the initially set period, is shown as the predetermined period, but the predetermined period may be repeatedly set. Also, the emotion determination unit may perform determination for each predetermined period. Here, the mental state of the subject HK includes a stable state, a surprised state, a grateful state, and an indignant state. Therefore, the emotion determination unit 32 determines which of the stable state, the surprised state, the grateful state, and the indignant state the mental state of the subject HK is based on the determination result by the emotion determination unit 31 and the number of fluctuations in the heart rate counted by the counting unit 26. Note that "grateful" means to feel strongly in the heart and be overwhelmed with emotion, and includes the meanings of other synonyms such as "being moved" (kan dou), "being impressed" (kan mei), "admiring" (kan shin), and "joy" (yorokobi). Also, "indignant" means to be enraged and includes the meanings of other synonyms such as "anger" (ikari).

[0052] When the number of fluctuations in the heart rate counted by the counting unit 26 is 0 and the state where the heart rate of the subject HK is less than the average heart rate is maintained in a predetermined period, the emotion determination unit 32 determines that the mental state of the subject HK is a stable state. Since the heart rate is in the resting range because it is less than the average heart rate and has never been equal to or more than the average heart rate in the predetermined period, the emotion determination unit 32 can determine that the mental state of the subject HK is a stable state.

[0053] When the number of heart rate fluctuations counted by the counting unit 26 is 1, the emotion determination unit 32 determines that the mental state is a surprised state. When the mental state of the subject HK is a surprised state, the heart rate is considered to fluctuate only once within a predetermined period and not continuously. Also, the subject HK can be in a surprised state when the subject HK's emotion is either a positive emotion or a negative emotion. Therefore, when the number of heart rate fluctuations counted by the counting unit 26 is 1, the emotion determination unit 32 can determine that the mental state of the subject HK is a surprised state regardless of whether the emotion of the subject HK determined by the emotion determination unit 31 is a positive emotion or a negative emotion.

[0054] When the number of heart rate fluctuations counted by the counting unit 26 is multiple, and the emotion of the subject HK determined by the emotion determination unit 31 is a positive emotion, the emotion determination unit 32 determines that the mental state of the subject HK is a grateful state. When the emotion of the subject HK is a positive emotion, the subject HK is considered to be in a pleasant mental state. Furthermore, when the subject HK is in a grateful mental state such as laughing because the video or the like that the subject HK has watched is interesting, the heart rate changes from the resting area below the average value to the emotion area above the average value, and this change is considered to continue to a certain extent. Therefore, when the number of heart rate fluctuations counted by the counting unit 26 is multiple, and the emotion of the subject HK determined by the emotion determination unit 31 is a positive emotion, the emotion determination unit 32 can determine that the mental state of the subject HK is a grateful state.

[0055] When the number of fluctuations in the heart rate counted by the counting unit 26 is plural, and the emotion of the subject HK determined by the emotion determination unit 31 is a negative emotion, the emotion determination unit 32 determines that the mental state of the subject HK is an angry state. When the emotion of the subject HK is a negative emotion, the subject HK is considered to be in an unpleasant mental state. Further, when the subject HK is in an angry mental state such as being angry with a person he dislikes, the heart rate changes from the resting area below the average value to the emotion area above the average value, and this change is considered to continue to a certain extent. Therefore, when the number of fluctuations in the heart rate counted by the counting unit 26 is plural, and the emotion of the subject HK determined by the emotion determination unit 31 is a negative emotion, the emotion determination unit 32 can determine that the mental state of the subject HK is an angry state.

[0056] When the number of fluctuations in the heart rate counted by the counting unit 26 is 0, and the state where the heart rate of the subject HK is equal to or higher than the average heart rate is maintained for a predetermined period, the emotion determination unit 32 determines that it is a state where the mental state of the subject HK cannot be determined, an emotion determination impossible state. FIGS. 5(A) and (B) are graphs showing examples of cases where it is difficult to determine emotions from the fluctuations in the heart rate over a predetermined period. FIG. 5(A) shows a state where the curve L4 representing the heart rate is stable in the emotion area over a predetermined period, and FIG. 5(B) shows a state where the curve L5 representing the heart rate fluctuates greatly in the emotion area over a predetermined period. However, in these cases, the heart rate is in the emotion area and is always equal to or higher than the average heart rate over a predetermined period. This is considered to be a state where the body of the subject HK is fatigued or is in a state of performing intense exercise, and it is considered to be a state where it is difficult to appropriately perform emotion determination. Therefore, when the number of fluctuations in the heart rate counted by the counting unit 26 is 0, and the state where the heart rate of the subject HK is equal to or higher than the average heart rate is maintained for a predetermined period, the emotion determination unit 32 can determine that it is a state where the mental state of the subject HK cannot be determined, an emotion determination impossible state.

[0057] Table 1 shows a list of the mental states of the subject HK determined by the emotion determination unit 32 based on the determination result of the emotion of the subject HK by the emotion determination unit 31 and the number of heart rate fluctuations counted by the counting unit 26. [Table 1] As described above, when the number of heart rate fluctuations is 0, the emotion determination unit 32 determines that the mental state of the subject is in a "stable" state regardless of whether the determination result of the emotion determination unit 31 is a positive emotion or a negative emotion. When the number of heart rate fluctuations is 1, the emotion determination unit 32 determines that the mental state of the subject is in a "surprised" state regardless of whether the determination result of the emotion determination unit 31 is a positive emotion or a negative emotion. Thus, when the number of heart rate fluctuations is 0 or 1, the emotion determination unit 32 can determine the mental state of the subject from the number of heart rate fluctuations regardless of whether the result of the emotion determination is a positive emotion or a negative emotion. Here, the mental state that can be determined based on the number of heart rate fluctuations regardless of whether it is a positive or negative emotion is defined as the "first mental state".

[0058] On the other hand, when the number of heart rate fluctuations is multiple, if the determination result of the emotion determination unit 32 is a positive emotion by the emotion determination unit 31, it is determined that the psychological state of the subject is in a state of "gratitude", and if the determination result of the emotion determination unit 31 is a negative emotion, it is determined that the psychological state of the subject is in a state of "anger". Thus, when the number of heart rate fluctuations is multiple, the emotion determination unit 32 can determine the psychological state of the subject based on either the positive emotion or the negative emotion, which is the result of the emotion determination, and the number of heart rate fluctuations. Here, the psychological state that can be determined based on either the positive or negative emotion and the number of heart rate fluctuations is defined as the "second psychological state". In this way, the psychological state of the subject includes the first psychological state and the second psychological state. When the emotion determination unit 32 determines whether the second psychological state is in a state of "gratitude" or a state of "anger" when the number of heart rate fluctuations is multiple, it uses the determination result of whether the emotion of the subject determined by the emotion determination unit 31 is a positive emotion or a negative emotion. That is, the emotion determination unit 32 uses the determination result by the emotion determination unit 31 at least when determining the second psychological state.

[0059] In this way, the emotion determination unit 32 can use the determination result by the emotion determination unit 31 for the determination of the psychological state according to the psychological state to be determined.

[0060] However, even when the number of heart rate fluctuations is 0 or 1, the emotion determination unit 32 may determine the psychological state (for example, a more detailed psychological state) using the determination result by the emotion determination unit 31. In this case as well, the emotion determination unit 32 is an example where the determination result by the emotion determination unit 31 can be used for the determination of the psychological state.

[0061] The notification unit 40 causes the display unit 41 to display the determination result of the emotion of the subject HK by the emotion determination unit 32. In particular, the notification unit 40 causes the display unit 41 to display the determination result of the emotion of the subject HK by the emotion determination unit 32 and transmits it externally via the transmission unit 42. The notification unit 40 is an example of an output unit that outputs the determination result of the emotion determination unit 32.

[0062] FIG. 6 is a flowchart showing the operation of the emotion determination device 1. First, in step S101, the imaging unit 10 captures an image Gr of the measurement frame Sa of the subject HK by the imaging element 11, and transmits the image data to the information terminal 5. Subsequently, the face recognition unit 21 identifies the measurement site from the image data of the subject HK.

[0063] Next, in step S102, the pulse wave extraction unit 22 extracts the pulse wave signal of the subject HK from the time-series signal E1 of the skin color of the measurement site identified by the face recognition unit 21. The interval detection unit 23 calculates the pulse wave interval from the pulse wave signal, generates the time-series data thereof, and stores the time-series data in the pulse wave memory 24.

[0064] Next, in step S103, the chaos analysis unit 25 calculates the maximum Lyapunov exponent λ of the pulse wave interval based on the time-series data of the pulse wave interval stored in S102.

[0065] Next, in step S104, the counting unit 26 determines whether there is a heart rate fluctuation of a predetermined value or more in a predetermined period based on the temporal change of the heart rate extracted from the pulse wave signal. If there is no heart rate fluctuation of a predetermined value or more, in step S105, it is determined whether the heart rate of the subject HK in the predetermined period is less than the average heart rate.

[0066] If the state where the heart rate of the subject HK is less than the average heart rate is maintained in the predetermined period, in step S106, the emotion determination unit 32 determines that the emotion of the subject HK is in a stable state. On the other hand, if the state where the heart rate of the subject HK is equal to or more than the average heart rate is maintained in the predetermined period, in step S107, the emotion determination unit 32 determines that it is in an emotion determination impossible state where the emotion determination of the subject HK cannot be performed.

[0067] In step S104, if it is determined that there is a heart rate fluctuation equal to or greater than a predetermined value, then in step S108, the counting unit 26 determines whether the number of heart rate fluctuations of the subject HK in a predetermined period is only once. If the number of heart rate fluctuations of the subject HK is only once, then in step S109, the emotion determination unit 32 determines that the emotion of the subject HK is in a state of surprise.

[0068] On the other hand, in step S108, if the number of heart rate fluctuations of the subject HK in a predetermined period is not only once, that is, if it is a plurality of times, then in step S110, the emotion determination unit 31 compares the maximum Lyapunov exponent λ calculated in step S103 with the threshold value λt, and determines whether the subject HK has a positive emotion or a negative emotion.

[0069] If it is determined that the subject HK has a positive emotion, then in step S111, the emotion determination unit 32 determines that the emotion of the subject HK is in a state of gratitude. On the other hand, if it is determined that the subject HK has a negative emotion, then in step S112, the emotion determination unit 32 determines that the emotion of the subject HK is in a state of anger.

[0070] As described above, according to the emotion determination device according to the embodiment of the present disclosure, it is possible to non-contact determine which of the states of the mental state of the subject HK being stable, surprised, grateful, and angry from the image information of the subject HK.

[0071] By using the emotion determination device according to the embodiment of the present disclosure, for example, in a nursing facility or the like, it is possible to grasp the emotions of a person who cannot communicate well. That is, it is possible to determine whether the elderly or the like are happy (grateful) or angry (irate) with respect to the nursing care provided to the elderly or the like. Therefore, according to the determination result, the nurse can determine what kind of nursing care should be provided to the elderly or the like next. Also, it is possible to store the emotional rhythm of the subject such as the elderly at the same time of the same behavior every day, and it is also possible to view the emotional rhythm at the same time of the same behavior. By knowing the emotional rhythm at the same time of the same behavior, it is possible to indirectly grasp the physical condition change of the subject. Furthermore, by incorporating the emotion determination device according to the embodiment of the present disclosure into an automatic training device for the elderly or the like, the automatic training device can determine the emotions of the elderly or the like with poor emotional expression and expressions, and can provide appropriate advice to the elderly or the like while performing training.

[0072] [Example 1] Next, the emotion determination device according to Example 1 will be described. FIG. 7 is a schematic configuration diagram of a Web conference system using an information terminal provided with the emotion determination device according to Example 1.

[0073] In a video call such as a Web conference, there is a problem that it is difficult to read emotions from the expressions of the other party displayed on the display screen. The emotion determination device according to Example 1 determines the emotion of the other party from the image information of the other party transmitted from the other party side in a Web conference or the like.

[0074] The first emotion determination device 101 installed in front of the first subject HK1 and the second emotion determination device 201 installed in front of the second subject HK2 are connected by the Internet 100 and can hold a web conference between them. The first emotion determination device 101 includes a first information terminal 105, a camera 111, and a microphone 112. The camera 111 captures an image of the first subject HK1 and transmits it, together with the voice data collected by the microphone 112, to the second information terminal 205 of the second subject HK2 via the Internet 100. In particular, the camera 111 captures an image of the measurement frame Sa1 of the first subject HK1, and the first information terminal 105 transmits the image captured by the camera 111 to the second information terminal 205. The real-time data of the heart rate b1 of the first subject HK1 is displayed on the display unit 241 of the second information terminal 205 together with the image of the first subject HK1. The second information terminal 205 compares the heart rate b1 of the first subject HK1 with the average heart rate b ave1 and can determine the emotion of the first subject HK1 from the temporal change of the heart rate b1 over a predetermined period. In addition, the determination result of the emotion (for example, positive emotion) can be displayed in the emotion determination area 243 of the display unit 241, or the determination result of the emotion (for example, stable state) can be displayed in the emotion display area 244.

[0075] Similarly, the second emotion determination device 201 includes a second information terminal 205, a camera 211, and a microphone 212. The camera 211 captures an image of the second subject HK2 and transmits it, together with the voice data collected by the microphone 212, to the first information terminal 105 of the first subject HK1 via the Internet 100. In particular, the camera 211 captures an image of the measurement frame Sa2 of the second subject HK2, and the second information terminal 205 transmits the image captured by the camera 211 to the first information terminal 105. The real-time data of the heart rate b2 of the second subject HK2 is displayed on the display unit 141 of the first information terminal 105 together with the image of the second subject HK2. The first information terminal 105 compares the heart rate b2 of the second subject HK2 with the average heart rate b ave2By comparing with this, the emotion of the second subject HK2 can be determined from the temporal change of the heart rate b2 during a predetermined period. Further, the determination result of the emotion (for example, positive emotion) can be displayed in the emotion determination area 143 of the display unit 141, or the determination result of the emotion (for example, a state of gratitude) can be displayed in the emotion display area 144.

[0076] FIG. 7 shows an example of conducting a Web conference using the emotion determination device among two subjects, but a Web conference can also be conducted using the emotion determination device among two or more subjects in the same manner.

[0077] FIG. 8 is a block diagram of the emotion determination device according to the first embodiment. In addition to the configuration of the information terminal 5 according to the embodiment shown in FIG. 2, the configurations of the first information terminal 105 and the second information terminal 205 include a receiving unit (191, 291), a voice reproduction unit (145, 245), a voice determination unit (127, 227), and a transmitting unit (192, 292).

[0078] The transmitting units (192, 292) respectively transmit the image information and voice information of the first subject HK1 or the second subject HK2, which are acquired by the cameras (111, 211) and the microphones (112, 212), to the receiving unit 291 of the second information terminal 205 and the receiving unit 191 of the first information terminal 105, which are the terminals on the other side, via the Internet 100.

[0079] The receiving units (191, 291) respectively receive the image information and voice information of the second subject HK2 and the first subject HK1 transmitted from the transmitting unit 292 of the second information terminal 205 and the transmitting unit 192 of the first information terminal 105.

[0080] The voice reproduction units (145, 245) respectively reproduce the voice data included in the information received by the receiving units (191, 291). A speaker can be used for the voice reproduction units (145, 245).

[0081] The voice determination units (127, 227) respectively determine, based on the voice information received by the reception units (191, 291), whether the first subject HK1 or the second subject HK2 is the speaker or the listener from the duration of the voice information.

[0082] On the speaker side, emotions may be generated by the very act of speaking oneself, and it may not be possible to perform accurate emotion determination. Therefore, when the relationship between the speaker and the listener continues for a certain period of time (for example, 10 seconds or more) by the voice determination units (127, 227), emotion determination and emotional arousal determination of the listener may be performed, and the determination result of the listener's emotion may be displayed on the display unit of the information terminal on the speaker side at any time. By doing so, the speaker side can grasp what kind of emotions the listener has when listening to the speaker's conversation.

[0083] In the above-described embodiment, an example of analyzing emotions on the receiver side based on the image information transmitted from the sender side is shown. Since the emotion determination device according to Embodiment 1 calculates the heart rate (pulse) from the acquired video information, the frame rate of the video becomes the sampling rate as it is. Therefore, the transmission rate of the received video may be monitored, and emotion determination may be performed only when the transmission rate is within an appropriate range.

[0084] Also, when multiple people participate in a meeting simultaneously in a web conference or the like, it is conceivable that the image information of the other party cannot be acquired at an appropriate timing due to the limitation of the capacity of the communication line. In such a case, the emotion determination may be switched between being performed on the transmission side or the reception side according to the transmission rate value. Specifically, instead of determining emotions based on the image information transmitted in real time, after transmitting the pre-acquired image information of the listener to the speaker, a method of determining emotions on the speaker side (the first method) and a method of transmitting the result of determining emotions on the listener side to the speaker side (the second method) are conceivable. The first method is considered to increase the amount of image information to be transmitted. On the other hand, in the second method, it is necessary to adjust the communication format because the data of the determination result is added to the image information. Therefore, it is preferable to appropriately switch between the first method and the second method according to the transmission rate.

[0085] In addition, when the sender is angry or in other situations, directly communicating the emotional judgment result may harm the other party's trust. In such cases, according to the judgment result, the emotional judgment result may be indirectly fed back, for example, by changing the color of the border of the sender's subject video. In addition to changing the color of the border of the subject video, the color of the curves of the graphs of the measurement frames (Sa1, Sa2) and heart rates (b1, b2) may also be changed. For example, the emotion may be displayed by the color of the measurement frames (Sa1, Sa2), and the emotion may be displayed by the color of the curves of the graphs of the heart rates (b1, b2).

[0086] According to the emotion judgment device according to Embodiment 1, in a Web conference or the like, it is possible to judge the emotion of a listener who is listening to the speaker's speech.

[0087] [Embodiment 2] Next, the emotion judgment device according to Embodiment 2 will be described. FIG. 9 is a diagram showing a display example of the screen of the emotion judgment device according to Embodiment 2. The emotion judgment device according to Embodiment 2 performs emotion judgment on a video displayed on the video display device 300 on existing software such as dedicated Web conference software, video playback applications, and video players.

[0088] Unlike the emotion judgment device according to Embodiment 1, the emotion judgment device according to Embodiment 2 captures the display video of the existing software on the screen from the video RAM (VRAM) 301 instead of using an imaging device to acquire the image of the subject, and executes emotion judgment. The image of the subject HK captured from the video RAM 301 is displayed on the display unit 41 of the information terminal 5, and the temporal change of the heart rate b from the pulse wave in the measurement frame Sa is the average heart rate b ave and is displayed on the display unit 41 together. Furthermore, the judgment result of emotion (for example, negative emotion) can be displayed in the emotion judgment area 43 of the display unit 41, or the judgment result of emotion (for example, an angry state) can be displayed in the emotion display area 44.

[0089] FIG. 10 is a block diagram of the emotion determination device 102 according to Example 2. The emotion determination device 102 includes a video display device 300 and an information terminal 5. The information terminal 5 includes a video acquisition unit 28 in addition to the configuration of the information terminal 5 according to the embodiment shown in FIG. 2, and includes an audio reproduction unit 45 instead of a transmission unit.

[0090] The video acquisition unit 28 automatically takes a plurality of consecutive images of the measurement frame Sa of the subject HK. The video acquisition unit 28 has a function of automatically tracking the measurement frame Sa on the forehead of the subject HK by a built-in face recognition application program. Thereby, even if the position of the measurement frame Sa of the subject HK moves within the display area on the display unit 41, it is possible to capture the pulse wave of the subject HK.

[0091] The audio reproduction unit 45 reproduces the audio data included in the information acquired by the video acquisition unit 28. A speaker can be used for the audio reproduction unit 45.

[0092] In the emotion determination device according to Example 1, the image of the subject was captured by a camera. However, in the emotion determination device according to Example 2, without acquiring an image again, using an existing video, the person included in the video is used as the subject, and the emotion of this person can be determined. Therefore, for example, it is possible to determine the emotion of a person shown in a video uploaded to a video sharing service. Specifically, for example, when a video of a person's apology press conference is uploaded to the video display device 300, the image data of that person is taken from the video RAM 301 into the video acquisition unit 28, and by performing emotion determination, it is possible to determine what kind of emotion the person had during the apology press conference.

[0093] Alternatively, when an image of a person with whom a video call is made using the video display device 300 is displayed, the image of the call partner is taken from the video RAM 301 into the video acquisition unit 28, and by performing emotion determination, it is possible to determine in real time what kind of emotion the person with whom the video call is being made has.

[0094] Furthermore, when video software such as movies or dramas is displayed using the video display device 300, the images of the actors appearing in the movie or drama are captured from the video RAM 301 by the video acquisition unit 28, and by performing emotion determination, it is possible to determine what kind of emotions the actor performing the acting had.

[0095] As described above, according to the emotion determination device according to the second embodiment, it is possible to determine the emotion of the speaker in a press conference for apology or the like, or to enjoy a video while analyzing the emotions of the actors in a drama or the like.

[0096] [Embodiment 3] Next, the emotion determination device according to the third embodiment will be described. FIG. 11 is a schematic configuration diagram of a marketing survey system using the emotion determination device according to the third embodiment. The emotion determination device 103 according to the third embodiment includes an information terminal 5 and an imaging unit 10. The subject HK observes the image of the product displayed on the display 400. The imaging element 11 of the imaging unit 10 captures the image of the measurement frame Sa of the subject HK, and the imaging unit 10 transmits the image of the subject HK captured by the imaging element 11 to the information terminal 5. The information terminal 5 determines the emotion when the subject HK observes the product displayed on the display 400, and transmits the determination result to the statistical analysis server 500 via the Internet 100. The statistical analysis server 500 can analyze the emotions when a plurality of subjects observe the same product, and analyze the appeal effect of the product on the consumer subjects.

[0097] FIG. 12 is a block diagram of the emotion determination device according to the third embodiment. The emotion determination result of the subject HK is transmitted from the transmission unit 42 to the statistical analysis server 500 via the Internet 100, and the statistical analysis server 500 analyzes the emotion determination result of the subject HK. Therefore, since it is not necessary to display the emotion determination result of the subject HK on the information terminal 5, the information terminal 5 may not be provided with a display unit.

[0098] The emotion determination device 103 according to Embodiment 3 can be applied to a marketing survey using digital signage, and it is possible to analyze whether a certain product is favorably regarded or not from the emotion determination results of the subjects (consumers). For example, by combining an information display device such as digital signage with the emotion determination device 103 and conducting product display, the emotions of consumers (subjects) towards a certain product can be observed in real time. When the frequency of emotions is high with positive emotions, it can be determined that the product is favored by consumers, and thus it can be utilized in a marketing survey.

[0099] Furthermore, since the emotion determination device 103 can extract only the emotion determination data of consumers (subjects) towards a certain product, a marketing survey can be conducted without handling personal information.

[0100] [Embodiment 4] Next, the emotion determination device according to Example 4 will be described. FIG. 13 is a schematic configuration diagram of a machine control system using the emotion determination device according to Example 4. The emotion determination device 104 according to Example 4 includes an information terminal 5 and an imaging unit 10. The emotion determination device 104 enables the information terminal 5 to determine the emotion of the subject HK who performs the machine operation, and based on the result, the information terminal 5 transmits a control signal to the machine 2 so that the subject HK can operate the machine safely. The imaging element 11 of the imaging unit 10 installed in the machine 2 acquires an image of the measurement frame Sa of the subject HK who is the operator of the machine 2 and transmits it to the information terminal 5. The information terminal 5 determines the emotion of the subject HK from the received image data, and transmits a control signal for controlling the machine 2 to the receiving unit 220 of the machine 2 based on the determination result. The receiving unit 220 inputs the received control signal to the control unit 210, and the control unit 210 controls the machine 2 according to the control signal. For example, when it is determined that the emotion of the subject HK who is the operator is in an angry state and the operator is in a state where the machine 2 cannot be operated normally, the information terminal 5 may transmit a control signal for forcibly stopping the machine 2 to the machine 2. Further, when forcibly stopping the machine 2 may instead endanger the subject HK, a signal for issuing a warning to return the emotion of the subject HK to a stable state may be transmitted.

[0101] FIG. 14 is a block diagram of the emotion determination device 104 according to Example 4. The information terminal 5 constituting the emotion determination device 104 according to Example 4 includes a control signal generation unit 46 that generates a signal for controlling the machine 2.

[0102] The control signal generation unit 46 generates a control signal for controlling the machine 2 based on the result of the emotion determination of the subject HK determined by the emotion determination unit 32. For example, when the emotion of the subject HK operating the machine 2 is in a stable state or a grateful state, it can be determined that there is no problem in continuing the operation of the machine 2 as it is, so no control signal for the machine 2 is generated. Alternatively, in this case, since it is determined that the subject HK can continue to operate the machine 2 without problems, a signal for continuing the control of the machine 2 may be generated.

[0103] On the other hand, when the emotion of the subject HK determined by the emotion determination unit 32 is in an angry state and there is a risk that the safety of the operator (subject) cannot be maintained if the operation of the machine 2 is continued as it is, the control signal generation unit 46 generates a signal to forcibly stop the machine 2 or a signal for issuing a warning to the operator based on the emotion determination result of the emotion determination unit 32.

[0104] The control signal for the machine 2 generated by the control signal generation unit 46 is transmitted to the receiving unit 220 of the machine 2 via the transmitting unit 42. The receiving unit 220 of the machine 2 inputs the received control signal to the control unit 210, and the control unit 210 controls the machine 2 based on the control signal.

[0105] Also, if the emotion determination unit 32 continuously determines the emotion of the operator (subject) and the state changes from an angry state to a stable state, the control signal generation unit 46 may generate a signal to resume the operation of the machine 2 based on the emotion determination result of the emotion determination unit 32 and transmit it to the machine 2.

[0106] In the above example, an example is shown in which the information terminal 5 transmits a control signal to the machine 2 based on the emotion determination result of the operator (subject) by the information terminal 5, but the example is not limited to this. For example, the emotion determination result of the operator determined by the information terminal 5 may be transmitted to a management center that manages a plurality of machines 2, and a control signal for the machine 2 may be transmitted from the management center side. By doing so, the management center can grasp the emotion determination result of the operator, and thus can also perform the health management of the operator at the same time.

[0107] By performing the control of the machine based on the emotion determination result of the operator using the emotion determination device according to the fourth embodiment, the safety of the operator can be ensured, and the emotion determination device can function as a so-called near-miss sensor.

[0108] In the embodiments described above, an example in which the emotion determination device is realized by an information terminal has been shown, but such examples are not limited thereto. For example, when realizing the operation of the emotion determination device on a control board and applying it to a digital signage, the control board may be incorporated into the display device. In this case, a camera may be installed in the display device, and the result of emotion determination executed on the control board may be transmitted to a server, and statistical analysis can be performed on the server side.

[0109] Also, although an example in which a mobile terminal such as a smartphone is used as the imaging unit has been shown, in order to image the images of the elderly and other subjects while providing care in a nursing facility or the like, a gaze camera may be used as the imaging unit to recognize the faces of the elderly and other subjects, and emotion determination may be performed based on the images obtained when the caregiver sees the elderly and other subjects. Also, the result of emotion determination may be notified by voice. Specifically, for example, the result of emotion determination may be notified by voice from the earphones worn by the caregiver or the speaker of the tablet terminal. When a gaze camera is used, it is assumed that the gaze target becomes the measurement target and it is difficult to confirm the result of emotion determination on the screen of the gaze camera. However, even in such a case, the result of emotion determination can be recognized by voice. When a gaze camera is used as the imaging device, although the gaze camera is small and convenient for carrying, there are cases where the processing ability is not sufficient. Therefore, the acquired video may be wirelessly transmitted to another terminal, and emotion determination may be performed on the receiving terminal side.

[0110] Note that the voice notification of the result of emotion determination may also be performed in other embodiments, particularly in a machine control system using the emotion determination device according to Embodiment 4. By notifying the result of emotion determination and warnings by voice, even when the operator of the machine is concentrating on the work and not gazing at the control screen, the result of emotion determination and warnings can be recognized by voice.

[0111] Alternatively, a camera provided in a smartwatch may be used as the imaging device. The image captured by the smartwatch may be subjected to emotion determination on the smartwatch body, or the captured image may be transmitted to another terminal and the emotion determination may be performed on the receiving terminal side.

[0112] [Embodiment 5] (Simultaneous measurement of multiple people) In recent years, the opportunities to hold Web conferences via the Internet or an intranet have been increasing. As an example of a Web conference, for example, when a company gives a presentation to a plurality of customers via the Internet, or when a teacher gives a lesson to a plurality of students via the Internet. At this time, the organizer of the Web conference such as a company or a teacher preferably grasps whether a plurality of listeners such as a plurality of customers or a plurality of students are concentrating on listening to the speech, or whether the presentation, lesson, etc. can be carried out in a good state. Since the listeners of the lecture are participants in the lecture, hereinafter, the "listener" will be referred to as the "participant".

[0113] However, also in the case of a seminar or the like using the Web, similar to the case of the Web conference of Embodiment 2, since the lecturer does not directly visually recognize the faces of a plurality of participants but sees the faces of a plurality of participants via the screen, there is a problem that it is difficult for the seminar organizer to grasp the state in which each participant is listening to the lecture. Here, the object presented by the organizer side of the Web conference to the participants may be referred to as "presentation", "lesson", "seminar", etc., but in this specification, these will be referred to as "lecture".

[0114] The emotion determination device according to Example 5 of the present disclosure aims to grasp the state in which a plurality of participants listened to a lecture even when there are a plurality of participants. The emotion determination device according to Example 1 of the present disclosure performs emotion analysis on a single subject, while the emotion determination device according to Example 5 of the present disclosure is characterized by simultaneously performing emotion analysis on a plurality of subjects. Furthermore, the emotion determination device according to Example 5 of the present disclosure calculates an optimal listening time representing the time when the participant is listening to the lecture in an ideal state, and determines the state in which the participant was listening to the lecture, as will be described later.

[0115] As an example, the emotion determination device according to Example 5 of the present disclosure will be described by taking the case where a lecture organizer gives a lecture to a plurality of participants. That is, the subject is a listener (participant) who takes the lecture. FIG. 15 shows a block diagram of a lecture organizing side PC 600, which is the emotion determination device according to Example 5 of the present disclosure. The lecture organizing side PC 600 transmits and receives data via the Internet 100 to and from terminals A (500a), B (500b),..., N (500n) used by each of a plurality of participants A, B,..., N when listening to the lecture.

[0116] Terminal A (500a) includes a camera 501, a microphone 502, a transmission / reception unit 503, and a display unit 504. The camera 501 captures an image of the face of participant A. The microphone 502 collects the voice of participant A. The camera 501 and the microphone 502 may be built into the terminal A (500a) or may be external. The transmission / reception unit 503 transmits and receives data via the Internet 100 to and from the lecture organizing side PC 600. The display unit 504 displays information related to the lecture transmitted from the lecture organizing side PC 600. The display unit 504 may display an image of the face of the lecturer who organizes the lecture, an image of the face of participant A himself / herself, etc. The configurations of terminals B (500b) and N (500n) are the same as that of terminal A (500a).

[0117] The lecture organizer's PC 600 includes a content distribution unit 601, a transmission / reception unit 602, an all-participants face recognition unit 603, each participant image extraction unit 604, participant emotion analysis units (605a, 605b, ···, 605n), each participant individual log storage unit 611, and a display / notification unit 612.

[0118] The content distribution unit 601 distributes content such as images and videos used by the lecturer who holds the lecture over a predetermined period of time. The content may be distributed in real time by the lecturer, or may be distributed by playing a pre-prepared video or the like.

[0119] The transmission / reception unit 602 receives data including images of the faces of participants A, B, ···, N from terminals A (500a), B (500b), ···, N (500n) via the Internet 100. For example, when there are 4 participants in the lecture, as shown in FIG. 16, images (41a to 41d) of the respective participants are displayed on the screen of the display / notification unit 612. Although FIG. 16 shows the case where there are 4 participants, such an example is not limited thereto.

[0120] The all-participants face recognition unit 603 recognizes the images of the faces of all of the plurality of participants. That is, the all-participants face recognition unit 603 is an example of a photographing unit and can photograph the faces of a plurality of subjects. Usually, the application software for web conferencing used in the lecture does not have a function of recognizing where on the screen the face of the participant is projected, and it is also unknown how many participants there are. For example, even if terminals (500a, 500b, ···, 500n) are connected to the lecture organizer's PC 600, it is impossible to recognize that the participants are attending the lecture when the images of the faces of the participants are not sent. Therefore, the all-participants face recognition unit 603 captures the image displayed by the application software, first scans the approximate position of the face to grasp how many participants there are, and also acquires the respective face position coordinates of the plurality of participants. The method of face recognition will be described later.

[0121] Each participant image extraction unit 604 extracts an image for use in the analysis of the pulse wave from the acquired participant images. This is because when there are multiple participants, the amount of data for image processing increases and it becomes difficult to process simultaneously in parallel. Therefore, it is to reduce the amount of data for image processing. Fig. 17 shows an example of extracting a partial image from the image of the participant's face. Each participant image extraction unit 604 extracts, for example, a part 41a' of the image 41a of participant A as shown in the upper left of Fig. 17. Similarly, each participant image extraction unit 604 extracts parts (41b', 41c', 41d') of the images (41b, 41c, 41d) of other participants B to D.

[0122] The participant A emotion analysis unit 605a includes an individual face recognition unit 606, a pulse wave extraction image processing unit 607, an RRI Lyapunov emotion determination unit 608, a heart rate emotion determination unit 609, and an optimal lecture time determination unit 610.

[0123] The individual face recognition unit 606 of the participant A emotion analysis unit 605a performs face recognition of participant A. Also, the individual face recognition unit 606 cuts out an area for extracting a pulse wave from the image 41a' of the face of participant A. For example, a part 412a of the image 41a' of participant A shown in the upper left of FIG. 17 is cut out. The cut-out image 412a may be displayed, for example, in the upper left of the image 41a' of participant A (412a'). Similarly, the individual face recognition units 606 of the emotion analysis units (605b, 605c, 605d) of the other participants (B, C, D) perform face recognition of the other participants (B, C, D) respectively. Also, the individual face recognition units 606 of the emotion analysis units (605b, 605c, 605d) of the other participants (B, C, D) cut out an area for extracting a pulse wave from the images (41b', 41c', 41d') of the faces of the other participants (B, C, D). For example, parts (412b, 412c, 412d) of the images (41b', 41c', 41d') of the other participants (B, C, D) are cut out. The cut-out images (412b, 412c, 412d) may be displayed, for example, in the upper left of the images (41b', 41c', 41d') of the other participants (B, C, D) (412b', 412c', 412d'). The method of cutting out the image will be described later. By cutting out a predetermined part of the entire image of the participant (for example, the image of the face part) and performing image processing, the amount of data processing during image processing can be reduced.

[0124] The pulse wave extraction image processing unit 607 is an example of a detection unit, and uses the image 412a in a predetermined range of the face image 41a' of participant A cut out by the individual face recognition unit 606 to detect heartbeat information including the heartbeat of participant A who is the subject.

[0125] The pulse wave extraction image processing unit 607 also has the function as the counting unit described in the first embodiment, and counts the number of heartbeat fluctuations that transition from a state below the average heartbeat to a state above the average heartbeat with an increase in the heartbeat above a predetermined value during a predetermined period.

[0126] The RRI Lyapunov emotion determination unit 608 is an example of an emotion determination unit, and determines whether the emotion of Participant A, who is the subject, is a negative emotion or a positive emotion based on the heartbeat information detected by the pulse wave extraction image processing unit 607. RRI is an abbreviation for R-R Interval and indicates the interval between heartbeats and pulse waves. Also, the RRI Lyapunov emotion determination unit 608 corresponds to the five blocks of the pulse wave extraction unit 22, the interval detection unit 23, the pulse wave memory 24, the chaos analysis unit 25, and the emotion determination unit 31 in FIG. 2 of the first embodiment. Here, generally speaking, it is said that the performance of participants listening to a lecture is maximized when the participants have a "moderate sense of tension" psychologically and are listening with a "normal mind". That is, it is not necessarily the case that the state where the participant does not feel stress is the best, and a state with a certain degree of "moderate sense of tension" is considered preferable. The state where the participant has a "moderate sense of tension" is a state where the participant has a "slightly negative emotion", that is, a state where the participant has a "slightly unpleasant emotion". The RRI Lyapunov emotion determination unit 608 can determine whether the participant has a "slightly unpleasant emotion", that is, whether the participant has a "moderate sense of tension" by determining whether the participant has a "slightly negative emotion".

[0127] FIG. 18 shows the relationship between the maximum Lyapunov exponent and the subjective exercise intensity RPE (Rating of Perceived Exertion). FIG. 18 is, as an example, a plot obtained by measuring the maximum Lyapunov exponent and the heart rate difference from the heartbeat information of five subjects at various levels of fatigue and determining the RPE at that time. RPE is an index that numerically represents the subjective intensity and fatigue of training or exercise. The Borg scale, which represents the intensity with numbers from 6 to 20, is generally used to quantify RPE. From FIG. 18, there is a correlation between the maximum Lyapunov exponent and RPE, and the value of RPE can be obtained from the maximum Lyapunov exponent calculated by the RRI Lyapunov emotion determination unit 608. For example, when the maximum Lyapunov exponent is 0, the RPE is 12, and when the maximum Lyapunov exponent is -0.7, the RPE is 15.

[0128] The Borg scale is shown in Fig. 19. In the Borg scale, when the RPE is between 12 and 15, the subject is considered to feel "somewhat strenuous", and this state is considered to be the state where the lecture participant, who is the subject, has a "moderate sense of tension". When the RPE is 5 or less, the subject is considered to feel "extremely pleasant", and when the RPE is 6 - 7, the subject is considered to feel "very pleasant". In these cases, the lecture participants are considered to feel "sleepy". When the RPE is 8 - 9, the subject is considered to feel "quite pleasant", and when the RPE is 10 - 11, the subject is considered to feel "pleasant". In these cases, the lecture participants are considered to feel "comfortable". When the RPE is 15 - 16, the subject is considered to feel "strenuous", when the RPE is 17 - 18, the subject is considered to feel "quite strenuous", and when the RPE is 19 - 20, the subject is considered to feel "extremely strenuous". In these cases, the lecture participants are considered to feel "uncomfortable".

[0129] The pulse rate emotion determination unit 609 determines the emotion of the participant based on whether the extracted pulse wave is within a predetermined range from the average resting pulse rate. Fig. 20 shows an example of the temporal change of the pulse rate when the pulse wave is within the predetermined range from the average resting pulse rate. The state where the above-mentioned participant has a "normal mind" is considered to be the state where the pulse wave is stable within the predetermined range from the average resting pulse rate. For example, when the participant's pulse is within a range of, for example, plus or minus 5 [bpm] centered on the average resting pulse rate, it may be determined that the participant has a normal mind. The average resting pulse rate may be 65 [bpm], which is the average value of the resting pulse rate of Japanese people, or the average resting pulse rate of each individual participant may be used.

[0130] Here, it is preferable to continuously analyze the emotions and feelings during the lecture period. When updating the Lyapunov index for emotion analysis, for example, every 30 seconds to 40 seconds, the emotion analysis may also be performed every 30 seconds to 40 seconds in accordance with this timing.

[0131] The optimal lecture time determination unit 610 is an example of an emotion determination unit. Based on the number of fluctuations in the heart rate counted by the pulse wave extraction image processing unit 607, it determines the mental state of the participant who is the subject, and can use the determination result by the RRI Lyapunov emotion determination unit 608 which is an emotion determination unit for the determination of the mental state. The optimal lecture time determination unit 610, which is an emotion determination unit, determines whether it is an ideal mental state in the lecture based on the number of fluctuations in the heart rate and the determination result of the RRI Lyapunov emotion determination unit 608 which is an emotion determination unit. That is, when both the first condition that the positive and negative emotions are in a state of "slightly unhappy", indicating that the participant has a "moderate sense of tension", and the second condition that the pulse is "within a predetermined range from the average pulse", indicating that the participant's emotion is in a "normal state" are satisfied, it is determined that the participant is in an ideal mental state. In this way, the time during which the participant is listening to the lecture in an ideal mental state is accumulated, and this accumulated time is set as the optimal lecture time. The longer the optimal lecture time is, the longer the time the participant is listening to the lecture in an ideal state can be known. The optimal lecture time can be calculated individually for each participant. An average value obtained by dividing the total value of the optimal lecture times of a plurality of participants by the number of participants may be calculated. Since the optimal lecture time is considered to be longer if the content of the lecture by the lecturer attracts the interest of the participants, the optimal lecture time may be used as an index representing the ability of the lecturer.

[0132] The participant B emotion analysis unit 605b, the participant N emotion analysis unit 605n, etc. have the same configuration as the participant A emotion analysis unit 605a.

[0133] The individual log storage unit 611 for each participant stores the emotions and emotions of each of the plurality of participants in time series from the start time to the end time or the leaving time of the lecture. By referring to the log, it is possible to know during which time period of the lecture the participant was listening in an ideal state. Alternatively, by referring to the log, it is possible to know at which point during the lecture the participant was sleepy, so it is possible to know after the lecture which part of the lecture had an explanation that was likely to make the participant sleepy.

[0134] The display / informing unit 612 is an example of an output unit, and outputs the determination results of the RRI Lyapunov emotion determination unit 608 and the pulse rate emotion determination unit 609. For example, as shown in FIG. 17, the determination results 411a to 411d of positive and negative emotions by the RRI Lyapunov emotion determination unit 608 are displayed in characters below the respective face images of a plurality of participants, and the determination results 413a to 413d of emotions by the pulse rate emotion determination unit 609 are displayed in a rectangular frame around the face images of the plurality of participants.

[0135] In the example shown in FIG. 17, an example is shown in which from the determination results 411a to 411d of positive and negative emotions, the emotion of participant A is "unhappy", the emotion of participant B is "slightly unhappy", the emotion of participant C is "happy", and the emotion of participant D is "sleepy". Also, when performing these character displays, the color of the characters may be made different according to the positive and negative emotions. For example, since it is normal to be slightly nervous during a lecture, the expression of positive and negative emotions may be such that "slightly negative emotion" is expressed in "green characters" of a safety color, and "negative emotion" is expressed in "red characters" of a warning color. Also, since extreme positive emotion induces drowsiness, it may be expressed in "black characters" of "sleepy".

[0136] Also, in the example shown in FIG. 17, it is possible to display whether the participant's pulse is within a predetermined range based on the average resting pulse rate, exceeds the upper limit of the predetermined range, or is less than the lower limit of the predetermined range, from the emotion determination results 413a to 413d. For example, when the pulse of Participant A is within the resting pulse rate range (average pulse ± 5 [bpm]), the frame of the emotion determination result 413a may be displayed in, for example, "green". In this case, the organizer of the lecture can recognize that Participant A is in a "normal state of mind" based on the fact that the emotion determination result 413a is green. Also, when the pulse of Participant B is less than the lower limit of the resting pulse rate range, the frame of the emotion determination result 413b may be displayed in, for example, "black". Also, when the pulse of Participant C exceeds the upper limit of the resting pulse rate range, the frame of the emotion determination result 413c may be displayed in, for example, "red". Also, when the pulse of Participant D is within the resting pulse rate range, the frame of the emotion determination result 413d may be displayed in, for example, "green". The colors of the frames of these emotion determination results 413a to 413d may be single colors or intermediate colors according to the pulse rate.

[0137] The above-described method of displaying the positive / negative emotion and emotion determination results is an example, and the positive / negative emotion and emotion determination results may be displayed by other display methods. For example, the positive / negative emotion determination results may be displayed using face marks corresponding to the emotions. Also, the emotion determination results may be displayed using characters, numbers, etc. according to the pulse rate. The organizer of the lecture can easily determine in what state each participant is listening to the lecture by referring to these positive / negative emotion determination results and emotion determination results.

[0138] In addition, by individually calculating the optimal lecture time for each participant and presenting it to the participants, the participants can easily grasp their own lecture status. Also, by calculating the optimal lecture time for each individual participant, summing up the optimal lecture times of all the participants, and dividing by the number of participants who attended the lecture to calculate the average value, the optimal lecture time for the entire lecture can be calculated. Further, since the lecture is held over a predetermined time such as one hour or two hours, it may be calculated as a standard value per unit time by dividing the optimal lecture time by the lecture holding time. For example, for a 60-minute lecture, if the cumulative value of the optimal lecture time is 12 minutes, the optimal lecture time for the entire lecture can be calculated as 20%.

[0139] Fig. 21 shows a flowchart for explaining the operation procedure of the lecture hosting side PC 600, which is an emotion determination device according to Embodiment 5. First, in step S201, a program for executing emotion sensing is started to acquire a Web conference screen. Here, the case where a plurality of participants participate in a Web conference will be described. Images of a plurality of participants are displayed on the screen of the PC of the Web conference host. For example, using Web conference software, only the faces of all the participants can be projected onto a monitor different from the monitor displaying the materials. When explaining by sharing the materials screen in a Web conference, the materials may be displayed across the entire screen, making it impossible to display the faces of the participants. When the expressions of the participants cannot be seen, even if the Web conference host is explaining, the reaction of the participants cannot be seen, so it is impossible to determine whether the participants understand the content being spoken. Therefore, the software for executing the Web conference may be equipped with a function that allows adding a screen separate from the screen of the PC displaying the materials to make two screens. By using this function, a plurality of participants' face images can be displayed on one screen, and the shared screen can be displayed on the other screen. By arranging and displaying a plurality of participants on one screen, the Web conference can be executed while confirming the expressions of the plurality of participants.

[0140] Next, in step S202, the entire face image is recognized, the number of participants' faces within the screen is counted, and the coordinate positions of each face are calculated. As described above, a screen showing all the participants is captured, and first, the entire face image is recognized. Here, since each participant may change their position within the screen, the face images of all the participants are recognized over a predetermined period, for example, about one minute. Here, recognizing the face images of all the participants means counting the number of recognized face images and calculating the coordinate positions where the faces are recognized within the screen. As the coordinate positions of the face image, the coordinate positions of the eyes and nose can be calculated, and the coordinate positions of a predetermined position within the face image, for example, the position between the eyebrows, can be used as the standard position for each participant. Also, when the camera captures the participants' faces, since the actual position of the face is slightly moving, by capturing images over a predetermined period, for example, one minute, a range where the participants exist within a predetermined number of pixels (for example, 300 pixels) in the X and Y directions on the XY coordinates is defined, and even if the coordinates of the face image vary within that range, it can be regarded as the same person.

[0141] Next, in step S203, starting from each inter - eyebrow coordinate, each individual face - surrounding image is cut out from the entire image. For example, as shown in FIG. 17, starting from the position of the inter - eyebrow coordinates, a range considering the movement of the participants' face positions, for example, a range of plus or minus 300 pixels in the X and Y directions centered on the inter - eyebrow coordinate position, is cut out, and this cut - out image is used as the image for determining emotions and feelings.

[0142] Next, in step S204, the face - surrounding image of participant A is captured and image recognition is performed.

[0143] Next, in step S205, it is determined whether a face image can be recognized in real time. If the face image of participant A can be recognized, in step S206, emotion and affect determination is executed, in step S207, the optimal listening time is accumulated, and then it returns to step S205 to determine whether a face image can be recognized in real time. Here, calculating the maximum Lyapunov exponent for emotion determination requires a predetermined period, for example, about 60 seconds. Since affect determination is also executed simultaneously, the optimal listening time is calculated, for example, every 60 seconds.

[0144] In step S205, if the face image of participant A cannot be recognized, in step S208, it is determined whether the face image cannot be recognized for a certain period of time. In step S208, if the face image can be recognized within a predetermined time, it returns to step S205 to determine whether the face image can be recognized in real time.

[0145] On the other hand, in step S208, if the face image cannot be recognized within a predetermined time (for example, 3 minutes), it can be determined that participant A has left. Therefore, in step S209, it is determined that participant A has left, and the log of the optimal listening time, emotion determination result, and affect determination result is saved. Here, the reason why the face image of participant A cannot be recognized is not only when participant A leaves in front of terminal A while maintaining the connection state between the terminal A connected to the lecturer's side PC for the Web conference, but also when the connection between participant A's terminal A and the lecturer's side PC is interrupted, and when the lecture ends.

[0146] Next, in step S210, the calculation of the optimal listening time for participant A is completed. The steps from S204 to S210 are the steps for calculating the optimal listening time for participant A, but for other participants, parallel processing is also performed simultaneously to calculate the optimal listening time. For example, for participant N, the optimal listening time for participant N is calculated by executing the steps from S211 to S217. The same applies to other participants such as participant B.

[0147] In the above description, an example was given in which each of a plurality of participants was determined whether they were attending or absent from the lecture, and the optimal listening time was calculated at the stage of absence. However, the example is not limited to this. When the face images of all the participants cannot be recognized at the end of the Web conference, it may be considered that the cumulative time of the optimal listening time is calculated at that time.

[0148] (Face image recognition) Next, a method for recognizing the face image of the subject will be described. FIG. 22(A) shows an output example of a feature coordinate fitting method for extracting feature coordinates from a face image, and FIG. 22(B) shows an output example of a method for detecting the position between the eyebrows from a captured face image.

[0149] FIG. 22(A) is an example in which the coordinate positions of the eyebrows 701, eyes 702, nasolabial folds 703, nose 704, mouth 705, and face contour 706 are extracted from the face image of the subject, and the image cut out for emotion determination and mood determination is set as a predetermined area 710 of the cheek. This method is effective when the subject does not wear a mask or the like, but when wearing a mask or the like, there is a problem that a part of the mouth, nose, and face contour is hidden and the coordinate position cannot be accurately measured.

[0150] Therefore, as a method of cutting out a specific area of the face image of the subject regardless of whether the mask is worn or not, as shown in FIG. 22(B), among the face images 720 of the subject HK, image recognition by deep learning is performed on the image 721 of the eyes and the face contour, and a predetermined area 722 extending from the glabella to the forehead is used as the image to be cut out for emotion determination and affect determination. The facial part for performing image recognition is not limited to the eyes and the contour, and other parts may be used. However, in order to be able to perform image recognition even when the mask 723 is worn, a part not covered by the mask 723 is preferable. In addition, the area cut out for emotion determination and affect determination is not limited to the area around the glabella, and may be the forehead or the like. Therefore, it is preferable that the lecture organizer PC 600, which is an emotion determination device, has measurement site specifying means for specifying each face from a screen on which a plurality of people are displayed and specifying the measurement site for each specified face. Each participant image cutout unit 604 is an example of the measurement site specifying means. The pulse wave extraction image processing unit 607 (see FIG. 15), which is a detection unit, can acquire heartbeat information based on the change in the image of the measurement site of each face.

[0151] Here, when extracting a specific area from images such as the eyes and the face contour included in the face image by deep learning, it is conceivable that the amount of computational processing increases and a burden is imposed on the processor that controls the lecture organizer PC 600, which is an emotion determination device. Therefore, in order to reduce the processing amount of the processor of the lecture organizer PC 600, the image processing by deep learning may be executed at the back end to speed it up.

[0152] (Image capture via HDMI) When software linked to a web conference that performs emotion determination using an image of a subject as described above is used, it is often used for business purposes within the intranet, and there may be cases where emotion sensing cannot be executed on the web conference PC due to security reasons. For example, in an emotion determination program, video information is directly captured or the web conference screen is acquired as it is. Therefore, if it is executed on a PC within the intranet, it may be recognized as being infected with a virus or being subject to illegal access. Here, there is HDMI output as an image output method that is not subject to intranet restrictions. Therefore, since HDMI video output can be output from the intranet so that HDMI output is performed from the PC to the projector, it is possible to capture the data of the subject's face image into the intranet PC using HDMI output and execute emotion determination.

[0153] Fig. 23 shows a configuration example for executing image capture using HDMI output. Assume that the web conference PC 620 is connected to the intranet. The HDMI output of the web conference PC 620 is input to the HDMI distribution unit 801. The HDMI distribution unit 801 distributes and outputs the HDMI signal input from the web conference PC 620 to "HDMI1" and "HDMI2". HDMI1 is input to the HDMI input unit 802, input to the emotion sensing PC 600, and emotion sensing is executed. The emotion sensing PC 600 is not connected to the intranet (non-intranet). The emotion sensing PC 600 can execute emotion sensing without being connected to the intranet by using the face image data of the subject included in the HDMI output. On the other hand, HDMI2 is input to the projector and the image of the web conference PC 620 can be displayed.

[0154] In the above example, an example was shown in which only the Web conference video was captured from the HDMI output of the Web conference PC 620, which is an intra-network PC, to the emotion sensing PC 600, which is a non-intra-network PC, and emotion analysis was performed on the emotion sensing PC 600. However, such examples are not limited to this. That is, emotion sensing may be performed using the HDMI output on the Web conference PC 620 that executes the Web conference. Also, although an example using the HDMI output was described as an example of taking out a video output from a PC connected to the intranet to a non-intranet PC not connected to the intranet, such examples are not limited to this. That is, other output methods other than the HDMI output may be used as the output for taking out the video output from the PC connected to the intranet to the non-intranet PC.

[0155] [Example 6] (Determination of Causes of Communication Impairment in the Elderly) There may be problems in communication between the elderly and caregivers. This problem is considered to be due to the caregiver's inability to determine the cause of the communication impairment with the elderly. Fig. 24 shows an example of the cause and countermeasure method for the communication impairment between the elderly and the caregiver. As the causes of the communication impairment in the elderly, the causes (1) to (3) can be considered in order from the less serious ones.

[0156] (1) The first cause is when the elderly dislike the caregiver due to their personality, that is, when the compatibility between the elderly and the caregiver is poor. As a countermeasure against this first cause, it is conceivable to change the caregiver who takes care of the elderly to another caregiver.

[0157] (2) The second cause is when it is due to simple vision and hearing loss. It is conceivable that the color cannot be distinguished due to vision loss, or the sound cannot be heard due to hearing loss. As a countermeasure against this second cause, it is conceivable to supplement vision by using glasses, supplement hearing by using a hearing aid, or take individual countermeasures for the way of communication by communicating by finger pointing.

[0158] (3) The third cause is when it is due to mental disorders such as dementia. It is conceivable that the elderly can see and hear things, but cannot recognize their meaning. As a countermeasure against this third cause, it is conceivable to take measures by a specialist doctor.

[0159] For example, regarding the second cause, if only a normal vision and hearing evaluation is performed, the first and third causes may be included, and the cause that inhibits communication cannot be accurately identified. Therefore, in order to accurately distinguish the first to third causes, three categories of video stimuli and acoustic stimuli as shown in Fig. 25 are periodically given to the subject, and the cause can be estimated based on whether the emotion or positive / negative feelings change synchronously with the period.

[0160] In order to determine whether the compatibility, which is the first cause, is the cause of inhibiting communication, it is conceivable to show the video of a specific caregiver etc. to the elderly as a video stimulus, and to let the elderly hear the voice of a specific caregiver etc. as an acoustic stimulus. The "specific caregiver" is, for example, a caregiver who is in charge of caring for an elderly person with communication problems.

[0161] In order to determine whether the vision and hearing, which is the second cause, is the cause of inhibiting communication, it is conceivable to show a video of only colors etc. to the elderly as a video stimulus, and to let the elderly hear a beat sound of a specific frequency etc. as an acoustic stimulus.

[0162] In order to determine whether a mental disorder, which is the third cause, is the cause of communication impairment, it is conceivable to show the elderly a deceptive picture video or a horror video without color stimulation as a visual stimulus, and it is conceivable to let the elderly hear meaningful words or the like as an acoustic stimulus.

[0163] Fig. 26 shows a block diagram of a subject state determination device 1000 which is a communication impairment cause determination device. The communication impairment cause determination device and the subject state determination device are examples of an emotion determination device. The subject state determination device 1000 which is a communication impairment cause determination device includes a display unit 1001, a periodic stimulus video generation unit 1002, a speaker or headphones 1003, a periodic stimulus sound generation unit 1004, a stimulus type switching unit 1005, a camera 1006, a face recognition unit 1007, a pulse wave extraction image processing unit 1008, an RRI Lyapunov emotion determination unit 1009, a pulse rate emotion determination unit 1010, a synchronization analysis unit 1011, a cause determination unit 1012, and a determination result notification unit 1013.

[0164] The periodic stimulus video generation unit 1002 and the periodic stimulus sound generation unit 1004 are examples of a stimulus generation unit, and repeatedly generate the same type of stimulus for a plurality of predetermined periods including a pause period as a stimulus to the visual or auditory sense of the subject. There are three types of stimuli, namely, a first stimulus, a second stimulus, and a third stimulus, which will be described later. For example, as the same type of stimulus, the first stimulus is repeatedly generated. However, each of the repeatedly generated stimuli may be different as long as it is classified as the first stimulus.

[0165] The display unit 1001 displays a video that serves as a visual stimulus for the elderly who are the subjects, generated by the periodic stimulus video generation unit 1002. As the display unit 1001, a liquid crystal display device, an organic EL display device, a projector, or the like can be used.

[0166] The speaker or headphones 1003 outputs a voice that serves as an acoustic stimulus for the elderly, generated by the periodic stimulus sound generation unit 1004.

[0167] The stimulus type switching unit 1005 switches whether the stimulus given to the elderly subject is a visual stimulus or an acoustic stimulus.

[0168] The camera 1006 captures an image of the face of the elderly subject.

[0169] The face recognition unit 1007 recognizes the face image from the image captured by the camera 1006.

[0170] The pulse wave extraction image processing unit 1008 is an example of a detection unit, and detects heartbeat information including the heart rate of the elderly subject.

[0171] The RRI Lyapunov emotion determination unit 1009 is an example of a counting unit, and counts the number of times of heartbeat fluctuations that transition from a state below the average heart rate to a state above the average heart rate with an increase in heart rate of a predetermined value or more every predetermined period.

[0172] The pulse rate emotion determination unit 1010 is an example of an emotion determination unit, and determines the presence or absence of the emotion of the subject corresponding to the stimulus every predetermined period based on the number of times of heartbeat fluctuations.

[0173] The synchrony analysis unit 1011 analyzes whether there is synchrony between a plurality of periods and the timing of the presence or absence of the emotion determined by the emotion determination unit.

[0174] The cause determination unit 1012 is an example of an inhibition factor determination unit, and makes a determination regarding the inhibition factor of communication with others for the subject based on the analysis result of the synchrony analysis unit 1011.

[0175] The determination result notification unit 1013 outputs the determination result of the inhibition factor of communication determined by the cause determination unit 1012.

[0176] The stimuli generated by the periodic stimulus video generator 1002 and the periodic stimulus sound generator 1004, which are stimulus generation units, are first stimuli whose generation can be sensually recognized by vision or hearing. The cause determination unit 1012, which is an inhibition factor determination unit, determines the presence or absence of abnormalities in vision or hearing. The first stimulus may be a sensory stimulus that stimulates vision or hearing and whose generation is recognized. Here, the "sensory stimulus" includes stimuli whose presence or absence of generation can be sensually recognized by vision or hearing. Note that a moving image may also be used as an example of a stimulus including an image or sound.

[0177] An example of the case where the first stimulus is a stimulus whose generation can be recognized by vision will be described. FIG. 27 shows an example of the temporal change of the emotion of an elderly person who is a subject when a color stimulus, which is a visual autonomic nerve response color such as red, is periodically shown as the first stimulus whose generation can be recognized by vision. First, the display unit 1001 is arranged so that an elderly person who is a subject can visually recognize an image, and an image 901 such as red is displayed on the display unit 1001 over a predetermined period (for example, 30 seconds) from time t1 to time t2. Next, a white image 902 that does not include a color stimulus is displayed on the display unit 1001 over a predetermined period (for example, 30 seconds) from time t2 to time t3. After time t3, an image including a color stimulus and an image without a color stimulus (no stimulus) are alternately and repeatedly displayed on the display unit 1001. This switching between the color stimulus and the no stimulus is executed by the periodic stimulus video generator 1002.

[0178] A face image of an elderly person who is a subject to whom a color stimulus and a no stimulus are applied is captured using the camera 1006.

[0179] The pulse wave extraction image processing unit 1008 is an example of a detection unit, and detects heartbeat information including the heartbeat of an elderly person from the image of the elderly person captured by the camera 1006.

[0180] The RRI Lyapunov emotion determination unit 1009 is an example of an emotion determination unit, and determines whether the emotion of an elderly person who is a subject is a negative emotion or a positive emotion based on the heartbeat information.

[0181] The pulse wave extraction image processing unit 1008 also has a function as a counting unit, and counts the number of times of heartbeat fluctuations that have transitioned from a state below the average heart rate to a state above the average heart rate with an increase in the heart rate of a predetermined value or more during a predetermined period.

[0182] The pulse rate emotion determination unit 1010, which is an emotion determination unit, determines the mental state of the subject, for example, the presence or absence of emotion, based on the number of times of heartbeat fluctuations.

[0183] The synchronization analysis unit 1011 analyzes whether there is synchronization between a plurality of periods of color stimulation and no stimulation and the timing of the presence or absence of emotion determined by the emotion determination unit.

[0184] The cause determination unit 1012, which is an inhibitory factor determination unit, determines the presence or absence of visual abnormality based on whether the timing of color stimulation and the timing when emotion occurs are synchronized. For example, in the example shown in FIG. 27, from time t1 to t2, emotion of the elderly person is detected when an image 901 including color stimulation is displayed, and from time t2 to t3, emotion of the elderly person is not detected when a white image 902 not including color stimulation is displayed. That is, emotion is detected in accordance with the timing of displaying the image 901 including color stimulation. In this case, it can be determined that the elderly person as the subject recognizes color, and it can be determined that there is no abnormality in vision.

[0185] Next, an example will be described in which the first stimulus is a stimulus whose generation can be recognized by hearing. Fig. 28 shows an example of the temporal change of the emotion of an elderly person who is a subject when a beat sound of a predetermined frequency (for example, 500 Hz) is periodically heard as the first stimulus whose generation can be recognized by hearing. First, a 500 Hz beat sound is output from the speaker or headphones 1003 over a predetermined period (for example, 30 sec) from time t1 to t2. Next, no sound (silence) is output from the speaker or headphones 1003 over a predetermined period (for example, 30 sec) from time t2 to t3. After time t3, the output of the 500 Hz beat sound and silence are alternately repeated. The switching between this 500 Hz beat sound and silence is executed by the periodic stimulus sound generation unit 1004.

[0186] An image of the face of an elderly person who is a subject to whom a 500 Hz beat sound and silence are added is captured using the camera 1006.

[0187] The pulse wave extraction image processing unit 1008 is an example of a detection unit, and detects heartbeat information including the heartbeat rate of an elderly person who is a subject from the image of the elderly person captured by the camera 1006.

[0188] The RRI Lyapunov emotion determination unit 1009, which is an example of an emotion determination unit, determines whether the emotion of an elderly person who is a subject is a negative emotion or a positive emotion based on the heartbeat information.

[0189] The pulse wave extraction image processing unit 1008 also has a function as a counting unit, and counts the number of heartbeat fluctuations that transition from a state below the average heartbeat rate to a state above the average heartbeat rate with an increase in the heartbeat rate of a predetermined value or more within a predetermined period.

[0190] The pulse rate emotion determination unit 1010, which is an emotion determination unit, determines the mental state of the subject, for example, the presence or absence of emotion, based on the number of heartbeat fluctuations.

[0191] The synchronization analysis unit 1011 analyzes whether there is synchronization between a plurality of periods of acoustic stimulation and no stimulation and the timing of the presence or absence of emotion determined by the emotion determination unit.

[0192] The cause determination unit 1012, which is an inhibition factor determination unit, determines the presence or absence of auditory abnormalities based on whether the timing of the output of the 500 Hz beat sound and the timing when emotion occurs are synchronized. For example, in the example shown in FIG. 28, from time t1 to t2, when the 500 Hz beat sound was output, the emotion of the elderly person was detected, and from time t2 to t3, when no sound was generated, the emotion of the elderly person was not detected. That is, emotion is detected in accordance with the timing of outputting the 500 Hz beat sound. In this case, it can be determined that the elderly person, who is the subject, recognizes the sound, and it can be determined that there is no abnormality in hearing.

[0193] In the above-described subject state determination apparatus 1000, the stimulus may be a first stimulus whose generation can be recognized only by visual or auditory senses. That is, the first stimulus does not require recognition of the information included in the stimulus, and is a stimulus whose generation can be recognized (sensibly) only by visual sense or only by auditory sense. The first stimulus may be a stimulus that also includes some information in the stimulus, and emotion is generated by sensibly recognizing the presence or absence of the stimulus, and further, emotion may be generated or may occur by recognizing the information therein.

[0194] The stimulus is a second stimulus including predetermined information that can be understood by the content of information given by vision or hearing, and the inhibition factor determination unit may determine the presence or absence of mental abnormalities of the subject. The second stimulus may be a cognitive stimulus involving the cognition of predetermined information. For example, as the second stimulus, even in the case of a video that has no stimulus in itself, such as a deceptive picture, a video including content that is likely to react when an abnormal form that is normally impossible to exist hidden in the deceptive picture is recognized and the meaning of the deceptive picture can be understood may be used. Alternatively, as the second stimulus, for example, even in the case where there is no stimulus in the sound itself, an audio including content that is likely to react when the meaning of words can be understood may be used.

[0195] First, a case of using a video containing predetermined information that can be understood in terms of content as the second stimulus will be described. FIG. 29 shows an example of the temporal change of the emotion of an elderly subject when a deceptive picture that makes one surprised or laugh because its meaning can be understood is periodically shown as the second stimulus containing predetermined information that can be understood visually. First, an image 903 in which the shape of an elephant's ear is the shape of a human profile is displayed on the display unit 1001 for a predetermined period (e.g., 30 seconds) from time t1 to t2. Next, a white image 902 without a deceptive picture is displayed on the display unit 1001 for a predetermined period (e.g., 30 seconds) from time t2 to t3. Next, an image 904 in which the hand holding the can that should be the right hand is the left hand is displayed on the display unit 1001 for a predetermined period (e.g., 30 seconds) from time t3 to t4. Next, a white image 902 without a deceptive picture is displayed on the display unit 1001 for a predetermined period (e.g., 30 seconds) from t4 to t5. Next, an image 905 in which the shape of the boundary of the shade of the color represented on the ground is the shape of a human profile is displayed on the display unit 1001 for a predetermined period (e.g., 30 seconds) from time t5. The switching between the image including the deceptive picture and the image without the deceptive picture is executed by the periodic stimulus video generation unit 1002.

[0196] An image of the face of an elderly subject who has seen an image including a deceptive picture and an image without a deceptive picture is captured using the camera 1006.

[0197] The pulse wave extraction image processing unit 1008 is an example of a detection unit, and detects heartbeat information including the heartbeat of an elderly subject from the image of the elderly subject captured by the camera 1006.

[0198] The RRI Lyapunov emotion determination unit 1009 is an example of an emotion determination unit, and determines whether the emotion of the elderly subject is a negative emotion or a positive emotion based on the heartbeat information.

[0199] The pulse wave extraction image processing unit 1008 also has a function as a counting unit, and counts the number of times of heartbeat fluctuations that have transitioned from a state below the average heart rate to a state above the average heart rate with an increase in heart rate of a predetermined value or more within a predetermined period.

[0200] The pulse rate emotion determination unit 1010, which is an emotion determination unit, determines the mental state of the subject, for example, the presence or absence of emotion, based on the number of times of heartbeat fluctuations.

[0201] The synchronization analysis unit 1011 analyzes whether there is synchronization between a plurality of periods in which an image including a deceptive picture and an image not including a deceptive picture are displayed and the timing of the presence or absence of emotion determined by the emotion determination unit.

[0202] The cause determination unit 1012, which is an inhibitory factor determination unit, determines the presence or absence of mental abnormality based on whether the timing when an image including a deceptive picture is displayed and the timing when emotion occurs are synchronized. For example, in the example shown in FIG. 29, emotion is detected when images 903, 904, and 905 including deceptive pictures are displayed, and emotion is not detected when an image 902 not including a deceptive picture is displayed. That is, emotion is detected in accordance with the timing of displaying images 903, 904, and 905 including deceptive pictures. In this case, it can be determined that the elderly person, who is the subject, recognizes the meaning of the deceptive picture, and it can be determined that there is no mental abnormality.

[0203] Note that since it may be difficult to understand the meaning of an image such as a deceptive picture, when a plurality of images of deceptive pictures are displayed, it is not limited to the case where emotion is detected for all images of deceptive pictures. Even if emotion is not detected for some deceptive pictures and emotion is detected for other deceptive pictures, it may be determined that there is no mental abnormality. Therefore, as the second stimulus, a plurality of types of images with different contents may be selected for a video including predetermined information whose content can be understood.

[0204] Next, a case where audio containing predetermined information whose content can be understood is used as the second stimulus will be described. Fig. 30 shows an example of the temporal change in the emotion of an elderly subject when a voice that makes people feel surprised or laugh when its meaning can be understood is periodically played as the second stimulus containing predetermined information whose content can be understood by hearing. First, the voice "There is a gas leak" is output from the speaker or headphones 1003 one or more times over a predetermined period (e.g., 30 seconds) from time t1 to time t2. Next, no sound is output (silence) over a predetermined period (e.g., 30 seconds) from time t2 to time t3. Next, the voice "A fire has occurred" is output from the speaker or headphones 1003 one or more times over a predetermined period (e.g., 30 seconds) from time t3 to time t4. Next, no sound is output (silence) over a predetermined period (e.g., 30 seconds) from time t4 to time t5. Next, the voice "An earthquake has occurred" is output from the speaker or headphones 1003 one or more times over a predetermined period (e.g., 30 seconds) from time t5. The switching between the output of this voice and silence is executed by the periodic stimulus video generation unit 1002.

[0205] The face image of the elderly subject when a voice with a specific meaning is output and when no voice is output is captured using the camera 1006.

[0206] The pulse wave extraction image processing unit 1008 is an example of a detection unit, and detects heartbeat information including the heartbeat rate of the elderly subject from the image of the elderly subject captured by the camera 1006.

[0207] The RRI Lyapunov emotion determination unit 1009 is an example of an emotion determination unit, and determines whether the emotion of the elderly subject is a negative emotion or a positive emotion based on the heartbeat information.

[0208] The pulse wave extraction image processing unit 1008 also has a function as a counting unit, and counts the number of heartbeat fluctuations that transition from a state below the average heartbeat rate to a state above the average heartbeat rate with an increase in the heartbeat rate of a predetermined value or more within a predetermined period.

[0209] The pulse rate emotion determination unit 1010, which is an emotion determination unit, determines the mental state of the subject, such as the presence or absence of emotion, based on the number of fluctuations in the heart rate.

[0210] The synchronization analysis unit 1011 analyzes whether there is synchronization between a plurality of periods of acoustic stimulation and no stimulation and the timing of the presence or absence of emotion determined by the emotion determination unit.

[0211] The cause determination unit 1012, which is an inhibitory factor determination unit, determines the presence or absence of mental abnormality based on whether the timing of outputting the voice and the timing of the occurrence of emotion are synchronized. For example, in the example shown in FIG. 30, emotion is detected when a voice having a specific meaning is output, and emotion is not detected when the voice is not output. That is, emotion is detected in accordance with the timing of outputting the voice having a specific meaning. In this case, it can be determined that the elderly person, who is the subject, recognizes the meaning of the words, and it can be determined that there is no mental abnormality.

[0212] The pulse wave extraction image processing unit 1008 also functions as a calculation unit that calculates the complexity of the change in the fluctuation of the inter-beat interval from the heart rate information. The pulse wave extraction image processing unit 1008 can calculate the complexity of the change in the fluctuation of the inter-beat interval from the heart rate information, for example, by using the maximum Lyapunov exponent as an index. The RRI Lyapunov emotion determination unit 1009 also functions as an emotion determination unit that determines whether the emotion of the subject is a negative emotion or a positive emotion based on the complexity. The RRI Lyapunov emotion determination unit 1009 may determine whether the emotion of the subject is a negative emotion or a positive emotion, for example, based on the maximum Lyapunov exponent calculated by the pulse wave extraction image processing unit 1008. The stimulus given to the elderly person, who is the subject, by the stimulus generation unit may be a third stimulus including at least one of an image or a voice of a specific person. The cause determination unit 1012, which is an inhibitory factor determination unit, may determine the quality of the compatibility with a specific person in the subject.

[0213] First, an example will be described in the case where the third stimulus is a stimulus including an image of a specific person. FIG. 31 shows an example of the temporal change of the emotion and positive / negative feelings of an elderly person as the subject when an image of a caregiver is periodically shown to the elderly person as the subject as the third stimulus including an image of a specific person. First, the image 906 of the caregiver is displayed on the display unit 1001 for a predetermined period (for example, 30 seconds) from time t1 to t2. Next, a white image 902 not including the image of the caregiver is displayed on the display unit 1001 for a predetermined period (for example, 30 seconds) from time t2 to t3. After time t3, the image 906 of the caregiver and the white image 902 not including the image of the caregiver are alternately and repeatedly displayed on the display unit 1001. The switching between the image 906 of the caregiver and the white image 902 not including the image of the caregiver is executed by the periodic stimulus video generation unit 1002.

[0214] An image of the face of the elderly person as the subject who has seen the image 906 of the caregiver and the white image 902 not including the image of the caregiver is captured using the camera 1006.

[0215] The pulse wave extraction image processing unit 1008 is an example of a detection unit, and detects heartbeat information including the heartbeat rate of the elderly person as the subject from the image of the elderly person captured by the camera 1006.

[0216] The RRI Lyapunov emotion determination unit 1009 is an example of an emotion determination unit, and determines whether the emotion of the elderly person as the subject is a negative emotion or a positive emotion based on the heartbeat information. The RRI Lyapunov emotion determination unit 1009 may determine whether the emotion of the elderly person as the subject is a negative emotion or a positive emotion based on the complexity of the change in the fluctuation of the inter-beat interval calculated by the pulse wave extraction image processing unit 1008 from the heartbeat information.

[0217] The pulse wave extraction image processing unit 1008 also has a function as a counting unit, and counts the number of heartbeat fluctuations that have transitioned from a state below the average heartbeat rate to a state above the average heartbeat rate with an increase in the heartbeat rate of a predetermined value or more within a predetermined period.

[0218] The pulse rate emotion determination unit 1010, which is an emotion determination unit, determines, for each predetermined period, the mental state of the elderly person, who is the subject corresponding to the stimulus, such as the presence or absence of emotion, based on the number of fluctuations in the heart rate.

[0219] The synchronization analysis unit 1011 analyzes whether there is synchronization between a plurality of periods in which the image 906 of the caregiver and the white image 902 that does not include the image of the caregiver are displayed, the timing of the presence or absence of emotion determined by the emotion determination unit, and the timing when the positive and negative emotions of the elderly person appear.

[0220] The cause determination unit 1012, which is an inhibition factor determination unit, determines whether the compatibility between the elderly person, who is the subject, and the caregiver is good or bad based on whether the timing of displaying the image 906 of the caregiver is synchronized with the timing when emotion occurs and which of positive and negative emotions appears.

[0221] Here, it is considered that emotion may appear whether the compatibility between the elderly person, who is the subject, and the caregiver is good or bad when the elderly person is shown the image of the caregiver. Therefore, when emotion appears when the elderly person is shown the image of the caregiver, it can be determined that the elderly person has some kind of emotion towards the caregiver. However, based only on the presence or absence of emotion, it cannot be determined whether the compatibility between the elderly person and the caregiver is good or bad.

[0222] Therefore, the cause determination unit 1012 determines whether the compatibility between the elderly person and the caregiver is good or bad using the determination result (positive and negative emotions) by the RRI Lyapunov emotion determination unit 1009.

[0223] For example, when emotion appears when the elderly person is shown the image of the caregiver and, at the same timing, positive emotion appears in the elderly person, the cause determination unit 1012 can determine that the elderly person likes the caregiver in terms of personality, that is, the compatibility between the elderly person and the caregiver is good.

[0224] On the one hand, when an emotion appears when showing an image of a caregiver to an elderly person, and if a negative emotion appears in the elderly person at the same timing, the cause determination unit 1012 can determine that the elderly person dislikes the caregiver in terms of personality, that is, the compatibility between the elderly person and the caregiver is poor.

[0225] In this way, the cause determination unit 1012 also has a function as an emotion determination unit, and it is possible to use the determination result (positive / negative emotion) by the RRI Lyapunov emotion determination unit 1009, which is an emotion determination unit, for the determination of the mental state (good or bad compatibility).

[0226] That is, in this case, the pulse rate emotion determination unit 1010 and the cause determination unit 1012 assume the function as an emotion determination unit.

[0227] Also, the good or bad compatibility is caused by the mental state that causes it in the elderly person who is the subject (for example, a mental state that the subject likes or dislikes the target person, which the subject has unconsciously or consciously towards the target person). Therefore, as described above, the emotion determination unit determines the good or bad compatibility as a mental state.

[0228] For example, in the example shown in FIG. 31, an emotion is detected in accordance with the timing of displaying the image 906 of the caregiver, and a negative emotion is detected. In this case, at least, it can be determined that the compatibility between the elderly person who is the subject and the caregiver is poor because the compatibility of the elderly person with respect to the caregiver is poor.

[0229] Next, an example will be described in which the third stimulus is a stimulus including the voice of a specific person. FIG. 32 shows an example of the temporal changes in the emotions and positive / negative feelings of an elderly person as the subject when the voice of a caregiver, as the third stimulus including the voice of a specific person, is periodically presented to the elderly person as the subject. First, the voice of the caregiver saying "I am XX" by pronouncing their own name is output from the speaker or headphones 1003 one or more times over a predetermined period (e.g., 30 seconds) from time t1 to time t2. Next, the output of the voice is stopped (silent) over a predetermined period (e.g., 30 seconds) from time t2 to time t3. After time t3, the state of outputting the voice of the caregiver themselves and the silent state are alternately repeated. This switching between the state of outputting the voice of the caregiver themselves and the silent state is executed by the periodic stimulus sound generation unit 1004.

[0230] An image of the face of the elderly person as the subject who is periodically presented with the voice of the caregiver saying "I am XX" by pronouncing their own name is captured using the camera 1006.

[0231] The pulse wave extraction image processing unit 1008 is an example of a detection unit, and detects heartbeat information including the heartbeat rate of the elderly person as the subject from the image of the elderly person captured by the camera 1006.

[0232] The RRI Lyapunov emotion determination unit 1009 is an example of an emotion determination unit, and determines whether the emotion of the elderly person as the subject is a negative emotion or a positive emotion based on the heartbeat information. The RRI Lyapunov emotion determination unit 1009 may determine whether the emotion of the elderly person as the subject is a negative emotion or a positive emotion based on the complexity of the change in the fluctuation of the heartbeat interval calculated by the pulse wave extraction image processing unit 1008 from the heartbeat information.

[0233] The pulse wave extraction image processing unit 1008 also has a function as a counting unit, and counts the number of heartbeat fluctuations that transition from a state below the average heartbeat rate to a state above the average heartbeat rate with an increase in the heartbeat rate of a predetermined value or more within a predetermined period.

[0234] The pulse rate emotion determination unit 1010, which is an emotion determination unit, determines, for each predetermined period, the mental state of the elderly person, who is the subject corresponding to the stimulus, based on the number of fluctuations in the heart rate, for example, the presence or absence of emotion.

[0235] The synchronization analysis unit 1011 analyzes whether there is synchronization between a plurality of periods of acoustic stimulus and no stimulus, and the timing of the presence or absence of emotion determined by the emotion determination unit and the timing when the positive or negative emotion of the elderly person appears.

[0236] The cause determination unit 1012, which is an inhibition factor determination unit, determines whether the compatibility between the elderly person, who is the subject, and the caregiver is good or bad based on whether the timing of the caregiver outputting a voice such as "I am XX" by pronouncing their own name synchronizes with the timing when emotion occurs, and which of positive or negative emotions appears.

[0237] Here, it is considered that emotion may appear whether the compatibility between the elderly person, who is the subject, and the caregiver is good or bad when the elderly person hears the voice of the caregiver. Therefore, when emotion appears when the elderly person hears the voice of the caregiver, it can be determined that the elderly person has some kind of emotion towards the caregiver. However, based only on the presence or absence of emotion, it cannot be determined whether the compatibility between the elderly person and the caregiver is good or bad.

[0238] Therefore, the cause determination unit 1012 determines whether the compatibility between the elderly person and the caregiver is good or bad using the determination result (positive or negative emotion) by the RRI Lyapunov emotion determination unit 1009.

[0239] For example, when emotion appears when the elderly person hears the voice of the caregiver, and at the same timing, positive emotion appears in the elderly person, the cause determination unit 1012 can determine that the elderly person likes the caregiver in terms of personality, that is, the compatibility between the elderly person and the caregiver is good.

[0240] On the one hand, when an elderly person shows an emotion when hearing the voice of a caregiver, and at the same time, if a negative emotion appears in the elderly person at the same timing, the cause determination unit 1012 can determine that the elderly person dislikes the caregiver in terms of personality, that is, the compatibility between the elderly person and the caregiver is poor.

[0241] In this way, the cause determination unit 1012 also has a function as an emotion determination unit, and it is possible to use the determination result (positive / negative emotion) by the RRI Lyapunov emotion determination unit 1009, which is an emotion determination unit, for the determination of the psychological state (good or bad compatibility).

[0242] That is, in this case, the pulse rate emotion determination unit 1010 and the cause determination unit 1012 assume the function as an emotion determination unit.

[0243] Also, the good or bad compatibility is caused by the psychological state that causes it in the elderly person who is the subject (for example, a psychological state that the subject likes or dislikes the target person, which the subject has unconsciously or consciously towards the target person). Therefore, as described above, the emotion determination unit determines the good or bad compatibility as a psychological state.

[0244] For example, in the example shown in FIG. 32, an emotion is detected in accordance with the timing when the caregiver himself / herself outputs a voice such as "I am XX" that pronounces his / her own name, and a negative emotion is detected. In this case, it can be determined that the compatibility between the elderly person who is the subject and the caregiver is poor.

[0245] The periodic stimulus video generation unit 1002 and the periodic stimulus sound generation unit 1004, which are stimulus generation units, repeat the operation of generating different stimuli among the first to third stimuli after generating any one of the first to third stimuli at predetermined intervals, at least until three stimuli among the first to third stimuli are generated. Note that the heart rate emotion determination unit 1010 determines that the subject has an emotion when the number of heart rate fluctuations is two or more during each stimulus generation period and each stimulus rest period (both corresponding to a predetermined period). This is because an emotion may occur even without generating a stimulus in response to the switching of the screen or the like between the stimulus generation period and the stimulus rest period. That is, this emotion is an emotion other than the first to third stimuli and should be excluded from the determination of the presence or absence of emotions caused by the first to third stimuli. The first to third stimuli generated by the periodic stimulus video generation unit 1002 and the periodic stimulus sound generation unit 1004 are stimuli that cause at least two or more emotions. This is also because the purpose is not to test reflexive reactions such as surprise determined by the number of heart rate fluctuations per stimulus, but to test the presence or absence of changes in mental states determined by the number of heart rate fluctuations of two or more times.

[0246] By determining the presence or absence of emotions of the elderly when the first and second stimuli are given to the elderly, and by determining the positive and negative emotions and the presence or absence of emotions of the elderly when the third stimulus is given to the elderly, it is possible to determine the cause of inhibiting the communication of the elderly. That is, the cause determination unit 1012, which is an inhibition factor determination unit, can make a plurality of determinations on the presence or absence of visual or auditory abnormalities, the presence or absence of mental abnormalities of the subject, and the compatibility with a specific person. As a determination regarding the inhibition factor, it is further possible to determine which of the plurality of determinations the inhibition factor possessed by the subject is caused by. FIG. 33 shows an example of determining the cause of inhibiting the communication of the elderly.

[0247] First, an example in which an elderly person is determined to be in a normal state will be described. When the elderly person shows a reaction when given a first stimulus for vision and hearing determination, it can be determined that the elderly person has no abnormalities in vision and hearing. Also, when the elderly person shows a reaction when given a second stimulus for mental disorder determination, it can be judged that the elderly person is in a state without mental abnormalities. Furthermore, when the elderly person is given a third stimulus for compatibility determination and does not react or shows a positive reaction (positive emotion) even if they do react, it can be determined that the compatibility between the elderly person and the caregiver is not bad.

[0248] Next, an example in which an elderly person is determined to be in a state of mental abnormality will be described. When the elderly person shows a reaction when given a first stimulus for vision and hearing determination, it can be determined that the elderly person has no abnormalities in vision and hearing. Also, when the elderly person does not show a reaction when given a second stimulus for mental disorder determination, since they are in a state where they cannot understand the meaning of the images and sounds included in the second stimulus, it can be judged that the elderly person is in a state of mental abnormality. Furthermore, when the elderly person is given a third stimulus for compatibility determination and does not react or shows a positive reaction (positive emotion) even if they do react, it can be determined that the compatibility between the elderly person and the caregiver is not bad.

[0249] Next, an example in which an elderly person is determined to be in a state of hearing abnormality will be described. When the elderly person shows a reaction when given a first stimulus for vision determination, it can be determined that the elderly person has no abnormality in vision. When the elderly person does not show a reaction when given a first stimulus for hearing determination, it can be determined that the elderly person is in a state of hearing abnormality. If the elderly person has an abnormality in hearing, even if a second stimulus including a voice for mental disorder determination is given to the elderly person, the presence or absence of mental abnormality cannot be determined from the presence or absence of a reaction. Therefore, regardless of the presence or absence of an acoustic reaction to the second stimulus including a voice, in order to determine hearing abnormality, it is indicated as "no / yes" in the table shown in FIG. 33. Furthermore, when the elderly person is given a third stimulus for compatibility determination and does not react or shows a positive reaction (positive emotion) even if they do react, it can be determined that the compatibility between the elderly person and the caregiver is not bad. In the example shown in FIG. 33, a case where the compatibility between the elderly person and the caregiver is good is shown, but there may also be a case where the elderly person has hearing abnormality and the compatibility with the caregiver is not good.

[0250] Next, an example in which an elderly person is determined to be in a visual state will be described. When an elderly person shows a reaction when a first stimulus for hearing ability determination is given, it can be determined that the elderly person has no hearing abnormality. When an elderly person shows no reaction when a first stimulus for visual ability determination is given, it can be determined that the elderly person is in a state of visual abnormality. When an elderly person has a visual abnormality, even if a second stimulus including an image for mental disorder determination is given to the elderly person, the presence or absence of mental abnormality cannot be determined from the presence or absence of a reaction. Therefore, regardless of the presence or absence of an image reaction to the second stimulus including an image, in order to determine visual abnormality, it is indicated as "no / yes" in the table shown in FIG. 33. Further, when the elderly person shows no reaction or shows a positive reaction (positive emotion) even when reacting when a third stimulus for compatibility determination is given to the elderly person, it can be determined that the compatibility between the elderly person and the caregiver is not bad. In the example shown in FIG. 33, the case where the compatibility between the elderly person and the caregiver is good is shown, but there may be a case where the elderly person has a hearing abnormality and the compatibility with the caregiver is not good either.

[0251] Next, an example of determining the goodness or badness of the compatibility of an elderly person will be described. When an elderly person shows a reaction when a first stimulus for visual and hearing ability determination is given, it can be determined that the elderly person has no abnormality in vision and hearing. Also, when an elderly person shows a reaction when a second stimulus for mental disorder determination is given, it can be determined that the elderly person is in a state without mental abnormality. Further, when the elderly person shows a reaction and shows a negative reaction (negative emotion) when a third stimulus for compatibility determination is given to the elderly person, it can be determined that the compatibility between the elderly person and the caregiver is bad. Note that when a negative reaction appears when at least one of an image stimulus and an acoustic stimulus is given as the third stimulus for compatibility determination to the elderly person, it may be determined that the compatibility between the elderly person and the caregiver is bad. Note that, unlike the case shown in FIG. 33, when the elderly person shows a reaction and shows a positive reaction (positive emotion) when a third stimulus for compatibility determination is given to the elderly person, it can be determined that the compatibility between the elderly person and the caregiver is good.

[0252] Next, a method for determining the presence or absence of emotion will be described. FIG. 34(A) shows an example of the true value of the reaction that appears when a stimulus is periodically applied. First, a predetermined period (e.g., 30 sec) from time t1 to t2 is set as a stimulus pause period during which no stimulus is applied to the subject. At this time, since the subject is predicted not to show a reaction, the true value is set to "0". Next, a predetermined period (e.g., 30 sec) from time t2 to t3 is set as a stimulus generation period during which a stimulus is applied to the subject. At this time, since the subject is predicted to show a reaction, the true value is set to "1". After time t3, when the stimulus pause period and the stimulus generation period are periodically repeated, accordingly, the true value "0" during the stimulus pause period and the true value "1" during the stimulus generation period are periodically repeated.

[0253] FIG. 34(B) shows an example of emotion determination when a stimulus is applied to the subject in accordance with the timing shown in FIG. 34(A). For example, when the period from time t1 to t2 is a stimulus pause period and no emotion is detected when no stimulus is applied to the subject, the emotion is "none", which is represented as "0". Since the true value at this time is "0" from FIG. 34(A), the emotion during this stimulus pause period coincides with the true value. Next, when the period from time t2 to t3 is a stimulus generation period and emotion is detected when a stimulus is applied to the subject, the emotion is "present", which is represented as "1". Since the true value at this time is "1" from FIG. 34(A), the emotion during this stimulus generation period coincides with the true value.

[0254] From time t3 to t11, the stimulation pause period and the stimulation generation period are periodically repeated, and the emotion is determined in a total of 10 periods. At this time, if no emotion is detected during the stimulation generation period from time t6 to t7, the emotion is "none" ("0"). However, since the true value during this period is "1", the determination result of the emotion does not match the true value. Among the total 10 periods from t1 to t11, excluding the period from t6 to t7, the determination result of the emotion matches the true value in the other 9 periods, so the synchronization rate is calculated to be 90%. The synchronization analysis unit 1011 determines based on whether the ratio of the presence or absence of emotion synchronized in the stimulation generation period and the stimulation pause period, that is, whether the synchronization rate is equal to or higher than a predetermined value. For example, when the predetermined value is 70%, since the calculated synchronization rate is 90%, it can be determined that the emotion is synchronized with the stimulation because it exceeds the predetermined value of 70%.

[0255] In the above example, an example of calculating and determining the synchronization rate of the presence or absence of emotion for a plurality of periods of applying stimulation and the stimulation pause period is shown, but it is not limited to such an example. That is, when the number of stimulation occurrences and the number of stimulation pauses are fixed, even if the synchronization rate is not calculated, the determination based on whether the number of synchronized times is equal to or more than a predetermined number can also be regarded as the determination by the synchronization rate (the ratio of synchronization). For example, as shown in FIGS. 34(A) and 34(B), when the total period of the stimulation generation period and the stimulation pause period is fixed to 10 periods, and the number of synchronized times in the stimulation generation period and the stimulation pause period is equal to or more than a predetermined number (for example, 7 times), it may be determined that it is synchronized by regarding the synchronization rate as being above 70% of the predetermined value.

[0256] By using the subject state determination device 1000, which is a communication inhibition cause determination device, for example, an examination can be conducted on the first day an elderly person moves into a nursing facility to identify what type of elderly person it is. Also, when determining the compatibility between an elderly person and a caregiver, since it may not be possible to determine that the caregiver and the elderly person have not interacted for a predetermined period, for example, an examination may be conducted about one month after the elderly person moves into the nursing facility. By determining the cause that inhibits communication between the elderly person and the caregiver, appropriate measures can be taken to promote communication.

[0257] [Appendix] The above-described Example 6 is a specific exemplification of the inventions described in each of the appendices shown below.

[0258] (Appendix 1) A stimulus generation unit that generates stimuli to the subject's vision or hearing at a plurality of predetermined periods each having a pause period in between; A detection unit that detects heartbeat information including the subject's heartbeat; A counting unit that counts, for each of the predetermined periods, the number of heartbeat fluctuations that have transitioned from a state below the average heartbeat to a state above the average heartbeat with an increase in the heartbeat of a predetermined value or more; An emotion determination unit that determines, for each of the predetermined periods, the presence or absence of the subject's emotion corresponding to the stimulus based on the number of heartbeat fluctuations; A synchrony analysis unit that analyzes whether there is synchrony between the plurality of periods and the timing of the presence or absence of the emotion determined by the emotion determination unit; The subject state determination device having an inhibition factor determination unit that determines, for the subject, a determination regarding an inhibition factor related to communication with others based on the analysis result of the synchrony determination unit. Characterized in that it is a subject state determination device.

[0259] As a device for determining communication inhibition factors known so far, there is a device that determines the presence or absence of abnormalities in vision, hearing, and cognitive abilities through the vision and hearing of a subject. However, in the case of the conventional device, when determining from the subject's response or reaction such as button pressing, there are difficulties in the reaction behavior itself and there are cases where determination cannot be made. In addition, there is a device that determines communication inhibition factors from the electroencephalogram of a subject. However, such a device is large-scale, so there is a problem that a large space is required as a place to install the device. Furthermore, such a device requires advanced analysis means, so there is a problem that the number of people who can handle the device is limited. Therefore, it is preferable to realize the communication inhibition factor of the subject with a simple configuration easily and to reduce the restrictions on the installation location and the people who can handle it. According to the subject state determination device described in the appended note, based on "analyzing whether or not there is synchronism at the timing of the number of heart rate fluctuations that have transitioned to a state equal to or higher than the average heart rate" and "the presence or absence of emotion", "judgment regarding the inhibition factor is performed". Therefore, the device configuration is simple, the restriction on the installation location can be reduced, and furthermore, since the communication inhibition factor of the subject can be determined by a simple test, the restriction on the people who can handle it can also be reduced.

[0260] (Appended Note 2) The stimulus is a first stimulus whose generation can be sensuously recognized by vision or hearing, The inhibition factor determination unit determines the presence or absence of abnormality in the vision or hearing, The subject state determination device described in Appended Note 1.

[0261] In the subject state determination device described in Appended Note 2, the stimulus may be a first stimulus whose generation can be recognized only by the sense of vision or hearing.

[0262] (Appended Note 3) The stimulus is a second stimulus including predetermined information that can be understood in content, The inhibition factor determination unit determines the presence or absence of mental abnormality of the subject, The subject state determination device according to Supplementary Note 1 or 2.

[0263] (Supplementary Note 4) Furthermore, a calculation unit that calculates the complexity of the change in the fluctuation of the heart rate interval from the heart rate information, and an emotion determination unit that determines whether the emotion of the subject is a negative emotion or a positive emotion based on the complexity, The stimulus is a third stimulus including at least one of an image or voice of a specific person, The inhibition factor determination unit determines the quality of the compatibility between the specific person and the subject. The subject state determination device according to any one of Supplementary Notes 1 to 3.

[0264] (Supplementary Note 5) The inhibition factor determination unit, can make a plurality of determinations including the presence or absence of visual or auditory abnormalities, the presence or absence of mental abnormalities in the subject, and the quality of the compatibility with the specific person, As the determination regarding the inhibition factor, it further determines which of the plurality of determinations the inhibition factor possessed by the subject is caused by. The subject state determination device according to Supplementary Note 4.

[0265] (Supplementary Note 6) After the stimulus generation unit generates any one of the first to third stimuli every predetermined period, it repeats the operation of generating different stimuli among the first to third stimuli until at least the three stimuli of the first to third are generated. The subject state determination device according to Supplementary Note 4.

[0266] (Supplementary Note 7) The synchronization analysis unit determines based on whether the ratio of synchronization of the presence or absence of the emotion during the stimulus generation period and the stimulus rest period is equal to or greater than a predetermined value. The subject state determination device according to any one of Supplementary Notes 1 to 6.

[0267] In the subject state determination device according to Supplementary Note 7, it also includes the case where the number of stimulus generations is fixed and the number of synchronized times is equal to or greater than a predetermined number.

[0268] (Appendix 8) The first stimulus is a sensory stimulus that stimulates vision or hearing and whose occurrence is recognized, and is the subject state determination device according to any one of Appendices 1 to 7.

[0269] (Appendix 9) The second stimulus is a cognitive stimulus accompanied by the recognition of predetermined information, and is the subject state determination device according to any one of Appendices 3 to 8.

Claims

1. A detection unit that detects heartbeat information including the heartbeat rate of a subject; An emotion determination unit that determines whether the emotion of the subject is a negative emotion or a positive emotion based on the heartbeat information; A counting unit that counts the number of heartbeat fluctuations that have transitioned from a state below the average heartbeat rate to a state above the average heartbeat rate with an increase in the heartbeat rate of a predetermined value or more within a predetermined period; An emotional state determination unit that determines the mental state of the subject based on the number of heartbeat fluctuations and uses the determination result by the emotion determination unit for the determination of the mental state; An output unit that outputs the determination result of the emotional state determination unit, and has, The negative emotion is an emotion in which the subject feels at least one of brain fatigue, anxiety, and depression, and the positive emotion is an emotion in which the subject does not feel any of the brain fatigue, the anxiety, and the depression, The mental state includes a state of being grateful and a state of being indignant, When the number of heartbeat fluctuations counted by the counting unit is plural and the emotion of the subject determined by the emotion determination unit is a positive emotion, the emotional state determination unit determines that the mental state is the state of being grateful, When the number of heartbeat fluctuations counted by the counting unit is plural and the emotion of the subject determined by the emotion determination unit is a negative emotion, the emotional state determination unit determines that the mental state is the state of being indignant, An emotion determination device characterized by the above.

2. Further, it has a photographing unit that photographs the face of the subject, The detection unit detects the heartbeat information based on the change in the image data acquired by the photographing unit, The emotion determination device according to claim 1.

3. The predetermined period is repeatedly set, The emotional state determination unit makes a determination for each predetermined period. The emotion determination device according to claim 1 or 2.

4. The photographing unit can photograph the faces of a plurality of subjects, Furthermore, it has a measurement site specifying means for specifying each face from a screen on which a plurality of people are displayed and specifying a measurement site for each specified face, The detection unit acquires the heartbeat information based on the change in the image of the measurement site of each face, The emotion determination device according to claim 2.

5. It further has a calculation unit that calculates a maximum Lyapunov exponent indicating the degree of fluctuation of the inter-beat interval from the heartbeat information, The subject is a learner who takes a lecture, When the maximum Lyapunov exponent of the attendee calculated by the calculation unit is between -0.7 and 0, and the pulse of the attendee detected by the detection unit is within a predetermined range from the average pulse of the attendee, the emotion determination unit determines that the attendee is in an ideal mental state during the lecture. The emotion determination device according to any one of claims 1 to 4.

6. A stimulus that can be sensuously recognized as to the presence or absence of generation by vision or hearing, A stimulus that can understand the content of information given by vision or hearing, A stimulus generation unit that generates a stimulus including at least one of an image or voice of a specific person, The stimulus generation unit repeatedly generates the same type of stimulus in a plurality of predetermined periods with a pause period in between, The emotion determination unit determines the mental state of the subject at least during the plurality of predetermined periods. The emotion determination device according to any one of claims 1 to 5.

7. The mental state includes a surprised state. When the number of fluctuations in the heart rate counted by the counting unit is 1, The emotion determination unit determines that the mental state is the surprised state. The emotion determination device according to any one of claims 1 to 3.

8. The mental state includes a stable state. When the number of fluctuations in the heart rate counted by the counting unit is 0 and the state where the heart rate is less than the average heart rate is maintained during the predetermined period, The emotion determination unit determines that the mental state is the stable state. The emotion determination device according to any one of claims 1 to 6.

9. The mental state includes an emotion determination impossible state in which the mental state cannot be determined. When the number of fluctuations in the heart rate counted by the counting unit is 0 and the state where the heart rate is greater than or equal to the average heart rate is maintained during the predetermined period, The emotion determination unit determines that it is the emotion determination impossible state. The emotion determination device according to any one of claims 1 to 6.

10. The step of detecting heart rate information including the heart rate of the subject, A first determination step of determining whether the emotion of the subject is a negative emotion or a positive emotion based on the heart rate information, The step of counting the number of fluctuations in the heart rate that has transitioned from a state less than the average heart rate to a state greater than or equal to the average heart rate with an increase in the heart rate of a predetermined value or more during a predetermined period. Based on the number of fluctuations in the heart rate, determine the mental state of the subject, and use the determination result of the first determination step in the determination of the mental state in a second determination step; Cause a computer to execute a step of outputting the determination result of the mental state; The negative emotion is an emotion in which the subject feels at least one of brain fatigue, anxiety, and depression, and the positive emotion is an emotion in which the subject does not feel any of the brain fatigue, the anxiety, and the depression; The mental state includes a state of being grateful and a state of being indignant; In the second determination step; When the number of fluctuations in the heart rate is multiple times and the emotion of the subject is a positive emotion, determine that the mental state is the state of being grateful; When the number of fluctuations in the heart rate is multiple times and the emotion of the subject is a negative emotion, determine that the mental state is the state of being indignant; A characteristic emotion determination program.

11. A detection unit detects heart rate information including the heart rate of a subject; An emotion determination unit determines whether the emotion of the subject is a negative emotion or a positive emotion based on the heart rate information; A counting unit counts the number of fluctuations in the heart rate that has transitioned from a state below the average heart rate to a state above the average heart rate with an increase in the heart rate of a predetermined value or more during a predetermined period; An emotional state determination unit determines the mental state of the subject based on the number of fluctuations in the heart rate, and uses the determination result by the emotion determination unit in the determination of the mental state; An output unit outputs the determination result of the emotional state determination unit; The negative emotion is an emotion in which the subject feels at least one of brain fatigue, anxiety, and depression, and the positive emotion is an emotion in which the subject does not feel any of the brain fatigue, the anxiety, and the depression; The mental state includes a state of being grateful and a state of being indignant; When the number of fluctuations in the heart rate counted by the counting unit is multiple times and the emotion of the subject determined by the emotion determination unit is a positive emotion, the emotional state determination unit determines that the mental state is the state of being grateful; When the number of fluctuations in the heart rate counted by the counting unit is multiple times and the emotion of the subject determined by the emotion determination unit is a negative emotion, the emotional state determination unit determines that the mental state is the state of being indignant; An emotion determination method characterized by this.

Images