Emotion determination device, emotion determination method, and emotion determination program

Abstract

The emotion determination device has: a detection unit that detects heartbeat information including a heart rate of a subject; an emotion determination unit that determines whether an emotion of the subject is a negative emotion or a positive emotion based on the heartbeat information; a counting unit that counts, for a prescribed period, a number of heart rate changes from a state below an average heart rate to a state of the average heart rate or more that accompanies an increase in the heart rate to a value of the average heart rate or more; a mood determination unit that determines a psychological state of the subject based on the number of heart rate changes, and uses a determination result of the emotion determination unit for the determination of the psychological state; and an output unit that outputs a determination result of the mood determination unit.

Description

Technical Field

[0001] This invention relates to an emotion determination device, emotion determination method, and emotion determination procedure for determining the emotions of an object. Background Technology

[0002] In elder care, understanding the emotions of the elderly is crucial for caregivers in determining subsequent care strategies. However, sometimes the elderly lack facial expressions due to aging and have difficulty conversing, making it challenging to glean emotions from their facial expressions, atmosphere, or conversational content.

[0003] Furthermore, in video calls and teleconferences, it is crucial for the speaker to understand the audience's emotional state in relation to what they are saying, which is essential for advancing the conversation. However, in teleconferences, the images reflecting the audience's expressions are limited to those captured by the camera, making it difficult to glean emotions from the audience's facial expressions and the atmosphere. Generally, the term "teleconference," "web conference," or other names are sometimes used depending on the context; in this manual, these conferences will be collectively referred to as "web conferences."

[0004] Patent Document 1 discloses a communication device comprising: an emotion analysis unit that analyzes the emotion of the other party during communication; a storage unit that stores the emotion data analyzed by the emotion analysis unit in correspondence with the other party who performed the emotion analysis; a notification unit that notifies the other party based on the emotion data stored in the storage unit; and a control unit that, when a other party is selected, reads the emotion data corresponding to that other party from the storage unit and notifies the other party through the notification unit.

[0005] Patent Document 2 discloses an image processing apparatus comprising: an image data acquisition unit that acquires image data of multiple meeting participants; a face image detection unit that detects the face image of each meeting participant from the image data acquired by the image data acquisition unit; an image composition unit that segments the detected face image and reconstructs it into an image; an emotion estimation unit that estimates the emotion of each participant based on the detected face image; and a display mode changing unit that changes the display mode of the face image of each participant based on the estimated emotion.

[0006] In the invention described in Patent Document 1, since the emotion is determined by analyzing the voice of the other party in the conversation, the other party needs to speak, which is not applicable to people with few conversations or audience members of a meeting. In addition, in the invention described in Patent Document 2, facial expressions are inferred from the facial images of the meeting participants, and emotions are inferred based on the inferred expressions, but there is a problem that it is difficult to infer the emotions of participants who lack facial expressions.

[0007] Existing technical documents

[0008] Patent documents

[0009] Patent Document 1: Japanese Patent Application Publication No. 2005-311915

[0010] Patent Document 2: Japanese Patent Application Publication No. 2020-48149 Summary of the Invention

[0011] The purpose of this invention is to provide an emotion determination device, emotion determination method, and emotion determination procedure that can detect the other party's emotions in a non-contact manner.

[0012] The emotion determination device according to the embodiments of this disclosure includes: a detection unit that detects heartbeat information including the heart rate of a subject; an emotion determination unit that determines whether the subject's emotion is negative or positive based on the heartbeat information; a counting unit that counts the number of heart rate fluctuations that change from a state below the average heart rate to a state above the average heart rate as the heart rate increases to a value above the average heart rate during a predetermined period; an emotion determination unit that can determine the subject's psychological state based on the number of heart rate fluctuations and uses the determination result of the emotion determination unit for the determination of psychological state; and an output unit that outputs the determination result of the emotion determination unit.

[0013] The emotion determination device also has a camera unit that captures the subject's face, and a detection unit that detects heartbeat information based on changes in the image data acquired by the camera unit.

[0014] In the emotion determination device, the psychological state includes: a first psychological state that can be determined based on the number of heart rate fluctuations, regardless of whether it is a positive or negative emotion; and a second psychological state that can be determined based on whether it is a positive or negative emotion and the number of heart rate fluctuations. The emotion determination unit uses the determination result of the emotion determination unit at least when determining the second psychological state.

[0015] In the emotion determination device, a specified period is repeatedly set, and the emotion determination unit makes a determination according to each specified period.

[0016] In the emotion determination device, the imaging unit can capture images of the faces of multiple subjects. The emotion determination device also has a measurement location determination unit, which determines each face from the image displaying multiple people and determines the measurement location for each determined face. The detection unit obtains heartbeat information based on the changes in the images of the measurement locations of each face.

[0017] In the aforementioned emotion determination device, the subject is a listener of a lecture. The emotion determination unit determines whether the subject is in the ideal psychological state for listening to the lecture based on the number of heart rate fluctuations and the determination results of the emotion determination unit.

[0018] The emotion determination device includes a stimulus generation unit that generates stimuli such as: stimuli that can be visually or auditorily identified as having occurred; stimuli that allow understanding of the content of information provided by visual or auditory means; and stimuli containing at least one of an image or sound of a specific person. The stimulus generation unit repeatedly generates the same type of stimulus at predetermined intervals with rest periods, and the emotion determination unit determines the psychological state of the subject at least at the predetermined intervals.

[0019] In the aforementioned emotion determination device, negative emotion is the emotion that the test subject feels, such as brain fatigue, anxiety, and depression.

[0020] In the emotion determination device, the psychological state of the subject determined by the emotion determination unit includes any one of the following: a stable state, a surprised state, a grateful state, and an angry state.

[0021] In the emotion determination device, if the number of heart rate changes counted by the counting unit is one, the emotion determination unit determines that the psychological state is one of surprise.

[0022] In the emotion determination device, if the number of heart rate changes counted by the counting unit is multiple and the emotion determination unit determines that the subject's emotion is positive, the emotion determination unit determines that the psychological state is one of gratitude.

[0023] In the emotion determination device, if the number of heart rate changes counted by the counting unit is multiple and the emotion determination unit determines that the subject's emotion is negative, the emotion determination unit determines that the psychological state is anger.

[0024] In the emotion determination device, if the number of heart rate changes counted by the counting unit is 0 and the heart rate remains below the average heart rate for a specified period, the emotion determination unit determines that the psychological state is stable.

[0025] In the emotion determination device, if the number of heart rate changes counted by the counting unit is 0 and the heart rate is maintained at or above the average heart rate for a specified period, the emotion determination unit determines that the psychological state cannot be determined and the emotion cannot be determined.

[0026] The emotion determination program of this disclosure causes a computer to perform the following steps: detect heartbeat information including the heart rate of the subject; determine whether the subject's emotion is negative or positive based on the heartbeat information; count the number of heartbeat changes during a specified period, which occur as the heart rate increases above a specified value, changing from a state below the average heart rate to a state above the average heart rate; determine the subject's psychological state based on the number of heartbeat changes, and use the determination result for the psychological state determination; and output the psychological state determination result.

[0027] In the emotion determination method of this disclosure, the detection unit detects heartbeat information including the heart rate of the subject; the emotion determination unit determines whether the subject's emotion is negative or positive based on the heartbeat information; the counting unit counts the number of heart rate changes during a specified period, which occur as the heart rate increases to a value above the average heart rate, from a state below the average heart rate to a state above the average heart rate; the emotion determination unit determines the subject's psychological state based on the number of heart rate changes and uses the determination result of the emotion determination unit for the determination of psychological state; and the output unit outputs the determination result of the emotion determination unit.

[0028] The emotion determination apparatus, emotion determination method, and emotion determination procedure according to the embodiments of this disclosure can detect the other party's emotions in a non-contact manner. Attached Figure Description

[0029] Figure 1 This is a perspective view showing the usage state of the emotion determination device according to the embodiments of this disclosure.

[0030] Figure 2 This is a block diagram of an emotion determination device according to an embodiment of the present disclosure.

[0031] Figure 3 (A) to (C) are diagrams used to illustrate the principle of judging negative emotions based on pulse.

[0032] Figure 4 (A) to (C) are diagrams used to illustrate the principle of judging emotions based on changes in heart rate over a specified period.

[0033] Figure 5 (A) and (B) are charts illustrating examples of situations where it is difficult to determine emotions based on changes in heart rate over a specified period.

[0034] Figure 6 This is a flowchart illustrating an example of the operation of the emotion determination device according to an embodiment of the present disclosure.

[0035] Figure 7 This is a diagram showing an example of a screen display in a web conference using the emotion determination device of Embodiment 1.

[0036] Figure 8 This is a block diagram of the emotion determination device in Embodiment 1.

[0037] Figure 9 This is a diagram showing an example of the display of the emotion determination device in Embodiment 2.

[0038] Figure 10 This is a block diagram of the emotion determination device in Embodiment 2.

[0039] Figure 11 This is a schematic diagram of the structure of a marketing survey system that uses the sentiment determination device of Example 3.

[0040] Figure 12 This is a block diagram of the emotion determination device in Embodiment 3.

[0041] Figure 13 This is a schematic diagram of the mechanical control system using the emotion determination device of Embodiment 4.

[0042] Figure 14 This is a block diagram of the emotion determination device in Embodiment 4.

[0043] Figure 15 This is a block diagram of the course-organizing PC, which is the emotion determination device in Embodiment 5.

[0044] Figure 16 This is an example of displaying the images of participants in a scenario with multiple participants.

[0045] Figure 17 This is an example of an image display where a portion of the image is cut from the facial images of multiple participants.

[0046] Figure 18 This is a graph showing the relationship between the maximum Lyapunov index and the subjective exercise intensity (RPE).

[0047] Figure 19 The Borg score is a measure of the relationship between RPE and subjective effort / fatigue during training and exercise.

[0048] Figure 20 This is an example of the change in pulse rate over time when the pulse rate moves from the average resting pulse rate into a specified range.

[0049] Figure 21 This is a flowchart illustrating the operational sequence of the emotion determination device in Embodiment 5.

[0050] Figure 22 (A) is an example of the output of a feature coordinate fitting method that extracts feature coordinates from a facial image. Figure 22 (B) is an example of the output method for detecting the position between the eyebrows from a captured facial image.

[0051] Figure 23 This is a configuration example for performing image acquisition using HDMI (registered trademark) output.

[0052] Figure 24 This is an example of the reasons and solutions that hinder communication between the elderly and their caregivers.

[0053] Figure 25 Examples of visual and auditory stimuli used to assess communication barriers in three categories: compatibility, visual / hearing impairment, and mental illness.

[0054] Figure 26 This is a block diagram of the subject status determination device of the communication obstruction cause determination device as an embodiment 6.

[0055] Figure 27 This is an example of the temporal changes in the emotions of elderly subjects who were periodically observed when the first stimulus, such as red, was seen to occur through visual recognition.

[0056] Figure 28 This is an example of the temporal changes in the emotions of an elderly person who was being tested, when they periodically heard a beat at a specified frequency as the first stimulus that could be recognized through auditory perception.

[0057] Figure 29 This is an example of the temporal changes in the emotions of elderly subjects who, as the subjects, periodically saw illusions of surprise or laughter when they saw a second stimulus containing prescribed information that could be understood visually.

[0058] Figure 30 This is an example of the temporal changes in the emotions of an elderly subject who periodically hears a sound that would elicit surprise or laughter if the meaning could be understood, as the second stimulus containing prescribed information that can be understood through hearing.

[0059] Figure 31 This is an example of the temporal changes in the emotions and positive and negative feelings of the elderly test subjects when they periodically see the image of their caregiver as a third stimulus containing an image of a specific person.

[0060] Figure 32 This is an example of the temporal changes in the emotions and positive and negative feelings of the elderly test subjects when they periodically hear the voice of their caregivers as a third stimulus containing the voice of a specific person.

[0061] Figure 33 This is an example of determining the reasons that hinder communication among the elderly.

[0062] Figure 34 (A) is an example of the truth value of the response that occurs when a stimulus is applied periodically. Figure 34 (B) is with Figure 34 Example of judging emotions when stimuli are applied to the subject at consistent intervals, as shown in (A). Detailed Implementation

[0063] The following description, with reference to the accompanying drawings, outlines the emotion determination device, emotion determination method, and emotion determination procedure of the present invention. However, the scope of the present invention is not limited to these embodiments; please note the invention described in the technical solutions and its equivalents.

[0064] Figure 1 This is a perspective view showing the usage state of the emotion determination device 1 according to an embodiment of the present disclosure. Figure 1 As shown, the emotion determination device 1 includes a camera unit 10 and an information terminal 5. In the illustrated example, the camera unit 10 is a portable terminal such as a smartphone, and the information terminal 5 is a notebook computer (PC) with a display unit 41. However, it is not limited to this example; a tablet computer or a digital camcorder may be used as the camera unit 10, and a tablet computer, a desktop PC, or a dedicated processing device may be used as the information terminal 5. The camera unit 10 and the information terminal 5 may also be integrated.

[0065] exist Figure 1 The image shows a state where the camera unit 10 is mounted on a support 90 that holds the camera unit 10. (Example) Figure 1 As shown, the camera unit 10 includes a camera element 11 and a touch panel 19 with a display for setting the operation of the camera unit 10.

[0066] Especially among the elderly, some people have a fear of the test, some resist the test itself, such as the sensor installation, and some experience negative emotions just from hearing the instructions. Therefore, the emotion determination device 1 uses a camera unit 10 with an imaging element (camera) 11 to photograph the exposed parts of the subject's skin (e.g., the forehead or cheeks) in a way that prevents the test from causing stress. Then, the emotion determination device 1 automatically detects the pulse signal, which serves as heartbeat information, from the obtained image by extracting brightness changes synchronized with blood flow, in a non-contact manner while the subject remains unconscious.

[0067] The imaging element 11 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) type image sensor. During each measurement, the imaging element 11, such as... Figure 1 As shown, for example, multiple images Gr of the measurement frame Sa of the forehead of the subject HK are automatically and continuously captured without the subject's operation. The imaging unit 10 has the function of automatically tracking the measurement frame Sa of the subject HK's forehead according to a built-in facial recognition application. Thus, even if the subject HK moves back and forth within the set area of ​​the imaging unit 10, the pulse of the subject HK can be captured. Figure 1 As shown, the imaging unit 10 uses its built-in wireless communication function to send the image data of the subject HK captured by the imaging unit to the information terminal 5 via radio wave RW.

[0068] Figure 2 This is a block diagram of the emotion determination device 1. (For example...) Figure 1 As shown, the information terminal 5 of the emotion determination device 1 includes an emotion detection unit 20, a determination unit 30, a notification unit 40, and a timing unit 50. The emotion detection unit 20 includes a face recognition unit 21, a pulse extraction unit 22, an interval detection unit 23, a pulse memory 24, a chaos analysis unit 25, and a counting unit 26. The determination unit 30 includes an emotion determination unit 31 and an emotion determination unit 32. The notification unit 40 includes a display unit 41 and a transmission unit 42. The pulse memory 24 can be constructed from a hard disk or semiconductor memory, the display unit 41 can be constructed from a liquid crystal display, and the timing unit 50 can be constructed from a known clock circuit. The other elements are implemented as software (programs) by a computer within the information terminal 5, which includes a CPU, ROM, and RAM.

[0069] The face recognition unit 21 uses a contour detection algorithm or a feature point extraction algorithm to analyze the facial shape of the subject HK in the image Gr captured by the imaging element 11, and identifies exposed skin areas such as the forehead as the measurement area. The face recognition unit 21 outputs the data representing the skin color of the measurement area, i.e., the timing signal E1, to the pulse extraction unit 22.

[0070] The pulse extraction unit 22 extracts the pulse signal of the subject HK from the timing signal E1 and outputs the signal to the interval detection unit 23. Since capillaries are concentrated within the measurement frame Sa on the forehead of the subject HK, the image Gr contains a brightness change component synchronized with the blood flow of the subject HK. In particular, since the pulse (blood flow change) is most reflected in the brightness change component of the green light in the image Gr, the pulse extraction unit 22 uses a bandpass filter that allows the pulse to pass through at a frequency of approximately 0.5 to 3 [Hz], and extracts the pulse signal based on the brightness change component of the green light in the timing signal E1.

[0071] The imaging unit 10, the facial recognition unit 21, and the pulse extraction unit 22 are examples of a detection unit that detects the heartbeat information of the subject. However, the function of the detection unit does not necessarily have to be divided into the imaging unit 10 and the information terminal 5. For example, the imaging unit 10 may have the functions of the facial recognition unit 21 and the pulse extraction unit 22, or the imaging unit 10 may be included in the information terminal 5.

[0072] Figure 3 Diagrams (A) through (C) illustrate the principle of determining negative emotions based on pulse. Among them, Figure 3 (A) represents the waveform of the pulse signal PW, where the horizontal axis t is time (milliseconds) and the vertical axis A is the intensity of the pulse amplitude. Figure 3 As shown in (A), the pulse signal PW is a triangular wave that reflects the change in blood flow caused by the heartbeat. The interval between the peak points P1 to P(n+1) representing the pulse with the strongest amplitude intensity, which indicates the pulse with the most blood flow, is set as the pulse interval d1 to dn.

[0073] The interval detection unit 23 detects the peak points P1 to P(n+1) of the pulse signal PW of the subject HK, and uses the timing unit 50 to calculate the pulse intervals d1 to dn in milliseconds, and then generates the pulse interval timing data based on the pulse intervals d1 to dn.

[0074] The pulse memory 24 stores the pulse intervals d1 to dn detected by the interval detection unit 23 as the timing data of the pulse intervals.

[0075] Figure 3 (B) is a graph representing an example of the fluctuation of the pulse interval. This graph is called a Lorenz scatter plot, with the pulse interval dn on the horizontal axis and the pulse interval dn-1 on the vertical axis (both in milliseconds). Time-series data of the pulse intervals are plotted on the coordinates (dn, dn-1) for n = 1, 2, ... . Since it is known... Figure 3 The degree of deviation of point R in the graph (B) reflects the brain fatigue level of the subject HK, so if it is displayed on display unit 41 Figure 3 The scatter plot of data (B) can also be used to easily monitor the brain fatigue level of the test subjects in the HK test.

[0076] The chaos analysis unit 25 uses the timing data of the pulse interval stored in the pulse memory 24, i.e. Figure 3 The coordinates (dn, dn-1) of the Lorentz scatter plot of (B) are used to calculate the maximum Lyapunov exponent λ according to the following formula (1).

[0077] [Formula 1]

[0078]

[0079] Here, M is the total sampling time in the pulse intervals d1 to dn, and d is the distance between the patterns at time k and time k-1 in the time series data (the distance on the two-dimensional plane in the Lorentz scatter plot). The interval detection unit 23 and the chaos analysis unit 25 are examples of calculation units that calculate the maximum Lyapunov exponent representing the volatility of the heartbeat interval based on the heartbeat information.

[0080] A brief explanation of the Maximum Lyapunov Index (MLI). If the heart of a mammal, including a human, operates cyclically like a machine, certain parts will tire and become prone to damage. To maintain continuous operation, fatigue is typically avoided due to fluctuations in the complex system. This is the function of the autonomic nervous system. If stress or psychological damage occurs here, the autonomic nervous system, focused on processing the stressful stimulus, cannot generate fluctuations in the complex system. Not only is the presence or absence of fluctuations in the complex system (chaos) related to the degree of these fluctuations, but the degree of such fluctuations is also associated with positive or negative emotions (autonomic nervous system). For example, fluctuations with a specific periodicity, represented by a circle or ellipse, do not involve fluctuations in the complex system. Therefore, a positive MLI indicates the presence of fluctuations in the complex system, suggesting positive emotions in the subject. Conversely, a negative MLI indicates the absence of fluctuations in the complex system, suggesting negative emotions in the subject. Thus, by using the MLI as an indicator, the degree of fluctuations in the complex system can be quantified, allowing determination of whether the subject experiences positive or negative emotions. This utilizes the principle of emotion sensing based on the maximum Lyapunov exponent.

[0081] Furthermore, the LF / HF method can be used instead of the maximum Lyapunov index when assessing emotions. The LF / HF method evaluates autonomic nervous activity based on heart rate fluctuations, using the ratio of low-frequency (LF) to high-frequency (HF) components (LF / HF) as a sympathetic nervous system indicator. When analyzing the frequency fluctuations of heartbeats or pulse intervals, with LF set to a power spectrum of 0.04–0.15 Hz and HF set to a power spectrum of 0.15–0.4 Hz, an LF / HF ratio less than 2.0 indicates a "positive emotion," a ratio greater than 2.0 but less than 5.0 indicates a "slightly negative emotion," and a ratio greater than 5.0 indicates a "negative emotion." However, while the maximum Lyapunov index can be analyzed using 30 seconds of pulse fluctuations, the LF / HF method requires approximately 3 minutes of measurement time to accurately measure the low-frequency component of LF.

[0082] Figure 3 (C) is a graph showing the relationship between the maximum Lyapunov index and negative feelings. The maximum Lyapunov index represents the variability of the heartbeat or pulse interval. This graph was created by conducting a questionnaire survey on 10 adult men and women, asking them to indicate the degree of fatigue they felt and whether the fatigue was related to mental fatigue, anxiety, or depression. The maximum Lyapunov index λ of the pulse interval was measured on the same subjects, and the relationship between the responses and the value of λ was summarized. F0 corresponds to "no fatigue," F1 to "age-appropriate fatigue," F2 to "temporary fatigue," F3 to "chronic fatigue," and F4 to "negative feelings." The vertical axis of the graph represents the maximum Lyapunov index λ.

[0083] from Figure 3 As shown in (C), the maximum Lyapunov index λ is a small absolute value close to 0 under simple fatigue, but becomes a large absolute value when there is negative emotion. For these 10 adult men and women, considering the measurement bias, the threshold for whether negative emotion is felt can be set to about -0.6.

[0084] If the maximum Lyapunov exponent λ obtained from the chaos analysis unit 25 satisfies the following equation (2), the emotion determination unit 31 determines that the subject has generated a negative emotion; if λ does not satisfy equation (2), it determines that the subject has not generated a negative emotion.

[0085] λ≤λt · · · (2)

[0086] Here, the threshold λt is -0.6, but sometimes other values ​​are used based on the characteristics obtained in the emotion determination device 1. The emotion determination unit 31 is an example of an emotion determination unit that determines, based on the maximum Lyapunov index, whether the subject's emotion is a negative emotion of feeling at least one of mental fatigue, anxiety, and depression, or a positive emotion of not feeling any of mental fatigue, anxiety, and depression.

[0087] The counting unit 26 counts the number of heart rate changes during a specified period, which occur as the heart rate increases above a specified value, and the heart rate changes from a state below the average heart rate to a state above the average heart rate. Figure 4 Charts (A) through (C) illustrate the principle of judging emotions based on changes in heart rate over a specified period. The vertical axis represents heart rate, and the horizontal axis represents time. ave This represents the average heart rate (HK) of the subject. When the average heart rate (HK) of the subject is unknown, an average of 65 bpm can be used. The heart rate is expressed as the average heart rate (b). ave The above area is called the "emotional zone," where the heart rate is lower than the average heart rate (b). ave The area is called the "quiet zone".

[0088] like Figure 4 As shown in (A) to (C), the heart rate varies over time. The counting unit 26 tracks the heart rate during a specified period, transitioning from a resting zone to an emotional zone, i.e., from a heart rate less than the average heart rate b. ave State towards average heart rate b ave The number of heart rate changes described above, where the heart rate fluctuation Δs is greater than a specified value, is counted. For example, the specified period can be set to 30 seconds and the specified value to 10 bpm. The specified period of 30 seconds is consistent with the minimum period required to maintain the accuracy of Lyapunov analysis. However, it is not limited to these values; the specified period and specified value can be appropriately set according to the degree of heart rate change in the subject accompanying the emotional change.

[0089] Figure 4 (A) indicates that the heart rate during the specified period is less than the average heart rate b. ave For example, in this case, the curve L1 representing the heart rate is in the quiet region during the specified period, and the counting unit 26 counts the number of heart rate changes from a state below the average heart rate to a state above the average heart rate as 0.

[0090] Figure 4 (B) represents an example where the heart rate changes dramatically only once. For example... Figure 4 As shown in (B), in curve L2 representing heart rate, heart rate s1 is less than the average heart rate b. ave Heart rate s2 is the average heart rate b aveTherefore, when the difference between s1 and s2, i.e. Δs, is a predetermined value or higher, the counting unit 26 counts the number of heart rate changes that occur during a predetermined period as the heart rate increases from a state below the average heart rate to a state above the average heart rate.

[0091] Figure 4 (C) represents an example of multiple heart rate fluctuations. For example... Figure 4 As shown in (C), in curve L3 representing heart rate, heart rates s1, s3, and s5 are less than the average heart rate b. ave Heart rates s2, s4, and s6 represent the average heart rate b. ave The above. Furthermore, when the differences between s1 and s2 (Δs1), s3 and s4 (Δs2), and s5 and s6 (Δs3) are all above a predetermined value, the counting unit 26 counts the number of heart rate changes from a state below the average heart rate to a state above the average heart rate as the heart rate increases above the predetermined value within a predetermined period, as 3 times. Figure 4 In (C), an example of three heart rate fluctuations is shown, but it is not limited to such an example. In the case of two or four or more heart rate fluctuations, the counting unit 26 also determines that the heart rate fluctuations are multiple.

[0092] The emotion determination unit 32 determines the psychological state of the subject HK based on the determination result of the emotion determination unit 31 and the number of heart rate changes counted by the counting unit 26. Figure 4 In sections (A) to (C), only the initially set period of 30 seconds is shown as the prescribed period, but the prescribed period can be set repeatedly. Furthermore, the emotion determination unit can make a determination during each prescribed period. Here, the psychological state of the test subject HK includes a stable state, a surprised state, a grateful state, and an angry state. Therefore, the emotion determination unit 32 determines whether the psychological state of the test subject HK is a stable state, a surprised state, a grateful state, or an angry state based on the determination result of the emotion determination unit 31 and the number of heart rate fluctuations counted by the counting unit 26. Furthermore, "gratitude" refers to a strong inner feeling and emotional excitement, including "being moved" (being touched), "remembering" (being moved by an impact), "admiration" (being surprised or agreeing), "joy," and other synonyms. Additionally, "anger" means indignation, including "being angry" and other synonyms.

[0093] If the heart rate fluctuation counted by the counting unit 26 is 0 times, and the subject HK's heart rate remains below the average heart rate during the specified period, then the emotion determination unit 32 determines that the subject HK's psychological state is stable. Since the heart rate is below the average heart rate during the specified period and never exceeds the average heart rate, the heart rate is in the resting range, and the emotion determination unit 32 can determine that the subject HK's psychological state is stable.

[0094] If the heart rate fluctuation counted by the counting unit 26 is one, the emotion determination unit 32 determines that the psychological state is one of surprise. When the test subject HK's psychological state is one of surprise, it is assumed that the heart rate fluctuated only once within the specified period and was not continuous. Furthermore, the test subject HK's state of surprise can occur even if the test subject HK's emotion is positive or negative. Therefore, if the heart rate fluctuation counted by the counting unit 26 is one, regardless of whether the emotion determined by the emotion determination unit 31 is positive or negative, the emotion determination unit 32 can determine that the test subject HK's psychological state is one of surprise.

[0095] If the heart rate fluctuations counted by the counting unit 26 are multiple, and the emotion determination unit 31 determines that the subject HK's emotion is positive, the emotion determination unit 32 determines that the subject HK's psychological state is one of gratitude. If the subject HK's emotion is positive, the subject HK is considered to be in a pleasant psychological state. Furthermore, if the subject HK is enjoying something interesting, such as an image, and experiences a state of gratitude like laughter, the heart rate is considered to fluctuate from the lower-than-average resting zone to the higher-than-average emotional zone, and this fluctuation is considered to persist to some extent. Therefore, if the heart rate fluctuations counted by the counting unit 26 are multiple, and the emotion determination unit 31 determines that the subject HK's emotion is positive, the emotion determination unit 32 can determine that the subject HK's psychological state is one of gratitude.

[0096] If the heart rate fluctuations counted by the counting unit 26 are multiple, and the emotion determination unit 31 determines that the subject HK's emotion is negative, the emotion determination unit 32 determines that the subject HK's psychological state is anger. When the subject HK's emotion is negative, it is considered that the subject HK is in an unpleasant psychological state. Furthermore, if the subject HK is in an angry psychological state, such as being angered by someone they dislike, it is considered that the heart rate fluctuates from the lower-than-average resting zone to the higher-than-average emotional zone, and this fluctuation persists to some extent. Therefore, if the heart rate fluctuations counted by the counting unit 26 are multiple, and the emotion determination unit 31 determines that the subject HK's emotion is negative, the emotion determination unit 32 can determine that the subject HK's psychological state is anger.

[0097] If the heart rate fluctuation counted by the counting unit 26 is 0 times, and the heart rate of the subject HK is maintained above the average heart rate during the specified period, the emotion determination unit 32 determines that the psychological state of the subject HK cannot be determined and the emotion cannot be determined. Figure 5 (A) and (B) are charts illustrating examples of situations where it is difficult to determine emotions based on changes in heart rate over a specified period. Figure 5 (A) indicates that the L4 curve, which represents the heart rate, is in a stable emotional state during the specified period. Figure 5 (B) indicates a state where the heart rate curve L5 fluctuates significantly within the emotional zone during the specified period. However, in these cases, the heart rate remains above the average heart rate throughout the specified period, even though it is within the emotional zone. This is considered to indicate that the subject HK is in a state of fatigue or has engaged in strenuous exercise, making it difficult to properly determine the emotion. Therefore, if the counting unit 26 counts 0 heart rate fluctuations and the subject HK's heart rate remains above the average heart rate during the specified period, the emotion determination unit 32 can determine that the subject HK's psychological state is in a state of emotional indeterminability.

[0098] Table 1 shows a summary of the psychological state of the subject HK as determined by the emotion determination unit 32 based on the emotion determination unit 31's determination of the subject HK's emotion and the number of heart rate changes counted by the counting unit 26.

[0099] [Table 1]

[0100]

[0101] As described above, when the heart rate fluctuation is 0, regardless of whether the emotion determination unit 31 determines a positive or negative emotion, the emotion determination unit 32 will determine the subject's psychological state as "stable." When the heart rate fluctuation is 1, regardless of whether the emotion determination unit 31 determines a positive or negative emotion, the emotion determination unit 32 will determine the subject's psychological state as "surprised." Thus, whether the heart rate fluctuation is 0 or 1, the emotion determination unit 32 can determine the subject's psychological state based on the heart rate fluctuation, regardless of whether the emotion determination result is positive or negative. Here, the psychological state in which both positive and negative emotions can be determined based on the heart rate fluctuation is defined as the "first psychological state."

[0102] On the other hand, when the heart rate fluctuates multiple times, if the emotion determination unit 31 determines the emotion as positive, the emotion determination unit 32 determines the test subject's psychological state as "gratitude," and if the emotion determination unit 31 determines the emotion as negative, the emotion determination unit 32 determines the test subject's psychological state as "anger." Thus, when the heart rate fluctuates multiple times, the emotion determination unit 32 can determine the test subject's psychological state based on either a positive or negative emotion (the result of the emotion determination) and the heart rate fluctuation count. Here, the psychological state that can be determined based on either a positive or negative emotion and the heart rate fluctuation count is defined as the "second psychological state." Thus, the test subject's psychological state includes a first psychological state and a second psychological state. When determining whether the second psychological state is "gratitude" or "anger" when the heart rate fluctuates multiple times, the emotion determination unit 32 uses the determination result of whether the test subject's emotion is positive or negative, as determined by the emotion determination unit 31. That is, the emotion determination unit 32 uses the determination result of the emotion determination unit 31 at least when determining the second psychological state.

[0103] In this way, the emotion determination unit 32 can correspond to the psychological state to be determined and use the determination result of the emotion determination unit 31 to determine the psychological state.

[0104] However, even when the heart rate fluctuation is 0 or 1, the emotion determination unit 32 can use the determination result of the emotion determination unit 31 to determine the psychological state (e.g., a more detailed psychological state). In this case, the emotion determination unit 32 can also use the determination result of the emotion determination unit 31 to determine the psychological state.

[0105] The notification unit 40 causes the display unit 41 to display the emotion determination result of the emotion determination unit 32 on the subject HK's feelings. Specifically, the notification unit 40 displays the emotion determination result of the emotion determination unit 32 on the display unit 41 and sends it externally via the transmission unit 42. The notification unit 40 is an example of an output unit that outputs the determination result of the emotion determination unit 32.

[0106] Figure 6 This is a flowchart illustrating the operation of the emotion determination device 1. First, in step S101, the imaging unit 10 captures an image Gr of the measurement frame Sa of the subject HK using the imaging element 11, and sends the image data to the information terminal 5. Next, the face recognition unit 21 determines the measurement location based on the image data of the subject HK.

[0107] Next, in step S102, the pulse extraction unit 22 extracts the pulse signal of the subject HK from the time-series signal E1 of the skin color at the measurement site determined by the face recognition unit 21. The interval detection unit 23 calculates the pulse interval based on the pulse signal, generates the time-series data, and stores the time-series data in the pulse memory 24.

[0108] Next, in step S103, the chaos analysis unit 25 calculates the maximum Lyapunov exponent λ of the pulse interval based on the temporal data of the pulse interval stored in S102.

[0109] Next, in step S104, the counting unit 26 determines whether there is a heart rate fluctuation exceeding a predetermined value within a specified period based on the time change of heart rate extracted from the pulse signal. If there is no heart rate fluctuation exceeding the predetermined value, then in step S105, it is determined whether the heart rate of the subject HK during the specified period is less than the average heart rate.

[0110] When the test subject HK's heart rate remains below the average heart rate for the specified period, in step S106, the emotion determination unit 32 determines that the test subject HK's emotion is stable. On the other hand, when the test subject HK's heart rate remains above the average heart rate for the specified period, in step S107, the emotion determination unit 32 determines that the emotion determination of the test subject HK cannot be performed, indicating an emotion determination inability state.

[0111] When it is determined in step S104 that there is a heart rate fluctuation exceeding a specified value, in step S108, the counting unit 26 determines whether the number of heart rate fluctuations of the subject HK during the specified period is only one. If the number of heart rate fluctuations of the subject HK is only one, in step S109, the emotion determination unit 32 determines that the subject HK's emotion is surprise.

[0112] On the other hand, in step S108, if the number of heart rate changes of the subject HK during the specified period is not just once, but multiple times, in step S110, the emotion determination unit 31 compares the maximum Lyapunov index λ calculated in step S103 with the threshold λt to determine whether the subject HK has a positive or negative emotion.

[0113] If the test subject HK is determined to have a positive emotion, in step S111, the emotion determination unit 32 determines that the test subject HK's emotion is gratitude. On the other hand, if the test subject HK is determined to have a negative emotion, in step S112, the emotion determination unit 32 determines that the test subject HK's emotion is anger.

[0114] As described above, the emotion determination device according to the embodiments of the present disclosure can determine, in a non-contact manner, whether the psychological state of the subject HK is a stable state, a surprised state, a grateful state, or an angry state, based on the image information of the subject HK.

[0115] By using the emotion determination device of the present disclosure, it is possible to understand the emotions of people who cannot communicate well in care facilities. Specifically, it is possible to determine whether an elderly person is happy (grateful) or angry (angry) during caregiving. Therefore, caregivers can decide what kind of caregiving behavior to provide to the elderly person based on the determination result. Furthermore, the emotional rhythms of elderly subjects under the same behavior at the same time each day can be stored, allowing observation of the emotional rhythms under the same behavior at the same time. By understanding the emotional rhythms under the same behavior at the same time, changes in the subject's physical condition can be indirectly understood. Furthermore, by assembling the emotion determination device of the present disclosure into an automatic training device for the elderly, the automatic training device can provide appropriate suggestions and training to elderly individuals who lack emotional expression or facial expressions, based on the determination of their emotions.

[0116] [Example 1]

[0117] Next, the emotion determination device of Embodiment 1 will be described. Figure 7 This is a schematic diagram of a web conferencing system using an information terminal with the emotion determination device of Embodiment 1.

[0118] In video calls such as web conferencing, there is a problem that it is difficult to read the emotions of the other party from the facial expressions displayed on the screen. The emotion determination device in Embodiment 1 is a device that determines the other party's emotions based on the image information sent from the other party's side in a web conferencing or similar event.

[0119] A first emotion determination device 101, positioned before the first subject HK1, and a second emotion determination device 201, positioned before the second subject HK2, are connected via the Internet 100, enabling them to conduct web conferencing. The first emotion determination device 101 includes a first information terminal 105, a camera 111, and a microphone 112. The camera 111 captures an image of the first subject HK1 and transmits it, along with sound data collected by the microphone 112, via the Internet 100 to the second information terminal 205 of the second subject HK2. Specifically, the camera 111 captures an image of the measurement frame Sa1 of the first subject HK1, and the first information terminal 105 transmits the image captured by the camera 111 to the second information terminal 205. The image of the first subject HK1 and real-time data of the first subject HK1's heart rate b1 are displayed together on the display unit 241 of the second information terminal 205. The second information terminal 205 compares the heart rate b1 of the first subject HK1 with the average heart rate b1. ave1 By comparison, the emotion of the first subject HK1 can be determined based on the time change of heart rate b1 during a specified period. In addition, the emotion determination result (e.g., positive emotion) can be displayed in the emotion determination area 243 of the display unit 241, or the emotion determination result (e.g., stable state) can be displayed in the emotion display area 244.

[0120] Similarly, the second emotion determination device 201 includes a second information terminal 205, a camera 211, and a microphone 212. The camera 211 captures an image of the second subject HK2 and transmits it, along with sound data collected by the microphone 212, via the Internet 100 to the first information terminal 105 of the first subject HK1. Specifically, the camera 211 captures an image of the measurement frame Sa2 of the second subject HK2, and the second information terminal 205 transmits the image captured by the camera 211 to the first information terminal 105. The image of the second subject HK2 and real-time data of the second subject HK2's heart rate b2 are displayed together on the display unit 141 of the first information terminal 105. The first information terminal 105 compares the second subject HK2's heart rate b2 with the average heart rate b... ave2 By comparison, the emotions of the second subject HK2 can be determined based on the temporal changes in heart rate b2 over a specified period. Furthermore, the emotion determination result (e.g., positive emotion) can be displayed in the emotion determination area 143 of the display unit 141, or the emotion determination result (e.g., a state of gratitude) can be displayed in the emotion display area 144.

[0121] exist Figure 7 The example shown is a web conference using an emotion assessment device between two subjects, but the same applies to web conferences between more than two subjects.

[0122] Figure 8 This is a block diagram of the emotion determination device in Embodiment 1. The structures of the first information terminal 105 and the second information terminal 205, besides... Figure 2 In addition to the structure of the information terminal 5 in the embodiment shown, it also includes a receiving unit (191, 291), a sound reproduction unit (145, 245), a sound determination unit (127, 227), and a transmitting unit (192, 292).

[0123] The transmitting units (192, 292) transmit the image and sound information of the first subject HK1 or the second subject HK2, which are acquired by the camera (111, 211) and the microphone (112, 212), to the receiving unit 291 of the second information terminal 205 and the receiving unit 191 of the first information terminal 105, which are the counterparty terminals, via the Internet 100.

[0124] The receiving units (191, 291) respectively receive image and sound information of the second subject HK2 and the first subject HK1 transmitted from the transmitting unit 292 of the second information terminal 205 and the transmitting unit 192 of the first information terminal 105.

[0125] The sound reproduction units (145, 245) reproduce the sound data contained in the information received by the receiving units (191, 291). The sound reproduction units (145, 245) can use loudspeakers.

[0126] The sound determination unit (127, 227) determines, based on the duration of the sound information received by the receiving unit (191, 291), whether the first subject HK1 or the second subject HK2 is the speaker or the listener.

[0127] The speaker's emotions arise from the act of speaking itself, sometimes making accurate emotion assessment impossible. Therefore, if the speaker-listener relationship continues for a certain period (e.g., more than 10 seconds), the voice assessment unit (127, 227) can determine the listener's emotions and feelings, and the assessment results can be displayed in real-time on the speaker's information terminal. This allows the speaker to understand the listener's emotional state while listening to the speaker's conversation.

[0128] In the above embodiments, an example of analyzing emotions on the receiver's side based on image information transmitted from the sender's side is shown. The emotion determination device of Embodiment 1 calculates heart rate (pulse) based on the acquired image information, so the frame rate of the image directly becomes the sampling rate. Therefore, it is also possible to monitor the transmission rate of the received image and perform emotion determination only when the transmission rate is within an appropriate range.

[0129] Furthermore, in web conferences or similar events where multiple people participate simultaneously, the possibility of not being able to acquire the other party's image information at the appropriate timeframe due to limitations in communication line capacity must also be considered. In such cases, the emotion determination can be switched between being performed at the sending and receiving sides, depending on the transmission rate. Specifically, the following methods can be considered: a method where emotion is determined at the speaker's side after pre-acquired audience image information is sent to the speaker, rather than based on real-time transmitted image information (first method); and a method where the emotion determination result is sent from the audience's side to the speaker's side (second method). The first method is considered to transmit more image information. On the other hand, the second method requires adjusting the communication format used to add the determination result to the image information. Therefore, it is preferable to appropriately switch between the first and second methods based on the transmission rate.

[0130] Additionally, situations such as anger on the sending side can be considered, where directly conveying the emotional assessment would damage the recipient's image. In these cases, the color of the border on the sender's image can be changed to indirectly reflect the emotional assessment result. Besides changing the border color of the recipient's image, the colors of the curves in the measurement frames (Sa1, Sa2) or heart rate (b1, b2) graphs can also be changed. For example, the colors of the measurement frames (Sa1, Sa2) can be used to display emotions, and the colors of the heart rate (b1, b2) curves can be used to display feelings.

[0131] According to the emotion determination device of Embodiment 1, in web conferences and the like, it is possible to determine the emotion of the audience who hear the speaker's words.

[0132] [Example 2]

[0133] Next, the emotion determination device of Embodiment 2 will be described. Figure 9 This is a diagram showing an example of the display of the emotion determination device of Embodiment 2. The emotion determination device of Embodiment 2 determines the emotion of a moving image displayed on the moving image display device 300 using existing software such as dedicated web conferencing software, moving image playback application, or moving image player.

[0134] The emotion determination device in Example 2 differs from that in Example 1. Instead of using an imaging element to acquire images of the subject, it retrieves the displayed image from the video RAM (VRAM) 301, which is based on existing software, to perform emotion determination. The image of the subject HK retrieved from the video RAM 301 is displayed on the display unit 41 of the information terminal 5, and the time variation of heart rate b is compared with the average heart rate b based on the pulse in the measurement frame Sa. aveThey are displayed together on the display unit 41. Furthermore, the emotion determination result (e.g., negative emotion) can be displayed in the emotion determination area 43 of the display unit 41, or the emotion determination result (e.g., angry state) can be displayed in the emotion display area 44.

[0135] Figure 10 This is a block diagram of the emotion determination device 102 in Embodiment 2. The emotion determination device 102 includes a dynamic image display device 300 and an information terminal 5. The information terminal 5, in addition to... Figure 2 In addition to the structure of the information terminal 5 in the embodiment shown, it also has an image acquisition unit 28 and an audio reproduction unit 45 instead of a transmission unit.

[0136] The image acquisition unit 28 continuously and automatically captures multiple images of the measurement frame Sa of the subject HK. The image acquisition unit 28 has the function of automatically tracking the measurement frame Sa of the subject HK's forehead through a built-in facial recognition application. Therefore, even if the position of the measurement frame Sa of the subject HK moves within the display area of ​​the display unit 41, the pulse of the subject HK can be captured.

[0137] The sound reproduction unit 45 reproduces the sound data contained in the information acquired by the image acquisition unit 28. The sound reproduction unit 45 can use a speaker.

[0138] In the emotion determination device of Embodiment 1, an image of the subject is captured by a camera. However, in the emotion determination device of Embodiment 2, instead of acquiring a new image, the person in the existing image is used as the subject to determine their emotion. Therefore, for example, it is possible to determine the emotion of a person shown in an image uploaded to a moving image sharing service. Specifically, for example, if an image of a person's apology is uploaded to the moving image display device 300, the image data of that person is retrieved from the video RAM 301 and entered into the image acquisition unit 28 to perform emotion determination, thereby determining the person's emotion at the apology.

[0139] Alternatively, when displaying the image of the other party in a video call using the dynamic image display device 300, the image of the other party in the video call is retrieved from the video RAM 301 and imported into the image acquisition unit 28 for emotion determination, thereby enabling real-time determination of the other party's emotions during the video call.

[0140] Furthermore, when using the dynamic image display device 300 to display video software for movies, TV series, etc., the images of the actors appearing in the movies or TV series are captured from the video RAM 301 and input into the image acquisition unit 28 for emotion determination, thereby determining the emotions of the actors performing.

[0141] As described above, the emotion determination device according to Embodiment 2 can determine the speaker's emotion in apology meetings, etc., or can analyze the emotions of actors in TV dramas, etc., while appreciating moving images.

[0142] [Example 3]

[0143] Next, the emotion determination device of Embodiment 3 will be described. Figure 11 This is a schematic diagram of a marketing survey system using the emotion determination device of Embodiment 3. The emotion determination device 103 of Embodiment 3 includes an information terminal 5 and an imaging unit 10. The subject HK observes an image of a product displayed on a display 400. The imaging element 11 of the imaging unit 10 captures an image of the subject HK's measurement frame Sa, and the imaging unit 10 sends the image of the subject HK captured by the imaging element 11 to the information terminal 5. The information terminal 5 determines the subject HK's emotion when observing the product displayed on the display 400 and sends the determination result to a statistical analysis server 500 via the Internet 100. The statistical analysis server 500 can analyze the emotions of multiple subjects observing the same product and analyze the appeal effect of the product on the subjects as consumers.

[0144] Figure 12 This is a block diagram of the emotion determination device in Embodiment 3. The emotion determination result of the subject HK is sent from the sending unit 42 to the statistical analysis server 500 via the Internet 100, where the emotion determination result of the subject HK is analyzed. Therefore, it is not necessary to display the emotion determination result of the subject HK on the information terminal 5, so a display unit is not required.

[0145] The emotion determination device 103 of Embodiment 3 can be applied to marketing surveys utilizing digital signage, analyzing whether a product is viewed favorably based on the emotion determination results of the test subjects (consumers). For example, by combining information display devices such as digital signage with the emotion determination device 103 to display products, consumers' (test subjects') emotions towards a product can be observed in real time. If the emotion is positive and the frequency of the emotion is high, it can be determined that the product is liked by consumers, and it can be flexibly applied to marketing surveys.

[0146] Furthermore, since the sentiment determination device 103 can extract only the sentiment determination data of consumers (test subjects) towards a certain product, it can conduct marketing research without processing personal information.

[0147] [Example 4]

[0148] Next, the emotion determination device of Example 4 will be described. Figure 13This is a schematic diagram of the mechanical control system using the emotion determination device of Embodiment 4. The emotion determination device 104 of Embodiment 4 includes an information terminal 5 and an imaging unit 10. The information terminal 5 of the emotion determination device 104 determines the emotion of the subject HK performing the mechanical operation. Based on this result, the information terminal 5 sends a control signal to the machine 2, thereby enabling the subject HK to operate the machine safely. The imaging element 11 of the imaging unit 10, installed on the machine 2, acquires an image of the measurement frame Sa of the subject HK, who is operating the machine 2, and sends it to the information terminal 5. The information terminal 5 determines the emotion of the subject HK based on the received image data and sends a control signal to the receiving unit 220 of the machine 2 based on the determination result. The receiving unit 220 inputs the received control signal to the control unit 210, which controls the machine 2 according to the control signal. For example, if it is determined that the subject HK, as the operator, is in an angry state and the operator is unable to operate the machine 2 normally, the information terminal 5 can send a control signal to the machine 2 to force it to stop. In addition, if forcibly stopping the machine 2 may expose the subject HK to danger, a warning signal can be sent to help the subject HK's emotions return to a stable state.

[0149] Figure 14 This is a block diagram of the emotion determination device 104 of Embodiment 4. The information terminal 5 constituting the emotion determination device 104 of Embodiment 4 has a control signal generation unit 46 that generates signals for controlling the machine 2.

[0150] The control signal generation unit 46 generates a control signal for controlling the machine 2 based on the emotion determination result of the subject HK determined by the emotion determination unit 32. For example, if the subject HK operating the machine 2 is in a stable or grateful state, it can be determined that there is no problem continuing to operate the machine 2, so no control signal is generated for the machine 2. Alternatively, in this case, since it is determined that there is no problem for the subject HK to continue operating the machine 2, a signal to continue controlling the machine 2 can be generated.

[0151] On the other hand, if the emotion determination unit 32 determines that the subject HK is in an angry state, and if the operation of the machine 2 continues in this way, the safety of the operator (subject) may not be maintained, the control signal generation unit 46 generates a signal to forcibly stop the machine 2 or a signal to warn the operator based on the emotion determination result of the emotion determination unit 32.

[0152] The control signal generated by the control signal generation unit 46 is sent to the receiving unit 220 of the machine 2 via the transmitting unit 42. The receiving unit 220 of the machine 2 inputs the received control signal to the control unit 210, and the control unit 210 controls the machine 2 according to the control signal.

[0153] Alternatively, if the emotion determination unit 32 continues to determine that the operator's (subject's) emotion has changed from an angry state to a stable state, the control signal generation unit 46 can generate a signal to restart the operation of the machine 2 based on the emotion determination result of the emotion determination unit 32, and send it to the machine 2.

[0154] The above example illustrates how information terminal 5 sends control signals to machine 2 based on the emotional assessment results of the operator (subject), but it is not limited to this example. For instance, the emotional assessment results of the operator determined by information terminal 5 can also be sent to a management center that manages multiple machines 2, and control signals to the machines 2 can be sent from the management center. In this way, since the management center can grasp the emotional assessment results of the operator, it can also simultaneously conduct health management of the operator.

[0155] By using the emotion determination device of Embodiment 4, mechanical control can be performed based on the determination result of the operator's emotion, thereby ensuring the operator's safety. The emotion determination device can function as a so-called potential hazard sensor.

[0156] The embodiments described above illustrate an example of implementing an emotion determination device in an information terminal, but are not limited to such examples. For instance, when the operation of the emotion determination device is implemented on a control board and applied to digital signage, the control board can also be assembled in a display device. In this case, a camera is installed on the display device, and the result of the emotion determination performed on the control board is sent to a server for statistical analysis.

[0157] Furthermore, examples of using portable terminals such as smartphones as the imaging unit have been shown. However, in order to capture images of elderly people as subjects while providing care in nursing facilities, a gaze camera can also be used as the imaging unit to identify the faces of the elderly people and to determine their emotions based on images obtained while the caregiver observes them. Additionally, the results of the emotion determination can be communicated via voice. Specifically, for example, the results can be communicated via voice from the speaker of an earphone worn by the caregiver or a tablet terminal. When using a gaze camera, the gaze target becomes the object of measurement. Although situations where it may be difficult to confirm the results of the emotion determination on the gaze camera's screen are considered, even in such cases, the results of the emotion determination can be identified through voice. When using a gaze camera as the imaging element, although gaze cameras are small and portable, their processing power is sometimes insufficient. Therefore, the acquired images can also be wirelessly transmitted to other terminals for emotion determination at the receiving terminal.

[0158] Furthermore, the ability to notify the results of emotion determination via voice can also be implemented in mechanical control systems using other embodiments, particularly the emotion determination device of Embodiment 4. By notifying the results and warnings of emotion determination via voice, the results and warnings of emotion determination can be recognized by voice even when the machine operator is focused on the work and not looking at the control screen.

[0159] Alternatively, a camera mounted on the smartwatch can be used as the capturing element. Images captured by the smartwatch can be used for emotion determination within the smartwatch itself, or the captured images can be sent to other terminals for emotion determination.

[0160] [Example 5]

[0161] (Simultaneous measurement of multiple individuals)

[0162] In recent years, opportunities for web conferencing via the Internet and intranets have increased. Examples of web conferencing include companies demonstrating to multiple customers via the Internet, and teachers lecturing to multiple students via the Internet. In these cases, it is preferable that the company or teacher hosting the web conferencing has a good understanding of whether the multiple listeners (customers or students) are paying attention, or whether the demonstration or lecture is being conducted effectively. Since the listeners of a course are participants, the term "listeners" will be referred to as "participants" below.

[0163] However, even in the case of a seminar or similar event utilizing the Web, similar to the Web conference in Example 2, the lecturer does not directly visually recognize the faces of multiple participants, but rather observes their faces through the screen. Therefore, it is difficult for the seminar organizer to grasp the state of each participant's listening. Here, the objects that the organizer of a Web conference presents to the participants are sometimes referred to as "demonstration," "lecture," or "seminar," but in this specification, these will be referred to as "courses."

[0164] The purpose of the emotion determination device of Embodiment 5 of this disclosure is to determine the listening state of multiple participants, even when there are multiple participants. In contrast to the emotion determination device of Embodiment 1 of this disclosure, which performs emotion analysis on a single subject, the emotion determination device of Embodiment 5 of this disclosure is characterized by simultaneously performing emotion analysis on multiple subjects. Furthermore, the emotion determination device of Embodiment 5 of this disclosure is characterized by, as described later, calculating the optimal listening time representing the time a participant listens in an ideal state, and determining the listening state of the participant.

[0165] As an example, the emotion determination device of Embodiment 5 of this disclosure will be described using a scenario where a course organizer conducts a course with multiple participants. That is, the subject being tested is a listener (participant) of the course. Figure 15 This is a block diagram of the course convening PC 600, which is the emotion determination device according to Embodiment 5 of this disclosure. The course convening PC 600 transmits and receives data via the Internet 100 between terminals A (500a), B (500b), ..., N (500n) used by multiple participants A, B, ..., N to listen to the course.

[0166] Terminal A (500a) includes a camera 501, a microphone 502, a transceiver unit 503, and a display unit 504. The camera 501 captures a facial image of participant A. The microphone 502 collects the voice of participant A. The camera 501 and microphone 502 can be built into terminal A (500a) or externally mounted. The transceiver unit 503 sends and receives data via the Internet 100 with the course organizer's PC 600. The display unit 504 displays course-related information sent from the course organizer's PC 600. The display unit 504 can also display a facial image of the course organizer's lecturer, a facial image of participant A, etc. The structures of terminals B (500b) and N (500n), etc., are the same as those of terminal A (500a).

[0167] The course organizer's PC 600 has a material publishing unit 601, a receiving and sending unit 602, a face recognition unit for all participants 603, an image cropping unit for each participant 604, a participant emotion analysis unit (605a, 605b, ..., 605n), an individual log storage unit for each participant 611, and a display / notification unit 612.

[0168] The Materials Publishing Department 601 publishes images, videos, and other materials used by the instructors during the courses within a specified timeframe. Materials can be published in real-time by the instructors or by reproducing pre-prepared animated images.

[0169] The transceiver unit 602 receives data containing facial images of participants A, B, ..., N from terminals A (500a), B (500b), ..., N via the Internet 100. For example, in the case of four participants in the course... Figure 16 As shown, images of each participant (41a-41d) are displayed on the screen of the display / notification unit 612. Figure 16 This refers to a case where there are 4 participants, but it is not limited to this example.

[0170] The all-participant face recognition unit 603 recognizes the facial images of all participants. That is, the all-participant face recognition unit 603 is an example of a camera unit capable of capturing the faces of multiple subjects. Application software used for web conferencing of courses typically lacks the ability to identify where a participant's face is reflected on the screen, and it also doesn't know how many participants there are. For example, even if the terminals (500a, 500b, ..., 500n) are connected to the course-hosting PC 600, it cannot recognize whether a participant is listening without sending images of their faces. Therefore, the all-participant face recognition unit 603 captures the image displayed by the application software and first scans the approximate facial positions to determine the number of participants. Additionally, the all-participant face recognition unit 603 acquires the individual facial coordinates of each participant. The face recognition method will be described later.

[0171] Each participant's image extraction unit 604 extracts an image for pulse analysis from the acquired participant's image. This is because when there are multiple participants, the amount of data for image processing increases, making it difficult to process simultaneously in parallel; therefore, it is necessary to reduce the amount of data for image processing. Figure 17 This represents an example of cropping a portion of an image from a participant's facial image. For example, ... Figure 17 As shown in the upper left, the image cropping unit 604 of each participant crops a portion 41a' of the image 41a of participant A. Similarly, the image cropping unit 604 of each participant crops a portion (41b', 41c', 41d') of the images (41b, 41c, 41d) of the other participants B to D.

[0172] Participant A's emotion analysis unit 605a includes an individual facial recognition unit 606, a pulse extraction image processing unit 607, an RRI Lyapunov emotion determination unit 608, a pulse count emotion determination unit 609, and an optimal listening time determination unit 610.

[0173] The individual face recognition unit 606 of the participant A emotion analysis unit 605a performs facial recognition on participant A. Additionally, the individual face recognition unit 606 extracts a region from participant A's facial image 41a' for pulse extraction. For example, it extracts... Figure 17 A portion 412a of the image 41a' of participant A shown in the upper left corner. The cropped image 412a can, for example, be displayed in the upper left corner (412a') of the image 41a' of participant A. Similarly, the individual facial recognition units 606 of the emotion analysis units (605b, 605c, 605d) of other participants (B, C, D) perform facial recognition of the other participants (B, C, D) respectively. In addition, the individual facial recognition units 606 of the emotion analysis units (605b, 605c, 605d) of other participants (B, C, D) crop a region for pulse extraction from the facial images (41b', 41c', 41d') of other participants (B, C, D). For example, a portion (412b, 412c, 412d) of the images (41b', 41c', 41d') of other participants (B, C, D) is cropped. The cropped images (412b, 412c, 412d) can, for example, be displayed to the upper left (412b', 412c', 412d') of the images (41b', 41c', 41d') of other participants (B, C, D). The image cropping method will be described later. By cropping a specified portion (e.g., an image of the face) from the overall image of the participant, the amount of data processed during image processing can be reduced.

[0174] The pulse extraction image processing unit 607 is an example of a detection unit. It uses an image 412a of a predetermined range of the facial image 41a' of participant A, which is sliced ​​by the individual facial recognition unit 606, to detect heartbeat information containing the heart rate of participant A as the subject.

[0175] The pulse extraction image processing unit 607 also has the function of a counting unit as described in Embodiment 1, which counts the number of heart rate changes during a specified period as the heart rate increases from a state below the average heart rate to a state above the average heart rate.

[0176] The RRI Lyapunov emotion determination unit 608 is one example of an emotion determination unit. Based on heartbeat information detected by the pulse extraction image processing unit 607, it determines whether the emotion of participant A, the subject of the test, is negative or positive. RRI is an abbreviation for RR Interval (heartbeat interval), representing the interval between heartbeats and pulses. Furthermore, the RRI Lyapunov emotion determination unit 608 corresponds to the one in Embodiment 1. Figure 2 The system comprises five functional blocks: pulse extraction unit 22, interval detection unit 23, pulse memory unit 24, chaos analysis unit 25, and emotion determination unit 31. Generally, the highest performance of a lecture participant occurs when they listen with a "normal mindset" while maintaining a "moderate level of tension" psychologically. That is, a state where the participant feels no pressure is not necessarily optimal; a state with a certain degree of "moderate tension" is preferred. A state of "moderate tension" refers to a state where the participant experiences "slightly negative emotions," i.e., a state of "slightly unpleasant emotions." The RRI Lyapunov emotion determination unit 608 determines whether the participant experiences "slightly negative emotions," thus determining whether they experience "slightly unpleasant emotions," i.e., whether they experience "moderate tension."

[0177] Figure 18 This represents the relationship between the maximum Lyapunov index and the subjective exercise intensity (RPE). As an example, Figure 18 This is a graph showing the maximum Lyapunov index and heart rate difference for five subjects, based on their heart rate information at various levels of fatigue, to determine and plot the Relative Physical Exertion (RPE) at these points. RPE is a numerical index representing the subjective effort / fatigue level of training or exercise. To quantify RPE, the Burger score, representing effort, is typically used with numbers ranging from 6 to 20. Figure 18 There is a correlation between the maximum Lyapunov index and the RPE, and the RPE value can be calculated from the maximum Lyapunov index calculated by the RRI Lyapunov sentiment determination unit 608. For example, when the maximum Lyapunov index is 0, the RPE is 12, and when the maximum Lyapunov index is -0.7, the RPE is 15.

[0178] Figure 19The Borg rating indicates the level of effort exerted. In the Borg rating system, a RPE of 12-15 indicates "slightly strenuous," representing a state of "moderate tension" for the participant. Furthermore, an RPE below 5 indicates "most relaxed," and an RPE of 6-7 indicates "very relaxed," suggesting "drowsy" for the participant. An RPE of 8-9 indicates "quite relaxed," and an RPE of 10-11 indicates "relaxed," suggesting "enjoyable" for the participant. An RPE of 15-16 indicates "strenuous," an RPE of 17-18 indicates "quite strenuous," and an RPE of 19-20 indicates "very strenuous," suggesting "unpleasant" for the participant.

[0179] The pulse rate emotion determination unit 609 determines the participant's emotion based on whether the extracted pulse is within a specified range from the average resting pulse rate. Figure 20 This section describes an example of the change in pulse rate over time as the pulse moves from the mean resting pulse rate into a defined range. The "normal" state of the participant described above is considered to be a state where the pulse stabilizes within the defined range from the mean resting pulse rate. For example, if a participant's pulse, centered on the mean resting pulse rate, is within ±5 bpm, the participant can be considered to have a normal resting pulse rate. The mean resting pulse rate can be the average resting pulse rate of Japanese people, i.e., 65 bpm, or it can be the average resting pulse rate of each participant.

[0180] Here, the analysis of feelings and emotions is preferably carried out during the course. Alternatively, the analysis of emotions can be performed every 30 to 40 seconds, in a manner where, for example, the Lyapunov index used for feelings analysis is updated every 30 to 40 seconds.

[0181] The optimal listening time determination unit 610, an example of an emotion determination unit, determines the psychological state of the participant based on the heart rate fluctuation counts recorded by the pulse extraction image processing unit 607, and uses the determination result of the RRI Lyapunov emotion determination unit 608 (an emotion determination unit) for psychological state assessment. The optimal listening time determination unit 610, as an emotion determination unit, determines whether the participant is in an ideal psychological state for listening to the course based on the heart rate fluctuation counts and the determination result of the RRI Lyapunov emotion determination unit 608. Specifically, if the first condition (indicating the participant has "moderate tension" and "slightly unpleasant" positive / negative emotions) and the second condition (indicating the participant's "normal" mood and pulse within a specified range from the average pulse) are met, the participant is judged to be in an ideal psychological state. The time the participant spends listening to the course in this ideal psychological state is accumulated, and this accumulated time is taken as the optimal listening time. It is evident that the longer the optimal listening time, the longer the participant can listen to the course in an ideal state. The optimal listening time can be calculated separately for each participant. Alternatively, the average value can be calculated by dividing the sum of the optimal listening times of multiple participants by the number of participants. Regarding optimal listening time, if the lecturer's course content is of interest to the participants, the optimal listening time is considered to be longer, so optimal listening time can also be used as an indicator of the lecturer's ability.

[0182] The emotion analysis unit 605b of participant B and the emotion analysis unit 605n of participant N have the same structure as the emotion analysis unit 605a of participant A.

[0183] The individual participant log storage unit 611 stores the feelings and emotions of each participant in a chronological order, from the start time of the course to the end time or the time of departure. By referring to the logs, it is possible to know which time period of the course the participant listened attentively in an ideal state. Alternatively, by referring to the logs, it is possible to know at which point in the course the participant was drowsy, so that it is possible to know after the course which part of the course might have been explained in a way that might have made them sleepy.

[0184] The display / notification unit 612 is an example of an output unit, outputting the judgment results of the RRI Lyapunov emotion determination unit 608 and the pulse rate emotion determination unit 609. For example, as Figure 17 As shown, the positive and negative emotion determination results 411a to 411d of the RRI Lyapunov emotion determination unit 608 can be displayed in text below the facial images of multiple participants, and the emotion determination results 413a to 413d of the pulse rate emotion determination unit 609 can be displayed in quadrilateral frames around the facial images of multiple participants.

[0185] exist Figure 17 In the examples shown, based on the judgment results 411a-411d for positive and negative emotions, it indicates that participant A's emotion is "unpleasant," participant B's emotion is "slightly unpleasant," participant C's emotion is "pleasant," and participant D's emotion is "drowsy." Furthermore, when displaying these texts, the text color can also be different according to the positive or negative emotion. For example, since it is common to feel slightly nervous while listening to a lecture, "slightly negative emotion" can be represented by the safe color "green text," and "negative emotion" by the warning color "red text." Additionally, since extreme positive emotions can induce drowsiness, "drowsy" can be represented by "black text."

[0186] In addition, Figure 17 In the example shown, based on the emotion assessment results 413a-413d, it can be displayed whether the participant's pulse is within a specified range based on the average resting pulse rate, exceeds the upper limit of the specified range, or is below the lower limit of the specified range. For example, if participant A's pulse is within the resting pulse rate range (average pulse ± 5 bpm), the emotion assessment result 413a can be displayed in green. In this case, the course organizer can recognize that participant A is "normal" based on the green color of the emotion assessment result 413a. Similarly, if participant B's pulse is below the lower limit of the resting pulse rate range, the emotion assessment result 413b can be displayed in black. If participant C's pulse exceeds the upper limit of the resting pulse rate range, the emotion assessment result 413c can be displayed in red. And if participant D's pulse is within the resting pulse rate range, the emotion assessment result 413d can be displayed in green. The color of the boxes for these emotion judgment results 413a-413d can be a single color or an intermediate color corresponding to the pulse rate.

[0187] The method described above for displaying the results of positive and negative emotions and feelings is one example; other methods can also be used. For instance, facial markings corresponding to emotions can be used to display the results. Alternatively, words or numbers corresponding to pulse rates can be used to display the results. By referring to these results of positive and negative emotion and feeling assessments, course organizers can easily determine the state in which each participant is listening to the course.

[0188] Furthermore, by calculating the optimal listening time for each participant individually and providing feedback, participants can easily monitor their own listening status. Additionally, by calculating the optimal listening time for each participant, summing the optimal listening times of all participants, and dividing this sum by the average of all participants, the overall optimal listening time for the course can be calculated. Moreover, since the course is conducted over a predetermined period of 1 or 2 hours, the optimal listening time can also be divided by the course duration to obtain a standard value per unit of time. For example, for a 60-minute course, if the cumulative optimal listening time is 12 minutes, the optimal listening time relative to the overall course time can be calculated as 20%.

[0189] Figure 21 This is a flowchart illustrating the operation sequence of the emotion determination device, i.e., the course convening PC 600, used in Embodiment 5. First, in step S201, a procedure for performing emotion sensing is initiated to acquire the web conference screen. Here, the scenario of multiple participants attending a web conference will be explained. Images of multiple participants are displayed on the screen of the PC hosting the web conference. For example, web conference software can be used to display the faces of all participants on a monitor different from the monitor displaying only the data. In cases where data is shared during a web conference, sometimes the data is displayed on the entire screen, and the participants' faces are not visible. Without seeing the participants' expressions, even if the web conference host provides explanations, the participants' reactions cannot be observed, making it impossible to determine whether the participants understand the content. Therefore, web conference software sometimes has the function of adding a screen different from the PC displaying the data, effectively creating two screens. By utilizing this function, multiple participants' facial images can be displayed on one screen, and the shared image can be displayed on another. By displaying multiple participants side-by-side on one screen, the web conference can be conducted while simultaneously checking the expressions of multiple participants.

[0190] Next, in step S202, overall facial image recognition is performed, counting the number of participants' faces in the frame and calculating the coordinates of each face. As described above, a frame displaying all participants is captured, and overall facial image recognition is performed first. Here, since participants sometimes change positions within the frame, facial images of all participants are recognized within a specified period, such as approximately one minute. Here, recognizing facial images of all participants means counting the number of recognized facial images and calculating the coordinates of the recognized faces within the frame. As the coordinates of the facial images, the coordinates of the eyes and nose can be calculated, and the coordinates of a specified position in the facial image, such as the position between the eyebrows, are used as the standard position for each participant. In addition, when a participant's face is captured by a camera, the actual position of the face moves subtly. Therefore, by capturing the image within a specified period, such as one minute, a range of participants is defined in the XY coordinates within a specified number of pixels (e.g., 300 pixels) in each of the X and Y coordinates. Even if the coordinates of the facial images change within this range, they can be considered as the same person.

[0191] Next, in step S203, starting from the coordinates of each eyebrow space, images of the surrounding facial features are extracted from the overall image. For example, as shown... Figure 17 As shown, starting from the coordinate position between the eyebrows, a range that takes into account the changes in the position of the participant's face is cut out, such as a range of ±300 pixels in the XY direction centered on the coordinate position between the eyebrows. This cut-out image is used as the image for determining emotions and feelings.

[0192] Next, in step S204, the facial surrounding image of participant A is acquired and image recognition is performed.

[0193] Next, in step S205, it is determined whether facial images can be recognized in real time. If participant A's facial image can be recognized, in step S206, emotion / feeling determination is performed. In step S207, the optimal listening time is accumulated. Then, the process returns to step S205 to determine whether facial images can be recognized in real time. Here, the calculation of the maximum Lyapunov index used for emotion determination requires a specified period, for example, about 60 seconds. Since emotion determination is also performed, the optimal listening time is calculated, for example, every 60 seconds.

[0194] In step S205, if participant A's facial image cannot be recognized, in step S208, it is determined whether the facial image cannot be recognized within a certain time. In step S208, if the facial image can be recognized within the specified time, the process returns to step S205 to determine whether the facial image can be recognized in real time.

[0195] On the other hand, in step S208, if a facial image cannot be recognized within a specified time (e.g., 3 minutes), it can be determined that participant A has left the room. Therefore, in step S209, participant A is determined to have left the room, and logs of the best listening time, emotion judgment result, and mood judgment result are saved. Here, the situations in which participant A's facial image cannot be recognized include not only situations where participant A leaves in front of terminal A while maintaining the connection between terminal A, which is connected to the course organizer's PC for the web conference, but also situations where the connection between participant A's terminal A and the course organizer's PC is cut off, and situations where the course ends.

[0196] Next, in step S210, the calculation of the optimal listening time for participant A is completed. Steps S204 to S210 are the steps for calculating the optimal listening time for participant A, but the optimal listening time for other participants is also calculated in parallel. For example, for participant N, the optimal listening time for participant N is calculated by executing steps S211 to S217. The same applies to other participants such as participant B.

[0197] The above explanation illustrates an example of determining whether multiple participants are listening to the lecture or leaving the room, and calculating the optimal listening time during the leaving phase. However, it is not limited to this example. Alternatively, the optimal listening time can be calculated by considering the end of the web conference as a point when it is impossible to recognize the facial images of all participants.

[0198] (Facial image recognition)

[0199] Next, the method for recognizing the facial images of the test subjects will be explained. Figure 22 (A) represents the output example of the feature coordinate fitting method for extracting feature coordinates from a facial image. Figure 22 (B) represents the output example of how the position between the eyebrows is detected from the captured facial image.

[0200] Figure 22 Example (A) is as follows: The coordinate positions of eyebrows 701, eyes 702, bridge of nose 703, nose 704, mouth 705, and facial contour 706 are extracted from the subject's facial image, and the image cropped for emotion and mood determination is set as a defined area 710 of the cheek. This method is effective when the subject is not wearing a mask, etc., but when wearing a mask, etc., part of the mouth, nose, and facial contour is hidden, resulting in the inability to accurately measure the coordinate positions.

[0201] Therefore, as a method of cropping a specific region of the subject's facial image regardless of whether or not a mask is being worn, such as Figure 22As shown in (B), in the facial image 720 of the subject HK, image recognition based on depth training is performed on the image 721 of the eyes and facial contours, and a defined area 722 from the brow to the temple is used as the image for emotion and mood determination. The facial parts used for image recognition are not limited to the eyes and contours, but can also be other parts, but in order to perform image recognition even when wearing a mask 723, it is preferable to use parts not covered by the mask 723. In addition, the area used for emotion and mood determination is not limited to the area around the brow, but can also be the forehead, etc. Therefore, the course meeting side PC600, which is the emotion determination device, preferably has a measurement part determination unit, which determines each face from the screen displaying multiple people and determines the measurement part for each determined face. The image cropping unit 604 for each participant is an example of the measurement part determination unit. The pulse extraction image processing unit 607, which is the detection unit (see reference 607), is also included. Figure 15 It can obtain heart rate information based on changes in images of various facial measurement sites.

[0202] Here, when specific regions are extracted from images including eyes and facial contours in a facial image through deep training, the computational workload increases, which is considered to burden the processor of the PC600 on the course convening side, which controls the emotion determination device. Therefore, in order to reduce the processing load on the processor of the PC600 on the course convening side, speed can also be achieved by performing deep training-based image processing in the backend.

[0203] (Image acquired via HDMI)

[0204] When using web conferencing software that performs emotion assessment using images of the test subject, as described above, such applications are frequently used within intranets for business purposes. However, due to security concerns, emotion sensing cannot always be performed on the PC used for the web conferencing. For example, in emotion assessment programs, since image information is directly captured or the web conferencing screen is directly viewed, execution on a personal computer within the intranet may sometimes be flagged as a virus infection or unauthorized access. Here, HDMI output is provided as an image output method that is not restricted by the intranet. Therefore, HDMI image output can be output from the intranet via HDMI from the PC to a projector, allowing the facial image data of the test subject to be captured and used to perform emotion assessment on the internal personal computer.

[0205] Figure 23This illustrates a configuration example using HDMI output to perform image acquisition. Assume a web conferencing PC 620 is connected to an intranet. The HDMI output of the web conferencing PC 620 is input to the HDMI distribution unit 801. The HDMI distribution unit 801 distributes the HDMI signal input from the web conferencing PC 620 into "HDMI1" and "HDMI2" for output. HDMI1 is input to the HDMI input unit 802 and then to the emotion sensing PC 600 to perform emotion sensing. The emotion sensing PC 600 is not connected to the intranet (non-internal network). The emotion sensing PC 600 can perform emotion sensing without an intranet connection by utilizing the facial image data of the subject contained in the HDMI output. Meanwhile, HDMI2 is input to a projector to display the image from the web conferencing PC 620.

[0206] In the above example, an example was shown where web conference images were captured from the HDMI output of a web conferencing PC 620 (which is a PC on an intranet) to an emotion-sensing PC 600 (which is a PC not on an intranet), and emotion analysis was performed in the emotion-sensing PC 600. However, this example is not limited to this one. That is, emotion sensing can also be performed on the web conferencing PC 620 that is conducting the web conference using HDMI output. Furthermore, as an example of image output from a PC connected to the intranet to a PC not connected to the intranet, an example using HDMI output was illustrated, but this example is not limited to this one. That is, as an example of image output from a PC connected to the intranet to a PC not connected to the intranet, other output methods besides HDMI output can also be used.

[0207] [Example 6]

[0208] (Determining the causes of communication difficulties in the elderly)

[0209] Communication difficulties sometimes arise between older adults and caregivers. These difficulties are believed to stem from caregivers' inability to determine the cause of the communication impediment. Figure 24 Examples of reasons that hinder communication between older adults and caregivers and corresponding countermeasures are given. As for the reasons that hinder communication between older adults, they are considered to be reasons (1) to (3) in order of decreasing severity.

[0210] (1) The primary reason is that the elderly person dislikes the caregiver's personality, that is, the elderly person and the caregiver have poor compatibility. As a countermeasure to this primary reason, it is possible to consider changing the caregiver of the elderly person to another type of caregiver.

[0211] (2) The second cause is simply a decrease in vision / hearing, such as the inability to recognize colors due to decreased vision or the inability to hear sounds due to decreased hearing. As a countermeasure for this second cause, individual countermeasures are considered, such as using glasses to supplement vision, using hearing aids to supplement hearing, or communicating by using fingers.

[0212] (3) The third cause is due to mental illnesses such as dementia, where the elderly may be able to see objects and hear sounds but cannot understand their meaning. As a countermeasure for this third cause, professional medical intervention may be considered.

[0213] For example, regarding the second cause, when only a routine vision / hearing assessment is conducted, sometimes both the first and third causes are included, making it impossible to accurately determine the cause hindering communication. Therefore, in order to accurately distinguish between the first and third causes, it is possible to periodically provide the test subject with... Figure 25 The causes of the three categories of visual and auditory stimuli shown are inferred based on whether the emotions or positive and negative feelings change synchronously with the cycle.

[0214] To determine whether the primary cause, compatibility, is hindering communication, one could consider showing the elderly person images of specific caregivers as visual stimuli, or letting them hear the voices of specific caregivers as auditory stimuli. "Specific caregivers" refers, for example, caregivers who provide care for elderly people with communication difficulties.

[0215] To determine whether the second cause, namely vision / hearing, is hindering communication, one could consider showing the elderly images of only colors as visual stimuli, or letting them hear rhythmic sounds of specific frequencies as auditory stimuli.

[0216] To determine whether the third cause, namely mental illness, is a barrier to communication, one could consider showing the elderly visual images such as illusion paintings or frightening images without color stimulation, or letting them hear meaningful language as auditory stimulation.

[0217] Figure 26This is a block diagram of a subject state determination device 1000, which serves as a device for determining the cause of communication obstruction. The device for determining the cause of communication obstruction and the subject state determination device are examples of emotion determination devices. The subject state determination device 1000, as a device for determining the cause of communication obstruction, includes a display unit 1001, a periodic stimulus image generation unit 1002, a speaker or headset 1003, a periodic stimulus sound generation unit 1004, a stimulus type switching unit 1005, a camera 1006, a face recognition unit 1007, a pulse extraction image processing unit 1008, an RRI Lyapunov emotion determination unit 1009, a pulse count emotion determination unit 1010, a synchronicity analysis unit 1011, a cause determination unit 1012, and a determination result notification unit 1013.

[0218] The periodic stimulus image generation unit 1002 and the periodic stimulus sound generation unit 1004 are examples of stimulus generation units that repeatedly generate the same type of stimulus to the visual or auditory senses of the subject during multiple predetermined periods with rest intervals. The stimuli can be of three types: first stimulus, second stimulus, and third stimulus, as described later. For example, the first stimulus is repeatedly generated as the same type of stimulus. However, the repeatedly generated stimuli can be different as long as they are classified as the first stimulus.

[0219] Display unit 1001 displays the image generated by periodic stimulus image generation unit 1002, which serves as the image stimulus for the elderly person being tested. Display unit 1001 can be a liquid crystal display device, an organic EL display device, or a projector, etc.

[0220] The speaker or headphones 1003 outputs the sound generated by the periodic stimulus sound generating unit 1004, which is a sound stimulus for the elderly.

[0221] The stimulation type switching unit 1005 switches the stimulation given to the elderly person as the test subject to visual stimulation or auditory stimulation.

[0222] Camera 1006 captures facial images of elderly people who are the subjects of the test.

[0223] The facial recognition unit 1007 recognizes facial images from images captured by the camera 1006.

[0224] The pulse extraction image processing unit 1008 is an example of a detection unit that detects heartbeat information, including the heart rate of an elderly person who is the subject of the test.

[0225] The RRI Lyapunov Emotion Judgment Unit 1009 is an example of a counting unit that counts the number of heart rate changes that occur at predetermined intervals, as the heart rate increases above a predetermined value, and the heart rate changes from a state below the average heart rate to a state above the average heart rate.

[0226] The pulse rate emotion determination unit 1010 is an example of an emotion determination unit. It determines the presence or absence of the subject's emotion corresponding to the stimulus at regular intervals based on the number of heart rate changes.

[0227] The synchronicity analysis unit 1011 analyzes whether there is synchronicity between multiple periods and the timing of the presence or absence of emotion determined by the emotion determination unit.

[0228] The cause determination unit 1012 is an example of an obstacle factor determination unit. For the test subject, the determination of obstacles to communication with other people is made based on the analysis results of the synchronicity analysis unit 1011.

[0229] The judgment result notification unit 1013 outputs the judgment result of the communication obstruction factors determined by the cause judgment unit 1012.

[0230] The stimuli generated by the periodic stimulus image generation unit 1002 and the periodic stimulus sound generation unit 1004, which are stimulus generation units, are first stimuli that can be sensorily identified through vision or hearing. The cause determination unit 1012, which is an obstacle factor determination unit, determines whether there is an abnormality in vision or hearing. The first stimulus can be a sensory stimulus that stimulates vision or hearing to identify the occurrence of the stimulus. Here, "sensory stimulus" includes stimuli that can be sensorily identified through vision or hearing. Furthermore, as an example of a stimulus that includes an image or sound, it can also be a moving image.

[0231] This will be illustrated with an example where the first stimulus is one that can be visually recognized. Figure 27 This example illustrates the temporal changes in the emotions of an elderly subject who periodically observed color stimuli such as red, which are the first stimuli that can be visually recognized. First, a display unit 1001 is configured such that the elderly subject can visually recognize the image. During a predetermined period (e.g., 30 seconds) from time t1 to t2, an image 901 (e.g., red) is displayed on the display unit 1001. Next, during a predetermined period (e.g., 30 seconds) from time t2 to t3, a white image 902 (without color stimulation) is displayed on the display unit 1001. After time t3, images containing color stimulation and images without color stimulation (no stimulation) are alternately and repeatedly displayed on the display unit 1001. This switching between color stimulation and no stimulation is performed by a periodic stimulation image generation unit 1002.

[0232] Camera 1006 was used to capture facial images of elderly subjects who were given color stimulation and those who were not.

[0233] The pulse extraction image processing unit 1008 is an example of a detection unit. It detects heartbeat information, including the heart rate of the elderly person, from an image of an elderly person as the subject captured by a camera 1006.

[0234] RRI Lyapunov Emotion Determination Unit 1009 is an example of an emotion determination unit that determines whether an elderly person being tested has a negative or positive emotion based on heart rate information.

[0235] The pulse extraction image processing unit 1008 also functions as a counting unit, counting the number of heart rate changes during a specified period as the heart rate increases above a specified value, transitioning from a state below the average heart rate to a state above the average heart rate.

[0236] The pulse rate function of the emotion determination unit 1010 determines the psychological state of the subject, such as the presence or absence of emotions, based on the number of heart rate fluctuations.

[0237] The synchronicity analysis unit 1011 analyzes whether there is synchronicity between multiple periods of color stimulation and no stimulation and the timing of the presence or absence of emotion determined by the emotion determination unit.

[0238] The cause determination unit 1012, which acts as an obstacle factor determination unit, determines whether there is a visual abnormality based on whether the timing of the color stimulus and the timing of the emotion generated are synchronized. For example, in Figure 27 In the example shown, the elderly person's emotion was detected from time t1 to t2 when image 901 containing color stimuli was displayed, but not from time t2 to t3 when a white image 902 without color stimuli was displayed. That is, the emotion was detected consistent with the timing of displaying image 901 containing color stimuli. In this case, it can be determined that the elderly person, as the subject, recognized the color, and it can be concluded that there was no visual abnormality.

[0239] Next, examples will be given of situations where the first stimulus is one that can be recognized auditorily. Figure 28 This example illustrates the temporal changes in the emotions of an elderly subject when periodically hearing a beat at a predetermined frequency (e.g., 500 Hz) as the first stimulus that can be recognized through auditory perception. First, during a predetermined period (e.g., 30 seconds) from time t1 to t2, a 500 Hz beat is output from a speaker or headphones 1003. Then, during a predetermined period (e.g., 30 seconds) from time t2 to t3, no sound is output from the speaker or headphones 1003 (silence). After time t3, the output of the 500 Hz beat and silence are alternately repeated. This switching between the 500 Hz beat and silence is performed by a periodic stimulus sound generation unit 1004.

[0240] Camera 1006 was used to capture facial images of elderly people who were being tested while being subjected to a 500Hz beat and were silent.

[0241] The pulse extraction image processing unit 1008 is an example of a detection unit. It detects heartbeat information, including the heart rate of the elderly person, from an image of an elderly person as the subject captured by a camera 1006.

[0242] RRI Lyapunov Emotion Determination Unit 1009 is an example of an emotion determination unit that determines whether an elderly person being tested has a negative or positive emotion based on heart rate information.

[0243] The pulse extraction image processing unit 1008 also functions as a counting unit, counting the number of heart rate changes during a specified period as the heart rate increases above a specified value, transitioning from a state below the average heart rate to a state above the average heart rate.

[0244] The pulse rate function of the emotion determination unit 1010 determines the psychological state of the subject, such as the presence or absence of emotions, based on the number of heart rate fluctuations.

[0245] The synchronicity analysis unit 1011 analyzes whether there is synchronicity between multiple periods of sound stimulation and no stimulation and the timing of the presence or absence of emotion determined by the emotion determination unit.

[0246] The cause determination unit 1012, which is the obstacle factor determination unit, determines whether there is an auditory abnormality based on whether the timing of the output of the 500Hz beat tone is synchronized with the timing of the emotion generated. For example, in Figure 28 In the example shown, the elderly person's emotion was detected from time t1 to t2 when a 500Hz beat was output, but not from time t2 to t3 when no sound was produced. That is, the emotion was detected precisely with the timing of the 500Hz beat output. In this case, it can be determined that the elderly person, as the subject, recognized the sound, and therefore, their hearing is not abnormal.

[0247] In the subject state determination device 1000 described above, the stimulus can be a first stimulus that can be identified solely through visual or auditory sensation. That is, the first stimulus is one that can be identified solely through visual or auditory sensation (sensory perception) without needing to recognize the information contained within it. The first stimulus can also contain some information, and the presence or absence of the stimulus can be sensorily recognized to generate emotions, and emotions can be generated by recognizing the information contained within it, or sometimes emotions can be generated by the stimulus.

[0248] A stimulus is a second stimulus containing prescribed information that enables the understanding of information provided by sight or hearing. The obstacle factor determination unit can determine whether the test subject has any mental abnormalities. The second stimulus can be a cognitive stimulus accompanying the cognition of the prescribed information. For example, as a second stimulus, an image like an optical illusion can be used—even if the colors themselves do not provide a stimulus, images that are easily responded to when recognizing normally impossible shapes hidden within the optical illusion and understanding its meaning. Alternatively, as a second stimulus, sounds containing materials that are easily responded to when understanding the meaning of language can be used, even if the sounds themselves do not provide a stimulus.

[0249] First, we will explain the situation where images containing prescribed information that can be understood are used as a second stimulus. Figure 29 The example illustrates the temporal changes in the emotions of an elderly subject who periodically viewed an illusionary painting—a second stimulus containing prescribed information that could be understood visually—when they were presented with such an illusionary painting that would elicit surprise or laughter if the meaning could be understood. First, during a prescribed period from time t1 to t2 (e.g., 30 seconds), an image 903 showing an elephant's ear in the shape of a human profile is displayed on display unit 1001. Next, during a prescribed period from time t2 to t3 (e.g., 30 seconds), a white image 902 without the illusionary painting is displayed on display unit 1001. Next, during a prescribed period from time t3 to t4 (e.g., 30 seconds), an image 904 showing a hand that should be holding a can in the right hand becoming a left hand is displayed on display unit 1001. Finally, during a prescribed period from t4 to t5 (e.g., 30 seconds), a white image 902 without the illusionary painting is displayed on display unit 1001. Next, for a predetermined period (e.g., 30 seconds) starting from time t5, an image 905 is displayed on display unit 1001 showing an image 905 in which the shape of the boundary of the color intensity displayed on the ground is the shape of a human profile. The switching between the image containing the illustrative image and the image not containing the illustrative image is performed by the periodic stimulus image generation unit 1002.

[0250] Camera 1006 was used to capture facial images of elderly subjects who saw both images containing and without optical illusions.

[0251] The pulse extraction image processing unit 1008 is an example of a detection unit. It detects heartbeat information, including the heart rate of the elderly person, from an image of an elderly person as the subject captured by a camera 1006.

[0252] RRI Lyapunov Emotion Determination Unit 1009 is an example of an emotion determination unit that determines whether an elderly person being tested has a negative or positive emotion based on heart rate information.

[0253] The pulse extraction image processing unit 1008 also functions as a counting unit, counting the number of heart rate changes during a specified period as the heart rate increases above a specified value, transitioning from a state below the average heart rate to a state above the average heart rate.

[0254] The pulse rate function of the emotion determination unit 1010 determines the psychological state of the subject, such as the presence or absence of emotions, based on the number of heart rate fluctuations.

[0255] The synchronization analysis unit 1011 analyzes whether there is synchronization between the multiple periods of displaying images containing and not containing illustrative images and the timing of the presence or absence of emotions determined by the emotion determination unit.

[0256] The cause determination unit 1012, which acts as an obstacle factor determination unit, determines whether there is a mental abnormality based on whether the timing of displaying the image containing the illusion and the timing of generating emotions are synchronized. For example, in Figure 29 In the example shown, emotion was detected when images 903, 904, and 905 containing the illusory image were displayed, but no emotion was detected when image 902, which does not contain the illusory image, was displayed. That is, emotion was detected at the same timing as the display of images 903, 904, and 905 containing the illusory image. In this case, it can be determined that the elderly person being tested recognized the meaning of the illusory image, and therefore, it can be determined that there is no mental abnormality.

[0257] Furthermore, since there are cases where the meaning of images like illustrative paintings is difficult to understand, when displaying multiple illustrative paintings, it is not limited to detecting emotions in all of them. Even if no emotion is detected in some illustrative paintings but in others, it can be determined that there is no mental abnormality. Therefore, as a second stimulus, multiple images with different content can be selected from images containing prescribed information that can be understood.

[0258] Next, regarding the second stimulus, the case involves using sound containing prescribed information that can be understood. Figure 30The example illustrates the temporal changes in the emotions of an elderly subject who periodically hears a sound that, if understood, would elicit surprise or laughter, as a second stimulus containing prescribed information that can be understood through hearing. First, during a prescribed period from time t1 to t2 (e.g., 30 seconds), the speaker or headphones 1003 outputs one or more sounds indicating a gas leak. Next, during a prescribed period from time t2 to t3 (e.g., 30 seconds), no sound is output (silence). Then, during a prescribed period from time t3 to t4 (e.g., 30 seconds), the speaker or headphones 1003 outputs one or more sounds indicating a fire. Next, during a prescribed period from time t4 to t5 (e.g., 30 seconds), no sound is output (silence). Then, from time t5 onwards, after a prescribed period (e.g., 30 seconds), the speaker or headphones 1003 outputs one or more sounds indicating an earthquake. The output of sound and the switching between silence are performed by the periodic stimulation image generation unit 1002.

[0259] The camera 1006 was used to capture facial images of elderly subjects, both when the camera output a sound with a specific meaning and when it did not output a sound.

[0260] The pulse extraction image processing unit 1008 is an example of a detection unit. It detects heartbeat information, including the heart rate of the elderly person, from an image of an elderly person as the subject captured by the camera 1006.

[0261] RRI Lyapunov Emotion Determination Unit 1009 is an example of an emotion determination unit that determines whether an elderly person being tested has a negative or positive emotion based on heart rate information.

[0262] The pulse extraction image processing unit 1008 also functions as a counting unit, counting the number of heart rate changes during a specified period as the heart rate increases above a specified value, transitioning from a state below the average heart rate to a state above the average heart rate.

[0263] The pulse rate function of the emotion determination unit 1010 determines the psychological state of the subject, such as the presence or absence of emotions, based on the number of heart rate fluctuations.

[0264] The synchronicity analysis unit 1011 analyzes whether there is synchronicity between multiple periods of sound stimulation and no stimulation and the timing of the presence or absence of emotion determined by the emotion determination unit.

[0265] The cause determination unit 1012, which acts as an obstacle factor determination unit, determines whether there is a mental abnormality based on whether the timing of the output sound and the timing of the generated emotion are synchronized. For example, in Figure 30In the example shown, emotion was detected when a sound with specific meaning was output, and not when no sound was output. That is, emotion was detected at the same timing as the output of the sound with specific meaning. In this case, it can be determined that the elderly person being tested recognized the meaning of the language and can be judged to have no mental abnormality.

[0266] The pulse extraction image processing unit 1008 also functions as a calculation unit for calculating the complexity of the fluctuations in the heart rate interval based on heart rate information. For example, by using the maximum Lyapunov index as an indicator, the pulse extraction image processing unit 1008 can calculate the complexity of the fluctuations in the heart rate interval based on heart rate information. The RRI Lyapunov emotion determination unit 1009 also functions as an emotion determination unit for determining whether the subject's emotion is negative or positive based on complexity. For example, the RRI Lyapunov emotion determination unit 1009 can determine whether the subject's emotion is negative or positive based on the maximum Lyapunov index calculated by the pulse extraction image processing unit 1008. The stimulus applied to the elderly subject by the stimulus generation unit can be a third stimulus containing at least one of an image or sound of a specific person. The cause determination unit 1012, which serves as an obstacle factor determination unit, can determine whether the subject's compatibility with the specific person is good.

[0267] First, an example will be given where the third stimulus is an image containing a specific person. Figure 31 This example illustrates the temporal changes in the emotions and positive / negative feelings of an elderly person being tested when they periodically see an image of a caregiver as a third stimulus containing an image of a specific person. First, during a predetermined period (e.g., 30 seconds) from time t1 to t2, an image 906 of the caregiver is displayed on display unit 1001. Next, during a predetermined period (e.g., 30 seconds) from time t2 to t3, a white image 902 without the caregiver is displayed on display unit 1001. After time t3, the image 906 of the caregiver and the white image 902 without the caregiver are alternately and repeatedly displayed on display unit 1001. The switching between the image 906 of the caregiver and the white image 902 without the caregiver is performed by a periodic stimulus image generation unit 1002.

[0268] The camera 1006 captures images of the elderly person's face as the subject, including an image 906 of the caregiver and a white image 902 that does not contain the caregiver.

[0269] The pulse extraction image processing unit 1008 is an example of a detection unit. It detects heartbeat information, including the heart rate of the elderly person, from an image of an elderly person as the subject captured by a camera 1006.

[0270] The RRI Lyapunov Emotion Determination Unit 1009 is one example of an emotion determination unit. Based on heartbeat information, it determines whether the emotion of the elderly subject is negative or positive. The RRI Lyapunov Emotion Determination Unit 1009 can determine whether the emotion of the elderly subject is negative or positive based on the complexity of the fluctuations in the heartbeat interval calculated by the pulse extraction image processing unit 1008 based on the heartbeat information.

[0271] The pulse extraction image processing unit 1008 also functions as a counting unit, counting the number of heart rate changes during a specified period as the heart rate increases above a specified value, transitioning from a state below the average heart rate to a state above the average heart rate.

[0272] The pulse rate determination unit 1010, as the emotion determination unit, determines the psychological state of the elderly person being tested, such as the presence or absence of emotion, at predetermined intervals based on the number of heart rate changes.

[0273] The synchronization analysis unit 1011 analyzes whether there is synchronization between multiple periods of displaying the image 906 of the caregiver and the white image 902 that does not contain the image of the caregiver and the timing of the presence or absence of emotions determined by the emotion determination unit and the timing of the occurrence of positive and negative emotions of the elderly.

[0274] The cause determination unit 1012, which is the obstacle factor determination unit, determines whether the compatibility between the elderly person being tested and the caregiver is good or bad based on whether the timing of the display of the caregiver's image 906 and the timing of the emotion generated are synchronized, and which emotion is positive or negative.

[0275] Here, it is argued that when elderly individuals, as test subjects, are shown images of their caregivers, emotions arise regardless of whether the compatibility between the elderly and caregivers is good or poor. Therefore, the presence or absence of emotions when elderly individuals see images of their caregivers indicates that they harbor some affection for the caregivers. However, the presence or absence of emotions alone cannot determine whether the compatibility between the elderly and caregivers is good or bad.

[0276] Therefore, the cause determination section 1012 uses the determination results (positive and negative emotions) of the RRI Lyapunov emotion determination section 1009 to determine whether the compatibility between the elderly and caregivers is good or bad.

[0277] For example, if an elderly person experiences an emotion when shown an image of their caregiver, and the elderly person exhibits a positive emotion at the same time, the cause determination unit 1012 can determine that the elderly person likes the caregiver in terms of personality, that is, the elderly person and the caregiver have good compatibility.

[0278] On the other hand, if an elderly person experiences an emotion when shown an image of a caregiver, and the elderly person experiences a negative emotion at the same time, the cause determination unit 1012 can determine that the elderly person dislikes the caregiver in terms of personality, that is, the elderly person and the caregiver have poor compatibility.

[0279] Thus, the cause determination unit 1012 also functions as an emotion determination unit, and can use the determination results (positive or negative emotions) of the RRI Lyapunov emotion determination unit 1009 (which is an emotion determination unit) to determine the psychological state (good or bad compatibility).

[0280] That is, in this case, the pulse rate emotion determination unit 1010 and the cause determination unit 1012 assume the function of emotion determination units.

[0281] Furthermore, the degree of compatibility is determined by the psychological state of the elderly test subject, which is the cause of that compatibility (e.g., the test subject's unconscious or conscious psychological state of liking or disliking the subject). Therefore, as mentioned above, the emotion judgment unit determines the degree of compatibility as a psychological state.

[0282] For example, in Figure 31 In the example shown, emotions were detected in accordance with the timing of the image 906 displaying the caregiver, and negative emotions were detected. In this case, since at least the elderly person being tested had poor compatibility with the caregiver, it can be determined that there is poor compatibility between the elderly person and the caregiver.

[0283] Next, examples will be given when the third stimulus is a stimulus containing the voice of a specific person. Figure 32 This example illustrates the temporal changes in the emotions and positive / negative feelings of an elderly person being tested when they periodically hear the voice of a caregiver, acting as a third stimulus containing the voice of a specific person. First, during a predetermined period from time t1 to t2 (e.g., 30 seconds), the caregiver's own voice, such as "I am 〇〇," is output once or multiple times from a speaker or headphones 1003. Then, during a predetermined period from time t2 to t3 (e.g., 30 seconds), the sound output stops (silence). After time t3, the state of outputting the caregiver's own voice and the state of silence alternately and repeatedly. The switching between the state of outputting the caregiver's own voice and the state of silence is executed by the periodic stimulus sound generation unit 1004.

[0284] Camera 1006 was used to capture facial images of elderly subjects who periodically heard their caregivers utter the sound of their own name, "I am 〇〇".

[0285] The pulse extraction image processing unit 1008 is an example of a detection unit. It detects heartbeat information, including the heart rate of the elderly person, from an image of an elderly person as the subject captured by a camera 1006.

[0286] The RRI Lyapunov Emotion Determination Unit 1009 is one example of an emotion determination unit. Based on heartbeat information, it determines whether the emotion of the elderly subject is negative or positive. The RRI Lyapunov Emotion Determination Unit 1009 can determine whether the emotion of the elderly subject is negative or positive based on the complexity of the fluctuations in the heartbeat interval calculated by the pulse extraction image processing unit 1008 based on the heartbeat information.

[0287] The pulse extraction image processing unit 1008 also functions as a counting unit, counting the number of heart rate changes during a specified period as the heart rate increases above a specified value, transitioning from a state below the average heart rate to a state above the average heart rate.

[0288] The pulse rate determination unit 1010, as the emotion determination unit, determines the psychological state of the elderly person being tested, such as the presence or absence of emotion, at predetermined intervals based on the number of heart rate changes.

[0289] The synchronicity analysis unit 1011 analyzes whether there is synchronicity between multiple periods of sound stimulation and no stimulation and the timing of the presence or absence of emotions determined by the emotion determination unit and the timing of the occurrence of positive and negative emotions in the elderly.

[0290] The Cause Determination Unit 1012, which is the Obstacle Factor Determination Unit, determines whether the compatibility between the elderly person being tested and the caregiver is good or bad based on whether the timing of the caregiver uttering the sound of "I am 〇〇" (I am 〇〇) is synchronized with the timing of the emotion generated, and whether it is a positive or negative emotion.

[0291] Here, it is argued that when elderly individuals, as test subjects, hear their caregivers' voices, emotions arise regardless of whether the compatibility between the elderly and caregivers is good or bad. Therefore, the presence or absence of emotions when elderly individuals hear their caregivers' voices indicates that they harbor some affection for the caregivers. However, the presence or absence of emotions alone cannot determine whether the compatibility between the elderly and caregivers is good or bad.

[0292] Therefore, the cause determination section 1012 uses the determination results (positive and negative emotions) of the RRI Lyapunov emotion determination section 1009 to determine whether the compatibility between the elderly and caregivers is good or bad.

[0293] For example, if an elderly person experiences an emotion when they hear the caregiver's voice, and the elderly person experiences a positive emotion at the same time, the cause determination unit 1012 can determine that the elderly person likes the caregiver in terms of personality, that is, the elderly person and the caregiver have good compatibility.

[0294] On the other hand, when an elderly person experiences an emotion upon hearing the caregiver's voice, and the elderly person experiences a negative emotion at the same time, the cause determination unit 1012 can determine that the elderly person dislikes the caregiver in terms of personality, that is, the elderly person and the caregiver have poor compatibility.

[0295] Thus, the cause determination unit 1012 also functions as an emotion determination unit, and can use the determination results (positive or negative emotions) of the RRI Lyapunov emotion determination unit 1009 (which is an emotion determination unit) to determine the psychological state (good or bad compatibility).

[0296] That is, in this case, the pulse rate emotion determination unit 1010 and the cause determination unit 1012 assume the function of emotion determination units.

[0297] Furthermore, the degree of compatibility is determined by the psychological state of the elderly test subject, which is the cause of that compatibility (e.g., the test subject's unconscious or conscious psychological state of liking or disliking the subject). Therefore, as mentioned above, the emotion judgment unit determines the degree of compatibility as a psychological state.

[0298] For example, in Figure 32 In the example shown, emotions were detected at the same time as the caregiver uttered their own name, "I am 〇〇," and negative emotions were detected. In this case, it can be determined that there is poor compatibility between the elderly person being tested and the caregiver.

[0299] The periodic stimulus image generation unit 1002 and the periodic stimulus sound generation unit 1004, acting as stimulus generation units, generate any one of the first to third stimuli at predetermined intervals, and then repeatedly generate different stimuli from the first to third stimuli until at least all three stimuli are generated. Furthermore, the pulse rate emotion determination unit 1010 determines that the subject has an emotion if the heart rate fluctuates more than twice during each stimulus generation period and each stimulus rest period (both equivalent to predetermined periods). This is because, based on the switching of images, etc., during the stimulus generation and rest periods, an emotion is generated even if no stimulus occurs. That is, the emotion is caused by stimuli other than the first to third stimuli and should be excluded from the determination of the presence or absence of emotions caused by the first to third stimuli. The first to third stimuli generated by the periodic stimulus image generation unit 1002 and the periodic stimulus sound generation unit 1004 are stimuli that generate emotions at least twice. This is because the purpose of each stimulus is not to test reflexive responses such as surprise, which are determined by a single heart rate change, but rather to test the presence or absence of changes in mental state, which are determined by two or more heart rate changes.

[0300] By determining whether the elderly person exhibits emotions when given the first and second stimuli, and whether they experience positive or negative feelings and emotions when given the third stimulus, the cause of the elderly person's communication difficulties can be determined. Specifically, the cause determination unit 1012, as the obstacle factor determination unit, can make multiple determinations regarding the presence or absence of visual or auditory abnormalities, the presence or absence of mental abnormalities in the test subject, and the degree of compatibility with a specific person. As a determination related to the obstacle factor, it can also determine which of the multiple determinations caused the obstacle factor possessed by the test subject. Figure 33 An example of determining the reasons that hinder communication in older adults.

[0301] First, let's explain examples of elderly individuals being judged to be in a normal state. If an elderly person responds to the first stimulus used to assess vision / hearing, it can be determined that their vision and hearing are normal. Similarly, if an elderly person responds to the second stimulus used to assess mental illness, it can be determined that they are not mentally ill. Furthermore, if an elderly person does not respond to the third stimulus used to assess compatibility, or if their response is positive (positive emotion), it can be determined that the compatibility between the elderly person and their caregiver is good.

[0302] Next, examples of elderly individuals being judged to be in a state of mental abnormality will be explained. If the elderly person responds to the first stimulus used to assess vision / hearing, it can be determined that their vision and hearing are not abnormal. Furthermore, if the elderly person does not respond to the second stimulus used to assess mental illness, it is because they are unable to understand the meaning of the images and sounds contained in the second stimulus, thus it can be determined that the elderly person is in a state of mental abnormality. Moreover, if the elderly person does not respond to the third stimulus used to assess compatibility, or if they respond but express a positive response (positive emotion), it can be determined that the compatibility between the elderly person and their caregiver is not poor.

[0303] Next, examples of elderly individuals being diagnosed with hearing impairment will be explained. If an elderly person responds to the first stimulus used for visual assessment, their vision can be determined to be normal. If an elderly person does not respond to the first stimulus used for hearing assessment, they can be diagnosed with hearing impairment. However, if an elderly person has hearing impairment, even if they are given a second stimulus containing sound used for assessing mental illness, the presence or absence of a response cannot determine the presence or absence of mental illness. Therefore, regardless of whether there is a sound response to the second stimulus containing sound, hearing impairment is diagnosed. Figure 33 The table shown is marked "None / Yes". Furthermore, when the older person is given the third stimulus used for compatibility assessment, if there is no response or, even if, a response is positive (positive emotion), the compatibility between the older person and caregiver can be determined to be not poor. Figure 33 The examples shown illustrate situations where there is good compatibility between the elderly person and their caregiver, but there are also cases where the elderly person has hearing impairment and poor compatibility with their caregiver.

[0304] Next, examples of elderly individuals being judged to have a visual state will be explained. If an elderly person responds to the first stimulus used for hearing assessment, it can be determined that their hearing is normal. If an elderly person does not respond to the first stimulus used for visual assessment, it can be determined that they have visual impairment. If an elderly person has visual impairment, even if they are given a second stimulus containing an image used for mental illness assessment, the presence or absence of a response cannot determine the presence or absence of mental illness. Therefore, regardless of whether there is an image response to the second stimulus containing an image, it is determined to be visual impairment. Figure 33 The table shown is marked "None / Yes". Furthermore, when the older person is given the third stimulus used for compatibility assessment, if there is no response or, even if, a response is positive (positive emotion), the compatibility between the older person and caregiver can be determined to be not poor. Figure 33The examples shown illustrate situations where there is good compatibility between the elderly person and their caregiver, but there are also cases where the elderly person has hearing impairment and poor compatibility with their caregiver.

[0305] Next, examples of determining compatibility between elderly individuals will be explained. If an elderly person responds to the first stimulus used to assess vision / hearing, it can be determined that their vision and hearing are normal. Furthermore, if an elderly person responds to the second stimulus used to assess mental illness, it can be determined that they are not mentally ill. Moreover, if an elderly person responds to the third stimulus used to assess compatibility and expresses a negative reaction (negative emotion), it can be determined that the compatibility between the elderly person and their caregiver is poor. Additionally, if a negative reaction occurs when at least one of the visual or auditory stimuli is presented as the third stimulus used to assess compatibility, it can be determined that the compatibility between the elderly person and their caregiver is poor. Furthermore, with... Figure 33 The situation is different. When the third stimulus used to determine compatibility is given to the elderly, if there is a response and a positive response (positive emotion), it can be determined that the compatibility between the elderly and the caregiver is good.

[0306] Next, the method for determining the presence or absence of emotions will be explained. Figure 34 (A) represents an example of the truth value of a response when a stimulus is applied periodically. First, a predetermined period from time t1 to t2 (e.g., 30 seconds) is taken as the stimulus rest period during which no stimulus is given to the subject. At this time, since no response is predicted from the subject, the truth value is set to "0". Next, a predetermined period from time t2 to t3 (e.g., 30 seconds) is taken as the stimulus generation period during which a stimulus is given to the subject. At this time, since a response is predicted from the subject, the truth value is set to "1". After time t3, when the stimulus rest period and stimulus generation period are periodically repeated, the truth value "0" in the stimulus rest period and the truth value "1" in the stimulus generation period are periodically repeated accordingly.

[0307] Figure 34 (B) indicates that it is related to Figure 34 Example (A) shows the determination of emotion when the subject is consistently given a stimulus at regular intervals. For example, if no emotion is detected when time t1 to t2 is taken as the stimulus rest period and no stimulus is given to the subject, the emotion is "none" and is represented as "0". The truth value at this time is based on... Figure 34 Since (A) is "0", the emotion during the stimulus rest period is consistent with the true value. Next, if the stimulus is generated between time t2 and t3, and the emotion is detected when the stimulus is given to the subject, the emotion is "present", represented as "1". The true value at this point is based on... Figure 34Since (A) is "1", the emotion during the period when the stimulus was generated is consistent with the true value.

[0308] From time t3 to t11, the stimulus rest period and stimulus generation period are periodically repeated, and the emotion is determined within a total of 10 periods. At this time, if no emotion is detected during the stimulus generation period from time t6 to t7, the emotion is "none" ("0"). However, since the true value for this period is "1", the emotion determination result is inconsistent with the true value. Since the emotion determination result is consistent with the true value in the 9 periods other than t6 to t7 out of the total 10 periods from t1 to t11, the synchronization rate is calculated to be 90%. The synchronization analysis unit 1011 determines the emotion based on whether the synchronization rate is above a predetermined value, i.e., whether the proportion of synchronization between the stimulus generation period and the stimulus rest period is synchronized. For example, if the predetermined value is set to 70%, and the calculated synchronization rate is 90%, since it exceeds 70% as the predetermined value, it can be determined that the emotion is synchronized with the stimulus.

[0309] The above example illustrates a determination based on the synchronization rate, calculated during multiple periods of stimulus delivery and stimulus rest, to assess the presence or absence of emotion. However, this is not an isolated example. That is, even with a fixed number of stimulus deliveries and rest periods, a determination based on the synchronization rate (the proportion of synchronization) can be considered valid, even without calculating the synchronization rate, simply determining whether the number of synchronization occurrences exceeds a predetermined number. For example, as... Figure 34 (A) and Figure 34 As shown in (B), the total duration of the stimulus generation period and the stimulus rest period can also be fixed at 10 periods. If the number of times synchronization occurs during the stimulus generation period and the stimulus rest period is more than a specified number (e.g., 7 times), it is considered that the synchronization rate is more than 70% of the specified value and is judged as synchronization.

[0310] By using the subject status determination device 1000, which serves as a means of determining the cause of communication barriers, it is possible to examine, for example, the elderly person admitted to a care facility on the first day and distinguish their type. Furthermore, in determining the compatibility between the elderly person and their caregiver, it is sometimes impossible to determine that the caregiver and the elderly person are unrelated within a specified period; therefore, it is also possible to conduct an examination approximately one month after the elderly person's admission to the care facility. By determining the cause of the communication barrier between the elderly person and their caregiver, appropriate countermeasures to restore communication can be taken.

[0311] [Postscript]

[0312] The above-described Embodiment 6 is a specific example of the invention described in the appendices shown below.

[0313] (Postscript 1)

[0314] A device for determining the state of a subject, characterized in that it has:

[0315] The stimulus-generating part produces visual or auditory stimuli to the subject at each of a predetermined plurality of prescribed periods with rest periods in between.

[0316] The testing department's tests include the subject's heart rate and heartbeat information;

[0317] The counting unit counts the number of heart rate changes during each specified period, which occur as the heart rate increases above a specified value and changes from a state below the average heart rate to a state above the average heart rate.

[0318] The emotion determination unit determines the presence or absence of an emotion in the subject corresponding to a stimulus during each of the specified periods, based on the number of heart rate fluctuations.

[0319] A synchronicity analysis unit analyzes whether synchronicity exists between the plurality of periods and the timing of the presence or absence of emotions determined by the emotion determination unit; and

[0320] The obstacle factor determination unit determines, based on the analysis results of the synchronicity determination unit, the obstacles to communication with other people for the subject.

[0321] Devices for determining communication impairments, as known to date, include those that assess visual, auditory, or cognitive abilities based on the subject's sense of sight and hearing. However, in conventional devices, when determining based on responses such as answers or button presses, the responses themselves are sometimes difficult, preventing accurate assessment. Additionally, devices that determine communication impairments based on the subject's brainwaves are known. However, such devices are large in scale, requiring a spacious environment for installation. Furthermore, the need for advanced analytical units limits the number of people capable of operating them.

[0322] Therefore, it is preferable to have a simple structure that can easily address the communication barriers of the test subjects and reduce the limitations of the setting location and the number of people who can handle it.

[0323] According to the appendix, the test subject status determination device determines the communication barriers by analyzing the number of heart rate changes that transition to a state above average heart rate and whether there is synchronicity in the timing of the presence or absence of emotions. Therefore, the device has a simple structure, which can reduce the limitation of the setting location. Furthermore, it can determine the communication barriers of the test subject through simple testing, which can also reduce the limitation of the people who can handle it.

[0324] (Postscript 2)

[0325] In the subject status determination device described in Appendix 1

[0326] The stimulus is the first stimulus that can be sensorily identified by sight or hearing.

[0327] The obstacle factor determination unit determines whether there is any abnormality in the vision or hearing.

[0328] In the subject status determination device described in Appendix 2

[0329] The stimulus may be the first stimulus that occurs and can only be identified based on visual or auditory sensations.

[0330] (Note 3)

[0331] In the subject status determination device described in Appendix 1 or 2

[0332] The stimulus is a second stimulus containing prescribed information that can be understood.

[0333] The obstacle factor determination unit determines whether the test subject has any mental abnormalities.

[0334] (Note 4)

[0335] The subject status determination device described in any one of Appendices 1 to 3 further comprises:

[0336] The computing unit calculates the complexity of the fluctuations in the heartbeat interval based on the heartbeat information; and

[0337] The emotion determination department, based on the aforementioned complexity, determines whether the test subject's emotion is negative or positive.

[0338] The stimulus is a third stimulus that contains at least one of the images or sounds of a specific person.

[0339] The obstacle factor determination unit determines whether the compatibility between the test subject and the specific person is good.

[0340] (Note 5)

[0341] In the subject status determination device described in Appendix 4

[0342] The obstacle factor determination unit can make multiple determinations, including the presence or absence of visual or auditory abnormalities, the presence or absence of mental abnormalities in the subject, and the degree of compatibility with the specific person.

[0343] As a determination related to the aforementioned obstacle, it is also determined which of the plurality of determinations the obstacle possessed by the test subject is caused by.

[0344] (Note 6)

[0345] In the subject status determination device described in Appendix 4

[0346] The stimulus generating unit generates any one of the first to third stimuli during each specified period, and then repeatedly performs the action of generating different stimuli among the first to third stimuli until at least the first to third stimuli are generated.

[0347] (Note 7)

[0348] In any of the subject status determination devices described in Appendices 1 to 6

[0349] The synchronicity analysis unit determines whether the proportion of synchronization of the emotions during the occurrence and cessation of the stimulus is above a predetermined value.

[0350] The subject state determination device described in Appendix 7 also includes a case where the number of stimuli generated is fixed and the number of times they are synchronized is more than a specified number.

[0351] (Postscript 8)

[0352] In any of the subject status determination devices described in Appendices 1 to 7

[0353] The first stimulus is a sensory stimulus that is visually or auditorily perceived to identify the occurrence.

[0354] (Note 9)

[0355] In any of the subject status determination devices described in Appendices 3 to 8

[0356] The second stimulus is a cognitive stimulus that accompanies the cognition of prescribed information.

Claims

1. An emotion determination device, characterized in that, The emotion determination device has the following features: The testing department's tests include the subject's heart rate and heartbeat information; The emotion determination department determines whether the subject's emotion is negative or positive based on the heartbeat information. The counting unit counts the number of heart rate changes that occur during a specified period, as the heart rate increases to a value above the specified value, from a state below the average heart rate to a state above the average heart rate. The emotion determination unit is able to determine the psychological state of the subject based on the number of heart rate changes, and use the determination result of the emotion determination unit to determine the psychological state. as well as The output unit outputs the determination result of the emotion determination unit. The psychological state of the test subject determined by the emotion determination unit includes any one of the following: a stable state, a surprised state, a grateful state, an emotion that cannot be determined, and an angry state.

2. The emotion determination device according to claim 1, characterized in that, The emotion determination device also includes a camera unit that captures images of the subject's face. The detection unit detects the heartbeat information based on changes in the image data acquired by the imaging unit.

3. The emotion determination device according to claim 1, characterized in that, The psychological state includes: a first psychological state in which both positive and negative emotions can be determined based on the number of heart rate fluctuations; and a second psychological state in which both positive and negative emotions and the number of heart rate fluctuations can be determined. The emotion determination unit uses the determination result of the emotion determination unit at least when determining the second psychological state.

4. The emotion determination device according to claim 1, characterized in that, The specified period was repeatedly set. The emotion determination unit makes a determination according to each of the specified periods.

5. The emotion determination device according to claim 2, characterized in that, The imaging unit is capable of capturing images of the faces of multiple subjects. The emotion determination device also includes a measurement location determination unit, which identifies each face from a screen displaying multiple people and determines the measurement location for each identified face. The detection unit acquires the heartbeat information based on the changes in the images of the measurement sites on each face.

6. The emotion determination device according to any one of claims 1 to 5, characterized in that, The test subjects were the listeners of the lecture course. The emotion determination unit determines whether the patient is in an ideal psychological state for listening to the course based on the number of heart rate fluctuations and the determination result of the emotion determination unit.

7. The emotion determination device according to any one of claims 1 to 5, characterized in that, The emotion determination device has a stimulation generating unit that generates the following stimuli: It can sense whether a stimulus has occurred through sight or hearing; Stimuli that enable the understanding of the content of information provided by sight or hearing; and A stimulus containing at least one of the images or sounds of a specific person. The stimulus-generating unit repeatedly generates the same stimulus at predetermined intervals with rest periods. The emotion determination unit determines the psychological state of the test subject at least during the predetermined plurality of periods.

8. The emotion determination device according to claim 1, characterized in that, The negative emotion is the feeling that the subject experiences, at least one of mental fatigue, anxiety, and depression.

9. The emotion determination device according to any one of claims 1 to 4, characterized in that, If the heart rate fluctuation counted by the counting unit is one, the emotion determination unit determines that the psychological state is one of surprise.

10. The emotion determination device according to any one of claims 1 to 5, characterized in that, If the number of heart rate changes counted by the counting unit is multiple and the emotion determination unit determines that the subject's emotion is positive, the emotion determination unit determines that the psychological state is one of gratitude.

11. The emotion determination device according to any one of claims 1 to 5, characterized in that, If the number of heart rate changes counted by the counting unit is multiple and the emotion determination unit determines that the subject's emotion is negative, the emotion determination unit determines that the psychological state is anger.

12. The emotion determination device according to any one of claims 1 to 5, characterized in that, If the number of heart rate fluctuations counted by the counting unit is 0, and the heart rate remains lower than the average heart rate during the specified period, the emotion determination unit determines that the psychological state is stable.

13. The emotion determination device according to any one of claims 1 to 5, characterized in that... If the number of heart rate changes counted by the counting unit is 0, and the heart rate is maintained at or above the average heart rate during the specified period, the emotion determination unit determines that the psychological state cannot be determined and the emotion cannot be determined.

14. An emotion determination program product, comprising an emotion determination program, characterized in that, The emotion determination program causes the computer to perform the following steps: The test includes the subject's heart rate and heartbeat information; Based on the heartbeat information, it is determined whether the subject's emotion is negative or positive. During the specified period, the number of heart rate changes that occur as the heart rate increases above a specified value, transitioning from a state below the average heart rate to a state above the average heart rate, is counted. The psychological state of the subject is determined based on the number of heart rate fluctuations, and the determination result is used to determine the psychological state. as well as Output the determination result of the psychological state. The psychological state of the test subject includes any one of the following: stable state, surprised state, grateful state, emotionally ambiguous state, and angry state.

15. A method for determining emotion, characterized in that, The testing department detects the subject's heart rate and heartbeat information; The emotion assessment department determines whether the subject's emotion is negative or positive based on the heartbeat information. The counting unit counts the number of heart rate changes that occur during a specified period, as the heart rate increases to a value above the specified value, from a state below the average heart rate to a state above the average heart rate. The emotion determination unit determines the psychological state of the test subject based on the number of heart rate changes, and uses the determination result of the emotion determination unit to determine the psychological state. as well as The output unit outputs the determination result of the emotion determination unit. The psychological state of the test subject determined by the emotion determination unit includes any one of the following: a stable state, a surprised state, a grateful state, an emotion that cannot be determined, and an angry state.

Images