Emotion estimation system, emotion estimation device, emotion estimation method, and program

Abstract

To estimate an emotion of a subject accurately, with a small amount of calculation on the basis of biological data.SOLUTION: An emotion estimation system comprises: an acquiring unit for acquiring biological data from a biological sensor; a neutral reference setting unit for generating a Poincare plot which is formed of a point group in two axes on the basis of the biological data in resting of a subject, out of pieces of biological data, then calculating a barycenter position of the Poincare plot and setting the calculated position as a neutral reference; a threshold setting unit for setting a first threshold to a prescribed position in an origin direction of the Poincare plot to the neutral reference, then setting a second threshold to a prescribed position in an opposite direction of the origin direction of the Poincare plot; a calculation unit for generating the Poincare plot on the basis of the biological data of an inspection object for a prescribed period out of the pieces of biological data, then calculating a barycenter position of the Poincare plot as an inspection point; and an estimation unit for estimating an emotion of the subject on the basis of the inspection point.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an emotion estimation system, an emotion estimation device, an emotion estimation method, and a program.

Background Art

[0002] In recent years, in a vehicle driving support system or the like, a technique for acquiring biometric data (such as heartbeat data) of a passenger has been used. Then, by applying various methods such as fast Fourier transform and Poincaré plot to the acquired biometric data, the emotion of the passenger can be estimated.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above prior art, for example, in a method using fast Fourier transform, there is a problem that the amount of calculation is large. Further, for example, in a method using Poincaré plot, there is room for improvement in terms of accuracy because the influence due to individual differences is not taken into account.

[0005] Therefore, an embodiment of the present invention aims to provide an emotion estimation system, an emotion estimation device, an emotion estimation method, and a program that can estimate the emotion of a target person with high accuracy with a small amount of calculation based on biometric data.

Means for Solving the Problems

[0006] The emotion estimation system according to an embodiment of the present invention includes a biosensor that measures predetermined biometric data relating to a subject that fluctuates periodically and whose period length tends to change depending on the subject's emotional changes; an acquisition unit that acquires the biometric data from the biosensor; a neutral reference setting unit that generates a Poincaré plot consisting of a point cloud in two axes based on the subject's resting biometric data, calculates its centroid position and sets it as a neutral reference; and a first threshold value set at a predetermined position in the direction of the origin of the Poincaré plot relative to the neutral reference, and a second threshold value set at a predetermined position in the opposite direction to the origin of the Poincaré plot. The system includes a threshold setting unit, a calculation unit that generates a Poincaré plot based on the biological data of the subject to be examined for a predetermined time period and calculates the centroid position as the examination point, and an estimation unit that estimates a neutral emotion if the state in which the examination point is between the first threshold and the second threshold continues for a predetermined time, estimates an unpleasant emotion if the state in which the examination point is less than the first threshold continues for a predetermined time, and if the state in which the examination point is greater than the second threshold continues for a predetermined time, calculates an area corresponding to the width of the distribution of the Poincaré plot corresponding to the examination point, and estimates whether or not it is a pleasant emotion based on the area and the distance between the neutral criterion and the examination point. This configuration allows for the calculation of a neutral criterion, comparison of the test score with the first and second thresholds, and a switch to a different estimation logic if the test score remains above the second threshold for a predetermined period of time. This enables the estimation of a subject's emotions with high accuracy and minimal computational effort based on biometric data.

[0007] Furthermore, if the state in which the inspection point is greater than the second threshold continues for a predetermined time, the estimation unit normalizes the area and the distance, substitutes the normalized area and distance into the parameters of a predetermined multiple regression equation, and estimates that the emotion is pleasant if the calculation result satisfies predetermined conditions. This configuration, by using normalized area and distance and a pre-defined multiple regression equation, further reduces the influence of individual differences and enables highly accurate sentiment estimation.

[0008] The system further includes a respiratory sensor that measures the respiratory data of the subject, the acquisition unit acquires the respiratory data from the respiratory sensor, and the estimation unit determines whether the subject is speaking or not based on the respiratory data, and if it determines that the subject is speaking, it invalidates the estimation result of the subject's emotions. This configuration allows for the disabling of emotion estimation during speech, when the accuracy of emotion estimation tends to decrease.

[0009] The aforementioned predetermined biometric data is the heart rate data of the subject. This configuration allows for the estimation of a subject's emotions using heart rate data.

[0010] Furthermore, the emotion estimation device of the embodiment of the present invention includes: an acquisition unit that acquires biological data from a biosensor that measures predetermined biological data relating to a subject that fluctuates periodically and whose period length tends to change depending on the subject's emotional changes; a neutral reference setting unit that generates a Poincaré plot consisting of a point cloud in two axes based on the biological data of the subject at rest, calculates the centroid position and sets it as a neutral reference; and a first threshold value set at a predetermined position in the direction of the origin of the Poincaré plot with respect to the neutral reference, and a second threshold value set at a predetermined position in the opposite direction to the origin of the Poincaré plot. The system includes a threshold setting unit, a calculation unit that generates a Poincaré plot based on the biological data of the subject to be examined for a predetermined time period and calculates the centroid position as the examination point, and an estimation unit that estimates a neutral emotion if the state in which the examination point is between the first threshold and the second threshold continues for a predetermined time, estimates an unpleasant emotion if the state in which the examination point is less than the first threshold continues for a predetermined time, and if the state in which the examination point is greater than the second threshold continues for a predetermined time, calculates an area corresponding to the width of the distribution of the Poincaré plot corresponding to the examination point, and estimates whether or not it is a pleasant emotion based on the area and the distance between the neutral criterion and the examination point.

[0011] Furthermore, the emotion estimation method of the embodiment of the present invention includes an acquisition step of acquiring biological data from a biosensor that measures predetermined biological data relating to a subject that fluctuates periodically and whose period length tends to change depending on the subject's emotional changes; a neutral reference setting step of generating a Poincaré plot consisting of a point cloud in two axes based on the biological data of the subject at rest, calculating its centroid position and setting it as a neutral reference; and a threshold setting step of setting a first threshold at a predetermined position in the direction of the origin of the Poincaré plot with respect to the neutral reference, and setting a second threshold at a predetermined position in the opposite direction to the origin of the Poincaré plot. The method includes a value setting step, a calculation step of generating a Poincaré plot based on the biological data of the subject to be tested for a predetermined time period and calculating the centroid position as the test point, and an estimation step of estimating whether or not it is a pleasant emotion if the state in which the test point is between the first threshold and the second threshold continues for a predetermined time, estimating a neutral emotion if the state in which the test point is less than the first threshold continues for a predetermined time, and calculating an area corresponding to the width of the distribution of the Poincaré plot corresponding to the test point if the state in which the test point is greater than the second threshold continues for a predetermined time, and estimating whether or not it is a pleasant emotion based on the area and the distance between the neutral criterion and the test point.

[0012] Furthermore, the emotion estimation system of the embodiment of the present invention includes a computer, an acquisition unit that acquires biological data from a biosensor that measures predetermined biological data relating to a subject that periodically fluctuates and whose period length tends to change depending on changes in the subject's emotions, a neutral reference setting unit that generates a Poincaré plot consisting of a point cloud in two axes based on the biological data of the subject at rest, calculates the centroid position and sets it as a neutral reference, and a threshold that sets a first threshold at a predetermined position in the direction of the origin of the Poincaré plot with respect to the neutral reference, and sets a second threshold at a predetermined position in the direction opposite to the origin of the Poincaré plot. This program is designed to function as a setting unit, a calculation unit that generates a Poincaré plot based on a predetermined amount of biological data of the subject being tested for a predetermined time, and calculates the centroid position as the test point, and an estimation unit that estimates a neutral emotion if the test point remains between the first threshold and the second threshold for a predetermined time, estimates an unpleasant emotion if the test point remains below the first threshold for a predetermined time, and if the test point remains above the second threshold for a predetermined time, calculates an area corresponding to the width of the distribution of the Poincaré plot corresponding to the test point, and estimates whether or not it is a pleasant emotion based on the area and the distance between the neutral criterion and the test point. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a schematic diagram showing the configuration of the vehicle system of the embodiment. [Figure 2] Figure 2 is a block diagram outlining the functional configuration of the vehicle system according to the embodiment. [Figure 3] Figure 3 shows an example of a Poincaré plot in an embodiment. [Figure 4] Figure 4 is a flowchart showing the first process performed by the information processing device of the embodiment. [Figure 5] Figure 5 is a flowchart showing the second process performed by the information processing device of the embodiment. [Figure 6]FIG. 6 is a flowchart showing a third process by the information processing apparatus according to the embodiment.

Embodiments of the Invention

[0014] Hereinafter, exemplary embodiments of the present invention will be disclosed. The configurations of the embodiments shown below, as well as the actions, results, and effects brought about by these configurations, are examples. The present invention can be realized by configurations other than those disclosed in the following embodiments, and it is possible to obtain at least one of various effects and derivative effects based on the basic configuration.

[0015] FIG. 1 is a diagram schematically showing the configuration of the vehicle system 1 according to the embodiment. FIG. 2 is a block diagram showing an overview of the functional configuration of the vehicle system 1 according to the embodiment. In the vehicle system 1, for example, the emotion of the occupant 2 sitting on the seat 21 is estimated. The vehicle system 1 includes each configuration shown in FIGS. 1 and 2. Note that there are also configurations shown in FIG. 2 but not shown in FIG. 1.

[0016] Also, in the present embodiment, as biometric data that periodically varies regarding the target person (occupant 2) and whose period length tends to change according to the change in the emotion of the target person, heartbeat data will be taken as an example for explanation.

[0017] The heartbeat sensor 11 (biometric sensor) transmits radio waves toward the occupant 2 and detects a heartbeat signal (heartbeat data), which is an electrical signal corresponding to the heartbeat of the occupant 2, by receiving the reflected wave from the occupant 2. The heartbeat sensor 11 is disposed on the backrest portion 22 of the seat 21, transmits radio waves (transmission waves) toward a predetermined portion of the occupant 2 (for example, the vicinity of the heart on the back), and receives the reflected wave generated by the reflection of the transmission wave by the occupant 2. The heartbeat sensor 11 detects a heartbeat signal corresponding to the heartbeat (pulse accompanying the pulsation) of the occupant 2 based on the change in the frequency of the radio waves transmitted and received. The frequency of the transmission wave of the heartbeat sensor 11 can be appropriately selected according to the usage situation, but is, for example, about 24 GHz. The heartbeat sensor 11 is, for example, a Doppler-type sensor.

[0018] The wearable device 12 is, for example, a smartwatch with a heart rate data acquisition function. In this embodiment, although the heart rate sensor 11 and the wearable device 12 are used as means for acquiring heart rate data, both are not essential, and at least one of them is sufficient. Further, the method for acquiring heart rate data is not limited to these. For example, a method of acquiring heart rate data by image processing of a captured image of the occupant by a camera, or a method of acquiring heart rate data by sensing the hand of the occupant holding the steering wheel by a steer touch sensor disposed on the steering wheel may also be used.

[0019] The authentication device 13 is a device that acquires authentication information from an authentication tag or an ID (Identifier) card.

[0020] The vehicle information sensor 14 is a sensor that acquires various vehicle information. Examples of the vehicle information include steering information, accelerator operation information, brake operation information, vehicle speed information, vehicle interior temperature information, vehicle surrounding information, vehicle position information, and the like.

[0021] The respiration sensor 15 measures the respiration data of the occupant 2. The respiration sensor 15 is realized, for example, by an image sensor or a sensor that detects changes in chest circumference and abdominal circumference in a belt type.

[0022] The display device 61 is a means for displaying various information, and is, for example, a liquid crystal display.

[0023] The audio device 62 is a means for generating various sounds, and is, for example, a speaker.

[0024] The aroma device 63 is a device that diffuses a predetermined aroma component into the vehicle interior. The occupant can adjust the autonomic nervous system, hormone balance, and the condition of various organs by absorbing this aroma component through the nose, lungs, skin, and the like.

[0025] The seat control mechanism 64 is a mechanism that changes the posture of the seat on which the occupant 2 sits or executes a massage function.

[0026] The drive mechanism 65 is a mechanism that drives the vehicle's power source (engine, motor, etc.) through the driver's accelerator operation or automatic drive.

[0027] The braking mechanism 66 is a mechanism that performs braking to decelerate and stop the vehicle through the driver's brake operation or automatic braking.

[0028] The steering mechanism 67 is a mechanism that changes the direction of travel of the vehicle through the driver's steering input or automatic steering.

[0029] The air conditioning mechanism 68 is a mechanism that performs air conditioning functions such as temperature control, airflow control, and ventilation.

[0030] Communication device 69 is a device that performs various communications, such as emergency notifications, to external devices.

[0031] The information processing device 5 is, for example, an ECU (Electronic Control Unit). The information processing device 5 may be implemented using an ECU for vehicle control, or it may be implemented using a different ECU. The information processing device 5 comprises a processing unit 51 and a storage unit 52. The processing unit 51 has the following functional configuration: an acquisition unit 511, a setting unit 512, a calculation unit 513, an estimation unit 514, and a control unit 515.

[0032] The acquisition unit 511 acquires various information from other components. For example, the acquisition unit 511 acquires heart rate data from the heart rate sensor 11 and the wearable device 12. The acquisition unit 511 also acquires authentication information from the authentication device 13, such as authentication tags and ID cards. Furthermore, the acquisition unit 511 acquires various vehicle information from the vehicle information sensor 14. Finally, the acquisition unit 511 acquires respiratory data from the respiratory sensor 15.

[0033] Please also refer to Figure 3 below. Figure 3 is a diagram showing an example of a Poincaré plot in the embodiment. The setting unit 512 sets various parameters and thresholds. For example, the setting unit 512 (neutral reference setting unit) generates a Poincaré plot consisting of point cloud data P on two axes based on the resting heart rate data of the occupant 2 (for example, heart rate data obtained by instructing the occupant 2 to rest for 5 minutes) from the heart rate data, calculates the centroid position CG, and sets the neutral reference LN based on the centroid position CG.

[0034] Furthermore, the setting unit 512 (threshold setting unit) sets a first threshold LD at a predetermined position in the direction of the origin of the Poincaré plot (for example, a position moved 3% from the origin) relative to the neutral reference LN, and sets a second threshold LU at a predetermined position in the opposite direction to the origin of the Poincaré plot (for example, a position moved 1% from the origin). When setting these thresholds, for example, information regarding personal characteristics may be acquired and used as personal data from a wearable device 12 or authentication device 13 (authentication tag or ID card) as pre-operation information.

[0035] Furthermore, the setting unit 512 may acquire health levels before boarding and sleep levels from the previous day as a life log to understand the driver's physical condition before starting to drive, and use this information to set thresholds. The cloud computer 7 stores, for example, the personal data, life log, and abnormality notification history of the occupant 2.

[0036] Furthermore, the setting unit 512 may acquire and use vehicle information from the vehicle information sensor 14 in order to understand the occupant's condition while driving.

[0037] The calculation unit 513 generates a Poincaré plot based on the heart rate data of the subject under examination for a predetermined period of time, and calculates the centroid position as the examination point. This process is repeated while slightly shifting the time period for processing the heart rate data of the subject under examination.

[0038] The estimation unit 514 estimates a neutral emotion if the test point remains between the first threshold LD and the second threshold LU for a predetermined time (for example, about 10 seconds). Furthermore, the estimation unit 514 estimates an unpleasant emotion if the test point remains below the first threshold LD for a predetermined time (for example, about 10 seconds).

[0039] Furthermore, if the test point remains above the second threshold LU for a predetermined time (for example, about 10 seconds), the estimation unit 514 switches the estimation logic to calculate an area corresponding to the width of the distribution of the Poincaré plot corresponding to the test point (the area of ​​ellipse E in Figure 3, calculated based on the distribution of the Poincaré plot). Based on this area and the distance between the neutral criterion and the test point, the unit estimates whether or not the emotion is pleasant.

[0040] Specifically, for example, if the state in which the test point is greater than the second threshold LD continues for a predetermined time, the estimation unit 514 normalizes the area and distance (for example, normalizes them so that the standard deviation is 1 and the mean is 0), substitutes the normalized area and distance into the parameters of a predetermined multiple regression equation, and estimates that the emotion is pleasant if the calculation result satisfies predetermined conditions (for example, when the value is 0 or greater).

[0041] For example, if we let S and m be the normalized area and distance, respectively, and α and β be predetermined values ​​obtained in advance, the multiple regression equation is given by equation (1) below. HF = αm + βS ... Equation (1)

[0042] Then, for example, by substituting normalized area and distance into this multiple regression equation, we estimate that the emotion is pleasant when the value becomes 0 or greater. In other words, in that case, we set α and β in the multiple regression equation to values ​​appropriate for that estimation.

[0043] Furthermore, the estimation unit 514 may determine whether or not crew member 2 is speaking based on the respiratory data, and if it determines that crew member 2 is speaking, it may invalidate the estimated result of crew member 2's emotion. The reason for invalidating is that, generally, the accuracy of emotion estimation decreases when the subject is speaking.

[0044] The control unit 515 performs various controls. For example, the control unit 515 controls the display device 61, sound equipment 62, fragrance equipment 63, seat control mechanism 64, drive mechanism 65, braking mechanism 66, steering mechanism 67, air conditioning mechanism 68, and communication equipment 69 according to the emotion estimation result.

[0045] The control unit 515 displays warnings, status information, stress levels, relaxation levels, and driving suitability levels via the display device 61, according to the emotion estimation results.

[0046] Furthermore, the control unit 515 generates warning sounds, etc., using the acoustic device 62 according to the emotion estimation result.

[0047] Furthermore, the control unit 515 controls the seat control mechanism 64 according to the emotion estimation result to change the seat's posture or perform a massage.

[0048] Furthermore, the control unit 515 controls the drive mechanism 65, braking mechanism 66, and steering mechanism 67 according to the emotion estimation result, thereby generating a sensory warning, avoiding an accident, or stopping the vehicle on the shoulder of the road.

[0049] Furthermore, the control unit 515 controls the air conditioning mechanism 68 to adjust the temperature and airflow, or to perform ventilation, according to the emotion estimation results.

[0050] Furthermore, if the control unit 515 detects an abnormality occurring inside the vehicle, for example, it will control the communication device 69 according to the emotion estimation result to send an emergency notification or registered user notification to a designated external organization.

[0051] Next, the first processing by the information processing device 5 will be described with reference to Figure 4. Figure 4 is a flowchart of the first processing by the information processing device 5 in this embodiment. The first processing is for determining the rest criteria (details will be described later). The heart rate data used is, for example, the heart rate data acquired when the occupant 2 is instructed to rest for 5 minutes.

[0052] First, in step S11, the acquisition unit 511 reads heart rate data (current value) from the heart rate sensor 11, wearable device 12, etc.

[0053] Next, in step S12, the setting unit 512 determines whether or not a heart rate peak has been detected based on the heart rate data. If yes, the unit proceeds to step S13; otherwise, it returns to step S11.

[0054] In step S13, the setting unit 512 calculates the heart rate interval (RRI (interval)).

[0055] Next, in step S14, the setting unit 512 outputs X-axis (horizontal axis in Figure 3) information (current value T).

[0056] Next, in step S15, the setting unit 512 determines whether a specified time (for example, about 1 to 3 seconds) has elapsed. If yes, proceed to step S16; otherwise, return to step S11.

[0057] In step S16, the setting unit 512 outputs Y-axis (vertical axis in Figure 3) information (T+N (for example, N=1~3)).

[0058] Next, in step S17, the setting unit 512 acquires point cloud data P (Figure 3) by repeating steps S11 to S16.

[0059] Next, in step S18, the setting unit 512 calculates the centroid CG (Figure 3) by calculating the average value using the point cloud data P.

[0060] Next, in step S19, the setting unit 512 calculates and determines a neutral reference LN as the resting reference, which is a straight line passing through the center of gravity CG and parallel to Y=-X. It also determines a predetermined position in the direction of the origin (for example, a position moved 3% from the origin) as the first threshold LD with respect to the neutral reference LN, and determines a predetermined position in the opposite direction to the origin (for example, a position moved 1% from the origin) as the second threshold LU.

[0061] Next, the second processing by the information processing device 5 will be described with reference to Figure 5. Figure 5 is a flowchart showing the second processing by the information processing device of the embodiment. The second processing is emotion estimation processing. The heart rate data used is, for example, heart rate data obtained from a driver while driving.

[0062] First, in step S201, the setting unit 512 sets the resting estimation region (the region between the first threshold LD and the second threshold LU).

[0063] Next, in step S202, the calculation unit 513 calculates the center of gravity position of the heart rate data of the new subject being examined.

[0064] Next, in step S203, the estimation unit 514 determines whether the center of gravity is within the resting estimation range. If yes, the unit proceeds to step S204; otherwise, it proceeds to step S206.

[0065] In step S204, the estimation unit 514 determines whether a specified time (for example, about 5 to 10 seconds) has elapsed since the center of gravity entered the resting estimation region. If yes, the unit proceeds to step S205; otherwise, it returns to step S203. Note that in step S204, a specified number of times may be used instead of a specified time.

[0066] In step S205, the estimation unit 514 estimates the emotion to be neutral. Next, the process proceeds to step S217.

[0067] In step S206, the estimation unit 514 determines whether the center of gravity is within the discomfort estimation region (lower left of the first threshold LD in Figure 3). If yes, the unit proceeds to step S207; if no (upper right of the second threshold LU in Figure 3), the unit proceeds to step S209.

[0068] In step S207, the estimation unit 514 determines whether a specified time (for example, about 5 to 10 seconds) has elapsed since the center of gravity entered the discomfort estimation region. If yes, the unit proceeds to step S208; otherwise, it returns to step S203. Note that in step S207, a specified number of times may be used instead of a specified time.

[0069] In step S208, the estimation unit 514 estimates that the emotion is unpleasant. Next, the process proceeds to step S217.

[0070] In step S209, the estimation unit 514 reads the point cloud data of the object to be inspected.

[0071] Next, in step S210, the estimation unit 514 calculates an area S (corresponding to the area of ​​ellipse E in Figure 3), which is a feature quantity corresponding to the width of the distribution of the point cloud data.

[0072] Next, in step S211, the estimation unit 514 reads the rest criteria.

[0073] Next, in step S212, the estimation unit 514 calculates the feature quantity, which is the distance m between the neutral criterion and the centroid.

[0074] Next, in step S213, the estimation unit 514 normalizes S and m so that their standard deviations are 1 and their mean is 0.

[0075] Next, in step S214, the estimation unit 514 calculates the value of the multiple regression equation by substituting the normalized S and m into the parameters of the multiple regression equation that have been set in advance.

[0076] Next, in step S215, the estimation unit 514 determines whether the value of the multiple regression equation is 0 or greater. If yes, it proceeds to step S216; otherwise, it returns to step S203.

[0077] In step S216, the estimation unit 514 estimates that the emotion is pleasant. Next, the process proceeds to step S217.

[0078] In step S217, the estimation unit 514 outputs the emotion estimation result. The control unit 515 then controls the display devices 61 to the communication devices 69 according to the emotion estimation result. For example, the control unit 515 displays a warning (such as fatigue, lack of sleep, or distraction) using a meter or sub-display on the display device 61 according to the emotion estimation result.

[0079] Furthermore, for example, the control unit 515 generates a warning sound from the sound device 62 according to the emotion estimation result. Also, for example, the control unit 515 emits a fragrance that soothes emotions from the fragrance device 63 according to the emotion estimation result. Also, for example, the control unit 515 changes posture or performs a massage by controlling the seat control mechanism 64 according to the emotion estimation result.

[0080] Furthermore, for example, the control unit 515 controls the drive mechanism 65, braking mechanism 66, and steering mechanism 67 according to the emotion estimation result to generate a sensory alarm, avoid an accident, or stop the vehicle on the shoulder of the road. Also, for example, the control unit 515 controls the air conditioning mechanism 68 to adjust the temperature and airflow or to ventilate the vehicle according to the emotion estimation result.

[0081] Furthermore, for example, if the control unit 515 detects an abnormality occurring inside the vehicle, it controls the communication device 69 according to the emotion estimation result to send an emergency notification or registered user notification to a designated external organization.

[0082] Furthermore, for example, the control unit 515 may notify the cloud computer 7 of the emotion estimation results and their associated information, and have them saved as personal data, life logs, anomaly notifications, etc., for future use.

[0083] Next, with reference to Figure 6, the third process performed by the information processing device 5 will be described. Figure 6 is a flowchart showing the third process performed by the information processing device 5 in this embodiment. The third process is to invalidate the emotion estimation result if the occupant 2 is speaking.

[0084] First, in step S31, the acquisition unit 511 reads respiratory data (current value) from the respiratory sensor 15.

[0085] Next, in step S32, the estimation unit 514 determines whether or not it has detected a respiratory peak based on the respiratory data. If yes, it proceeds to step S33; otherwise, it returns to step S31.

[0086] In step S33, the estimation unit 514 calculates the respiratory interval (current value).

[0087] Next, in step S34, the estimation unit 514 determines whether or not crew member 2 is speaking based on the breathing interval (current value). If yes, the unit proceeds to step S35; otherwise, it returns to step S31. Generally, the breathing interval shortens during speech. Therefore, in step S34, for example, if the breathing interval (current value) is below a predetermined breathing interval threshold, the unit determines that crew member 2 is speaking.

[0088] In step S35, the estimation unit 514 invalidates the emotion estimation result of crew member 2.

[0089] Thus, according to the vehicle system 1 of this embodiment, the neutral reference LN (Figure 3) at rest is calculated, the test point (the centroid of the Poincaré plot of the subject being tested) is compared with the first threshold LD and the second threshold LU, and if the state in which the test point is greater than the second threshold LU continues for a predetermined time, a different estimation logic is switched, thereby enabling the estimation of the occupant's emotions with high accuracy and with less computational effort based on biometric data. In other words, generally, the determination of pleasant emotions tends to be less accurate than the determination of unpleasant emotions, but in this embodiment, the determination of pleasant emotions can be made more accurate by switching to a different estimation logic for pleasant emotions.

[0090] Furthermore, as described above, the estimation unit 514 can further reduce the influence of individual differences and perform highly accurate emotion estimation by using the normalized area S and distance m and a pre-set multiple regression equation.

[0091] Furthermore, emotion estimation can be disabled during speech, when the accuracy of emotion estimation tends to decrease.

[0092] Furthermore, by avoiding computationally intensive methods such as the Fast Fourier Transform, the computational cost is significantly reduced.

[0093] Furthermore, by using the Poincaré plot method, the system is less susceptible to the effects of outliers caused by body movements. In other words, it has high noise immunity.

[0094] The program for realizing the functions of the above embodiment may be provided as a file in a format installable or executable format on the information processing device 5, recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk). Alternatively, the program may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Furthermore, the program may be provided or distributed via a network such as the Internet.

[0095] Although embodiments of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0096] For example, the biometric data used to estimate a subject's emotions is not limited to heart rate data; other biometric data, such as respiratory data, may also be used, as long as they fluctuate periodically and the length of the period tends to change depending on the subject's emotional changes.

[0097] In addition, in parallel with emotion estimation, the subject's level of arousal may be estimated using information such as electroencephalograms (EEGs) and skin action potentials (TEA). [Explanation of symbols]

[0098] 1...Vehicle system, 2...Occupant, 5...Information processing device, 7...Cloud computer, 11...Heart rate sensor, 12...Wearable device, 13...Authentication device, 14...Vehicle information sensor, 15...Respiratory sensor, 21...Seat, 22...Backrest, 51...Processing unit, 61...Display device, 62...Audio equipment, 63...Fragrance equipment, 64...Seat control mechanism, 65...Drive mechanism, 66...Braking mechanism, 67...Steering mechanism, 68...Air conditioning mechanism, 69...Communication equipment, 511...Acquisition unit, 512...Setting unit, 513...Calculation unit, 514...Estimation unit, 515...Control unit

Claims

1. A biosensor that measures the heart rate data of a subject, wherein the biosensor measures predetermined biometric data relating to the subject that fluctuates periodically, and the length of the period tends to change depending on the subject's emotional changes. An acquisition unit that acquires the biological data from the biological sensor, A neutral reference setting unit generates a Poincaré plot consisting of a point cloud on two axes, using the current value on the X axis and the value after a specified time has elapsed on the Y axis, based on the biological data of the subject at rest, and calculates the centroid position and sets it as the neutral reference. A threshold setting unit sets a first threshold at a predetermined position in the direction of the origin of the Poincaré plot with respect to the neutral criterion, and sets a second threshold at a predetermined position in the direction opposite to the direction of the origin of the Poincaré plot. A calculation unit generates a Poincaré plot based on the biological data of the subject to examination for a predetermined time period from the aforementioned biological data, and calculates the centroid position as the examination point. If the state in which the test point is between the first threshold and the second threshold continues for a predetermined period of time, it is presumed to be a neutral emotion. If the state in which the aforementioned test point is below the first threshold continues for a predetermined period of time, it is presumed to be an unpleasant emotion. An emotion estimation system comprising: an estimation unit that, if the state in which the test point is greater than the second threshold continues for a predetermined time, calculates the area of ​​an ellipse corresponding to the width of the distribution of the Poincaré plot corresponding to the test point, normalizes the area and the distance based on the area and the distance between the neutral criterion and the test point, and estimates that the emotion is pleasant if the value calculated by a multiple regression equation with the normalized area and distance as variables is greater than or equal to a predetermined threshold.

2. The system further includes a respiratory sensor that measures the respiratory data of the subject, The acquisition unit acquires the respiratory data from the respiratory sensor, The emotion estimation system according to claim 1, wherein the estimation unit determines whether the subject is speaking based on the respiratory data, and if it determines that the subject is speaking, it invalidates the estimation result of the subject's emotion.

3. An acquisition unit that acquires biological data from a biosensor that measures the heart rate data of a subject, wherein the biological data is predetermined biological data that fluctuates periodically and whose period length tends to change depending on changes in the subject's emotions, A neutral reference setting unit generates a Poincaré plot consisting of a point cloud on two axes, using the current value on the X axis and the value after a specified time has elapsed on the Y axis, based on the biological data of the subject at rest, and calculates the centroid position and sets it as the neutral reference. A threshold setting unit sets a first threshold at a predetermined position in the direction of the origin of the Poincaré plot with respect to the neutral criterion, and sets a second threshold at a predetermined position in the direction opposite to the direction of the origin of the Poincaré plot. A calculation unit generates a Poincaré plot based on the biological data of the subject to examination for a predetermined time period from the aforementioned biological data, and calculates the centroid position as the examination point. If the state in which the test point is between the first threshold and the second threshold continues for a predetermined period of time, it is presumed to be a neutral emotion. If the state in which the aforementioned test point is below the first threshold continues for a predetermined period of time, it is presumed to be an unpleasant emotion. An emotion estimation device comprising: an estimation unit that, if the state in which the test point is greater than the second threshold continues for a predetermined time, calculates the area of ​​an ellipse corresponding to the width of the distribution of the Poincaré plot corresponding to the test point, normalizes the area and the distance based on the area and the distance between the neutral standard and the test point, and estimates that the emotion is pleasant if the value calculated by a multiple regression equation with the normalized area and distance as variables is greater than or equal to a predetermined threshold.

4. A method for estimating emotion using an information processing device, wherein the information processing device is An acquisition step of acquiring biometric data from a biosensor that measures the heart rate data of a subject, wherein the biometric data is predetermined biometric data that fluctuates periodically and whose period length tends to change depending on changes in the subject's emotions, A neutral reference setting step is performed in which, based on the biological data of the subject at rest, a Poincaré plot consisting of a point cloud on two axes is generated using the current value on the X axis and the value after a specified time has elapsed on the Y axis, the centroid position is calculated and set as the neutral reference, A threshold setting step in which, with respect to the neutral criterion, a first threshold is set at a predetermined position in the direction of the origin of the Poincaré plot, and a second threshold is set at a predetermined position in the direction opposite to the direction of the origin of the Poincaré plot; A calculation step of generating a Poincaré plot based on the biological data of the subject to examination for a predetermined time period from the aforementioned biological data, and calculating the centroid position as the examination point, If the state in which the test point is between the first threshold and the second threshold continues for a predetermined period of time, it is presumed to be a neutral emotion. If the state in which the aforementioned test point is below the first threshold continues for a predetermined period of time, it is presumed to be an unpleasant emotion. An emotion estimation method comprising: an estimation step in which, if the state in which the test point is greater than the second threshold continues for a predetermined time, the area of ​​an ellipse corresponding to the width of the distribution of the Poincaré plot corresponding to the test point is calculated, the area and the distance are normalized based on the area and the distance between the neutral criterion and the test point, and if the value calculated by a multiple regression equation with the normalized area and distance as variables is greater than or equal to a predetermined threshold, the emotion is estimated to be pleasant.

5. Computers, An acquisition unit that acquires biological data from a biosensor that measures the heart rate data of a subject, wherein the biological data is predetermined biological data that fluctuates periodically and whose period length tends to change depending on changes in the subject's emotions, A neutral reference setting unit generates a Poincaré plot consisting of a point cloud on two axes, using the current value on the X axis and the value after a specified time has elapsed on the Y axis, based on the biological data of the subject at rest, and calculates the centroid position and sets it as the neutral reference. A threshold setting unit sets a first threshold at a predetermined position in the direction of the origin of the Poincaré plot with respect to the neutral criterion, and sets a second threshold at a predetermined position in the direction opposite to the direction of the origin of the Poincaré plot. A calculation unit generates a Poincaré plot based on the biological data of the subject to examination for a predetermined time period from the aforementioned biological data, and calculates the centroid position as the examination point. If the state in which the test point is between the first threshold and the second threshold continues for a predetermined period of time, it is presumed to be a neutral emotion. If the state in which the aforementioned test point is below the first threshold continues for a predetermined period of time, it is presumed to be an unpleasant emotion. A program to function as an estimation unit that, when the state in which the aforementioned test point is greater than the second threshold continues for a predetermined time, calculates the area of ​​an ellipse corresponding to the width of the distribution of the Poincaré plot corresponding to the test point, normalizes the area and the distance based on the area and the distance between the neutral criterion and the test point, and estimates that the emotion is pleasant if the value calculated by a multiple regression equation with the normalized area and distance as variables is greater than or equal to a predetermined threshold.

Images