Relaxation level determination device, relaxation level determination method, and relaxation induction device

Abstract

To achieve a technology of accurately determining a degree of relaxation from biological information and information on a facial expression of a subject.SOLUTION: In a relaxation degree determination device (control unit 1), a relaxation degree estimation unit (13) estimates a degree of relaxation of a subject by referring to a heart rate measurement value of the subject acquired by a heart rate measurement value acquisition unit (11), a relaxation degree correction unit (14) corrects the degree of relaxation on the basis of information on a facial expression of the subject acquired by a facial expression information acquisition unit (12), and the corrected degree of relaxation is determined as a degree of relaxation of the subject.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a relaxation level determination device, a relaxation level determination method, and a relaxation induction device. [Background technology]

[0002] A technique for estimating human emotions is known as one technique for controlling machines that takes human emotions into consideration. One such technique is to estimate the emotion of a subject based on the facial expression and biometric information of the subject, and combine the obtained estimation results based on weighting by the subject to obtain a final estimation result of the subject's emotion (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-59107 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional techniques leave room for further study in terms of estimating the subject's degree of relaxation (relaxation level). For example, the above-mentioned Patent Document 1 does not disclose a method for determining the subject's degree of relaxation. In addition, in the technique of the above-mentioned Patent Document 1, the final estimation result includes weighting by the subject for the estimation results based on each index, and therefore may lack objectivity. For this reason, if the subject is not aware of the relationship between the subject's emotions and the detected indexes, inappropriate weighting may be applied, which may reduce the accuracy of the final estimation result.

[0005] An object of one aspect of the present invention is to realize a technique for accurately determining a level of relaxation from a subject's biometric information and facial expression information. [Means for solving the problem]

[0006] In order to solve the above problems, a relaxation level assessment device according to one embodiment of the present invention comprises a heart rate measurement value acquisition unit that acquires a heart rate measurement value of the person being assessed, a facial expression information acquisition unit that acquires information on the facial expression of the person being assessed, a relaxation level estimation unit that estimates the relaxation level of the person being assessed by referring to the heart rate measurement value acquired by the heart rate measurement value acquisition unit, and a relaxation level correction unit that corrects the relaxation level estimated by the relaxation level estimation unit based on the facial expression information acquired by the facial expression information acquisition unit, and judges the relaxation level corrected by the relaxation level correction unit to be the relaxation level of the person being assessed.

[0007] In addition, in order to solve the above-mentioned problems, a relaxation level assessment method according to one embodiment of the present invention includes a heart rate measurement value acquisition step of acquiring a heart rate measurement value of the person being assessed, a facial expression information acquisition step of acquiring information on the facial expression of the person being assessed, a relaxation level estimation step of estimating the relaxation level of the person being assessed by referring to the heart rate measurement value acquired in the heart rate measurement value acquisition step, and a relaxation level correction step of correcting the relaxation level estimated in the relaxation level estimation step based on the facial expression information acquired in the facial expression information acquisition step, and assessing the relaxation level corrected in the relaxation level correction step as the relaxation level of the person being assessed.

[0008] With these configurations, it is possible to determine the degree of relaxation by appropriately using the heart rate measurement values ​​and facial expression information of the person being assessed, thereby realizing a technology that can accurately determine the degree of relaxation from the biometric information and facial expression information of the person being assessed.

[0009] The relaxation level correction unit may correct the level of relaxation of the person being assessed to be higher in accordance with facial expression information of the person being assessed that is more strongly correlated with a high level of relaxation, and may correct the level of relaxation of the person being assessed to be lower in accordance with facial expression information of the person being assessed that is less correlated with a high level of relaxation.

[0010] This configuration is even more effective in terms of increasing the accuracy of the determination of the relaxation level by appropriately reflecting facial expression information in the determination of the relaxation level.

[0011] The relaxation level correction unit may further classify the type of high relaxation level of the person being assessed according to facial expression information that is more strongly correlated with a high level of relaxation among facial expression information of the person being assessed.

[0012] This configuration is even more effective in terms of increasing the accuracy of determining the relaxation level by appropriately classifying the type of relaxation level from facial expression information.

[0013] The relaxation level assessment device may further include a frequency component extraction unit that extracts ultra-low frequency components and high frequency components of the heart rate fluctuations of the person being assessed by referring to the heart rate measurement values ​​acquired by the heart rate measurement value acquisition unit, and the relaxation level estimation unit may estimate the relaxation level of the person being assessed based on the ultra-low frequency components and high frequency components extracted by the frequency component extraction unit.

[0014] This configuration is more effective in terms of reducing the information processing load in the process of estimating the relaxation level, and increasing the accuracy of estimating the relaxation level.

[0015] The frequency component extraction unit may determine the magnitude of the heart rate fluctuation of the person being assessed by referring to the heart rate measurement value acquired by the heart rate measurement value acquisition unit, and the relaxation level estimation unit may estimate the relaxation level of the person being assessed based on the result of the heart rate fluctuation determination by the frequency component extraction unit, based on the ultra-low frequency components and the high frequency components when the heart rate fluctuation is large, and may estimate the relaxation level of the person being assessed based on the ultra-low frequency components when the heart rate fluctuation is small.

[0016] This configuration is even more effective in terms of reducing the information processing load in the process of estimating the relaxation level, and increasing the accuracy of estimating the relaxation level.

[0017] The frequency component extraction section may extract ultra-low frequency components of the heart rate fluctuation of the person to be assessed, based on a moving average of the heart rate fluctuation of the person to be assessed.

[0018] This configuration can clarify the characteristics of the heart rate fluctuations of the person being assessed, and is even more effective in terms of increasing the accuracy of estimating the degree of relaxation.

[0019] The relaxation level estimation section may estimate the relaxation level of the person to be determined by referring to the reference data acquired by the reference data acquisition section.

[0020] This configuration is even more effective in terms of accurately grasping the state of the subject's heart rate measurement value when estimating the relaxation level, and increasing the accuracy of estimating the relaxation level.

[0021] The relaxation level assessment device may further include an external factor information acquisition unit that acquires external factor information including at least one of environmental information indicating the environment in which the person being assessed is present and date and time information, and the relaxation level estimation unit may estimate the relaxation level of the person being assessed by referring to the external factor information acquired by the external factor information acquisition unit.

[0022] This configuration allows the influence of external factors to be appropriately reflected in the estimation of the degree of relaxation of the person to be assessed, and is therefore even more effective in terms of improving the accuracy of the estimation of the degree of relaxation.

[0023] The person to be determined may be a passenger in a moving body, and the heart rate measurement value acquisition unit may acquire the heart rate measurement value of the person to be determined from a heart rate sensor mounted on the moving body, and the facial expression information acquisition unit may acquire facial expression information of the person to be determined from a camera mounted on the moving body.

[0024] This configuration is even more effective from the standpoint of more easily determining the degree of relaxation of the person being assessed in the moving body, and from the standpoint of constantly utilizing the results of the determination of the degree of relaxation of the person being assessed in the moving body in the operation of the moving body.

[0025] Furthermore, in order to solve the above-mentioned problems, a relaxation induction device according to one embodiment of the present invention comprises a heart rate measurement value acquisition unit that acquires heart rate measurement values ​​of the person being assessed, a facial expression information acquisition unit that acquires information on the facial expression of the person being assessed, a relaxation level estimation unit that estimates the relaxation level of the person being assessed by referring to the heart rate measurement values ​​acquired by the heart rate measurement value acquisition unit, a relaxation level correction unit that corrects the relaxation level estimated by the relaxation level estimation unit based on the facial expression information acquired by the facial expression information acquisition unit, and a stimulation provision unit that provides the person being assessed with a stimulation according to the relaxation level corrected by the relaxation level correction unit.

[0026] According to this configuration, since it is possible to determine the relaxation level by appropriately using the measured heart rate value and facial expression information of the person to be assessed, it is possible to realize a technology for accurately determining the relaxation level from the biological information and facial expression information of the person to be assessed. Also, according to the above configuration, it is possible to induce the person to be assessed into a more desirable relaxed state by providing the person to be assessed with a stimulus determined by referring to the highly accurate relaxation level. Effect of the Invention

[0027] According to one aspect of the present invention, it is possible to realize a technique for accurately determining a relaxation level of a person based on biometric information and facial expression information of the person to be determined. [Brief description of the drawings]

[0028] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a relaxation degree determination device according to a first embodiment of the present invention. [Diagram 2] 5 is a flowchart showing the flow of an example of a relaxation degree determination process performed by the relaxation degree determination device according to the first embodiment of the present invention. [Diagram 3] 5 is a flowchart showing the flow of a first example of a process for correcting a relaxation level based on facial expression information according to the first embodiment of the present invention. [Figure 4]FIG. 2 is a diagram showing a typical example of a screen displaying the result of determination of a relaxation level according to a first example of correction of a relaxation level in the first embodiment of the present invention. [Diagram 5] 13 is a flowchart showing the flow of a second example of the process of correcting a relaxation level based on facial expression information according to the first embodiment of the present invention. [Figure 6] FIG. 13 is a diagram showing a typical example of a screen displaying the result of determination of a relaxation level in accordance with a second example of correction of a relaxation level in the first embodiment of the present invention. [Figure 7] FIG. 11 is a block diagram illustrating a schematic functional configuration of a relaxation degree determination device according to a second embodiment of the present invention. [Figure 8] 10 is a flowchart showing the flow of a first example of a process for estimating a relaxation level based on a heart rate measurement value according to the second embodiment of the present invention. [Figure 9] 13 is a flowchart showing the flow of a second example of a process for estimating a relaxation level based on a heart rate measurement value according to the second embodiment of the present invention. [Figure 10] FIG. 11 is a block diagram illustrating a schematic functional configuration of a relaxation degree determination device according to a third embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing a schematic example of an arrangement of devices around a subject in a third embodiment of the present invention. [Figure 12] 10 is a flowchart showing the flow of an example of a process for determining a relaxation level by a relaxation level determining device according to a third embodiment of the present invention. [Figure 13] FIG. 11 is a block diagram illustrating a functional configuration of a relaxation induction device according to a fourth embodiment of the present invention. [Figure 14] 10 is a flowchart showing an example of a flow from a relaxation degree determination process to a relaxation induction process by a relaxation induction device according to embodiment 4 of the present invention. [Figure 15] 13 is a flowchart showing the flow of an example of a relaxation induction process according to the fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] [Embodiment 1] Hereinafter, one embodiment of the present invention will be described in detail.

[0030] [Functional configuration] Fig. 1 is a block diagram showing a schematic functional configuration of a relaxation level assessment device according to a first embodiment of the present invention. As shown in Fig. 1, the relaxation level assessment device of this embodiment is composed of a control unit 1. The control unit 1 includes a heart rate measurement value acquisition unit 11 that acquires a heart rate measurement value of a person to be assessed, a facial expression information acquisition unit 12 that acquires information on a facial expression of the person to be assessed, a relaxation level estimation unit 13 that estimates a relaxation level of the person to be assessed by referring to the heart rate measurement value acquired by the heart rate measurement value acquisition unit 11, and a relaxation level correction unit 14 that corrects the relaxation level estimated by the relaxation level estimation unit 13 based on the facial expression information acquired by the facial expression information acquisition unit 12.

[0031] The control unit 1 is connected to a heartbeat measuring device 15, a camera 16, a storage unit 17, and a notification device 18. The heartbeat measuring device 15 can be appropriately selected within a range in which the heartbeat of the person to be assessed can be measured. Examples of the heartbeat measuring device 15 include heartbeat sensors arranged in accordance with each seat of a moving body (vehicle) described below, and a wristwatch-type vital information measuring device that can be worn by the person to be assessed.

[0032] The camera 16 is a device that captures the facial expression of the person to be judged. The camera 16 is usually placed in front of the person to be judged so that the entire face of the person to be judged can be captured. The image captured by the camera 16 may be a still image of the face of the person to be judged, or a video image.

[0033] The storage unit 17 stores information for executing the determination (estimation and correction) of the degree of relaxation. Examples of the information include a program for carrying out the determination of the degree of relaxation, and numerical values, formulas, and maps used in the determination. The storage unit 17 can also store the measured values ​​and acquired values ​​in the determination of the degree of relaxation in association with the person to be determined.

[0034] The notification device 18 is a device for notifying the person to be judged. Examples of the notification device 18 include an audio output device that outputs audio for notifying the person to be judged, and a display device that displays characters and images for notifying the person to be judged.

[0035] [Example of relaxation level assessment] 2 is a flowchart showing an example of a process for determining a degree of relaxation performed by the relaxation degree determination device according to the first embodiment of the present invention. The heartbeat measurement device 15 measures the heartbeat of the person to be determined. The camera 16 captures the facial expression of the person to be determined from the front.

[0036] In step S101, the heartbeat measurement value acquisition unit 11 obtains output information of the subject's heartbeat being measured by the heartbeat measurement device 15. In this way, the heartbeat measurement value acquisition unit 11 acquires the subject's heartbeat measurement value. The "heartbeat measurement value" is information on the measurement results of the subject's heartbeat and its components. More specifically, the heartbeat measurement value may be the heartbeat data itself measured by the heartbeat measurement device 15, or may be data extracted from the heartbeat data. Examples of the latter include the RR-I value and HF value described later.

[0037] In step S102, the facial expression information acquisition unit 12 acquires output information of a front image of the face of the person being judged captured by the camera 16. In this way, the facial expression information acquisition unit 12 acquires facial expression information of the person being judged. "Facial expression information" is information that indicates the facial expression of the person being judged, and is, for example, information of an image of a main part or substantially the entire face of the person being judged.

[0038] In step S103, the relaxation level estimation unit 13 estimates the relaxation level of the person to be assessed by referring to the heart rate measurement value acquired by the heart rate measurement value acquisition unit 11. The estimation of the relaxation level in step S103 may be performed by any method within the scope of methods that refer to the heart rate measurement value of the person to be assessed. The method of estimating the relaxation level in step S103 may be a method in which the quantitatively expressed relaxation level is uniquely determined by the heart rate measurement value. For example, the relaxation level estimation unit 13 may use the heart rate of the person to be assessed, which is substantially unchanged, as a reference value, and estimate a lower relaxation level as the heart rate increases, and estimate a higher relaxation level as the heart rate decreases. The estimation of the relaxation level using the heart rate measurement value will be described in more detail in a later embodiment.

[0039] In step S104, the relaxation level correction section 14 corrects the relaxation level estimated in step S103 based on the facial expression information acquired in step S102. Then, the control section 1 determines the relaxation level corrected by the relaxation level correction section 14 as the relaxation level of the person to be determined.

[0040] <First example of correction of relaxation level based on facial expression information> Here, the correction of the relaxation level based on facial expression information will be described. Fig. 3 is a flow chart showing the flow of a first example of the process of correcting the relaxation level based on facial expression information in the first embodiment of the present invention.

[0041] In step S201, the relaxation level correction section 14 judges the emotion of the person to be judged based on the facial expression.

[0042] Here, determination of emotions from facial expressions will be described. There are no limitations on the method of determining emotions from facial expressions, and any known method can be adopted. For example, emotion determination from facial expressions can be performed using "faceproc." "faceproc" is a facial expression API (application programming interface), and is an expression recognition API that uses artificial intelligence to infer basic emotions from facial images.

[0043] The types and number of emotions to be determined from facial expressions can be appropriately determined within a range in which the degree of relaxation of the person to be determined can be determined. The emotions may be only emotions that represent the state in which the person to be determined is relaxed, or may include both emotions that represent the state in which the person to be determined is relaxed and emotions that represent the state in which the person to be determined is not relaxed. Examples of the types of emotions used in determining facial expressions include seven emotions, namely anger, disgust, fear, happiness, neutral or emotionless or indifferent (hereinafter also referred to as "neutral, etc."), sadness, and surprise, or eight emotions that may further include contempt in some cases.

[0044] In step S202, the relaxation level correction unit 14 acquires a correction value for the relaxation level according to the emotion judgment result determined from the facial expression of the person being judged. The emotion judgment result based on the facial expression and the correction of the relaxation level will be described below, and an example of acquiring the correction value will be described below.

[0045] Usually, when the person being evaluated is relaxed, the parasympathetic nervous system becomes dominant, and the person's heart rate slows down. In a relaxed state, the accuracy of the determination based on the heart rate measurement value tends to be higher than the accuracy of the determination based on the facial expression. On the other hand, when the person being evaluated is irritated, the heart rate increases, but in daily life, the feeling of irritation strong enough to actually cause the heart rate to increase rarely occurs. Therefore, the accuracy of the determination of the feeling of irritation based on the facial expression tends to be higher than the accuracy of the determination based on the heart rate measurement value.

[0046] In this way, when judging the degree of relaxation of a person, for emotions for which the accuracy of judging emotions based on facial expressions tends to be higher, the judgment result based on heart rate measurements is actively corrected by the judgment result based on emotions based on facial expressions, thereby making it possible to further increase the accuracy of judging the degree of relaxation.

[0047] When emotion determination is performed for multiple emotions, the emotion determination result can be expressed as a ratio of the emotion expressed by the facial expression. In this case, the total number of determination values ​​for each emotion is 1. The emotion value with the highest determination value among the determination values ​​for multiple emotions is the emotion determination result based on the facial expression of the person being determined. The quantification of emotions will be described later.

[0048] On the other hand, the types of emotions are classified into two types: facial expression information that is more strongly correlated with the level of relaxation, and facial expression information that is less strongly correlated with the level of relaxation. For example, the former is information on emotions that represent a state in which the person being assessed is relaxed (hereinafter also referred to as "positive emotions"), and the latter is information on emotions that represent a state in which the person being assessed is not relaxed (hereinafter also referred to as "negative emotions"). Such classification of emotions can be performed, for example, by classifying emotions that are more strongly related to a relaxed state as positive emotions and other emotions as negative emotions. In the case of the above eight emotions, it is possible to classify two emotions, happiness and neutral, as positive emotions and other emotions as negative emotions.

[0049] Then, a correction value for a positive emotion and a correction value for a negative emotion are set in advance. For example, the correction value is set to +0.5 for a positive emotion and to -0.5 for a negative emotion.

[0050] A correction value for the degree of relaxation corresponding to the judgment result of the emotion based on the facial expression can be obtained by determining whether the emotion is positive or negative from the judgment result of the emotion based on the facial expression and selecting a correction value corresponding to the judgment result.

[0051] The following describes the processing example of correcting the relaxation level based on facial expression information. In step S203, the relaxation level is corrected by adding the correction value acquired in step S202 to the relaxation level based on the heart rate measurement value estimated in step S103. The corrected relaxation level is determined to be the relaxation level of the person being assessed.

[0052] For example, if the relaxation level based on the heart rate measurement value is 0.4 and the correction value for positive emotion is +0.5, the relaxation level of the person to be assessed is determined to be 0.9. Also, for example, if the upper limit of the relaxation level is 1 and the lower limit is -1, and the relaxation level after adding the correction value exceeds the upper limit, the relaxation level after correction may be set to the upper limit, i.e., 1, and if the relaxation level after adding the correction value falls below the lower limit, the relaxation level after correction may be set to the lower limit, i.e., -1.

[0053] The degree of relaxation can be notified to the person being assessed by notification device 18. If notification device 18 is a display device such as a liquid crystal display, it is possible to notify the person being assessed of the result of the relaxation degree assessment by an image, as will be described below.

[0054] <First example of notification of judgment result> FIG. 4 is a diagram showing a typical example of a screen displaying the result of determination of the relaxation level according to the first example of correction of the relaxation level in the first embodiment of the present invention.

[0055] 4, screen 180 is the screen of notification device 18, which is a liquid crystal display device. Screen 180 is vertically divided into two sections according to the level of relaxation level.

[0056] The character image 181 represents the subject in a relaxed state, and is displayed when the degree of relaxation is greater than a predetermined value, that is, when the subject is determined to be in a relaxed state. The character image 181 is displayed, for example, according to the determined degree of relaxation. The character image 181 may be displayed, for example, at a position in the vertical direction of the screen 180 according to the degree of relaxation, or may be displayed at a predetermined position at the top of the screen 180, but may be displayed larger as the degree of relaxation increases.

[0057] The character image 182 represents a state in which the person being assessed is not relaxed, and is displayed when the degree of relaxation is less than a predetermined value. Like the character image 181, the character image 182 may be displayed at a position in the vertical direction of the screen 180 according to the degree of relaxation, or may be displayed at a predetermined position on the screen 180 but may be displayed larger as the degree of relaxation decreases.

[0058] Thus, in a first example of the relaxation degree correction in this embodiment, the relaxation degree correction unit 14 corrects the relaxation degree of the person being judged to be higher in accordance with facial expression information that is more strongly correlated with the level of relaxation degree among the facial expression information of the person being judged, and corrects the relaxation degree of the person being judged to be lower in accordance with facial expression information that is less strongly correlated with the level of relaxation degree. Therefore, since the facial expression information that is more strongly correlated with the relaxation degree is reflected in the relaxation degree judgment result, it is possible to judge the relaxation degree with higher accuracy compared to a judgment of the relaxation degree based only on the heart rate measurement value or only on the facial expression information.

[0059] <Second example of correction of relaxation level based on facial expression information> In this embodiment, the correction of the relaxation level based on the facial expression information may involve classifying the judgment result according to the type of emotion based on the facial expression of the person being judged. Fig. 5 is a flowchart showing the flow of a second example of the process of correcting the relaxation level based on the facial expression information in the first embodiment of the present invention. The flowchart in Fig. 5 is the same as the flowchart shown in Fig. 4, except that step S301 is included between steps S201 and S202.

[0060] In step S301, the relaxation level correction unit 14 classifies the determination result of the emotion determined by the facial expression in step S201 according to the type of the emotion. For example, in step S301, the relaxation level correction unit 14 classifies the determination of a high relaxation level, which will be described below, into active relaxation and calm relaxation.

[0061] A relaxed state usually includes a state of enjoying something. In this case, the heart rate measurement value increases, but the feeling of irritation does not actually occur. For such feelings that are more strongly correlated with a relaxed state, even if the heart rate increases, the accuracy of the judgment based on facial expression tends to be higher than that based on the heart rate measurement value.

[0062] For example, a person may feel calm and refreshed by talking with a like-minded friend or watching a comedy video or other content that makes the person laugh. The state of relaxation in which the parasympathetic nervous system is dominant and the heart rate slows down can be called sedative relaxation including drowsiness, whereas the state of relaxation in which the person is having fun can be called active relaxation. Reflecting such states of relaxation that can be classified by emotions in determining the degree of relaxation is also included in the embodiments of the present invention.

[0063] For example, of the two emotions, happiness and neutral, which are classified as positive emotions, happiness tends to be expressed as a clear smile on the face. Therefore, when the judgment value of the positive emotion is the judgment value of happiness, the state of high relaxation in this case can be classified as a state of active relaxation among the high relaxation states. Note that, when the judgment value of the positive emotion is the judgment value of neutral, the state of high relaxation in this case can be classified as a state of calm relaxation among the high relaxation states.

[0064] <Second example of notification of judgment result> Fig. 6 is a diagram showing a typical example of a screen showing the result of determining the relaxation level according to a second example of the correction of the relaxation level in the first embodiment of the present invention. In Fig. 6, the diagram on the left side of the page is an example of a screen when notifying the relaxation level of the above-mentioned sedative relaxation, and the diagram on the right side is an example of a screen when notifying the relaxation level of the above-mentioned active relaxation.

[0065] In this example, the character image 181 represents a state of calm relaxation among the states in which the person being judged is relaxed. In this example, the character image 183 represents a state of active relaxation among the states in which the person being judged is relaxed. Like the character images 181 and 182, the character image 183 may be displayed at a position in the vertical direction of the screen 180 according to the degree of relaxation, or may be displayed at a predetermined position at the top of the screen 180 with a relative size according to the numerical value of the degree of relaxation.

[0066] Thus, in the second example of relaxation degree correction in this embodiment, the relaxation degree correction unit 14 further classifies the type of high relaxation state of the person being assessed according to the facial expression information of the person being assessed that has a stronger correlation with the level of relaxation among the facial expression information of the person being assessed. Therefore, it becomes possible to more clearly grasp the high relaxation state of the person being assessed from the relaxation degree judgment, and the accuracy of the relaxation degree judgment is improved.

[0067] [Summary of embodiment 1] The control unit (1, relaxation level assessment device) of this embodiment includes a heart rate measurement value acquisition unit (11) that acquires the heart rate measurement value of the person being assessed, a facial expression information acquisition unit (12) that acquires information on the facial expression of the person being assessed, a relaxation level estimation unit (13) that estimates the relaxation level of the person being assessed by referring to the heart rate measurement value acquired by the heart rate measurement value acquisition unit, and a relaxation level correction unit (14) that corrects the relaxation level estimated by the relaxation level estimation unit based on the facial expression information acquired by the facial expression information acquisition unit, and judges the relaxation level corrected by the relaxation level correction unit to be the relaxation level of the person being assessed.

[0068] In addition, the relaxation level assessment method according to this embodiment includes a heart rate measurement value acquisition step (step S101) for acquiring a heart rate measurement value of the person being assessed, a facial expression information acquisition step (step S102) for acquiring information on the facial expression of the person being assessed, a relaxation level estimation step (step S103) for estimating the relaxation level of the person being assessed by referring to the heart rate measurement value acquired in the heart rate measurement value acquisition step, and a relaxation level correction step (step S104) for correcting the relaxation level estimated in the relaxation level estimation step based on the facial expression information acquired in the facial expression information acquisition step, and the relaxation level corrected in the relaxation level correction step is judged to be the relaxation level of the person being assessed.

[0069] Therefore, according to the present embodiment, it is possible to determine the relaxation level of the subject by appropriately using both the heart rate measurement value and the facial expression information of the subject, and thus it is possible to realize a technology for accurately determining the relaxation level of the subject from the biometric information and the facial expression information of the subject.

[0070] In this embodiment, the relaxation level correction unit corrects the degree of relaxation of the person being assessed to be higher in accordance with facial expression information that is more strongly correlated with the level of relaxation among the facial expression information of the person being assessed, and corrects the degree of relaxation of the person being assessed to be lower in accordance with facial expression information that is less correlated with the level of relaxation. This configuration is even more effective in terms of improving the accuracy of the determination of the degree of relaxation by appropriately reflecting the facial expression information in the determination of the degree of relaxation.

[0071] In addition, in this embodiment, the relaxation level correction unit further classifies the type of the highly relaxed state of the person being assessed according to the facial expression information of the person being assessed that is more strongly correlated with the level of relaxation. This configuration is even more effective in terms of increasing the accuracy of the assessment of the relaxation level by appropriately classifying the type of relaxation level from the facial expression information.

[0072] [Embodiment 2] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0073] Fig. 7 is a block diagram showing a schematic functional configuration of a relaxation level assessment device according to embodiment 2 of the present invention. As shown in Fig. 7, the control unit 2 has the same functional configuration as the control unit 1 in embodiment 1, except that the control unit 2 further includes a frequency component extraction unit 21 that extracts ultra-low frequency components and high frequency components of the heart rate fluctuations of the person to be assessed by referring to the heart rate measurement values ​​acquired by the heart rate measurement value acquisition unit 11.

[0074] [Example of relaxation level assessment] The control unit 2 determines the relaxation level in the same manner as the control unit 1 of embodiment 1, except that the control unit 2 performs a process of estimating the relaxation level based on heart rate measurement values ​​(step S103 in FIG. 2) as described below. The estimation of the relaxation level based on heart rate measurement values ​​in this embodiment will be described below.

[0075] <First example of estimating relaxation level based on heart rate measurement> FIG. 8 is a flowchart showing the flow of a first example of a process for estimating a relaxation level based on a heart rate measurement value according to the second embodiment of the present invention.

[0076] The frequency component extraction unit 21 judges the magnitude of the heartbeat fluctuation of the subject with reference to the heartbeat measurement value acquired by the heartbeat measurement value acquisition unit 11. For example, in step S401, the frequency component extraction unit 21 extracts an ultra-low frequency component (RR-I value) and a high frequency component (HF value) of the heartbeat fluctuation of the subject with reference to the heartbeat measurement value acquired by the heartbeat measurement value acquisition unit 11.

[0077] The RR-I value is the length (seconds) of the heartbeat interval of the subject. The RR-I value of heartbeat fluctuation is the heartbeat fluctuation expressed in a band of 0.0033 to 0.04 Hz. The HF value is a value representing the high frequency component (0.15 to 0.4 Hz) in the heartbeat fluctuation of the subject, and is the sum of the power values ​​in the range of 0.15 to 0.4 (Hz) (the area of ​​the corresponding range in the spectrum data). The RR-I value and the HF value can be extracted from the heartbeat data by known techniques.

[0078] In step S402, the frequency component extraction unit 21 judges whether the acquired heartbeat fluctuation is greater than or smaller than a threshold. The "threshold" here is a threshold for classifying data with large heartbeat fluctuation from data with small heartbeat fluctuation based on a moving average of measurement results of multiple RR-I values, which will be described later.

[0079] The relaxation level estimation section 13 estimates the relaxation level of the person to be assessed based on the RR-I value and the HF value extracted by the frequency component extraction section 21. For example, if the heart rate fluctuation is equal to or greater than the threshold value (YES in step S402), the control section 2 reads out the following formula 1 from the storage section 17, and the relaxation level estimation section 13 calculates the relaxation level using the following formula 1 in step S403. Note that in the following formulas 1 and 2, "D relax " indicates the degree of relaxation, and "V RR-I " represents the RR-I value, and "V HF " represents the HF value. Also, α 1、 α2, β1, β 2、 γ1 is a constant that can be positive or negative. D relax =α1×V RR-I +β1×V HF +γ1 (formula 1)

[0080] If the heartbeat fluctuation is less than the threshold value (NO in step S402), the control unit 2 reads out the following formula 2 from the storage unit 17, and the relaxation level estimation unit 13 calculates the relaxation level using the following formula 2 in step S404. D relax =α2×V RR-I +β2 (formula 2)

[0081] The above formulas 1 and 2 are obtained as follows. First, the moving average of the measurement results of a plurality of RR-I values ​​is used to stratify data into data with large heartbeat fluctuations and data with small heartbeat fluctuations. The moving average of the RR-I value can be obtained by calculating the moving average of the RR-I at an appropriate interval (for example, 20 seconds). Then, 0.04 seconds is set as a threshold value for the amplitude of the smoothed RR-I value per 160 seconds, and data whose amplitude per 160 seconds appropriately extracted exceeds the threshold value is classified as data with large heartbeat fluctuations, and data below the threshold value is classified as data with small heartbeat fluctuations. The above formulas 1 and 2 are derived by performing multiple regression analysis on each of the stratified data groups. Alternatively, the above formulas 1 and 2 are derived by performing multiple regression analysis on each of the stratified data groups using the difference from the reference value.

[0082] As described above, when a person is relaxed, the parasympathetic nerves become dominant and the heart rate tends to slow down. In the first example of the present embodiment, as described above, the degree of relaxation of the person is estimated by utilizing the correlation between the magnitude of the amplitude of the heart rate fluctuation of the person. Therefore, the first example described above is advantageous in further reducing the load of information processing in the estimation process of the degree of relaxation by omitting the frequency analysis in the calculation of HF, and further increasing the accuracy of the estimation of the degree of relaxation.

[0083] <Second example of estimating relaxation level based on heart rate measurement> In this embodiment, the estimation of a relaxation degree based on a heart rate measurement value may be performed as follows: Fig. 9 is a flowchart showing the flow of a second example of the process of estimating a relaxation degree based on a heart rate measurement value in embodiment 2 of the present invention. The second example differs from the first example described above in the process after the RR-I value and the HF value are acquired in step S401.

[0084] In step S501, the frequency component extraction unit 21 smoothes the RR-I value. For example, the frequency component extraction unit 21 calculates a moving average of the RR-I value at an appropriate interval (for example, 20 seconds). By calculating the moving average at an appropriate time interval, the RR-I value of the subject is appropriately and clearly represented.

[0085] In step S502, the relaxation level estimation section 13 judges whether the RR-I value smoothed by the frequency component extraction section 21 is larger than or smaller than a threshold value. For example, the relaxation level estimation section 13 sets a threshold value of, for example, 0.04 seconds for the amplitude of the RR-I value per 160 seconds, appropriately extracts the amplitude of the RR-I value per 160 seconds, and judges whether the extracted amplitude exceeds the threshold value.

[0086] If the amplitude per unit time of the RR-I value is greater than the threshold value in step S502 (YES in step S502), then in step S503, the relaxation level estimation section 13 judges the relaxation level of the person to be assessed based on the HF value. More specifically, if the amplitude is greater than the threshold value, the relaxation level estimation section 13 reads out an equation showing the correlation between the HF value and the relaxation level from the storage section 17, and calculates the relaxation level of the person to be assessed using the equation. Note that the "equation showing the correlation" can be found, for example, by the least squares method from multiple measurement results.

[0087] If the amplitude per unit time of the RR-I value is equal to or less than the threshold value in step S502 (NO in step S502), the relaxation level estimation section 13 judges the relaxation level of the person to be assessed based on the RR-I value in step S504. More specifically, the relaxation level estimation section 13 reads out an equation indicating the correlation between the RR-I value and the relaxation level from the storage section 17 when the amplitude is equal to or less than the threshold value, and calculates the relaxation level of the person to be assessed using the equation.

[0088] Thus, in the second example, the frequency component extraction section 21 extracts the RR-I value of the heart rate fluctuation of the person to be assessed based on the moving average of the heart rate fluctuation of the person to be assessed. Using this as an index, the relaxation level estimation section 13 estimates the relaxation level of the person to be assessed based on the RR-I value and the HF value extracted by the frequency component extraction section 21.

[0089] In the second example, the type of heart rate measurement value having a higher correlation with the subject's relaxation level is determined from the RR-I value and HF value of the subject according to the magnitude of the amplitude of the subject's heart rate fluctuation.Then, the correlation between the determined type of heart rate measurement value and the relaxation level is used to estimate the subject's relaxation level.Therefore, like the first example, the second example also reduces the information processing load in the relaxation level estimation process and is advantageous in increasing the accuracy of the relaxation level estimation.

[0090] [Summary of embodiment 2] The control unit (2) of this embodiment further includes a frequency component extraction unit (21) that extracts ultra-low frequency components and high frequency components of the heartbeat fluctuations of the subject by referring to the heartbeat measurement values ​​acquired by the heartbeat measurement acquisition unit.The relaxation level estimation unit estimates the relaxation level of the subject based on the ultra-low frequency components and high frequency components extracted by the frequency component extraction unit.Such a configuration is more effective from the viewpoints of reducing the information processing load in the relaxation level estimation process and increasing the accuracy of the relaxation level estimation.

[0091] In this embodiment, the frequency component extraction unit may determine the magnitude of the heart rate fluctuation of the subject by referring to the heart rate measurement value acquired by the heart rate measurement value acquisition unit. In this case, the relaxation level estimation unit may estimate the relaxation level of the subject based on the result of the heart rate fluctuation determination by the frequency component extraction unit, based on the ultra-low frequency component and the high frequency component when the heart rate fluctuation is large, and may estimate the relaxation level of the subject based on the ultra-low frequency component when the heart rate fluctuation is small. Such a configuration is even more effective in terms of reducing the information processing load in the relaxation level estimation process and increasing the accuracy of the relaxation level estimation.

[0092] In addition, in this embodiment, the frequency component extracting section may extract ultra-low frequency components of the heart rate fluctuation of the subject based on a moving average of the heart rate fluctuation of the subject. Such a configuration can clarify the characteristics of the heart rate fluctuation of the subject, and is more effective in terms of improving the accuracy of the estimation of the relaxation level.

[0093] [Embodiment 3] Another embodiment of the present invention will be described below. For ease of explanation, the same reference numerals are used for members having the same functions as those described in the above embodiment, and the description thereof will not be repeated. In this embodiment, the technology for determining the relaxation level of the present invention is applied to a passenger in a vehicle, which is a moving object.

[0094] [Functional configuration] Fig. 10 is a block diagram showing a schematic functional configuration of a relaxation degree assessment device according to a third embodiment of the present invention. As shown in Fig. 10, the control unit 3 has the same functional configuration as the control unit 1 of the first embodiment described above, except that it further includes a reference data acquisition unit 31 that acquires reference data that is a reference for the heart rate measurement value of the person to be assessed, and an external factor information acquisition unit 32 that acquires external factor information. The external factor information is information that includes at least one of environmental information that indicates the environment in which the person to be assessed exists and date and time information.

[0095] The control unit 4 is further connected to an in-vehicle sensor 33 and an input device 34. The in-vehicle sensor 33 is a sensor mounted on the vehicle for controlling the driving of the vehicle and detecting the environment inside and outside the vehicle. Examples of the in-vehicle sensor 33 include a temperature and humidity sensor, a seat pressure sensor, and a seat reclining angle sensor.

[0096] The input device 34 is a device for inputting information to the control unit 3 from the subject or from outside the control unit 3, such as the Internet. Examples of the input device 34 include a touch panel, a voice input device, and a receiving device for Internet communication.

[0097] Fig. 11 is a diagram showing an example of the arrangement of devices around a subject 100 in the third embodiment of the present invention. As shown in Fig. 11, the subject 100 is a passenger of a moving body and is seated in a seat 35 of the vehicle.

[0098] The heartbeat measuring device 15 is a heartbeat sensor mounted on a vehicle, and is, for example, wrapped around the left wrist of the subject 100. The heartbeat measuring device 15 measures the heartbeat of the subject 100, and transmits the obtained heartbeat information, that is, the heartbeat measurement value, to the control unit 4 by wired or wireless communication.

[0099] Camera 16 is a camera mounted on a vehicle (on-board camera) and is disposed, for example, at a position that captures the face of person 100 from a position slightly above and in front of person 100. Camera 16 continuously captures images of facial expressions of person 100 and transmits facial expression information, which is information on the acquired facial images, to control unit 4 via wired or wireless communication.

[0100] [Example of relaxation level assessment] FIG. 12 is a flowchart showing an example of the flow of a process for determining a relaxation degree performed by the device for determining a relaxation degree according to the third embodiment of the present invention.

[0101] In step S601, the heartbeat measurement value acquiring unit 11 starts acquiring heartbeat measurement values ​​from the heartbeat measuring device 15. In this manner, in this embodiment, the heartbeat measurement value acquiring unit 11 acquires the heartbeat measurement values ​​of the subject from a heartbeat sensor mounted on a moving object.

[0102] In step S602, the facial expression information acquisition unit 12 starts acquiring image data of facial expressions from the camera 16. In this manner, in this embodiment, the facial expression information acquisition unit 12 acquires facial expression information of the person to be determined from the camera 16 mounted on the moving object.

[0103] In step S603, the external factor information acquisition unit 32 acquires external factor information from the in-vehicle sensor 33 and the input device 34. For example, the external factor information acquisition unit 32 acquires information on the temperature outside the vehicle, the temperature and humidity inside the vehicle, and the seat reclining angle from the in-vehicle sensor 33, and acquires information on the date and time when the person to be determined is riding in the moving object from the input device 34.

[0104] In step S604, the control unit 3 judges whether the heart rate of the person to be judged is stable or not by referring to the heart rate measurement value acquired by the heart rate measurement acquisition unit 11. For example, the control unit 3 judges that the heart rate is stable when the fluctuation in the heart rate at a specific time is equal to or less than a threshold, and judges that the heart rate is not stable when the fluctuation in the heart rate at a specific time exceeds the threshold.

[0105] If the control unit 3 determines in step S604 that the heart rate is stable (YES in step S604), the reference data acquiring unit 31 acquires reference data of the heart rate measurement value in step S605. For example, the reference data acquiring unit 31 acquires the heart rate measurement value for a specific time period (e.g., 2 minutes) acquired by the heart rate measurement value acquiring unit 11 as basic data.

[0106] On the other hand, if the control unit 3 determines in step S604 that the heart rate has not settled down, the control unit 3 waits for a predetermined time (for example, one minute) in step S606, and returns to step S604.

[0107] In step S607, the relaxation level estimation section 13 executes a process of estimating a relaxation level using the reference data acquired by the reference data acquisition section 31 as a heart rate measurement value. The relaxation level estimation process is performed in the same manner as the process in step S103 described above, except that the basic data is used as the heart rate measurement value. In this manner, in this embodiment, the relaxation level estimation section 13 estimates the relaxation level of the person to be determined by referring to the reference data acquired by the reference data acquisition section 31.

[0108] Furthermore, in estimating the relaxation level in step S607, the relaxation level estimation section 13 estimates the relaxation level of the person to be assessed by referring to the external factor information acquired by the external factor information acquisition section 32. For example, the relaxation level estimation section 13 adds a constant to the formula read out from the storage section 17 described above or changes a coefficient, depending on the external factor information acquired by the external factor information acquisition section 32, to calculate the relaxation level.

[0109] Next, the relaxation level correction unit 14 corrects the relaxation level estimated in step S607 based on the facial expression information. This process of correcting the relaxation level based on the facial expression information is performed in the same manner as the process in step S104 described above.

[0110] In this embodiment, reference data of the subject is acquired when estimating the relaxation level, so that it is possible to more accurately determine the relaxation level of the subject who is in a moving body such as a vehicle. When the subject is a driver, it is possible to more accurately determine the relaxation level of the subject who is driving a car, which is advantageous from the viewpoint of realizing a stable driving state of the car.

[0111] Furthermore, in the present embodiment, external factor information is acquired when estimating the relaxation level of the person being assessed, which is advantageous in that the estimation of the relaxation level appropriately reflects the influence of external factors acting on the person being assessed on the relaxation level of the person being assessed.

[0112] [Summary of embodiment 3] The control unit (3) of this embodiment further includes a reference data acquisition unit (31) that acquires reference data that is a reference for the heart rate measurement value of the person being assessed. The relaxation level estimation unit estimates the relaxation level of the person being assessed by further referring to the reference data acquired by the reference data acquisition unit. This configuration is even more effective in terms of accurately grasping the state of the heart rate measurement value of the person being assessed when estimating the relaxation level and improving the accuracy of the estimation of the relaxation level.

[0113] In this embodiment, the control unit further includes an external factor information acquisition unit (32) that acquires external factor information including at least one of environmental information indicating the environment in which the person to be assessed is present and date and time information. The relaxation level estimation unit further references the external factor information acquired by the external factor information acquisition unit to estimate the relaxation level of the person to be assessed. This configuration is even more effective in terms of improving the accuracy of the estimation of the relaxation level, since it allows the influence of external factors to be appropriately reflected in the estimation of the relaxation level of the person to be assessed.

[0114] In this embodiment, the subject is a passenger in the moving body. The heart rate measurement value acquisition unit acquires the heart rate measurement value of the subject from a heart rate sensor (heart rate measurement device 15) mounted on the moving body, and the facial expression information acquisition unit acquires facial expression information of the subject from a camera (16) mounted on the moving body. This configuration is even more effective from the viewpoint of more easily judging the relaxation level of the subject in the moving body and from the viewpoint of constantly utilizing the judgment result of the relaxation level of the subject in the moving body in the operation of the moving body.

[0115] [Embodiment 4] Another embodiment of the present invention will be described below. For ease of explanation, the same reference numerals are used for components having the same functions as those described in the above embodiment, and the description thereof will not be repeated. In this embodiment, the technology of the present invention for inducing a subject to a relaxed state is applied to a passenger in a moving body, such as an automobile.

[0116] [Functional configuration] Fig. 13 is a block diagram showing a schematic functional configuration of a relaxation induction device according to embodiment 4 of the present invention. As shown in Fig. 13, the relaxation induction device of this embodiment includes a control unit 4. The control unit 4 is configured similarly to the control unit 3 of embodiment 3 described above, except that it further includes a stimulus determination unit 41. The stimulus determination unit 41 determines a stimulus according to the relaxation degree corrected by the relaxation degree correction unit 14.

[0117] The control unit 4 is further connected to a stimulus providing device 42. The stimulus providing device 42 is a device that provides the subject with the stimulus determined by the stimulus determining unit 41. The stimulus providing device 42 may be a standalone device, or may be various in-vehicle devices. Examples of the stimulus provided by the stimulus providing device 42 include light, sound, vibration, heat, humidity, and smell, and the stimulus providing device 42 may be any device that can appropriately control and present such stimuli.

[0118] For example, the stimulus providing device 42 that presents light may be a device capable of changing the brightness or color of light presented in a vehicle, and examples thereof include interior lights, displays, and control panels. An example of the stimulus providing device 42 that presents sound includes a car audio system. An example of the stimulus providing device 42 that presents vibration includes a seat with a massage function. An example of the stimulus providing device 42 that presents heat includes an air conditioner and a seat with a heating function. An example of the stimulus providing device 42 that presents humidity includes an air conditioner. An example of the stimulus providing device 42 that presents an odor includes a deodorizing device or a fragrance generating device that can be attached to a cigarette lighter socket.

[0119] [Example of relaxation induction processing] Fig. 14 is a flow chart showing an example of the flow from the relaxation degree determination process to the relaxation induction process by the relaxation induction device according to embodiment 4 of the present invention. The relaxation induction process is executed in the same manner as the relaxation degree determination process in embodiment 3 shown in Fig. 12, except that it includes a process of providing a stimulus based on the result of the relaxation degree determination. In the relaxation induction process in this embodiment, as shown in Fig. 14, in step S701, the control unit 4 executes a process of providing the subject with a stimulus based on the result of the relaxation degree determination determined by the correction process of the relaxation degree based on facial expression in step S104.

[0120] FIG. 15 is a flowchart showing the flow of an example of a relaxation induction process according to the fourth embodiment of the present invention.

[0121] In step S801, the stimulus determination unit 41 selects a stimulus to be provided to the subject based on the determination result of the subject's relaxation level. For example, the stimulus determination unit 41 selects a stimulus to be provided to the subject by referring to a response model stored in the storage unit 17. The response model is a model that specifies the relationship between the subject's relaxation level and the content and intensity of the stimulus to be provided to the subject. Examples of the response model include a model formula. The model formula specifies at least the relationship between the subject's relaxation level and the content and intensity of the stimulus to be provided. By referring to such a reference model, the stimulus determination unit 41 determines the content and intensity of the stimulus to be provided to the subject.

[0122] In step S802, the stimulus providing device 42 provides the stimulus selected by the stimulus determination unit 41 to the person to be determined, in accordance with the information of the stimulus selected by the stimulus determination unit 41.

[0123] After providing the stimulus, the control unit 4 executes a determination of the relaxation level of the subject to whom the stimulus was provided. This determination of the relaxation level is performed in the same manner as in steps S103 and S104 of FIG. 2 described above, as shown in FIG.

[0124] In step S803, the control unit 4 generates notification information of the result of the determination of the relaxation level, and transmits it to the notification device 18. The notification device 18 notifies the subject of the notification information in the form of sound or image.

[0125] In step S804, the control unit 4 inquires of the person being assessed as to whether or not to end the relaxation induction processing. This inquiry is made by sending inquiry information to the notification device 18, and is made by the notification device 18 using a voice or an image. When the person being assessed inputs information to the effect that he or she wishes to end the relaxation induction processing via the input device 34 (in the case of YES), the control unit 4 ends the relaxation induction processing. When the person being assessed inputs information to the effect that he or she does not wish to end the relaxation induction processing (in the case of NO), the control unit 4 returns to step S801, and executes the above-described processing for relaxation induction again.

[0126] In this embodiment, it is possible to provide the subject with both a refreshing active relaxation state and a drowsy calm relaxation state. Therefore, during automatic driving, it is possible to guide the subject, who is a passenger of a moving body, to a calm relaxation state on the way and to an active relaxation state near the destination. This makes it possible to guide the subject to a state where he or she can be active as soon as he or she arrives at the destination.

[0127] In addition, in this embodiment, it is possible to induce the subject to be assessed into an active or calm relaxed state on a regular basis over a long period of time, thereby reducing the stress of the subject traveling on a mobile object and improving the mental health of the subject.

[0128] Furthermore, if you only use sedative relaxation, the parasympathetic nervous system will always be dominant. If the parasympathetic nervous system is dominant and overactive, you may feel heavy and lethargic, or abnormally sleepy. By providing both sedative relaxation and active relaxation, you can keep your autonomic nervous system in good condition.

[0129] [Summary of the fourth embodiment] The control unit (4, relaxation induction device) of this embodiment includes the above-mentioned heart rate measurement value acquisition unit, facial expression information acquisition unit, relaxation level estimation unit, and relaxation level correction unit, as well as a stimulus provision unit (21) that provides the subject with a stimulus according to the relaxation level corrected by the relaxation level correction unit. Thus, the relaxation level of the subject can be accurately determined from the biometric information and facial expression information, and further, by providing the subject with a stimulus determined with reference to the relaxation level, the subject can be induced to a more desirable relaxed state.

[0130] [Modifications] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in the different embodiments are also included in the technical scope of the present invention.

[0131] In the first to fourth embodiments of the present invention, the facial expression information may be information on an image of a part of the face of the person to be judged. For example, the facial expression information may be information on an image of the main parts of the person to be judged, such as the eyes and mouth.

[0132] In the first to fourth embodiments of the present invention, the correction value for correcting the relaxation level based on facial expression information may be different for each type of emotion. The absolute value of the correction value may be larger for emotions that are more likely to be clearly expressed by facial expressions than by measured heart rate values, so that the result of the determination based on facial expressions is more strongly reflected in the determination of the relaxation level of the person being determined.

[0133] Furthermore, in the first to fourth embodiments of the present invention, information other than emotion-based information may be used as facial expression information in the relaxation level determination process. For example, "eyes closed" may be added as facial expression information, and the "eyes closed" information may be referenced in the relaxation level determination process. As an example of the reference of such information, when the measurement result of the "eyes closed" information (for example, a percentage or a number of consecutive occurrences) exceeds a specific threshold, the control unit may end the relaxation determination process and provide a notification or a stimulus to wake the person being determined.

[0134] Furthermore, in the "second example of correction of relaxation level based on facial expression information" in the above-mentioned embodiment 1, the relaxation level in the above-mentioned active relaxation state may be lower than the relaxation level in the sedative relaxation state, when the relaxation level is estimated based on the heart rate measurement value. Therefore, when classified as an active relaxation state, the correction value may be set to a larger value than when classified as a sedative relaxation state. In this way, setting the correction value appropriately depending on the type of relaxation level is also included in preferred embodiments of the present invention.

[0135] Furthermore, in the "first example of estimating a relaxation level based on heart rate measurement values" in the above-mentioned embodiment 2, the RR-I value may be smoothed in the same manner as in the "second example of estimating a relaxation level based on heart rate measurement values" in the same embodiment. In this case, in the first example as well, the characteristics of the heart rate fluctuations of the subject can be made clearer, as in the second example.

[0136] In addition, in the above-mentioned embodiment 4, when the result of the relaxation level determination is notified to the person being determined by the notification device 18, the amount of change in the relaxation level after the provision of a stimulus relative to the relaxation level before the provision of a stimulus may also be notified to the person being determined.

[0137] In the above-described fourth embodiment, the stimulus determination section may determine the stimulus according to input of stimulus information (content, intensity, etc.) from the subject.

[0138] In the fourth embodiment, the control unit may select a response model based on one or more factors indicated by the external factor information. By selecting a response model based on the factors, a response model appropriately weighted by the factors is selected. Therefore, it is possible to guide the subject to a relaxed state in which the external factor information is appropriately reflected.

[0139] In the fourth embodiment, the control unit may further include a reaction model update unit that updates the reaction model according to the result of the determination of the degree of relaxation of the subject and the stimulus provided to the subject at that time. Such a configuration is more advantageous from the viewpoint of increasing the accuracy when the subject is guided to a desired relaxed state in the relaxation induction process.

[0140] [Software implementation example] The control blocks (control units 1 to 4) of the relaxation level assessment device or relaxation induction device according to the embodiments of the present invention may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or may be realized by software.

[0141] In the latter case, the relaxation degree assessment device or relaxation induction device includes a computer that executes instructions of a program, which is software that realizes each function. This computer includes, for example, one or more processors, and a computer-readable recording medium that stores the program. The object of the present invention is achieved by having the processor in the computer read and execute the program from the recording medium.

[0142] The processor may be, for example, a CPU (Central Processing Unit). The recording medium may be a "non-transitory tangible medium" such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like. The device may further include a RAM (Random Access Memory) for expanding the program.

[0143] The program may be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or a broadcast wave). Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0144] [Experimental example for acquiring information on the subject's emotions] In relation to the first to fourth embodiments of the present invention, an experiment for acquiring information on the emotion of the person to be assessed used in the first to fourth embodiments of the present invention will be described below. The emotion information of the person to be assessed can be acquired by the following experiment. Note that the "subject" in the following description may or may not be the person to be assessed.

[0145] First, in the first step, the subject is allowed to rest for three minutes and the first psychological evaluation is performed. Then, in the second step, the subject's emotions are controlled to a specific emotion for three minutes, and the second psychological evaluation is performed. The emotion control is performed by providing the subject with a specific stimulus, or by controlling the subject to reach a specific emotion by himself. In the emotion control, for example, the subject is controlled to a neutral emotional state, or a state in which the subject is irritated. Then, in the third step, a specific stimulus is provided to the subject for ten minutes, and the third psychological evaluation is performed. The stimulus provided to the subject is, for example, heat vibration, hot and cold, or a comedy video.

[0146] The subject's heart rate is measured throughout each step: the initial 3 minutes of rest, followed by 3 minutes of emotional control, and then the 10 minutes of stimulation.

[0147] In measuring the degree of relaxation based on the measured heart rate, the degree of relaxation is calculated based on the measured heart rate for the last two minutes of each step, for example using the above-mentioned formulas 1 and 2.

[0148] To measure emotions based on facial expressions, the subject's facial expressions are recorded on video at a frame rate of 30 fps for the last two minutes of each step, and the emotion estimation API is used to obtain a measurement value of the subject's emotions based on the image data of each frame of the captured facial images. The average value of each emotion measurement for each of the seven frames during the two minutes of recording is calculated. The highest average value is then used as the measurement result of the emotion based on facial expressions for that step.

[0149] In the psychological evaluation after each step, the subject's emotions are quantitatively evaluated. The psychological evaluation can be performed using the Two-Dimensional Mood Scale (TDMS). TDMS is a test that quantifies the subject's psychological state through self-monitoring of the psychological state.

[0150] In each step, the degree of relaxation is measured using heart rate measurements, or emotions are measured using facial expressions, or both are measured, thereby making it possible to obtain individual results of the measurement of the degree of relaxation using heart rate measurements, the measurement of emotions using facial expressions, and the determination of the degree of relaxation using these.

[0151] [Reference Experimental Example] Among the items of the psychological evaluation in the third step, the score of the ratio of positive emotions minus the score of the ratio of negative emotions was corrected between -1 and 1 for each individual, and the first result (actual value) of the relaxation level in the experiment was determined. In addition, the relaxation level calculated from the heart rate measurement value in the third step using the above-mentioned formula 1 or formula 2 was determined as the second result (estimated result of the relaxation level based on the heart rate measurement value). Furthermore, the above-mentioned correction value was calculated from the result of the emotion estimation API based on the facial expression image in the third step, and the above-mentioned measurement result of the relaxation level based on the heart rate measurement value was corrected to obtain the third result (determined result of the relaxation level). The tolerance range for the first result was set to ±0.25, and if the second result and the third result were within the above-mentioned tolerance range, they were determined to be correct, and if they were outside the above-mentioned tolerance range, they were determined to be incorrect.

[0152] In 24 experiments, the accuracy rate of the second result compared to the first result was 62.5% (15 correct answers). Of the nine incorrect answers, six were results that estimated that the subject was in an irritated state. Meanwhile, the accuracy rate of the third result compared to the first result was 83.3% (20 correct answers). In this way, the accuracy of judging the relaxation level was improved by correcting the relaxation level estimation result based on heart rate measurements with facial expression information. [Explanation of symbols]

[0153] 1~4 Control section 11 Heart rate measurement value acquisition section 12 Facial Expression Information Acquisition Unit 13 Relaxation level estimation unit 14 Relaxation degree correction section 15 Heart rate measuring device 16 Cameras 17 Memory section 18 notification device 21 Frequency component extraction section 31 Reference data acquisition section 32 External factor information acquisition unit 33 In-vehicle sensors 34 Input Devices 35 Reclining seat 41 Stimulus determining part 42 Stimulation delivery device 100 Judgment person 180 screens 181~183 Character images

Claims

1. a heart rate measurement value acquisition unit for acquiring a heart rate measurement value of a subject; A facial expression information acquisition unit that acquires information on the facial expression of the person to be determined; a frequency component extraction unit that extracts ultra-low frequency components and high frequency components of the heartbeat fluctuation of the subject by referring to the heartbeat measurement value acquired by the heartbeat measurement value acquisition unit, and judges the magnitude of the heartbeat fluctuation of the subject; Based on the result of the determination of heartbeat fluctuation by the frequency component extraction unit, When the heart rate fluctuation is large, a relaxation level of the person to be determined is estimated based on the extremely low frequency component and the high frequency component; a relaxation level estimation unit that estimates a relaxation level of the subject based on the ultra-low frequency component when the heart rate fluctuation is small; a relaxation level correction unit that corrects the relaxation level estimated by the relaxation level estimation unit based on the facial expression information acquired by the facial expression information acquisition unit, The relaxation degree determination device determines the relaxation degree of the person to be determined as the relaxation degree corrected by the relaxation degree correction unit.

2. The relaxation degree correction unit is configured to: correcting the degree of relaxation of the person to be determined to be higher in accordance with facial expression information that is more strongly correlated with the degree of relaxation; correcting the degree of relaxation of the person to be determined to be lower in accordance with facial expression information that is less correlated with the level of relaxation; The relaxation level determination device according to claim 1 .

3. The relaxation level assessment device of claim 1 or 2, wherein the relaxation level correction unit further classifies the type of the highly relaxed state of the person being assessed according to facial expression information of the person being assessed that is more strongly correlated with a high level of relaxation, among the facial expression information of the person being assessed.

4. The relaxation level assessment device according to claim 1 , wherein the frequency component extracting section extracts ultra-low frequency components of the heart rate fluctuation of the person to be assessed based on a moving average of the heart rate fluctuation of the person to be assessed.

5. A reference data acquisition unit that acquires reference data that is a reference for the heart rate measurement value of the subject, the relaxation level estimation unit estimates a relaxation level of the person to be determined by referring to the reference data acquired by the reference data acquisition unit; The relaxation degree determination device according to any one of claims 1 to 4.

6. An external factor information acquisition unit that acquires external factor information including at least one of environmental information indicating an environment in which the person to be determined exists and date and time information, the relaxation level estimation unit estimates a relaxation level of the person to be determined by referring to the external factor information acquired by the external factor information acquisition unit; The relaxation degree determination device according to any one of claims 1 to 5.

7. The subject is a passenger of a moving body, the heartbeat measurement value acquisition unit acquires a heartbeat measurement value of the subject from a heartbeat sensor mounted on a moving object; 7. The relaxation level determination device according to claim 1, wherein the facial expression information acquisition unit acquires the facial expression information of the person to be determined from a camera mounted on a moving object.

8. A relaxation level determination method executed by a relaxation level determination device having a processor, The processor performs the following processes (1) to (6): (1) A process of acquiring a heart rate measurement value of a subject; (2) acquiring information on the facial expression of the person to be determined; (3) referring to the heart rate measurement value, extracting ultra-low frequency components and high frequency components of the heart rate fluctuation of the subject, and determining the magnitude of the heart rate fluctuation of the subject; (4) Based on the result of the determination of the heartbeat fluctuation, When the heart rate fluctuation is large, a relaxation level of the person to be determined is estimated based on the extremely low frequency component and the high frequency component; When the heart rate fluctuation is small, a process of estimating a relaxation level of the person to be determined based on the ultra-low frequency component; (5) a process of correcting the estimated relaxation level based on the facial expression information; (6) executing a process of determining the corrected relaxation degree as the relaxation degree of the person to be determined; A method for assessing relaxation level.

9. a heart rate measurement value acquisition unit for acquiring a heart rate measurement value of a subject; A facial expression information acquisition unit that acquires information on the facial expression of the person to be determined; a frequency component extraction unit that extracts ultra-low frequency components and high frequency components of the heartbeat fluctuation of the subject by referring to the heartbeat measurement value acquired by the heartbeat measurement value acquisition unit, and judges the magnitude of the heartbeat fluctuation of the subject; Based on the result of the determination of heartbeat fluctuation by the frequency component extraction unit, When the heart rate fluctuation is large, a relaxation level of the person to be determined is estimated based on the extremely low frequency component and the high frequency component; a relaxation level estimation unit that estimates a relaxation level of the subject based on the ultra-low frequency component when the heart rate fluctuation is small; a relaxation level correction unit that corrects the relaxation level estimated by the relaxation level estimation unit based on the facial expression information acquired by the facial expression information acquisition unit; a stimulus providing unit that provides the subject with a stimulus according to the degree of relaxation corrected by the relaxation degree correcting unit; A relaxation induction device comprising:

Images