Cognitive ability estimation apparatus and program

Abstract

A cognitive ability estimation apparatus including: a biological information acquisition means that acquires biological information from which at least a heartbeat of a subject who performs driving or operation of a moving body can be specified; a video information acquisition means that acquires video information of the subject; a state estimation means that estimates the state of an autonomous nerve of the subject on the basis of the heartbeat to be specified from the biological information; and a cognitive ability estimation means that estimates an cognitive ability of the subject on the basis of an estimation result of the autonomous nerve by the state estimation means and the video information.

Description

Cognitive ability estimation device and program

[0001] The present invention relates to a cognitive ability estimation device and a program.

[0002] There are various types of mobile objects that are driven or operated by humans. Such mobile objects are heavy, and if an accident occurs due to improper driving or operation, the damage can be severe. For this reason, for example, in the case of automobiles, the driver is treated as the target, and the cognitive ability of the driver is estimated. Based on the estimation results, safe driving is supported by providing stimuli to the driver as needed (see, for example, Patent Document 1). For the sake of convenience, the mobile object will be referred to as the automobile, the target as the driver, and the operation for moving the mobile object as driving.

[0003] A driver's cognitive ability can be estimated using video information from a camera capturing an image of the driver in the driver's seat (see, for example, Patent Document 1). However, changes in the driver's visually identifiable features, such as blink frequency, gaze range, and gaze speed, are manifested as drowsiness. Therefore, there is a time lag between the actual onset of drowsiness and the confirmation of changes in the features. Because of this time lag, drowsiness has also been estimated from the state of the autonomic nervous system (see, for example, Patent Document 2).

[0004] JP 2019-200544 A JP 2009-039167 A JP 2008-125801 A

[0005] Conventionally, some stimuli are provided to a driver who is estimated to be drowsy by an in-vehicle device to alleviate the drowsiness (wakefulness). Examples of such stimuli include airflow from an air conditioner and sound warnings. Drivers who notice their own drowsiness typically consciously attempt to alleviate the drowsiness. This awareness can sometimes increase the activity of the sympathetic nervous system even when the driver is drowsy. In other words, an abnormal state can occur in which both the sympathetic and parasympathetic nervous systems are activated (see, for example, Patent Document 3).

[0006] Since drivers must drive safely, it is thought that their autonomic nervous system is relatively prone to abnormal states. For this reason, it can be said that the driver's drowsiness, or cognitive ability, cannot always be estimated with high accuracy from the state of the autonomic nervous system. Therefore, it is thought to be important to be able to estimate cognitive ability with higher accuracy.

[0007] An object of the present invention is to provide a cognitive ability estimation device that can estimate with higher accuracy the cognitive ability of a subject who drives or operates a mobile object.

[0008] A cognitive ability estimation device according to one aspect of the present disclosure includes a biometric information acquisition means for acquiring biometric information capable of identifying at least the heart rate of a subject driving or operating a mobile vehicle, a video information acquisition means for acquiring video information of the subject, a state estimation means for estimating the state of the autonomic nervous system of the subject based on the heart rate identified from the biometric information, and a cognitive ability estimation means for estimating the cognitive ability of the subject based on the estimation result of the autonomic nervous system state by the state estimation means and the video information.

[0009] The present invention makes it possible to estimate with higher accuracy the cognitive ability of a subject who drives or operates a mobile object.

[0010] 1 is a diagram illustrating an example of a mechanism by which a cognitive ability estimation device according to one embodiment of the present invention estimates the cognitive ability of a subject, and an example of control performed according to the estimated cognitive ability. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a cognitive ability estimation device according to one embodiment of the present invention. FIG. 3 is a functional block diagram illustrating an example of a functional configuration realized on a cognitive ability estimation device according to one embodiment of the present invention. FIG. 4 is a flowchart illustrating an example of a safe driving support process. FIG. 5 is a flowchart illustrating an example of a safe driving support process (continued).

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments described below are merely examples, and the technical scope of the present invention is not limited to these. Various modifications are also included within the technical scope of the present invention.

[0012] 1 is a diagram illustrating an example of a mechanism by which a cognitive ability estimation device according to one embodiment of the present invention estimates the cognitive ability of a subject, and an example of control performed in accordance with the estimated cognitive ability. The subject whose cognitive ability is estimated by the cognitive ability estimation device 1 is a person who drives or operates a mobile object. Therefore, the mobile object is equipped with a power source, such as an engine, that enables movement.

[0013] In Fig. 1, the moving body is assumed to be an automobile, and the subject is assumed to be a driver 3 who sits in the driver's seat 2 and drives the automobile. Therefore, unless otherwise specified, the following description will be given assuming that the moving body is an automobile and the subject is the driver 3. The moving body may be a train, a ship, an airplane, or the like. The subject may be a pilot, etc.

[0014] In this embodiment, biometric information, video information, and status information are used to estimate the cognitive ability of the driver 3. Of these pieces of information, the status information can be omitted. The biometric information is information that indicates the heart rate of the driver 3 or information that can identify the heart rate. FIG. 1 shows a steering wheel 5 equipped with a pulse sensor and a smart watch 6 worn on the wrist of the driver 3 as examples of devices that generate biometric information.

[0015] The pulse sensor may be, for example, a capacitance type, optical type, or radio wave type, and processes a signal output from the sensing sensor to identify the heartbeat from the pulsation of blood flow. The identification result is transmitted from the pulse sensor as biometric information. This biometric information is input to the cognitive ability estimation device 1 via, for example, an ECU (Electronic Control Unit). The pulse sensor may be provided in the driver's seat 2 instead of the steering wheel 5. There are no particular limitations on the type of sensor, installation location, etc., as long as it can identify the heartbeat.

[0016] The smartwatch 6 is equipped with, for example, an optical pulse sensor. The smartwatch 6 detects the pulsation of the driver's 3's blood flow as a heartbeat, and transmits the detection result as biometric information. The biometric information wirelessly transmitted from the smartwatch 6 is received, for example, directly by the cognitive ability estimation device 1.

[0017] The camera 4 is installed inside the automobile to capture images mainly of the face of the driver 3 sitting in the driver's seat 2. The image information obtained by the image capture is transmitted to the cognitive ability estimation device 1 via the ECU, similar to the pulse sensor provided on the steering wheel 5. From the image information, it is also possible to identify the heart rate of the driver 3. It is also possible to identify the breathing pattern. Therefore, the image information may be considered as biometric information.

[0018] The camera 7 and radar 8 are devices for generating status information or information used to generate status information. This status information is information that represents the status of a vehicle, which is a moving object. The camera 7 is used, for example, to capture images of the vehicle's direction of travel. The image information obtained by this capture is used, for example, to check the status of the road in the vehicle's direction of travel. The road status that is checked specifically includes the curves of the road, the presence or absence of lines drawn on the road, the presence or absence of obstacles, the type of obstacle, etc. All of these are identified by image analysis using the image information. Therefore, the camera 7 is a device for generating status information.

[0019] The radar 8 is used to measure the distance to an object present in the traveling direction, and generates and outputs, for example, distance information indicating the measured distance as status information. Therefore, the radar 8 is a device capable of generating status information. Note that the status information is not limited to that which can be generated by the camera 7 or the radar 8. Measured position information, information on the traveling speed of the vehicle, and the steering angle of the steering wheel 5 can also be used as status information. The status information to be used may be determined depending on the type of the mobile object, its purpose, etc.

[0020] The distance information output from the radar 8 and the video information output from the camera 7 are processed by the corresponding ECU 9. The ECU 9 recognizes, for example, the presence or absence of lines drawn on the road, the type of lines, the presence or absence of objects in the direction of travel, and the type of objects from the video information. Based on these recognition results, the ECU 9 determines whether the vehicle is swaying or whether the vehicle is straying from the lines. In addition, by combining this with the distance information, the ECU 9 also determines whether the distance between the vehicle and another vehicle traveling in front is appropriate, whether an obstacle is present, and whether the obstacle can be avoided or not. The ECU 9 then outputs, as status information, for example, the video information, each recognition result, and each determination result to the cognitive ability estimation device 1. The status information (distance information) generated by the radar 8 is converted into a processing result using the status information.

[0021] The video information and state information received by the cognitive ability estimation device 1 are used in the video analysis process of step S1. This video analysis process is a process for evaluating multiple indices that can be used to estimate cognitive ability. The types of multiple indices and their combinations are not particularly limited, but at least drowsiness must be used. This is because drowsiness has a very large impact on cognitive ability. As other indices, for example, fatigue and attention may be used, as in Patent Document 1. Here, the multiple indices are assumed to be three: drowsiness, fatigue, and attention.

[0022] Well-known methods can be used to evaluate each index. Specifically, for example, the method described in Patent Document 1 may be used. For this reason, detailed explanations will be omitted here. Note that a method that focuses on eye movement, specifically blinking frequency and eye movement, etc., is generally used to estimate the drowsiness of the driver 3. From the eye movement, the speed of the line of sight movement, the range of line of sight movement, etc. can be confirmed.

[0023] The faster the vehicle's traveling speed, the narrower the range of movement of the driver's 3's line of sight tends to be. Furthermore, when the driver 3 is trying to avoid an obstacle ahead, he or she may visually move his or her line of sight significantly to confirm the direction in which the vehicle should proceed. When turning the vehicle left or right, the driver 3 also typically moves his or her line of sight relatively significantly. For these reasons, state information is useful for estimating the driver's 3 cognitive ability with higher accuracy.

[0024] The biological information received by the cognitive ability estimation device 1 is used in the autonomic nerve state analysis process in step S2. This state analysis process evaluates the state of the autonomic nerves, that is, the activity levels of the sympathetic nerves and the parasympathetic nerves, and estimates the levels of drowsiness and stress using the evaluation results.

[0025] It is known that the spectrum obtained by frequency analysis of heartbeat intervals has a single peak structure. It is believed that the activity level can be relatively evaluated by comparing the power of the sympathetic nervous system in the range of 0.05-0.15 Hz and the power of the parasympathetic nervous system in the range of 0.15-0.45 Hz. Drowsiness and stress can be evaluated from their relative activity levels. Therefore, bioinformation that represents heart rate or allows identification of heart rate can be used to estimate the state of the autonomic nervous system, i.e., drowsiness and stress. Therefore, in this embodiment, bioinformation is used as information necessary for estimating the state of the autonomic nervous system. Since well-known methods may be used to estimate the state of the autonomic nervous system from the heart rate, a detailed description thereof will be omitted.

[0026] The structure present in the range of 0.05-0.15 Hz is called the Myer Wave Sinus Arrhythmia (NWSA), and the structure present in the range of 0.15-0.45 Hz is called the Respiratory Sinus Arrhythmia (RSA). Information related to respiration and blood pressure changes can also be obtained from the spectrum of heartbeat fluctuations.

[0027] The state of the driver 3 estimated from the activity levels of the autonomic nerves, i.e., the sympathetic nerves and the parasympathetic nerves, can be roughly divided into the following four states (see, for example, Patent Document 3): (1) A state in which the sympathetic nerves are dominant (2) A state in which the parasympathetic nerves are dominant (3) A state in which the activity levels of both the sympathetic nerves and the parasympathetic nerves are increased (4) A state in which the activity levels of both the sympathetic nerves and the parasympathetic nerves are decreased

[0028] (1) is a state in which it is estimated that the driver 3 is not feeling drowsy. A state in which the stress level is high, that is, a state in which the driver 3 is excited, is also included in (1). Hereinafter, this will be referred to as the first state. (2) is a state in which it is estimated that the driver 3 is feeling drowsy or tired. If the driver 3 is very relaxed, this state may be estimated. Hereinafter, this will be referred to as the second state. (3) is a state that appears when the driver 3 tries to overcome drowsiness. This is an abnormal state that does not usually appear. A driver 3 who is aware of drowsiness will be conscious of at least trying to suppress that drowsiness. Due to this consciousness, this state is thought to be relatively likely to occur in the driver 3. Hereinafter, this will be referred to as the third state. (4) is a state that is likely to be estimated in a driver 3 who is in a depressed state. It can be assumed that the state of the autonomic nervous system is unstable. Hereinafter, this will be referred to as the fourth state.

[0029] Even if the autonomic nervous system is roughly divided into four states in this way, it does not necessarily mean that the level of each state can be estimated. Furthermore, it does not necessarily mean that the level of cognitive ability can be estimated with high accuracy. In the first state, the cognitive ability of the driver 3 is considered to be high. If the driver 3 is feeling strong stress, the stress may cause a decrease in attention. A decrease in attention means that the actual cognitive ability is low due to not paying attention to driving or looking away, etc. However, even if the driver is feeling stressed, it is possible that the driver is able to control himself and maintain high cognitive ability. For these reasons, the level of cognitive ability cannot necessarily be estimated appropriately.

[0030] In the second state, it is highly likely that the driver 3's cognitive ability is reduced due to drowsiness, fatigue, or the like. However, depending on the road on which the vehicle is traveling or the current situation, the driver 3 may simply be relaxed. For this reason, even in the second state, the driver 3 may be able to drive the vehicle safely. The third state is a state seen when the driver 3 is trying to overcome drowsiness, as described above. Since the driver 3 is aware of his or her drowsiness and is consciously trying to overcome it, the cognitive ability itself may not have declined significantly. Even in the fourth state, it is considered that the driver 3 is appropriately aware of the situation. Therefore, it is not always possible to estimate the level of cognitive ability with high accuracy.

[0031] When focusing on the state of the autonomic nervous system, it is possible to estimate changes in the level of cognitive ability at an earlier timing than by video analysis (for example, Patent Document 3). However, it may not be possible to accurately estimate the level of cognitive ability from the state of the autonomic nervous system. For this reason, in this embodiment, the indicators evaluated from the video information and the results obtained by analyzing the state of the autonomic nervous system are used in a complementary manner to estimate the cognitive ability of the driver 3 with higher accuracy. To estimate the complementary cognitive ability, the cognitive ability estimation device 1 executes an actual drowsiness level analysis process in step S3 and a stress effect analysis process in step S4.

[0032] Drowsiness experienced by the driver 3 in the second state has the most significant impact on cognitive decline. This is because drowsiness is not only the most significant cause of cognitive decline, but also because drowsiness typically continues for a long period of time. In other words, drowsiness is likely to cause a state of extremely low cognitive ability to continue for a long period of time. For this reason, in this embodiment, the actual level of drowsiness estimated from the state of the autonomic nervous system is evaluated as the actual drowsiness level by the actual drowsiness level analysis process of step S3. The driver 3 to be evaluated is the driver 3 whose autonomic nervous system is estimated to be in the second state. This actual drowsiness level analysis process uses video information and state information.

[0033] A driver 3 who tries to overcome drowsiness intentionally moves his or her body to stimulate the body, for example, within a range that does not interfere with safe driving. Examples of such body movements include intentionally moving the head or shoulders to stimulate the body, intentionally moving the eyes (blinking, etc.), operating an air conditioner or the like to make the body more susceptible to stimuli from wind or sound, opening a window to let outside air into the vehicle, etc. Even if a driver 3 who makes such movements feels drowsy, it can be estimated that the decline in cognitive ability level is relatively small. In other words, it can be estimated that the actual level of drowsiness is relatively low and that a sufficient level of cognitive ability is maintained.

[0034] However, if such movements are not confirmed or are barely confirmed, even if the driver is consciously trying to overcome drowsiness, that is, even if sympathetic nerve activation can be confirmed, the driver is not taking any action to wake up from drowsiness. Therefore, it can be estimated that the actual drowsiness level of the driver 3 is high and the cognitive ability level is not sufficient.

[0035] As described above, the movements of the driver 3 are affected by the state of the vehicle. Furthermore, the movements of the driver 3 affect the state of the vehicle. For example, if the driver 3 is driving on a road marked with lines, there is a risk that the vehicle will stray from the lines if the driver 3 does not operate the steering wheel 5 appropriately. There is also a risk that the vehicle will be driven at a speed that significantly exceeds the speed limit. Such information can also be used as state information to estimate the actual drowsiness level of the driver 3. The presence or absence of lines on the road, the type of lines, and whether the vehicle is straying from the lines can all be confirmed from the video information from the camera 7. If a sign or the like indicating a speed limit is captured by the camera 7, the speed limit can be confirmed from the video information.

[0036] Furthermore, attention, which is evaluated as an index, is related to drowsiness. For example, a driver 3 who is estimated to have low attention by frequently moving his / her gaze over a relatively large range is unlikely to be feeling drowsy. This is because such movements would not be observed without brain activity. For this reason, the actual drowsiness level analysis process uses video information and state information.

[0037] In the above description, for convenience, actual drowsiness levels are divided into two stages, high and low. However, in reality, each of these stages of actual drowsiness levels can be further divided into two or more stages. At a high actual drowsiness level, that is, in a situation where it is estimated that the driver 3 is feeling very drowsy, it is not necessary to divide the levels into multiple stages, taking safety into consideration. In other words, as a numerical value representing the actual drowsiness level, a numerical value representing that safe driving cannot be expected may be set, and the level at which safe driving can be expected may be expressed by two or more numerical values. Here, for convenience, it is assumed that the numerical value representing that safe driving cannot be expected is 3, and the estimated result of actual drowsiness level is expressed as an integer between 0 and 3. The smaller the value, the lower the actual drowsiness level, that is, the less drowsy the driver 3 is feeling.

[0038] In the stress effect analysis process of step S4, the effect of the stress felt by the driver 3 on the driver 3 is evaluated. For this effect evaluation, image information and state information are used. The result of the autonomic nerve state analysis process is used to determine whether or not it is necessary to evaluate this effect. As a result, only drivers 3 who are considered to be in the first state are subject to the effect evaluation. More specifically, only drivers 3 whose stress level evaluated in the autonomic nerve state analysis process of step S2 is equal to or higher than a set value, that is, only drivers 3 who are estimated to be experiencing stress equal to or higher than a stress level that is deemed to have a negative effect on safe driving are subject to the effect evaluation.

[0039] The stress influence analysis process mainly evaluates the influence of stress on attention. As a result, movements of the driver 3's arms, upper body, head, etc. are not emphasized unless they are at a level that adversely affects the driver's ability to see the direction they should be looking in or the driving operation. This is because, unless they are at that level, it is considered that stress is not actually affecting the decline in attention, or that the degree of the influence is relatively small. If the degree of influence is within a small range, it can be estimated that the driver 3 is in a state where he or she can drive safely.

[0040] On the other hand, when the driver 3 moves their arms, upper body, or head violently or widely, it becomes difficult for them to properly direct their gaze in the direction they should be looking, properly recognize the images that enter their eyes, or immediately and appropriately perform the driving operations that they should be performing. Such movements are thought to be the result of the effects of stress manifesting as the driver 3's body movements. If such movements are repeatedly observed, it can be assessed that the driver 3's attention level is low. It is highly likely that the driver 3 is unable to control stress. Therefore, it can be estimated that stress is affecting the driver 3 to the extent that it is preventing them from driving safely.

[0041] The degree of impact is also evaluated, for example, in multiple stages. In evaluating the degree of impact, as described above, a determination is also made as to whether safe driving is possible. As a result, the degree of impact may also be assigned a numerical value, for example, indicating that safe driving cannot be expected. In this case, the level at which safe driving can be expected may be expressed, for example, by two or more numerical values. Here, for the sake of convenience, it is assumed that the numerical value indicating that safe driving cannot be expected is 3, and the evaluation result of the degree of impact is expressed by an integer between 0 and 3. The smaller the value, the lower the degree of impact, i.e., it indicates that stress is not actually affecting the driver 3.

[0042] As described above, the movements of the driver 3 are affected by the state of the vehicle. For example, when the vehicle is stopped, there are no restrictions on the movements of the driver 3 from the perspective of safe driving. The driver 3 can make any movements he or she likes. Even when the vehicle is moving, the permissible movements of the driver 3 change depending on the driving speed, etc. For this reason, the stress effect analysis process also uses video information and state information.

[0043] The results of the stress impact analysis process and the actual drowsiness level analysis process are passed to the cognitive ability estimation process in step S5 and the device control process in step S6, respectively. The cognitive ability estimation process is a process for estimating the cognitive ability of the driver 3 as well as changes in cognitive ability. The cognitive ability estimation uses the actual sleep level, the evaluation results of each index except for drowsiness, and the degree of impact of stress. Here again, for convenience, it is assumed that cognitive ability is evaluated on a four-point scale from 0 to 3, with 3 representing the worst state. Under such assumptions, the smaller the number, the higher the cognitive ability. Because cognitive ability is estimated numerically, the target of estimation will hereinafter also be referred to as the "cognitive ability level."

[0044] As described above, when the actual drowsiness level and the degree of impact are evaluated on a four-point scale, with 3 representing the worst state in terms of safe driving, if the actual drowsiness level or the value of the degree of impact is 3, for example, the cognitive ability level may also be set to 3. If both the actual drowsiness level and the degree of impact are not 3, the cognitive ability level may be calculated using a calculation formula in which the actual drowsiness level and the evaluation results of each index other than drowsiness are used as variables. Each variable may be multiplied by a coefficient determined according to the degree of impact on safe driving, for example.

[0045] Changes in cognitive ability are evaluated using, for example, actual sleepiness level, degree of influence, stress level, and each index other than sleepiness, which are taken into consideration in controlling one or more devices that make up device group 20. For this reason, these will be collectively referred to as "control indexes" hereinafter to distinguish them from the above-mentioned indexes.

[0046] 1 shows only an air conditioning equipment control device 21 and a steering control device 22 as devices constituting the equipment group 20. The air conditioning equipment control device 21 is a device that controls an air conditioning device capable of blowing hot air and cold air, and is also capable of controlling the temperature and blowing direction of the air. By controlling this control device 21, it is possible to provide a stimulus to the driver 3 by blowing air. The control device 21 is shown in the equipment group 20 because it is a device necessary for controlling the air conditioning device.

[0047] The steering control device 22 is a device that controls the generation of a driving force that allows the steering wheel 5 to be turned more easily. It is also part of a device that enables automatic driving. The control device 22 can use the driving force to generate vibrations or the like in the hands holding the steering wheel 5. By controlling this control device 22 to generate vibrations or the like in the steering wheel 5, it is possible to provide a stimulus to the driver 3. Like the control device 21, the control device 22 is also a device necessary for providing a stimulus from the steering wheel 5, and is therefore shown in the equipment group 20.

[0048] The devices to be controlled are not limited to those shown in FIG. 1. Navigation devices that provide navigation by display and voice, interior lighting, power windows, etc. may also be controlled. The navigation device can be used for voice output or message display. In FIG. 1, the cognitive ability estimation device 1 is shown to directly instruct the device group 20. However, in reality, the cognitive ability estimation device 1 requests control from the corresponding ECU. The ECU 9 is shown in FIG. 1 because the ECU 9 generates the status information.

[0049] In this embodiment, a desirable stimulus is provided to the driver 3 in a timely manner so that the driver 3 can drive safely. The stimulus that is desirable for the driver 3 may differ depending on the state of the driver 3. The content of the information to be provided to the driver 3 also differs depending on the state of the driver 3. For example, a driver 3 with low attentional ability may be made more aware of the low level of attentional ability. To this end, it is conceivable to inform the driver 3 of the low level of attentional ability by voice or the like. However, it is considered that providing such information by voice or the like is insufficient for a driver 3 who is feeling drowsy or fatigued. It is also considered necessary to provide the driver 3 with a stimulus to alleviate the level of drowsiness or fatigue that they are feeling.

[0050] If information is provided by voice or other means, the information will also be provided to passengers in the same vehicle, which is expected to encourage the passengers to encourage the driver 3. For this reason, it is desirable to provide information in a way that can be easily recognized by people other than the driver 3.

[0051] In this way, it is desirable to select the content of the information to be provided to the driver 3 and the stimuli to be given, taking into consideration the state of the driver 3. However, even if desirable information and stimuli are provided to the driver 3, it is not guaranteed that the state of the driver 3 will improve. For this reason, in this embodiment, changes in each control index are evaluated, and the evaluation results are reflected in the control of the device group 20, so that the state of the driver 3 will improve more reliably. In the cognitive ability estimation process, it is determined whether or not it is necessary to control the device group 20, and if it is determined that it is necessary, it selects a device to be controlled from the device group 20, and also determines the control content, etc. of the selected device.

[0052] The device control process of step S6 is a process for controlling devices that should be controlled among the device group 20. The device control process is passed the results of the actual drowsiness level analysis process, the stress impact analysis process, and the cognitive ability estimation process. The device control process corresponds to a case where the actual drowsiness level or the impact degree is 3 when the results of the actual drowsiness level analysis process and the stress impact analysis process are passed to it. Control of the device to wake up from drowsiness or to control stress is started. After the control is started, or in a situation where both the actual drowsiness level and the impact degree are not 3, the device control process controls the device in accordance with the instructions passed as the processing result of the cognitive ability estimation process. Control of the device also includes ending control of the device.

[0053] In the cognitive ability estimation process, if no improvement in the condition of the driver 3 is confirmed even after providing stimuli, stronger stimuli are provided in stages. Therefore, after the actual drowsiness level or the degree of influence reaches 3 and device control is started, the device is controlled in accordance with the instructions passed from the cognitive ability estimation process.

[0054] In this embodiment, the state of the driver 3 estimated from the autonomic nervous system is confirmed from the video information, and the actual state of the driver 3 is estimated. Therefore, the actual state of the driver 3 can be estimated with higher accuracy. Individual differences can also be more appropriately addressed. The state information obtained from the vehicle improves estimation accuracy, enabling more appropriate responses. As a result, it becomes possible to provide the driver 3 with information according to their state, and even provide physical stimulation, etc., in a timely and more appropriate manner. By providing stimulation, etc., it is expected that the state of the driver 3 will improve. As a result, it becomes possible to more appropriately support the driver 3 so that they can drive the vehicle safely.

[0055] Fig. 2 is a block diagram showing an example of the hardware configuration of a cognitive ability estimation device according to one embodiment of the present invention. Fig. 2 also shows various sensors related to the cognitive ability estimation device 1 and each device constituting the device group 20. Note that this configuration example is just one example, and the hardware configuration of the cognitive ability estimation device 1 and the information processing device that can be used as the estimation device 1 is not limited to this.

[0056] 2, the cognitive ability estimation device 1 includes, for example, a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an SSD (Solid State Drive) 14, an IFC (Interface Controller) 15, and a communication unit 16. These are connected to a bus.

[0057] The CPU 11 executes, for example, programs stored in the ROM 12 and / or programs stored in the SSD 14 to realize various processes. Both programs are loaded into the RAM 13 and executed. The programs loaded from the SSD 14 to the RAM 13 include, for example, an operating system (OS) and various application programs that run on the OS. The various application programs include one or more programs developed to enable the information processing device to function as the cognitive ability estimation device 1. Hereinafter, this developed application program will be referred to as a "developed application."

[0058] The development application may be recorded on removable media and distributed. It may also be possible to distribute it via a network such as the Internet. Therefore, the recording medium on which the development application is recorded may be one that is installed or attached to an information processing device that is directly or indirectly connected to a network, or one that is installed or attached to an externally accessible device.

[0059] The IFC group 15 enables connection with various external devices. The IFC group 15 may include IFCs for audio output and image output. It may also include an IFC for directly connecting in-vehicle devices. It may also include an IFC that enables reception of biometric information or video information. The RAM 13 also stores data and the like necessary for the CPU 11 to execute various processes, as appropriate. This data includes data used in various programs executed by the CPU 11. This data also includes video information, biometric information, and status information.

[0060] When each ECU including the ECU 9 is connected to an in-vehicle LAN (Local Area Network), the communication unit 16 enables communication with each ECU via the in-vehicle LAN. When supporting reception of biological information from the smart watch 6, the communication unit 16 may also be capable of wireless communication.

[0061] FIG. 2 shows a camera 4 and a pulse sensor 31 as a sensor group 30 capable of acquiring biometric information. The pulse sensor 31 is provided, for example, on the steering wheel 5. This pulse sensor 31 may be provided on the driver's seat 2 instead of the steering wheel 5. FIG. 2 also shows two other sensor groups 40 and 50 for acquiring status information. The sensor group 40 is particularly used to check the status of the vehicle, and the sensor group 50 is used to check the environment in which the vehicle is placed. As described above, the ECU also includes an ECU 9 that processes information output from one or more sensors included in each of the sensor groups 40 and 50, and generates and outputs status information. This ECU 9 is omitted from FIG. 2.

[0062] The sensor group 40 includes a steering angle sensor 41, a brake / accelerator sensor 42, a G sensor 43, and a speedometer 44. The steering angle sensor 41 detects the angle at which the steering wheel 5 is turned as the steering angle. The brake / accelerator sensor 42 detects the amount of operation of an accelerator pedal and a brake pedal (not shown). The G sensor 43 is an acceleration sensor. It detects the acceleration generated in the vehicle. The speedometer 44 measures the traveling speed of the vehicle. All of these detection results are treated as vehicle body status information. The vehicle body status information will hereinafter be referred to as "vehicle information."

[0063] The sensor group 5 includes a camera 7, a radar 8, and a locator 51. The locator 51 outputs position information indicating the vehicle's position determined by positioning. This position information makes it possible to confirm the road on which the vehicle is traveling, whether or not there is a curve in the road in the traveling direction, the degree (radius) of the curve, etc. The image information from the camera 7, the status information generated by the ECU 9 from the image information and the distance information from the radar 8, and the position information from the locator 51 will hereinafter be referred to as "environmental information." The type, number, and combination of sensors constituting each of the sensor groups 30 to 50 are not particularly limited. The example shown in FIG. 2 is an example.

[0064] The device group 20 includes an air conditioning equipment control device 21, a steering control device 22, a message output control device 23, and an automatic driving device 24. The message output control device 23 enables information to be provided to the driver 3 by audio output. This control device 23 may be mounted on a navigation device. The automatic driving device 24 enables automatic driving of the vehicle. If the automatic driving device 24 cannot confirm an improvement in the driver's condition even after providing stimuli, specifically if it is determined that the driver cannot be expected to drive safely due to drowsiness or stress, the automatic driving device 24 is requested to switch to automatic driving.

[0065] 3 is a functional block diagram showing an example of a functional configuration implemented on a cognitive ability estimation device according to an embodiment of the present invention. Next, with reference to FIG. 3, an example of a functional configuration implemented on the cognitive ability estimation device 1 will be described in detail.

[0066] 3 , the CPU 11 of the cognitive ability estimation device 1 has a functional configuration including a biological information acquisition unit 111, a vehicle information acquisition unit 112, an environmental information acquisition unit 113, an autonomic nerve state analysis unit 114, an image information analysis unit 115, an index evaluation unit 116, an actual sleep level evaluation unit 117, a stress effect evaluation unit 118, a cognitive ability estimation unit 119, and an equipment control processing unit 120. The CPU 11 can transmit and receive information (data) to and from each ECU group 60 including the ECU 9 via the communication unit 16.

[0067] Each sensor included in the sensor group 30 to 50 is connected to one of the ECUs that make up the ECU group 60. The same is true for each device that makes up the device group 20. Therefore, the reception of information obtained by each sensor and the control of each device are performed via one of the ECUs.

[0068] The functional components on the CPU 11 are realized by the CPU 11 executing various programs including the development application. As a result, the following information storage areas are secured in the SSD 14: a biometric information storage unit 141, a state information storage unit 142, a state analysis result storage unit 143, an index evaluation result storage unit 144, a level evaluation result storage unit 145, an impact assessment result storage unit 146, a cognitive ability estimation result storage unit 147, a state evaluation information storage unit 148, an index evaluation information storage unit 149, a level evaluation information storage unit 150, an impact assessment information storage unit 151, an estimation information storage unit 152, and a device control information storage unit 153.

[0069] Note that information that only needs to be temporarily stored may be stored in the RAM 13 instead of the SSD 14. Various pieces of information to be stored in the SSD 14 are stored in the RAM 13, and then transferred to and stored in the SSD 14. For convenience, the process of storing information will be ignored here, and only the SSD 14 will be considered as the destination for storing information.

[0070] The biological information acquisition unit 111 acquires biological information necessary for analyzing the state of the autonomic nerves. The biological information here includes not only that acquired by the pulse sensor 31 but also image information acquired by the camera 4. The biological information acquired by the biological information acquisition unit 111 is stored in the biological information storage unit 141 secured in the SSD 14.

[0071] The vehicle information acquisition unit 112 and the environmental information acquisition unit 113 acquire vehicle information and environmental information, respectively. The acquired vehicle information and environmental information are both stored in the state information storage unit 142 secured in the SSD 14. Note that the information acquisition by the biometric information acquisition unit 111, the vehicle information acquisition unit 112, and the environmental information acquisition unit 113 is performed at predetermined timing, for example, at predetermined time intervals. This is because pulse rate and the like are unlikely to undergo repeated rapid changes within a short period of time.

[0072] The autonomic nerve state analyzer 114 estimates the state of the autonomic nerves by analysis using the biological information stored in the biological information storage unit 141. Through this analysis, the activity levels of the sympathetic nerves and the parasympathetic nerves are evaluated, and the evaluation results are used to estimate the levels of drowsiness and stress. The estimation results are stored as state analysis results in the state analysis result storage unit 143 secured in the SSD 14. The autonomic nerve state analysis process shown as step S2 in FIG. 1 is executed by the autonomic nerve state analyzer 114.

[0073] The state evaluation information storage unit 148 secured in the SSD 14 stores information for estimating the state of the autonomic nerves as state evaluation information. This evaluation information is, for example, as described above, information for evaluating the activity levels of the sympathetic nerves and the parasympathetic nerves from a spectrum obtained by frequency analysis of the heartbeat interval, and further evaluating the levels of drowsiness and stress from the evaluation results. The autonomic nerve state analyzer 114 refers to this evaluation information to estimate the state of the autonomic nerves.

[0074] The video information analysis unit 115 generates motion feature information representing the features of the movements of the driver 3, for example, by analysis using video information stored in the biometric information storage unit 141 as biometric information. The index evaluation unit 116 evaluates each index by analysis using the generated motion feature information and the status information stored in the status information storage unit 142. The evaluation results of each index are stored in the index evaluation result storage unit 144 secured in the SSD 14.

[0075] The index evaluation information storage unit 149 secured in the SSD 14 stores information for evaluating each index as index evaluation information. This evaluation information is prepared for each status category represented by, for example, status information. The categories are determined by taking into account, for example, the type of road (including whether it is a highway or not), the driving area, and the driving speed, etc. The index evaluation unit 116 identifies the evaluation information to be referenced from the status information and evaluates each index by referencing the identified evaluation information using the motion feature information. For this reason, the video analysis process shown as step S1 in FIG. 1 is realized by the video information analysis unit 115 and the index evaluation unit 116.

[0076] The actual drowsiness level evaluation unit 117 evaluates the actual drowsiness level using, for example, the state analysis result stored in the state analysis result storage unit 143, the state information stored in the state information storage unit 142, the evaluation results of each index except for drowsiness stored in the index evaluation result storage unit 144, and the actual drowsiness level evaluation information stored in the level evaluation information storage unit 150.

[0077] The actual drowsiness level evaluation information is information for evaluating the actual drowsiness level. This evaluation information, like the index evaluation information, is prepared for each category of state represented by the state information. Thus, the state information is used to identify the actual drowsiness level evaluation information stored in the level evaluation information storage unit 150 to refer to. Each actual drowsiness level evaluation information represents the correspondence between each index other than drowsiness, i.e., each evaluation result of fatigue and alertness, and the actual drowsiness level. The actual drowsiness level evaluated with reference to such actual drowsiness level evaluation information is stored in the level evaluation result storage unit 145 secured in the SSD 14. The actual drowsiness level analysis process shown as step S3 in FIG. 1 is implemented by the actual drowsiness level evaluation unit 117.

[0078] The stress effect assessment unit 118 assesses the actual degree of impact of stress using the stress level stored as the analysis result in the state analysis result storage unit 143, the state information stored in the state information storage unit 142, and the stress effect assessment information stored in the impact assessment information storage unit 151.

[0079] The stress impact assessment information is information for assessing the impact level of stress. For people with low stress tolerance, even low stress levels may have a strong impact on their actual behavior. For this reason, it is believed that there are relatively large individual differences in the impact of stress. Therefore, in this embodiment, the impact level is defined as the stress level that actually appears in the behavior of the driver 3, and the impact level is assessed by referring to the stress impact assessment information.

[0080] Even if there are individual differences, if the stress estimated to be felt is at a low level, it is considered that the impact of that stress on the behavior of the driver 3 will also be small. For this reason, the target persons may be, for example, only drivers 3 whose estimated stress level is 2 or higher. This also applies to the actual drowsiness level evaluation unit 117.

[0081] For example, like the index evaluation information, the stress impact assessment information is prepared for each category of the state represented by the state information. As a result, the state information is also used to identify the stress impact assessment information to be referenced from among the stress impact assessment information stored in the impact assessment information storage unit 151.

[0082] The effects of stress may manifest themselves in behaviors that do not or hardly affect attention. Examples of such behaviors include relatively small, continuous movements of the upper body, arms, or head. Facial expressions of anger or sadness may also be observed. For this reason, the degree of impact is evaluated from perspectives other than attention, etc. Each piece of stress impact assessment information makes such an evaluation possible. The impact degree evaluated with reference to such stress impact assessment information is stored in the impact assessment result storage unit 146 secured in the SSD 14. The stress impact analysis process shown as step S4 in FIG. 1 is implemented by the stress impact assessment unit 118.

[0083] The cognitive ability estimation unit 119 estimates the cognitive ability level of the driver 3 from the control indices, i.e., the actual sleep level, the evaluation results of each index excluding drowsiness, and the degree of influence of stress, and determines whether or not it is necessary to control the device group 20 based on the estimation result. The cognitive ability estimation unit 119 checks changes in each control index, selects devices to be controlled from among the devices constituting the device group 20, and determines the control content. For this purpose, the cognitive ability estimation information stored in the estimation information storage unit 152 secured in the SSD 14 is referenced.

[0084] The cognitive ability estimation information is information that defines the type of device to be controlled and the content of that control, for example, by control index, by level represented by that control index, and by the content of changes in the level of that control index. Therefore, the cognitive ability estimation unit 119 outputs the type of device to be controlled and the content of that control as a cognitive ability estimation result by referring to the cognitive ability estimation information using each control index. This estimation result is stored in the cognitive ability estimation result storage unit 147 secured in the SSD 14. The cognitive ability estimation process shown as step S5 in FIG. 1 is realized by the cognitive ability estimation unit 119.

[0085] The device control processing unit 120 performs processing to control the device group 20 in accordance with the type of device to be controlled and the content of the control stored as the estimation result in the cognitive ability estimation result storage unit 147. The device control processing shown as step S6 in FIG. 1 is realized by the device control processing unit 120.

[0086] Control of each device constituting the device group 20 is actually performed by the corresponding ECU. Therefore, the device control processing unit 120 transmits a control request including control information indicating the specified device and the actual control content to the corresponding ECU according to the type of device to be controlled and the control content. This control information is determined by referring to the device control information stored in the device control information storage unit 153 secured in the SSD 14. Therefore, the device control information to be transmitted is defined for each device and each control content, for example. The control request including the control information is transmitted by the device control processing unit 120 via the communication unit 16. Note that the control of the device group 20 is performed as needed, and the device group 20 is not constantly controlled.

[0087] The above-mentioned units 111 to 120 operate while the automobile is in a state in which it can travel. For example, they operate at predetermined time intervals. This allows for timely support to be provided to the driver 3 so that the driver can drive safely. This time interval may be changed depending on the situation, for example, depending on the actual drowsiness level estimated by the driver 3.

[0088] 4 and 5 are flowcharts showing an example of safe driving support processing. This safe driving support processing is processing executed to support safe driving by the driver 3 by controlling necessary equipment through analysis of video information and estimation of cognitive ability based on analysis of the state of the autonomic nerves using biological information. For example, the processing is executed every time a predetermined time interval elapses. Each unit 114 to 120 is realized by executing this processing. Next, the safe driving support processing will be described in detail with reference to FIGS. 4 and 5. The processing is executed mainly by the CPU 11.

[0089] First, in step S11, the CPU 11 analyzes the state of the autonomic nerves using biological information. In the following step S12, the CPU 11 evaluates each index through video analysis using video information. Then, the process proceeds to step S13. In step S13, the CPU 11 determines whether or not there is an abnormal result that may impair safe driving in the result of the autonomic nerve state analysis or in any of the evaluated indexes. For example, if the autonomic nerve state analysis estimates drowsiness or stress, or if the video analysis identifies any of drowsiness, fatigue, or decreased attention, it is determined that there is an abnormality, the determination in step S13 is YES, and the process proceeds to step S16. On the other hand, if there is no abnormality, the determination in step S13 is NO, and the process proceeds to step S14.

[0090] In step S14, the CPU 11 determines whether or not the device is currently being controlled. If the device is currently being controlled, the determination in step S14 becomes YES, and the process proceeds to step S15. If the device is not currently being controlled, that is, if the driver 3 is maintaining a state in which safe driving is possible, the determination in step S14 becomes NO, and the safe driving support process ends here.

[0091] In step S15, the CPU 11 terminates the control of the device that is being controlled. The termination of the control is realized by transmitting a request to the corresponding ECU. After transmitting the request, the safe driving support process ends. Note that the target device here does not include the automatic driving device 24.

[0092] In step S16, the CPU 11 determines whether the vehicle is currently in automatic driving mode. If the vehicle is currently in automatic driving mode, the determination in step S16 is YES, and the safe driving support process ends here. If the vehicle is not currently in automatic driving mode, that is, if the vehicle is currently in manual driving mode, the determination in step S16 is NO, and the process proceeds to step S17.

[0093] In step S17, the CPU 11 determines whether drowsiness has been detected or estimated by analyzing the state of the autonomic nerves. If drowsiness has been detected, that is, estimated, the determination in step S17 is YES, and the process proceeds to step S18. If drowsiness has not been detected, the determination in step S17 is NO, and the process proceeds to step S31 in FIG. 5.

[0094] In step S18, the CPU 11 evaluates the actual drowsiness level using the video analysis results. In the following step S19, the CPU 11 determines whether the evaluation result of the actual drowsiness level is equal to or greater than a set value. If the evaluation result is equal to or greater than the set value, for example, if the actual drowsiness level is 2 or greater, the determination in step S19 is YES and the process proceeds to step S20. If the evaluation result is less than the set value, the determination in step S19 is NO and the process proceeds to step S31 in FIG. 5.

[0095] In step S20, the CPU 11 determines whether or not any device other than the automatic driving device 24 is being controlled. If any device is being controlled, the determination in step S20 becomes YES and the process proceeds to step S22. If no device is being controlled, the determination in step S20 becomes NO and the process proceeds to step S21.

[0096] In step S21, the CPU 11 controls the device to be selected depending on the actual drowsiness level. After that, the safe driving support process ends. The device selected here is for providing a relatively small stimulus to the driver 3. If the actual drowsiness level does not improve, a stronger stimulus will be provided to the driver 3.

[0097] In step S22, the CPU 11 determines whether or not an effect of improving the actual drowsiness level has been recognized. If such an effect has been recognized, the determination in step S22 becomes YES, and the process proceeds to step S21. As a result, the driver 3 continues to be stimulated as before. On the other hand, if such an effect has not been recognized, the determination in step S22 becomes NO, and the process proceeds to step S23. Note that, since it takes a certain amount of time for the effect to appear in the driver 3, this time is taken into consideration when determining whether or not an effect has been recognized. This also applies to step S38, which will be described later.

[0098] In step S23, the CPU 11 determines whether or not there are other options for controlling the device. The other options are options for devices or control contents that provide a stronger stimulus to the driver 3. If there are other options, the determination in step S23 is YES and the process proceeds to step S24. If there are no other options, the determination in step S23 is NO and the process proceeds to step S25.

[0099] In step S24, the CPU 11 selects one of the other options. The option selected here may be, for example, the option that provides the least stimulation among the other options. After the selection, the process proceeds to step S21. As a result, the device is controlled based on the selection result. Meanwhile, in step S25, the CPU 11 requests the automatic driving device 24 to switch to automatic driving. In this way, the vehicle is shifted from manual driving to automatic driving. Thereafter, the safe driving support process ends. For this reason, automatic driving is treated separately from the other options as the option to be finally selected.

[0100] 5, the CPU 11 determines whether stress is detected by analyzing the state of the autonomic nerves. If stress is detected, the determination in step S31 becomes YES, and the process proceeds to step S32. If stress is not detected, the determination in step S31 becomes NO, and the process proceeds to step S34.

[0101] In step S32, the CPU 11 evaluates the degree of impact of stress. In the following step S33, the CPU 11 determines whether the evaluated degree of impact is equal to or greater than a set value. If the evaluated degree of impact is equal to or greater than the set value, for example, if the degree of impact is 2 or greater, the determination in step S33 is YES and the process proceeds to step S36. If the degree of impact is less than the set value, the determination in step S33 is NO and the process proceeds to step S34.

[0102] In step S34, the CPU 11 estimates cognitive ability, and determines whether or not it is necessary to control an appliance based on the estimation result. If it is determined that it is necessary to control an appliance, it determines the appliance to be controlled and the control content. In the following step S35, the CPU 11 performs processing based on the determination result of whether or not it is necessary, the appliance to be controlled, and the control content. Thereafter, the safe driving support processing ends. Note that when proceeding to step S34, it is considered that the state of the autonomic nervous system has a relatively small effect on the driving of the driver 3. For this reason, in step S34, cognitive ability may be estimated using only the evaluation results of each index.

[0103] In step S36, the CPU 11 determines whether or not any device other than the automatic driving device 24 is being controlled. If any device is being controlled, the determination in step S36 becomes YES and the process proceeds to step S38. If no device is being controlled, the determination in step S36 becomes NO and the process proceeds to step S37.

[0104] In step S37, the CPU 11 controls the device to be selected depending on the degree of influence. After that, the safe driving support process ends. The device selected here is intended to provide a relatively small stimulus to the driver 3 to reduce the impact of stress. If the degree of influence is not improved, a stronger stimulus will be provided to the driver 3.

[0105] In step S38, the CPU 11 determines whether or not an effect of improving the degree of influence has been recognized. If such an effect has been recognized, the determination in step S38 becomes YES and the process proceeds to step S37. As a result, the driver 3 continues to be stimulated as before. On the other hand, if such an effect has not been recognized, the determination in step S38 becomes NO and the process proceeds to step S39.

[0106] In step S39, the CPU 11 determines whether or not there are other options for controlling the device. The other options are options for devices or control contents that provide a stronger stimulus to the driver 3 and reduce the effects of stress. If there are other options, the determination in step S39 is YES, and the process proceeds to step S40. If there are no other options, the determination in step S39 is NO, and the safe driving support process ends here. As a result, the same stimulus as before continues to be provided to the driver 3.

[0107] In step S40, the CPU 11 selects one of the other options. For example, the option selected here is the one that provides the least stimulation among the other options. After the selection, the process proceeds to step S37. The device is then controlled based on the selection result.

[0108] In this manner, in this embodiment, a stimulus is given to the driver 3 who is estimated to be in an undesirable state for safe driving. If the stimulus does not result in improvement, gradually stronger stimuli are given. If the driver 3 is so sleepy that the stimulus does not result in improvement, the system forcibly shifts to automated driving. Note that the stimulus includes providing information by audio output. If the provision of audio information does not result in improvement, physical stimuli other than audio output are also provided. The reason for the process flow of gradually giving stronger stimuli is that it is considered unlikely that an inappropriate control index will change while the device is being controlled. For example, if the driver 3 is feeling very sleepy, it is considered unlikely that he or she will experience strong stress in a relatively short period of time.

[0109] In the functional configuration exemplified in Fig. 4, the biological information acquisition unit 111 corresponds to biological information acquisition means and image information acquisition means. The autonomic nerve state analysis unit 114 corresponds to state estimation means. The image information analysis unit 115, index evaluation unit 116, actual drowsiness level evaluation unit 117, stress effect evaluation unit 118, and cognitive ability estimation unit 119 correspond to cognitive ability estimation means. Furthermore, the vehicle information acquisition unit 112 and environmental information acquisition unit 113 correspond to state information acquisition means. The device control processing unit 120 corresponds to control processing means.

[0110] In this embodiment, a vehicle is assumed as the mobile device, but the mobile device is not limited to a vehicle. The mobile device may be a train, an airplane, a helicopter, or other flying vehicle, or a ship. Such mobile devices differ not only in their characteristics but also in the constraints imposed on the subject when driving or operating them. For example, a train can only run on rails. Furthermore, the presence of obstacles, etc., usually does not need to be taken into consideration. For these reasons, it is necessary to determine the method for evaluating each index and the method for estimating cognitive ability depending on the type of mobile device. However, this embodiment can also be applied to mobile devices other than automobiles.

[0111] Furthermore, in this embodiment, as described above, the indices evaluated from the video information and the results obtained by the autonomic nervous state analysis are used in a complementary manner. However, the estimation of drowsiness, etc., can be performed more quickly using the autonomic nervous state analysis. For this reason, control may be divided into control using the estimation results from the autonomic nervous state analysis and control using both the estimation results and the evaluation results using the video information. As a result, for example, if drowsiness is estimated by the autonomic nervous state analysis, a warning may be issued by audio output, and if drowsiness is also estimated in the evaluation using the video information, a physical stimulus may also be given by controlling the device. Various modifications, including those described above, are possible.

[0112] 1 Cognitive ability estimation device, 2 Driver's seat, 3 Driver (subject), 4, 7 Camera, 6 Smart watch, 8 Radar, 9 ECU, 11 CPU, 14 SSD, 16 Communication unit, 20 Equipment group, 21 Air conditioning equipment control device, 22 Steering control device, 23 Message output control device, 24 Autonomous driving device, 31 Pulse sensor, 41 Steering angle sensor, 42 Brake / access sensor, 43 G sensor, 44 Speedometer, 111 Biometric information acquisition unit, 112 Vehicle information acquisition unit, 113 Environmental information acquisition unit, 114 Autonomic nervous state analysis unit, 115 Video information analysis unit, 116 Index evaluation unit, 117 Actual sleep level evaluation unit, 118 Stress impact evaluation unit, 119 Cognitive ability estimation unit, 120 Equipment control processing unit.

Claims

1. A means for acquiring biometric information that acquires biometric information that can at least identify the heart rate of a person driving or operating a mobile vehicle, A video information acquisition means for acquiring video information of the aforementioned subject, A state estimation means for estimating the state of the subject's autonomic nervous system based on the heart rate identified from the aforementioned biological information, A cognitive ability estimation means for estimating the cognitive ability of the subject based on the estimation result of the state of the autonomic nervous system by the state estimation means and the video information, Equipped with, The cognitive ability estimation means evaluates the subject's state using multiple indicators, including drowsiness, based on the video information, and estimates the cognitive ability using the evaluation results of each indicator and the drowsiness level of the subject represented by the estimated state of the autonomic nervous system. Cognitive ability estimation device.

2. A means for acquiring biometric information that acquires biometric information that can at least identify the heart rate of a person driving or operating a mobile vehicle, A video information acquisition means for acquiring video information of the aforementioned subject, A state estimation means for estimating the state of the subject's autonomic nervous system based on the heart rate identified from the aforementioned biological information, A cognitive ability estimation means for estimating the cognitive ability of the subject based on the estimation result of the state of the autonomic nervous system by the state estimation means and the video information, Equipped with, The cognitive ability estimation means evaluates the subject's state using one or more indicators based on the video information, and uses the evaluation results of each indicator and the estimated results of the autonomic nervous system state to estimate the degree to which the stress level affects each evaluation result of multiple indicators, including drowsiness. Cognitive ability estimation device.

3. A means for acquiring biometric information that acquires biometric information that can at least identify the heart rate of a person driving or operating a mobile vehicle, A video information acquisition means for acquiring video information of the aforementioned subject, A state estimation means for estimating the state of the subject's autonomic nervous system based on the heart rate identified from the aforementioned biological information, A cognitive ability estimation means for estimating the cognitive ability of the subject based on the estimation result of the state of the autonomic nervous system by the state estimation means and the video information, A control processing means for performing processing for controlling a first device capable of providing the subject with stimuli to improve their cognitive ability based on the results of the cognitive ability estimation means, Equipped with, If the mobile body is equipped with an automatic driving function, the control processing means performs processing for the automatic driving of the mobile body by the automatic driving function based on the result of the cognitive ability estimation by the cognitive ability estimation means after controlling the first device. Cognitive ability estimation device.

4. The cognitive ability estimation means evaluates the actual sleepiness level of the subject, which is the subject's actual sleepiness level, using the sleepiness level of the subject represented by the estimation result of the autonomic nervous system state and the evaluation results of the multiple indicators. The cognitive ability estimation device according to claim 1.

5. The system further comprises a state information acquisition means capable of acquiring state information representing the state of the moving body, The cognitive ability estimation means estimates the cognitive ability of the subject based on the estimation result of the autonomic nervous system state by the state estimation means, the video information, and the state information. A cognitive ability estimation device according to any one of claims 1 to 3.

6. The system further comprises control processing means for controlling a first device capable of providing the subject with stimuli to improve their cognitive abilities based on the results of the cognitive ability estimation means. The cognitive ability estimation device according to claim 1 or 2.

7. The control processing means performs processing for controlling a second device capable of notifying people other than the target person. The cognitive ability estimation device according to claim 6.

8. In an information processing device, To acquire at least biometric information that allows identification of the heart rate of the person driving or operating the mobile vehicle, Obtain video information of the aforementioned subject, Based on the heart rate identified from the aforementioned biological information, the state of the subject's autonomic nervous system is estimated. Based on the aforementioned video information, the subject's condition is evaluated using multiple indicators, including drowsiness, and the subject's cognitive ability is estimated using the evaluation results of each indicator and the estimated drowsiness level of the subject, which are represented by the estimated state of the autonomic nervous system. A program that executes a process.