State determination method, state determination device, and learned model for state determination
A cerebral blood flow sensor and trained model using a convolutional neural network address the issue of noise interference in brain activity detection, enabling accurate driver state determination for safe vehicle operation.
Patent Information
- Application Number
- JP2023082607
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-28
AI Technical Summary
Existing techniques for determining a vehicle driver's state using brain activity sensors fail when noise, such as body movements, interferes with accurate detection of brain activity.
A method involving a cerebral blood flow sensor to measure brain activity, combined with a trained model that uses teacher data to determine the driver's condition, even in areas where brain activity is inaccurately detected, utilizing a convolutional neural network to improve robustness and accuracy.
Enables accurate determination of the driver's condition, including states requiring driving assistance, by improving detection robustness against noise and deviations, ensuring safe vehicle operation.
Smart Images

Figure 2024165972000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a state determination method, a state determination device, and a trained model for state determination. [Background technology]
[0002] A technology is known that uses the detection results of a brain activity sensor that can detect activated areas inside the brain of a vehicle driver to determine whether the driver's level of anxiety exceeds a threshold, and if it is determined that the threshold is exceeded, controls the vehicle to respond to the driver's high level of anxiety (Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2016-64773 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, the technology of Patent Document 1 has a problem in that, for example, when there are parts of the brain where brain activity cannot be accurately detected due to noise such as the driver's body movements, the driver's state cannot be determined.
[0005] The problem that the present invention aims to solve is to provide a state determination method, a state determination device, and a trained model for state determination that can determine a driver's state even if there are parts of the driver's brain where brain activity cannot be accurately detected. [Means for solving the problem]
[0006] The present invention solves the above problem by inputting a distribution image including the distribution of cerebral blood flow in at least a portion of a vehicle driver into a trained model for state determination, and outputting the driver's state from the trained model for state determination, wherein the trained model for state determination is a trained model that has been trained to determine the driver's state based on teacher data including the distribution image and the driver's state. Effect of the Invention
[0007] According to the present invention, even if there is a part in the driver's brain where brain activity cannot be detected accurately, the driver's state can be determined. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a block diagram showing a configuration of a driving support system including a state determination device according to this embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a flowchart of a control process of a state determining method executed by the state determining device according to the present embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a flowchart of a control process for generating a trained model for state determination according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. A state determination device according to an embodiment of the present invention will be described with reference to an example in which the state determination device is applied to a driving assistance system. Note that the state determination device is not limited to being applied to a driving assistance system, and may be applied to other systems, etc.
[0010] FIG. 1 is a diagram showing a block configuration of a driving assistance system including a state determination device according to this embodiment. The driving assistance system 1 includes a cerebral blood flow sensor 20, a state determination device 100, a vehicle controller 200, and an output device 300. Each device is mounted on a vehicle and is connected to each other by wire or wirelessly so that information can be exchanged between them. For example, each device is connected by a CAN (Controller Area Network) or other in-vehicle LAN. The driving assistance system 1 controls the traveling of the vehicle to assist the driving of the vehicle. The driving assistance system 1 can be applied not only to the traveling of the vehicle by autonomous driving control, but also to the case of assisting the traveling of the vehicle by manual driving by the driver. In addition, when applied to the autonomous driving control of a vehicle, the driving assistance system 1 can be applied to the case of autonomously controlling both the speed control and the steering control, as well as the case of autonomously controlling one of the speed control and the steering control and manually controlling the other.
[0011] In this embodiment, the driving assistance system 1 assists in driving the vehicle based on the determination result of the driver's state by the state determination device 100. The state determination device 100 determines the driver's state and transmits the determination result of the driver's state to the vehicle controller 200 and / or the output device 300. The vehicle controller 200 executes autonomous driving control of the vehicle according to the determination result of the driver's state. The output device 300 outputs notification information to the occupants of the vehicle, including the driver, according to the determination result of the driver's state.
[0012] In the case of autonomous driving control, the system performs part or all of the driving, but the driver needs to monitor the surrounding traffic conditions and the operation of the system. Therefore, the state determination device 100 determines in what state the driver is monitoring the autonomous driving. For example, when the driver is not concentrating on manual driving while the vehicle is running, the driving assistance system 1 switches the driving of the vehicle to autonomous control. In addition, when the driver is monitoring the autonomous driving and is not awake, the driving assistance system 1 outputs notification information to wake the driver up. In this embodiment, a cerebral blood flow sensor is used to measure brain activity such as the driver's thoughts other than driving, and the state of the driver is determined from the brain activity. A specific method will be described below.
[0013] The cerebral blood flow sensor 20 is a sensor that is attached to the head of the driver and detects the cerebral blood flow of the driver. For example, the cerebral blood flow sensor 20 is a functional near infrared spectroscopy (fNIRS). The functional near infrared spectroscopy irradiates near infrared light with high biological permeability from above the scalp and receives the light that has passed through the brain tissue to simultaneously detect changes in oxygenated hemoglobin concentration (Oxy-Hb) and deoxygenated hemoglobin concentration (deOxy-Hb) in the brain. In this embodiment, the cerebral blood flow sensor 20 detects changes in cerebral blood flow using an Ox-Hb signal, which is said to be the best indicator of changes in cerebral blood flow coupled with neural activity. The cerebral blood flow sensor 20 detects the cerebral blood flow at a predetermined position on the driver's head, such as the forehead. There are a plurality of predetermined positions, and each predetermined position is arranged at a predetermined interval. A probe sensor is installed at each predetermined position. That is, the probe sensor is arranged at a plurality of positions on the driver's head at a predetermined interval to detect a signal.
[0014] The cerebral blood flow sensor 20 detects the cerebral blood flow rate at a predetermined time in a chronological order at a constant cycle, and obtains time series data showing the change in the cerebral blood flow rate at the predetermined time. For example, the cerebral blood flow sensor 20 detects time series data showing the change in the cerebral blood flow rate while the vehicle is running. The cerebral blood flow sensor 20 also obtains a distribution image including the distribution of the cerebral blood flow at the same time based on the cerebral blood flow rate at each predetermined position detected at the same time. As the distribution image, a plurality of images are obtained in chronological order. The distribution image is a two-dimensional matrix showing the increase or decrease in the cerebral blood flow rate in each region in the brain. The distribution image includes the distribution of the region where the cerebral blood flow rate is increased and the region where the cerebral blood flow rate is decreased. For example, in the distribution image, the region where the cerebral blood flow rate is increased is shown in a warm color, and the region where the cerebral blood flow rate is decreased is shown in a cool color. The distribution image also shows the amount of change in the cerebral blood flow in each region. For example, the greater the amount of change, the darker the color. The distribution of the cerebral blood flow is a distribution including the cerebral blood flow at at least a part of the head, that is, all or some of the predetermined positions.
[0015] The state determination device 100 according to this embodiment will be described below. The state determination device 100 determines the state of the driver of the vehicle. The determination of the driver's state is a determination of whether or not the driver needs driving assistance. The state in which the driver needs driving assistance is a state in which the driver cannot operate and / or monitor driving, such as a state in which the driver is not concentrating on operating and / or monitoring driving, a state in which the driver is not awake, and a state in which the driver feels anxious. The state in which the driver does not need driving assistance is a state in which the driver can operate and / or monitor driving, such as a state in which the driver is concentrating on operating and / or monitoring driving, a state in which the driver is awake, and a state in which the driver does not feel anxious. The state determination device 100 acquires a distribution image including the distribution of the driver's cerebral blood flow from the cerebral blood flow sensor 20, and determines the state of the driver based on the distribution image. The state determination device 100 includes a processor 10 and a storage device 30.
[0016] The processor 10 is a computer including a ROM 12 in which a program for executing a state determination process for determining the state of the driver is stored, a CPU 11 as an operating circuit that functions as a state determination device 100 by executing the program stored in the ROM 12, and a RAM 13 that functions as an accessible storage device. The processor 10 exchanges information with the cerebral blood flow sensor 20, the vehicle controller 200, and the output device 300.
[0017] The processor 10 includes, as functional blocks, an acquisition unit 110, a determination unit 111, a control instruction unit 112, and a learning unit 113. The processor 10 of this embodiment executes each function by cooperation between the software for implementing each of the above functions or executing each process and the above hardware. In this embodiment, the functions of the processor 10 are divided into four blocks, and the functions of each functional block are described. However, the functions of the processor 10 do not necessarily need to be divided into four blocks, and may be divided into three or less functional blocks, or five or more functional blocks. In addition, although these functional blocks are provided in the processor 10, this is not limited thereto, and they may be provided in other devices. For example, although the processor 10 includes the learning unit 113, this is not limited thereto, and the learning unit 113 may be provided in a device external to the processor 10. In such a case, the processor 10 may acquire a trained model for state determination trained externally from an external device and store it in the storage device 30.
[0018] The acquisition unit 110 executes an image acquisition process to acquire a distribution image including the distribution of at least a part of the driver's cerebral blood flow. For example, the acquisition unit 110 starts the image acquisition process when the vehicle starts to run. That is, the acquisition unit 110 acquires a distribution image including the distribution of the driver's cerebral blood flow from the cerebral blood flow sensor 20 at a constant period while the vehicle is running. The acquisition unit 110 may acquire time series data indicating a change in the cerebral blood flow from the cerebral blood flow sensor 20.
[0019] The acquisition unit 110 may also acquire a distribution image that has been subjected to a process for removing noise. The acquisition unit 110 performs a process for removing noise on the time series data acquired from the cerebral blood flow sensor 20, and acquires a distribution image based on the time series data after the process for removing noise. For example, the acquisition unit 110 performs a process for removing noise caused by the body movement of the driver on the time series data of the cerebral blood flow using a body movement filter. The acquisition unit 110 also performs a process for removing noise caused by optical disturbance on the time series data of the cerebral blood flow using a light countermeasure filter. Specifically, the acquisition unit 110 calculates the frequency of the body movement, the frequency of the light, and the frequency of the cerebral blood flow, and removes noise caused by the body movement and / or noise caused by optical disturbance using a bandpass filter. The acquisition unit 110 acquires a distribution image by generating a distribution image including the distribution of the cerebral blood flow detected at the same time point from the time series data of the cerebral blood flow that has been subjected to a process for removing noise at multiple positions.
[0020] The acquisition unit 110 may also acquire teacher data including the distribution image and the driver's state. For example, the acquisition unit 110 acquires the teacher data when the teacher data is input by the user. The teacher data will be described later.
[0021] The determination unit 111 executes a determination process for determining the state of the driver. In the determination process, the determination unit 111 inputs input data including a distribution image acquired by the acquisition unit 110 to a trained model for state determination, and determines the state of the driver by outputting the state of the driver from the trained model for state determination as output data corresponding to the input data. The determination unit 111 may also input input data including a distribution image that has been subjected to a process for removing noise to the trained model for state determination. The trained model for state determination will be described later.
[0022] The driver's state includes a state in which the driver is concentrating on driving, a state in which the driver is awake, and a state in which the driver feels anxious. The trained model for state determination is trained so that a distribution image corresponds to each state of the driver. As a result, when a distribution image is input, the trained model for state determination outputs output data including the driver's state corresponding to the distribution image among each state of the driver.
[0023] In this embodiment, the determination unit 111 determines whether the driver is in a state requiring driving assistance based on the state of the driver output from the trained model for state determination. For example, the determination unit 111 determines that the driver is in a state requiring driving assistance when the trained model for state determination outputs output data indicating that the driver is not concentrating on driving, is not awake, and / or is feeling anxious. Furthermore, the determination unit 111 determines that the driver is in a state not requiring driving assistance when the trained model for state determination outputs output data indicating that the driver is concentrating on driving, is awake, and / or is not feeling anxious.
[0024] The control instruction unit 112 executes at least one of driving assistance control of the vehicle and notification to the occupant according to the determination result determined by the determination unit 111. The control instruction unit 112 transmits a control instruction to the vehicle controller 200 and / or the output device 300 according to the determination result. In this embodiment, the control instruction unit 112 transmits a control instruction to the vehicle controller 200 and / or the output device 300 when the driver is in a state requiring driving assistance. The control instruction is, for example, a control instruction to the vehicle controller 200 to execute autonomous control of the vehicle. In addition, the control instruction is, for example, a control instruction to the output device 300 to notify the occupant of notification information including the driver's state.
[0025] In addition, the control instruction unit 112 may change the assistance level related to the driving assistance control according to the judgment result, and execute the driving assistance control according to the changed assistance level. For example, when the control instruction unit 112 judges that the driver is in a state requiring driving assistance, the control instruction unit 112 changes the assistance level to a higher level than when the control unit 112 judges that the driver is in a state requiring no driving assistance. The control instruction unit 112 generates a control instruction for executing the driving assistance control at the changed assistance level, and outputs it to the vehicle controller 200. The assistance level defines the extent to which the driving assistance system 1 intervenes in the control of the vehicle's driving operation, in other words, the extent to which the driver's manual operation intervenes. The driving assistance system 1 realizes the assistance of the driving operation defined by each assistance level when autonomously controlling the driving of the vehicle by combining the autonomous control of the driving speed using the autonomous speed control function and the autonomous control of the steering operation using the autonomous steering control function.
[0026] The assistance level in this embodiment can be set, for example, based on the driving automation level defined in SAE J3016: SEP2016, Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles published by the Society of Automotive Engineers (SAE).
[0027] The control instruction unit 112 may also execute a notification to the driver. For example, when the control instruction unit 112 determines that the driver is in a state requiring driving assistance, the control instruction unit 112 outputs, to the output device 300, a control instruction to output image information and / or audio information indicating that the driver is in a state requiring driving assistance to an occupant.
[0028] In this embodiment, at least one of the vehicle driving assistance control and the notification to the occupant is executed according to the determination result of whether or not the driver needs driving assistance. However, the present invention is not limited to this as long as at least one of the vehicle driving assistance control and the notification to the occupant is executed according to the driver's state. For example, at least one of the vehicle driving assistance control and the notification to the occupant may be executed according to the determination result of the driver not concentrating on driving, the driver not awake, and / or the driver feeling anxious. For example, when the driver is not awake, the control instruction unit 112 may output notification information for waking up the driver. In this embodiment, the control instruction unit 112 is not a required configuration and may be applied as necessary. For example, the processor 10 is not limited to having the control instruction unit 112 as a function, and may transmit only the determination result by the determination unit 111 to the vehicle controller 200 and / or the output device 300. In this case, the vehicle controller 200 and / or the output device 300 executes each function according to the determination result.
[0029] The learning unit 113 uses teacher data including the distribution image and the driver's state to train a neural network and generate a trained model for state determination for determining the driver's state. For example, the learning unit 113 uses an open source machine learning library PyTorch to train using a convolutional neural network (CNN). The teacher data is data in which the distribution image and the driver's state are associated with each other. The distribution image included in the teacher data is also referred to as a distribution image for training. The distribution image for training is a distribution image acquired to generate a trained model for state determination, and is a distribution image acquired in a simulation and / or an experiment. For example, the distribution image for training includes a distribution of cerebral blood flow acquired by a cerebral blood flow sensor attached to the head of an experiment participant. As the distribution image for training, a plurality of distribution images are acquired in a predetermined time (for example, one minute) in an experiment or the like. The distribution image for training is a distribution image including at least a part of the distribution of cerebral blood flow in a human head.
[0030] Here, acquisition of teacher data will be described by taking as an example a case where the driver's state is determined to be whether or not the driver is concentrating. For example, in an experiment for acquiring teacher data, a time period during which the participant only monitors driving (a time period during which the participant concentrates on monitoring driving) and a time period during which the participant monitors driving and performs other tasks (a time period during which the participant is not concentrating on monitoring driving) are provided, and learning time series data showing changes in the participant's cerebral blood flow rate at each time period (e.g., one minute each) is acquired. Also, at each time period, a distribution image including the distribution of the cerebral blood flow rate at each time point is acquired. For example, one learning distribution image is acquired every second, i.e., 60 learning distribution images are acquired. Then, each distribution image is associated with a state of concentrating on monitoring or a state of not concentrating on monitoring, respectively.
[0031] In this embodiment, for example, in the convolutional neural network used for learning, the kernel of the convolutional layer is a 3×3 matrix, and the activation function is Re-LU. In the pooling layer, max pooling is used to reduce the feature amount and the calculation cost. The convolutional neural network is configured to apply the convolutional layer three times and finally perform full connection. The loss function is cross-entropy error. In addition, the convolutional neural network is configured as an end-to-end learning configuration that learns everything from the distribution image of the input layer to binary data on the driver's state of the output layer (for example, whether the driver is concentrating on monitoring the driving or not). Note that the configuration of the convolutional neural network is not limited to these configurations, and other configurations may be applied as appropriate as necessary.
[0032] In this embodiment, by using a convolutional neural network for learning, a trained model can be generated so that the driver's state can be judged more accurately even if there is a part in the distribution image where the cerebral blood flow cannot be accurately detected due to noise or deviation. Since the convolutional neural network is composed of a convolutional layer and a pooling layer, the robustness of the judgment can be improved by the convolution operation and pooling process. The convolution operation provides translation invariance, so that the robustness against local numerical deviations is improved. In addition, the pooling process uses max pooling, so that even if noise occurs in some signals, the pooling of the maximum value does not change, so that the robustness against local noise is improved.
[0033] In this embodiment, the learning unit 113 is not limited to learning the correspondence between the activated area in the distribution image and the driver's state, but also learns the correspondence between the distribution image of the entire brain including the activated area and the non-activated area and the driver's state. As a result, the trained model for state determination can more accurately determine the driver's state by learning using the distribution characteristics of the cerebral blood flow in both the activated area and the non-activated area, rather than learning using the distribution characteristics of the activated area.
[0034] The learning unit 113 may also use teacher data in which the time series data of cerebral blood flow is associated with the driver's state to learn the neural network and generate a trained model for state determination for determining the driver's state. The time series data included in the teacher data is time series data for learning. For example, the time series data for learning is data including changes in cerebral blood flow over a predetermined time. The predetermined time is, for example, one minute. In this embodiment, the learning unit 113 performs learning based on the time series data using linear discrimination analysis (LDA). As described above, the learning unit 113 generates a trained model for state determination based on the distribution image and / or the time series data. The learning unit 113 stores the generated trained model for state determination in the storage device 30.
[0035] In the present embodiment, the learning unit 113 is not limited to performing learning using the time series data in the activated area, but may also perform learning including the time series data in the inactivated area. As a result, the driver's state can be determined more accurately by performing learning including the time series data in both the activated area and the inactivated area, rather than by performing learning using the time series data in the activated area.
[0036] As described above, in this embodiment, the learning unit 113 performs learning using time series data of cerebral blood flow and a distribution image of cerebral blood flow. That is, learning is performed using two-sided data of temporal changes and spatial changes in the brain. In the judgment by machine learning, feature amounts are extracted including areas not related to specific brain activity, and the accuracy of the judgment can be improved. For example, even if there is a part in the driver's brain where brain activity cannot be accurately detected, such as an area related to specific brain activity, the judgment accuracy of the trained model for state judgment is improved by learning based on feature amounts including other areas not related to the specific brain activity, so that the state judgment device 100 can judge the state of the driver using the trained model for state judgment.
[0037] The storage device 30 is a storage medium that stores various information. The storage device 30 stores a trained model for state determination 31. The trained model for state determination 31 is a trained model that has been trained by the learning unit 113 to determine the state of the driver based on teacher data including a distribution image and the state of the driver. For a specific learning method, the description of the functions of the learning unit 113 described above is used as appropriate.
[0038] The trained model for state determination is composed of a neural network including an input layer to which input data including a distribution image including the distribution of at least a portion of the cerebral blood flow of the driver of the vehicle is input, and an output layer to which output data including the state of the driver is output. The trained model for state determination is trained to determine the state of the driver using teacher data that associates the input data with the output data. When input data is input to the input layer, the trained model for state determination causes the computer to function so as to output output data corresponding to the input data from the output layer.
[0039] The trained model for state determination may be a trained model trained based on the distribution features of cerebral blood flow extracted from a distribution image (distribution image for training) included in the teacher data. For example, the training unit 113 may divide the distribution image into a predetermined number of regions (meshes) and extract the ratio or correlation coefficient between each region as a feature. When two regions are correlated, the ratio or correlation coefficient between the two regions is calculated to be higher than when the regions are not correlated. For example, when the cerebral blood flow in both regions increases or decreases, the two regions are correlated. When the cerebral blood flow in one of the two regions increases and the other decreases, the two regions are not correlated. The training unit 113 may extract the distribution features of cerebral blood flow from the distribution image using a CNN algorithm. The method of extracting features from the distribution image is not limited to this, and other known methods may be used.
[0040] The teacher data may include a distribution image that has been processed to emphasize the characteristics of the distribution of cerebral blood flow. For example, the distribution image included in the teacher data is a distribution image that indicates the difference between the distribution of the cerebral blood flow of the driver under stable conditions as a reference (baseline) and the distribution of the cerebral blood flow in other states of the driver. The stable condition is, for example, a DMN (Default Mode Network) state. The other states of the driver include, for example, a state in which the driver is concentrating on driving, a state in which the driver is awake, and a state in which the driver feels anxious. In this embodiment, the learning unit 113 acquires, for example, in an experiment or the like, a distribution image including the distribution of the cerebral blood flow of the driver under stable conditions and a distribution image including the distribution of the cerebral blood flow in a state in which the driver is concentrating on driving from a cerebral blood flow sensor attached to the head of the driver. Based on the acquired distribution image, the learning unit 113 acquires a distribution image for learning that indicates the difference between the distribution in a state in which the driver is concentrating on driving and the distribution under stable conditions.
[0041] Here, an example of a method for acquiring a distribution image processed to emphasize the distribution characteristics of cerebral blood flow will be described. The learning unit 113 acquires, for example, a distribution image A of the cerebral blood flow of a driver in a predetermined state other than stable conditions, and a distribution image B of the cerebral blood flow of a driver under stable conditions. For example, in distribution image A, the left and right outer parts are more activated than the center part, and the change in cerebral blood flow is large, but in distribution image B, the left and right outer parts are also more activated than the center part, and the change in cerebral blood flow is large. In distribution image C showing the difference between both distribution images, the change in cerebral blood flow in the left and right outer parts is small because it is compared with the change in cerebral blood flow under stable conditions.
[0042] The distribution image processed to emphasize the distribution characteristics of the cerebral blood flow may be a distribution image showing the difference between the distribution of the cerebral blood flow at a predetermined time point and a baseline, which is an average value of a plurality of distributions acquired at a predetermined time before the predetermined time point. The predetermined time is, for example, 30 seconds.
[0043] The distribution image processed to emphasize the distribution characteristics of the cerebral blood flow may be a distribution image divided into a predetermined number of regions (meshes) and each region is weighted. In this embodiment, a weight is set in advance for a specific region among the regions in the brain. For example, since the region to be activated in the brain is predicted for each state of the driver, the region to be activated in the brain when the driver is in a predetermined state is set as the specific region. In this case, the distribution image processed to emphasize the distribution characteristics of the cerebral blood flow is an image including a distribution of cerebral blood flow obtained by integrating the weight corresponding to each region in the brain.
[0044] The teacher data may include a distribution image that does not include signal fluctuations other than the distribution of cerebral blood flow. The distribution image is also referred to as a pre-processing distribution image. The distribution image that does not include signal fluctuations other than the distribution of cerebral blood flow may be a distribution image from which noise other than the distribution of cerebral blood flow has been removed, or may be a distribution image including the distribution of the driver's cerebral blood flow detected under conditions in which noise due to external disturbance does not occur. The distribution image from which noise other than the distribution of cerebral blood flow has been removed is, for example, a distribution image to which a low-pass filter of 0.1 Hz has been applied after trend removal to remove noise caused by blood pressure fluctuations, respiratory fluctuations, body movements, etc. As a result, since the distribution image is noise-free, it is easier to extract the characteristics of the distribution more accurately, and the learning speed can be increased.
[0045] The teacher data may include a distribution image including a variation in a signal other than the distribution of cerebral blood flow. For example, the distribution image for learning is a distribution image that has been processed to generate noise. The distribution image for learning may be a distribution image in which some signals of the distribution of cerebral blood flow are small or does not include some signals of the distribution of cerebral blood flow. For example, the distribution image is a distribution image in which a part of the distribution of cerebral blood flow is masked, or a distribution image in which some of the distribution of cerebral blood flow includes an error. Specifically, the learning unit 113 performs learning using a distribution image in which Gaussian noise is added to some of the cerebral blood flow signals detected at each predetermined position of the head. In addition, the teacher data may include a distribution image before processing and a distribution image after processing to generate noise in the distribution image before processing. By using a distribution image including noise for learning, overlearning of the learning model can be suppressed, robustness can be increased, and the accuracy of judgment can be improved.
[0046] The learning distribution image may be a distribution image having noise in a part of the cerebral blood flow distribution. The distribution image having noise in a part of the cerebral blood flow distribution is, for example, a distribution image in which noise is generated by performing a geometric transformation process on a part of the cerebral blood flow distribution. The geometric transformation process is a transformation process such as enlarging, reducing, or rotating an image. The distribution image having noise in a part of the cerebral blood flow distribution may be a distribution image including white noise in a part of the cerebral blood flow distribution.
[0047] The distribution image including the fluctuation of a signal other than the distribution of cerebral blood flow may be a distribution image including the distribution of the driver's cerebral blood flow detected under a condition where noise due to disturbance occurs, such as a condition where there is solar radiation, a condition where the driver moves, and / or a condition where the cerebral blood flow is detected from above the driver's hair.
[0048] The vehicle controller 200 is an in-vehicle computer such as an electronic control unit (ECU), and electronically controls a drive mechanism that governs the operation of the vehicle. The vehicle controller 200 controls a drive device, a braking device, and a steering device included in the drive mechanism to run the vehicle.
[0049] The drive mechanism includes an electric motor and / or an internal combustion engine as a driving source for driving the vehicle, a power transmission device including a drive shaft and an automatic transmission that transmits the output from these driving sources to the drive wheels, a drive device that controls the power transmission device, and a braking device that brakes the wheels. The vehicle controller 200 generates control signals for each of these drive mechanisms and controls driving including acceleration and deceleration of the vehicle. By sending control information to the drive mechanism, driving control including acceleration and deceleration of the vehicle can be automatically performed. The drive device and the braking device control the acceleration and deceleration of the vehicle based on the control signals obtained from the vehicle controller 200 or input signals due to the driver's accelerator and brake operations.
[0050] The vehicle controller 200 also controls the steering device so that the vehicle travels while maintaining a predetermined lateral position with respect to the travel path (trajectory). The travel path in this specification is a path corresponding to the trajectory along which the vehicle travels, and can be identified by road, uphill / downhill of the road, and lane of the road. The steering device includes a steering actuator. The steering actuator includes a motor and the like attached to a steering column shaft. The steering device executes steering control of the vehicle based on a control signal acquired from the vehicle controller 200 or an input signal by the driver's steering operation.
[0051] In this embodiment, the vehicle controller 200 executes autonomous driving control in accordance with the state of the driver determined by the state determination device 100. For example, the vehicle controller 200 executes autonomous driving control including acceleration / deceleration control and steering control of the vehicle based on a control instruction transmitted from the state determination device 100.
[0052] The output device 300 is a device for outputting information and providing the information to the driver. The output device 300 may be a device mounted on a vehicle, or may be a device mounted on a portable terminal such as a mobile phone or a tablet terminal used by the driver. For example, the output device 300 is a device for displaying image information, and is configured by a liquid crystal display, a projector, or the like. The output device 300 may also be a device for outputting audio information. For example, the output device 300 is configured by a speaker.
[0053] In the present embodiment, the output device 300 outputs notification information to the driver in accordance with the state of the driver determined by the state determination device 100. For example, the output device 300 notifies the driver of image information and / or audio information based on a control instruction transmitted from the state determination device 100.
[0054] Next, an example showing a control procedure for executing the state determination method according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of a flowchart of the control process of the state determination method executed by the state determination device according to this embodiment. In this embodiment, when the vehicle starts to travel, the control flow starts from step S1. Also, while the vehicle is traveling, the control flow shown in Fig. 2 is repeatedly executed at a constant cycle. As a result, while the vehicle is traveling, the state of the driver is monitored by the state determination device 100.
[0055] In step S101, the state determination device 100 acquires a distribution image including the distribution of cerebral blood flow of at least a part of the driver of the vehicle. For example, the state determination device 100 acquires the distribution image from the cerebral blood flow sensor 20. The cerebral blood flow sensor 20 is a sensor attached to the head of the driver. In step S102, the state determination device 100 inputs input data including the distribution image acquired in step S101 to the trained model for state determination 31. In step S103, the state determination device 100 judges the state of the driver. For example, the state determination device 100 judges the state of the driver by outputting the state of the driver from the trained model for state determination as output data corresponding to the input data.
[0056] In FIG. 2, the control flow is ended after the state of the driver is determined in step S103, but this is not limited thereto. After step S103, the state determination device 100 may execute at least one of the driving assistance control of the vehicle and the notification to the occupant according to the determination result, i.e., the state of the driver. For example, when the driver is in a state requiring driving assistance, the state determination device 100 transmits a control instruction to the vehicle controller 200 to execute the autonomous driving control of the vehicle, and / or a control instruction to the output device 300 to output notification information indicating that the driver is in a state requiring driving assistance. In addition, the state determination device 100 is not limited to transmitting a control instruction, and may transmit only the determination result to the vehicle controller 200 and / or the output device 300, or may transmit the determination result to another control device that controls the vehicle controller 200 and / or the output device 300.
[0057] Furthermore, the state determination device 100 may end the control flow when the driver does not require driving assistance, or may stop the driving assistance when, for example, driving assistance is being performed by the driving assistance system 1. Specifically, when autonomous driving control is being performed, the state determination device 100 may output a control instruction to switch to manual driving by the driver.
[0058] Next, an example of a control procedure for generating a trained model for state determination according to the present embodiment will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of a flowchart of a control process for generating a trained model for state determination according to the present embodiment. In the present embodiment, for example, when teacher data is input to the state determination device 100 by a user, the control flow starts from step S201. Note that in FIG. 3, the state determination device 100 is configured to perform learning of the convolutional neural network, but this is not limited thereto, and the state determination device 100 may acquire a trained model for state determination trained outside the state determination device 100. In addition, the neural network may be configured as an end-to-end learning that learns everything from the distribution image of the input layer to the binary data related to the driver's state of the output layer all at once.
[0059] In step S201, the state determination device 100 acquires a distribution image (distribution image for learning) including the distribution of cerebral blood flow of at least a part of the driver of the vehicle, and teacher data including the state of the driver. The distribution image for learning is a distribution image of cerebral blood flow acquired by simulation and / or experiment. The distribution image acquired by simulation and / or experiment may be a distribution image acquired under conditions where noise due to disturbance occurs, or may be a distribution image acquired under conditions where noise due to disturbance does not occur. The distribution image for learning may be a distribution image that has been pre-processed. The pre-processing may be, for example, a process of removing noise, or a process of generating noise.
[0060] In step S202, the state determination device 100 executes a process of emphasizing the distribution characteristics of the cerebral blood flow in the distribution image for learning. The process of emphasizing the distribution characteristics of the cerebral blood flow is, for example, a process of generating a distribution image showing the difference between the distribution image of the cerebral blood flow obtained in an experiment and the distribution image of the cerebral blood flow obtained under stable conditions. The distribution image of the cerebral blood flow obtained in an experiment is a distribution image of the cerebral blood flow in a state of the driver other than the stable condition. The state of the driver other than the stable condition includes, for example, a state in which the driver is concentrating on driving, a state in which the driver is awake, and a state in which the driver feels anxious.
[0061] In step S203, the neural network executes a process of extracting the cerebral blood flow distribution characteristics from the distribution image in which the cerebral blood flow distribution characteristics are emphasized. For example, the neural network extracts the cerebral blood flow distribution characteristics using a CNN algorithm. In step S204, the neural network executes machine learning for determining the driver's state. For example, the neural network learns to reduce the error between an estimated value estimated from the cerebral blood flow distribution characteristics and a true value given by the training data. The learned neural network is stored in the storage device 30 as a trained model for state determination.
[0062] As described above, the state determination device and the state determination method according to the present embodiment obtain a distribution image including the distribution of cerebral blood flow of at least a part of the driver of a vehicle, input input data including the distribution image to a trained model for state determination, and output the driver's state as output data corresponding to the input data from the trained model for state determination, thereby determining the driver's state, and the trained model for state determination is a trained model that has been trained to determine the driver's state based on teacher data including the distribution image and the driver's state. This makes it possible to determine the driver's state even if there is a part in the driver's brain where brain activity cannot be accurately detected.
[0063] In the state determination device and the state determination method according to the present embodiment, the teacher data includes a distribution image that has been processed to emphasize the distribution characteristics of cerebral blood flow, thereby improving the accuracy of determining the driver's state using the trained model for state determination.
[0064] In the state determination device and the state determination method according to the present embodiment, the trained model for state determination is a trained model trained based on the distribution features of cerebral blood flow extracted from the distribution image included in the teacher data, thereby improving the accuracy of determining the driver's state using the trained model for state determination.
[0065] In the state determination device and the state determination method according to the present embodiment, the teacher data includes a distribution image that has been processed to generate noise. By introducing noise, the trained model for state determination can be prevented from over-learning, thereby improving robustness and increasing the accuracy of determining the driver's state.
[0066] In the state determination device and the state determination method according to the present embodiment, the teacher data includes a distribution image that does not include fluctuations in signals other than the distribution of cerebral blood flow, which makes it easier to extract the characteristics of the distribution of cerebral blood flow and increases the learning speed because there is no disturbance in the distribution image.
[0067] In the state determination device and the state determination method according to the present embodiment, the teacher data includes a distribution image including fluctuations in signals other than the distribution of cerebral blood flow. This allows for the incorporation of noise to suppress overlearning of the trained model for state determination, thereby increasing robustness and improving the accuracy of determining the driver's state.
[0068] In the state determination device and state determination method according to the present embodiment, the teacher data includes an unprocessed distribution image, which is a distribution image that does not include fluctuations in signals other than the distribution of cerebral blood flow, and a distribution image that has been processed to generate noise in the unprocessed distribution image. This allows the introduction of noise to suppress overlearning of the trained model for state determination, thereby increasing robustness and improving the accuracy of determining the driver's state.
[0069] In the state determination device and the state determination method according to the present embodiment, the teacher data includes a distribution image including the distribution of the driver's cerebral blood flow detected under conditions where noise due to disturbance occurs. This allows the introduction of noise to suppress overlearning of the trained model for state determination, thereby increasing robustness and improving the accuracy of determining the driver's state.
[0070] In addition, the state determination device and the state determination method according to the present embodiment input data including a distribution image that has been subjected to a noise removal process to the trained model for state determination, thereby improving the accuracy of determining the driver's state using the trained model for state determination.
[0071] The trained model for state determination according to the present embodiment is a trained model for state determination configured by a neural network including an input layer to which input data including a distribution image including at least a part of the cerebral blood flow distribution of the driver of the vehicle is input, and an output layer to which output data including the state of the driver is output, and the trained model for state determination is configured to function as a computer that is trained to determine the state of the driver using teacher data that associates the input data with the output data, and that outputs output data corresponding to the input data from the output layer when the input data is input to the input layer. This allows the state of the driver to be determined even if there is a part in the driver's brain where brain activity cannot be accurately detected.
[0072] The above-described embodiments are described for the purpose of facilitating understanding of the present invention, and are not described for the purpose of limiting the present invention. Therefore, the elements disclosed in the above embodiments are intended to include all design modifications and equivalents that fall within the technical scope of the present invention. [Explanation of symbols]
[0073] 1. Driving assistance systems 100...Status determination device 10…Processor 110…Acquisition Department 111...Judgment section 112...Control instruction section 113…Learning Department 30…Storage device 31…Trained model for state determination 20…Cerebral blood flow sensor 200…Vehicle controller 300…Output device
Claims
1. A state determination method executed by a state determination device, comprising: The state determination device is Obtaining a distribution image including a distribution of cerebral blood flow of at least a part of a driver of the vehicle; Inputting input data including the distribution image into a trained model for state determination, The state of the driver is determined by outputting the state of the driver from the trained model for state determination as output data corresponding to the input data; A state determination method, wherein the trained model for state determination is a trained model that has been trained to determine the driver's state based on teacher data including the distribution image and the driver's state.
2. The state determination method according to claim 1, A condition determination method in which the training data includes a distribution image that has been processed to emphasize characteristics of the distribution of cerebral blood flow.
3. The state determination method according to claim 1 or 2, A state determination method, in which the trained model for state determination is a trained model trained based on the characteristics of the distribution of cerebral blood flow extracted from the distribution image included in the training data.
4. The state determination method according to claim 1 or 2, A condition determination method, wherein the training data includes the distribution image that has been processed to generate noise.
5. The state determination method according to claim 1 or 2, A condition determination method in which the teacher data includes a distribution image that does not contain signal fluctuations other than the distribution of cerebral blood flow.
6. The state determination method according to claim 1 or 2, A condition determination method in which the teacher data includes a distribution image that includes fluctuations in signals other than the distribution of cerebral blood flow.
7. The state determination method according to claim 1 or 2, A state determination method in which the training data includes an unprocessed distribution image, which is a distribution image that does not contain signal fluctuations other than the distribution of the cerebral blood flow, and the distribution image that has been processed to generate noise in the unprocessed distribution image.
8. The state determination method according to claim 1 or 2, A condition determination method, wherein the teacher data includes a distribution image including a distribution of the driver's cerebral blood flow detected under conditions in which noise due to external disturbance occurs.
9. The state determination method according to claim 1 or 2, The state determination method includes inputting the input data including the distribution image from which noise has been removed into the trained model for state determination.
10. an acquisition unit that acquires a distribution image including a distribution of cerebral blood flow of at least a part of a driver of the vehicle; a determination unit that determines a state of the driver by inputting input data including the distribution image into a trained model for state determination and outputting a state of the driver from the trained model for state determination as output data corresponding to the input data, The state determination device, wherein the trained model for state determination is a trained model that has been trained to determine the state of the driver based on teacher data including the distribution image and the state of the driver.
11. an input layer to which input data including a distribution image including a distribution of cerebral blood flow of at least a part of a driver of a vehicle is input; A trained model for state determination configured by a neural network including an output layer that outputs output data including the driver's state, learning is performed to determine the state of the driver using teacher data in which the input data and the output data are associated with each other, A trained model for state determination that causes a computer to function such that, when the input data is input to the input layer, the output layer outputs the output data corresponding to the input data.
Citation Information
Patent Citations
On-vehicle system, vehicle control device, and program for vehicle control device
JP2016064773A
