Information processing device, information processing system, information processing method and information processing program

Abstract

An information processing device is provided, comprising an eye behavior analysis unit (300) for analyzing the eye behavior of a driver driving a moving object, wherein the eye behavior analysis unit dynamically switches the analysis mode according to the driving mode of the moving object.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and an information processing program. Background Art

[0002] In recent years, the autonomous driving technology for controlling the vehicle by the vehicle control system (information processing system) has been actively developed. However, even in the case where such autonomous driving technology has become widespread, it is expected that there may be a situation where the autonomous driving allowed section and the manual driving allowed section are mixed together, depending on the actual road infrastructure maintenance conditions, the autonomous driving allowed section being a section where the system can independently perform autonomous driving control, and the manual driving allowed section being a section where autonomous driving is not allowed. Not only may there be a situation where the system continuously performs autonomous driving driving completely independently, but there may also be a situation where it is necessary to switch from the autonomous driving mode as described above to the manual driving mode where the driver performs control.

[0003] Furthermore, when switching from the automatic driving mode to the manual driving mode, in order to avoid causing an accident, etc., it is desirable that the system determines the recovery correspondence level of the driver's manual driving mode, and performs the switch only when it is determined that recovery to manual driving is allowed. Thus, for example, it is considered that the system detects the driver's awakening level by analyzing eyeball behavior, and uses a means for determining the recovery correspondence level of the manual driving mode as one of the means for determining the recovery correspondence level. Eyeball behavior is considered to be a behavior that reflects the activity results of the human brain such as cognitive results. Moreover, in recent years, with the advancement of image analysis technology, it has become easier to detect eyeball behavior with high precision by analyzing captured images of the eyeballs. For example, in the technology described in the following patent document 1, the driver's eyeball behavior is observed by an image capture device.

[0004] In addition, the above-mentioned eyeball behavior not only has a common tendency for each person according to the state (e.g., feeling sleepy), but also reflects the activity in the brain. Therefore, the eyeball behavior is also affected by the driver's own innate tendency and the driver's unique experience. That is, the eyeball behavior indicates the behavior that is unique to each person. Therefore, in the case of expecting to accurately determine the driver's awakening level by analyzing the eyeball behavior, the vehicle control system is required to continuously identify and learn the eyeball behavior that is unique to each driver.

[0005] Citation List

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2015-127937

[0008] Patent Document 2: WO 2019 / 188398 A

[0009] Patent Document 3: WO 2019 / 097944 A

[0010] Patent Document 4: WO 2019 / 202881 A

[0011] Patent Document 5: WO 2019 / 082774 A Summary of the invention

[0012] Problems to be solved by the present invention

[0013] On the other hand, eye behaviors such as microsaccades and tremors are high-speed movements. For this reason, in the case where it is desired to detect such eye behaviors with high precision, sampling and analysis at a high frame rate (e.g., 250 frames per second (fps)) is required. Note that microsaccades refer to high-speed jumping movements of the eyeballs that are performed unconsciously when a person is looking at an object and are observed as involuntary subtle eye tremors. However, in the case where the vehicle control system (information processing system) continuously performs such high-speed sampling and analysis, the load of imaging processing, analysis processing, etc. increases. This leads to an increase in power consumption and an increase in device temperature caused by high-speed driving. In addition, in this case, the noise generated by the increase in temperature will also lead to a decrease in detection sensitivity.

[0014] Therefore, the present disclosure proposes an information processing device, an information processing system, an information processing method, and an information processing program that can accurately determine a driver's recovery correspondence level while reducing the load of imaging processing, analysis processing, and the like.

[0015] Solution to the problem

[0016] According to the present disclosure, there is provided an information processing device including: an eye behavior analysis unit that analyzes the eye behavior of a driver driving a mobile body, wherein the eye behavior analysis unit dynamically switches an analysis mode according to a driving mode of the mobile body.

[0017] In addition, according to the present disclosure, there is provided an information processing system including an eye behavior analysis unit that analyzes the eye behavior of a driver driving a mobile body, wherein the eye behavior analysis unit dynamically switches an analysis mode according to a driving mode of the mobile body.

[0018] Furthermore, according to the present disclosure, there is provided an information processing method, comprising: analyzing the eye behavior of a driver driving a mobile body via an eye behavior analysis unit, wherein an analysis mode of the analysis is dynamically switched according to a driving mode of the mobile body.

[0019] Furthermore, according to the present disclosure, there is provided an information processing program that causes a computer to execute: an analysis function of analyzing eye behavior of a driver driving a mobile body, wherein an analysis mode of the analysis function is dynamically switched according to a driving mode of the mobile body. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an explanatory diagram illustrating an example of the autonomous driving level.

[0021] Figure 2 is a flowchart illustrating an example of traveling according to an embodiment of the present disclosure.

[0022] Figure 3 : is an explanatory diagram illustrating an example of transition of the autonomous driving level according to an embodiment of the present disclosure.

[0023] Figure 4 is a flow chart illustrating an example of a monitoring operation according to an embodiment of the present disclosure.

[0024] Figure 5 2 is an explanatory diagram illustrating an example of a detailed configuration of the vehicle control system 100 according to the embodiment of the present disclosure.

[0025] Figure 6 is an explanatory diagram illustrating an example of the installation position of the image capturing device included in the sensor unit 113 according to the embodiment of the present disclosure.

[0026] Figure 7 1 is an explanatory diagram illustrating an example of each sensor included in the sensor unit 113 according to the embodiment of the present disclosure.

[0027] Figure 8 : is an explanatory diagram illustrating an example of a unit that performs determination of a driver's awakening level according to an embodiment of the present disclosure.

[0028] Fig. 9 : is an explanatory diagram illustrating details of an operation example of the eye behavior analysis unit 300 according to an embodiment of the present disclosure.

[0029] Fig.10 is a flowchart of an information processing method according to an embodiment of the present disclosure.

[0030] Fig.11 yes Fig.10 A sub-flowchart of step S104 is shown in the diagram.

[0031] Fig.12 1 is a hardware configuration diagram illustrating an example of a computer 1000 that implements some functions of the vehicle control system 100 . DETAILED DESCRIPTION

[0032] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the accompanying drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and repeated descriptions thereof are omitted.

[0033] Note that the description will be given in the following order.

[0034] 1. Examples of autonomous driving levels

[0035] 2. Driving example

[0036] 3. Examples of transitions in autonomous driving levels

[0037] 4. Monitoring Examples

[0038] 5. Background leading to the embodiments of the present disclosure

[0039] 6. Examples

[0040] 6.1 Detailed Structure of Vehicle Control System 100

[0041] 6.2 Detailed Structure of Sensor Unit 113

[0042] 6.3 Detailed Structure of the Unit for Executing the Determination of the Driver's Awakening Level

[0043] 6.4 Operation Example of Eye Behavior Analysis Unit 300

[0044] 6.5 Information Processing Methods

[0045] 6.6 Summary

[0046] 7. Hardware composition

[0047] 8. Supplement

[0048] <<1. Examples of autonomous driving levels>>

[0049] First, before describing the details of the embodiments of the present disclosure, Figure 1 Describe the level of autonomy of the autonomous driving technology. Figure 1 This is an explanatory diagram illustrating an example of the autonomous driving level. Figure 1 The following description is based on the automated driving levels defined by the Society of Automotive Engineers (SAE). Figure 1 The discussion of the autonomous driving levels shown in the diagram does not thoroughly explore the problems and reliability of the widespread use of autonomous driving technology. Therefore, in the following description, there are places that are not necessarily explained in accordance with the interpretation defined in SAE based on the above problems, etc.

[0050] like Figure 1 As shown in the diagram, the autonomous driving level is divided into five levels, for example, from level 0 to level 4. First, autonomous driving level 0 corresponds to manual driving (direct driving manipulation by the driver) without driving support from a vehicle control system, and the driver performs all driving tasks and also performs monitoring for safe driving (e.g., avoiding dangerous actions).

[0051] Next, automated driving level 1 corresponds to manual driving (direct driving control) in which driving support (automatic braking, adaptive cruise control (ACC), lane keeping assist system (LKAS) etc.) by the vehicle control system can be performed, and the driver performs all driving tasks and also performs monitoring for safe driving.

[0052] Next, autonomous driving level 2 is also called "autonomous driving function under specific conditions", and the vehicle control system performs subtasks of driving tasks related to vehicle control in both the front and rear directions and the left and right directions of the vehicle under specific conditions. For example, in autonomous driving level 2, the vehicle control system cooperatively controls both steering operation and acceleration / deceleration (for example, cooperation between ACC and LKAS). On the other hand, even in autonomous driving level 2, the execution subject of the driving task is basically the driver, and the monitoring subject of safe driving is also the driver.

[0053] In addition, autonomous driving level 3 is also called "conditional autonomous driving", and the vehicle control system can perform all driving tasks within a limited range. In autonomous driving level 3, the vehicle control system is the main body for executing driving tasks, and the main body for monitoring safe driving is basically the vehicle control system.

[0054] On the other hand, in the automated driving level 3 defined by SAE, the types of secondary tasks that the driver can actually perform are not clearly defined (here, "secondary tasks" means operations other than driving operations performed by the driver while driving). Specifically, it is considered that the driver can perform operations or actions other than manipulation while driving at automated driving level 3, for example, secondary tasks such as operating a mobile terminal, teleconferencing, watching videos, playing games, thinking, or talking with other passengers. On the other hand, within the scope of the definition of automated driving level 3 by SAE, the driver is expected to appropriately respond such as driving operations in accordance with requests from the vehicle control system due to system failures, deterioration of the driving environment, etc. Therefore, in automated driving level 3, in order to ensure safe driving, even when performing secondary tasks as described above, it is expected that the driver is always in a state of readiness that the driver can immediately return to manual driving.

[0055] In addition, the autonomous driving level 4 is also called "full autonomous driving under specific conditions", and the vehicle control system performs all driving tasks within a limited range. In the autonomous driving level 4, the execution subject of the driving task is the vehicle control system, and the monitoring subject of safe driving is also the vehicle control system. On the other hand, unlike the autonomous driving level 3 described above, in the autonomous driving level 4, the driver is not expected to respond such as driving operations (manual driving) according to requests from the vehicle control system due to system failures, etc. Therefore, in the autonomous driving level 4, the driver can perform secondary tasks as described above, and can, for example, take a nap according to the situation.

[0056] As described above, in the automated driving level 0 to the automated driving level 2, the vehicle travels in a manual driving mode in which the driver is the subject of execution of all or part of the driving tasks. Thus, in these three automated driving levels, the driver is not allowed to engage in secondary tasks, which are actions other than manual driving and operations related to manual driving, during driving, which may cause the driver's attention to drop and cause the driver to not pay attention to the front.

[0057] On the other hand, in the autonomous driving level 3, the vehicle is driven in the autonomous driving mode in which the vehicle control system is the main body for executing all driving tasks. On the other hand, as described above, in the autonomous driving level 3, there may be a situation where the driver performs driving operations. Therefore, in the autonomous driving level 3, when the driver is allowed to perform secondary tasks, the driver is required to be in a ready state in which the driver can return to manual driving from the secondary tasks.

[0058] Furthermore, in the autonomous driving level 4, the vehicle travels in an autonomous driving mode in which the vehicle control system performs all driving tasks. On the other hand, there may be sections where the autonomous driving level 4 cannot be applied to a part of the driving route due to the maintenance status of the actual road infrastructure, etc. In such sections, for example, since it is assumed that the autonomous driving level is set to level 2 or lower, the driver is required to autonomously perform the driving task. For this reason, even in the autonomous driving level 4, since there is a transition to the autonomous driving level below level 2 as described above, the driver is required to be in a ready state in which the driver can return to manual driving from the secondary task.

[0059] Note that the actual usage range of each autonomous driving level allowed for each autonomous driving level that is different from each other is referred to as an "operational design domain (ODD)".

[0060] <<2. Driving Example>>

[0061] Next, based on the autonomous driving levels described above, refer to Figure 2 An example of traveling according to an embodiment of the present disclosure will be described. Figure 2is a flow chart illustrating an example of driving according to an embodiment of the present disclosure. Figure 2 As shown in the diagram, during the driving according to the embodiment of the present disclosure, the vehicle control system executes, for example, steps from step S11 to step S17. The details of each of these steps are described below.

[0062] First, the vehicle control system performs driver authentication (step S11). Driver authentication can be performed by knowledge authentication using a password, personal identification number, etc., biometric authentication using a face, fingerprint, iris of the pupil, voiceprint, etc., or authentication using both knowledge authentication and biometric authentication. In an embodiment of the present disclosure, by performing such driver authentication before starting driving, even in the case where multiple drivers drive the same vehicle, unique biometric information of each driver, such as the eye behavior of each driver, can be obtained in association with each driver.

[0063] Next, the driver or the like operates the input unit 101 (see FIG. Figure 5 ), set the destination (S12). Note that the embodiment describes an example in which the driver gets on the vehicle and sets the destination. On the other hand, the embodiments of the present disclosure are not limited to this. The vehicle control system can pre-set the destination based on the destination information or calendar information manually input into a smart phone or the like (assuming that it can communicate with the vehicle control system) before the driver gets on the vehicle. Alternatively, the vehicle control system can automatically pre-set the destination by obtaining itinerary information, etc. pre-stored in a smart phone, a cloud server, etc. (assuming that it can communicate with the vehicle control system) via a concierge service. Then, the vehicle control system makes pre-planned settings such as a driving route based on the set destination. In addition, the vehicle control system updates and obtains information such as the road environment of the set driving route that is frequently updated, that is, local dynamic map (LDM) information obtained by high-precision driving map information of the road on which the vehicle is traveling. In addition, the vehicle control system sets the level of autonomous driving suitable for each section on the driving route based on the acquired LDM information, etc.

[0064] Next, the vehicle control system starts to display the driving interval on the driving route. Then, the vehicle control system starts driving according to the set automatic driving level (step S13). Note that when starting driving, the display of the driving interval is updated based on the position information of the vehicle (host vehicle) and the acquired LDM update information.

[0065] Next, the vehicle control system appropriately performs monitoring (observation) of the driver's state (step S14). In an embodiment of the present disclosure, for example, monitoring is performed to obtain training data for determining the driver's recovery corresponding level (e.g., awakening level), or monitoring is performed to switch the driving mode according to the automatic driving level set for each section on the driving route. Note that the details of the monitoring will be described later.

[0066] Next, when the vehicle reaches a switching point from the automatic driving mode to the manual driving mode based on the automatic driving level set for each section on the driving route, the vehicle control system determines whether the driving mode can be switched (step S15). Note that the details of the determination made at this time will be described later. Then, in the case where it is determined that the driving mode can be switched (step S15: Yes), the vehicle control system proceeds to the processing of step S16, and in the case where it is determined that the driving mode cannot be switched (step S15: No), the vehicle control system returns to the processing of step S13.

[0067] Next, the vehicle control system switches the driving mode (step S16). In addition, the vehicle control system determines whether the vehicle (host vehicle) has arrived at the destination (step S17). In the case where the vehicle has arrived at the destination (step S17: Yes), the vehicle control system ends the processing, and in the case where the host vehicle has not arrived at the destination (step S17: No), the vehicle control system returns to the processing of step S13. Thereafter, the vehicle control system appropriately repeats the processing from step S13 to step S17 until the vehicle arrives at the destination.

[0068] <<3. Example of transition of autonomous driving levels>>

[0069] Next, refer to Figure 3 An example of transition of the autonomous driving level according to an embodiment of the present disclosure is described in more detail. Figure 3 : is an explanatory diagram illustrating an example of transition of the autonomous driving level according to an embodiment of the present disclosure.

[0070] like Figure 3 As shown in the diagram, it is assumed that, for example, when a transition is made from the sections of the autonomous driving level 3 and the autonomous driving level 4 on the driving route to the sections of the autonomous driving level 0 and 1 and the autonomous driving level 2, the autonomous driving mode ( Figure 3 in the lower area) to manual driving mode ( Figure 3 ) in the upper area of ​​the

[0071] On the other hand, it is difficult for the driver to consciously maintain a state of readiness in which the driver can return to manual driving while the vehicle is traveling in the automatic driving mode. For example, while the vehicle is traveling in the automatic driving mode, it is believed that the driver will be immersed in secondary tasks such as sleeping (napping), watching TV or videos, playing games, etc. In addition, for example, the driver may be looking in front of or around the vehicle as if driving manually when just taking his hands off the steering wheel, may be reading a book, or may be dozing off. Thus, the driver's awakening level (level of consciousness) varies with the differences between these secondary tasks.

[0072] Furthermore, in the case where the driver falls into a sleep state while the vehicle is traveling in the automatic driving mode, the driver is in a state where the driver's level of consciousness or judgment is reduced, that is, a state where the awakening level is reduced. Therefore, since the driver cannot perform normal manual driving in the state where the awakening level is reduced, if the driving mode is switched to the manual driving mode in this state, an accident may occur in the worst case. Therefore, even in the state where the awakening level is reduced, before switching to the manual driving mode, the driver is required to recover to a high awakening state (internal awakening recovery state) in which the driver can drive the vehicle with normal consciousness. That is, in order to ensure safe driving, it is required that the switch from the automatic driving mode to the manual driving mode is performed only in the state where the driver can be observed to have recovered to the internal awakening state.

[0073] Therefore, in the embodiment of the present disclosure, in order to avoid causing an accident, etc., only when the driver is in the recovery corresponding level of the manual driving mode, that is, when an active response indicating the recovery of internal awakening (a state in which the driver recovers to the internal awakening state) can be observed (such as Figure 3 In addition, in the embodiment of the present disclosure, as shown in the center of Figure 3 As shown in the diagram, in the case where an active response indicating internal awakening recovery cannot be observed, the driving mode is switched to an emergency evacuation mode such as a minimum risk strategy (MRM). Note that in the emergency evacuation mode, processing such as deceleration, stopping, or parking on the road, roadside, or evacuation space is performed. In addition, in Figure 3 In the transition from the autonomous driving level 4 to the autonomous driving level 3, the driving mode is not switched. Therefore, the observation of the active response indicating the internal awakening recovery as described above is not performed. On the other hand, the present embodiment is not limited to Figure 3 In the case of a transition from automated driving level 4 to automated driving level 3, a transition based on the observations or observation results as described above may be made.

[0074] Specifically, in the case where an active response indicating internal awakening recovery is not observed when a transition is made from the autonomous driving level 4 to the autonomous driving level 3, even if the driver is obliged to return to manual driving according to the legal system, it is not always possible for the driver to be in a state where the intervention request (RTI) as a request to return to the autonomous driving level 3 from the vehicle control system can be appropriately responded to. More specifically, in response to the RTI as a request to return to the autonomous driving level 3, it is not always possible for the driver to be in a state where the brain is restored to the awakening state and can be restored to a physical state that allows manual driving without numbness of the body, etc. In this case, when the transition is made from the autonomous driving level 4 to the autonomous driving level 3, there may be a condition in the vehicle control system that exceeds the design concept previously envisioned, and as a result, there may be a concern that an accident or the like may be caused. Therefore, in the following present embodiment, in order to reduce the concern as described above, even at a stage where the vehicle control system does not need to issue an RTI as a return request to the driver, in order to check the return correspondence level (e.g., awakening level) of the driver, a virtual RTI as a preventive return request can be appropriately performed, and an active response indicating internal awakening recovery of the driver can be observed.

[0075] Notice, Figure 3 The arrows indicating the transition of the automatic driving level shown in the diagram indicate the direction of the transition in which the automatic switching is allowed. In addition, it is noted that the transition in the opposite direction of the arrows is not recommended because the transition may cause the driver to misunderstand the state of the vehicle control system. That is, in the vehicle control system according to the embodiment of the present disclosure, once an automatic transition of the automatic driving level such as switching from the automatic driving mode to the manual driving mode with driver intervention is performed, it is desirable that the vehicle control system is designed not to automatically return to the automatic driving mode again without the intentional instruction of the driver. As described above, providing the directionality (irreversibility) of the switching of the driving mode means that the vehicle control system is designed to prevent the driving mode from being switched to the automatic driving mode without the explicit intention of the driver. Thus, according to the vehicle control system, the automatic driving mode can be enabled only when the driver has a clear intention. For example, in the case where the driving mode is not the automatic driving mode, it is possible to prevent the driver from misunderstanding that the driving mode is the automatic driving mode and taking actions such as easily starting a secondary task.

[0076] As described above, in the embodiment of the present disclosure, in order to ensure safe driving, the switch from the automatic driving mode to the manual driving mode is performed only when the driver is observed to be in an internal recovery state.

[0077] <<4. Monitoring Examples>>

[0078] So, reference Figure 4An example of monitoring (observation) of switching from the automatic driving mode to the manual driving mode will be described. Figure 4 FIG. 1 is a flow chart illustrating an example of a monitoring operation according to an embodiment of the present disclosure. Figure 4 As shown in the diagram, in the embodiment of the present disclosure, for example, when switching from the automatic driving mode to the manual driving mode, the vehicle control system performs steps from step S21 to step S27. The details of each of these steps are described below.

[0079] First, assume that the vehicle is traveling in autonomous driving mode and the driver is completely disengaged from driving control. Also, assume that the driver is performing a secondary task such as taking a nap, watching a video, concentrating on playing a game, or working with a visual tool such as a tablet or smartphone. On the other hand, assume that the work is performed using a visual tool such as a tablet or smartphone, for example, with the driver's seat tilted back or in a seat different from the driver's seat.

[0080] First, the vehicle control system appropriately intermittently performs passive monitoring and / or active monitoring of the driver (step S21). Here, active monitoring and passive monitoring will be explained.

[0081] First of all, active monitoring is a method of observing the driver's intelligence, such as perception, cognition, judgment and action ability, in which the vehicle control system inputs active information to the driver and observes the driver's conscious response to the active information. For example, examples of active information input to the driver include visual information, auditory information, tactile information, olfactory information and (taste) information. These active information cause the driver's perception and cognitive actions. In the case where the active information is information that affects the risk, the driver will make (response) judgments or actions according to the risk. Thus, by observing such responses of the driver, the perception state, cognitive state, judgment state and action state of the driver's brain can be determined. Specifically, in active monitoring, for example, when the vehicle control system performs steering control with a steering amount that does not affect the safe driving of the vehicle and corresponds to pseudo-active information in order to promote the driver's feedback, as a conscious response, (when the driver is normally awake) the driver's action of restoring the steering with an appropriate steering amount is expected (specifically, the driver perceives and recognizes the unnecessary steering amount, makes a judgment to restore the steering relative to the unnecessary steering amount, and takes a series of actions, thereby causing an action corresponding to the above response. Thus, by observing the above response, the perception state, cognitive state, judgment state and action state of the driver's brain can be determined. Note that during the period when the vehicle is traveling in manual driving mode, since the driver continuously performs a series of operations of perception, cognition, judgment and action on the road environment, etc. in order to perform manipulation, the vehicle control system can observe the driver's conscious response (that is, it can be said that there is a response to the driving manipulation) without directly inputting active information to the driver. In addition, when the above active information is repeatedly input, this information is intelligently filtered in the driver's brain and is regarded as unnecessary (that is, "habituated"). Therefore, in the embodiment of the present disclosure, the frequency of active information input is preferably set to an appropriate frequency in order to avoid the above-mentioned "habituation".

[0082] Next, in the case where the driver's conscious response to direct active information input cannot be observed (actively monitored), passive monitoring is performed. There are various observation methods for passive monitoring of the driver's state, for example, observation of the driver's biometric information can be used. More specifically, for example, in passive monitoring, when the driver is sitting in the driver's seat and in a posture that allows driving, detailed observation evaluation of indicators related to the ratio of open eyes (PERCLOS), head posture behavior, eyeball behavior (saccades (rapid eye movements), gaze, microsaccades, etc.), blinking, facial expressions, facial directions, etc. is expected. In addition, in postures other than sitting, by using wearable devices, etc., extended observations for observing heart rate, pulse rate, blood flow, breathing, brain waves, sweating status, sleepiness level estimated from heart rate and breathing, etc. can be performed. In addition, in passive monitoring, the driver's sitting or leaving the driver's seat, the driver's movement, the moving destination, the posture, etc. can be observed. In addition, the steering amount associated with the driver's attention driving state (the state of manual driving while keeping proper attention to driving) can be directly observed. In addition, the information observed through passive monitoring can be used to estimate the time required for the driver to return to manual driving when the driving control system issues a driving mode switching notification, alarm, etc. during the vehicle driving in the automatic driving mode. In addition, the information observed through passive monitoring can be used to determine whether to switch to the emergency evacuation mode when the driver does not want to return to manual driving within a predetermined time.

[0083] Next, we will return Figure 4 The explanation continues. The vehicle control system notifies the driver of the RTI as a resumption request for manual driving (step S22). At this time, for example, the driver is notified of the RTI as a resumption request for manual driving using a dynamic tactile sense such as vibration, or in a visual or audible manner. Then, in response to such notification of the RTI as a resumption request, the driver returns to the driver's seat in a normal state of wakefulness, and recovers to a high state of wakefulness in which the driver is able to drive the vehicle under normal consciousness. Note that the RTI as a resumption request can be performed multiple times in stages. In this case, the RTI as a resumption request can be performed by different means at each stage, and can be dynamically changed, for example, according to the state of the driver.

[0084] Next, the vehicle control system monitors the driver's sitting state, sitting posture, etc. (step S23). In addition, the vehicle control system centrally performs active monitoring of the driver who is properly seated in the driver's seat (step S24). For example, in order to prompt the driver to return to a high state of arousal in which the driver can drive the vehicle under normal consciousness, the active monitoring includes active information input, such as alarming the driver, inputting pseudo-noise steering to the manual steering control of the vehicle in a pseudo manner, and the like.

[0085] Next, the vehicle control system intensively monitors the driver's facial and eye behaviors, such as saccades (intensive monitoring of eye behaviors) (step S25).

[0086] Here, the monitoring of eye behavior will be explained. Although information related to the driver's state can be observed by various means, it is difficult to directly observe the cognition, judgment and action in the driver's brain. For example, when using functional magnetic resonance imaging (fMRI), electroencephalogram, etc., it is necessary to restrain the object being measured (driver). For this reason, in the embodiment of the present disclosure, this method is not suitable as a means of observing the driver's state. Therefore, in the embodiment of the present disclosure, as one of the various biometric information observation means, a means of observing the driver's eye behavior is used. For example, when the driver is sitting in the driver's seat, eye behavior observation can be performed without specially restraining the driver. That is, it can be said that eye behavior observation is a non-invasive and non-wearable observation means.

[0087] In addition, eyeball behavior is a part of the behavior expressed by biological reflexes, and occurs in the case where a reaction occurs by gazing through a cyclic adjustment adaptive response in which the response to the event change does not contain a thinking element, and a behavior that is not a reflexive response but is used for feature tracking to identify the features of the visual object and promote the understanding of the features, the reaction includes smooth pursuit (sliding tracking eyeball movement) of relatively chasing visual moving objects, slow convergence due to the vehicle approaching the front background and high-speed recovery dispersion movement for solving the convergence, sliding tracking eyeball movement of tracking objects in the target direction regardless of the rotation of the driver's body and head, etc., and the behavior includes unique microsaccades, etc. In these behaviors, many phenomena that appear by reflecting the neural transmission and processing in the brain are also seen at the same time. That is, it is believed that the activity results such as the recognition of the gaze object obtained by referring to the memory of the brain are reflected. Therefore, by using the fact that the cognitive function activity in the brain is reflected in the eyeball behavior, the driver's awakening level can be estimated with high accuracy based on the analysis of the eyeball behavior. That is, by performing eye behavior observation, when switching from the automatic driving mode to the manual driving mode is performed (in detail, immediately before the switching), it is possible to indirectly observe whether the driver has recovered to a high level of awakening (recovery corresponding level) at which the driver can drive the vehicle under normal consciousness. In particular, in the case where the driver once leaves the driving control work and resumes the driving control work after a period of time, it is considered that the driver does not have sufficient memory related to the surrounding environment and the vehicle state required to resume manual driving. Thus, for example, the driver tries to quickly perform actions for grasping information that the driver would grasp if the driver were to perform continuous manual driving, such as visually checking the conditions ahead of the road, or visually checking the cause of an RTI from the vehicle control system as a request for resumption of manual driving. Such actions for grasping information are reflected in the driver's eye behavior.

[0088] In addition, as described above, eyeball behavior indicates specific behavior for each person and further for each state of the person. Therefore, in the case of expecting to accurately determine the driver's awakening level by analyzing eyeball behavior, it is necessary to constantly identify and learn the eyeball behavior unique to each driver, and determine the driver's awakening level based on such learning. In addition, when the driver returns to manual driving, what the driver focuses on checking and what priority the driver checks in are greatly affected by the memory based on the driver's past risk experience, etc., and therefore, will change due to various factors such as road conditions and driving speed during driving. Thus, eyeball behavior not only shows behavior unique to each person, but also changes due to the influence of memory based on various experiences of the driver. In this embodiment, instead of using a unified judgment for the driver to perform awakening level judgment, the recovery ability of the same driver is determined based on the learning obtained by intermittent learning in the active observation interval. Thus, the recovery ability judgment can be more appropriately performed for each driver.

[0089] Then, the vehicle control system determines the driver's recovery response level by determining the driver's awakening level based on the monitoring in the above-mentioned step S25 (step S26). Then, the vehicle control system determines whether the driver is at a recovery reaction level at which the driver can cope with returning to manual driving. The vehicle control system according to the present embodiment observes the driver's recovery process in stages and observes the driver's response in each stage of the recovery process. Thus, a composite determination can be made. Then, based on the driver's internal awakening recovery and the inspection of whether the driver can perform manual driving actions, the vehicle control system performs a switch from the automatic driving mode to the manual driving mode when it is determined with a predetermined accuracy that recovery to manual driving is allowed (step S27).

[0090] Notice, Figure 4 The steps in the above description do not necessarily have to be performed in the order described, and may be performed in an appropriately modified order, or may be performed partially in parallel. For example, the active monitoring in step S24 and the concentrated monitoring of eyeball behavior in step S25 may be performed in parallel, or may be performed in the same order as described above. Figure 4 The order shown in the diagram is different.

[0091] <<5. Background leading to the embodiments of the present disclosure>>

[0092] In addition, before describing the details of the present invention, the background that led the inventor to produce the embodiments of the present invention will be described.

[0093] Meanwhile, during the period when the driver returns to the driver's seat from a nap state or an absent state and actually holds the steering wheel to mainly perform manual driving, there are multiple recovery processes and stages, and ideally, the driver's recovery correspondence level (awakening level) increases in each stage. Thus, in the vehicle control system, even in a situation where the driver's recovery correspondence level does not increase ideally, a check means for checking the driver's various internal awakening recoveries and the driver's ability to perform manual driving actions can be used according to the recovery process and stage. For example, in the case where the driver recovers from a nap in a nap place in the car or the driver recovers from a nap after leaving the driver's seat, by detecting a change in the posture for returning to the driver's seat from a nap posture, standing of the feet, movement of the seat, etc., as a body movement used as a response indicating recovery from a sleep state, the driver's internal awakening recovery and the driver's ability to perform manual driving actions can be checked. Furthermore, in the case where the driver returns from a working state such as a state where the driver leaves the driver's seat for a TV conference, etc., by detecting the driver's seating in the driver's seat, the movement action of returning to the driver's seat in response to the check operation of the RTI as a recovery request from the vehicle control system in the state where the driver leaves the driver's seat, etc., the driver's internal awakening recovery and the driver's ability to perform manual driving actions can be checked. Furthermore, in the case where the driver arrives near the driver's seat, the driver's internal awakening recovery and the driver's ability to perform manual driving actions can be checked by the driver's check operation of detecting the road ahead, the check operation of pointing and calling the road ahead when necessary, the change of the driving posture in the driver's seat and the movement of the feet caused by the change of the driving posture, the orientation of the head or face caused by the road ahead check operation, etc. Furthermore, by detecting the driver's visual check of the RTI as a recovery request from the vehicle control system, the line of sight movement (eyeball behavior) for checking the message content involved in the RTI as a recovery request or the time of full recovery completion, etc., the driver's internal awakening recovery and the driver's ability to perform manual driving actions can be checked. In addition, by detecting the driver's line of sight movement (eyeball behavior) used to check the situation around the vehicle when starting manual driving, the operation of the automatic driving release switch, the step-by-step input of torque to the steering wheel, the pressing action of the brake pedal, the pressing action of the accelerator pedal, the acceleration command and the deceleration command, the correction operation of the virtual noise related to the steering angle input from the vehicle control system, etc., the driver's internal awakening recovery and the driver's ability of manual driving action can be checked. That is, the vehicle control system is required to observe the recovery correspondence level of the driver's recovery process step by step and in a complex manner. Then, at the final stage of this step-by-step and complex observation, as a means of estimating the cognitive level, judgment level, and action level in the driver's brain, sampling and analysis of eyeball behavior are performed at a high frame rate.

[0094] Specifically, in the stage before manual driving, as a means for accurately determining the driver's awakening level, it is necessary to observe eye behavior that reflects the activity in the brain, especially microsaccades. Therefore, the following description will focus on eye behavior. As mentioned above, eye behavior indicates behavior that is unique to each person. Therefore, in the vehicle control system, in order to accurately determine the driver's awakening level using analysis of eye behavior, etc., it is necessary not only to analyze the eye behavior in detail, but also to continuously identify and learn the unique eye behavior of each driver, and determine the driver's awakening level based on such learning.

[0095] Therefore, in order to detect the difference in the dominance of the eyeball behavior of each driver, it is necessary to sample and analyze the eyeball behavior as a high-speed movement such as microsaccades at a high frame rate through an image capture device or the like. However, when such high-speed sampling and analysis are continuously performed, the load of imaging processing, analysis processing, etc. increases. This leads to an increase in power consumption and an increase in the temperature of the device. In addition, when the amount of signal that can be obtained by performing high-speed sampling is reduced, the noise generated due to the increase in temperature increases. As a result, the detection sensitivity decreases.

[0096] Specifically, in order to detect the difference in the dominance of each driver's eyeball behavior, it is necessary to sample and analyze the high-speed movement of the driver's eyeball at a high frame rate of, for example, 250fps or higher, preferably 1000fps. Therefore, not only the increase in power consumption and the transmission of large-capacity data are required, but also the signal needs to be read at high speed. Thereby, the integration time of the image signal will be shortened, and the amount of signal obtained by integration will be reduced. In addition, when sampling and analyzing are performed regularly at a high frame rate, the temperature of the device rises due to such operation, and as a result, there are side effects such as an increase in noise caused by the temperature rise. For this reason, it cannot be said that it is best to always sample and analyze the eyeball behavior at a high frame rate.

[0097] In addition, due to the increase in noise caused by the temperature increase, the detection sensitivity decreases. In order to compensate for the decrease in detection sensitivity, it is considered to increase the luminous brightness of the light source. In this case, the light source with increased luminous brightness always illuminates the driver's eyes. However, for example, in the case where the infrared light source with increased luminous brightness continuously illuminates the driver's eyes, the driver's irradiation dose increases, which is undesirable from the perspective of the driver's eye health.

[0098] Therefore, in view of such a situation, the inventors have produced an embodiment of the present disclosure that dynamically changes the operation of observing and analyzing eye behavior, such as dividing the driver's eye behavior into time periods for sampling and analyzing eye behavior at a high frame rate and other time periods. According to an embodiment of the present disclosure, the driver's awakening level (recovery corresponding level) can be accurately determined while reducing the load of imaging processing, analysis processing, etc. Thus, according to an embodiment of the present disclosure, the load of imaging processing, analysis processing, etc. can be reduced, thereby avoiding an increase in power consumption and an increase in device temperature. In addition, a temperature increase can be avoided, so the detection sensitivity will not be reduced.

[0099] Specifically, during driving in the automatic driving mode, the driver does not perform manual driving. Therefore, it includes a period of time in which only passive monitoring is appropriately performed, and a period in which the eye behavior is observed and analyzed in order to determine the driver's awakening level in the event of a switch from the automatic driving mode to the manual driving mode. That is, during driving in the automatic driving mode, in the event of a switching event mainly occurring, sampling and analysis of eye behavior can be performed at a high frame rate, otherwise, such sampling and analysis does not need to be performed. Therefore, in an embodiment of the present disclosure, the operation of sampling and analyzing eye behavior is dynamically changed, so that in the event of a switch from automatic driving to manual driving, sampling and analysis of eye behavior is performed at a high frame rate, otherwise, sampling and analysis of eye behavior is performed at a low frame rate. As described above, in the present embodiment, sampling and analysis of eye behavior is not always performed at a high frame rate. Thereby, the driver's awakening level can be accurately determined while reducing the load of imaging processing, analysis processing, etc.

[0100] More specifically, in the period when the switching event occurs (in detail, the switching preparation period), the eye behavior is sampled and analyzed at a high frame rate of, for example, 250fps or higher, preferably 1000fps. Thus, the driver's wakefulness level is accurately determined. On the other hand, in other periods, the eye behavior is sampled and analyzed at a normal frame rate such as 60fps. Thus, the driver's fatigue and sleepiness (passive monitoring) are identified. In addition, at this time, sleepiness evaluation values ​​such as the eye closure ratio can be obtained from the degree of blinking or opening the eyes. In addition, in the present embodiment, the vehicle control system not only obtains the eye behavior, but also obtains various information from the captured image of the driver's face or head at a normal frame rate, such as the information obtained through the above-mentioned passive monitoring, the information obtained through expression analysis, and the careless driving state, and uses such information to determine the driver's wakefulness level (recovery corresponding level).

[0101] In addition, when sampling (imaging) the eyeball behavior at a high frame rate, the time period in which the signal can be integrated in each frame (each imaging period) becomes shorter. Thus, in order to integrate the signal amount required for accurate analysis, during the imaging period, a light source with high luminous brightness illuminates the driver's eyes. In the present embodiment, during manual driving, when the driver's eyes are illuminated with visible light, the driver's sight is obstructed. Thus, infrared light that does not affect the visibility of the retina is used for illumination. However, in the case where the infrared light source with increased luminous brightness continuously illuminates the driver's eyes, even if the infrared light is invisible to the driver's eyes, the infrared light will also burden the retina. Thus, from the perspective of the driver's eye health, excessive irradiation is not desirable. Therefore, according to the present embodiment, sampling of the eyeball behavior is performed at a high frame rate within a limited period of time. Thus, the irradiation period of the light source with increased luminous brightness is limited, so that the burden on the driver's retina can be reduced.

[0102] Note that in the examples described in this specification, the case of using an image capture device that samples at a predetermined frame rate is described. On the other hand, the present embodiment is not limited thereto. For example, in the present embodiment, in the case of identifying a change in a continuously acquired captured image, an event-driven imaging device that generates a signal can be used for observation. In this case, for example, the signal generated by the event-driven imaging device due to event detection can be used to control the light emission timing of a light source that illuminates the driver's face, eyeballs, etc., or the light emission timing and imaging timing can be synchronized. For example, by using the signal of the event-driven imaging device as a trigger, sampling and analysis at a high frame rate within a predetermined period can be started, or irradiation of light with increased luminous brightness within a predetermined period can be started. In addition, when a change is detected, the quantization difference amount can be controlled together. That is, by adjusting the light source and the quantization control parameters to appropriately perform observations of eyeball saccades and microsaccades, and using the event-driven imaging device, effective control can be performed, such as dividing the driver's eyeball behavior into a period for analyzing eyeball behavior at a high sampling cycle and other periods.

[0103] In addition, once the driver's awakening level is reduced, the driver's attention to the surrounding environment is reduced, and the attention to the surrounding environment will be reduced. As a result, the inertness to the surrounding stimulation is aggravated, so that the eye behaviors such as microsaccades and gaze tremors are reduced. Therefore, in the present embodiment, even in the period other than the period of the event of switching the driving mode, once the driver's awakening level is identified by passive monitoring, the eye behavior can be sampled and analyzed at a high frame rate according to such a state. The eye behavior observed at this time can be used as training data associated with each driver and each driver's awakening level for learning. Then, in the present embodiment, during the period of the switching event, the driver's awakening level (recovery reaction level) is determined with reference to the eye behavior unique to each driver obtained based on this learning. Thus, the accuracy of the determination can be improved. Even in this case, in the present embodiment, the sampling and analysis of the eye behavior is not always performed at a high frame rate. Thus, the driver's awakening level (recovery reaction level) can be accurately determined while reducing the load of imaging processing, analysis processing, etc. Below, the details of the embodiments of the present disclosure generated by the present invention will be described in sequence.

[0104] <<6. Example>>

[0105] <6.1 Detailed Configuration of Vehicle Control System 100>

[0106] First, refer to Figure 5 and Figure 6 A detailed configuration of a vehicle control system (information processing system) 100 according to an embodiment of the present disclosure will be described. Figure 5 FIG. 1 is an explanatory diagram illustrating an example of a detailed configuration of the vehicle control system 100 according to the present embodiment. Figure 6 In the present invention, when distinguishing between a vehicle provided with the vehicle control system 100 and other vehicles, the vehicle is hereinafter referred to as an own vehicle or a self-vehicle.

[0107] like Figure 5As shown in the diagram, the vehicle control system 100 mainly includes an input unit 101, a data acquisition unit 102, a communication unit 103, an in-vehicle device 104, an output control unit 105, an output unit 106, a drive system control unit 107, a drive system 108, a body system control unit 109, a body system 110, a storage unit 111, an automatic driving control unit 112, and a sensor unit 113. The input unit 101, the data acquisition unit 102, the communication unit 103, the output control unit 105, the drive system control unit 107, the body system control unit 109, the storage unit 111, and the automatic driving control unit 112 are connected to each other via a communication network 121. The communication network 121 includes, for example, an in-vehicle communication network, a bus, etc. that conforms to a certain standard, such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), or FlexRay (registered trademark). Note that the various units of the vehicle control system 100 can be directly connected to each other without the communication network 121.

[0108] Note that in the following description, in the case where the various units of the vehicle control system 100 communicate via the communication network 121, the description of the communication network 121 will be omitted. For example, in the case where the input unit 101 and the automatic driving control unit 112 communicate with each other via the communication network 121, it is simply described that the input unit 101 and the automatic driving control unit 112 communicate with each other.

[0109] Hereinafter, details of respective functional units included in the vehicle control system 100 according to the present embodiment will be described in sequence.

[0110] The input unit 101 includes devices used when an occupant such as a driver inputs various data, instructions, etc. For example, the input unit 101 includes an operation device such as a touch panel, a button, a microphone, a switch, or a joystick, an operation device that allows input by a method other than manual operation such as voice or gesture, etc. In addition, for example, the input unit 101 may be a remote control device using infrared or other radio waves, or an external connection device such as a mobile device or a wearable device that responds to the operation of the vehicle control system 100. Thus, the input unit 101 can generate an input signal based on the data, instructions, etc. input by the occupant, and provide the input signal to each functional unit of the vehicle control system 100.

[0111] The data acquisition unit 102 may acquire data used for processing of the vehicle control system 100 from the sensor unit 113 including various sensors and the like, and provide the data to each functional unit of the vehicle control system 100 .

[0112] For example, the sensor unit 113 includes various sensors for detecting the condition of the vehicle (host vehicle), etc. Specifically, for example, the sensor unit 113 includes a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU), and sensors for detecting the operation amount of the accelerator pedal, the operation amount of the brake pedal, the steering angle of the steering wheel, the engine speed, the motor speed, the wheel speed, etc.

[0113] In addition, for example, the sensor unit 113 may include various sensors for detecting information related to the exterior of the vehicle (host vehicle). Specifically, for example, the sensor unit 113 may include image capture devices such as a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, or other cameras. In addition, for example, the sensor unit 113 may include an environmental sensor for detecting weather, climate conditions, etc., a surrounding information detection sensor for detecting objects around the host vehicle, etc. Examples of environmental sensors include raindrop sensors, fog sensors, sunshine sensors, and snow sensors. In addition, examples of surrounding information detection sensors include ultrasonic sensors, radars, LiDAR (light detection and ranging, laser imaging detection and ranging), sonars, etc.

[0114] In addition, for example, the sensor unit 113 may include various sensors for detecting the current position of the vehicle (host vehicle). Specifically, for example, the sensor unit 113 may include a GNSS receiver that receives a GNSS signal from a GNSS satellite.

[0115] In addition, for example, the sensor unit 113 may include various sensors for detecting information related to the interior of the vehicle. Specifically, for example, the sensor unit 113 may include an image capture device that captures an image of the driver, a biometric information sensor that detects the driver's biometric information, a microphone that collects sounds inside the vehicle, etc. The biometric information sensor is, for example, arranged on a seat surface, a steering wheel, etc., and can detect the biometric information of an occupant sitting on the seat or a driver holding the steering wheel. Examples of the driver's biometric information include heart rate, pulse rate, blood flow, respiration, brain waves, skin temperature, skin resistance, sweating state, head posture behavior, and eye behavior (gaze, blink, saccade, microsaccade, fixation, drift, gaze, pupil reaction of the iris, etc.). These biometric information can be detected by using electric potentials between predetermined positions on the body surface of a driver or the like, alone or in combination, signals obtained by contact observation such as blood flow using infrared light, non-contact microwaves or millimeter waves, signals obtained by non-contact observation using frequency modulated (FM) waves, detection of eye behavior using a captured image of the eye obtained by an image capturing device (monitoring unit) using infrared wavelengths, or overload torque measurement information of a steering or pedal steering device for checking steering response.

[0116] The communication unit 103 communicates with the in-vehicle device 104, various devices outside the vehicle, servers, base stations, etc., and can send data supplied from various functional units of the vehicle control system 100, and provide received data to various functional units of the vehicle control system 100. Note that in the embodiments of the present disclosure, the communication protocol supported by the communication unit 103 is not particularly limited, and the communication unit 103 can support multiple types of communication protocols.

[0117] For example, the communication unit 103 may perform wireless communication with the in-vehicle device 104 through wireless LAN, Bluetooth (registered trademark), near field communication (NFC), wireless universal serial bus (WUSB), etc. In addition, for example, the communication unit 103 may perform wired communication with the in-vehicle device 104 through USB, High-Definition Multimedia Interface (HDMI) (registered trademark), Mobile High-Definition Link (MHL), etc. via a connection terminal (not shown) (including a cable if necessary).

[0118] In addition, for example, the communication unit 103 can communicate with a device (e.g., an application server or a control server) existing on an external network (e.g., the Internet, a cloud network, or a specific network provided by a company) via a base station or an access point. In addition, for example, the communication unit 103 can communicate with a terminal (e.g., a terminal of a pedestrian or a store, or a machine type communication (MTC) terminal) existing near the vehicle by using a peer-to-peer (P2P) technology. In addition, for example, the communication unit 103 can perform V2X communication such as vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-residential communication, or vehicle-to-pedestrian communication. In addition, for example, the communication unit 103 may include a beacon receiving unit that receives radio waves or electromagnetic waves emitted from a wireless station or the like set on a road, thereby acquiring information such as the current position, traffic congestion, traffic control, and required time. Note that the communication unit 103 can be used in a supplementary manner by pairing with a leading vehicle that is traveling in the section and is a leading vehicle, acquiring information acquired from a data acquisition unit set in the leading vehicle as pre-driving information, and supplementing the data acquired by the data acquisition unit 102 of the vehicle. In particular, it is noted that the communication unit 103 may be a means for further ensuring the safety of a following vehicle following a leading vehicle.

[0119] The in-vehicle device 104 may include, for example, a mobile device or wearable device owned by an occupant, an information device brought into or attached to the vehicle, a navigation device for searching for a route to a certain destination, etc. Note that, considering that with the popularization of autonomous driving, occupants are not necessarily fixed in a fixed seat position, the in-vehicle device 104 may be expanded to a video player, a game console, or other devices that are set in the vehicle and can be removed from the vehicle for use.

[0120] The output control unit 105 can control the output of various information to the occupants of the vehicle or the outside of the vehicle. For example, the output control unit 105 controls the output of visual information and auditory information from the output unit 106 by generating an output signal including at least one of visual information (e.g., image data) or auditory information (e.g., audio data) and providing the output signal to the output unit 106. Specifically, for example, the output control unit 105 generates a bird's-eye view image, a panoramic image, etc. by synthesizing image data captured by different image capture devices included in the sensor unit 113, and provides the output signal including the generated image to the output unit 106. Note that in the case of generating such a bird's-eye view image, a panoramic image, etc., by recording and storing the image before the synthesis process using the compound eye in a form of permitted use, the event can be reproduced more accurately. In addition, the recording and storage of the image before the synthesis process depends on whether to store information and the transmission load. In addition, for example, the output control unit 105 generates sound data including an alarm sound, an alarm message, etc. about dangers such as collision, contact, or entering a dangerous zone, and provides the output signal including the generated sound data to the output unit 106.

[0121] The output unit 106 may include a device capable of outputting visual information or auditory information to the occupants of the vehicle or the outside of the vehicle. For example, the output unit 106 includes a display device, a dashboard, an audio speaker, a headset, a wearable device such as a glasses-type display worn by the occupants, a projector, a lamp, etc. In addition to the device including the ordinary display, the display device included in the output unit 106 may be a device that displays visual information in the driver's field of view, such as a head-up display, a transmissive display, or a device with an augmented reality (AR) display function. Note that the output unit 106 may include various devices that provide olfactory stimulation (providing a predetermined smell) or tactile stimulation (providing cold air, providing vibration, providing electrical stimulation, etc.) to the driver in order to prompt the driver to wake up when the driver is deeply out of the driving control work due to sleep, etc. In addition, the output unit 106 may include a device that provides a somatosensory uncomfortable stimulation, such as a device that forces the driver to be in an uncomfortable posture by moving the backrest of the driver's seat.

[0122] The drive system control unit 107 can control the drive system 108 by generating various control signals and providing the control signals to the drive system 108. In addition, the drive system control unit 107 can provide control signals to various functional units other than the drive system 108 when necessary, and notify the control state of the drive system 108, etc.

[0123] The drive system 108 may include various devices related to the drive system of the vehicle. For example, the drive system 108 includes a drive force generating device such as an internal combustion engine or a drive motor for generating a drive force, a drive force transmitting mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting a steering angle, a brake device for generating a brake force, an anti-lock brake system (ABS), an electronic stability control (ESC), an electric power steering device, etc.

[0124] The body system control unit 109 can control the body system 110 by generating various control signals and providing the control signals to the body system 110. In addition, the body system control unit 109 provides control signals to various functional units other than the body system 110 when necessary, and notifies the control state of the body system 110, etc.

[0125] The body system 110 may include various devices of the body system arranged on the body, for example, the body system 110 includes a keyless entry system, a smart key system, an electric window device, an electric seat, a steering wheel, an air conditioner, various lights (such as headlights, taillights, brake lights, turn signals, fog lights, etc.), and the like.

[0126] The storage unit 111 may include, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. In addition, the storage unit 111 may store various programs, data, etc. used by various functional units of the vehicle control system 100. For example, the storage unit 111 stores map data, such as a three-dimensional high-precision map such as a dynamic map, a global map with lower precision than a high-precision map but covering a larger area, and a local map containing surrounding information of the vehicle.

[0127] The automatic driving control unit 112 can perform control related to automatic driving such as autonomous driving or driving support. Specifically, for example, the automatic driving control unit 112 performs collaborative control of functions of an advanced driver assistance system (ADAS) including collision avoidance or impact reduction of the vehicle, follow-up driving based on the distance between vehicles, vehicle constant speed driving, collision warning of the vehicle, lane departure warning of the vehicle, etc. In addition, for example, the automatic driving control unit 112 can perform collaborative control of automatic driving, etc., in which the vehicle is driven autonomously without relying on the operation of the driver. Specifically, the automatic driving control unit 112 includes a detection unit 131, a self-position estimation unit 132, a situation analysis unit 133, a planning unit 134, and an operation control unit 135.

[0128] The detection unit 131 can detect various information required for controlling the automatic driving. The detection unit 131 includes an external vehicle information detection unit 141, an internal vehicle information detection unit 142, and a vehicle state detection unit 143.

[0129] The vehicle exterior information detection unit 141 can perform detection processing of information related to the exterior of the vehicle based on data or signals from various units of the vehicle control system 100. For example, the vehicle exterior information detection unit 141 performs detection processing, identification processing, and tracking processing of objects around the vehicle, as well as detection processing of the distance to the objects. Examples of objects to be detected include vehicles, people, obstacles, buildings, roads, traffic lights, traffic signs, road signs, etc.

[0130] In addition, for example, the vehicle exterior information detection unit 141 performs detection processing of the surrounding environment of the vehicle. The surrounding environment as the detection object includes, for example, weather, temperature, humidity, brightness, road conditions, etc. For example, the vehicle exterior information detection unit 141 provides data indicating the result of the detection processing to the self-position estimation unit 132, the map analysis unit 151 of the situation analysis unit 133, the traffic rule recognition unit 152 and the situation recognition unit 153, the emergency avoidance unit 171 of the operation control unit 135, etc.

[0131] Note that in the case where the driving section is a section where the vehicle can travel mainly by automatic driving, and the LDM that is constantly updated is supplied from the infrastructure, the information acquired by the vehicle outside information detection unit 141 can be mainly provided from the infrastructure. In the case where there is a vehicle or a convoy traveling ahead in the corresponding section, the vehicle can receive the constantly updated information from the vehicle or the convoy in advance before entering the section, and can travel in the section. In addition, in the case where the latest LDM is not constantly updated from the infrastructure, in order to obtain road information from the vehicle when traveling in a convoy and further ensure safety, the road environment information that is the information immediately before the vehicle enters the corresponding section and is obtained from the preceding vehicle entering the section can be supplementarily further used. Whether the section is a section where automatic driving is allowed is generally determined by the presence or absence of prior information provided from these infrastructures. The updated new LDM provided from the infrastructure for determining the permission of automatic driving on the route is equivalent to providing an invisible track as so-called "information". Note that for convenience, the vehicle outside information detection unit 141 is illustrated on the premise that it is set in the own vehicle. By using information regarded as "information" by the preceding vehicle, the predictability of dangers that may occur while traveling can be further improved.

[0132] The in-vehicle information detection unit 142 can perform detection processing of information related to the interior of the vehicle based on data or signals from various functional units of the vehicle control system 100. For example, the in-vehicle information detection unit 142 performs driver authentication processing and driver identification processing, driver state detection processing, passenger detection processing, in-vehicle environment detection processing, etc. The state of the driver as the detection object includes, for example, physical condition, awakening level, concentration, fatigue, line of sight direction, detailed eye behavior, etc. The in-vehicle environment as the detection object includes, for example, temperature, humidity, brightness, odor, etc. The in-vehicle information detection unit 142 provides data indicating the result of the detection processing to the situation recognition unit 153 of the situation analysis unit 133, the emergency avoidance unit 171 of the operation control unit 135, etc. Note that, for example, when the driver is notified of an RTI as a request to resume manual driving and it is subsequently determined that the driver cannot achieve manual driving within a predetermined deadline, and the driver determines that even if deceleration control is performed to extend the time, manual driving cannot be restored in time, the in-vehicle information detection unit 142 can output instructions to the emergency avoidance unit 171, etc., and a vehicle evacuation process including deceleration, evacuation and parking can be started.

[0133] In addition, as described above, since it is assumed that the driver completely disengages from the driving control work and performs another work, the driver may temporarily fall asleep or start other work (secondary task). Therefore, it is necessary to identify to what extent the awakening level of consciousness required for resuming driving has been restored. Therefore, the above-mentioned in-vehicle information detection unit 142 mainly has two main functions, including a first function and a second function. The first function is passive monitoring of the state of the driver during driving, and the second function is active monitoring to detect and determine whether the driver is at a recovery reaction level that allows manual driving through the driver's conscious response after the notification of RTI as a recovery request for manual driving is performed.

[0134] The vehicle state detection unit 143 can perform detection processing of the state of the vehicle (host vehicle) based on the data or signals from each unit of the vehicle control system 100. The state of the host vehicle as the detection object includes, for example, speed, acceleration, steering angle, presence / absence of abnormality and the content of abnormality, state of driving operation, position and tilt of electric seat, state of door lock, state of other in-vehicle equipment, etc. The vehicle state detection unit 143 provides data indicating the result of the detection processing to the state recognition unit 153 of the state analysis unit 133, the emergency avoidance unit 171 of the operation control unit 135, etc.

[0135] Note that the state of the vehicle (the vehicle) to be identified may include, for example, the position, posture and movement (e.g., speed, acceleration, direction of movement, etc.) of the vehicle (the vehicle), the cargo load related to the determination of the motion characteristics of the vehicle (the vehicle) and the movement of the center of gravity of the vehicle body due to the cargo load, tire pressure, braking distance movement due to the wear condition of the brake pads of the brake, the maximum allowable deceleration braking to prevent the movement of the cargo caused by the braking of the load, the centrifugal force relaxation limit speed when driving along a curve with a liquid load, etc. Note that in this embodiment, the recovery start timing required for control varies with the specific characteristics of the vehicle and further with the load, etc. (such as the cargo loaded, the friction coefficient of the road surface, and the curvature and slope of the road) even in the completely same road environment. Therefore, it is necessary to collect and learn various conditions and reflect these conditions at the optimal timing for performing control.

[0136] The self-position estimation unit 132 can perform estimation processing of the position, posture, etc. of the vehicle (host vehicle) based on data or signals from various functional units of the vehicle control system 100, such as the vehicle exterior information detection unit 141 and the situation recognition unit 153 of the situation analysis unit 133. In addition, if necessary, the self-position estimation unit 132 can generate a local map for estimating the self-position (hereinafter referred to as a self-position estimation map).

[0137] The self-position estimation map is, for example, a high-precision map using a technique such as simultaneous localization and mapping (SLAM). The self-position estimation unit 132 provides data indicating the result of the estimation process to the map analysis unit 151, the traffic rule recognition unit 152, the situation recognition unit 153, etc. of the situation analysis unit 133. In addition, the self-position estimation unit 132 may also store the self-position estimation map in the storage unit 111.

[0138] The situation analysis unit 133 can analyze the situation of the vehicle (host vehicle) and the situation around the vehicle. The situation analysis unit 133 includes a map analysis unit 151 , a traffic regulation recognition unit 152 , a situation recognition unit 153 , and a situation prediction unit 154 .

[0139] The map analysis unit 151 can perform analysis processing of various maps stored in the storage unit 111 and construct a map including information required for the automatic driving process while using data or signals from various functional units of the vehicle control system 100, such as the self-position estimation unit 132 and the vehicle exterior information detection unit 141, as needed. The map analysis unit 151 provides the constructed map to the traffic rule recognition unit 152, the situation recognition unit 153, the situation prediction unit 154, and the route planning unit 161, the action planning unit 162, and the operation planning unit 163 of the planning unit 134.

[0140] The traffic regulation recognition unit 152 can perform recognition processing of traffic regulations around the vehicle (host vehicle) based on data or signals from various units of the vehicle control system 100, such as the self-position estimation unit 132, the vehicle exterior information detection unit 141, and the map analysis unit 151. Through this recognition processing, for example, the position and condition of traffic lights around the vehicle (host vehicle), the content of traffic regulations around the host vehicle, the lanes in which the host vehicle can travel, etc. are recognized. The traffic regulation recognition unit 152 provides data indicating the result of the recognition processing to the situation prediction unit 154 and the like.

[0141] The situation recognition unit 153 can perform recognition processing of the situation of the vehicle (host vehicle) based on data or signals from various functional units of the vehicle control system 100, such as the self-position estimation unit 132, the vehicle-outside information detection unit 141, the vehicle-inside information detection unit 142, the vehicle state detection unit 143, and the map analysis unit 151. For example, the situation recognition unit 153 performs recognition processing of the situation of the vehicle (host vehicle), the situation around the vehicle (host vehicle), the situation of the driver of the vehicle (host vehicle), etc. In addition, when necessary, the situation recognition unit 153 generates a local map (hereinafter referred to as a situation recognition map) for recognizing the situation around the vehicle (host vehicle). The situation recognition map is, for example, an occupancy grid map. In addition, the situation recognition unit 153 provides data indicating the result of the recognition processing (including the situation recognition map when necessary) to the self-position estimation unit 132, the situation prediction unit 154, etc. In addition, the situation recognition unit 153 stores the situation recognition map in the storage unit 111.

[0142] The situation prediction unit 154 can perform prediction processing of the situation of the vehicle (own vehicle) based on the data or signals from the various units of the vehicle control system 100, such as the map analysis unit 151, the traffic rule recognition unit 152, and the situation recognition unit 153. For example, the situation prediction unit 154 performs prediction processing of the situation of the vehicle (own vehicle), the situation around the vehicle (own vehicle), the situation of the driver, etc. Note that the situation of the vehicle (own vehicle) as the prediction object includes, for example, the behavior of the vehicle (own vehicle), the occurrence of abnormalities, the drivable distance, etc. The situation around the vehicle (own vehicle) as the prediction object includes, for example, the behavior of the moving body around the vehicle (own vehicle), the change of the state of the traffic light, the change of the environment such as the weather, etc. The situation of the driver as the prediction object includes, for example, the behavior and physical condition of the driver. Then, the situation prediction unit 154 provides the data indicating the result of the prediction processing together with the data from the traffic rule recognition unit 152 and the situation recognition unit 153 to the route planning unit 161, the action planning unit 162, the operation planning unit 163, etc. of the planning unit 134.

[0143] The route planning unit 161 can plan a route to a destination based on data or signals from various functional units of the vehicle control system 100, such as the map analysis unit 151 and the condition prediction unit 154. For example, the route planning unit 161 sets a route from a current location to a specified destination based on a global map. In addition, the route planning unit 161 sets an automatic driving level for each section on the driving route based on LDM, etc. In addition, for example, the route planning unit 161 can appropriately change the route based on conditions such as traffic congestion, accidents, traffic control and construction, the driver's physical condition, etc. The route planning unit 161 provides data indicating the planned route to the action planning unit 162, etc.

[0144] The action planning unit 162 may plan the action of the vehicle (host vehicle) for safe driving on the route planned by the route planning unit 161 within the planning time based on data or signals from various functional units of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. For example, the action planning unit 162 plans starting, stopping, driving direction (e.g., forward, backward, left turn, right turn, direction change, etc.), driving lane, driving speed, overtaking, etc. The action planning unit 162 provides data indicating the planned action of the vehicle (host vehicle) to the operation planning unit 163, etc.

[0145] The operation planning unit 163 can plan the operation of the vehicle (host vehicle) for implementing the action planned by the action planning unit 162 based on the data or signals from the various functional units of the vehicle control system 100, such as the map analysis unit 151 and the situation prediction unit 154. For example, the operation planning unit 163 plans acceleration, deceleration, driving trajectory, etc. In addition, the operation planning unit 163 can plan the setting of the driving mode, the timing of executing the switching, etc. The operation planning unit 163 provides data indicating the planned operation of the vehicle (host vehicle) to the acceleration / deceleration control unit 172, the direction control unit 173, etc. of the operation control unit 135.

[0146] The operation control unit 135 may control the operation of the vehicle (host vehicle). The operation control unit 135 includes an emergency avoidance unit 171 , an acceleration / deceleration control unit 172 , and a direction control unit 173 .

[0147] The emergency avoidance unit 171 can detect emergency situations such as collision, contact, entering a dangerous area, driver abnormality, and vehicle abnormality based on the detection results of the vehicle exterior information detection unit 141, the vehicle interior information detection unit 142, and the vehicle state detection unit 143. When the occurrence of an emergency situation is detected, the emergency avoidance unit 171 plans the operation of the vehicle to avoid the emergency situation, such as an emergency stop or a sharp turn. The emergency avoidance unit 171 provides data indicating the planned vehicle operation to the acceleration / deceleration control unit 172, the direction control unit 173, and the like.

[0148] The acceleration / deceleration control unit 172 can perform acceleration / deceleration control for implementing the operation of the vehicle (host vehicle) planned by the operation planning unit 163 or the emergency avoidance unit 171. For example, the acceleration / deceleration control unit 172 calculates the control target value of the driving force generating device or the braking device for implementing the planned acceleration, deceleration or emergency stop, and provides a control command indicating the calculated control target value to the drive system control unit 107. Note that, for example, there are mainly two situations in which an emergency situation may occur. That is, there is a situation in which an accident occurs due to a sudden reason during automatic driving on a road that is originally considered to be safe according to the driving route in the automatic driving mode such as LDM acquired from the infrastructure, and the driver's emergency recovery cannot be performed in time, and it is difficult to switch from the automatic driving mode to the manual driving mode.

[0149] The direction control unit 173 may perform direction control for realizing the operation of the vehicle (host vehicle) planned by the operation planning unit 163 or the emergency avoidance unit 171. For example, the direction control unit 173 calculates a control target value of a steering mechanism for realizing a driving track or a sharp turn planned by the operation planning unit 163 or the emergency avoidance unit 171, and provides a control command indicating the calculated control target value to the drive system control unit 107.

[0150] also, Figure 6 1 is a diagram illustrating an example of the setting position of the image capturing device included in the sensor unit 113. Figure 6 As shown in the diagram, in vehicle 7900, the image capture device can be respectively applied to imaging units 7910, 7912, 7914, 7916 and 7918, for example, which are set at at least one of the positions of the front nose, the exterior rearview mirror, the rear bumper, the rear door or the upper part of the windshield inside the vehicle.

[0151] The imaging unit 7910 provided at the front nose and the imaging unit 7918 provided at the upper portion of the windshield inside the vehicle mainly acquire images in front of the vehicle 7900. The imaging units 7912 and 7914 provided at the exterior rearview mirrors mainly acquire images on the sides of the vehicle 7900. The imaging unit 7916 provided at the rear bumper or rear door mainly acquires images behind the vehicle 7900. The imaging unit 7918 provided at the upper portion of the windshield inside the vehicle is mainly used to detect the preceding vehicle, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc. In addition, in future autonomous driving, when the vehicle turns right or left, the imaging unit 7918 can be used to detect pedestrians crossing the road ahead of the right or left turn in a wider range, as well as objects approaching the intersection in an extended range.

[0152] Notice, Figure 6 The diagram illustrates an example of the imaging range of each of the imaging units 7910, 7912, 7914, and 7916. Imaging range a indicates the imaging range of the imaging unit 7910 provided at the front nose, imaging ranges b and c respectively indicate the imaging ranges of the imaging units 7912 and 7914 provided at the exterior rearview mirrors, and imaging range d indicates the imaging range of the imaging unit 7916 provided at the rear bumper or the rear door. For example, by superimposing the image data captured by the imaging units 7910, 7912, 7914, and 7916, a bird's-eye view image of the vehicle 7900 when viewed from above can be obtained. For example, by superimposing the image data captured by the imaging units 7910, 7912, 7914, and 7916, a bird's-eye view image of the vehicle 7900 when viewed from above, an omnidirectional stereoscopic display image in which various parts around the vehicle are surrounded by a curved plane, and the like can be obtained.

[0153] The vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928 and 7930 disposed at the front, rear, sides, corners, and upper portion of the windshield inside the vehicle 7900 may be, for example, ultrasonic sensors or radar devices. The vehicle exterior information detection units 7920, 7926 and 7930 disposed at the front nose, rear bumper, rear door, and upper portion of the windshield inside the vehicle 7900 may be, for example, LiDAR devices. These vehicle exterior information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, and the like. These detection results may also be applied to improve the display of three-dimensional objects such as the above-mentioned bird's-eye view display and omnidirectional stereoscopic display.

[0154] <6.2 Detailed Configuration of Sensor Unit 113>

[0155] Next, refer to Figure 7 Examples of various sensors included in the sensor unit 113 for obtaining information related to the driver in the vehicle will be described. Figure 71 is an explanatory diagram illustrating examples of various sensors included in the sensor unit 113 according to the present embodiment. Figure 7 1 is a diagram illustrating an example of various sensors included in the sensor unit 113 for obtaining information related to the driver in the vehicle. Figure 7 As shown in the diagram, the sensor unit 113 includes a position / posture detection unit 200, which is a detector for detecting the position and posture of the driver, for example, including a ToF camera, a stereo camera, a seat strain gauge, etc. In addition, the sensor unit 113 includes a face recognition unit 202, a face tracking unit 204, and an eye tracking unit (monitoring unit) 206 as detectors for obtaining biometric information of the driver. Hereinafter, the details of various sensors included in the sensor unit 113 according to the present embodiment will be described in sequence.

[0156] The facial recognition unit 202, the facial tracking unit 204, and the eye tracking unit (monitoring unit) 206 may be constituted by, for example, various sensors such as an image capture device. For example, the facial recognition unit 202 recognizes and detects the face of the driver, etc. from the captured image, and outputs the detected information to the facial tracking unit 204. The facial tracking unit 204 detects the movement of the driver's face or head based on the information detected by the facial recognition unit 202. In addition, the eye tracking unit 206 detects the eye behavior of the driver. In addition, the eye tracking unit 206 may include, for example, a sensor such as an electrooculogram (EOG) that measures the electric potential of the eyeball.

[0157] In addition, the sensor unit 113 may include a biometric information detection unit 208 as another detector for obtaining the biometric information of the driver. In addition, the sensor unit 113 may include an authentication unit 210 that performs authentication of the driver. Note that in addition to knowledge authentication using a password, a personal identification number, etc., the authentication method of the authentication unit 210 may also be biometric authentication using a face, a fingerprint, an iris of a pupil, a voice print, etc., without particular limitation. In the above description, the main sensors included in the sensor unit 113 are described. On the other hand, the sensor unit 113 may include various sensors other than these sensors.

[0158] <6.3 Detailed Configuration of Unit for Executing Determination of Driver's Awakening Level>

[0159] Next, refer to Figure 8 A specific configuration example of a unit that performs determination of the driver's awakening level (recovery reaction level) according to an embodiment of the present disclosure will be described. Figure 81 is an explanatory diagram illustrating an example of a unit for performing determination of a driver's awakening level according to an embodiment of the present disclosure. Specifically, the unit for performing determination of a driver's awakening level includes Figure 5 Part of the in-vehicle information detection unit 142 of the detection unit 131 illustrated in the figure, the situation recognition unit 153 of the situation analysis unit 133, and the storage unit 111. More specifically, Figure 8 The diagram illustrates an eye behavior analysis unit 300 and an eye behavior learning device 302 included in the in-vehicle information detection unit 142, a determination unit 320 included in the situation recognition unit 153, and a database (DB) 310 stored in the storage unit 111, which cooperate to determine the driver's awakening level. Figure 8 The various functional blocks are shown in the diagram.

[0160] (Eye behavior analysis unit 300)

[0161] The eye behavior analysis unit 300 acquires and analyzes the driver's eye behavior detected by the eye tracking unit 206 of the sensor unit 113 via the data acquisition unit 102. For example, the eye behavior analysis unit 300 detects and analyzes eye behaviors such as saccades (movements of the eye), gazes, and microsaccades (micromovements of the eye) of the driver's eyes. The eye behavior information analyzed by the eye behavior analysis unit 300 is output to the eye behavior learning device 302 and the determination unit 320 described later.

[0162] In addition, in the present embodiment, the eye behavior analysis unit 300 can dynamically switch the analysis mode according to the driving mode, etc. For example, the eye behavior analysis unit 300 can switch between at least two analysis modes (a first analysis mode and a second analysis mode). Specifically, the eye behavior analysis unit 300 can perform analysis at a high frame rate (first frame rate) of, for example, 250fps or higher in one analysis mode (first analysis mode), and can perform analysis at a low frame rate (second frame rate) of, for example, 60fps in another analysis mode (second analysis mode).

[0163] More specifically, in the event that the driving mode switches from the automatic driving mode to the manual driving mode, the eye behavior analysis unit 300 performs concentrated sampling and analysis of eye behavior at a high frame rate during the period of the preparation mode for the switch (the period of the driving mode change preparation mode) (the first analysis mode). In the eye behavior analysis, the eye behavior analysis unit 300 performs observation (sampling) and analysis of the driver's microsaccades or gaze tremors. Then, the determination unit 320 described later determines the driver's awakening level (recovery reaction level) based on the analysis results. In addition, preferably, the time length of the eye behavior analysis period within the period of the above-mentioned preparation mode is determined so that before the vehicle reaches the switching point of the driving mode corresponding to the automatic driving level, it is ensured that there is enough time to determine the driver's awakening level (i.e., the corresponding level of recovery of the driver's manual driving) with high accuracy, and the switching point is set on a route based on LDM, etc. Therefore, in this embodiment, the start point (monitoring point) of the eyeball behavior analysis period within the period of the preparation mode is determined based on the trip (journey), LDM, road conditions, driving speed, vehicle type (trailer or ordinary passenger car), driver's seating state (state information), etc. That is, the time length of the period is dynamically changed.

[0164] In addition, when the driving mode is the automatic driving mode, the eye behavior analysis unit 300 samples and analyzes the eye behavior at a low frame rate (second analysis mode). The eye behavior analysis is performed by the above-mentioned passive monitoring. For example, the eye closure ratio, saccades, gazes, etc. are also observed and analyzed to determine the driver's drowsiness or fatigue through eye behavior analysis. In addition, in the case of the automatic driving mode, the eye behavior analysis unit 300 can dynamically switch the analysis frequency according to the automatic driving level (automatic driving level 3 or automatic driving level 4). For example, in automatic driving level 3, the analysis can be performed more frequently than in the analysis in automatic driving level 4. This is because, as described above, in automatic driving level 3, in order to ensure safe driving, it is expected that the driver is always in a ready state so that the driver can immediately return to manual driving. Therefore, in automatic driving level 3, it is preferred to perform eye behavior analysis at a high frequency in order to detect the driver's drowsiness and fatigue, and to determine whether the driver can immediately return to manual driving.

[0165] On the other hand, in the above description, specific autonomous driving levels such as autonomous driving level 3 and autonomous driving level 4 are distinguished and recorded. This is for the sake of convenience, and actual application is not necessarily limited to the example of clearly distinguishing and controlling the autonomous driving levels. That is, since the autonomous driving level and the driving mode change according to the ever-changing conditions, in order to determine whether the driver has the perception ability, cognitive ability, judgment ability and action ability required to return to manual driving, each device that observes the driver's state, such as the eye behavior analysis unit 300, appropriately performs observations corresponding to the conditions and obtains observable evaluation values.

[0166] In addition, even in the case of the automatic driving mode, in order to obtain training data for learning by the eye behavior learning device 302 described later, for example, the eye behavior analysis unit 300 can perform eye behavior analysis at a high frame rate or a low frame rate in a time period whose length is shorter than the time length of the eye behavior analysis period in the period of the above-mentioned preparation mode. For example, in a case where the driving mode is the automatic driving mode and a reduction in the driver's awakening level is detected through passive monitoring, the eye behavior analysis unit 300 can perform eye behavior analysis at a high frame rate (first analysis mode). The analysis result in this case is training data for learning by the eye behavior learning device 302 described later (training data marked as eye behavior when the awakening level is reduced).

[0167] On the other hand, when the driving mode is a manual driving mode, the eye behavior analysis unit 300 performs an analysis of eye behavior at a low frame rate (second analysis mode). This eye behavior analysis is performed by the passive monitoring described above. For example, the eye closure ratio, saccades, gazes, etc. are also observed and analyzed to determine the driver's drowsiness or fatigue through eye behavior analysis. In addition, even in the case of a manual driving mode, based on the driver's driving operation, it is determined that the driver is performing manual driving in a normal state. In order to obtain training data for learning by the eye behavior learning device 302 described later (marked as training data of eye behavior when the awakening level is normal), the eye behavior analysis unit 300 can also perform eye behavior analysis. In order to obtain training data, for example, the eye behavior analysis unit 300 performs eye behavior analysis at a high frame rate or a low frame rate in a period of time shorter than the time length of the eye behavior analysis period in the period of the preparation mode described above. In addition, even when the driving mode is the manual driving mode and a decrease in the driver's arousal level is detected by passive monitoring, the eye behavior analysis unit 300 can perform eye behavior analysis at a high frame rate (first analysis mode). The analysis result in this case is also training data (marked as training data of eye behavior when the arousal level decreases) learned by the eye behavior learning device 302 described later.

[0168] That is, in the present embodiment, the eye behavior analysis unit 300 does not always perform eye behavior analysis at a high frame rate. Therefore, the load of imaging processing, analysis processing, etc. can be reduced. In addition, in the present embodiment, the eye behavior analysis unit 300 performs eye behavior analysis at a high frame rate when necessary. Thus, the driver's awakening level (recovery reaction level) can be accurately determined.

[0169] (Eye behavior learning device 302)

[0170] The eyeball behavior learning device 302 uses the analysis results of the eyeball behavior of the driver indicated at each awakening level obtained in the past as training data for learning, generates a database 310 for determination by a determination unit 320 described later, and outputs the database to the storage unit 111 (see Figure 5 ). In this embodiment, for example, the eye behavior learning device 302 can be a supervised learning device such as support vector regression or a deep neural network. In this case, the analysis result (eye behavior) and the arousal level indicating the analysis result (when the arousal level is normal or reduced) are respectively input to the eye behavior learning device 302 as an input signal and a training signal, and the eye behavior learning device 302 performs machine learning of the relationship between these input information according to a predetermined rule. Then, the eye behavior learning device 302 receives multiple pairs of input signals and training signals as inputs, and performs machine learning of these inputs to generate a database (DB) 310 storing relationship information, wherein the relationship information indicates the relationship between the analysis result (eye behavior) and the arousal level. Note that the generated DB 310 is not limited to being stored in the storage unit 111, and can be stored in a server (not shown) on the cloud in association with the identification information for identifying the driver. In the case where the driver changes to a commercial vehicle or uses a shared car or a rental car, the stored DB 310 can also be used for different vehicles. In addition, it is preferred that the information in DB 310 is frequently updated regardless of where the information is stored. Note that, in the case where the restoration requirement obtained by the sport characteristics or the like varies depending on the type of vehicle, normalization of the evaluation determination criteria corresponding to the vehicle may be further performed.

[0171] (Determination unit 320)

[0172] The determination unit 320 determines the driver's awakening level (recovery reaction level) based on the analysis result of the eye behavior analyzed by the eye behavior analysis unit 300. For example, in the case where it is confirmed that the driver performs eye behaviors such as saccades, gazes, or microsaccades to solve the problem, the determination unit 320 can determine that the driver's awakening level is high. On the other hand, in the case where these eye behaviors are not observed or these eye behaviors are few, the determination unit 320 can determine that the driver's awakening level is low.

[0173] Specifically, eyeball behavior indicates different behaviors when a person is in a normal wakefulness state and when a person is in a state of consciousness or a decrease in wakefulness level. In addition, each person exhibits a unique eyeball behavior. Thus, in the present embodiment, the determination unit 320 makes a determination with reference to a database (DB) 310 generated in association with each driver by the eyeball behavior learning device 302. More specifically, in the present embodiment, the determination unit 320 determines the wakefulness level (recovery corresponding level) by comparing the analysis results of the driver's eyeball behavior during the period of the preparation mode with a database (DB) 310 based on the analysis results of the driver's eyeball behavior acquired in the past. Thus, in the present embodiment, the determination is made with reference to the unique eyeball behavior of each driver obtained by learning. Therefore, the accuracy of the determination can be improved.

[0174] In addition, the determination unit 320 may output the determination result to the status prediction unit 154 (see Figure 5 ) and planning unit (mobile driving control unit) 134 (see Figure 5 For example, the planning unit 134 may plan the switching of the driving mode based on the determination result of the determination unit 320 .

[0175] <6.4 Operation Example of Eye Behavior Analysis Unit 300>

[0176] Next, refer to Fig. 9 The details of the operation example of the eye behavior analysis unit 300 according to the embodiment of the present disclosure are further explained. Fig. 9 3 is an explanatory diagram illustrating details of an operation example of the eyeball behavior analysis unit 300 according to an embodiment of the present disclosure. Fig. 9 In the figure, it is assumed that the left end is the departure point (departure point) and the right end is the destination point (destination). In the following, the description will be made from the departure point to the destination point.

[0177] First, in this embodiment, Fig. 9As shown in the diagram, passive monitoring is intermittently performed at a predetermined frequency regardless of the driving mode. For example, passive monitoring includes eye behavior analysis at a low frame rate, performing eye behavior analysis to observe and analyze, for example, eye closure ratio, saccades, gazes, etc., and determining the driver's drowsiness and fatigue.

[0178] In addition, in this embodiment, Fig. 9 In the driving interval in the manual driving mode of the automatic driving level 0 and 1 shown in the left diagram of , not only the above passive monitoring is performed, but also eye behavior analysis (e.g., microsaccades) can be performed when it is determined based on the driver's driving operation that the driver is manually driving in a normal state, so as to obtain training data (marked as training data of eye behavior when the arousal level is normal) for learning by the eye behavior learning device 302. In order to obtain training data, for example, the eye behavior analysis unit 300 can perform eye behavior analysis at a high frame rate.

[0179] In addition, for example, in Fig. 9 In the driving interval of normal driving and traffic congestion, etc. in the manual driving mode of autonomous driving level 2 shown in the left diagram, it is assumed that a reduction in the driver's arousal level is detected by passive monitoring. In this case, passive monitoring can be performed more frequently, and eye behavior analysis (e.g., microsaccades) can be performed at a higher frame rate. The analysis result of the eye behavior in this case is training data for learning by the eye behavior learning device 302 (training data marked as eye behavior when the arousal level is reduced). In addition, in response to detecting a reduction in the driver's arousal level, an alarm or notification can be issued to the driver as active monitoring. Active monitoring can be performed in the case where the driver consciously responds to the alarm or notification. For example, as described above, in the case where the vehicle control system 100 performs steering control with an unnecessary steering amount as active information, a notification is issued. In this case, the driver's action to resume steering with an appropriate steering amount is a conscious response. In addition, as Fig. 9 As shown by the dotted line in , active monitoring is not limited to being performed when the driver's alertness level is reduced, and can be performed regularly at a predetermined frequency. On the other hand, in the manual driving mode, the driver constantly performs road environment recognition judgment in order to perform manual driving. Therefore, the active monitoring described above is not necessarily performed except when the driver's alertness level is reduced.

[0180] Additionally, eye behavior analysis can be performed periodically during sections in manual driving mode.

[0181] Then, in the driving section in the autonomous driving mode of autonomous driving level 4, the driver may leave the driver's seat. During this period, it is difficult to perform eye behavior analysis regularly (observation prohibited period). Therefore, during this period, passive monitoring can be performed more frequently.

[0182] Next, in the interval in the preparation mode for switching the driving mode from the automatic driving mode to the manual driving mode, the eye behavior analysis is intensively performed at a high frame rate. In addition, preferably, the time length of the eye behavior analysis period is determined so that before the vehicle reaches the switching point of the driving mode corresponding to the automatic driving level, sufficient time is ensured to determine the driver's awakening level with high accuracy, and the switching point is set on the route based on LDM, etc. Therefore, in this embodiment, the starting point (monitoring point) of the eye behavior analysis period is determined based on the itinerary (journey), LDM, road conditions, driving speed, vehicle type (trailer or ordinary passenger car), driver's sitting state (state information), etc. That is, the time length of the period is dynamically changed. At this time, in the case of detecting that the driver's awakening level is reduced, the driver is restored to the internal awakening of the driver in the order of notification, alarm, MRM alarm, noise injection (questions in the form of a test), etc., and at the same time, active monitoring for detecting the driver's conscious response is intensively performed.

[0183] That is, in the present embodiment, the eye behavior analysis is not always performed at a high frame rate. Therefore, the load of imaging processing, analysis processing, etc. can be reduced. In addition, in the present embodiment, the eye behavior analysis is performed at a high frame rate when necessary. Therefore, the driver's awakening level (recovery reaction level) can be accurately determined.

[0184] Note that the present applicant has submitted multiple patent applications (Patent Documents 2 to 4) for such a configuration for determining the driver's recovery reaction level (awakening level), specifically, a configuration related to observations used for determination, and the contents disclosed in these patent applications are also included in the embodiments of the present disclosure. In addition, the present applicant has submitted a patent application (see Patent Document 4) for a configuration for estimating the time required for the driver to properly return to manual driving (manual driving recovery possible time) based on a driver's state evaluation value obtained through observations, etc. before switching from an automatic driving mode to a manual driving mode, and this configuration can also be used in the present embodiment.

[0185] In this embodiment, various observations of the driver recorded in patent documents, etc. are continuously performed, the tendency of observation is learned, and the recovery reaction level such as the awakening level is determined taking into account the change of the tendency at this time. This is because even for the same driver, the driver's previous experience and resume will affect the driver's perception, cognition, judgment, action, etc. Specifically, for example, the eyeball behavior to be observed changes greatly according to the method in which the driver senses the necessity of recovering (switching) from automatic driving to manual driving. For example, in the case of insufficient information required to reduce risks through intuitive actions, the number of times that information that can be obtained visually is searched is increased. For this reason, in the case of fully preparing the information required for judgment, the transition from search to action can be performed without searching too much information. Therefore, in the case of insufficient information, in order to search for insufficient information through the recognition of individual visual information, the driver repeats the operation of fixing the driver's line of sight to the object, such as gaze characterized by saccades, microsaccades, drifts and tremors.

[0186] In particular, at the stage of risk determination in the case of a recovery action from automatic driving to manual driving, the information remaining in the memory has a high risk and is not sufficient for determination. Therefore, the driver is required to search for information visually. For example, in the case of a driver who temporarily does not look ahead and keeps watching a video or operating a mobile terminal, the driver first looks in front of the vehicle to grasp the situation, and then pays attention to the lanes, obstacles, parallel vehicles, or the movement of oncoming vehicles that affect the driving direction of the vehicle, and performs processes such as the formation of a gaze for understanding the situation and the check of message information of RTI (notification) as a recovery request. In addition, for example, in an urban area where roads on which pedestrians jump and school districts where children rush out are mixed, the visual behavior of checking whether anyone enters the road from the peripheral parts of the road is dominant.

[0187] The system that allows humans to temporarily accumulate information and process information during the execution of cognitive tasks is called working memory (working storage). Thus, the information required for the judgment of human actions is accumulated and processed in the working memory of humans. On the other hand, it is believed that there are limits on the capacity and period of information accumulation. Specifically, the information accumulated in the working memory decays over time. For example, information with low importance decays first, so the working memory works like a dynamic cache memory. With the popularization of autonomous driving and the improvement of the performance of autonomous driving, the range of available driving designs is expanded. Therefore, it is believed that the necessity for users to constantly check the surrounding environment information required for safe manual driving is gradually decreasing. Thus, the number of forward checks required for driving is reduced, or even no forward check is performed. Thus, when the work of disengaging from the driving control work is gradually increased, the prior visual information of the driving road required for judgment is reduced. Originally, the reason why the driver regularly checks the road ahead during manual driving is that there is a weighted stimulus of the importance of the risk that can become a risk factor. Thus, the driver continues to perform prior visual search without slacking, thereby preventing careless driving. On the other hand, in the case of an increase in the work of disengaging from the driving control work, there are few risk factors in the memory of the decaying working memory. As a result, the necessity of regularly observing changes in conditions is reduced. Therefore, the observation of the driver's behavior of turning his / her eyes toward the road and rechecking the situation is also reduced. In this embodiment, considering the characteristics of human working memory that information with low risk importance as memory information decays first over time, information provision and notification for appropriate manual driving are performed at appropriate timing for the driver. Therefore, the driver's state or the driver's response is observed.

[0188] In addition, in order for the driver to normally start actual manual driving in response to the RTI (notification) as a request to resume (switch) from the automatic driving to the manual driving, a period of time is required for acquiring information required for action determination. In addition, the visual information acquisition includes not only visual inspection as an action to acquire the latest information, but also an action to acquire information for feedback on the action control. Note that in the present embodiment, as the information required for action determination, for example, a human-machine interface such as the means disclosed in Patent Document 5 filed by the present applicant can be used. For example, Patent Document 5 discloses a technology that divides a driving route into various sections (manual driving section, driver interference required section, etc.) and displays the driving route to the driver with different colors and different line widths for each section. In addition, by designing a display time axis of approach information, the driver can visually know when the vehicle approaches each section requiring each measure as time passes (= the driving of the vehicle). That is, as described in Patent Document 5, the driver regularly updates and provides approach information during the driving process along the vehicle's itinerary. Therefore, as time passes, the fact that the vehicle approaches the takeover point is visually incorporated into the working memory of thinking as a risk. Thus, the driver can regard the approaching takeover approach information as an imminent risk. In this way, the provision of visual proximity information including semantic information corresponds to the stimulation of the state before the check taking over of the working memory. Thus, the method of giving updated information also affects the visual behavior of the driver. Thus, in the present embodiment, in the case of estimating the state of the driver based on the eye behavior analysis and the observation evaluation value of the eye behavior, the observation behavior can be evaluated by using the information provided to the driver as an influencing factor. By updating and presenting information at any time using the human-machine interface, the driver will recognize the importance of the necessity of returning to manual driving from automatic driving with a sense of time approaching. Therefore, the visual information required for recovery is accumulated in the working memory of the driver. Then, in the driver's action judgment for recovery, based on the presented information and the urgency caused by the driver's sense of risk, the action of acquiring insufficient information is performed before turning to action. Thus, by using the presented information of the human-machine interface, the prediction information required for driving is effectively input (stored) into the working memory of the driver, and by appropriately providing information that makes the driver feel the necessity of re-checking midway, the driver's unconscious significant departure can be prevented. As a result, the driver's judgment is accelerated, so the driver's recovery delay time (manual driving recovery possible time) disclosed in Patent Document 3 can be reduced.

[0189] In the present disclosure, the main focus is on the analysis of eyeball behavior. On the other hand, in the case where the driver's awakening state is insufficient, the feedback of obtaining information described above may be incomplete. As a result, in addition to the eyeball behavior, various behaviors and the like may occur, which may lead to excessive control by the driver in some cases. Thus, while regularly checking the predictive information for predicting the conditions at the location where the autonomous driving is switched to manual driving, the eyeball behavior for the RTI (notification) as a recovery request is different between the driver who uses the autonomous driving function and the driver who completely ignores such regular checks. In addition, when the time required for the driver to return to manual driving is insufficient, and the driver switches to the control action without fully understanding the conditions for manual driving, the feedback in the control action is often incomplete, and excessive control with an inappropriate amount of control, such as oversteering, may occur.

[0190] Therefore, in order to make the estimation of the driver's recovery reaction level (awakening level) and recovery delay time by analyzing the eye behavior function properly, the vehicle control system 100 according to the embodiment of the present disclosure is constructed as a system that determines the driver's recovery action based on the approach information presented to the driver as the vehicle travels, the additional information added to the risk information to be presented, the notification to the driver, the actual recovery action and eye behavior caused by the notification, the handling stability as a result of the recovery action, etc.

[0191] <6.5 Information processing methods>

[0192] Next, refer to Fig.10 The information processing method according to the embodiment of the present disclosure is described. Here, the method will be described with a focus on the acquisition of training data. Fig.10 is a flow chart of an information processing method according to an embodiment of the present disclosure. Specifically, Fig.10 As shown in the diagram, the information processing method according to the present embodiment includes steps from step S101 to step S114. The details of each of these steps according to the present embodiment will be described below.

[0193] First, the vehicle control system 100 determines whether the current section corresponds to the automatic driving mode section (step S101). The vehicle control system 100 proceeds to the process of step S102 when the current section corresponds to the automatic driving mode section (step S101: Yes), and proceeds to the process of step S111 when the current section does not correspond to the automatic driving mode section (i.e., the manual driving mode section) (step S101: No).

[0194] Here, the observation and analysis of eyeball behavior will be mainly focused on for explanation. In the automatic driving mode section, the vehicle control system 100 regularly performs various observations (passive monitoring) for estimating the position, posture and activity content of the driver before awakening, and predicting the time required for the driver to recover to the corresponding level (awakening level) that allows manual driving. Then, the vehicle control system 100 determines whether a driving mode switching event has occurred, and at the timing of the event, determines whether the current time has reached a preset monitoring point based on the predicted information (step S102). In the case of a driving mode switching event occurring and the current time reaching a preset monitoring point (step S102: Yes), the vehicle control system 100 proceeds to the processing of step S103. On the other hand, in cases other than the case of a driving mode switching event occurring and the current time reaching a preset monitoring point (step S102: No), the vehicle control system 100 proceeds to the processing of step S104.

[0195] Next, according to the switching of the driving mode, the vehicle control system 100 performs an RTI (notification) as a request to restore to the initial stage to the driver, detects that the driver is sitting in the driver's seat, observes the driver's posture (body or face direction), etc., and centrally performs eye behavior analysis at a high frame rate (step S103). Here, one purpose of performing eye behavior analysis is to estimate the amount of activity in the driver's brain and check whether the driver has recovered to the recovery corresponding level (awakening level) that allows manual driving immediately before switching to the manual driving mode. The sampling and analysis of the eye behavior is performed within a limited period of time based on a captured image of a narrow area containing only the face and eyeballs of the driver sitting in the driver's seat. At this time, in addition to the eye behavior analysis at a high frame rate, the vehicle control system 100 can also centrally perform active monitoring, including notifications, alarms or questions (for responses involving cognition and judgment, such as questions in the form of a test) and responses to notifications, alarms or questions (speaking, gestures, manual input, movement of sight, etc.), input of pseudo-R steering, and detection of viewing of display information provided by the vehicle control system 100, etc. In addition, the vehicle control system 100 can detect the object to be viewed, the viewing order of the objects, the torque input of the steering wheel, the pressing action of the brake pedal, the pressing action of the accelerator pedal, etc. By performing such detection, it can be detected whether the driver is properly performing the manual driving recovery process. In addition, active monitoring can be performed simultaneously with the eye behavior analysis at a high frame rate, or it can be performed with a time difference.

[0196] In addition, the vehicle control system 100 periodically performs passive monitoring, for example, including eye behavior analysis at a low frame rate (step S104). Fig.10In the flowchart, passive monitoring including eye behavior analysis is performed. On the other hand, the present embodiment is not limited thereto, and passive monitoring without eye behavior analysis may be performed periodically. Note that in step S104, it is preferred that the vehicle control system 100 continuously observes changes in events that require switching from automatic driving to manual driving, and appropriately changes monitoring content, monitoring timing, monitoring cycle, etc. according to the changes. Note that the details of step S104 will be described later.

[0197] Next, the vehicle control system 100 detects the driver's awakening level based on the result of the eye behavior analysis in the above step S103, and further determines whether the driver can resume the corresponding level of recovery to manual driving (step S105). If the vehicle control system 100 determines that the driver is awake enough to resume manual driving (step S105: yes), it proceeds to step S106, and if it determines that the driver is not awake enough to resume manual driving (step S105: no), it proceeds to step S109.

[0198] Next, the vehicle control system 100 performs a switch from the automatic driving mode to the manual driving mode (step S106). In addition, the vehicle control system 100 determines whether the switch is normally performed (step S107). The vehicle control system 100 proceeds to the process of step S108 when the switch is normally performed (normal) (step S107: yes), and proceeds to the process of step S109 when the switch is not normally performed (failure) (step S107: no).

[0199] Next, the vehicle control system 100 marks the result of the eye behavior analysis in the above step S103 as training data when the arousal level is normal, performs learning (step S108), and then returns to the process of step S101.

[0200] Next, the vehicle control system 100 starts the MRM (step S109 ). In addition, the vehicle control system 100 marks the result of the eye behavior analysis in step S103 as training data when the arousal level is abnormal, performs learning (step S110 ), and then ends the process.

[0201] Next, the vehicle control system 100 determines whether a decrease in the driver's arousal level is observed (step S111). The vehicle control system 100 proceeds to the process of step S112 when a decrease in the driver's arousal level is observed (step S111: Yes), and proceeds to the process of step S114 when a decrease in the driver's arousal level is not observed (step S111: No). Note that in the present embodiment, in the manual driving mode section, it is preferable to periodically perform observations for estimating the driver's arousal level according to the automatic driving level, etc.

[0202] Next, the vehicle control system 100 notifies and warns the driver (active monitoring) (step S112). In addition, the vehicle control system 100 performs eye behavior analysis at a high frame rate, marks the results of the eye behavior analysis as training data when the arousal level is reduced, performs learning (step S113), and then returns to the process of step S101.

[0203] Next, the vehicle control system 100 performs eye behavior analysis, marks the results of the detailed analysis of the eye behavior as training data when the arousal level is normal, performs learning (step S114), and then returns to the process of step S101.

[0204] Next, refer to Fig.11 The details of the above-mentioned step S104 are described. Fig.11 yes Fig.10 The sub-flow chart of step S104 shown in the diagram is shown in FIG. Specifically, Fig.11 As shown in the diagram, the information processing method according to this embodiment may include steps from step S201 to step S205. The details of each of these steps are described below.

[0205] First, the vehicle control system 100 obtains the journey road environment, local weather information, RRR information such as the predetermined RRR from the infrastructure and the preceding vehicle (step S201). Here, the required recovery rate (RRR) refers to the predetermined success rate at which the driver is expected to start manual driving appropriately at the location where the switch from automatic driving to manual driving is made, and the location is determined based on LDM, etc. In addition, it is assumed that the estimated extended time (manual driving recovery possible time) for allowing the switch to be made at the RRR at the arrival timing of the switching location from automatic driving to manual driving is defined as ΔT RRR (T) (Note that the estimated extended time is disclosed as the restoration delay time in Patent Document 3). That is, when ΔT elapses from the RTI (notification) as the restoration request RRR(T), it is expected that the driver will appropriately return to manual driving with the success rate of RRR. Note that RRR is embedded information assigned to each section of the road. For example, the vehicle control system 100 can obtain RRR from the infrastructure together with the update information of LDM. In addition, RRR can be appropriately changed and set from the numerical value assigned to each section according to the driver's attribute information (driving experience, driving skills, etc.), etc. More specifically, for example, in the case where the switch to manual driving cannot be made due to a narrow main road, etc., and the vehicle stops in the corresponding section, when the vehicle may cause road closure or traffic jam, the RRR assigned to the section is set to 100% (that is, the switch to manual driving is required to be reliably made in the section). On the other hand, for example, in the case where the switch to manual driving cannot be made due to the road being a rural road with very little traffic, and the vehicle stops in the corresponding section, when the vehicle does not cause road closure or traffic jam, the RRR assigned to the section can be set to a lower value, such as 50% (that is, in the section, the switch to manual driving is not required to be reliably made). In addition, the estimated extension time ΔT RRR (T) can be calculated based on the set RRR, the driver's attributes (driving experience, driving skills, etc.) and status (content of the secondary task, arousal level, physical condition, posture, etc.).

[0206] Next, after confirming that the driver has returned to the driver's seat according to the first-stage notification included in the RTI (notification) as the multi-stage return request, the vehicle control system 100 starts the process for estimating ΔT RRR (T) observation (step S202).

[0207] Next, the vehicle control system 100 compares the remaining time until the vehicle reaches the point where the switch from automatic driving to manual driving is performed (takeover limit point) with the predetermined time (step S203). ob1 The timed backoff time ΔT from reaching the takeover limit point has passed RRR +ΔT cyc Timing T bac In the case of the next observation, the observation timing T ob2 The timed backoff time ΔT from reaching the takeover limit point has passed RRR Timing T nt (Note that ΔT cyc is the observation period). In this case, the observation timing T ob2 The time T for notification has passed. nt As a result, the timing of performing RTI (notification) as a recovery request to the driver is missed. Therefore, in the present embodiment, in the above step S203, the vehicle control system 100 compares the remaining time with ΔTRRR +2×ΔT cyc (predetermined time), and check the observation period ΔT according to the comparison result cyc Thereafter, the vehicle control system 100 performs several observations or determines the timing of RTI (notification) as a restoration request.

[0208] More specifically, the vehicle control system 100 controls the vehicle when the remaining time is less than ΔT. RRR +2×ΔT cyc In the case of (step S203: Yes), the process proceeds to step S204, and the remaining time is greater than ΔT RRR +2×ΔT cyc (Step S203: No), return to the processing of step S201.

[0209] The vehicle control system 100 observes the period ΔT cyc Reset it to be shorter, and then observe several times, or change the observation method that can more accurately determine the recovery corresponding level (awakening level) (step S204). Note that this embodiment is not limited to the observation period ΔT based on the remaining time. cyc For example, in the case of a recent switch from automatic driving mode to manual driving mode, the observation period ΔT can be checked. cyc Alternatively, the observation means may be changed. In the present embodiment, for example, when the RRR is changed due to a change in the traffic volume at the location where the switch from the automatic driving to the manual driving is made (the takeover limit point), a change in the probability of an accident caused by the change in the traffic volume, a filling degree of the evacuated parking space, etc., when the RRR changes, the observation period ΔT may be checked. cyc Alternatively, the observation means may be changed. In addition, in the present embodiment, for example, in the case where the RRR is changed based on the driver's arousal level or the content of non-driving related activities (NDRA) such as secondary tasks, the observation period ΔT may be checked. cyc Alternatively, the observation means may be changed. In addition, in the present embodiment, the observation means may be changed based on factors that complicate the situation recognition performed when the driver is driving manually (straight roads, mixed traffic, icy roads, and complexity risks).

[0210] Furthermore, in the present embodiment, the vehicle control system 100 determines the timing of performing RTI (notification) as a recovery request (step S204). In the present embodiment, for example, the notification timing may be changed based on the driver's awakening level, the content of NDRA such as a secondary task, the driver's position and posture, etc. (for example, in the case where the awakening level is low, the notification timing is set to be earlier). Furthermore, the present embodiment is not limited thereto. For example, in the case where the RRR is changed due to the occurrence of a new event of switching from the automatic driving mode to the manual driving mode, the change in the traffic volume at the takeover limit point, the change in the probability of accident occurrence caused by the change in traffic volume, the degree of filling of the evacuated parking space, etc., the notification timing may be changed. Note that in the case where the notification is performed immediately before switching from the automatic driving mode to the manual driving mode, the time required for the driver to return to the manual driving mode may not be enough depending on the state of the driver, so that the switching at the takeover limit point (the start of the manual driving) may be difficult. On the other hand, in the case where the notification is performed early, the driver's awareness of the importance of the necessity of starting the switching action (the takeover action) from the automatic driving mode to the manual driving mode becomes low. As a result, it is considered that the preparation for switching to the manual driving may be neglected. Therefore, in this embodiment, it is required to perform RTI (notification) as a recovery request at the optimal timing. In addition, in this embodiment, notification can be performed multiple times.

[0211] Next, the vehicle control system 100 determines whether to start the operation for switching from the automatic driving mode to the manual driving mode (step S205). In the case where it is determined that the operation is to be started (step S205: Yes), the vehicle control system 100 ends step S104 to start the operation and proceeds to Fig.10 On the other hand, in the case where it is determined that the operation is not to be started (step S205: No), the vehicle control system 100 returns to the process of step S201.

[0212] As described above, in the present embodiment, the analysis result of the eyeball behavior peculiar to each driver during driving is obtained, and learning is performed, and the awakening level (recovery reaction level) of the driver is determined with reference to the eyeball behavior peculiar to each driver obtained based on such learning. As a result, according to the present embodiment, the accuracy of determination can be improved.

[0213] <6.6 Summary>

[0214] As described above, according to the embodiments of the present disclosure, it is possible to accurately determine the driver's awakening level (recovery reaction level) while reducing the load of imaging processing, analysis processing, and the like.

[0215] Note that in the embodiments of the present disclosure, a vehicle is described as an example. On the other hand, the present embodiment is not limited to the example of application to a vehicle, and can be applied to mobile bodies such as vehicles, electric vehicles, hybrid electric vehicles, motorcycles, personal mobility devices, airplanes, ships, construction machinery, and agricultural machinery (tractors).

[0216] <<7. Hardware Configuration>>

[0217] A portion of the vehicle control system 100 according to the above-described various embodiments is composed of, for example, Fig.12 The configuration shown in the diagram is implemented by a computer 1000. Fig.12 1 is a hardware configuration diagram illustrating an example of a computer 1000 that implements some functions of the vehicle control system 100. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The various units of the computer 1000 are connected to each other via a bus 1050.

[0218] The CPU 1100 operates based on the programs stored in the ROM 1300 or the HDD 1400 and controls the respective units. For example, the CPU 1100 expands the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the respective programs.

[0219] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, a program depending on the hardware of the computer 1000, and the like.

[0220] The HDD 1400 is a non-transitory computer-readable recording medium that records a program executed by the CPU 1100 , data used by the program, etc. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure as an example of program data 1450 .

[0221] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from other devices via the communication interface 1500, or transmits data generated by the CPU 1100 to other devices.

[0222] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from the input / output device 1650 such as a keyboard, a mouse, and a microphone (microphone) via the input / output interface 1600. In addition, the CPU 1100 sends data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can play the role of a medium interface for reading a program recorded in a predetermined recording medium (medium), etc. The medium is, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase-change rewritable optical disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, etc.

[0223] For example, in the case where the computer 1000 functions as a part of the vehicle control system 100 according to the embodiment of the present disclosure, the CPU 1100 of the computer 1000 implements the functions of the automatic driving control unit 112 and the like by executing the program stored in the RAM 1200. In addition, the HDD 1400 stores the information processing program and the like according to the present disclosure. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data. On the other hand, as another example, these programs may be acquired from other devices via the external network 1550.

[0224] In addition, the automatic driving control unit 112 according to the present embodiment, etc., can be applied to a system including multiple devices, such as cloud computing, which is premised on being connected to a network (or communication between devices). That is, the information processing device according to the present embodiment described above can be implemented as an information processing system according to the present embodiment through multiple devices, for example. An example of the hardware configuration of a part of the vehicle control system 100 is described above. The above-mentioned various components can be configured using general-purpose parts, or can be configured by hardware dedicated to the functions of each component. Such a configuration can be appropriately changed according to the technical level at the time of implementation.

[0225] <<8.Supplement>>

[0226] Note that the embodiments of the present disclosure described above may include, for example, an information processing method executed by an information processing device or an information processing system as described above, a program for causing the information processing device to function as a vehicle control system, and a non-transitory tangible medium recording the program. In addition, the program may be distributed via a communication line (including wireless communication) such as the Internet.

[0227] In addition, the various steps in the information processing method of the embodiment of the present disclosure described above may not necessarily be performed in the order of the records. For example, the various steps may be performed in an appropriately changed order. In addition, some steps may be performed in parallel or individually, rather than sequentially. In addition, the processing of each step does not necessarily have to be performed in accordance with the recorded method, for example, it may be performed by other methods via other functional units.

[0228] Note that in the description of the embodiment of the present disclosure described above, it is assumed that the image capture device is controlled by a fixed or variable operating frame rate. On the other hand, in a type of image capture device called an event-driven imaging device, an event-based imaging device, a neuromorphic retina, or an event camera, the concept of frame rate does not necessarily exist. These event-driven imaging devices are based on pulses recognized as changes in brightness, convert the occurrence time and coordinates of the quantized changes into signals, and record the signals. In the image capture control of the event-driven imaging device, when the changes in information captured as light appear drastically throughout the screen, the amount of information processing performed in parallel becomes too much, thereby increasing the amount of processing. As a result, an increase in the operating temperature of the device, an excessive number of events occurring in parallel at the same time, etc. may occur. Therefore, in the case of observing the state of the driver using these event-driven imaging devices in which the concept of frame rate does not exist, in order to optimize the detection frequency of the event (change) by the event-driven imaging device during the observation period for evaluation, the following steps are preferably performed. In the present embodiment, for example, in order to adjust the illumination brightness of the lighting device that illuminates the driver with light having a wavelength in a narrow infrared wavelength region, the light receiving conversion gain or the quantization threshold for the detection of the event (change) is adjusted. By making adjustments, it is possible to perform observations such as detection of eye behaviors such as microsaccades and drifts of the eye, detection of blinks, or observation of changes in facial expressions at an optimal frequency while suppressing an increase in the amount of processing, so as to identify changes in the observed object. That is, in the case of using an event-driven imaging device, by controlling a threshold value, etc., switching of the analysis mode according to an embodiment of the present disclosure can be performed.

[0229] In addition, in the description of the embodiment of the present disclosure, the embodiment is described in detail based on the automatic driving level defined by the SAE. On the other hand, the concept of classifying the use of automatic driving according to the automatic driving level is a classification divided from the design perspective of the vehicle. On the other hand, from the user's perspective, after the user always correctly understands and grasps the automatic driving level within the driving design range that allows driving according to each level in the automatic driving level, it is not easy for the driver to drive the vehicle according to the automatic driving level in the available automatic driving level of the vehicle. That is, this design can be called a machine-centric design, in which the vehicle is used according to the function or instruction of the machine. That is, in the case where the corresponding conditions of the vehicle system dynamically change over time due to various external and internal factors, and the automatic driving level during driving is not uniquely determined by only the physical road section, etc., it can be said that the driver is required to independently cope with the level allowed by the road conditions determined by the vehicle control system 100 each time. On the other hand, in the case of observing the relationship between the driver and the vehicle control system 100 from the perspective of ergonomics, in order to achieve the purpose of using the vehicle such as moving, and to obtain the secondary benefits obtained during the moving, the user considers the burden of driving and the balance between various risks caused by driving to make an action decision. Here, the burden refers to the vehicle control work performed for moving or a certain risk caused by the vehicle control work. Originally, the benefit of autonomous driving from the driver's perspective is that it is freed from the constraints of driving, and the time used for driving can be used for meaningful time that is not related to driving and can be used independently of driving. In order to provide such a benefit, it can be said that it is necessary to convert the idea of ​​supporting autonomous driving control into an idea called human-centered design obtained by reversing the relationship among the ideas called machine-centered design in the prior art. Then, in the case of examining the relationship between the vehicle control system 100 of the vehicle and the driver as a user from the perspective of this idea, it can be said that from the perspective of ergonomics, the use of autonomous driving is a desirable form of use, and as a design of the vehicle, autonomous driving is used to correspond to the available autonomous driving levels in the "driving design range", allowing various actual autonomous driving functions according to the driver's state of awakening and physical readiness.

[0230] A person makes his / her action selection while maintaining a balance between the selectable benefits and the losses or risks involved in the benefits, and further learns the action. By introducing control based on human-centered design, action learning is performed so that the driver can prepare appropriate recovery in accordance with the upper limit of the autonomous driving control environment allowed for each road on which the driver drives the vehicle. In addition, for the driver who performs such action learning, the driver is allowed to disengage at a higher level such as autonomous driving level 4, that is, disengage from the driving control work through advanced autonomous driving driving that can obtain the benefits of performing NDRA, etc. On the other hand, in the case where a state indicating the driver's expected appropriate recovery cannot be observed, or in the case where the recovery level is determined to be lowered based on the driver's observed state with reference to the driver's past recovery response history or learning data, the use of autonomous driving can be greatly restricted by prohibiting the use of autonomous driving level 4, etc., where the driver can disengage from the driving control work, or limiting the use of autonomous driving level 3 to a short time, even within the "designed driving range".

[0231] That is, instead of allowing the level of autonomous driving to be determined as a "driving design range" based on the performance standards of the sensing equipment set in the vehicle and the performance standards of the judgment processing, the autonomous driving control that allows the use of autonomous driving based on the corresponding ability of people is a form of vehicle use that is easy for people. That is, by changing the control concept of the vehicle's autonomous driving system from the concept called machine-centric design to human-centric design, a form of use that is easy for people can be provided through autonomous driving control. In addition, in the description of the present disclosure, applicable control using the driver's state observation means is explained based on the former machine-centric design. On the other hand, even in the case where the control concept is replaced with a human-centric design, the driver's switching action (takeover action) from autonomous driving to manual driving can similarly occur. Therefore, applicable control using the driver's state observation means described above can be performed.

[0232] As described above, the preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings. On the other hand, the technical scope of the present disclosure is not limited to such examples. It should be understood by those skilled in the art that various modifications and various changes can be made within the scope of the technical ideas recorded in the claims, and such modifications or changes are within the scope and spirit of the present disclosure.

[0233] In addition, the effects described in this specification are merely illustrative or exemplary, and not restrictive. That is, together with or in place of the above effects, the technology disclosed herein may exhibit other effects that are obvious to those skilled in the art based on the description of this specification.

[0234] Note that the following configurations are also within the technical scope of the present disclosure.

[0235] (1) An information processing device comprising:

[0236] an eyeball behavior analysis unit, the eyeball behavior analysis unit analyzing the eyeball behavior of a driver driving the mobile body,

[0237] The eye behavior analysis unit dynamically switches the analysis mode according to the driving mode of the mobile object.

[0238] (2) The information processing device according to (1), wherein the eye behavior includes at least one of saccade, gaze, or microsaccade of the eye.

[0239] (3) The information processing device according to (1) or (2), wherein the eye behavior analysis unit

[0240] switching between at least a first analysis mode and a second analysis mode,

[0241] In the first analysis mode, analysis is performed at a first frame rate, and

[0242] In the second analysis mode, analysis is performed at a second frame rate that is lower than the first frame rate.

[0243] (4) The information processing device according to (3), wherein in a driving mode change preparation mode in which the driving mode of the mobile body is switched from the automatic driving mode to the manual driving mode, the eye behavior analysis unit performs analysis in the first analysis mode.

[0244] (5) The information processing device according to (4), wherein a starting point of the first analysis mode is determined based on at least one of a travel distance, a local dynamic map, and a driver's state information.

[0245] (6) The information processing device according to (4) or (5) further includes: a determination unit, which determines the driver's corresponding level of recovery to manual driving based on the analysis result of the driver's eye behavior in the first analysis mode.

[0246] (7) The information processing device according to (6), wherein the determination unit determines the restoration correspondence level by comparing an analysis result of the driver's eye behavior in the first analysis mode with a previously acquired analysis result of the driver's eye behavior.

[0247] (8) The information processing apparatus according to (7) further includes: a learning device that generates a database for the determination by learning previously acquired analysis results of the driver's eye behavior.

[0248] (9) The information processing device according to any one of (6) to (8), further comprising: a mobile vehicle driving control unit, the mobile vehicle driving control unit switching the driving mode of the mobile vehicle based on the determination result of the recovery corresponding level.

[0249] (10) The information processing device according to any one of (4) to (9), wherein when the driving mode of the mobile body is the automatic driving mode, the eye behavior analysis unit performs analysis in the second analysis mode.

[0250] (11) An information processing device according to (10), wherein in the autonomous driving mode, the eye behavior analysis unit dynamically switches the analysis frequency according to the autonomous driving level.

[0251] (12) An information processing device according to any one of (4) to (11), wherein when the driving mode of the mobile body is a manual driving mode, the eye behavior analysis unit performs analysis in the first analysis mode or the second analysis mode.

[0252] (13) The information processing device according to any one of (4) to (12), wherein the eye behavior analysis unit performs analysis in the first analysis mode as the driver's arousal level decreases.

[0253] (14) An information processing system comprising:

[0254] an eyeball behavior analysis unit, the eyeball behavior analysis unit analyzing the eyeball behavior of a driver driving the mobile body,

[0255] The eye behavior analysis unit dynamically switches the analysis mode according to the driving mode of the mobile object.

[0256] (15) The information processing system according to (14), further comprising: a monitoring unit that monitors the eye behavior of the driver.

[0257] (16) An information processing method comprising:

[0258] The eye behavior of a driver driving a mobile object is analyzed via an eye behavior analysis unit, wherein an analysis mode of the analysis is dynamically switched according to a driving mode of the mobile object.

[0259] (17) An information processing program causing a computer to execute:

[0260] An analysis function that analyzes the eye behavior of a driver driving a mobile object,

[0261] The analysis mode of the analysis function is dynamically switched according to the driving mode of the mobile body.

[0262] Reference numerals list

[0263] 100 Vehicle Control Systems

[0264] 101 Input Unit

[0265] 102 Data acquisition unit

[0266] 103 Communication Unit

[0267] 104 In-vehicle equipment

[0268] 105 Output control unit

[0269] 106 Output Units

[0270] 107 Drive system control unit

[0271] 108 Drive system

[0272] 109 Body system control unit

[0273] 110 Body System

[0274] 111 Storage Unit

[0275] 112 Automatic driving control unit

[0276] 113 Sensor unit

[0277] 121 Communication Network

[0278] 131 Detection Unit

[0279] 132 Self-position estimation unit

[0280] 133 Situation Analysis Unit

[0281] 134 Planning Units

[0282] 135 Operation control unit

[0283] 141 External vehicle information detection unit

[0284] 142 In-vehicle information detection unit

[0285] 143 Vehicle status detection unit

[0286] 151 Map Analysis Unit

[0287] 152 Traffic rules recognition unit

[0288] 153 Situation Recognition Unit

[0289] 154 Situation Prediction Unit

[0290] 161 Route Planning Unit

[0291] 162 Action Planning Unit

[0292] 163 Operation Planning Unit

[0293] 171 Emergency Shelter Unit

[0294] 172 Acceleration / deceleration control unit

[0295] 173 Direction control unit

[0296] 200 Position / attitude detection unit

[0297] 202 Facial Recognition Unit

[0298] 204 Face Tracking Unit

[0299] 206 Eye Tracking Unit

[0300] 208 Biometric Information Detection Unit

[0301] 210 Certification Units

[0302] 300 Eye Behavior Analysis Unit

[0303] 302 Eye Behavior Learning Equipment

[0304] 310 Database

[0305] 320 judgment unit

Claims

1. An information processing device, comprising: An eye behavior analysis unit, the eye behavior analysis unit analyzes the eye behavior of a driver driving a mobile object, wherein the eye behavior analysis unit dynamically switches between at least a first analysis mode and a second analysis mode according to the driving mode of the mobile object, wherein The eye behavior analysis unit In the first analysis mode, analysis is performed at a first frame rate, and In the second analysis mode, analyzing is performed at a second frame rate lower than the first frame rate; Based on the driving mode being a driving mode change preparation mode in which the driving mode of the mobile body is switched from an automatic driving mode to a manual driving mode, the eye behavior analysis unit performs analysis in the first analysis mode; as well as A determination unit, which determines the driver's recovery correspondence level to manual driving based on an analysis result of the driver's eye behavior in the first analysis mode, wherein the determination unit determines the recovery correspondence level by comparing the analysis result of the driver's eye behavior in the first analysis mode with a previously acquired analysis result of the driver's eye behavior. 2 . The information processing device according to claim 1 , wherein the eye behavior includes at least one of saccade, gaze or microsaccade of the eye. 3 . The information processing device according to claim 1 , wherein a starting point of the first analysis mode is determined based on at least one of a journey, a local dynamic map, and a driver's status information.

4. The information processing device according to claim 1, further comprising: A learning device generates a database for the determination by learning previously acquired analysis results of the driver's eye behavior.

5. The information processing device according to claim 1, further comprising: A vehicle driving control unit switches a driving mode of the vehicle based on a determination result of the restoration corresponding level. 6 . The information processing device according to claim 1 , wherein when the driving mode of the mobile body is an automatic driving mode, the eye behavior analysis unit performs analysis in the second analysis mode.

7. An information processing device according to claim 6, wherein in the automatic driving mode, the eye behavior analysis unit dynamically switches the analysis frequency according to the automatic driving level. 8 . The information processing device according to claim 1 , wherein when the driving mode of the mobile body is a manual driving mode, the eye behavior analysis unit performs analysis in the first analysis mode or the second analysis mode. 9 . The information processing device according to claim 1 , wherein the eye behavior analysis unit performs analysis in the first analysis mode according to a decrease in the driver's arousal level.

10. An information processing system comprising: An eye behavior analysis unit, the eye behavior analysis unit analyzes the eye behavior of a driver driving a mobile object, wherein the eye behavior analysis unit dynamically switches between at least a first analysis mode and a second analysis mode according to the driving mode of the mobile object, wherein The eye behavior analysis unit In the first analysis mode, analysis is performed at a first frame rate, and In the second analysis mode, analyzing is performed at a second frame rate lower than the first frame rate; Based on the driving mode being a driving mode change preparation mode in which the driving mode of the mobile body is switched from an automatic driving mode to a manual driving mode, the eye behavior analysis unit performs analysis in the first analysis mode; as well as A determination unit, which determines the driver's recovery correspondence level to manual driving based on an analysis result of the driver's eye behavior in the first analysis mode, wherein the determination unit determines the recovery correspondence level by comparing the analysis result of the driver's eye behavior in the first analysis mode with a previously acquired analysis result of the driver's eye behavior.

11. The information processing system according to claim 10, further comprising: A monitoring unit monitors the eye behavior of the driver.

12. An information processing method, comprising: Analyzing the eye behavior of a driver driving the mobile object via an eye behavior analysis unit; dynamically switching the analysis mode of the eye behavior analysis unit between at least a first analysis mode and a second analysis mode according to the driving mode of the mobile body; In the first analysis mode, analyzing is performed at a first frame rate; In the second analysis mode, analyzing is performed at a second frame rate lower than the first frame rate; performing analysis in the first analysis mode based on the driving mode being a driving mode change preparation mode in which the driving mode of the mobile body is switched from an automatic driving mode to a manual driving mode; as well as Based on the analysis results of the driver's eye behavior in the first analysis mode, the driver's recovery correspondence level to manual driving is determined, wherein the recovery correspondence level is determined by comparing the analysis results of the driver's eye behavior in the first analysis mode with the analysis results of the driver's eye behavior previously acquired.

13. A computer program product, comprising an information processing program, wherein the information processing program causes a computer to perform the following operations: Analyze the eye behavior of drivers driving mobile objects; dynamically switching an analysis mode for analyzing eyeball behavior between at least a first analysis mode and a second analysis mode according to a driving mode of the mobile body; In the first analysis mode, analyzing is performed at a first frame rate; In the second analysis mode, analyzing is performed at a second frame rate lower than the first frame rate; performing analysis in the first analysis mode based on the driving mode being a driving mode change preparation mode in which the driving mode of the mobile body is switched from an automatic driving mode to a manual driving mode; as well as Based on the analysis results of the driver's eye behavior in the first analysis mode, the driver's recovery correspondence level to manual driving is determined, wherein the recovery correspondence level is determined by comparing the analysis results of the driver's eye behavior in the first analysis mode with the analysis results of the driver's eye behavior previously acquired.

Images