Driving status determination device, driving support system, and driving status determination program
The driving state determination device enhances accuracy in detecting driver conditions by focusing on specific scenarios, improving the precision of distracted driving assessments.
Patent Information
- Application Number
- JP2025077521
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-19
AI Technical Summary
Existing technologies for determining a vehicle driver's condition, such as distracted driving, lack accuracy in various driving scenarios, leading to decreased state determination precision.
A driving state determination device that includes a controller to detect driving ability and alertness level, focusing on specific scenarios where accuracy is high, and determines the driver's state based on these factors.
Improves the accuracy of determining a driver's state, particularly in scenarios where detection is challenging, by enhancing the precision of distracted driving assessments.
Smart Images

Figure 2026181812000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a driving state determination device, a driving support system, and a driving state determination program. [Background technology]
[0002] Technologies have been proposed to determine and detect the state of a vehicle driver (see Patent Documents 1 and 2 below). For example, technologies have been proposed to detect a driver's distracted state (inattentive driving state) and issue warnings. [Prior art documents] [Patent Documents]
[0003] [Patent Document 1] Patent No. 7578021 [Patent Document 2] Patent No. 6324262 [Overview of the Initiative] [Problems that the invention aims to solve]
[0004] These technologies require high accuracy in determining the driver's condition.
[0005] The present invention aims to improve the accuracy of determining the driver's condition in a vehicle. [Means for solving the problem]
[0006] The driving state determination device according to the present invention is a driving state determination device for determining the state of a vehicle driver, and includes a controller, the controller detects the driver's driving ability when the driving scene of the vehicle corresponds to a detection target scene, and determines which of a plurality of states the driver's state corresponds to based on the detection result of the driving ability and the driver's level of alertness. The plurality of states include a first state and a second state in which the driver's degree of concentration on driving is lower than that of the first state. [Effects of the Invention]
[0007] Based on the detection results of the user's driving ability and the estimation results of their alertness level, it is possible to determine the user's state (for example, whether the user's state corresponds to a state of distracted driving). However, there are various driving scenarios, and it is not easy to achieve high accuracy in detecting driving ability in all scenarios. If driving ability is detected in driving scenarios where it is difficult to improve the accuracy of detection, and the user's state is determined based on the results, it will lead to a decrease in the accuracy of the state determination. Taking this into consideration, the controller according to the present invention detects driving ability when the vehicle's driving scenario corresponds to a specific scenario (detection target scenario), and then determines the user's state based on the detection results of driving ability and the estimation results of their alertness level. By setting scenarios where it is easy to improve the accuracy of driving ability detection as specific scenarios (detection target scenarios), the accuracy of the user's state determination (for example, the accuracy of determining a state of distracted driving) can be improved. [Brief explanation of the drawing]
[0008] [Figure 1] This is a schematic overall configuration diagram of a driver assistance system according to an embodiment of the present invention. [Figure 2] This is a functional block diagram of a controller in a driver assistance device according to an embodiment of the present invention. [Figure 3] This is an explanatory diagram relating to an embodiment of the present invention and concerning the estimation of the level of arousal. [Figure 4] This is an explanatory diagram illustrating the determination of the user's state as a driver, relating to an embodiment of the present invention. [Figure 5] This is an operation flowchart of a controller in a driver assistance device, relating to Embodiment EX1 of the present invention. [Figure 6] This diagram illustrates multiple types of operating scenes, relating to Example EX1, which is an embodiment of the present invention. [Figure 7] This diagram illustrates the determination of the user's state as a driver, relating to Embodiment EX1, which is an embodiment of the present invention. [Figure 8]This diagram relates to Embodiment EX2 of the present invention and shows the correspondence between multiple types of driving scenes and multiple detection algorithms. [Figure 9] This is an operation flowchart of a controller in a driver assistance device, relating to Embodiment EX2 of the present invention. [Figure 10] This diagram relates to Example EX2, which is an embodiment of the present invention, and shows the internal configuration of the driving capacity detection unit. [Figure 11] This is an operation flowchart of a controller in a driver assistance device, relating to Embodiment EX3 of the present invention. [Figure 12] This diagram relates to Embodiment EX3, which is an embodiment of the present invention, and shows the relationship between the intersection of interest and the vehicle. [Figure 13] This diagram relates to Embodiment EX3, which is an embodiment of the present invention, and shows the configuration of a traffic signal. [Modes for carrying out the invention]
[0009] Hereinafter, examples of embodiments of the present invention will be specifically described with reference to the drawings. In each of the referenced drawings, the same parts are denoted by the same reference numerals, and redundant descriptions relating to the same parts are omitted as a general rule. In addition, in this specification, for the sake of simplification of the description, symbols or reference numerals that refer to information, signals, physical quantities, functional parts, circuits, elements, or components may be indicated, and the names of the information, signals, physical quantities, functional parts, circuits, elements, or components corresponding to such symbols or reference numerals may be omitted or abbreviated.
[0010] Figure 1 shows a schematic overall configuration of the driver assistance system SYS according to an embodiment of the present invention. The driver assistance system SYS comprises a driver assistance device 1, an HMI 2, a vehicle state detection unit 3, a surrounding situation detection unit 4, and an in-vehicle camera 5. The driver assistance device 1, HMI 2, vehicle state detection unit 3, surrounding situation detection unit 4, and in-vehicle camera 5 are arranged in the vehicle V1 (installed inside the vehicle V1). The driver assistance device 1 is mounted in the vehicle V1 and used by user U1. In this embodiment, the idea that the HMI 2 is provided outside the driver assistance device 1 is adopted, but it may also be understood that the HMI 2 is provided inside the driver assistance device 1. The driver assistance device 1 may also be an information terminal device (hereinafter referred to as the user terminal device) brought into the vehicle V1 by user U1. The user terminal device may be a smartphone or a tablet terminal. The HMI 2 may also be provided in the user terminal device.
[0011] Vehicle V1 is any type of vehicle. Here, Vehicle V1 is assumed to be an automobile or similar vehicle that travels on the road surface. User U1 is the driver who operates Vehicle V1. Hereafter, when simply referred to as "driver," it refers to the driver of Vehicle V1 (and therefore User U1). The direction from the driver's seat of Vehicle V1 toward the steering wheel is defined as "forward," and the direction from the steering wheel of Vehicle V1 toward the driver's seat is defined as "rear." The direction perpendicular to the front-rear direction and parallel to the road surface is defined as the left-right direction. In Vehicle V1, left refers to the left side as seen from the perspective of a driver sitting in the driver's seat facing forward, and right refers to the right side as seen from the perspective of a driver sitting in the driver's seat facing forward.
[0012] The driver assistance device 1, HMI 2, vehicle status detection unit 3, surrounding situation detection unit 4, and in-vehicle camera 5 are connected to an in-vehicle network formed within the vehicle V1. The driver assistance device 1 can communicate bidirectionally or unidirectionally with each of the HMI 2, vehicle status detection unit 3, surrounding situation detection unit 4, and in-vehicle camera 5 via the in-vehicle network. The in-vehicle network includes, for example, CAN (Controller Area Network) and AVCLAN (Audio Visual Communication Local Area Network).
[0013] The driver assistance device 1 is an information processing device equipped with a controller 11, a memory 12, and a communication unit 13.
[0014] The controller 11 is equipped with a processing unit as a hardware resource, including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). This processing unit may include an NPU (Neural Processing Unit). The controller 11 may implement any functions, operations, and processes to be realized by the controller 11 by executing a program recorded in memory 12 or any other recording medium (not shown). All or part of the operations mainly performed by the controller 11 shown below may be understood as operations performed by the processing unit within the controller 11.
[0015] Memory 12 is comprised of non-volatile memory such as ROM (Read-only memory) or flash memory, and volatile memory such as RAM (Random Access Memory). Memory 12 stores various data referenced by the controller 11, as well as various programs to be executed by the controller 11.
[0016] The communication unit 13 is a communication circuit (communication module) that transmits and receives arbitrary signals between the driver assistance device 1 and a different other device. The communication unit 13 may be a communication circuit (communication module) located outside the driver assistance device 1. The communication circuit located outside the driver assistance device 1 may be a device installed in the vehicle V1 and may be a communication circuit shared by the driver assistance device 1 and other devices other than the driver assistance device 1. The communication unit 13 may also be a communication circuit (communication module) located in the user terminal device. The other device for the communication unit 13 includes any device connected to the in-vehicle network and the external network. The external network includes mobile communication networks, the internet, and intranets. The controller 11 can transmit and receive arbitrary information with the other device using the communication unit 13, but the description of the communication unit 13 may be omitted below.
[0017] HMI2 is a Human-Machine Interface, an interface for mutual information transmission between the driver assistance device 1 and the occupant of the vehicle V1 (at least user U1). User U1 and the driver assistance device 1 can mutually transmit any information through HMI2. HMI2 includes a microphone 21, a speaker 22, a display device 23, and a vibration device 24.
[0018] Microphone 21 picks up ambient sounds around its installation location and converts them into an electrical signal, which is an audio signal. Microphone 21 is installed in an appropriate location inside the vehicle V1 so that the speech sounds of each occupant (at least user U1) of vehicle V1 are included in the sound picked up by microphone 21. Multiple microphones 21 may be installed inside the vehicle V1. Microphone 21 may also be a microphone provided on a user terminal device.
[0019] Speaker 22 outputs any sound (message, warning sound, music, etc.) under the control of the driver assistance device 1 or an audio device (not shown). Speaker 22 is installed in an appropriate location in the passenger compartment of vehicle V1 so that each occupant (at least user U1) of vehicle V1 can hear the output sound of speaker 22. Multiple speakers 22 may be installed in the passenger compartment of vehicle V1. Speaker 22 may also be a speaker provided in a user terminal device.
[0020] The display device 23 has a display screen such as a liquid crystal display panel and displays any image (video) according to the control of the driver assistance device 1 or a display control device (not shown). The display device 23 is installed in an appropriate place in the passenger compartment of the vehicle V1 so that each occupant of the vehicle V1 (at least user U1) can see the display content of the display device 23. Multiple display devices 23 may be installed in the passenger compartment of the vehicle V1. The display device 23 may be a head-up display. The display device 23 may be a display device provided on a user terminal device.
[0021] The vibration device 24 is installed in the driver's seat of the vehicle V1 and vibrates the driver's seat under the control of the driver assistance device 1. The driver's seat is the seat in which the driver of the vehicle V1 (and therefore user U1) should sit. Therefore, the vibration device 24 can apply vibrations to user U1 by vibrating the driver's seat.
[0022] User U1 can input arbitrary information to the driver assistance device 1 through the microphone 21 by speaking. The content of User U1's speech is picked up by the microphone 21, and an audio signal (hereinafter referred to as the speech audio signal) indicating the content of User U1's speech is generated by the microphone 21. The speech audio signal is supplied from the microphone 21 to the driver assistance device 1, thereby inputting information from User U1's speech into the driver assistance device 1. The driver assistance device 1 can output (transmit) arbitrary information to User U1 as sound using the speaker 22, and can output (transmit) arbitrary information to User U1 as video using the display device 23. In addition, the driver assistance device 1 can transmit information associated with vibration to User U1 by applying vibration to User U1 using the vibration device 24. The display device 23 is equipped with a touch panel. User U1 may also input arbitrary information to the driver assistance device 1 by operating the touch panel. In addition, user U1 may input any information to the driving support device 1 via any operating member (keyboard, mechanical switch) that may be provided on the HMI2.
[0023] The vehicle state detection unit 3 has multiple sensors that detect various states of the vehicle V1, and generates vehicle state information indicating the detection results of each sensor and supplies it to the driver assistance device 1. The multiple sensors provided in the vehicle state detection unit 3 include sensors 31 to 38. Sensors 31, 32, 33, and 34 are the accelerator pedal sensor, brake pedal sensor, steering wheel sensor, and turn signal sensor, respectively. The vehicle V1 is equipped with driver operation components that receive driving operations from the driver (user U1), and these driver operation components include the accelerator pedal, brake pedal, steering wheel, and turn signal control unit. Based on the driver's operations on the accelerator pedal and brake pedal, the speed of the vehicle V1 is adjusted through acceleration or deceleration, and based on the driver's operations on the steering wheel, the steering angle of the vehicle V1 is adjusted. The turn signal control unit has the form of a lever or switch. Based on the driver's operations on the turn signal control unit, the illumination or de-illumination of the turn signals provided on the vehicle V1 is controlled. The turn signals provided on vehicle V1 include a right turn signal and a left turn signal. Sensors 35, 36, 37, and 38 are a vehicle speed sensor, steering angle sensor, G sensor, and GPS sensor, respectively. Sensors other than sensors 31-38 (e.g., a shift lever sensor and a door lock sensor) are also provided on the vehicle state detection unit 3. The vehicle state information includes information generated by each sensor provided on the vehicle state detection unit 3.
[0024] The accelerator pedal sensor 31 detects the operation of the accelerator pedal of vehicle V1 by the driver of vehicle V1 and generates and outputs accelerator pedal operation information indicating the operation of the accelerator pedal. The brake pedal sensor 32 detects the operation of the brake pedal of vehicle V1 by the driver of vehicle V1 and generates and outputs brake pedal operation information indicating the operation of the brake pedal. The steering wheel sensor 33 detects the operation of the steering wheel of vehicle V1 by the driver of vehicle V1 and generates and outputs steering wheel operation information indicating the operation of the steering wheel. The turn signal sensor 34 detects the operation of the turn signal control unit of vehicle V1 by the driver of vehicle V1 and generates and outputs turn signal operation information indicating the operation of the turn signal control unit. The vehicle status information supplied to the driver assistance device 1 includes accelerator pedal operation information, brake pedal operation information, steering wheel operation information, and turn signal operation information. Each of the accelerator pedal operation information, brake pedal operation information, steering wheel operation information, and turn signal operation information is a component of the driving operation information. Driving operation information indicates the details of the driving operations performed by the driver (user U1) on the vehicle V1.
[0025] The vehicle speed sensor 35 detects the speed of the vehicle V1 and generates and outputs vehicle speed information (vehicle speed pulse) representing the detected speed. The steering angle sensor 36 detects the steering angle of the vehicle V1 and generates and outputs steering angle information representing the detected steering angle. The G sensor 37 detects the acceleration applied to the vehicle V1 in a predetermined axial direction and generates and outputs the acceleration detection result as acceleration information. The G sensor 37 may detect acceleration in two mutually orthogonal axial directions, or acceleration in three mutually orthogonal axial directions. The GPS sensor 38 receives signals from multiple GPS satellites forming a GPS (Global Positioning System) and generates and outputs vehicle position information based on the reception results. The vehicle position information generated by the GPS sensor 38 represents the current location of the vehicle V1 by longitude and latitude, or represents the current location of the vehicle V1 by longitude, latitude and altitude. The vehicle status information supplied to the driving support device 1 includes vehicle speed information, steering angle information, acceleration information, and vehicle position information.
[0026] The surrounding conditions detection unit 4 detects the surrounding conditions of the vehicle V1 and generates surrounding conditions information indicating the detection results of those conditions. The surrounding conditions information generated by the surrounding conditions detection unit 4 is supplied to the driver assistance device 1. The surrounding conditions detection unit 4 includes an external camera 41, a distance measuring sensor 42, an illuminance sensor 43, and a rainfall sensor 44. The external image information, distance measuring information, illuminance information, and rainfall information described later are included in the surrounding conditions information.
[0027] The external camera 41 consists of one or more cameras that capture the outside of the vehicle V1. The external camera 41 has a shooting area set outside the vehicle V1, and generates an external camera image by capturing the scene within the shooting area. The external camera image is an image of the shooting area captured by the external camera 41. Image information indicating the external camera image is called external image information. The external camera 41 takes pictures at a predetermined frame rate. The external camera 41 may consist of a front camera, a rear camera, a left camera, and a right camera. The front camera has a shooting area set outside the vehicle V1 and on the front side of the vehicle V1. The rear camera has a shooting area set outside the vehicle V1 and on the rear side of the vehicle V1. The right camera has a shooting area set outside the vehicle V1 and on the right side of the vehicle V1. The left camera has a shooting area set outside the vehicle V1 and on the left side of the vehicle V1.
[0028] The distance measuring sensor 42 generates and outputs distance measurement information by measuring distance. In the distance measuring sensor 42, the distance between the vehicle V1 and a three-dimensional object located within the distance measuring area around the vehicle V1 is detected, as well as the orientation of the object relative to the vehicle V1. These detection results are included in the distance measurement information. The distance measuring sensor 42 may be composed of a LIDAR (Light Detection and Ranging) that uses light to measure distance, or it may be composed of a radar that uses radio waves to measure distance. The distance measuring sensor 42 may also be composed of a combination of LIDAR and radar.
[0029] The illuminance sensor 43 detects the illuminance outside and inside the vehicle V1, and generates and outputs illuminance information representing the detected illuminance. The rainfall sensor 44 detects whether it is raining outside the vehicle V1, and generates and outputs rainfall information representing the detection result. The rainfall sensor 44 may also detect whether it is raining by detecting whether water droplets are adhering to the windshield of the vehicle V1. In addition, a temperature sensor and a road surface freezing detection sensor may also be included in the surrounding conditions detection unit 4.
[0030] The in-vehicle camera 5 consists of one or more cameras that capture images of the interior of the vehicle V1. The in-vehicle camera 5 has a shooting area set inside the vehicle V1 (i.e., the passenger compartment of the vehicle V1), and generates an in-vehicle camera image by capturing images of the shooting area. The in-vehicle camera image is an image of the shooting area captured by the in-vehicle camera 5. Image information representing the in-vehicle camera image is called in-vehicle image information. The in-vehicle camera 5 takes pictures at a predetermined frame rate. The in-vehicle image information obtained from each shot taken by the in-vehicle camera 5 is sequentially supplied from the in-vehicle camera 5 to the driver assistance device 1. User U1 is located within the shooting area of the in-vehicle camera 5, and therefore the in-vehicle camera image includes an image of user U1. User U1's face is included within the shooting area of the in-vehicle camera 5, and therefore the in-vehicle camera image includes an image of user U1's face (an image of user U1's face).
[0031] Figure 2 shows a functional block diagram of the controller 11. The controller 11 comprises functional blocks 111 to 115. Each of the functions of functional blocks 111 to 115 may be realized by executing a program recorded in memory 12 or any other recording medium (not shown) on the controller 11. Functional blocks 111, 112, 113, 114, and 115 are the alertness estimation unit, the driving ability detection unit, the scene determination unit, the user state determination unit, and the corresponding processing unit, respectively.
[0032] [Awakening level estimation unit 111] The arousal level estimation unit 111 will now be described. The arousal level estimation unit 111 performs an arousal level estimation process. In the arousal level estimation process, the arousal level estimation unit 111 estimates the arousal level of user U1 based on the arousal level estimation information J1. The arousal level estimation information J1 may be abbreviated as information J1 below. In the following description, arousal level refers to the arousal level of user U1 unless otherwise specified. The arousal level of user U1 is the arousal level of the central nervous system in user U1. Information J1 is information that correlates with the arousal level of user U1. Several methods are known for estimating the arousal level of a person based on information that correlates with the arousal level of that person. The arousal level estimation unit 111 may estimate the arousal level of user U1 based on information J1 in accordance with any of these known methods.
[0033] For example, information J1 may be electroencephalogram (EEG) data of user U1, in which case a known method for estimating a person's level of alertness based on their EEG data can be used. Specifically, an EEG sensor can be attached to user U1, and the EEG sensor attached to user U1 detects user U1's brainwaves and outputs EEG data indicating the detection result of the brainwaves. The EEG data includes information about the detected brainwaves. For example, a headgear-type EEG sensor can be used as the EEG sensor. When information J1 is EEG data, user U1's level of alertness is derived based on the β / α wave ratio of user U1's brainwaves. That is, a value corresponding to the β / α wave ratio in user U1's brainwaves (for example, a value proportional to the β / α wave ratio of the brainwaves) can be derived as user U1's level of alertness. In user U1, the β / α wave ratio of the brainwaves is the ratio of the β wave to the α wave of the brainwaves. More specifically, for example, the beta / alpha waves of the electroencephalogram (EEG) in user U1 are “INT_β1 / INT_α1” or “PW_β1 / PW_α1”. INT_β1 and INT_α1 represent the intensity of the beta and alpha waves of the EEG in user U1, respectively. PW_β1 and PW_α1 represent the power of the beta and alpha waves of the EEG in user U1, respectively. EEG data is an example of biometric information that can be used as information J1. The biometric information used as information J1 may also be heart rate data representing user U1's heart rate or body temperature data representing user U1's body temperature. Two or more data from user U1's EEG data, heart rate data, and body temperature data may be used as information J1.
[0034] Alternatively, for example, information J1 may be a facial image of user U1, in which case a known method for estimating a person's level of alertness based on a facial image can be used. Estimating the level of alertness using user U1's facial image eliminates the need to attach sensors to user U1 to detect brainwave data, etc., thus offering greater convenience. As described above, the in-car camera images generated by the in-car camera 5 include a facial image of user U1, which is a captured image of user U1's face. The in-car camera 5 sequentially captures images at a predetermined frame rate, and the in-car camera images are periodically supplied to the alertness estimation unit 111 at intervals that are the reciprocal of the frame rate of the in-car camera 5. The alertness estimation unit 111 may extract user U1's facial image from each in-car camera image, or a facial image extraction unit (not shown) responsible for such extraction may be provided separately in the controller 11 from the alertness estimation unit 111. If the facial image of user U1 is information J1, an estimation AI model that has been trained by machine learning to recognize the correlation between a person's facial image and their level of alertness can be created and installed in the alertness estimation unit 111. In this case, the alertness estimation unit 111 can use the estimation AI model to estimate the alertness of user U1 from the facial image of user U1. In this embodiment, AI is an abbreviation for Artificial Intelligence.
[0035] Furthermore, for example, information J1 may be dialogue information between user U1 and the agent. The dialogue information includes at least the content of the dialogue between user U1 and the agent, and may include the tempo of the dialogue, etc. The agent is a software agent formed by the controller 11, and is a pseudo-personality having artificial intelligence. The agent may have a chatbot equipped with a generation AI. The agent can communicate with user U1 through dialogue and can also autonomously operate equipment in the vehicle V1, etc. Dialogue between user U1 and the agent is realized through HMI2. A method for estimating the cognitive ability of a first person or a second person based on dialogue information (content and tempo of the dialogue, etc.) between a first person and a second person is known, and this method can be used in the arousal level estimation unit 111. The arousal level estimation unit 111 may then estimate user U1's arousal level by estimating user U1's cognitive ability based on the dialogue information between user U1 and the agent. In this case, the arousal level estimation unit 111 can estimate that the higher user U1's cognitive ability, the higher user U1's arousal level.
[0036] The alertness level estimation unit 111 periodically estimates the alertness level of user U1 and periodically outputs data indicating the alertness level derived from the estimation as alertness level data AL. The alertness level data AL is numerical data, and the higher the estimated degree of alertness of user U1, the higher the value of the alertness level data AL.
[0037] Referring to Figures 3(a) to 3(c), the periodic derivation and output of the arousal level data AL will be explained. The arousal level estimation unit 111 has multiple output timings t that are discretely located on the time series. AL The i-th output timing t is defined and set. AL In particular, the output timing t AL [i] is denoted as such. i represents any integer. Output timing t AL The arousal level data AL output from the arousal level estimation unit 111 in [i] is specifically referred to as arousal level data AL[i]. Interval INT AL This refers to two adjacent output timings t ALis an interval therebetween, and has a predetermined time length. As shown in Fig. 3(a), the arousal level estimation unit 111 sets the interval INT AL Every time the time of minutes elapses, the latest arousal level data AL is output. In Fig. 3(b), the arousal level data AL output from the arousal level estimation unit 111 is shown at the interval INT AL The state of being updated at a cycle of minutes is shown.
[0038] In the arousal level estimation unit 111, for each output timing t AL An estimation period is defined and set in association with. Output timing t AL The estimation period corresponding to [i] is the i-th estimation period. As shown in Fig. 3(c), the i-th estimation period is a period having a length of a fixed estimation time, and is the output timing t AL A predetermined minute time Δt from [i] AL It is a period that ends at a timing only earlier than that. For each estimation period, the arousal level estimation unit 111 derives the arousal level data AL for the estimation period based on the information J1 in the estimation period. Therefore, the arousal level estimation unit 111 derives the arousal level data AL[i] based on the information J1 in the i-th estimation period. When the information J1 is the face image of the user U1, the arousal level estimation unit 111, based on the face image of the user U1 in the i-th estimation period (specifically, based on the face image of the user U1 obtained by photographing with the in-vehicle camera 5 in the i-th estimation period), derives the arousal level data AL[i]. The (i+1)-th estimation period is a period later than the i-th estimation period. The (i+1)-th estimation period may be started after the end of the i-th estimation period. Alternatively, the (i+1)-th estimation period may be started after the start of the i-th estimation period and before the end of the i-th estimation period. That is, the i-th estimation period and the (i+1)-th estimation period may partially overlap.
[0039] [Driving capability detection unit 112] The driving ability detection unit 112 will now be described. The driving ability detection unit 112 performs driving ability detection processing. In the driving ability detection processing, the driving ability detection unit 112 detects the driving ability of user U1 based on the driving ability detection information J2, and generates and outputs driving ability data DA that indicates the detected driving ability. User U1's driving ability is an indicator that represents the level of user U1's ability to drive vehicle V1, and is the overall ability of user U1 to judge the situation, make decisions, and operate the vehicle V1. In the following description, driving ability refers to user U1's driving ability unless otherwise specified. Driving ability may be read as driving performance. The driving ability data DA is numerical data, and the higher user U1's driving ability (i.e., the higher the detected ability of user U1 to drive vehicle V1), the higher the value of the driving ability data DA. The driving ability detection unit 112 detects user U1's driving ability based on the driving ability detection information J2 according to a predetermined detection algorithm, and derives the driving ability data DA accordingly.
[0040] The driving ability detection information J2 may be abbreviated as "information J2" below. Information J2 includes vehicle status information output from the vehicle status detection unit 3 (see Figure 1). The vehicle status information included in information J2 includes all or part of the accelerator pedal operation information, brake pedal operation information, steering wheel operation information, turn signal operation information, vehicle speed information, steering angle information, acceleration information, and vehicle position information shown in Figure 1.
[0041] Information J2 also includes gaze information. Gaze information refers to the gaze information of user U1. Multiple functional blocks provided in the controller 11 include a gaze detection unit (not shown) that performs gaze detection processing. In the gaze detection processing, the gaze detection unit detects the gaze of user U1 based on the in-vehicle camera image and generates gaze information that indicates the gaze detection result. The gaze information indicates where user U1 is looking (i.e., the direction of user U1's gaze) and the movement of user U1's gaze. Known gaze detection techniques for detecting a person's gaze can be used in the gaze detection processing. The gaze detection unit may be included in the driving ability detection unit 112, in which case the driving ability detection unit 112 obtains the gaze information in information J2 by performing gaze detection processing on the in-vehicle camera image. A dedicated gaze detection camera for detecting user U1's gaze may be provided as a component of the in-vehicle camera 5 or separately in the driving support system SYS. In this case, the gaze detection camera can acquire the eye image of user U1, and gaze detection processing can be performed based on the eye image of user U1.
[0042] Information J2 may also include information other than vehicle status information and gaze information, for example, surrounding situation information output from the surrounding situation detection unit 4 (see Figure 1). Specifically, for example, the driving ability detection unit 112 can change the driving ability data DA according to the degree of vehicle V1 swaying, and in this case, the degree of vehicle V1 swaying can be detected based on external image information (see Figure 1) while utilizing known white line detection technology, etc.
[0043] [Scene determination unit 113] The scene determination unit 113 will now be described. The scene determination unit 113 performs scene determination processing. In the scene determination processing, the scene determination unit 113 determines the driving scene of vehicle V1 based on the scene determination information J3. The scene determination information J3 may be abbreviated as information J3 below. There are multiple scenes (hereinafter referred to as multiple candidate scenes) to which the driving scene of vehicle V1 may be. In the scene determination processing, the scene determination unit 113 determines, based on information J3, which of the multiple candidate scenes the current driving scene of vehicle V1 corresponds to. The scene determination unit 113 outputs the result of the scene determination processing as scene determination result information J4 to the driving ability detection unit 112. The scene determination result information J4 indicates which of the multiple candidate scenes the current driving scene of vehicle V1 corresponds to. Specific examples of each candidate scene will be described later.
[0044] Information J3 includes vehicle status information output from the vehicle status detection unit 3 and surrounding situation information output from the surrounding situation detection unit 4 (see Figure 1), and also includes map information. The map information is electronic map data showing geographical information of each point within the area that the vehicle V1 can travel. The map information may be stored in memory 12. Alternatively, the map information may be provided to the driver assistance device 1 from an external device (not shown) connected to an external network outside the vehicle.
[0045] In addition, information J3 may include various other types of information. For example, information J3 may further include traffic congestion information and weather information, and may further include navigation information. Based on vehicle position information and map information, the controller 11 can perform navigation operations to provide user U1 with driving guidance to guide vehicle V1 to its destination. In the navigation operation, a driving navigation route (driving guidance route) to the destination of vehicle V1 is set, and the direction in which vehicle V1 should proceed along the driving navigation route is sequentially notified to user U1 by displaying an image and outputting sound. The display of an image refers to the display of an image on the display device 23, and the output of sound refers to the output of sound from the speaker 22. Navigation information that can be included in information J3 includes information indicating the driving navigation route.
[0046] Various scene determination methods are known for determining the driving scene of a vehicle, and the scene determination unit 113 may perform scene determination processing using any known scene determination method based on the information J3. As will be described in detail later, the driving ability detection unit 112 controls the execution or non-execution of the driving ability detection process according to the scene determination result information J4. In general terms, the driving ability detection unit 112 can perform control such as executing the driving ability detection process only when the driving scene of the vehicle V1 corresponds to a scene (detection target scene) in which the driving ability of user U1 can be detected with relatively high accuracy. The scene determination result information J4 may be abbreviated as information J4 below.
[0047] [User status determination unit 114] The user state determination unit 114 will now be described. The user state determination unit 114 receives the alertness data AL output from the alertness estimation unit 111 and the driving ability data DA output from the driving ability detection unit 112 as input. The user state determination unit 114 performs user state determination processing. In the user state determination processing, the user state determination unit 114 determines the state of user U1 based on the alertness data AL and the driving ability data DA. There are multiple candidate states as candidates for the state of user U1. In the user state determination processing, the user state determination unit 114 determines which of the multiple candidate states the state of user U1 corresponds to based on the alertness data AL and the driving ability data DA.
[0048] The total number of candidate states can be any number of two or more. Here, as an example, the multiple candidate states are the four states STa, STb1, STb2, and STb3 shown in Figure 4. The user state determination unit 114 can binarize the alertness data AL and the driving ability data DA, and based on the result of this binarization, it can determine which of the states STa, STb1, STb2, and STb3 the user U1's state corresponds to. Binarization of the alertness data AL distinguishes whether the alertness data AL is relatively large or relatively small, and binarization of the driving ability data DA distinguishes whether the driving ability data DA is relatively large or relatively small. If both the alertness data AL and the driving ability data DA are relatively large, the user U1's state is determined to correspond to state STa. If both the alertness data AL and the driving ability data DA are relatively large, the user U1's state is determined to correspond to state STb1. If the alertness data AL is relatively low and the driving ability data DA is relatively high, the user U1's state is determined to be state STb2. If the alertness data AL is relatively low and the driving ability data DA is also relatively low, the user U1's state is determined to be state STb3.
[0049] State STa is a state in which user U1 is driving vehicle V1 safely while concentrating on driving operations (normal driving state). States STb1, STb2, and STb3 are states in which user U1 is driving vehicle V1 while drowsy or while thinking about something else. In any case, the degree of user U1's concentration on driving is higher in state STa than in states STb1, STb2, and STb3. That is, in states STb1, STb2, and STb3, the degree of user U1's concentration on driving is lower than in state STa. States STb1, STb2, and STb3 correspond to distracted driving states. State STa does not correspond to distracted driving states.
[0050] The alertness estimation unit 111 sequentially outputs multiple alertness data AL by repeatedly executing the alertness estimation process at intervals. The driving ability detection unit 112 sequentially outputs multiple driving ability data DA by repeatedly executing the driving ability detection process at intervals. The user state determination unit 114 may perform user state determination processing based on the multiple alertness data AL and the multiple driving ability data DA. In this case, the user state determination unit 114 can determine which of states STa, STb1, STb2, and STb3 the state of user U1 corresponds to based on the multiple alertness data AL and the multiple driving ability data DA (specific examples will be described later). An storage unit (not shown) for storing the multiple alertness data AL and the multiple driving ability data DA is provided within the user state determination unit 114 or prior to the user state determination unit 114.
[0051] Furthermore, the driving ability data DA output from the driving ability detection unit 112 may itself be binary data. In this case, the driving ability detection unit 112 detects whether user U1's driving ability is relatively high or relatively low based on the information J2. When the driving ability detection unit 112 detects that user U1's driving ability is relatively high, it outputs driving ability data DA with a value of "1", and when it detects that user U1's driving ability is relatively low, it outputs driving ability data DA with a value of "0".
[0052] The result of the determination by the user state determination unit 114 is supplied to the corresponding processing unit 115 as state determination data SJ. The state determination data SJ indicates whether the state of user U1 corresponds to state STa, STb1, STb2, or STb3.
[0053] [Corresponding Processing Unit 115] The response processing unit 115 will now be described. The response processing unit 115 can perform necessary response processing, such as distracted driving response processing, according to the state determination data SJ. If the state of user U1 corresponds to state STb1, STb2, or STb3, the response processing unit 115 can execute the first, second, and third distracted driving response processing, respectively. In each distracted driving response processing, the response processing unit 115 issues a warning notification to user U1 using the HMI2. The warning notification may include at least one of the following: outputting a warning sound from the speaker 22 and displaying a warning image on the display device 23. The warning notification may include a notification that applies vibration to user U1 by vibrating the vibration device 24 in a predetermined vibration pattern. The warning notification is a notification that informs user U1 that their state corresponds to a distracted driving state or that there is a risk that user U1's state will correspond to a distracted driving state, and serves to remind user U1 to concentrate on driving operations. Warning notifications can prevent dangerous situations related to the operation of vehicle V1 from occurring. The content of the warning notifications may be the same or different between the first, second, and third distracted driving response processes.
[0054] The distracted driving response process may include processing to control the driving state of vehicle V1. The processing to control the driving state of vehicle V1 may be included in the distracted driving response process only when the state of user U1 corresponds to state STb3. For example, in the distracted driving response process (especially in the distracted driving response process when the state of user U1 corresponds to state STb3), the response processing unit 115 may execute processing to reduce the driving speed of vehicle V1 or to stop vehicle V1 on the shoulder of the road as processing to control the driving state of vehicle V1.
[0055] In this case, the user state determination unit 114 distinguishes between states STb1 to STb3, but it is also possible to treat a combination of states STb1 to STb3 as state STb. In this case, if it is determined that the state of user U1 corresponds to state STb, the corresponding processing unit 115 performs the first, second, or third idle operation response processing described above based on the state determination data SJ indicating that.
[0056] The following describes specific operational examples, application technologies, and modification technologies related to the SYS driver assistance system within several practical examples. Unless otherwise specified and without contradiction, the matters described above apply to each of the following examples. In the event of any inconsistency between the above and the above in any example, the description in the example may take precedence. Furthermore, unless there is a contradiction, the matters described in any of the following examples can be applied to any other example (i.e., any two or more examples from the multiple examples can be combined).
[0057] <<Example EX1>> Example EX1 will now be described. Figure 5 is an operation flowchart of the driver assistance device 1 according to Example EX1. Referring to Figure 5, the operation flow of the driver assistance device 1 according to Example EX1 will be explained.
[0058] Each of the processes in steps S11 to S21 shown in Figure 5 is executed by the controller 11. Step S11 is reached when the driver assistance device 1 is activated. For example, the driver assistance device 1 is activated in conjunction with the start of the engine mounted on the vehicle V1 by operating the ignition switch of the vehicle V1. Alternatively, the driver assistance device 1 is activated when power for driving is supplied to the driver assistance device 1. In step S11, the controller 11 starts sequential execution of the alertness estimation process by the alertness estimation unit 111. Therefore, after reaching step S11, multiple alertness estimation processes are executed sequentially (for example, the alertness estimation process is executed repeatedly in a periodic manner), and the latest alertness data AL is output one after another from the alertness estimation unit 111. The storage unit provided in the controller 11 stores and holds the latest fixed number of alertness data AL. In step S12 following step S11, 1 is assigned to the variable i managed by the controller 11. The variable i has an integer value. Note that the execution order of steps S11 and S12 is arbitrary. After steps S11 and S12, proceed to step S13.
[0059] In step S13, the scene determination unit 113 performs a scene determination process. As described above, in the scene determination process, the scene determination unit 113 determines, based on information J3, which of the multiple candidate scenes the current driving scene of vehicle V1 corresponds to. Of the multiple candidate scenes, one or more will be the detection target scene, and the rest will be the non-detection target scenes. Here, we assume that there are four types of detection target scenes SN1, SN2a, SN2b, and SN2c, and the multiple candidate scenes will be the 1st to 5th candidate scenes. Specifically, as shown in Figure 6, the 1st, 2nd, 3rd, and 4th candidate scenes are the detection target scenes SN1, SN2a, SN2b, and SN2c, respectively, and the 5th candidate scene is the non-detection target scene. Then, in the scene determination process, the scene determination unit 113 determines, based on information J3, which of the detection target scenes SN1, SN2a, SN2b, and SN2c and the non-detection target scene the current driving scene of vehicle V1 corresponds to. When the driving scene of vehicle V1 is determined to correspond to the j-th candidate scene, the scene determination unit 113 outputs scene determination result information J4 with the value "j" to indicate this (see Figure 6). j represents any integer.
[0060] The detection target scene SN1 is when the vehicle V1's speed reaches a predetermined speed SP. H The above conditions are met, and the vehicle V1 is in a straight-ahead driving state at high speed. A straight-ahead driving state is a state in which the vehicle V1 is moving straight forward. However, a state in which the absolute value of the steering angle of the vehicle V1 is maintained at or below a predetermined angle value (e.g., 2°) may also be considered a straight-ahead driving state. Speed SP H This is a speed selected and determined from a speed range of, for example, 40 km / h to 80 km / h, and in this case, it is assumed to be 60 km / h.
[0061] The scene determination unit 113 determines that the speed of vehicle V1 is speed SP H If the above conditions are met and the vehicle V1 remains in a straight-line driving state for a predetermined determination time or longer, it can be determined that the driving scene of vehicle V1 corresponds to the detection target scene SN1. Alternatively, if the speed of vehicle V1 is speed SP HWhen the above conditions are met, and the vehicle V1 is determined to continue driving straight for a certain period of time or longer based on the map information and the driving navigation route, the scene determination unit 113 can determine that the driving scene of the vehicle V1 corresponds to the detected scene SN1.
[0062] The detection target scene SN2a is when the vehicle V1's speed reaches a predetermined speed SP. L The following conditions apply, and the vehicle V1 is traveling towards the intersection while slowing down, in an intersection deceleration scene. For the sake of clarity, any single intersection of interest will be referred to as the "intersection of interest" below. In this case, the detection target scene SN2a is when the speed of vehicle V1 reaches a predetermined speed SP L The following conditions apply, and the driving scene involves vehicle V1 slowing down as it approaches the intersection in question.
[0063] Regarding the detection target scene SN2a, "vehicle V1 is driving while decelerating towards the intersection of interest" can be understood as meaning that vehicle V1 is driving while decelerating in preparation for stopping before the intersection of interest, in accordance with the traffic signals of the traffic signals installed at the intersection of interest. The scene determination unit 113 can determine whether vehicle V1 is decelerating in preparation for stopping before the intersection of interest based on vehicle speed information, map information, and vehicle position information, while also referring to navigation information, turn signal operation information, and the traffic signals mentioned above. A distance condition, such as the distance between vehicle V1 and the intersection of interest being less than or equal to a predetermined distance, may be added as a condition for determining that the driving scene of vehicle V1 corresponds to the detection target scene SN2a. Speed SP L SP is a speed selected and determined from a speed range of, for example, 20 km / h or less, and in this case, it is assumed to be 10 km / h. L The above speed SP H Lower.
[0064] The detection target scene SN2b is when the vehicle V1's speed reaches a predetermined speed SP. L The following is a scene of a vehicle V1 making a right turn at a designated intersection.
[0065] The driving scene of vehicle V1 during the period from the start of the right turn action to the end of the right turn action is the intersection right turn scene. In the process of vehicle V1 traveling toward the intersection of interest, the start of the right turn action is the timing when user U1 inputs an operation to the turn signal control unit to indicate the intention to turn right at the intersection of interest, or a predetermined time before that timing. Subsequently, the end of the right turn action is the timing when the right turn at the intersection of interest is completed and vehicle V1 leaves the intersection of interest, or when the right turn signal of vehicle V1 switches from flashing to unlit. The above distance condition may be added as a condition for determining that the driving scene of vehicle V1 corresponds to the detection target scene SN2b. Based on vehicle speed information, map information, vehicle position information, and turn signal operation information, the scene determination unit 113 can determine whether the driving scene of vehicle V1 corresponds to the detection target scene SN2b, while also referring to navigation information and the above traffic signals.
[0066] The detection target scene SN2c is when the vehicle V1's speed reaches a predetermined speed SP. L The following is a scene of a vehicle V1 making a left turn at a designated intersection.
[0067] The driving scene of vehicle V1 during the period from the start of the left turn action to the end of the left turn action is the intersection left turn scene. In the process of vehicle V1 driving toward the intersection of interest, the start of the left turn action is the timing when user U1 inputs an operation to the turn signal control unit to indicate the intention to turn left at the intersection of interest, or a predetermined time before that timing. Subsequently, the end of the left turn action is the timing when the left turn at the intersection of interest is completed and vehicle V1 leaves the intersection of interest, or when the left turn signal of vehicle V1 switches from flashing to unlit. The above distance condition may be added as a condition for determining that the driving scene of vehicle V1 corresponds to the detection target scene SN2c. Based on vehicle speed information, map information, vehicle position information, and turn signal operation information, the scene determination unit 113 can determine whether the driving scene of vehicle V1 corresponds to the detection target scene SN2c, while also referring to navigation information and the above traffic signals.
[0068] A non-detectable scene is a driving scene that is different from any of the detectable scenes SN1, SN2a, SN2b, and SN2c. When the scene determination unit 113 determines that the current driving scene of vehicle V1 does not correspond to any of the detectable scenes SN1, SN2a, SN2b, and SN2c, it determines that the current driving scene of vehicle V1 corresponds to a non-detectable scene. A non-detectable scene may consist of multiple subdivided scenes. In this case, when the scene determination unit 113 determines that the current driving scene of vehicle V1 does not correspond to any of the detectable scenes SN1, SN2a, SN2b, and SN2c, it may determine which of the multiple subdivided scenes the current driving scene of vehicle V1 corresponds to. However, in this embodiment, since it is important whether or not the driving scene of vehicle V1 is determined to correspond to a detectable scene, the distinction between multiple subdivided scenes is ignored.
[0069] Let's return to the explanation of the operation flowchart in Figure 5. After step S13, proceed to step S14. The driving scene determined by the scene determination process will be referred to as the determined driving scene below. In step S14, the driving capability detection unit 112 checks whether the determined driving scene in step S13 matches any of the detection target scenes SN1, SN2a, SN2b, and SN2c. If the determined driving scene matches any of the detection target scenes SN1, SN2a, SN2b, and SN2c (Y in step S14), proceed to step S15. That is, if the scene determination result information J4 from the scene determination process in step S13 has a value of "1", "2", "3", or "4" (see Figure 6), proceed to step S15. If the determined driving scene is not a detection target scene (N in step S14), return to step S13. That is, if the scene determination result information J4 from the scene determination process in step S13 has a value of "5" (see Figure 6), return to step S13.
[0070] In step S15, the driving ability detection unit 112 generates driving ability data DA by executing a driving ability detection process based on the information J2. The information J2 used in step S15 is the information J2 during the period when the driving scene of vehicle V1 is in the detection target scene SN1, SN2a, SN2b, or SN2c, that is, the information J2 during the period when the driving scene of vehicle V1 corresponds to a high-speed straight driving scene, an intersection deceleration scene, an intersection right turn scene, or an intersection left turn scene.
[0071] For example, consider the first case in which the driving scene determined by the scene determination process in step S13 matches the detected scene SN1. In the first case, the speed of the vehicle V1 is a predetermined speed SP H The above conditions are met, and the information J2 obtained during the first period when vehicle V1 is in a straight-line driving state is used in step S15 to generate driving capability data DA. For example, consider a second case in which the driving scene determined by the scene determination process in step S13 matches the detected target scene SN2a. In the second case, the speed of vehicle V1 is a predetermined speed SP LThe information J2 obtained during the second period when the vehicle V1 is traveling while decelerating towards the intersection is used in step S15 to generate driving capability data DA. For example, consider a third case in which the driving scene determined by the scene determination process in step S13 matches the detected scene SN2b. In the third case, the speed of the vehicle V1 is a predetermined speed SP L The information J2 obtained during the third period when the vehicle V1 is driving to make a right turn at the intersection of interest is used in step S15 to generate driving capability data DA. For example, consider a fourth case in which the driving scene determined by the scene determination process in step S13 matches the detected scene SN2c. In the fourth case, the speed of the vehicle V1 is a predetermined speed SP L The following information J2 is used in step S15 to generate driving capability data DA during the fourth period when vehicle V1 is driving to make a left turn at the intersection of interest.
[0072] The driving capability data DA generated in step S15 is specifically referred to as driving capability data DA[i]. For example, when "i=1", the driving capability data DA[i] generated in step S15 is driving capability data DA[1], and when "i=2", the driving capability data DA[i] generated in step S15 is driving capability data DA[2]. The same applies when "i≧3". After step S15, proceed to step S16.
[0073] In step S16, the controller 11 determines that the variable i is the reference number n A Check if it matches. Reference number n A n has a predetermined positive integer value. A It would also be acceptable if it were =1, but here it is "n A Assume that ≥ 2. A If the condition "i=n" is met (Y in step S16), proceed to step S18. AIf the condition is not met (N in step S16), the process proceeds to step S17. In step S17, the controller 11 adds 1 to the variable i, and then the process returns to step S13. Note that in step S17, the controller 11 may wait for a certain amount of time to elapse before proceeding to step S13. This is to prevent the consecutive generation of two or more driving capability data DAs for a single and common driving scene.
[0074] In step S18, the user state determination unit 114 performs user state determination processing. At the stage of proceeding to step S18, n A The processing of step S15 for each batch results in the operation capacity data DA[1]~DA[n A ] has already been generated. The user state determination process in step S18 is performed using the most recently generated n A Individual driving ability data DA[1]~DA[n A ] and the most recently generated n B Individual Awakening Level Data AL[1]~AL[n B This is done based on the most recently generated n B Individual Awakening Level Data AL[1]~AL[n B ] is the most recent n from the group of arousal level data AL generated by the arousal level estimation unit 111 at the time of execution of the process in step S18. B This is individual arousal level data AL. B n represents the base number. B n has a predetermined positive integer value. B It would also be acceptable if it were =1, but here it is "n B Assume that ≥ 2''.
[0075] In step S18, the user state determination unit 114 determines the driving capacity data DA[1]~DA[n A Each of the values of ] is set to a predetermined operating capacity threshold DA TH It compares with the user state determination unit 114, n A The individual driving ability data DA is the driving ability data DA[1]~DA[n A ] Among them, the driving ability threshold DA TH The number of driving ability data DA with the above values DAGOOD The following are counted, and the driving ability evaluation value DA_EV is derived. The driving ability evaluation value DA_EV is calculated from the driving ability data DA[1]~DA[n] used in the user state determination process. A ] Count DA relative to the total number GOOD This is the ratio. That is, “DA_EV=DA GOOD / n A ". Alternatively, the driving ability evaluation value DA_EV is the driving ability data DA[1]~DA[n A ] Average value (driving ability data DA[1]~DA[n A It may also be the average value of the values that ] possesses. A If = 1, the value of the driving ability data DA[1] itself is set as the driving ability evaluation value DA_EV.
[0076] On the other hand, in step S18, the user state determination unit 114 determines the level of alertness data AL[1]~AL[n B Each of the values of ] is set to a predetermined arousal threshold AL TH It compares with the user state determination unit 114, n B Individual arousal level data AL is arousal level data AL[1]~AL[n B Among them, the arousal threshold AL TH The number of AL (algorithm data) entries with the above values. GOOD The system counts the number of arousal levels and derives the alertness evaluation value AL_EV. The alertness evaluation value AL_EV is calculated using the alertness data AL[1]~AL[n] used in the user state determination process. B ] Number of items relative to the total number AL GOOD This is the ratio. That is, "AL_EV=AL GOOD / n B ". Alternatively, the arousal level evaluation value AL_EV is the arousal level data AL[1]~AL[n B ] Average value (Awakening level data AL[1]~AL[n B It may also be the average value of the values that ] possesses. B If it is =1, the value of the arousal level data AL[1] itself is set to the arousal level evaluation value AL_EV.
[0077] Figure 7 is a conceptual diagram of the determination of the user U1's state by the user state determination process in step S18. In step S18, the user state determination unit 114 determines the driving ability evaluation value DA_EV to a predetermined determination threshold TH. EV1 The arousal level evaluation value AL_EV is compared with a predetermined judgment threshold TH. EV2 It compares with the user state determination unit 114, and the user state determination unit 114 determines that “DA_EV≧TH EV1 " and "AL_EV≧TH EV2 When the condition " is met, it is determined that the state of user U1 corresponds to state STa. The user state determination unit 114 determines "DA_EV <TH EV1 " and "AL_EV≧TH EV2 When the condition " is met, it is determined that the state of user U1 corresponds to state STb1. The user state determination unit 114 determines that "DA_EV≧TH EV1 " and "AL_EV <TH EV2 When the condition " is met, it is determined that the state of user U1 corresponds to state STb2. The user state determination unit 114 determines "DA_EV <TH EV1 " and "AL_EV <TH EV2 When the condition is met, it is determined that the state of user U1 corresponds to state STb3. The result of the user state determination unit 114's determination in step S18 is supplied to the corresponding processing unit 115 as state determination data SJ. After step S18, the process proceeds to step S19.
[0078] In step S19, the corresponding processing unit 115 refers to the state determination data SJ and checks whether the state of user U1 is determined to be one of the states STb1, STb2, or STb3. If the state of user U1 is determined to be one of the states STb1, STb2, or STb3 (Y in step S19), the process proceeds to step S20. If the state of user U1 is determined to be state STa (N in step S19), the process proceeds to step S21 without proceeding to step S20.
[0079] In step S20, the response processing unit 115 executes a distracted driving response process according to the state determination data SJ. If the state determination data SJ indicates that the state of user U1 corresponds to state STb1, STb2, or STb3, the response processing unit 115 in step S20 executes the first, second, and third distracted driving response processes described above, respectively. In step S21, the controller 11 assigns "1" to the variable i, and then the process returns to step S13.
[0080] Furthermore, in the flowchart of Figure 5, when moving from step S16 to step S18, n B Individual awakening level data AL(AL[1]~AL[n B It is assumed that ]) has always been generated and accumulated. If the process proceeds to step S18, the total number of generated and accumulated awakening level data AL is n B If the number is less than one, you may skip the process in step S18, assign 1 to the variable i, and return to step S13.
[0081] Furthermore, the period from the timing of moving from step S14 to step S15 with "i=1" to the timing of moving from step S16 to step S18 is measured using driving capability data DA[1]~DA[n A This is an evaluation period for generating and accumulating ]. Awakening level data AL[1]~AL[n B ] may be the total arousal level data AL output from the arousal level estimation unit 111 during the evaluation period, in which case n B The value of this parameter will be an undefined value that depends on the length of the evaluation period.
[0082] Based on the detection results of user U1's driving ability and the estimation results of their alertness level, it is possible to determine the state of user U1 (for example, whether user U1's state corresponds to a state of distracted driving). However, there are various driving scenarios, and it is not easy to achieve high accuracy in detecting driving ability in all scenarios. If driving ability is detected in a driving scenario where it is difficult to improve the accuracy of driving ability detection, and the state of user U1 is determined based on the results, it will lead to a decrease in the accuracy of the state determination. Taking this into consideration, the controller 11 detects driving ability when the driving scenario of the vehicle V1 corresponds to a specific scenario (detection target scenario), and then determines the state of user U1 based on the driving ability detection result (DA) and the alertness level estimation result (AL) (S13~S18). In this way, by setting a specific scenario (detection target scenario) where it is easy to improve the accuracy of driving ability detection, the accuracy of determining the state of user U1 (for example, the accuracy of determining a state of distracted driving) can be improved.
[0083] More specifically, the controller 11 executes the driving ability detection process when the driving scene matches a specific scene (detection target scene), while suppressing the execution of the driving ability detection process (not executing the driving ability detection process; S13~S15) when the driving scene does not match a specific scene (detection target scene). This makes it possible to improve the accuracy of determining the state of user U1 (for example, the accuracy of determining a state of distracted driving) by setting scenes that are easier to improve the accuracy of driving ability detection as specific scenes (detection target scenes).
[0084] The controller 11 (scene determination unit 113) refers to scene determination information J3 (see Figure 2), which includes vehicle status information, surrounding situation information, and map information, when determining whether the driving scene of vehicle V1 corresponds to a specific scene (detection target scene). By referring to the vehicle status information, surrounding situation information, and map information, it is possible to correctly determine whether the driving scene of vehicle V1 corresponds to a specific scene that makes it easier to improve the accuracy of detecting driving ability.
[0085] When the controller 11 (driving capability detection unit 112) derives one piece of driving capability data DA, it may refer to both the vehicle state information and the line-of-sight information, or may refer to only one of the vehicle state information and the line-of-sight information. As already described, the controller 11 (alertness level estimation unit 111) can estimate the alertness level of the user U1 based on the biological information of the user U1, the face image of the user U1, or the interaction information between the user U1 and the agent. Through the driving capability detection processing and the alertness level estimation processing based on these pieces of information, the driving capability and the alertness level can be favorably detected and estimated, and thus the state of the user U1 can be favorably determined. The biological information of the user U1, the face image of the user U1, and the interaction information between the user U1 and the agent are the first, second, and third indices for estimating the alertness level. The controller 11 (alertness level estimation unit 111) may estimate the alertness level of the user U1 based on any two or more of the first, second, and third indices.
[0086] The controller 11 (user state determination unit 114) uses n A driving capability evaluation value DA_EV based on the results of the driving capability detection processing times as the determination threshold TH EV1 has a function of classifying the driving capability of the user U1 into a high driving capability or a low driving capability by comparison (see FIG. 7). At this time, if "DA_EV≧TH EV1 ", the driving capability of the user U1 is classified as the high driving capability, and if "DA_EV<TH EV1 ", the driving capability of the user U1 is classified as the low driving capability which is lower than the high driving capability. The low driving capability is lower than the high driving capability. On the other hand, the controller 11 (user state determination unit 114) uses n B alertness level evaluation value AL_EV based on the results of the alertness level estimation processing times as the determination threshold TH EV2 has a function of classifying the alertness level of the user U1 into a high alertness level or a low alertness level by comparison (see FIG. 7). At this time, if "AL_EV≧TH EV2 ", the alertness level of the user U1 is classified as the high alertness level, and if "AL_EV<TH EV2In this case, user U1's level of alertness is classified as low alertness, which is lower than high alertness. The controller 11 (user state determination unit 114) then determines which of several states (in this case, states STa and STb1 to STb3) user U1's state corresponds to, based on the classification result for driving ability and the classification result for alertness. This method makes it possible to make an appropriate user state determination according to the detection result of driving ability and the estimation result of alertness.
[0087] In this case, the controller 11 determines that user U1's state corresponds to the first state if user U1's driving ability is classified as high driving ability and user U1's level of alertness is classified as high alertness; otherwise, it determines that user U1's state corresponds to the second state. The first state is state STa, and the second state is state STb1, STb2, or STb3. The first state corresponds to a normal driving state, and the second state corresponds to a distracted driving state. With this determination method, user U1's state can be appropriately determined according to the detection results of driving ability and the estimation results of alertness.
[0088] In the operation described above in Example EX1, states STb1 to STb3 are distinguished, but it is also acceptable to consider a combination of states STb1 to STb3 as state STb. In this case, if it is determined that the state of user U1 corresponds to state STb (Y in step S19), the corresponding processing unit 115 may perform the first, second, or third idle operation response processing described above in step S20 based on the state determination data SJ indicating that fact.
[0089] By using HMI2 to handle distracted driving (warning notifications, etc.), it is possible to prevent dangerous situations related to the operation of vehicle V1 from occurring.
[0090] <<Example EX2>> Example EX2 will now be described. Example EX2 and the examples described below are based on Example EX1, and unless otherwise specified in Example EX2 and the examples described below, the description in Example EX1 shall apply to Example EX2 and the examples described below, unless there is a contradiction. However, when interpreting the description in Example EX2, the description in Example EX2 may take precedence in the event of any contradiction between Example EX1 and EX2 (the same applies to the examples described below).
[0091] The information that should be considered when detecting driving ability changes depending on the driving scene. For example, when vehicle V1 is driving straight at high speed, the driver's gaze is influenced by the movement of the background in front of them due to human habit, making it difficult to use eye-tracking information to detect driving ability. On the other hand, when vehicle V1 is driving straight at high speed, driving ability can be easily evaluated by vehicle speed, steering operation, and the degree of vehicle V1's sway. Also, for example, when vehicle V1 is approaching an intersection or turning right or left at an intersection at low speed, the degree of awareness of the surroundings is important for determining the quality of driving ability, making eye-tracking information useful for detecting driving ability. However, when vehicle V1 is approaching an intersection or turning right or left at an intersection at low speed, vehicle speed, steering operation, and the degree of vehicle V1's sway change only slightly between cases where the driving operation is appropriate and cases where it is not, making it difficult to use them to evaluate the quality of driving ability.
[0092] Considering these factors, the driving capability detection unit 112 in Example EX2 is provided with multiple detection algorithms to be used for the driving capability detection process. These multiple detection algorithms include detection algorithms ALG1, ALG2a, ALG2b, and ALG2c. Figure 8 is a diagram showing the correspondence between multiple types of driving scenes and multiple detection algorithms. The driving capability detection unit 112 has table data (table data as shown in Figure 8) that shows the correspondence between multiple types of driving scenes and multiple detection algorithms, and may decide which detection algorithm to use based on this table data. Detection algorithms ALG1, ALG2a, ALG2b, and ALG2c are algorithms that correspond to the detection target scenes SN1, SN2a, SN2b, and SN2c, respectively. Therefore, if the determined driving scene by the scene determination process in step S13 of Figure 5 matches the detection target scene SN1, SN2a, SN2b, or SN2c, the driving capability detection process is performed as follows.
[0093] Let's consider the first case where the driving scene determined by the scene determination process in step S13 matches the target scene SN1. In the first case, the driving capability detection process (in other words, the driving capability detection process by the first detection method) is executed by the detection algorithm ALG1 in step S15, generating the driving capability data DA. Let's consider the second case where the driving scene determined by the scene determination process in step S13 matches the target scene SN2a. In the second case, the driving capability detection process (in other words, the driving capability detection process by the second detection method) is executed by the detection algorithm ALG2a in step S15, generating the driving capability data DA. Let's consider the third case where the driving scene determined by the scene determination process in step S13 matches the target scene SN2b. In the third case, the driving capability detection process (in other words, the driving capability detection process by the third detection method) is executed by the detection algorithm ALG2b in step S15, generating the driving capability data DA. Let's consider the fourth case where the driving scene determined by the scene determination process in step S13 matches the target scene SN2c. In the fourth case, the driving capability detection process (in other words, the driving capability detection process by the fourth detection method) is executed in step S15 by the detection algorithm ALG2c, thereby generating the driving capability data DA. Note that if the flowchart in Figure 5 is rewritten to clearly show the technology shown in Example EX2, it will look like Figure 9. The only difference between Figure 5 and Figure 9 is the description of the processing content in step S15. Step S15 in Figure 9 may be specifically referred to as step S15a.
[0094] In the driving ability detection process using the detection algorithm ALG1, the driving ability detection unit 112 detects the user U1's driving ability primarily based on vehicle state information, without using gaze information. Therefore, in step S15 (S15a) of the first case, the driving ability detection unit 112 detects the user U1's driving ability primarily based on vehicle state information, without using gaze information. As a result, the driving ability data DA derived in step S15 (S15a) of the first case depends on vehicle state information but not on gaze information.
[0095] In the driving capability detection process by any of the detection algorithms ALG2a, 2b and 2c, the driving capability detection unit 112 detects the driving capability of the user U1 mainly based on line-of-sight information without using vehicle state information. Accordingly, in step S15 (S15a) according to any of the second to fourth cases, the driving capability detection unit 112 detects the driving capability of the user U1 mainly based on line-of-sight information without using vehicle state information. As a result, the driving capability data DA derived in step S15 (S15a) according to any of the second to fourth cases depends on the line-of-sight information, but does not depend on the vehicle state information.
[0096] FIG. 10 shows the internal configuration of the driving capability detection unit 112 according to the embodiment EX2. The driving capability detection unit 112 shown in FIG. 10 includes four detection units 112[1] to 112[4]. The driving capability detection unit 112 operates the detection units 112[1], 112[2], 112[3], and 112[4] in the first, second, third, and fourth cases, respectively.
[0097] The detection unit 112[1] performs the driving capability detection process using the detection algorithm ALG1 in the first case. The detection unit 112[1] detects the driving capability of the user U1 through the driving capability detection process based on the input information IN[1], and generates and outputs data indicating the detected driving capability as driving capability data OUT[1]. The driving capability detection unit 112 generates and outputs the driving capability data OUT[1] as the driving capability data DA in step S15 (S15a) according to the first case. The input information IN[1] corresponds to the speed SP at which the speed of the vehicle V1 is a predetermined value H or higher and includes vehicle state information and surrounding situation information during the first period in which the vehicle V1 is in a straight traveling state, and does not include line-of-sight information during the first period.
[0098] In the second case, the detection unit 112[2] performs driving ability detection processing using the detection algorithm ALG2a. The detection unit 112[2] detects the driving ability of user U1 through driving ability detection processing based on input information IN[2], and generates and outputs data indicating the detected driving ability as driving ability data OUT[2]. In step S15 (S15a) related to the second case, the driving ability detection unit 112 generates and outputs the driving ability data OUT[2] as driving ability data DA. The input information IN[2] is the speed of the vehicle V1 at a predetermined speed SP L The following includes line-of-sight information and surrounding situation information during the second period when vehicle V1 is traveling while decelerating towards an intersection, but does not include vehicle status information during the second period.
[0099] In the third case, the detection unit 112[3] performs driving ability detection processing using the detection algorithm ALG2b. The detection unit 112[3] detects the driving ability of user U1 through driving ability detection processing based on input information IN[3], and generates and outputs data indicating the detected driving ability as driving ability data OUT[3]. In step S15(S15a) related to the third case, the driving ability detection unit 112 generates and outputs the driving ability data OUT[3] as driving ability data DA. The input information IN[3] is that the speed of vehicle V1 is a predetermined speed SP L The following information includes line-of-sight information and surrounding situation information during the third period when vehicle V1 is driving to make a right turn at the intersection in question, but does not include vehicle status information during the third period.
[0100] In the fourth case, the detection unit 112[4] performs driving ability detection processing using the detection algorithm ALG2c. The detection unit 112[4] detects the driving ability of user U1 through driving ability detection processing based on input information IN[4], and generates and outputs data indicating the detected driving ability as driving ability data OUT[4]. In step S15(S15a) related to the fourth case, the driving ability detection unit 112 generates and outputs the driving ability data OUT[4] as driving ability data DA. The input information IN[4] is the speed of the vehicle V1 at a predetermined speed SP LThe following includes line-of-sight information and surrounding situation information during the fourth period when vehicle V1 is driving to turn left at the intersection in question, but does not include vehicle status information during the fourth period.
[0101] Let's explain the detection unit 112[1]. The detection unit 112[1] identifies the time-series change in the speed of the vehicle V1 during the first period based on the vehicle speed information during the first period included in the input information IN[1], and can detect and evaluate the driving ability of user U1 based on this time-series change. In this case, the detection unit 112[1] can detect the driving ability of user U1 by determining whether user U1 is driving the vehicle V1 at a speed appropriate for the driving scene during the first period. Hereinafter, the speed of the vehicle V1 will be simply referred to as vehicle speed. For example, the detection unit 112[1] derives the cumulative amount of vehicle speed fluctuation by accumulating the amount of fluctuation in vehicle speed per unit time during the first period, and the larger the cumulative amount of vehicle speed fluctuation, the lower the driving ability data OUT[1], and the smaller the cumulative amount of vehicle speed fluctuation, the higher the driving ability data OUT[1].
[0102] Furthermore, for example, the detection unit 112[1] can detect and evaluate the driving ability of user U1 based on the driving operation information during the first period included in the input information IN[1]. In this detection and evaluation, surrounding situation information during the first period may also be referenced. As described above, the driving operation information consists of accelerator pedal operation information, brake pedal operation information, steering wheel operation information, and turn signal operation information. The detection unit 112[1] can detect the driving ability of user U1 by determining whether user U1 is performing driving operations appropriate to the driving scene during the first period. For example, the detection unit 112[1] may increase or decrease the driving ability data OUT[1] based on the smoothness of steering wheel operation (derivative value of change in the amount of steering wheel operation) during the first period.
[0103] For example, the detection unit 112[1] evaluates the degree of vehicle V1 sway during the first period based on the external image information (external camera image) during the first period included in the input information IN[1], and can detect and evaluate the driving ability of user U1 based on the degree of sway. In this case, the detection unit 112[1] decreases the driving ability data OUT[1] as the degree of sway increases, and increases the driving ability data OUT[1] as the degree of sway decreases. Regarding vehicles, image-based sway detection technology is widely put into practical use, and such sway detection technology can be used in the detection unit 112[1]. The detection unit 112[1] may evaluate the degree of sway only when the speed of vehicle V1 is above a certain speed.
[0104] The detection units 112[2] to 112[4] will be explained below. The detection unit 112[2] can detect and evaluate the driving ability of user U1 based on the gaze information during the second period included in the input information IN[2]. In this detection and evaluation, surrounding situation information during the second period may also be referenced. The detection unit 112[2] can detect the driving ability of user U1 by determining whether user U1 is able to pay appropriate attention to the area in front of vehicle V1 (other vehicles and people etc. located in front of vehicle V1) during the second period.
[0105] Similarly, the detection unit 112[3] can detect and evaluate the driving ability of user U1 based on the gaze information during the third period included in the input information IN[3]. In this detection and evaluation, surrounding situation information during the third period may also be referenced. The detection unit 112[3] can detect the driving ability of user U1 by determining whether user U1 is paying appropriate attention to the direction of travel of vehicle V1 and its surroundings (other vehicles and people located in the direction of travel of vehicle V1 and its surroundings) during the third period. Similarly, the detection unit 112[4] can detect and evaluate the driving ability of user U1 based on the gaze information during the fourth period included in the input information IN[4]. In this detection and evaluation, surrounding situation information during the fourth period may also be referenced. The detection unit 112[4] can detect the driving ability of user U1 by determining whether user U1 is paying appropriate attention to the direction of travel of vehicle V1 and its surroundings (other vehicles and people located in the direction of travel of vehicle V1 and its surroundings) during the fourth period.
[0106] Simply put, for a variable i that satisfies "i=2", "i=3", or "i=4", the detection unit 112[i] can derive the cumulative gaze change amount by accumulating the amount of change in the user U1's gaze (amount of change in gaze direction) per unit time during the i-th period. The detection unit 112[i] may decrease the driving ability data OUT[i] as the cumulative gaze change amount decreases, and may increase the driving ability data OUT[i] as the cumulative gaze change amount increases.
[0107] The detection unit 112[1] may perform driving ability detection processing using a rule-based detection algorithm ALG1. Similarly, the detection units 112[2] to 112[4] may each perform driving ability detection processing using rule-based detection algorithms ALG2a, ALG2b, and ALG2c, respectively. Various technologies have been proposed for evaluating and detecting driving ability for a vehicle based on various information related to the vehicle. A known algorithm that evaluates and detects driving ability based on information equivalent to vehicle state information and information equivalent to surrounding situation information may be used as the detection algorithm ALG1. A known algorithm that evaluates and detects driving ability based on information equivalent to gaze information and information equivalent to surrounding situation information may be used as the detection algorithms ALG2a, ALG2b, or ALG2c.
[0108] The detection unit 112[1] may perform driving ability detection processing using a first AI model formed by machine learning. Similarly, the detection units 112[2], 112[3], and 112[4] may perform driving ability detection processing using a second AI model, a third AI model, and a fourth AI model formed by machine learning. The first, second, third, and fourth AI models can be formed by machine learning using the first, second, third, and fourth training datasets, respectively. A brief explanation of each training dataset and machine learning is provided below.
[0109] In the data collection process for creating each training dataset, a drive simulator (not shown) and a data collection device (not shown) are used with the cooperation of the subjects. The drive simulator is a device that simulates the driving and operation of an automobile with a configuration equivalent to that of vehicle V1. For convenience, the automobile whose driving and operation are simulated in the drive simulator is referred to as a virtual automobile. The drive simulator is equipped with a display device that shows the view from the driver's seat of the virtual automobile, speakers that output the sounds of the virtual automobile driving, and driving control components for operating the virtual automobile, as well as a simulation computer. The driving control components of the drive simulator include the steering wheel, accelerator pedal, brake pedal, and turn signal controls of the virtual automobile. In the data collection process, the subjects operate the virtual automobile using the drive simulator. The driving operations of the virtual automobile are performed on the driving control components of the drive simulator. The simulation computer displays images on the display device in the drive simulator and outputs sounds from the speakers in the drive simulator in response to the driving operations performed by the subjects.
[0110] In the first test environment, where the subject's driving ability is expected to be relatively high, the subject is instructed to operate a virtual vehicle using a driving simulator. The data acquisition device acquires vehicle status information, surrounding environment information, and the subject's gaze information of the virtual vehicle in the first test environment, and stores the acquired information as first test data in a database (not shown). Separately, in the second test environment, where the subject's driving ability is expected to be relatively low, the subject is instructed to operate a virtual vehicle using a driving simulator. The data acquisition device acquires vehicle status information, surrounding environment information, and the subject's gaze information of the virtual vehicle in the second test environment, and stores the acquired information as second test data in a database (not shown).
[0111] The virtual vehicle's vehicle status information corresponds to the vehicle status information of vehicle V1 and includes the virtual vehicle's speed information, etc. The virtual vehicle's surrounding environment information corresponds to the surrounding environment information of vehicle V1 and includes image information showing the outside of the virtual vehicle, etc. The subject's gaze information corresponds to the gaze information of user U1.
[0112] For example, in the second test environment, subjects are asked to drive a virtual car while performing an additional task, while in the first test environment, subjects are asked to drive a virtual car without performing an additional task. In each test environment, subjects are responsible for the task of driving a virtual car. The additional task is a task different from the task of driving a virtual car (for example, a task that requires subjects to perform some kind of work or thinking). In the second test environment, the presence of the additional task makes it more likely that subjects' driving ability will be reduced compared to the first test environment. In the above database, the first test data is labeled as high ability, indicating that the subject has high driving ability, and the second test data is labeled as low ability, indicating that the subject has low driving ability.
[0113] In the data collection process, the cooperation of numerous subjects is sought to collect first and second test data from each subject and store them in a database. In the machine learning process following the data collection process, first to fourth AI models are created by performing machine learning on first to fourth DNNs (Deep Neural Networks) based on the first and second test data sets using a machine learning device (not shown). The machine learning is performed with each DNN indicating that the first test data represents data from a subject with relatively high driving ability, and the second test data represents data from a subject with relatively low driving ability. When creating the i-th AI model, machine learning is performed using the first and second test data under the conditions corresponding to case i (where i is 1, 2, 3, or 4).
[0114] Let's consider the technology shown in Example EX2. The controller 11 uses different detection algorithms for detecting driving ability depending on whether the driving scene of the vehicle V1 corresponds to the first detection target scene (SN1) or the second detection target scenes (SN2a, SN2b, SN2c), thereby differentiating the method of detecting the user U1's driving ability. As a result, driving ability can be detected using a detection method appropriate to the driving scene, thereby improving the accuracy of driving ability detection.
[0115] Specifically, when the driving scene of vehicle V1 corresponds to the first detection target scene, the driving ability of user U1 is detected based on vehicle state information, and when the driving scene of vehicle V1 corresponds to the second detection target scene, the driving ability of user U1 is detected based on user U1's gaze information. There are situations in which referencing vehicle state information is suitable for detecting driving scenes, and situations in which referencing gaze information is suitable for detecting driving scenes. Therefore, as described above, by switching the information referenced for detecting driving ability, the accuracy of driving ability detection can be improved.
[0116] Here, the first detection target scene is preferably detection target scene SN1, which corresponds to high-speed, straight-line driving (see Figures 6 and 8). When vehicle V1 is driving straight at high speed, the driver's gaze is influenced by the movement of the background in front of them due to human habit, making it difficult to use gaze information to detect driving ability. On the other hand, when vehicle V1 is driving straight at high speed, it is easy to evaluate driving ability based on vehicle speed, steering wheel operation, and the degree of vehicle V1's swaying. The second detection target scene is preferably detection target scenes SN2a, SN2b, or SN2c, which relate to approaching an intersection or turning right or left at an intersection at low speed (see Figures 6 and 8). When vehicle V1 is approaching an intersection or turning right or left at an intersection at low speed, the degree to which the driver pays attention to their surroundings is important for determining the quality of their driving ability, so gaze information is useful for detecting driving ability.
[0117] Furthermore, the detection algorithms ALG2b and ALG2c may be a common detection algorithm, in which case the detection units 112[3] and 112[4] are considered to be a common detection unit (see Figure 10). Alternatively, the detection algorithms ALG2a, ALG2b, and ALG2c may be a common detection algorithm, in which case the detection units 112[2] to 112[4] are considered to be a common detection unit (see Figure 10).
[0118] <<Example EX3>> Example EX3 will now be described. Example EX3 describes an application technology applicable to Example EX2. Figure 11 is an operation flowchart of the controller 11 according to Example EX3. Based on the operation flowchart of the controller 11 according to Example EX1 (Figure 5), in Example EX3, step S14 is replaced by a group of steps S14_1 to S14_5. Except for this replacement, the operation flow of the controller 11 according to Example EX3 is the same as the operation flow of the controller 11 according to Example EX1. However, in Example EX3, as with Example EX2 (see Figure 9), step S15 may be referred to as step S15a.
[0119] The operation of the controller 11 in Example EX3 will be explained with reference to Figure 11. The operation from the start of the driving support device 1 through steps S11 and S12 to step S13 is the same as in Example EX1. The processing content in step S13 is also as shown in Example EX1. However, in Example EX3, after step S13, the process proceeds to step S14_1.
[0120] In step S14_1, the driving ability detection unit 112 checks whether the current driving scene of vehicle V1 was determined to be one of the detection target scenes SN1, SN2a, SN2b, and SN2c in the scene determination process in step S13. If it is determined that the current driving scene of vehicle V1 is one of the detection target scenes SN1, SN2a, SN2b, and SN2c (Y in step S14_1), the process proceeds to step S14_2. If it is determined that the current driving scene of vehicle V1 is one of the non-detection target scenes (N in step S14_1), the process returns to step S13.
[0121] In step S14_2, the driving capability detection unit 112 checks whether the determined driving scene by the scene determination process in step S13 is the target scene SN1. If the determined driving scene by the scene determination process in step S13 is the target scene SN1 (Y in step S14_2), the process proceeds to step S15 (S15a). If the determined driving scene by the scene determination process in step S13 is not the target scene SN1 (N in step S14_2), the process proceeds to step S14_3.
[0122] In step S14_3, the driving ability detection unit 112 checks whether the driving scene determined by the scene determination process in step S13 is the target scene SN2a and whether the current surrounding conditions of the vehicle V1 correspond to specific condition PS2a. At this time, the scene determination unit 113 determines whether the current surrounding conditions of the vehicle V1 correspond to specific condition PS2a based on the surrounding condition information, and the result of this determination is provided to the driving ability detection unit 112, enabling the confirmation in step S14_3. Specific condition PS2a will be described later. In step S14_3, if the driving scene determined by the scene determination process in step S13 is the target scene SN2a and the current surrounding conditions of the vehicle V1 correspond to specific condition PS2a (Y in step S14_3), the process returns to step S13; otherwise (N in step S14_3), the process proceeds to step S14_4.
[0123] In step S14_4, the driving ability detection unit 112 confirms whether the driving scene determined by the scene determination process in step S13 is the target scene SN2b and whether the current surrounding conditions of the vehicle V1 correspond to specific condition PS2b. At this time, the scene determination unit 113 determines whether the current surrounding conditions of the vehicle V1 correspond to specific condition PS2b based on the surrounding condition information, and the result of this determination is provided to the driving ability detection unit 112, enabling the confirmation in step S14_4. Specific condition PS2b will be described later. In step S14_4, if the driving scene determined by the scene determination process in step S13 is the target scene SN2b and the current surrounding conditions of the vehicle V1 correspond to specific condition PS2b (Y in step S14_4), the process returns to step S13; otherwise (N in step S14_4), the process proceeds to step S14_5.
[0124] In step S14_5, the driving ability detection unit 112 checks whether the driving scene determined by the scene determination process in step S13 is the target scene SN2c and whether the current surrounding conditions of the vehicle V1 correspond to the specific condition PS2c. At this time, the scene determination unit 113 determines whether the current surrounding conditions of the vehicle V1 correspond to the specific condition PS2c based on the surrounding condition information, and the result of this determination is provided to the driving ability detection unit 112, enabling the confirmation in step S14_5. The specific condition PS2c will be described later. In step S14_5, if the driving scene determined by the scene determination process in step S13 is the target scene SN2c and the current surrounding conditions of the vehicle V1 correspond to the specific condition PS2c (Y in step S14_5), the process returns to step S13; otherwise (N in step S14_5), the process proceeds to step S15.
[0125] Step S15 in Example EX3 is the same as step S15(S15a) in Example EX2. Therefore, in step S15(S15a) of Example EX3, the driving capability data DA is generated according to the method shown in Example EX2. Then, in step S15 when the judgment driving scene is the detection target scene SN1, SN2a, SN2b, or SN2c, the driving capability data OUT[1], OUT[2], OUT[4], and OUT[4] are generated as the driving capability data DA, respectively (see Figure 10). The operation after the completion of step S15(S15a) is as shown in Example EX1.
[0126] The significance of the processing performed between steps S13 and S15 in Figure 11 will be explained. As shown in Example EX2, when the judgment driving scene is a second detection target scene (SN2a, SN2b, or SN2c), the driving ability detection processing is performed mainly by referring to the gaze information of user U1. However, depending on the surrounding conditions of vehicle V1, there is little difference in gaze information between when user U1 is concentrating on driving and when driving absentmindedly, and therefore it is difficult to distinguish between the former and the latter cases using gaze information. If the driving ability detection processing is performed in a situation where it is difficult to distinguish between the former and the latter cases using gaze information, it is difficult to obtain high judgment accuracy, and as a result, it is difficult to improve the accuracy of the judgment of user U1's state. Taking this into consideration, in Example EX3, even if the driving scene of vehicle V1 corresponds to a second detection target scene, the execution of the driving ability detection processing is suppressed (driving ability detection processing is not executed: S14_3~S14_5). This suppresses the driving capability detection process in situations where accuracy is difficult to improve, thereby reducing the degradation of accuracy in user U1 state determination due to false detections (and consequently ensuring accuracy in state determination).
[0127] Examples of specific situations PS2a, PS2b, or PS2c are given below as the first to fifth specific situations. When the scene determination unit 113 performs scene determination processing, it determines whether the current surrounding situation of the vehicle V1 corresponds to one of the first to fifth specific situations based on the surrounding situation information, and supplies the determination result together with the scene determination result information J4 to the driving ability detection unit 112.
[0128] The first specific situation is a situation in which a vehicle is present ahead. Here, the vehicle ahead is another vehicle (such as a car) traveling in front of vehicle V1. However, the distance between the other vehicle and vehicle V1 (inter-vehicle distance) is assumed to be less than or equal to a predetermined distance (for example, about 5 to 10 meters). In the first specific situation, user U1 basically just looks straight ahead in order to make vehicle V1 follow the vehicle ahead, and therefore, there is little difference in eye-tracking information between when user U1 is concentrating on driving and when driving absentmindedly.
[0129] Refer to Figure 12 for an explanation of the second to fourth specific situations. Figure 12 illustrates the relationship between the intersection 610 of interest and vehicle V1. In relation to the intersection 610 of interest, the driving scene of vehicle V1 is determined by the scene determination process to be the detection target scene SN2a, SN2b, or SN2c. The intersection 610 of interest is an intersection where vehicle V1 can go straight, turn right, or turn left. In the situation shown in Figure 12, after driving toward the intersection 610 of interest, vehicle V1 will turn right or left at the intersection 610 of interest, go straight through the intersection 610, or stop before the intersection 610 of interest, based on the driving operations of user U1 while following the lighting status of the lights on the traffic light 620. The traffic light 620 is a traffic light installed at the intersection 610 of interest that indicates permission for vehicle V1 to proceed or an instruction to stop at the intersection 610 of interest.
[0130] Figure 13 shows the configuration of the traffic signal 620. The traffic signal 620 is an arrow-type traffic signal having a main signal unit 621 and an arrow signal unit 622. The main signal unit 621 has a red lamp 621R, a yellow lamp 621Y, and a blue lamp 621B. The arrow signal unit 622 has a right arrow lamp 622R, a straight arrow lamp 622S, and a left arrow lamp 622L.
[0131] Traffic light 620 illuminates only one of the following: red lamp 621R, yellow lamp 621Y, or blue lamp 621B. When red lamp 621R is illuminated, it emits a red light and functions as a red signal for vehicle V1. The illumination of red lamp 621R indicates a red light and suggests that vehicle V1 should stop before intersection 610. When yellow lamp 621Y is illuminated, it emits a yellow light and functions as a yellow signal for vehicle V1. The illumination of yellow lamp 621Y indicates a yellow light and suggests that vehicle V1 should stop before intersection 610, except in exceptional circumstances. When blue lamp 621B is illuminated, it emits a blue or green light and functions as a green signal for vehicle V1. The illumination of blue lamp 621B indicates a blue light and means that vehicle V1 is permitted to enter intersection 610.
[0132] At traffic light 620, one or more of the following lamps may be illuminated: right arrow lamp 622R, straight arrow lamp 622S, and left arrow lamp 622L. When the right arrow lamp 622R is illuminated, it lights up in blue in the shape of a rightward arrow and functions as a rightward arrow signal. The illumination of the right arrow lamp 622R is a blue light arrow that points to the right and, regardless of the illumination status of lamps 621R, 621Y, and 621B, it signifies permission for vehicle V1 to enter the designated intersection 610 and to turn right at the designated intersection 610. When the left arrow lamp 622L is illuminated, it lights up in blue in the shape of a leftward arrow and functions as a leftward arrow signal. When the left arrow lamp 622L is illuminated, it is a blue light arrow pointing to the left, and regardless of the illumination status of lamps 621R, 621Y, and 621B, it means that vehicle V1 is permitted to enter the intersection 610 and to turn left at the intersection 610. When the straight arrow lamp 622S is illuminated, it is a forward-pointing arrow shaped blue light and functions as a forward arrow signal. When the straight arrow lamp 622S is illuminated, it is a blue light arrow pointing to the forward (arrow in the direction of going straight), and regardless of the illumination status of lamps 621R, 621Y, and 621B, it means that vehicle V1 is permitted to enter the intersection 610 and to pass straight through the intersection 610.
[0133] Hereinafter, the situation when the right arrow lamp 622R is lit will be referred to as a right-turn signal situation, the situation when the left arrow lamp 622L is lit will be referred to as a left-turn signal situation, and the situation when the straight arrow lamp 622S is lit will be referred to as a straight-ahead signal situation. The scene determination unit 113 can determine whether the situation around the vehicle V1 corresponds to a right-turn signal situation, a left-turn signal situation, or a straight-ahead signal situation based on external camera images, including images of the traffic signal 620.
[0134] The second specific situation is the right-turn signal situation. When vehicle V1 turns right at intersection 610 in the right-turn signal situation, the traffic light at the pedestrian crossing that vehicle V1 crosses during the right-turn process is red. Therefore, user U1 performs the right-turn driving maneuver while basically only looking in the direction of vehicle V1's movement, without paying much attention to pedestrians. Consequently, there is little difference in eye-tracking information between when user U1 is concentrating on driving and when they are driving absentmindedly.
[0135] The third specific situation is the left-turn signal situation. When vehicle V1 turns left at intersection 610 in the left-turn signal situation, the traffic light at the pedestrian crossing that vehicle V1 crosses during the left turn is red. Therefore, user U1 performs the left-turn driving maneuver while basically only looking in the direction of vehicle V1's movement, without paying much attention to pedestrians. Consequently, there is little difference in eye-tracking information between when user U1 is concentrating on driving and when they are driving absentmindedly.
[0136] The fourth specific situation is a straight-ahead signal situation. In a straight-ahead signal situation, when vehicle V1 proceeds straight through intersection 610, the traffic light at the pedestrian crossing that vehicle V1 crosses during its straight-ahead journey is red. Therefore, user U1 performs straight-ahead driving operations while basically only looking in the direction of vehicle V1's movement, without paying much attention to pedestrians. Consequently, there is little difference in eye-tracking information between when user U1 is concentrating on driving and when driving absentmindedly.
[0137] The fifth specified situation is a situation where the visibility is sufficiently good for user U1. The scene determination unit 113 determines whether the visibility for user U1 in the direction of travel of vehicle V1 is sufficiently good based on the external camera image, and determines that the surrounding situation of vehicle V1 falls under the fifth specified situation when it determines that the visibility is sufficiently good. In a situation with poor visibility, user U1 frequently moves their gaze to grasp the details of what is happening inside and outside the intersection 610 of interest. In contrast, in a situation with sufficiently good visibility, the movement of the gaze becomes relatively smaller. Therefore, there is little difference in gaze information between when user U1 is concentrating on driving and when they are driving absentmindedly.
[0138] The relationship between the specific situations PS2a, PS2b, and PS2c shown in the operation flowchart of Figure 11 and the first to fifth specific situations will be explained. In Example EX3, the detection target scene SN2a is when the speed of the vehicle V1 is a predetermined speed SP L Scene SN2a_1 is understood to be the following, and in which vehicle V1 enters and passes through the intersection 610 of interest. However, the detection target scene SN2a is the same as shown in Examples EX1 and EX2 (see Figures 6 and 8), where the speed of vehicle V1 is a predetermined speed SP L The following scenario SN2a_2 may also occur, in which vehicle V1 is driving towards intersection 610 in order to stop just before it.
[0139] The first specific situation is an example of specific situation PS2a. In addition, the first specific situation may also be an example of specific situation PS2b or specific situation PS2c. The second specific situation (right-turn signal situation) is an example of specific situation PS2b corresponding to a right turn, and the third specific situation (left-turn signal situation) is an example of specific situation PS2c corresponding to a left turn (see Figures 6, 8, and 11). The fourth specific situation (straight-ahead signal situation) is an example of specific situation PS2a corresponding to going straight. The fifth specific situation is an example of specific situation PS2a, a specific situation PS2b, or a specific situation PS2c.
[0140] <<Example EX4>> Example EX4 will be described. Example EX4 will describe application techniques or modification techniques for the various operations and configurations described above.
[0141] In the embodiments described above, four types of detection target scenes (SN1, SN2a, SN2b, and SN2c) are listed as detection target scenes, but the number of detection target scene types is arbitrary as long as it is one or more.
[0142] Each operation of the controller 11 shown in Examples EX1 to EX3 may be performed only under predetermined service application conditions. For example, the service provision condition may be that the vehicle V1 is located within a predetermined service provision area.
[0143] While passenger cars or trucks that travel on public roads are primarily envisioned as Vehicle V1, the type of Vehicle V1 is arbitrary. Vehicle V1 may also be an industrial vehicle such as a forklift, or an agricultural vehicle such as a tractor.
[0144] A program that causes a computer device to execute any method described in the embodiments of the present invention, and a non-volatile recording medium on which such program is recorded, are included within the scope of the embodiments of the present invention. The program that causes a computer (computer device) to execute any method described in the embodiments of the present invention may be a subprogram incorporated into any main program or called by any main program. Any processing in the embodiments of the present invention may be implemented by hardware such as a semiconductor integrated circuit, software corresponding to the above program, or a combination of hardware and software.
[0145] When focusing on the function of determining the driving state of user U1, the driver assistance device 1 functions as a driving state determination device, and the driver assistance system SYS functions as a driving state determination system. The controller 11 has a function of determining whether user U1 is driving carelessly (i.e., whether user U1's state corresponds to a careless driving state). When focusing on the function of determining whether user U1 is driving carelessly, the driver assistance device 1 functions as a careless driving determination device, and the driver assistance system SYS functions as a careless driving determination system. The driving state determination device or the careless driving determination device includes a computer capable of executing any program. The controller 11 itself or the arithmetic processing unit provided in the controller 11 can be considered as the computer. The method executed by the driver state determination device according to the present invention can be called a driving state determination method, and the program that causes the computer to execute the driving state determination method can be called a driving state determination program. The method executed by the careless driving determination device according to the present invention can be called a careless driving determination method, and the program that causes the computer to execute the careless driving determination method can be called a careless driving determination program.
[0146] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical idea set forth in the claims. The embodiments described above are merely examples of embodiments of the present invention, and the meaning of the terms of the present invention or each constituent element is not limited to those described above. The specific numerical values shown in the above description are merely examples and can, of course, be changed to various numerical values. [Explanation of Symbols]
[0147] SYS Driver Assistance System V1 Vehicle U1 User 1. Driving assistance system 11 Controllers 12 memory 13 Communications Department 2 HMI 21 Microphone 22 speakers 23 Display device 24 Vibration device 3. Vehicle status detection unit 31. Accelerator pedal sensor 32 Brake pedal sensor 33 Handle Sensor 34 Turn signal sensor 35. Vehicle speed sensor 36. Steering angle sensor 37 G-sensor 38 GPS sensors 4. Surrounding Condition Detection Unit 41. Exterior car camera 42 Distance measuring sensor 43 Illuminance Sensor 44 Rainfall Sensor 5. In-car camera 111 Awakening Level Estimation Unit 112 Driving ability detection unit 113 Scene determination unit 114 User State Determination Unit 115 Corresponding Processing Unit
Claims
1. A driving state determination device for determining the state of the driver of a vehicle, comprising a controller, The controller detects the driver's driving ability when the vehicle's driving scene matches the detection target scene, and determines which of several states the driver's state corresponds to based on the detection result of the driving ability and the driver's level of alertness. The aforementioned plurality of states include a first state and a second state in which the driver's level of concentration on driving is lower than that of the first state. Operating status determination device.
2. There are multiple scenes to be detected. The controller detects the driving ability when the vehicle's driving scene corresponds to the first detection target scene or the second detection target scene. The controller makes the method for detecting the driving ability different depending on whether the vehicle's driving scene corresponds to the first detection target scene or the second detection target scene. The operating state determination device according to claim 1.
3. The controller detects the driving ability based on vehicle status information indicating the state of the vehicle when the vehicle driving scene corresponds to the first detection target scene, and detects the driving ability based on the driver's gaze information when the vehicle driving scene corresponds to the second detection target scene. The operating state determination device according to claim 2.
4. The first detection target scene is a driving scene in which the vehicle's speed is equal to or greater than a first predetermined speed and the vehicle is traveling in a straight line. The second detection target scene is a driving scene in which the vehicle's speed is less than or equal to the second predetermined speed which is less than the first predetermined speed, and the vehicle is decelerating as it approaches an intersection, or a driving scene in which the vehicle's speed is less than or equal to the second predetermined speed, and the vehicle is driving to turn left or right at the intersection. The operating state determination device according to claim 3.
5. The controller will not perform the process of detecting the driving ability if the vehicle's driving scene corresponds to the second detection target scene, but the surrounding conditions of the vehicle correspond to a specific condition. or an operating state determination device according to any one of claims 2 to 4.
6. The controller executes the process for detecting the driving ability when the vehicle's driving scene matches the detection target scene, and does not execute the process when the vehicle's driving scene does not match the detection target scene. or an operating state determination device according to any one of claims 1 to 4.
7. The controller determines whether the vehicle's driving scene corresponds to the detected scene based on vehicle status information indicating the vehicle's state, surrounding situation information indicating the vehicle's surroundings, and map information. or an operating state determination device according to any one of claims 1 to 4.
8. The controller detects the driving ability based on at least one of the vehicle status information indicating the state of the vehicle and the driver's gaze information, and estimates the level of alertness based on at least one of the driver's biometric information, the driver's facial image, and dialogue information between the driver and the simulated personality formed by the controller. or an operating state determination device according to any one of claims 1 to 4.
9. The controller classifies the driving ability into a first driving ability or a second driving ability lower than the first driving ability through a process for detecting the driving ability, and classifies the level of alertness into a first level of alertness or a second level of alertness lower than the first level through a process for estimating the level of alertness, and determines which of the multiple states the driver's state falls into based on the former and latter classification results. or an operating state determination device according to any one of claims 1 to 4.
10. The controller determines that the driver's state corresponds to the first state when the driving ability is classified as the first driving ability and the level of alertness is classified as the first level of alertness, and determines that the driver's state corresponds to the second state in all other cases. The second state described above is a state of aimless driving. The operating state determination device according to claim 9.
11. An operating state determination device according to any one of claims 1 to 4, A driver assistance system comprising an interface for providing notifications to the driver, The controller notifies the driver using the interface, in accordance with the result of the determination regarding the driver's state. , driver assistance system.
12. A driving state determination program that causes a computer to execute a driving state determination method for determining the state of the driver of a vehicle, wherein the driving state determination method is: A step of detecting the driver's driving ability when the driving scene of the vehicle corresponds to the detection target scene, The process includes determining which of a plurality of states the driver's condition falls into, based on the detection results of the driving ability and the driver's level of alertness. The aforementioned plurality of states include a first state and a second state in which the driver's level of concentration on driving is lower than that of the first state. , a program for determining the operating status.
Citation Information
Patent Citations
Electrode structure for image forming device
JP1988024262A
Driving state detection method
JP7578021B2
