Vehicle control device and vehicle control method

Abstract

To further safely perform automatic driving.SOLUTION: A vehicle control device comprises: an automation level setting unit which sets an automation level; a driver monitoring unit which detects a driving intervention level showing a degree that a driver intervenes driving of a vehicle and in which a demand degree is set lower as the automation level is higher, and determines whether or not the driving intervention level lacks to the set automation level; and a penalty application unit which applies a penalty which becomes a disadvantage of the driver to correct the intervention level of the driver on the basis of occurrence frequencies of a state in which the driving intervention level lacks in the case where the state in which the driving intervention level lacks occurs for a plurality of times. The present technology can be applied to, for example, the vehicle control device which controls the driving.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present technology relates to a vehicle control device and a vehicle control method, and more particularly to a vehicle control device and a vehicle control method that enable safer automatic driving. [Background technology]

[0002] It has been proposed to determine whether a driver's poor posture is due to the driver's habit, and to notify the driver about the poor posture in different ways depending on whether the poor posture is determined to be due to a habit or not (see, for example, Patent Document 1).

[0003] In addition, it has been proposed to prohibit autonomous driving of a vehicle if it is determined that the driver does not have the driving ability to return from autonomous driving to manual driving before the vehicle begins autonomous driving (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-38793 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-115356 Summary of the Invention [Problem to be solved by the invention]

[0005] However, as autonomous driving becomes more widespread, there are concerns that drivers may become dependent on autonomous driving. If drivers become too dependent on autonomous driving, for example, if the autonomous driving system is no longer able to operate normally, the driver may not be able to intervene quickly, increasing the risk of an accident.

[0006] However, Patent Documents 1 and 2 do not consider preventing the driver from becoming dependent on automated driving.

[0007] This technology was developed in light of these circumstances, and aims to enable safer autonomous driving. [Means for solving the problem]

[0008] A vehicle control device according to one aspect of the present technology includes an automation level setting unit that sets an automation level, a driver monitoring unit that detects a driving intervention level that indicates the degree to which the driver is involved in driving the vehicle, with the required level being set lower the higher the automation level, and determines whether the driving intervention level is insufficient compared to the set automation level, and a penalty application unit that, if a state in which the driving intervention level is insufficient occurs multiple times, applies a penalty that is detrimental to the driver and causes the driver's intervention level to be corrected based on the frequency of occurrence of the state in which the driving intervention level is insufficient.

[0009] A vehicle control method according to one aspect of the present technology comprises a vehicle control device that sets an automation level, indicates the degree to which the driver is involved in driving the vehicle, detects a driving intervention level that is set lower the higher the automation level, determines whether the driving intervention level is insufficient compared to the set automation level, and if a state in which the driving intervention level is insufficient occurs multiple times, applies a penalty that is detrimental to the driver and forces the driver to correct their intervention level based on the frequency of occurrence of the state in which the driving intervention level is insufficient.

[0010] In one aspect of the present technology, an automation level is set to indicate the degree to which the driver is involved in driving the vehicle, and a driving intervention level is detected that is set to a lower level as the automation level is higher.It is determined whether the driving intervention level is insufficient compared to the set automation level, and if a state in which the driving intervention level is insufficient occurs multiple times, a penalty is applied that is detrimental to the driver and forces the driver to correct their intervention level based on the frequency of occurrence of the state in which the driving intervention level is insufficient. [Effects of the Invention]

[0011] According to one aspect of the present technology, the driver's level of driving intervention is corrected, thereby enabling safer automated driving.

[0012] The effects described here are not necessarily limited to those described herein, and may be any of the effects described in this disclosure. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a configuration example of an autonomous driving system to which the present technology is applied. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a driver monitoring unit and a vehicle control unit. [Figure 3] FIG. 1 is a diagram illustrating an example of installation of a portion of a vehicle control system. [Figure 4] FIG. 1 is a diagram for explaining automation levels. [Figure 5] 10 is a flowchart illustrating an automatic driving control process. [Figure 6] 10 is a flowchart illustrating an automatic driving control process. [Figure 7] FIG. 10 is a diagram showing a specific example of a distribution of allowable automation levels. [Figure 8] 10 is a flowchart for explaining details of a driving intervention level correction process. [Figure 9] 10 is a flowchart for explaining details of a manual driving return support process. [Figure 10] FIG. 10 is a diagram illustrating a specific example of a log. [Figure 11] 10 is a flowchart for explaining an autonomous driving use restriction process. [Figure 12] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, a mode for carrying out the invention (hereinafter referred to as an "embodiment") will be described in detail with reference to the drawings. The description will be made in the following order. 1. Embodiment 2. Variations 3.Other

[0015] <<1. Embodiment>> <Example of autonomous driving system configuration> FIG. 1 shows an example of the configuration of an autonomous driving system 10 to which the present technology is applied.

[0016] The automatic driving system 10 includes a vehicle control system 11 and a mobile terminal 12.

[0017] The vehicle control system 11 includes a surrounding image capturing unit 21, a surrounding information acquisition unit 22, a position measurement unit 23, an input unit 24, a vehicle information acquisition unit 25, a driver monitoring unit 26, a communication unit 27, a vehicle control unit 28, a display unit 29, an audio output unit 30, a light emitting unit 31, an odor generating unit 32, a driving control unit 33, an in-vehicle device control unit 34, and a memory unit 35.

[0018] The surroundings photographing unit 21 includes various photographing devices such as a mono camera, a stereo camera, a Tof (Time of Flight) camera, a polarization camera, a time-gated camera, a multispectral camera, and an infrared or other invisible light camera, and photographs the surroundings of the vehicle including the traveling direction of the vehicle. The surroundings photographing unit 21 supplies the image obtained by photographing (hereinafter referred to as a surrounding image) to the vehicle control unit 28.

[0019] The surrounding information acquisition unit 22 is equipped with various sensors such as sonar, radar, lidar, temperature sensor, humidity sensor, etc., and acquires information about the surroundings of the vehicle. For example, the surrounding information acquisition unit 22 acquires information about the environment around the vehicle such as temperature, humidity, weather, road surface condition, etc., as well as information about objects around the vehicle such as the type and position of the objects around the vehicle. The surrounding information acquisition unit 22 supplies the acquired surrounding information to the vehicle control unit 28.

[0020] The position measurement unit 23 measures the current position of the vehicle using a satellite navigation system such as the Global Navigation Satellite System (GNSS), which measures the current position using artificial satellites. The position measurement unit 23 supplies the measurement result to the vehicle control unit 28.

[0021] The input unit 24 includes input devices such as a microphone, buttons, switches, a touch panel, and a gesture recognition device, and receives input of instructions, data, etc. from vehicle occupants including the driver. The input unit 24 supplies the input instructions, data, etc. to the vehicle control unit 28.

[0022] The vehicle information acquisition unit 25 acquires vehicle information including various types of information related to the vehicle. For example, the vehicle information acquisition unit 25 acquires information related to the movement of the vehicle, such as the vehicle speed, acceleration, angular velocity, and traveling direction. The vehicle information acquisition unit 25 also acquires information related to driving operations, such as the operation timing and operation amount of the accelerator pedal, brake pedal, steering wheel, parking brake, shift lever, turn signal lever, power (ignition) switch, lamp switch, wiper switch, etc. Furthermore, the vehicle information acquisition unit 25 acquires information related to the state of the vehicle, such as the state of each part of the vehicle and whether or not there is a malfunction. The vehicle information acquisition unit 25 supplies the acquired vehicle information to the vehicle control unit 28.

[0023] The driver monitoring unit 26 monitors the driver as will be described later with reference to FIG. 2, and supplies the monitoring results to the vehicle control unit 28.

[0024] The communication unit 27 includes communication devices for various communication methods.

[0025] For example, the communication unit 27 includes a communication device that performs wireless communication using DSRC (Dedicated Short Range Communications), and communicates with ITS (Intelligent Transport Systems) spots installed along roads to acquire a Local Dynamic Map (LDM). The LDM includes, for example, static information including road surface information, lane information, and three-dimensional structure information, quasi-static information including traffic regulation information, road construction information, and wide-area weather information, quasi-dynamic information including accident information, congestion information, and narrow-area weather information, and dynamic information including information on surrounding vehicles and pedestrians, traffic signal information, and the like.

[0026] For example, the communication unit 27 includes a communication device that communicates in accordance with the communication standards (3G / 4G / LTE (Long Term Evolution), etc.) that mobile phones use to communicate, and acquires various information such as map data from a server or the like via a network such as the Internet.

[0027] For example, the communication unit 27 includes a beacon device, communicates with roadside devices installed on the roadside to support safe driving, and acquires various types of traffic information.

[0028] For example, the communication unit 27 includes a short-range wireless communication device such as Bluetooth (registered trademark), and communicates with the mobile terminal 12 and the like to transmit and receive various types of information.

[0029] The communication unit 27 supplies the acquired information to the vehicle control unit 28. The communication unit 27 also acquires information from the vehicle control unit 28 to be transmitted to other communication devices or the like.

[0030] The vehicle control unit 28 includes an ECU (Electronic Control Unit) and controls each part of the vehicle control system 11, as will be described later with reference to FIG.

[0031] The display unit 29 includes, for example, various display devices, and displays various images and information under the control of the vehicle control unit 28. For example, the display unit 29 includes a head-up display or a transmissive display provided in part of the windshield, and displays images and information superimposed in the driver's field of vision. Further, for example, the display unit 29 includes an instrument panel, a display of a car navigation system, etc.

[0032] The audio output unit 30 includes, for example, a speaker, an alarm, a buzzer, etc., and outputs audio information, warning sounds, etc. under the control of the vehicle control unit 28.

[0033] The light emitting unit 31 is equipped with a light emitting device such as an LED (Light Emitting Diode) or a lamp, and under the control of the vehicle control unit 28, lights up or flashes light for the purpose of notifying the driver of various information or alerting them.

[0034] Odor generating unit 32 includes, for example, a generator that generates a bad odor and a deodorizer that eliminates odors, and generates odors and deodorizes the generated odors under the control of vehicle control unit 28. Note that a deodorizer is not necessarily required, and the bad odor may be deodorized by self-neutralization or the like after it has diffused into the air, making the generation of the bad odor temporary.

[0035] The driving control unit 33 controls devices related to the driving of the vehicle among various devices mounted on the vehicle under the control of the vehicle control unit 28. For example, the driving control unit 33 includes an engine control device that controls the operation of the engine, a motor control device that controls the operation of the motor, a brake control device that controls the operation of the brakes, a steering control device that controls the operation of the steering, and the like.

[0036] The in-vehicle device control unit 34 controls various devices mounted on the vehicle other than those related to vehicle driving. For example, the in-vehicle device control unit 34 controls an actuator that controls the tilt of the seat, an actuator that vibrates the seat, an actuator that vibrates the steering wheel, etc.

[0037] The storage unit 35 stores programs and data required for processing by the vehicle control system 11. For example, the storage unit 35 stores a log relating to the running of the vehicle, a facial image and recognition / identification extraction information used for authenticating the driver, learning results of various characteristics of the driver, etc. It is not necessary to store all the information in the storage unit 35, and for example, the information may be transmitted to a remote server or the like via the communication unit 27 and stored therein.

[0038] <Configuration Example of Driver Monitoring Unit 26 and Vehicle Control Unit 28> FIG. 2 shows an example of the configuration of the driver monitoring unit 26 and the vehicle control unit 28 of the vehicle control system 11.

[0039] The driver monitoring unit 26 includes a driver image capturing unit 101, a biometric information acquiring unit 102, a gaze detecting unit 103, and an authentication unit 104.

[0040] The driver photographing unit 101 is equipped with a photographing device and photographs the driver. The photographing range of the driver photographing unit 101 includes at least the area above the neck of the driver while driving, and may include a wider range. The driver photographing unit 101 supplies the image obtained by photographing (hereinafter referred to as the driver image) to the vehicle control unit 28. When photographing the driver, in order to obtain more accurate and specific information, the driver may be illuminated by a dedicated light source, such as a light source that emits structured light or a light source of a specific wavelength including infrared light.

[0041] The biometric information acquisition unit 102 includes sensors and the like that detect various types of biometric information of the driver. The biometric information acquired by the biometric information acquisition unit 102 includes, for example, pulse rate, electrocardiogram, body temperature, body odor, skin temperature, respiratory condition, alcohol content, etc. The biometric information acquisition unit 102 supplies the acquired biometric information of the driver to the vehicle control unit 28.

[0042] The gaze detection unit 103 detects the driver's facial direction, gaze direction, blinking, and eye movement (for example, fixation, saccade, etc.) based on the driver image. The gaze detection unit 103 further performs an assessment of the driver's attention to the outside world and an analysis of the driver's alertness based on dynamic analysis of the gaze, and supplies the detection results and analysis results to the vehicle control unit 28.

[0043] The authentication unit 104 authenticates the driver based on, for example, iris authentication or fingerprints from a driver image or gaze analysis image, and further based on the individual pulse waveform included in the pulse wave. The authentication unit 14 supplies the authentication result to the vehicle control unit 28.

[0044] The vehicle control unit 28 includes a periphery monitoring unit 121, a driver monitoring unit 122, an automatic driving control unit 123, a notification control unit 124, a penalty application unit 125, a log generation unit 126, and a learning unit 127.

[0045] The periphery monitoring unit 121 monitors the periphery of the vehicle based on the periphery image from the periphery photographing unit 21 , the periphery information from the periphery information acquiring unit 22 , and various types of information from the communication unit 27 .

[0046] The driver monitoring unit 122 monitors the driver based on the vehicle information from the vehicle information acquisition unit 25, the driver image from the driver photographing unit 101, the driver's biometric information from the biometric information acquisition unit 102, the detection result from the gaze detection unit 103, the authentication result from the authentication unit 104, and the learning result from the learning unit 127. The driver monitoring unit 122 includes a driving behavior analysis unit 141, a driver state detection unit 142, and a driving intervention level detection unit 143.

[0047] The driving behavior analysis unit 141 analyzes the driver's driving behavior (for example, the characteristics and properties unique to the authenticated driver, such as operations and behaviors related to driving) based on the driver image, vehicle information, and learning results by the learning unit 127.

[0048] The driver state detection unit 142 detects the state of the authenticated unique driver based on the driver image, the driver's biometric information, the detection results by the gaze detection unit 103, the authentication results by the authentication unit 104, and the learning results by the learning unit 127, etc.

[0049] The driving intervention level detection unit 143 detects the driving intervention level, which indicates the degree to which the driver is involved in driving, based on the analysis results of the driver's driving behavior, the detection results of the driver's state, and the automation level set by the automation level setting unit 152.

[0050] The autonomous driving control unit 123 controls autonomous driving. The autonomous driving control unit 123 includes a route setting unit 151, an automation level setting unit 152, and a driving assistance control unit 153.

[0051] The route setting unit 151 corrects the current position of the vehicle measured by the position measurement unit 23, based on the acceleration and angular velocity of the vehicle included in the vehicle information of the vehicle information acquisition unit 25. The route setting unit 151 also sets a driving route to the destination input via the input unit 24, based on the surrounding information from the surrounding information acquisition unit 22, the LDM, map data and map update information acquired via the communication unit 27, and the map data stored in the memory unit 35.

[0052] The automation level setting unit 152 sets the distribution of automation levels on the driving route based on surrounding information from the surrounding information acquisition unit 22, and LDM and traffic information acquired via the communication unit 27. The automation level setting unit 152 also sets the automation level based on the distribution of automation levels, user settings input via the input unit 24, and instructions from the penalty application unit 125, etc.

[0053] Here, the automation level indicates the level of automated driving, in other words, the degree of automation of driving, and will be described in detail later with reference to FIG.

[0054] The driving assistance control unit 153 controls the driving control unit 33 according to the set automation level to assist the driver in driving, thereby realizing autonomous driving. For example, the driving assistance control unit 153 performs driving assistance such as ACC (Adaptive Cruise Control), LKAS (Lane Keep Assist System), TJA (Traffic Jam Assist), and AEBS (Advanced Emergency Braking System). Furthermore, in driving sections with automation level 3 (described below) or higher, the driving assistance control unit 153 may perform driving assistance such as more advanced and complex control than the above driving assistance (e.g., overtaking including lane changes) or autonomous driving with advanced situational judgment including pedestrians and bicycles in urban areas.

[0055] The notification control unit 124 controls the display unit 29, the audio output unit 30, and the light emitting unit 31 to notify the driver of various information, call attention, etc. The notification control unit 124 may also notify the driver of various information, call attention, etc., using, for example, an actuator controlled by the in-vehicle device control unit 34.

[0056] Based on the detection result of the driving intervention level, the penalty application unit 125 controls the display unit 29, the audio output unit 30, the light emission unit 31, the odor generation unit 32, the driving control unit 33, and the in-vehicle device control unit 34 to apply a penalty to the driver as necessary. In addition, the penalty application unit 125 controls the log generation unit 126 to record information for applying the penalty in a log.

[0057] Here, a penalty is, for example, a penalty that is imposed on a driver to the extent that it does not impede driving, in order to correct the driver's level of involvement in driving. Examples of methods for imposing a penalty on a driver include making the driver feel uncomfortable or reducing convenience for the driver.

[0058] The log generating unit 126 generates and updates a log that records various events that occur in the vehicle. The log generating unit 126 stores the generated or updated log in the storage unit 35.

[0059] <Example of installation of part of vehicle control system 11> FIG. 3 shows an example of the installation of a part of the vehicle control system 11.

[0060] The driver monitoring unit 26 is provided, for example, near the dashboard in front of the steering wheel 171.

[0061] The odor generating unit 32 is provided, for example, in an air conditioner provided on the dashboard.

[0062] An actuator for vibrating the steering wheel 171 is provided inside the steering wheel 171 under the control of the in-vehicle device control unit .

[0063] An actuator for vibrating the seat 172 under the control of the in-vehicle device control unit 34 is provided inside the driver's seat 172 .

[0064] <Examples of automation levels> Figure 4 shows examples of automation levels, as defined by the Society of Automotive Engineers (SAE).

[0065] There are six automation levels, ranging from level 0 to level 5.

[0066] Automation level 0 is referred to as "no driving automation." At automation level 0, the driver performs all driving tasks.

[0067] Automation level 1 is called "driver assistance." At automation level 1, the automated driving system (hereinafter simply referred to as the system) controls the vehicle in either the forward / backward or left / right directions. Perform subtasks of the rotation task.

[0068] Level 2 automation is referred to as "partial driving automation." At level 2 automation, the system performs subtasks of the driving task related to both forward / backward and left / right vehicle control.

[0069] Automation level 3 is called "conditional driving automation." At automation level 3, the system performs all driving tasks within a limited area. Therefore, the driver can The driver can perform tasks and activities other than driving, such as operating the mobile terminal 12, holding a conference call, watching videos, playing games, thinking, talking with other passengers, etc. However, during a backup response (fallback) due to a system failure or a worsening driving environment, the driver is expected to respond appropriately to system requests, such as by performing driving operations.

[0070] Automation level 4 is called "highly automated driving." At automation level 4, the system performs all driving tasks within a limited area. Also, during backup (fallback) The driver is not expected to take any action such as operating the vehicle during the emergency. Therefore, the driver can take a nap while the vehicle is running.

[0071] Automation level 5 is referred to as "full driving automation." At automation level 5, the system performs all driving tasks in all areas. Furthermore, the driver is not expected to take any measures, such as operating the vehicle during backup (fallback) situations. Furthermore, at automation level 5, it is anticipated that the driver will not be present in the vehicle at all times while driving, so the driver cannot be expected to drive the entire journey. Therefore, fallback is not permitted, and the system must be able to handle all abnormalities, or a complete route must be developed to prevent unexpected situations from occurring.

[0072] Therefore, at automation levels 0 to 2, the driver performs all or part of the driving task, and is responsible for monitoring and responding to safety driving. At these three automation levels, the driver is required to have the ability to always return to driving if necessary. Therefore, the driver is not permitted to engage in secondary tasks other than driving that would reduce their attention while driving or impair their focus on the road ahead.

[0073] On the other hand, at automation levels 3 to 5, the system performs all driving tasks and is responsible for monitoring and responding to safety driving. However, at automation level 3, the driver may be required to perform driving operations. Also, there may be sections of the driving route where automation levels 3 and 4 cannot be applied, and in such sections, automation level 2 or lower is set and the driver must intervene in driving.

[0074] In the following, driving that requires the driver to intervene in some way and directly influence the driving of the vehicle will be referred to as manual driving. Therefore, manual driving is performed at automation levels 0 to 2.

[0075] Meanwhile, hereinafter, driving that does not require any driver intervention will be referred to as autonomous automated driving. Therefore, at automation levels 3 to 5, autonomous automated driving is basically performed. However, at automation level 3, manual driving may be required depending on the system's requirements. In other words, at automation level 3, the driver's withdrawal from driving operations must be limited.

[0076] Furthermore, by enabling the driver to safely and smoothly return to manual driving in necessary sections, it is possible to extend routes that contain a mixture of sections where autonomous driving is possible and sections where manual driving is required. Also, by preventing the driver from completely withdrawing from driving operations and enabling them to safely and smoothly return to manual driving, it becomes possible to implement autonomous driving in key sections of the driving route.

[0077] <Automatic driving control processing> Next, the automatic driving control process executed by the vehicle control system 11 will be described with reference to the flowcharts of Figures 5 and 6. This process is started, for example, when the power (ignition) switch of the vehicle is turned on.

[0078] In step S1, the driver monitoring unit 26 authenticates the driver. Specifically, the driver capturing unit 101 of the driver monitoring unit 26 captures an image of the driver. The authentication unit 104 recognizes the face of the driver in the captured image of the driver. Then, for example, the authentication unit 104 identifies the driver by searching for a face image that matches the face of the driver from among face images of one or more users stored in the memory unit 35. If the authentication unit 104 can identify the driver, it determines that the authentication has been successful, and if the authentication unit 104 cannot identify the driver, it determines that the authentication has been unsuccessful. The authentication unit 104 supplies the driver authentication result to the vehicle control unit 28.

[0079] For example, if authentication of the driver fails, the vehicle may be prohibited from traveling. In this case, the vehicle may be permitted to travel if the driver performs a predetermined operation in a secure environment to register as a new user.

[0080] However, the main purpose of authenticating a driver is to correlate the characteristics of the driving operation of the authenticated driver with the driver's state, and to control the vehicle or apply a penalty accordingly. Therefore, the authentication result does not necessarily need to be used to control whether the vehicle is permitted to travel. This allows the vehicle to travel in an unauthorized state, for example, in an emergency. Note that the fact that the vehicle is traveling in an unauthorized state may be notified to those around it by, for example, a warning light or vehicle-to-vehicle communication.

[0081] In step S2, the log generating unit 126 starts recording a log. Details of the log will be described later with reference to FIG.

[0082] In step S3, the vehicle control unit 28 acquires a destination. Specifically, a vehicle occupant (not necessarily the driver) inputs the destination via the input unit 24. The input unit 24 supplies information indicating the acquired destination to the vehicle control unit 28.

[0083] In step S4, the vehicle control system 11 starts acquiring surrounding information.

[0084] For example, the surroundings photographing unit 21 starts photographing the traveling direction of the vehicle and the surroundings, and supplies the surroundings images obtained by photographing to the vehicle control unit .

[0085] The surrounding information acquisition unit 22 starts acquiring surrounding information relating to the environment and objects around the vehicle, and supplies the surrounding information to the vehicle control unit 28 .

[0086] The vehicle information acquisition unit 25 starts acquiring vehicle information and supplying it to the vehicle control unit 28.

[0087] The position measurement unit 23 starts measuring the current position of the vehicle and supplying the measurement result to the vehicle control unit 28 .

[0088] The communication unit 27 starts receiving an LDM (Local Dynamic Map) from an ITS spot (not shown) and supplying the LDM to the vehicle control unit 28. The communication unit 27 also starts receiving map data and the like from a server (not shown) and supplying the map data and the like to the vehicle control unit 28. Note that the map data may be stored in advance in the storage unit 35, and the vehicle control unit 28 may acquire the map data from the storage unit 35. The communication unit 27 also starts receiving various types of traffic information from a roadside unit (not shown) and supplying the traffic information to the vehicle control unit 28.

[0089] In the following, information relating to maps such as LDM and map data will be collectively referred to as map information.

[0090] The periphery monitoring unit 121 starts monitoring the periphery of the vehicle based on the periphery image from the periphery photographing unit 21 , the periphery information from the periphery information acquiring unit 22 , and various information from the communication unit 27 .

[0091] The route setting unit 151 appropriately corrects the current position of the vehicle based on the information acquired from the periphery monitoring unit 121 and the vehicle acceleration, angular velocity, etc. included in the vehicle information supplied from the vehicle information acquisition unit 25. This corrects, for example, an estimation error in the current position of the vehicle due to information in the map information that does not reflect changes over time, an error in the position measurement unit 23, etc.

[0092] In step S5, the route setting unit 151 starts setting a travel route. Specifically, the route setting unit 151 sets a travel route from the current location to the destination based on map information, taking into consideration the driver's driving ability, etc. The route setting unit 151 also changes the travel route as needed, based on information such as the time of day, weather to the destination, traffic congestion, and traffic regulations.

[0093] In step S6, the automation level setting unit 152 starts updating the automation level.

[0094] Specifically, the automation level setting unit 152 sets the distribution of permissible automation levels (hereinafter referred to as "permissible automation levels") along the driving route based on map information, surrounding area information, etc. Here, the permissible automation level refers to the maximum automation level that can be set in the target section. For example, in a section where the permissible automation level is level 3, the vehicle can be driven at automation level 3 or lower.

[0095] For example, the automation level setting unit 152 sets the distribution of allowable automation levels on the driving route to a default value indicated in map information, etc. Then, the automation level setting unit 152 appropriately updates the distribution of allowable automation levels on the driving route based on information about the environment on and around the driving route, such as weather, road conditions, accidents, construction, and traffic regulations, obtained from the map information and surrounding information.

[0096] For example, in sections where road markings such as road studs, paint, and stones on the road surface, symbols, and road markings such as letters are difficult to recognize due to snow accumulation, the allowable automation level may be lowered from the original level 3 to level 2, and the use of LKAS may be prohibited. For example, in sections where visibility is poor due to smoke from a fire or thick fog, the allowable automation level may be lowered from the original level 3 to level 2, and the maximum speed may be restricted. For example, in sections where an accident has occurred or a fallen object has been detected, the allowable automation level may be lowered to level 1 or level 0. For example, in sections where the road surface is frozen or on bridges with strong crosswinds, the speed limit may be lowered and the allowable automation level may be lowered to level 1 or level 0.

[0097] Now, with reference to FIG. 7, a specific example of the distribution of allowable automation levels will be described.

[0098] The section S1 between points P1 and P2 has updated LDM information, is a straight section with clear road markings, and has an allowable automation level set to level 3.

[0099] The section S2 between points P2 and P3 has updated LDM information, but is a section where sudden departure accidents occur frequently, and the allowable automation level is set to level 2.

[0100] The section S3 between points P3 and P4 has updated LDM information, is a straight section with clear road markings, and has an allowable automation level set to level 3.

[0101] The section S4 between points P4 and P5 is a section where LDM updates are delayed or insufficient and road surfaces may freeze during winter nights, and the allowable automation level is set to level 1.

[0102] In addition, near the boundary between section S3, which includes point P4, and section S4, an emergency parking lane 201 is provided for vehicles whose drivers are unable to return to driving operations when transitioning from automation level 3 to automation level 2 or lower, and for parking broken-down vehicles, emergency vehicles, etc.

[0103] The section S5 between points P5 and P6 has updated LDM information, but it is a section with many parked vehicles and poor visibility, and the allowable automation level is set to level 1.

[0104] The section S6 between points P6 and P7 has updated LDM information, is a straight section with clear road markings, and has an allowable automation level set to level 3.

[0105] Within section S6, a service area 202 is provided where autonomous vehicles can take refuge in an emergency.

[0106] In the section S7 between points P7 and P8, LDM updates are delayed or insufficient, the safety level is undefined, and the allowable automation level is set to level 1.

[0107] An emergency parking lane 203 is provided within a predetermined range from point P7 in section S7.

[0108] The section S8 between points P8 and P9 has updated LDM information, is a straight section with clear road markings, and is a section for which the allowable automation level is normally set to level 3. However, section S8 is currently experiencing poor visibility due to bad weather, and the allowable automation level is restricted to level 2.

[0109] In addition, even in sections where automation level 3 is normally available, emergency parking lane 204 is provided along the way in sections where there is a possibility that the automation level may be frequently changed to a lower level due to weather conditions, etc., such as section S8. Also, for example, if a decrease in the driver's driving performance level is predicted before entering point P9, emergency parking lane 204 is used as an advance evacuation point for the vehicle.

[0110] Section S9 between points P9 and P10 is a section where automated driving is prohibited, and the allowable automation level is set to level 0.

[0111] The section S10 between points P10 and P11 has updated LDM information, is a straight section with clear road markings, and has an allowable automation level set to level 3.

[0112] A service area 205 where autonomously driven vehicles can take refuge in an emergency is provided near point P10 within section S10. Service area 202 and service area 205 are also used, for example, as places where vehicles are forced to stop (described later).

[0113] Then, the automation level setting unit 152 starts the process of appropriately updating the automation level of the vehicle based on the distribution of allowable automation levels on the driving route and the current location.

[0114] For example, the automation level setting unit 152 automatically varies the automation level of the vehicle in accordance with the allowable automation level. Alternatively, for example, the automation level setting unit 152 may automatically lower the automation level in sections where the allowable automation level is lower than the automation level of the vehicle, and may raise the automation level in response to a user operation in sections where the allowable automation level is higher than the automation level of the vehicle.

[0115] In step S7, the vehicle control system 11 starts monitoring the driver.

[0116] Specifically, the driver photographing unit 101 of the driver monitoring unit 26 starts photographing the driver and supplying the driver image obtained by photographing to the vehicle control unit 28.

[0117] The biometric information acquisition unit 102 starts acquiring the biometric information of the driver and supplying it to the vehicle control unit 28.

[0118] The gaze detection unit 103 starts detecting the driver's facial direction, gaze direction, blinking, and eye movement (e.g., fixation, saccade, etc.) based on the driver image, and supplies the detection results to the vehicle control unit 28.

[0119] The driving behavior analysis unit 141 starts analyzing the driving behavior of the driver based on the driver image, the vehicle information, the learning result by the learning unit 127, and the like.

[0120] The driver state detection unit 142 starts detecting the driver state based on the driver image, the driver's biometric information, the detection result by the gaze detection unit 103, the authentication result by the authentication unit 104, and the learning result by the learning unit 127, etc.

[0121] For example, the driver state detection unit 142 starts detecting the driver's posture, behavior, and the like.

[0122] Further, for example, the driver state detection unit 142 detects the responsiveness and alertness of the driver. Here, the responsiveness of the driver is defined based on, for example, whether or not the driver responds to external requests, instructions, and stimuli, as well as obstacles in the vehicle's traveling direction, the speed of the response, and the appropriateness of the response. The responsiveness of the driver decreases not only when the driver's alertness decreases, but also when the driver's attention is not focused on driving, or when the driver does not respond intentionally, etc.

[0123] Methods for detecting the driver's reactivity and alertness include, for example, passive monitoring and active monitoring.

[0124] Passive monitoring detects the driver's responsiveness and alertness by passively observing the driver's state.

[0125] For example, the reactivity and alertness of the driver are detected based on the driver's movements such as transitions in face direction, transitions in gaze direction, blink frequency, transitions in eye movement, etc. For example, the driver's attention, such as gaze movement and fixation to an object, is observed based on the driver's gaze movement in the direction of information obtained by the peripheral imaging unit 21, the peripheral information acquisition unit 22, etc., and the reactivity and alertness of the driver are detected based on the results.

[0126] For example, the driver's level of alertness is detected based on biological information such as the driver's heart rate and body odor.

[0127] For example, changes in the driver's responsiveness and alertness are detected by observing the time-series changes in the driver's driving operations, such as the steering stability and operation speed of the steering wheel, and the operation stability and operation speed of the accelerator pedal and brake pedal. Note that, since these driver reactions have unique characteristics for each driver, the characteristics according to the driver's situation may be learned, and the alertness of the driver may be detected based on the learning results.

[0128] Active monitoring detects the driver's responsiveness and alertness by providing visual, auditory, tactile, and other stimuli and instructions to the driver and observing the driver's response to the stimuli and instructions. For example, active monitoring is used when it is difficult to detect the driver's responsiveness and alertness using passive monitoring or to improve detection accuracy. For example, at automation level 3 or higher, the driver's involvement in the steering device may be completely cut off, and even if the steering device operation status is monitored, the driver's response cannot be detected. Therefore, active monitoring is an effective means to reliably grasp the driver's state. In other words, active monitoring has a function to complement passive monitoring. Also, active monitoring is used, for example, to stimulate the driver and wake them up.

[0129] For example, the driver state detection unit 142 detects the driver's responsiveness and alertness by controlling the display unit 29 to display short words or numbers within the driver's field of vision and have the driver read them aloud, or by displaying simple mathematical formulas and having the driver speak the calculation results.

[0130] For example, the driver state detection unit 142 controls the display unit 29 to display a pseudo target within the driver's field of vision as a target for the driver's gaze, and detects the driver's responsiveness and alertness by tracking the driver's gaze movement.

[0131] For example, the driver state detection unit 142 controls the audio output unit 30 to give the driver simple instructions (such as shaking the head from side to side), and detects the driver's responsiveness and alertness by observing the driver's reaction to the instructions.

[0132] For example, the driving assistance control unit 153 controls the driving control unit 33 in accordance with instructions from the driver state detection unit 142 to make the vehicle drive unnaturally within a range that ensures safety. Then, the driver state detection unit 142 detects the responsiveness and alertness of the driver based on the driver's reaction to the unnatural driving.

[0133] For example, the driving control unit 33 controls the steering wheel so that the vehicle's direction of travel is unbalanced. For example, when the steering wheel is not being operated, the driving control unit 33 causes the vehicle to meander for a predetermined period of time by changing the direction of the wheels or applying unbalanced braking loads to the left and right. In response to this, the driver state detection unit 142 detects the driver's responsiveness and alertness based on whether the driver operates the steering wheel to correct the meandering and on the driver's reaction speed, etc. Note that the amount of meandering of the vehicle is desirably set within a range in which the driver unconsciously performs driving operations to correct the meandering. Furthermore, for example, when the vehicle is traveling normally along a lane, the in-vehicle device control unit 34 applies a rotational load to the steering wheel equivalent to a simulated vehicle meandering. In response to this, the driver state detection unit 142 detects the driver's responsiveness and alertness based on whether the driver operates the steering wheel to stop the rotation and on the driver's reaction speed, etc.

[0134] For example, if the driver continues to meander without reacting, the following vehicle can be notified that some kind of danger may occur due to a decrease in the driver's reactivity or alertness, etc. Furthermore, if the driver continues to meander without reacting, an abnormality may be notified to the outside via the communication unit 27, etc.

[0135] For example, the driving control unit 33 changes the vehicle's direction of travel in a direction that slightly deviates from the lane for a predetermined period of time. On the other hand, if the driver is paying normal attention to the road ahead, it is expected that the driver will steer the vehicle to correct its direction. Therefore, the driver state detection unit 142 detects the driver's responsiveness and alertness based on whether the driver operates the steering wheel to return the vehicle to the lane, as well as the driver's reaction speed, etc. However, if the vehicle's direction of travel is changed unconditionally, a dangerous situation may occur depending on the positional relationship with surrounding vehicles. Furthermore, there is a possibility that a following vehicle is tailgating the vehicle. Therefore, it is desirable to implement this detection method within a range that does not adversely affect surrounding vehicles, by comprehensively assessing a combination of factors such as the state of surrounding vehicles and the psychological impact on the driver.

[0136] For example, when the ACC is enabled, the driving assistance control unit 153 sets the distance from the preceding vehicle to be longer than normal. In response to this, the driver state detection unit 142 detects the driver's responsiveness and alertness based on whether the accelerator pedal is operated to return the distance to the normal length and on the driver's reaction speed.

[0137] For example, the driving control unit 33 may change the amount of change in the vehicle's direction of travel in response to the steering amount of the steering wheel more or less than normal. In response to this, the driver state detection unit 142 detects the driver's responsiveness and alertness based on whether the steering wheel is operated to adjust the vehicle's direction of travel to a desired direction and on the driver's reaction speed. Note that the difference in the amount of change in the vehicle's direction of travel compared to normal conditions is preferably set within a range in which the driver unconsciously performs driving operations to correct the vehicle's direction of travel.

[0138] For example, the cruise control unit 33 increases or decreases the vehicle acceleration relative to normal in response to the amount of depression of the accelerator pedal. In response to this, the driver state detection unit 142 detects the driver's responsiveness and alertness based on whether the accelerator pedal is operated to adjust the vehicle speed to a desired speed and on the driver's reaction speed. It is desirable to set the difference in vehicle acceleration relative to normal within a range in which the driver unconsciously performs driving operations to correct the acceleration.

[0139] For example, the cruise control unit 33 increases or decreases the deceleration of the vehicle relative to the normal deceleration in response to the amount of depression of the brake pedal. In response to this, the driver state detection unit 142 detects the driver's responsiveness and alertness based on whether the brake pedal is operated to adjust the vehicle speed to a desired speed and on the driver's reaction speed. It is desirable to set the difference in vehicle deceleration relative to the normal deceleration within a range in which the driver unconsciously performs driving operations to correct the deceleration.

[0140] For example, when the vehicle is performing autonomous driving and there is no need for the driver to intervene in driving, and the driver is operating the mobile terminal 12 (information processing device), the driver state detection unit 142 displays a subwindow showing instructions to the driver on the screen of the mobile terminal 12 via the communication unit 27. Then, the driver state detection unit 142 detects the responsiveness and alertness of the driver based on whether or not the driver has responded normally to the instructions, and the reaction speed, etc.

[0141] For example, if a driver is looking ahead but is distracted by other things, it may be difficult to detect the driver's responsiveness and alertness using passive monitoring. In contrast, active monitoring improves the accuracy of detecting the driver's responsiveness and alertness.

[0142] In addition to the above, other types of driver states, such as state of consciousness, mental state, state of tension, and degree of drug influence, may also be detected.

[0143] The driving intervention level detection unit 143 starts detecting the driving intervention level of the driver based on the analysis result of the driving behavior and the detection result of the driver's state.

[0144] In step S8, the learning unit 127 starts the learning process.

[0145] For example, the learning unit 127 starts learning the correlation between the driver's driving ability and various detectable driver states based on the analysis results of the driving behavior analysis unit 141.

[0146] For example, the learning unit 127 starts learning biometric information, driver movements, and the driver's driving operation tendencies when the driver is performing normal manual driving. For example, the learning unit 127 constantly learns the correlation between the driver's unique characteristics according to driving conditions and normal driving characteristics when awake, such as the driver's gaze behavior, head posture, body posture, pulse waveform, respiratory state, and pupil response to external light, when the driver is performing stable driving operations such as stably driving in the center of the lane, stably stopping the vehicle at stop lights, and appropriately decelerating around curves. By using the learning results, for example, the accuracy of passive monitoring can be improved.

[0147] For example, the learning unit 127 starts learning the driver's reaction characteristics to active monitoring so that it can distinguish between normal and abnormal conditions. By using the learning results, for example, the accuracy of active monitoring is improved.

[0148] For the above learning, any learning method can be used, from simple correlation learning to complex artificial intelligence learning such as CNN (Convolutional Neural Network). In this way, the driver's unique characteristics are learned according to each state, so that the driver's driving ability can be detected more accurately based on the authenticated driver's state (for example, the driver's health state, degree of fatigue, etc.).

[0149] The learning unit 127 then stores the learning results in the storage unit 35. The learning results may not only be stored in the vehicle that was used and reused, but may also be stored separately from the vehicle in an electronic key or a remote server, etc., so that they can be easily used in another vehicle. In addition, in a vehicle that is used repeatedly by a driver, the learning results from the previous use may be imported, their obsolescence may be determined, and the learning dictionary obtained up to the previous use may be used as initial data.

[0150] In step S9, the driving assistance control unit 153 starts driving assistance. That is, the driving assistance control unit 153 starts processing to assist driving, such as ACC, LKAS, TJA, and AEBS, by controlling the driving control unit 33 in accordance with the current automation level.

[0151] In step S10, the driver intervention level detection unit 143 determines whether the driver intervention level is insufficient.

[0152] The level of driver intervention required here varies depending on the automation level that is set.

[0153] When automation level 0 is set, as mentioned above, the driver must perform all driving tasks. Therefore, the driver's level of driving intervention is required to be able to direct their gaze and attention to the direction of travel of the vehicle and to be able to perform the necessary driving operations.

[0154] When automation level 1 is set, as mentioned above, the driver is required to perform some driving tasks. Therefore, the driver's level of driving intervention is required to be able to direct their eyes and attention to the direction of travel of the vehicle and perform the necessary driving operations. Here, the required level of driving intervention is almost the same as for automation level 0, except that the number of required driving operations is reduced.

[0155] When automation level 2 is set, as mentioned above, the driver is required to perform some driving tasks. Therefore, the driver's level of driving intervention is required to be able to direct their eyes and attention to the direction of travel of the vehicle and perform the necessary driving operations. Here, the required level of driving intervention is almost the same as for automation level 0 and automation level 1, except that the number of required driving operations is reduced.

[0156] When the vehicle is set to automation level 3, as described above, the automated driving system basically performs all driving tasks. Meanwhile, the driver is expected to respond appropriately to system requests, such as by operating the vehicle during a backup response (fallback). Therefore, the driver's level of driving intervention does not require them to direct their gaze or attention to the direction of travel of the vehicle, but they are required to respond to requests from the vehicle within a specified time frame. Furthermore, when transitioning from automation level 3 to a lower level, it is desirable for the driver to return to the level of driving intervention of the target automation level before changing the automation level.

[0157] When set to automation level 4, as described above, the automated driving system performs all driving tasks. Furthermore, during backup measures (fallback), the driver is not expected to take any action, such as operating the vehicle. Therefore, the driver does not generally need to intervene in the driving of the vehicle. However, when transitioning from automation level 4 to a lower level, it is desirable for the driver to return to the driving intervention level of the destination automation level before changing the automation level.

[0158] When set to automation level 5, as described above, the automated driving system performs all driving tasks, so the driver does not need to intervene in driving the vehicle.

[0159] Therefore, when the automation level is set to 4 or 5, the driver does not need to intervene in driving, so it is unconditionally determined that the level of driving intervention is sufficient.

[0160] On the other hand, if the automation level is set to any of automation levels 0 to 3, the driver intervention level detection unit 143 determines that the driver intervention level is insufficient when the driver intervention level does not satisfy the conditions of the current automation level for a predetermined period of time or more, and processing proceeds to step S11.

[0161] For example, when the automation level is set to any of automation levels 0 to 2, if the driver does not direct their gaze or attention in the direction of travel of the vehicle for a predetermined period of time or longer, for example, if the operation of the steering device is delayed more than normal and the vehicle deviates from its trajectory, and a sudden steering operation is performed to correct the trajectory, so-called swaying is detected, it is determined that the level of driving intervention is insufficient, and processing proceeds to step S11.

[0162] For example, if automation level 3 is set, and instructions are given to the driver through active monitoring, but the driver does not respond normally within a specified time, it is determined that the level of driving intervention is insufficient, and processing proceeds to step S11.

[0163] Note that even if there is a section that can be driven at automation level 4, it does not necessarily mean that the entire journey can be driven at automation level 4. Therefore, even when automation level 4 is set, the driver intervention level may be determined using active monitoring, just as when automation level 3 is set, assuming that the automation level will be shifted to a lower level.

[0164] In step S11, the vehicle control system 11 executes a driving intervention level correction process, and then the process proceeds to step S15.

[0165] Here, the details of the driver intervention level correction process will be described with reference to the flowchart of FIG.

[0166] In step S41, the driving intervention level detection unit 143 determines whether or not insufficient driving intervention levels occur frequently. For example, if the number of occurrences of insufficient driving intervention levels since monitoring of the driver began in the process of step S7 in Fig. 5 is less than a predetermined threshold, and if the frequency of occurrences of insufficient driving intervention levels within a predetermined period of time immediately preceding the occurrence (e.g., 10 minutes) is less than the predetermined threshold, the driving intervention level detection unit 143 determines that insufficient driving intervention levels do not occur frequently, and the process proceeds to step S42.

[0167] In step S42, the vehicle control system 11 calls the driver's attention.

[0168] For example, the display unit 29 displays a warning screen or the like to call the driver's attention within the driver's field of vision under the control of the notification control unit 124. At this time, if the driver is operating the mobile terminal 12, for example, the warning screen or the like is displayed on the screen of the mobile terminal 12.

[0169] As a warning screen displayed on the mobile terminal 12, for example, a sub-screen of the route display screen shows a state in which a point prompting the driver to return to manual driving is approaching in step S71 of Fig. 9, which will be described later, etc. This route display screen may be, for example, a two-dimensional map display or a one-dimensional bar display showing the position on the driving route.

[0170] Furthermore, the display mode of the warning screen of the mobile terminal 12 may change depending on the time or distance required to reach the point where it is necessary to return to manual driving. For example, if the time required to reach the point where it is necessary to return to manual driving exceeds five minutes, a warning screen with a small green frame is displayed, and five minutes beforehand the size of the warning screen changes to occupy 10% of the screen of the mobile terminal 12, and two minutes beforehand the size of the warning screen changes to occupy 1 / 3 of the screen of the mobile terminal 12, and the frame of the warning screen changes to red and starts flashing.

[0171] Furthermore, for example, in order to prevent the driver from hastily operating the mobile terminal 12 when a warning screen is displayed, for example, the state of operation may be forcibly saved, and the mobile terminal 12 may be automatically transitioned to a standby state so that the operation can be resumed from the same state later, or the screen of the mobile terminal 12 may be forcibly turned off.

[0172] For example, under the control of the notification control unit 124, the audio output unit 30 outputs audio messages to alert the driver, alarms, buzzers, beeps, a pseudo car horn (horn) sound that can only be heard inside the vehicle, and the like.

[0173] For example, under the control of the notification control unit 124, the light emitting unit 31 lights or flashes a lamp or the like to alert the driver.

[0174] For example, the in-vehicle device control unit 34 performs haptic feedback such as vibrating the driver's seat or steering wheel or pulling the seat belt under the control of the notification control unit 124. When vibrating the seat, the driver may be provided with vibrations similar to those experienced when the vehicle crosses rumble strips or road studs, for example.

[0175] For example, the driving assistance control unit 153 controls the driving control unit 33 in accordance with instructions from the notification control unit 124 to control the steering of the vehicle so that vibrations similar to those experienced when crossing rumble strips or road studs are transmitted to the driver.

[0176] It is desirable that this warning be given in a manner that does not cause discomfort to the driver.

[0177] In step S43, the driver intervention level detection unit 143 determines whether the driver intervention level deficiency has been resolved. If the driver intervention level does not yet satisfy the conditions for the current automation level, the driver intervention level detection unit 143 determines that the driver intervention level deficiency has not been resolved, and the process proceeds to step S45.

[0178] On the other hand, in step S41, if the number of occurrences of insufficient driving intervention level since monitoring of the driver began is equal to or greater than a predetermined threshold, or if the frequency of occurrences of insufficient driving intervention level within the previous predetermined time period is equal to or greater than a predetermined threshold, the driving intervention level detection unit 143 determines that insufficient driving intervention level is occurring frequently, and processing proceeds to step S44.

[0179] In step S44, the penalty application unit 125 determines whether or not the application of a penalty is occurring frequently. For example, if the number of times a penalty has been applied since monitoring of the driver began is less than a predetermined threshold, and if the frequency of application of a penalty within a predetermined period of time immediately preceding the start of monitoring (e.g., 30 minutes) is less than the predetermined threshold, the penalty application unit 125 determines that the application of a penalty is not occurring frequently, and the process proceeds to step S45.

[0180] In step S45, the driver state detection unit 142 determines whether or not the vehicle is in a dangerous state. A dangerous state here refers to a state in which the driver cannot immediately return to the required level of driving intervention, or a state in which the driver cannot drive normally, for example, because the driver is unconscious, deep asleep, or drunk. If it is determined that the vehicle is not in a dangerous state, the process proceeds to step S46.

[0181] In step S46, the vehicle control system 11 applies a penalty.

[0182] Here, there are penalties that are applied while driving and penalties that are applied after driving. More specifically, penalties that are applied while driving are penalties that are applied until the vehicle arrives at the destination or until a forced stop is made, and are hereinafter referred to as in-driving penalties. Penalties that are applied after driving are penalties that are applied after the vehicle arrives at the destination or after a forced stop is made, and are hereinafter referred to as ex-post penalties. Penalties that are applied in this step are in-driving penalties.

[0183] Below, specific examples of driving penalties will be explained.

[0184] Driving penalties include, for example, a physical penalty, a penalty that increases the burden on the driver, a penalty that requires the driver to change the driving route, etc. These are not necessarily clearly distinguished, and one penalty may correspond to multiple types of penalty.

[0185] For example, the penalty application unit 125 may make the driver feel uncomfortable by making the warning given to the driver in step S42 stronger. For example, the volume of a voice message, alarm, buzzer, beep, pseudo horn, etc. may be increased or an unpleasant tone such as a discordant sound may be output, compared to when a warning was given. For example, the light intensity of a lamp or the like used to warn the driver may be increased, compared to when a warning was given. For example, the intensity of haptic feedback may be increased, compared to when a warning was given.

[0186] For example, the penalty application unit 125 controls the in-vehicle device control unit 34 to move the backrest of the driver's seat closer to a direction perpendicular to the seat surface, thereby forcing the driver into an uncomfortable driving posture.

[0187] For example, the penalty application unit 125 controls the odor generation unit 32 to generate a malodor for a short period of time. This malodor is preferably deodorized after it lasts for a short period of time. For example, after a malodorous liquid is sprayed, a deodorizing liquid is sprayed after a predetermined period of time.

[0188] For example, the penalty application unit 125 controls the driving control unit 33 to limit the upper speed limit of the vehicle for a predetermined time (for example, 2 minutes, 5 minutes, or 10 minutes, etc.), which makes the driver feel uncomfortable about not being able to go fast and causes a delay in arrival at the destination.

[0189] For example, the penalty application unit 125 controls the driving control unit 33 to perform discomfort braking control. For example, even if the driver depresses the brake pedal, the vehicle does not decelerate smoothly, but the deceleration increases and decreases, forcing the driver to drive in a manner that is uncomfortable for the driver. However, if there is a following vehicle, it is desirable to avoid affecting the following vehicle.

[0190] For example, the penalty application unit 125 instructs the route setting unit 151 to change the driving route to a route different from the ideal route. As a result, for example, the driving route is changed to a detour route so as to delay arrival at the destination. Also, for example, the driving route is changed to take a route that requires manual driving. Or, for example, the driving route is changed to avoid a route where driving at automation level 3 or higher is permitted. This increases the burden on the driver and delays arrival at the destination.

[0191] For example, the penalty application unit 125 instructs the driving assistance control unit 153 to lower the automation level. As a result, the automation level is lowered to level 2 or higher, increasing the burden on the driver. Also, for example, the penalty application unit 125 instructs the driving assistance control unit 153 to stop some driving assistance functions (for example, ACC, LKAS, etc.).

[0192] This reduction in automation level or restriction of driving assistance functions may be forcibly continued for a predetermined period of time, or, for example, it may be possible to make it impossible to return to the original automation level without going through complex operations or thought routines. In the latter case, the effect of awakening the driver can also be expected.

[0193] In order to prevent drivers from becoming accustomed to penalties, it is desirable that the type, degree, duration, etc. of the penalty applied be changed appropriately when the penalty is applied repeatedly.

[0194] Furthermore, for example, penalties may be ranked according to the degree of disadvantage suffered by the driver. Furthermore, even for the same type of penalty, the ranking may be based on the degree, duration, etc. In this way, for example, the greater the disadvantage suffered by the driver, the higher the penalty rank.

[0195] For example, the rank of the penalty to be applied may be increased as the number of times the penalty is applied increases.

[0196] Also, for example, the lower the driver's reactivity or alertness, the higher the rank of the penalty to be applied. Furthermore, in accordance with the higher rank of the penalty to be applied, the point at which the driver is prompted to return to manual driving, for example, in step S71 of FIG. 9 (described later), may be brought forward. This produces a penalty effect that shortens the autonomous automatic driving section. Also, the transition period until returning to manual driving is lengthened, making the transition to manual driving safer.

[0197] In step S47, similar to the processing in step S43, it is determined whether the insufficient driver intervention level has been resolved. If the driver intervention level satisfies the conditions for the current automation level, it is determined that the insufficient driver intervention level has been resolved, and processing proceeds to step S48. In other words, if the driver intervention level has been corrected by applying a penalty, processing proceeds to step S48.

[0198] If a penalty is still being applied, it will cease to be applied at this point.

[0199] On the other hand, if the driving intervention level satisfies the conditions for the current automation level in step S43, it is determined that the insufficient driving intervention level has been resolved, and the process proceeds to step S48. In other words, if the driving intervention level has been corrected by calling the driver's attention, the process proceeds to step S48.

[0200] If the driver is still being warned, the warning will stop at this point.

[0201] In step S48, the driving assistance control unit 153 temporarily restricts the driving operation as necessary.

[0202] For example, if a driver wakes up from a doze and, in a state of semi-consciousness, hastily operates the steering wheel, accelerator pedal, or brake pedal, the vehicle may be turned inappropriately, accelerated suddenly, or stopped suddenly, which may lead to an accident.

[0203] Therefore, the driving assistance control unit 153 temporarily imposes restrictions on driving operations depending on the driver's state before the driving intervention level is corrected. For example, when the driver wakes up from a state with a very low level of alertness, such as drowsiness, the driving assistance control unit 153 temporarily disables the driver's operation of the steering wheel, accelerator pedal, or brake pedal, and continues autonomous driving during that time. Then, the driving assistance control unit 153 imposes on the driver an operation or action to enable the operation of the steering wheel, accelerator pedal, or brake pedal.

[0204] For example, the notification control unit 124 controls the display unit 29 or the audio output unit 30 in accordance with an instruction from the driving assistance control unit 153 to notify the driver of an operation or action to be performed by an image or sound.

[0205] For example, instructions such as "Place your foot on the accelerator pedal (or brake pedal) and lightly press it down," "Repeat lightly pressing the accelerator pedal (or brake pedal) 10 times," "Lightly press the accelerator pedal (or brake pedal) in time with the light on," "Hold the steering wheel and turn it lightly left and right," etc. may be given to the driver. Alternatively, instructions for operations and actions similar to those given when the driver's reactivity and alertness are detected by the above-mentioned active monitoring may be given to the driver.

[0206] Then, if the driver is able to perform the specified operations and actions in accordance with the instructions successfully, the driving assistance control unit 153 enables the driver to operate the steering wheel, accelerator pedal, or brake pedal, and transitions from autonomous driving to manual driving.

[0207] In this way, the driving operation is restricted until the driver responds appropriately to the given instructions.

[0208] Thereafter, the driving intervention level correction process ends.

[0209] On the other hand, if it is determined in step S47 that the insufficient driver intervention level has not been resolved, that is, if the driver intervention level has not been corrected despite the application of the penalty, the process proceeds to step S49.

[0210] Also, in step S44, if the number of times a penalty has been applied since monitoring of the driver began is equal to or greater than a predetermined threshold, or if the frequency of application of a penalty within the previous predetermined time period is equal to or greater than a predetermined threshold, the penalty application unit 125 determines that penalties have been applied frequently, and processing proceeds to step S49.

[0211] In step S49, the automation level setting unit 152 determines whether the vehicle is currently in autonomous automatic driving. If the current automation level is level 2 or lower, the automation level setting unit 152 determines that the vehicle is not currently in autonomous automatic driving, and the process proceeds to step S50.

[0212] If it is determined in step S45 that the situation is dangerous, the process proceeds to step S50.

[0213] In step S50, the vehicle control system 11 performs a forced stop process. For example, in the case of an emergency, the route setting unit 151 searches for the nearest location on the driving route where an emergency vehicle or the like can park (for example, an emergency parking lane, a safety zone, a store parking lot, or the like) based on map information, and sets the detected location (for example, the emergency parking lane 201 in FIG. 7, or the like) as the location where the vehicle is to be forced to stop (hereinafter referred to as a forced stop location).

[0214] Furthermore, for example, when the need is not particularly urgent, the route setting unit 151 searches for the nearest parking area or service area (for example, service area 202 in FIG. 7 ) based on map information. When there is a parking area or service area within a predetermined range and it is possible to reach there without taking a route that requires manual driving, the route setting unit 151 sets the parking area or service area as a forced stop location. On the other hand, when there is no parking area or service area within the predetermined range, or when it is not possible to reach the parking area or service area without taking a route that requires manual driving, the route setting unit 151 searches for and sets a forced stop location in the same manner as in the case of an emergency.

[0215] The driving assistance control unit 153 controls the traveling control unit 33 and the like to stop the vehicle at the set forced stopping location. At this time, the vehicle is decelerated or driven slowly as necessary. Furthermore, an SOS is sent as necessary after the vehicle has stopped.

[0216] It is also possible that the driver may forcefully switch to manual driving and resume driving before the vehicle stops at a mandatory stopping point. In this case, since the driver may not be fully awake, it is desirable to gradually switch to manual driving.

[0217] Thereafter, the driving intervention level correction process ends.

[0218] On the other hand, in step S49, if the current automation level is level 3 or higher, the automation level setting unit 152 determines that the vehicle is in autonomous automatic driving, and the process proceeds to step S51.

[0219] In step S51, the manual driving return support process is executed, and the driving intervention level correction process ends.

[0220] Here, the manual driving return support process will be described in detail with reference to the flowchart of FIG.

[0221] In step S71, the notification control unit 124 prompts the driver to return to manual driving.

[0222] For example, under the control of the notification control unit 124, the display unit 29 displays a message within the driver's field of vision that urges the driver to return to manual driving.

[0223] For example, under the control of the notification control unit 124, the audio output unit 30 outputs a message, an alarm, a buzzer, a beep, or the like to prompt the driver to return to manual driving.

[0224] It is desirable that this notification not cause discomfort to the driver as much as possible. However, if the volume is gradually increased from a comfortable level to a comfortable level, for example, if the driver is napping, the driver may mistakenly think that the notification is an extension of their dream. In response to this, multiple notification methods may be switched in sequence, such as an alarm sound, a message, haptic vibration, and a buzzer.

[0225] In step S72, the driver intervention level detection unit 143 determines whether or not it is possible to return to manual driving. If the driver intervention level does not satisfy the conditions for the automation level after the change (after returning to manual driving), the driver intervention level detection unit 143 determines that it is not possible to return to manual driving, and the process proceeds to step S73.

[0226] In step S73, a penalty is applied in the same manner as in step S46 of FIG.

[0227] In step S74, similar to the processing in step S72, it is determined whether or not it is possible to return to manual driving. If the driving intervention level satisfies the conditions for the changed automation level (after returning to manual driving), it is determined that it is possible to return to manual driving, and processing proceeds to step S75. In other words, if the driver corrects the driving intervention level by applying a penalty and is ready to return to manual driving, processing proceeds to step S75.

[0228] If a penalty is still being applied, it will cease to be applied at this point.

[0229] On the other hand, if it is determined in step S72 that manual driving can be resumed, the processes of steps S73 and S74 are skipped and the process proceeds to step S75.

[0230] In step S75, similar to the process in step S48 of FIG. 8, driving operations are temporarily restricted as necessary.

[0231] Thereafter, the manual driving return support process ends.

[0232] On the other hand, if it is determined in step S74 that the driver is not able to return to manual driving, i.e., if the driver does not correct the level of driving intervention despite the application of a penalty and is not ready to return to manual driving, processing proceeds to step S76.

[0233] In step S76, a forced vehicle stop process is carried out in the same manner as in step S50 of FIG.

[0234] Thereafter, the manual driving return support process ends.

[0235] Returning to FIG. 6, on the other hand, if it is determined in step S10 that the driving intervention level is not insufficient, the process proceeds to step S12.

[0236] In step S12, the automation level setting unit 152 determines whether or not to change the automation level. If it is determined that the automation level is to be changed, the process proceeds to step S13.

[0237] For example, if the distance from the current location to the next automation level change point is less than a predetermined threshold, it is determined that the automation level should be changed. Alternatively, if the predicted value of the time required from the current location to the next automation level change point is less than a predetermined threshold, it is determined that the automation level should be changed.

[0238] For example, in the example of FIG. 7, the allowable automation level changes from level 3 to level 2 at point P2. Therefore, if the vehicle is traveling at automation level 3 in section S1, it must switch to automation level 2 before entering section S2. Therefore, for example, if the distance to point P2 is less than a predetermined threshold, or if the predicted value of the required time to reach point P2 is less than a predetermined threshold, it is determined that the automation level should be changed.

[0239] These thresholds are set based on, for example, the time required for the driver to return to manual driving, which is predicted based on the driver's driving ability, age, condition, etc., and within a range that allows the driver to be prepared to return to manual driving in sufficient time.

[0240] Furthermore, for example, in the case of automation level 3, if it becomes difficult to continue autonomous automated driving during preliminary response, it is determined that the automation level should be changed. For example, when performing autonomous automated driving by following a preceding vehicle using ACC, if a notification is received from the preceding vehicle that it is leaving, it is determined that the automation level should be changed.

[0241] In step S13, the automation level setting unit 152 determines whether a return to manual driving is necessary. For example, when the automation level transitions from automation level 3 or automation level 4 to automation level 2 or lower, that is, when the automation level transitions from a level where the driver is not involved in driving to a level where the driver is involved, the automation level setting unit 152 determines that a return to manual driving is necessary, and the process proceeds to step S14.

[0242] In step S14, the manual driving return support process is executed as described above with reference to Fig. 9. Thereafter, the process proceeds to step S15.

[0243] In step S15, the driving assistance control unit 153 determines whether or not a forced stop was performed in the driving intervention level correction process in step S11 or the manual driving return assistance process in step S14. If it is determined that a forced stop was performed, the process proceeds to step S17.

[0244] On the other hand, if it is determined in step S13 that a return to manual driving is not necessary, the process proceeds to step S16.

[0245] In step S16, the notification control unit 124 notifies the change in the automation level.

[0246] For example, under the control of the notification control unit 124, the display unit 29 displays a message indicating a change in the automation level within the driver's field of vision.

[0247] For example, under the control of the notification control unit 124, the audio output unit 30 outputs a message, an alarm, a buzzer, a beep, or the like indicating a change in the automation level.

[0248] Thereafter, the process proceeds to step S17.

[0249] On the other hand, if it is determined in step S12 that the automation level is not to be changed, the processes of steps S13 to S16 are skipped and the process proceeds to step S17.

[0250] In step S17, the route setting unit 151 determines whether or not the destination has been reached. If it is determined that the destination has not been reached yet, the process returns to step S10.

[0251] Thereafter, the processes of steps S10 to S17 are repeatedly executed until it is determined in step S15 that a forced stop has been made, or until it is determined in step S17 that the vehicle has arrived at the destination.

[0252] On the other hand, if it is determined in step S15 that a forced stop has been made, or if it is determined in step S17 that the vehicle has arrived at the destination, the process proceeds to step S18.

[0253] In step S18, it is determined whether or not the insufficient driver intervention level has occurred frequently, similar to the process in step S41 of Fig. 8. If it is determined that the insufficient driver intervention level has occurred frequently, the process proceeds to step S19.

[0254] In step S19, the penalty application unit 125 applies a penalty as necessary. The penalty applied here is an ex-post penalty. Specific examples of ex-post penalties will be described below.

[0255] For example, the driving assistance control unit 153 controls the driving control unit 33 under the instruction of the penalty application unit 125 to prohibit the vehicle from starting for a predetermined period after the forced stop, which causes a delay in arrival at the destination.

[0256] For example, under the instructions of the penalty application unit 125, the driving assistance control unit 153 controls the driving control unit 33 to prohibit autonomous driving at automation level 3 or higher after a forced stop until the vehicle arrives at the destination or for a specified period of time.

[0257] For example, the log generating unit 126 records information about the ex-post penalty in a log under the instruction of the penalty applying unit 125.

[0258] For example, information for deducting discount points for automobile insurance is recorded in the log. Based on the information recorded in this log, discount points are deducted when signing up for or renewing automobile insurance.

[0259] For example, information for making it mandatory to take a driving ability test when renewing a driver's license is recorded in the log. Based on the information recorded in this log, the driver is required to take a driving ability test when renewing a driver's license.

[0260] For example, information for adding points to a driver's license is recorded in a log, and points are added to the driver's license based on the information recorded in the log.

[0261] For example, information to increase the scope of inspections during vehicle inspections is recorded in logs. Based on the information recorded in these logs, more detailed inspections are carried out during vehicle inspections to improve the safety of autonomous driving. Furthermore, in order to improve the safety of autonomous driving, high redundancy is required for the equipment used to realize autonomous driving, so emphasis is placed on inspections of this equipment.

[0262] After that, the automatic driving control process ends.

[0263] On the other hand, if it is determined in step S18 that the decrease in the driving intervention level has not occurred frequently, the processing of step S19 is skipped and the automatic driving control processing ends.

[0264] In this way, correcting the driver's level of driving intervention suppresses the driver's reliance on automated driving. In particular, the driver is motivated to avoid in advance the disadvantages that may be caused by the application of a penalty. This reliably prevents, for example, the driver from concentrating on actions or tasks other than driving during autonomous automated driving, reducing their awareness of driving, or from becoming habitual in this state. As a result, for example, even if the driver needs to perform manual driving during autonomous automated driving, the driver can respond appropriately and quickly, improving the safety of automated driving.

[0265] <Example of log> Here, a specific example of a log will be described with reference to FIG.

[0266] The log includes items such as number, date, time, monitoring status, event, penalty, and reason for application, for example.

[0267] The number records a serial number assigned to each record in the log.

[0268] The date and time when the event occurred are recorded.

[0269] The monitoring status records the implementation status of monitoring the driver's status. "Start" indicates the start of monitoring the driver's status, "Monitoring" indicates that the driver's status is being monitored, and "End" indicates the end of monitoring the driver.

[0270] In the above example, the driver's state is constantly monitored, but monitoring of the driver's state may be stopped when there is little need for it. For example, as shown in this example, when autonomous driving is being performed, monitoring of the driver may be stopped except before switching to manual driving, etc.

[0271] The events briefly record the content of the event that occurred. For example, events of sending and receiving various information such as LDM requests, reception, and updates are recorded. Also, events such as the detection results of the driver's state, warnings to the driver, and application of penalties are recorded. Details of the driver's state, the content of the warnings, the content of the applied penalties, etc. may also be recorded. Also, transitions in automation levels are recorded.

[0272] The penalty column records whether a penalty was applied and the rank of the applied penalty. "Not Applicable" indicates that no penalty was applied to the corresponding event.

[0273] The reason for the application is recorded in the application reason field.

[0274] Security measures are implemented on the logs to prevent alteration, fabrication, eavesdropping, etc.

[0275] Furthermore, by increasing the amount of information in the log by including images, etc., it is expected that the log will become more useful in analyzing the causes of accidents, etc. On the other hand, logs in which no particular events have occurred are likely not worth saving. Therefore, for example, logs for a period of several minutes to several tens of minutes may be temporarily stored in a buffer, and older logs may be deleted as needed. In addition, when an event such as an impact, alarm, warning, or accident occurs, logs for a predetermined period before and after the event may be copied from the buffer to the storage unit 35 and saved.

[0276] Furthermore, for example, the log may be recorded separately from the vehicle on an electronic key or a remote server, so that it can be easily used in another vehicle.

[0277] <Autonomous driving usage restriction processing> Next, the automatic driving control process executed by the vehicle control system 11 will be described with reference to the flowchart of Fig. 11. Note that this process is executed, for example, before the automatic driving control process described above with reference to Figs.

[0278] In step S101, driver authentication is performed in the same manner as in step S1 of FIG.

[0279] In step S102, the penalty application unit 125 determines whether or not penalties have been applied frequently within a recent predetermined period. Specifically, the penalty application unit 125 acquires a log of the authenticated driver for a recent predetermined period (for example, one month, three months, six months, or one year) from the storage unit 35. Next, the penalty application unit 125 calculates the number of times penalties have been applied to the driver within that period and the cumulative value of the ranks of the applied penalties. Then, if the number of times penalties have been applied is equal to or greater than a predetermined threshold, or if the cumulative value of the ranks of the applied penalties is equal to or greater than a predetermined threshold, the penalty application unit 125 determines that penalties have been applied frequently, and the process proceeds to step S103.

[0280] In step S103, the automation level setting unit 152 restricts the use of automated driving in accordance with the instructions of the penalty application unit 125. For example, the automation level setting unit 152 restricts the available automation level to level 2 or lower. Alternatively, the automation level setting unit 152 fixes the automation level to level 0. This prevents the driver from using autonomous automated driving and forces the driver to drive manually.

[0281] For example, if a driver using a vehicle for the first time and making extensive use of the automated driving function becomes overly dependent on automated driving, there is a high possibility that the driver will be delayed in returning to manual driving when an emergency occurs. Therefore, it may be possible to make it easier to apply restrictions on the use of automated driving to such drivers, even if there is no record of frequent application of penalties.

[0282] It is desirable that the driver cannot release the restriction on the use of the automation level. For example, it may be possible for only the dealer to release the restriction. Alternatively, the restriction on the use of the automation level may not be released until a predetermined period of time (for example, one month) has passed.

[0283] After that, the automatic driving usage restriction process ends.

[0284] On the other hand, in step S103, if the number of times penalties have been applied is less than a predetermined threshold and the cumulative value of the ranks of the applied penalties is less than a predetermined threshold, the penalty application unit 125 determines that penalties have not been applied frequently, the processing of step S103 is skipped, and the autonomous driving use restriction processing ends.

[0285] For example, if a driver has only been using an autonomous vehicle for a short period of time and is not yet accustomed to autonomous driving, it is expected that they will get into the autonomous vehicle with a sense of tension, without reducing their level of driving intervention. On the other hand, as the period of use of self-driving vehicles increases and drivers become accustomed to autonomous driving, there is a concern that tension will decrease, dependence on autonomous driving will increase, and the level of driver intervention will decrease.

[0286] In contrast, limiting the use of automated driving based on the number of penalties applied and the cumulative value of the rank of applied penalties motivates the driver to avoid the application of penalties. As a result, even if the period of use of an automated driving vehicle increases, a decrease in the driver's level of driving intervention and reliance on automated driving are prevented. Furthermore, for example, it is prevented that a driver becomes accustomed to the penalties applied while driving, ignores the penalties, and intentionally uses automated driving with a reduced level of driving intervention. As a result, whether using automated driving for a short, medium, or long period of time, the driver is prevented from becoming overly dependent on automated driving, improving safety.

[0287] Furthermore, by establishing a complex and hierarchical system for applying penalties, it will be possible to build a social system in which each driver is always conscious of safety at all driving levels, and it will become possible to create a motorized society in which automated and manual vehicles coexist harmoniously.

[0288] <<2. Modifications>> Below, a description will be given of modifications of the above-described embodiment of the technology according to the present disclosure.

[0289] The classification of automation levels is not limited to the examples given above, and classification can be applied based on other criteria.

[0290] In addition, the driver state detection unit 142 may detect one of the driver's reactivity and alertness, and the driving intervention level detection unit 143 may detect the driving intervention level using one of the driver's reactivity and alertness.

[0291] Furthermore, for example, a combination of multiple penalties may be applied, and the combination of penalties to be applied may be changed depending on, for example, the driver's reactivity or alertness, the number of times or frequency of application of penalties, etc.

[0292] In addition, it is expected that there will be cases where it is necessary to move by vehicle urgently in an emergency, to avoid danger, in a depopulated area, etc. In such cases, it may be possible to cancel the driving penalty. However, for example, the cancellation of the driving penalty will be recorded in a log, and an ex-post penalty will be applied instead.

[0293] Furthermore, incentives equivalent to rewards may be given to drivers who maintain high levels of responsiveness and alertness, drivers who need attention, drivers who transition safely and smoothly to manual driving, etc. Similar to penalties, these incentives are expected to have the effect of preventing a decline in the level of driver intervention and preventing drivers from becoming overly dependent on automated driving.

[0294] The above-mentioned penalties and incentives may be managed as points in a log, and the data of these points may be analyzed and recorded or output as a report.

[0295] Furthermore, for example, although this is a tax system that does not currently exist, log points may be reflected in additional points such as the autonomous driving countermeasures promotion tax.

[0296] Furthermore, for example, the vehicle control system 11 may create map data using a technique such as SLAM (Simultaneous Localization and Mapping).

[0297] This technology can be applied to various vehicles that can automate at least part of their driving, regardless of the vehicle's power source or energy supply source. For example, this technology can be applied to gasoline-powered vehicles, hybrid vehicles, plug-in hybrid vehicles, electric vehicles, fuel cell vehicles, etc. In addition to general automobiles, this technology can also be applied to buses, trucks, motorcycles, etc. In particular, this technology is particularly effective when applied to various vehicles that can switch between autonomous driving and manual driving.

[0298] <<3.Others>> <Computer configuration example> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0299] FIG. 12 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0300] In the computer, a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, and a RAM (Random Access Memory) 403 are interconnected by a bus 404.

[0301] An input / output interface 405 is further connected to the bus 404. An input unit 406, an output unit 407, a recording unit 408, a communication unit 409, and a drive 410 are connected to the input / output interface 405.

[0302] The input unit 406 includes an input switch, a button, a microphone, an image sensor, etc. The output unit 407 includes a display, a speaker, etc. The recording unit 408 includes a hard disk, a nonvolatile memory, etc. The communication unit 409 includes a network interface, etc. The drive 410 drives a removable recording medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0303] In a computer configured as described above, the CPU 401 performs the above-described series of processes by, for example, loading a program recorded in the recording unit 408 into the RAM 403 via the input / output interface 405 and the bus 404 and executing it.

[0304] The program executed by the computer (CPU 401) can be provided by being recorded on a removable recording medium 411 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0305] In a computer, the program can be installed in the recording unit 408 via the input / output interface 405 by inserting the removable recording medium 411 into the drive 410. The program can also be received by the communication unit 409 via a wired or wireless transmission medium and installed in the recording unit 408. Alternatively, the program can be installed in the ROM 402 or the recording unit 408 in advance.

[0306] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0307] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0308] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0309] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0310] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0311] <Configuration combination example> The present technology can also be configured as follows.

[0312] (1) a driver monitoring unit that detects a driving intervention level that indicates the degree to which the driver is intervening in driving the vehicle; a penalty application unit that applies a penalty that is detrimental to the driver when the driving intervention level is insufficient; A vehicle control device comprising: (2) The penalty application unit applies the penalty when the driving intervention level is insufficient when the automated driving level transitions from a level where the driver does not intervene in driving to a level where the driver intervenes in driving. The vehicle control device according to (1) above. (3) a driving assistance control unit that restricts driving operations until the driver appropriately responds to given instructions when the insufficient driving intervention level is resolved after applying the penalty; The vehicle control device according to (2) above is further provided. (4) The vehicle control device described in any one of (1) to (3), wherein the penalty application unit applies the penalty if the driving intervention level is insufficient even after processing to alert the driver before applying the penalty. (5) The penalty application unit applies the penalty based on at least one of the number of occurrences and the frequency of occurrences of a state in which the driving intervention level is insufficient. The vehicle control device according to any one of (1) to (4). (6) The penalty application unit changes the penalty to be applied when the application of the penalty is repeated. The vehicle control device according to any one of (1) to (5). (7) The penalty causes discomfort to the driver or reduces convenience for the driver. The vehicle control device according to any one of (1) to (6). (8) The vehicle control device according to (7), wherein the penalty includes moving the backrest of the driver's seat closer to a direction perpendicular to the seat surface. (9) The penalty includes limiting the maximum speed of the vehicle. The vehicle control device according to (7) or (8). (10) The penalty includes limiting the autonomous driving function of the vehicle. The vehicle control device according to any one of (7) to (9). (11) The penalty includes changing the driving route to take a route that requires driving intervention by the driver. The vehicle control device according to any one of (7) to (10) above. (12) The penalty application unit applies the penalty after the vehicle arrives at the destination or after the vehicle is forcibly stopped. The vehicle control device according to any one of (1) to (11). (13) The penalty application unit prohibits the vehicle from starting for a predetermined period of time. The vehicle control device according to (12) above. (14) The penalty application unit records information about the penalty to be applied after the vehicle arrives at the destination or after the vehicle is forcibly stopped in a log. The vehicle control device according to (12) or (13) above. (15) The driver monitoring unit determines whether the level of driver intervention is insufficient based on conditions that vary depending on the level of autonomous driving. The vehicle control device according to any one of (1) to (14). (16) The penalty application unit stops application of the penalty when the insufficiency of the driving intervention level is resolved. The vehicle control device according to any one of (1) to (15). (17) a driver state detection unit that detects at least one of the responsiveness and the alertness of the driver; Further preparation, The driver monitoring unit detects the driving intervention level using at least one of the driver's reactivity and alertness. The vehicle control device according to any one of (1) to (16). (18) The driver state detection unit detects at least one of the responsiveness and the alertness of the driver based on the response of the driver to a stimulus or an instruction given to the driver. The vehicle control device according to (17) above. (19) The driver state detection unit issues an instruction via an information processing device used by the driver. The vehicle control device according to (18) above. (20) a driver monitoring step of detecting a driving intervention level indicative of a level of driver intervention in driving the vehicle; a penalty application step of applying a penalty that is disadvantageous to the driver when the driving intervention level is insufficient; A vehicle control method comprising: [Explanation of symbols]

[0313] 10 Autonomous driving system, 11 Vehicle control system, 12 Mobile terminal, 21 Surrounding area photographing unit, 22 Surrounding area information acquisition unit, 23 Position measurement unit, 25 Vehicle information acquisition unit, 26 Driver monitoring unit, 27 Communication unit, 28 Vehicle control unit, 29 Display unit, 30 Audio output unit, 31 Light-emitting unit, 32 Odor generation unit, 33 Driving control unit, 34 In-vehicle device control unit, 101 Driver photographing unit, 102 Biometric information acquisition unit, 103 Gaze detection unit, 104 Authentication unit, 121 Surrounding area monitoring unit, 122 Driver monitoring unit, 123 Autonomous driving control unit, 124 Notification control unit, 125 Penalty application unit, 126 Log generation unit, 127 Learning unit, 141 Driving behavior analysis unit, 142 Driver state detection unit 143 driving intervention level detection unit, 151 route setting unit, 152 automation level setting unit, 153 driving assistance control unit

Claims

1. an automation level setting unit that sets an automation level; a driver monitoring unit that detects a driving intervention level that indicates the degree to which the driver is involved in driving the vehicle, the higher the automation level, the lower the required level is set, and when the automation level is set to a level at which the driver needs to perform all or part of the driving tasks, determines through passive monitoring whether the driving intervention level is insufficient for the automation level, and when the automation level is set to a level at which the automated driving system performs all driving tasks, determines through active monitoring whether the driving intervention level is insufficient for the automation level; a penalty application unit that applies a penalty to the driver based on the frequency of occurrence of the state in which the level of driver intervention is insufficient, when the state in which the level of driver intervention is insufficient has occurred multiple times, to cause the driver to correct the level of intervention of the driver, which is disadvantageous to the driver; A vehicle control device comprising:

2. The level of automation at which the system performs all driving tasks is a level that provides more advanced driving assistance than driving assistance including ACC (Adaptive Cruise Control) and LKAS (Lane Keep Assist System). The vehicle control device according to claim 1 .

3. When the automation level is set to a level at which the driver needs to perform all or part of the driving tasks, the driver monitoring unit determines whether the level of driving intervention is insufficient based on the direction of the driver's line of sight and attention, and the direction of travel of the vehicle. The vehicle control device according to claim 1 .

4. The automation level at which the driver is required to perform all or part of the driving tasks includes a first automation level and a second automation level, the automation level being higher than the first automation level; the automation level at which the system performs all driving tasks includes a third automation level, the automation level being higher than the second automation level; If the automation level is the first automation level, a first driver intervention level is required as the driver intervention level; When the automation level is the second automation level, a second driver intervention level that is less demanding than the first driver intervention level is required as the driver intervention level; When the automation level is the third automation level, autonomous driving is possible. The vehicle control device according to claim 1 .

5. When the automation level is the third automation level, the driver monitoring unit determines whether the driver intervention level is insufficient based on a response to an instruction given to the driver. The vehicle control device according to claim 4.

6. The automation level setting unit restricts the driver from using automated driving when the number of penalties applied to the driver within a predetermined period or the cumulative value of the penalty ranks is equal to or greater than a predetermined threshold. The vehicle control device according to claim 1 .

7. The penalty application unit applies the penalty when the driver intervention level is insufficient when the automation level transitions from a level in which the driver does not intervene in driving to a level in which the driver intervenes in driving. The vehicle control device according to claim 1 .

8. a driving assistance control unit that restricts driving operations until the driver appropriately responds to given instructions when the insufficient driving intervention level is resolved after applying the penalty; The vehicle control device according to claim 7 further comprising:

9. The penalty application unit applies the penalty when the driving intervention level is still insufficient even after a process of calling the driver's attention before applying the penalty. The vehicle control device according to claim 1 .

10. The penalty provides a stronger warning to the driver than before the penalty was applied. The vehicle control device according to claim 9.

11. The penalty application unit changes the penalty to be applied when the application of the penalty is repeated. The vehicle control device according to claim 1 .

12. The penalty application unit changes the combination of the penalties to be applied based on at least one of the number of times the penalties are applied and the application frequency. The vehicle control device according to claim 11.

13. The penalty causes discomfort to the driver or reduces convenience for the driver. The vehicle control device according to claim 1 .

14. The penalty includes moving the driver's seat back closer to a perpendicular direction to the seat surface. The vehicle control device according to claim 13.

15. The penalty includes limiting the maximum speed at which the vehicle may travel. The vehicle control device according to claim 13.

16. The penalty includes changing the driving route so as to increase the burden on the driver. The vehicle control device according to claim 13.

17. The penalty includes changing the driving route to take a route that requires driving intervention by the driver. The vehicle control device according to claim 16.

18. The penalty may include changing the driving route to delay arrival at the destination. The vehicle control device according to claim 13.

19. The penalty includes causing a foul odor. The vehicle control device according to claim 13.

20. The penalty includes performing brake control to increase or decrease deceleration. The vehicle control device according to claim 13.

21. The penalty may include a reduction in the automation level or a restriction on driver assistance functions. The vehicle control device according to claim 13.

22. The penalty application unit applies the penalty after the vehicle arrives at the destination or after the vehicle is forcibly stopped. The vehicle control device according to claim 1 .

23. The penalty application unit prohibits the vehicle from starting for a predetermined period after the vehicle is forcibly stopped. The vehicle control device according to claim 22.

24. The penalty application unit restricts the use of autonomous driving until the vehicle arrives at the destination or for a predetermined period after the vehicle is forcibly stopped. The vehicle control device according to claim 22.

25. The penalty application unit records information about the penalty to be applied after the vehicle arrives at the destination or after the vehicle is forcibly stopped in a log. The vehicle control device according to claim 22.

26. The penalty application unit stops application of the penalty when the insufficiency of the driving intervention level is resolved. The vehicle control device according to claim 1 .

27. a driver state detection unit that detects at least one of the responsiveness and the alertness of the driver; Further preparation, The driver monitoring unit detects the driving intervention level using at least one of the driver's reactivity and wakefulness. The vehicle control device according to claim 1 .

28. The driver state detection unit detects at least one of the responsiveness and the alertness of the driver based on the response of the driver to a stimulus or an instruction given to the driver.

28. The vehicle control device according to claim 27.

29. The driver state detection unit detects at least one of the responsiveness and the alertness of the driver based on the driver's response to an instruction given via an information processing device used by the driver when the driver does not need to intervene in driving.

28. The vehicle control device according to claim 27.

30. The penalty application unit controls the rank of the penalty to be applied to the driver based on at least one of the reactivity and the alertness of the driver.

28. The vehicle control device according to claim 27.

31. a log generation unit that generates and updates a log including information about events that have occurred in the vehicle; The vehicle control device according to claim 1 further comprising:

32. The automation level setting unit sets a distribution of the automation levels that are allowed on the travel route based on at least one of surrounding information, map information, and traffic information. The vehicle control device according to claim 1 further comprising:

33. The vehicle control device Setting an automation level; and Detecting a driving intervention level that indicates the degree to which the driver is involved in driving the vehicle, the higher the automation level, the lower the required level is set; When the automation level is set to a level that requires the driver to perform all or part of the driving tasks, determine by passive monitoring whether the level of driver intervention is insufficient for the automation level, and when the automation level is set to a level that requires the driver to perform all driving tasks, determine by active monitoring whether the level of driver intervention is insufficient for the automation level; If the state in which the level of driving intervention is insufficient occurs multiple times, a penalty is applied based on the frequency of occurrence of the state in which the level of driving intervention is insufficient, which is detrimental to the driver and causes the driver to correct the level of intervention. A vehicle control method comprising:

Images