Information processing device, information processing method, and information processing program

Abstract

This information processing device comprises an acquisition unit (101) that acquires the state of an operator of a vehicle and an automatic operation control unit (10112) that controls automatic operation that makes the vehicle travel autonomously. On the basis of the state of the operator as acquired by the acquisition unit, the automatic operation control unit finds a return quality that is the quality of behavior when the vehicle returns to manual operation by the operator from automatic operation. The automatic operation control unit quantifies the return quality found and thereby monitors the operator.

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] In recent years, there has been a great deal of development in autonomous driving technology in which vehicle control systems (information processing systems) control vehicles. However, even if such autonomous driving technology becomes widespread, it is believed that there will still be many technical challenges before autonomous driving using only autonomous single-control can achieve the same driving speed as existing manual driving. Therefore, it is being considered to try cooperative autonomous driving that utilizes prior monitoring information, for example, by limiting autonomous driving to driving sections where road environment improvements and constant prior monitoring information of the road environment can be obtained.

[0003] In that case, depending on the actual road infrastructure development status, it is expected that a situation will arise where sections of road where autonomous automatic driving control by the vehicle control system is possible (autonomous driving sections) and sections of road where autonomous driving is not permitted (manual driving sections) coexist. In other words, it is not always the case that the vehicle control system will operate completely autonomously and continuously; there may also be situations where driving control must be handed over from the aforementioned autonomous driving to manual driving where the driver takes over steering, etc.

[0004] Patent Document 1 describes a technology related to the transfer of control from automated driving to manual driving. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2018-180594 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] If such a transfer of control from autonomous driving to manual driving is executed when the driver is not sufficiently prepared for manual driving, there is a risk of causing social harms such as inducing accidents to following vehicles or the like.

[0007] An object of the present disclosure is to provide an information processing apparatus, an information processing method, and an information processing program capable of appropriately executing a transfer from autonomous driving to manual driving.

Means for Solving the Problems

[0008] The information processing apparatus according to the present disclosure includes an acquisition unit that acquires the state of a driver of a vehicle, and an autonomous driving control unit that controls autonomous driving for the vehicle to autonomously travel. The autonomous driving control unit obtains a return quality, which is the quality of an action when the vehicle's travel returns from autonomous driving to manual driving by the driver's driving, based on the state of the driver acquired by the acquisition unit, and performs driver monitoring on the driver by quantifying the obtained return quality.

Brief Description of the Drawings

[0009] [Figure 1] It is a block diagram showing a schematic functional configuration example of a vehicle control system applicable to an embodiment of the present disclosure. [Figure 2] It is a block diagram showing a configuration example of an example of an information processing apparatus in which an autonomous driving control unit applicable to an embodiment is configured. [Figure 3] It is a schematic diagram for explaining each autonomous driving level of SAE from the perspective of a user as a usage state. [Figure 4] It is a schematic diagram for schematically explaining the application of autonomous driving level 3. [Figure 5] It is an example flowchart schematically showing a handover process from autonomous driving to manual driving according to the prior art. [Figure 6] It is an example flowchart schematically showing a handover process from autonomous driving to manual driving according to an embodiment. [Figure 7A]This is an example flowchart illustrating the flow from itinerary setting to the transition to autonomous driving mode according to the embodiment. [Figure 7B] This is an example flowchart showing the processing flow in the automated driving mode according to the embodiment. [Figure 7C] This is an example flowchart illustrating how to respond to events that occur during autonomous driving at Level 4 according to the embodiment. [Figure 8] This is a schematic diagram illustrating an example of an overview view of a travel itinerary applicable to the embodiment. [Figure 9A] This is a schematic diagram showing an example of an overhead view in which each section is color-coded according to the embodiment. [Figure 9B] This is a schematic diagram showing an example of a circular overhead view according to the embodiment. [Figure 9C] This is a schematic diagram showing an example of an overhead view including road information according to an embodiment. [Figure 10] This is an example of a functional block diagram illustrating the control function by HCD in the automated driving control unit according to the embodiment. [Figure 11] This is an example of a functional block diagram illustrating the function of the driver return delay evaluation unit according to the embodiment. [Figure 12] This is a schematic diagram illustrating a high-precision update LDM applicable to the embodiment. [Figure 13] This is a schematic diagram illustrating the acquisition of information by a remote support control interface applicable to the embodiment. [Figure 14] This is an example of a functional block diagram illustrating the function of the driver behavior change achievement level estimation unit according to the embodiment. [Figure 15] This is a schematic diagram illustrating the basic structure of Level 4 autonomous driving applicable to the embodiment. [Figure 16] This is a schematic diagram illustrating the ODD in autonomous driving level 4 according to the embodiment. [Figure 17A] This is an example flowchart illustrating an example of operation of Level 4 autonomous driving according to the embodiment. [Figure 17B] This is an example flowchart illustrating an example of operation of Level 4 autonomous driving according to the embodiment. [Figure 18A] This diagram schematically illustrates how a driver traveling on Road 7 in their own vehicle extends the section of autonomous driving at Level 3. [Figure 18B] This is an example flowchart illustrating the processing in a section where conditional autonomous driving level 3 is available, according to the embodiment. [Figure 19A] This is an example flowchart illustrating the processing flow of autonomous driving applicable to the embodiment, focusing on the ODD (Orientation Distance). [Figure 19B] This is an example flowchart illustrating in more detail an example of an ODD setting process applicable to the embodiment. [Figure 20] This is a schematic diagram to more specifically illustrate an example of setting an ODD interval applicable to the embodiment. [Figure 21] This is an example of a functional block diagram illustrating the functions of a driver behavior evaluation unit applicable to the DMS according to the embodiment. [Figure 22] This is an example functional block diagram illustrating the functions of an offline learning unit applicable to the embodiment. [Figure 23A] This is a schematic diagram illustrating the generation of a 3D head model applicable to the embodiment. [Figure 23B] This is a schematic diagram illustrating the generation of a body model applicable to the embodiment. [Figure 24] This is a schematic diagram illustrating a method for determining the driver's alertness level, applicable to the embodiment. [Figure 25] This is an example flowchart showing a process for evaluating the quality of actions according to the embodiment. [Modes for carrying out the invention]

[0010] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0011] The embodiments of this disclosure will be described below in the following order. 0. Summary of this Disclosure 1. Configurations applicable to embodiments of this disclosure 2. Overview of SAE's Levels of Autonomous Driving 3. Embodiments relating to this disclosure 3-1. Overview of the Embodiment 3-2. About the Human-Centered Design (HCD) according to the embodiment 3-2-1. Overview of the HCD according to the embodiment 3-2-2. Advantages of HCD in Autonomous Driving 3-2-2-1. On excessive dependence 3-2-2-2. About HCD 3-2-2-3. Benefits for Drivers 3-2-2-4. Driver's working memory and thinking during driving 3-2-2-5. Regarding the "Contract" between the System and the Driver 3-2-2-6. Operation of Autonomous Driving Level 4 3-2-2-7. Effects of adopting HCD 3-2-3. Specific Examples of HCDs According to the Embodiment 3-2-3-1. Example of Automated Driving Operation Applying the HCD According to the Embodiment 3-2-3-2. Evaluation of the driver's recovery actions 3-2-3-3. Overview of the itinerary applicable to the embodiment 3-2-4. Example of HCD control configuration according to the embodiment 3-3. Automated driving level 4 applicable to the embodiment 3-3-1. Basic structure 3-3-2. About ODD in Autonomous Driving Level 4 3-3-3. Example of operation of autonomous driving level 4 according to the embodiment 3-4. Examples of HCD application to Level 3 autonomous driving 3-5. Determinants of ODD 3-6. Driver Monitoring System (DMS) according to the embodiment 3-6-1. Overview of the DMS according to the embodiment 3-6-2. More specific description of the DMS according to the embodiment 3-6-3. Quantification of Quality of Action (QoA) according to the embodiment 3-6-4. Configurations applicable to the DMS according to the embodiment 3-6-5. Specific Examples of Evaluation of the Quality of Actions According to the Embodiment 3-6-6. Summary of DMS according to the embodiment

[0012] <<0. Summary of this Disclosure>> This disclosure relates to the process of transferring control of a vehicle from an automated driving system to a driver when the vehicle's driving control is transferred from automated driving, where the vehicle is driven autonomously, to manual driving, where the driver takes over steering and other vehicle operations. More specifically, this disclosure provides a mechanism that supports the driver's self-learning through repeated use, enabling a smooth transfer of driving control from the vehicle to the driver.

[0013] In other words, for drivers, the autonomous driving function is initially just theoretical knowledge gained from explanations and documents, and is an unfamiliar function in terms of actual experience. Therefore, it is likely that they are still skeptical of a system they have no experience with due to psychological anxiety. Furthermore, as a normal part of behavioral judgment, when people take action and take a certain risk to gain something, they make choices and decisions while balancing that risk.

[0014] Therefore, drivers who have started using autonomous driving will maintain a certain level of vigilance even while using the autonomous driving function, as long as they still have some anxiety about driving autonomously, in order to mitigate that sense of risk. The necessary awareness required when using autonomous driving functions will not completely disappear but will be retained.

[0015] As the event handling capabilities of autonomous driving systems gradually improve, and the anxiety associated with repeated use of autonomous driving decreases, users of autonomous driving will no longer feel anxious about becoming overly dependent on it. In particular, the advanced autonomous driving functions that are beginning to be introduced in recent years are required to have the ability to avoid accidents and take appropriate action even if the driver is required to switch from autonomous driving to manual control under suitable conditions and it is difficult for the driver to return to manual control, and to minimize the impact even in situations where an accident is unavoidable.

[0016] If autonomous driving with such advanced functions becomes a reality, drivers may gradually lose their anxiety about the risks of excessive reliance on autonomous driving, potentially leading to situations where drivers are insufficiently prepared when requested to take over manual driving. Therefore, in cases where it is difficult for drivers to take the necessary actions within the given timeframe, emergency stops using autonomous driving are being considered as a safe measure to minimize the impact risk.

[0017] However, if vehicles frequently decelerate or make emergency stops in many road conditions, situations that hinder the movement of other vehicles may occur in ways that are not directly visible to the driver, such as sudden deceleration of following vehicles, stopping in road conditions with poor visibility, and blocking narrow roads such as bridges with limited traffic lanes. These effects could lead to a decrease in the efficiency of the road environment, which is the artery of social activity. In other words, existing autonomous driving control systems did not have a means for drivers to reflect these social impacts as a sense of risk in their decision-making when using autonomous driving functions.

[0018] This disclosure aims to provide a mechanism that allows drivers to reflect the aforementioned social impacts as a sense of risk in their decision-making when using autonomous driving functions.

[0019] <<1. Configurations applicable to embodiments of this disclosure>> First, a configuration applicable to the embodiments of this disclosure will be described.

[0020] Figure 1 is a block diagram showing a schematic configuration example of the functions of a vehicle control system 10100, which is an example of a mobile control system applicable to embodiments of this disclosure.

[0021] In the following, when distinguishing a vehicle equipped with the vehicle control system 10100 from other vehicles, it will be referred to as "this vehicle" or "this vehicle."

[0022] The vehicle control system 10100 includes an input unit 10101, a data acquisition unit 10102, a communication unit 10103, in-vehicle equipment 10104, an output control unit 10105, an output unit 10106, a drive system control unit 10107, a drive system 10108, a body system control unit 10109, a body system 10110, a storage unit 10111, and an automatic driving control unit 10112.

[0023] Of these, the input unit 10101, data acquisition unit 10102, communication unit 10103, output control unit 10105, drive system control unit 10107, body system control unit 10109, storage unit 10111, and automatic driving control unit 10112 are interconnected via the communication network 10121. The communication network 10121 consists of an in-vehicle communication network or bus conforming to any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay (registered trademark). In some cases, the various parts of the vehicle control system 10100 may be directly connected without going through the communication network 10121.

[0024] In the following, when each part of the vehicle control system 10100 communicates via the communication network 10121, the description of the communication network 10121 will be omitted. For example, when the input unit 10101 and the automatic driving control unit 10112 communicate via the communication network 10121, it will simply be described as the input unit 10101 and the automatic driving control unit 10112 communicating.

[0025] The input unit 10101 is equipped with devices used by the passenger to input various data and instructions. For example, the input unit 10101 includes operating devices such as a touch panel, buttons, switches, and levers, and operating devices such as microphones and cameras that allow input by methods other than manual operation, such as voice or gestures. The input unit 10101 may also be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile device or wearable device that is compatible with the operation of the vehicle control system 10100. The input unit 10101 generates input signals based on data and instructions input by the passenger (e.g., the driver) and supplies them to each part of the vehicle control system 10100.

[0026] The data acquisition unit 10102 is equipped with various sensors and other devices for acquiring data used in the processing of the vehicle control system 10100, and supplies the acquired data to each part of the vehicle control system 10100.

[0027] For example, the data acquisition unit 10102 is equipped with various sensors for detecting the state of the vehicle, etc. Specifically, for example, the data acquisition unit 10102 is equipped with a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU), and sensors for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine speed, the motor speed, or the rotational speed of the wheels, etc.

[0028] Furthermore, for example, the data acquisition unit 10102 is equipped with various sensors for detecting information from outside the vehicle. Specifically, for example, the data acquisition unit 10102 is equipped with imaging devices such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. Furthermore, for example, the data acquisition unit 10102 is equipped with environmental sensors for detecting weather or climate, and ambient information detection sensors for detecting objects around the vehicle. Environmental sensors include, for example, raindrop sensors, fog sensors, sunlight sensors, snow sensors, etc. Ambient information detection sensors include, for example, ultrasonic sensors, radar, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), sonar, etc.

[0029] Furthermore, for example, the data acquisition unit 10102 is equipped with various sensors for detecting the vehicle's current position. Specifically, for example, the data acquisition unit 10102 is equipped with a GNSS receiver that receives GNSS signals from GNSS (Global Navigation Satellite System) satellites.

[0030] Furthermore, for example, the data acquisition unit 10102 is equipped with various sensors for detecting information inside the vehicle. Specifically, for example, the data acquisition unit 10102 includes an imaging device for imaging the driver, a biosensor for detecting the driver's biological information, and a microphone for collecting sounds inside the vehicle. In this case, it is preferable that the imaging device is capable of imaging the driver's head, upper body, waist, lower body, and feet. Multiple imaging devices can also be provided to image each of these parts. The biosensor is provided, for example, on the seat or steering wheel, and detects the biological information of the passenger sitting in the seat or the driver holding the steering wheel.

[0031] The communication unit 10103 communicates with in-vehicle equipment 10104, as well as various external devices, servers, base stations, etc., and transmits data supplied from various parts of the vehicle control system 10100, and supplies received data to various parts of the vehicle control system 10100. The communication protocols supported by the communication unit 10103 are not particularly limited, and it is possible for the communication unit 10103 to support multiple types of communication protocols.

[0032] For example, the communication unit 10103 communicates wirelessly with the in-vehicle equipment 10104 via Wi-Fi, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). Alternatively, the communication unit 10103 can communicate via wired connection terminals (and cables if necessary), such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), or MHL (Mobile High-definition Link).

[0033] Furthermore, for example, the communication unit 10103 communicates with devices (e.g., application servers or control servers) located on an external network (e.g., the Internet, a cloud network, or a carrier-specific network) via a base station or access point. Also, for example, the communication unit 10103 communicates with terminals located near its own vehicle (e.g., terminals of pedestrians or shops, or MTC (Machine Type Communication) terminals) using P2P (Peer To Peer) technology. Furthermore, for example, the communication unit 10103 performs V2X communication such as vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication. Also, for example, the communication unit 10103 is equipped with a beacon receiver and receives radio waves or electromagnetic waves transmitted from radio stations installed on roads, etc., to acquire information such as current location, congestion, traffic restrictions, or travel time.

[0034] The in-vehicle equipment 10104 includes, for example, mobile devices or wearable devices owned by passengers, information devices brought into or installed in the vehicle, and navigation devices for searching for routes to any destination.

[0035] The output control unit 10105 controls the output of various types of information to the occupants of the vehicle or to the outside. For example, the output control unit 10105 controls the output of visual and auditory information from the output unit 10106 by generating an output signal that includes at least one of visual information (e.g., image data) and auditory information (e.g., audio data) and supplying it to the output unit 10106. Specifically, for example, the output control unit 10105 synthesizes image data captured by different imaging devices of the data acquisition unit 10102 to generate an overhead image or panoramic image, and supplies an output signal containing the generated image to the output unit 10106. Also, for example, the output control unit 10105 generates audio data that includes warning sounds or warning messages for dangers such as collisions, contacts, or entering dangerous areas, and supplies an output signal containing the generated audio data to the output unit 10106.

[0036] The output unit 10106 is equipped with a device capable of outputting visual or auditory information to the occupants of the vehicle or to those outside the vehicle. For example, the output unit 10106 may include a display device, an instrument panel, a HUD (Head Up Display), audio speakers, headphones, wearable devices such as glasses-type displays worn by occupants, a projector, a lamp, etc. The display device equipped with the output unit 10106 may be a device that displays visual information within the driver's field of view, in addition to a device with a normal display, such as a head-up display, a transparent display, or a device with an AR (Augmented Reality) display function.

[0037] The drive system control unit 10107 controls the drive system system 10108 by generating various control signals and supplying them to the drive system system 10108. In addition, the drive system control unit 10107 supplies control signals to parts other than the drive system system 10108 as needed, and notifies them of the control status of the drive system system 10108, etc.

[0038] The drivetrain system 10108 includes various devices related to the vehicle's drivetrain. For example, the drivetrain system 10108 includes a drive force generating device for generating driving force such as an internal combustion engine or drive motor, a drive force transmission mechanism for transmitting driving force to the wheels, a steering mechanism for adjusting the steering angle, a braking device for generating braking force, an ABS (Antilock Brake System), an ESC (Electronic Stability Control), and an electric power steering device.

[0039] The body control unit 10109 controls the body system 10110 by generating various control signals and supplying them to the body system 10110. The body control unit 10109 also supplies control signals to other parts of the body system 10110 as needed, and notifies the control status of the body system 10110, etc.

[0040] The body system 10110 includes various body-related devices mounted on the vehicle body. For example, the body system 10110 includes a keyless entry system, a smart key system, power window devices, power seats, a steering wheel, an air conditioning system, and various lamps (e.g., headlights, reverse lights, brake lights, turn signals, fog lights).

[0041] The storage unit 10111 includes a storage medium for storing data and a controller for controlling the reading and writing of data to the storage medium. The storage medium included in the storage unit 10111 can be one or more of the following: magnetic storage devices such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disc Drive), semiconductor storage devices, optical storage devices, and magneto-optical storage devices. The storage unit 10111 stores various programs and data used by each part of the vehicle control system 10100. For example, the storage unit 10111 stores map data such as three-dimensional high-precision maps like dynamic maps, global maps with lower precision than high-precision maps but covering a wider area, and local maps containing information about the vehicle's surroundings.

[0042] One of the maps stored in the memory unit 10111 is the Local Dynamic Map (hereinafter referred to as LDM). Conceptually, the LDM consists of four layers of data, depending on the rate of change: static data (Type 1), quasi-static data (Type 2), quasi-dynamic data (Type 3), and dynamic data (Type 4).

[0043] In Figure 1, the automatic driving control unit 10112 comprises a detection unit 10131, a self-position estimation unit 10132, a situation analysis unit 10133, a planning unit 10134, and an operation control unit 10135. These detection unit 10131, self-position estimation unit 10132, situation analysis unit 10133, planning unit 10134, and operation control unit 10135 are realized by a predetermined program running on a CPU (Central Processing Unit). However, it is also possible to realize some or all of these detection unit 10131, self-position estimation unit 10132, situation analysis unit 10133, planning unit 10134, and operation control unit 10135 by hardware circuits that work together.

[0044] The automated driving control unit 10112 performs control related to automated driving, such as autonomous driving or driver assistance. Specifically, for example, the automated driving control unit 10112 performs cooperative control aimed at realizing ADAS (Advanced Driver Assistance System) functions, including collision avoidance or impact mitigation, following based on distance from the vehicle ahead, maintaining vehicle speed, collision warning, and lane departure warning. Furthermore, for example, the automated driving control unit 10112 performs cooperative control aimed at automated driving, such as driving autonomously without driver intervention.

[0045] The detection unit 10131 detects various types of information necessary for controlling autonomous driving. The detection unit 10131 includes an external information detection unit 10141, an internal information detection unit 10142, and a vehicle state detection unit 10143.

[0046] The external information detection unit 10141 performs detection processing of information outside the vehicle based on data or signals from various parts of the vehicle control system 10100. For example, the external information detection unit 10141 performs detection processing, recognition processing, and tracking processing of objects around the vehicle, as well as processing of the distance to objects. Objects to be detected include, for example, vehicles, people, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. Also, for example, the external information detection unit 10141 performs detection processing of the environment around the vehicle. The surrounding environment to be detected includes, for example, weather, temperature, humidity, brightness, and road surface conditions, etc.

[0047] The external information detection unit 10141 supplies data indicating the results of the detection process to the self-position estimation unit 10132, the map analysis unit 10151 of the situation analysis unit 10133, the traffic rule recognition unit 10152, and the situation recognition unit 10153, as well as the emergency avoidance unit 10171 of the motion control unit 10135, etc.

[0048] The in-vehicle information detection unit 10142 performs detection processing of in-vehicle information based on data or signals from various parts of the vehicle control system 10100. For example, the in-vehicle information detection unit 10142 performs driver authentication and recognition processing, driver status detection processing, passenger detection processing, and in-vehicle environment detection processing. The driver status to be detected includes, for example, physical condition, alertness level, concentration level, fatigue level, and gaze direction. The in-vehicle environment to be detected includes, for example, temperature, humidity, brightness, and odor. The in-vehicle information detection unit 10142 supplies data indicating the results of the detection processing to the situation recognition unit 10153 of the situation analysis unit 10133 and the emergency avoidance unit 10171 of the operation control unit 10135, etc.

[0049] The vehicle state detection unit 10143 performs detection processing of the vehicle's state based on data or signals from various parts of the vehicle control system 10100. The vehicle's state to be detected includes, for example, speed, acceleration, steering angle, presence and nature of abnormalities, driving operation status, power seat position and tilt, door lock status, and the status of other in-vehicle equipment. The vehicle state detection unit 10143 supplies data indicating the results of the detection processing to the situation recognition unit 10153 of the situation analysis unit 10133 and the emergency avoidance unit 10171 of the operation control unit 10135, etc.

[0050] The self-position estimation unit 10132 performs estimation processing of the vehicle's position and attitude based on data or signals from various parts of the vehicle control system 10100, such as the external information detection unit 10141 and the situation recognition unit 10153 of the situation analysis unit 10133. The self-position estimation unit 10132 also generates a local map (hereinafter referred to as the self-position estimation map) used for estimating the self-position as needed. The self-position estimation map is a high-precision map using, for example, SLAM (Simultaneous Localization and Mapping) technology. The self-position estimation unit 10132 supplies data showing the results of the estimation processing to the map analysis unit 10151, traffic rule recognition unit 10152, and situation recognition unit 10153 of the situation analysis unit 10133. The self-position estimation unit 10132 also stores the self-position estimation map in the storage unit 10111.

[0051] The situation analysis unit 10133 performs analysis processing of the vehicle's own situation and its surroundings. The situation analysis unit 10133 comprises a map analysis unit 10151, a traffic rule recognition unit 10152, a situation recognition unit 10153, and a situation prediction unit 10154.

[0052] The map analysis unit 10151 analyzes various maps stored in the memory unit 10111, using data or signals from various parts of the vehicle control system 10100, such as the self-position estimation unit 10132 and the external information detection unit 10141, as needed, and constructs a map containing information necessary for autonomous driving processing. The map analysis unit 10151 supplies the constructed map to the traffic rule recognition unit 10152, the situation recognition unit 10153, the situation prediction unit 10154, and the route planning unit 10161, action planning unit 10162, and operation planning unit 10163 of the planning unit 10134.

[0053] The traffic rule recognition unit 10152 performs recognition processing of traffic rules around the vehicle based on data or signals from various parts of the vehicle control system 10100, such as the self-position estimation unit 10132, the external information detection unit 10141, and the map analysis unit 10151. Through this recognition processing, for example, the location and status of traffic signals around the vehicle, the content of traffic regulations around the vehicle, and the lanes that can be driven on are recognized. The traffic rule recognition unit 10152 supplies data indicating the results of the recognition processing to the situation prediction unit 10154, etc.

[0054] The situation recognition unit 10153 performs situation recognition processing regarding the vehicle based on data or signals from various parts of the vehicle control system 10100, such as the self-position estimation unit 10132, the external information detection unit 10141, the internal information detection unit 10142, the vehicle state detection unit 10143, and the map analysis unit 10151. For example, the situation recognition unit 10153 performs recognition processing of the vehicle's situation, the situation around the vehicle, and the situation of the driver of the vehicle. In addition, the situation recognition unit 10153 generates a local map (hereinafter referred to as the situation recognition map) used to recognize the situation around the vehicle, as needed. The situation recognition map is, for example, an occupancy grid map.

[0055] The conditions of the vehicle to be recognized include, for example, the vehicle's position, posture, movement (e.g., speed, acceleration, direction of movement, etc.), and the presence and nature of any abnormalities. The conditions surrounding the vehicle to be recognized include, for example, the types and positions of stationary objects in the vicinity, the types, positions and movements (e.g., speed, acceleration, direction of movement, etc.) of animals in the vicinity, the composition and condition of the surrounding roads, and the surrounding weather, temperature, humidity, and brightness. The conditions of the driver to be recognized include, for example, physical condition, level of alertness, level of concentration, level of fatigue, eye movements, and driving operations.

[0056] The situation recognition unit 10153 supplies data indicating the results of the recognition process (including a situation recognition map if necessary) to the self-position estimation unit 10132 and the situation prediction unit 10154, etc. The situation recognition unit 10153 also stores the situation recognition map in the storage unit 10111.

[0057] The situation prediction unit 10154 performs predictive processing on the situation of the vehicle based on data or signals from various parts of the vehicle control system 10100, such as the map analysis unit 10151, the traffic rule recognition unit 10152, and the situation recognition unit 10153. For example, the situation prediction unit 10154 performs predictive processing on the situation of the vehicle, the situation around the vehicle, and the situation of the driver.

[0058] The predicted conditions of the vehicle itself include, for example, the vehicle's behavior, the occurrence of malfunctions, and the remaining driving range. The predicted conditions of the area around the vehicle include, for example, the behavior of animals around the vehicle, changes in traffic light status, and changes in the environment such as weather. The predicted conditions of the driver include, for example, the driver's behavior and physical condition.

[0059] The situation prediction unit 10154 supplies data indicating the results of the prediction process, along with data from the traffic rule recognition unit 10152 and the situation recognition unit 10153, to the route planning unit 10161, action planning unit 10162, and operation planning unit 10163 of the planning unit 10134.

[0060] The planning unit 10134 comprises a route planning unit 10161, an action planning unit 162, and an operation planning unit 163.

[0061] The route planning unit 10161 plans the route (journey) to the destination based on data or signals from various parts of the vehicle control system 10100, such as the map analysis unit 10151 and the situation prediction unit 10154. For example, the route planning unit 10161 sets the route from the current location to the specified destination based on the global map. Also, for example, the route planning unit 10161 modifies the route as appropriate based on conditions such as traffic congestion, accidents, road restrictions, construction, and the driver's physical condition. The route planning unit 10161 supplies data indicating the planned route to the action planning unit 10162, etc.

[0062] The action planning unit 10162 plans the vehicle's actions to safely travel the route planned by the route planning unit 10161 within a planned time, based on data or signals from various parts of the vehicle control system 10100, such as the map analysis unit 10151 and the situation prediction unit 10154. For example, the action planning unit 10162 plans starting, stopping, direction of travel (e.g., forward, reverse, left turn, right turn, change of direction, etc.), lane, speed, and overtaking. The action planning unit 10162 supplies data indicating the planned actions of the vehicle to the operation planning unit 10163, etc.

[0063] The motion planning unit 10163 plans the vehicle's actions to realize the actions planned by the action planning unit 10162, based on data or signals from various parts of the vehicle control system 10100, such as the map analysis unit 10151 and the situation prediction unit 10154. For example, the motion planning unit 10163 plans acceleration, deceleration, and the driving trajectory. The motion planning unit 10163 supplies data indicating the planned vehicle's actions to the acceleration / deceleration control unit 10172 and the direction control unit 10173 of the motion control unit 10135.

[0064] The motion control unit 10135 controls the vehicle's movements. The motion control unit 10135 includes an emergency avoidance unit 10171, an acceleration / deceleration control unit 10172, and a direction control unit 10173.

[0065] The emergency avoidance unit 10171 performs emergency detection processing such as collision, contact, entry into a dangerous zone, driver abnormality, and vehicle abnormality based on the detection results of the external information detection unit 10141, the internal information detection unit 10142, and the vehicle state detection unit 10143. When the emergency avoidance unit 10171 detects the occurrence of an emergency, it plans the vehicle's actions to avoid the emergency, such as sudden braking or sharp turns. The emergency avoidance unit 10171 supplies data indicating the planned vehicle actions to the acceleration / deceleration control unit 10172 and the direction control unit 10173, etc.

[0066] The acceleration / deceleration control unit 10172 performs acceleration / deceleration control to realize the vehicle's actions planned by the motion planning unit 10163 or the emergency avoidance unit 10171. For example, the acceleration / deceleration control unit 10172 calculates the target control value for the drive force generating device or braking device to achieve the planned acceleration, deceleration, or emergency stop, and supplies a control command indicating the calculated target control value to the drive system control unit 10107.

[0067] The direction control unit 10173 performs direction control to realize the vehicle's operation planned by the motion planning unit 10163 or the emergency avoidance unit 10171. For example, the direction control unit 10173 calculates the target control value for the steering mechanism to realize the travel trajectory or sharp turn planned by the motion planning unit 10163 or the emergency avoidance unit 10171, and supplies a control command indicating the calculated target control value to the drive system control unit 10107.

[0068] Figure 2 is a block diagram showing an example configuration of an information processing device in which the automatic driving control unit 10112 of Figure 1 is configured.

[0069] In Figure 2, the information processing device 10000 comprises a CPU 10010, a ROM (Read Only Memory) 10011, a RAM (Random Access Memory) 10012, a storage device 10013, an input / output interface 10014, and a control interface 10015, all of which are connected to each other via a bus 10020 so as to be able to communicate with one another.

[0070] The storage device 10013 is a storage medium that stores data in a non-volatile manner, and can be a hard disk drive, flash memory, or the like. The CPU 10010 controls the operation of the information processing device 10000 using the RAM 10012 as work memory, according to the programs stored in the storage device 10013 and the ROM 10011.

[0071] Input / Output I / F 10014 is an interface that controls the input and output of data to and from this information processing device 10000. Control I / F 10015 is an interface to the devices controlled by this information processing device 10000. For example, Input / Output I / F 10014 and Control I / F 10015 are connected to the communication network 10121.

[0072] For example, when the CPU 10010 and the information processing program according to the embodiment are executed, the above-mentioned detection unit 10131, self-position estimation unit 10132, situation analysis unit 10133, planning unit 10134, and operation control unit 10135 are configured, for example, as modules on the main memory area of ​​RAM 10012.

[0073] The information processing program is pre-installed on the information processing device 10000 when it is installed in a vehicle and shipped. However, it is not limited to this, and the information processing program may be installed on the information processing device 10000 after it has been installed in a vehicle and shipped. Alternatively, the information processing program can be supplied from the input / output interface 10014 via communication with an external device (such as a server) by the communication unit 10103 and installed on the information processing device 10000.

[0074] <<2. Overview of SAE's Levels of Autonomous Driving>> Next, we will describe the autonomous driving of vehicles applied to the embodiments. The Society of Automotive Engineers (SAE) defines levels of autonomous driving for vehicles. Table 1 shows the autonomous driving levels defined by the SAE.

[0075] [Table 1]

[0076] The following explanation will primarily refer to the SAE definitions of autonomous driving levels shown in Table 1. However, since the challenges and validity of autonomous driving technology when it becomes widespread have not been fully considered in the examination of the autonomous driving levels shown in Table 1, there are parts of the following explanation that do not necessarily follow the SAE definitions exactly, taking these challenges into account.

[0077] As shown in Table 1, according to the SAE, levels of autonomous driving that require human intervention in steering are classified into five stages, for example, from Level 0 to Level 4. The SAE also defines Level 5, which assumes unmanned autonomous driving only; however, this disclosure excludes Level 5 because it involves no driver involvement in steering.

[0078] Autonomous driving level 0 (Level 0) is manual driving without driver assistance from the vehicle control system (direct driving and steering by the driver), where the driver performs all driving tasks and constantly monitors for safe driving (e.g., actions to avoid danger).

[0079] Level 1 autonomous driving is manual driving (direct steering) where the vehicle control system can provide driving assistance (such as automatic braking, ACC (Adaptive Cruise Control), and LKAS (Lane Keeping Assistant System)), but the driver performs all driving tasks except for the single assisted function, and also monitors for safe driving.

[0080] Level 2 autonomous driving, also known as "autonomous driving under specific conditions," involves the vehicle control system performing subtasks related to both longitudinal and lateral vehicle control under specific conditions. For example, in Level 2 autonomous driving, the vehicle control system coordinates steering and acceleration / deceleration (e.g., coordination between ACC and LKAS). However, even in Level 2 autonomous driving, the driver is fundamentally the primary performer of the driving tasks, and the driver is also primarily responsible for monitoring safe driving.

[0081] Level 3 autonomous driving, also known as "conditional autonomous driving," allows the vehicle control system to perform all driving tasks within a limited area. In Level 3 autonomous driving, the vehicle control system is the primary entity responsible for performing driving tasks, and it is also primarily responsible for monitoring safe driving.

[0082] In Level 3 of autonomous driving as defined by the SAE, it is not clearly defined what secondary tasks the driver is actually capable of performing. "Secondary tasks" refer to actions performed by the driver during driving that are not related to driving, and are also known as NDRA (Non-driving related activity).

[0083] More specifically, it is considered that the driver may perform secondary tasks other than steering while driving at Level 3 autonomous driving, such as operating a mobile device, participating in conference calls, watching videos, playing games, thinking, or conversing with other passengers. On the other hand, within the scope of the SAE's definition of Level 3 autonomous driving, the driver is expected to appropriately take action, such as taking driving operations, in response to requests from the vehicle control system due to system failures or deterioration of the driving environment. Therefore, in Level 3 autonomous driving, in order to ensure safe driving, the driver is expected to always be in a state of readiness to immediately return to manual driving, even when performing the secondary tasks mentioned above.

[0084] Level 4 autonomous driving, also known as "fully autonomous driving under specific conditions," involves the vehicle control system performing all driving tasks within a limited area. In Level 4 autonomous driving, the vehicle control system is the primary entity responsible for executing driving tasks, and it is also the primary entity responsible for monitoring safe driving.

[0085] However, unlike Level 3 of autonomous driving, Level 4 autonomous driving does not require the driver to take action such as manually operating the vehicle in response to requests from the vehicle control system due to system failures or other issues. Therefore, in Level 4 autonomous driving, the driver can perform the secondary tasks mentioned above, and depending on the situation, they may even be able to take a nap, for example.

[0086] As described above, in autonomous driving levels 0 through 2, the vehicle operates in a manual driving mode in which the driver takes the lead in performing all or some of the driving tasks. Therefore, in these three levels of autonomous driving, the driver is not permitted to engage in secondary tasks other than manual driving and related actions that may impair attention or forward focus while driving.

[0087] On the other hand, in autonomous driving level 3, the vehicle operates in an autonomous driving mode in which the vehicle control system proactively performs all driving tasks. However, as explained earlier, situations may arise in autonomous driving level 3 where the driver needs to take control of the vehicle. Therefore, in autonomous driving level 3, if the driver is allowed to perform a secondary task, the driver is required to be prepared to return to manual driving from that secondary task.

[0088] Furthermore, even at autonomous driving level 4, the vehicle operates in an automated driving mode where the vehicle control system performs all driving tasks. However, in sections where autonomous driving level 4 is normally applicable, there may be parts of the section where it cannot be applied due to the actual road infrastructure conditions, etc. Such sections are expected to be set to autonomous driving level 2 or lower, requiring the driver to actively perform driving tasks. Therefore, even if autonomous driving at autonomous driving level 4 is being used during the planning stage of the journey or after the start of the journey, if circumstances arise that deviate from the conditions under which its use is permitted, a request to transition to autonomous driving level 2 or lower may occur as described above. For this reason, drivers are required to be prepared to return to manual driving from secondary tasks as needed, even if it was not planned at the beginning of the journey, if these changes in conditions become apparent.

[0089] Here, the actual range of use permitted for each of these different levels of autonomous driving is called the ODD (Operation Design Domain). More specifically, the ODD is the driving environment conditions that are assumed to be the basis for the operation of the autonomous driving system in terms of design. The autonomous driving system will operate normally and the vehicle will be driven autonomously only when all the conditions shown in the ODD are met. Furthermore, if the conditions shown in the ODD are not met during driving, it is necessary to switch the vehicle's driving control from autonomous driving to manual driving. Note that the conditions shown in the ODD generally differ depending on each autonomous driving system, as well as the deterioration, contamination of sensors, etc., and performance fluctuations at any given time based on the self-diagnostic results of onboard devices that control autonomous driving.

[0090] Figure 3 is a schematic diagram illustrating each SAE autonomous driving level from the user's perspective as a usage scenario.

[0091] The environments in which autonomous driving level 0 is applicable are generally all types of public road infrastructure, including private roads, public roads, and highways. The environments in which autonomous driving level 1 is applicable are roads equipped with driver assistance devices and environments, such as some main roads and highways. In autonomous driving level 1, existing manually driven vehicles must be equipped with driver assistance systems such as ACC and LKAS, which are provided by the vehicle control system. In this case, decreased driver attention becomes an intuitive risk because the driver assistance system does not provide comprehensive support.

[0092] Furthermore, Level 2 autonomous driving is applicable to certain road sections, such as highways, provided that the necessary driving assistance devices and environment are in place. In sections where Level 2 autonomous driving is applicable, it is permitted to combine automatic acceleration and deceleration in the direction of travel using driving control systems such as ACC, with automatic lateral control relative to the direction of travel, such as LKAS, which keeps the vehicle in its lane. This requires the driver to continue to pay close attention to their driving. On the other hand, in sections where Level 2 autonomous driving is applicable, driving itself can proceed as usual if there are no interfering factors, so if driving assistance becomes too sophisticated, it may lead to a decrease in the driver's sense of risk. Therefore, Level 2 autonomous driving can also be said to be an autonomous driving level that requires preventative measures to avoid a decrease in driver attention.

[0093] As mentioned above, in these sections of autonomous driving from level 0 to 2, the vehicle's movement is controlled by the driver's manual operation.

[0094] On the other hand, sections designated as Level 3 and Level 4 autonomous driving are, as mentioned above, sections where autonomous driving control by the vehicle's autonomous driving system is possible. Of these, an environment where Level 4 autonomous driving is applicable may be achieved, for example, by ensuring sections where information of each type in LDM is constantly updated and road predictability is guaranteed.

[0095] In contrast, environments in which Level 3 autonomous driving is applicable may include sections where Level 4 autonomous driving is normally possible, but which for some reason cannot meet the ODD conditions corresponding to Level 4 autonomous driving. For example, sections where only quasi-static data can be obtained in LDM, or sections where the driving conditions for Level 4 autonomous driving cannot be continuously secured due to a decrease or deficiency in the system's environmental adaptability. Sections where Level 4 autonomous driving cannot be secured may include sections under temporary construction, flooded sections, complex intersection sections, sections with missing LDM, sections with temporary communication bandwidth shortages, and sections where a risk alert has been issued by the vehicle ahead.

[0096] Furthermore, environments where Level 3 autonomous driving is applicable may include sections that are functionally passable with Level 4 autonomous driving control, but where the application of Level 4 autonomous driving is canceled for some reason. Examples of such sections include sections where stopping or slowing down for emergency evacuation due to Minimum Risk Maneuver (MRM) poses a risk of causing flow stack, disrupting the smooth flow of traffic infrastructure, construction zones, railway crossings, etc. In addition, sections where driver-unassisted passage is prohibited by law, whether fixed or proactively set (where violations are subject to penalties due to institutional preventative measures), can also be environments where Level 3 autonomous driving is applicable.

[0097] Furthermore, as shown by the black arrows in Figure 3, even in sections where, from the user's perspective, only autonomous driving up to Level 2 is permitted at high speeds under normal road usage conditions with smooth traffic flow and no congestion, if conditions are met that temporarily allow autonomous driving at Level 3 or higher due to congestion, then an operation is being considered that would enable the use of autonomous driving functions such as ACSF (Automatically Commanded Steering Function). For example, even in sections where autonomous driving is not originally intended, such as congested sections on expressways where autonomous driving Level 2 is applied, autonomous driving at Level 3 or Level 4 may become possible. In particular, if autonomous driving at Level 4 is permitted, the safe execution of NDRA becomes possible. In this case, it is necessary to be able to predict the end of congestion and manage the return to manual driving.

[0098] Here, we consider a scenario in which autonomous driving is used when a vehicle enters a Level 4 autonomous driving section from a Level 2 autonomous driving section, and the vehicle's driving control switches from manual driving by the driver to autonomous driving by the vehicle control system. In this case, the driver no longer needs to concentrate on driving the vehicle, and their ability to maintain attention decreases. In other words, the switch from manual driving to autonomous driving can be considered a usage scenario that leads to a decrease in the driver's ability to maintain continuous attention.

[0099] Furthermore, considering the switch from autonomous driving level 4 to autonomous driving level 2 (return to manual driving), the section where autonomous driving at autonomous driving level 4 is performed is a usage area where the driver's ability to maintain attention is reduced, and it is necessary to formulate a detailed time budget for the return to autonomous driving based on prior monitoring information of the driver in a steady state. In existing technologies, autonomous driving level 4 uniquely notified the driver of the return to autonomous driving level 2 or lower, i.e., manual driving.

[0100] The role of functions equivalent to autonomous driving level 3 can be described as a bridge to prevent the sections where autonomous driving at autonomous driving level 4 is performed from being separated from sections where manual driving at autonomous driving levels 2 or lower is performed. In other words, the usage of autonomous driving at autonomous driving level 3 is a usage pattern in which the driver is expected to maintain attention to driving and take corrective action in a short time (e.g., a few seconds). In autonomous driving level 3, the detection of a decline in the driver's attention by the DMS (Driver Monitoring System) that monitors the driver, and the driver's continued attention are essential requirements for using autonomous driving levels 3 or lower.

[0101] Figure 4 is a schematic diagram illustrating the application of Level 3 autonomous driving. The map in Figure 4 shows a scenario where the vehicle travels from the starting point ST to the ending point EP, following the journey TP (shown as a shaded area in the diagram) in the direction indicated by the arrows (counterclockwise, left-hand direction).

[0102] In Figure 4, sections RA1, RA2, and RA3 represent sections corresponding to, for example, autonomous driving levels 0 to 2, where manual driving is mandatory. When a vehicle enters sections RA1 to RA3 while driving autonomously at, for example, autonomous driving level 4, the autonomous driving system must transfer driving control from autonomous driving by the system to manual driving by the driver, such as steering. On the other hand, sections RB1 to RB5 represent sections where autonomous driving can be continued under careful monitoring of the transition from autonomous to manual driving. Sections RB1 to RB5 correspond to, for example, autonomous driving level 3.

[0103] In order to enter sections RA1, RA2, and RA3, where manual driving is mandatory, the driver needs to prepare themselves to switch from automated driving back to manual driving. For this reason, sections RB1, RB4, and RB5, corresponding to, for example, automated driving level 3, are set up on the entry side of sections RA1, RA2, and RA3.

[0104] On the other hand, sections RB2 and RB3 are sections that, functionally, can be traversed using Level 4 autonomous driving control, but are designated as sections where the application of Level 4 autonomous driving is canceled for some reason. Section RB2 is, for example, a temporary construction zone or a flooded zone, and section RB3 is, for example, a section where caution is required when driving due to sharp curves.

[0105] Thus, if a return to manual driving occurs while driving at Level 4 autonomous driving, if there is sufficient time before reaching the point where the system requests a return and the handover is required, the driver will pass through a Level 3 autonomous driving section or a state where the driver's driving ability has been restored, allowing them to pay attention to their surroundings. In Level 3 autonomous driving, the driver is not directly involved in driving but must maintain attention to the situation. Therefore, in operations where the driver is required to wait for a long time without actually steering, only paying attention, it may become a source of discomfort for the driver.

[0106] <<3. Embodiments relating to this disclosure>> Next, embodiments relating to this disclosure will be described. In the following, unless otherwise specified, ODD refers to an ODD corresponding to autonomous driving level 4 and indicates the conditions for autonomous driving level 4. Furthermore, a driving section that satisfies the conditions indicated by the ODD will simply be referred to as an ODD section.

[0107] <3-1. Overview of Embodiments> First, we will explain the outline of the embodiment in comparison with existing technologies. Figure 5 is a flowchart that schematically shows an example of the process of handing over from automated driving to manual driving using existing technologies. Prior to the start of the process shown in the flowchart of Figure 5, it is assumed that the vehicle with the driver on board is traveling in an ODD section corresponding to automated driving level 4.

[0108] As the end of the ODD section approaches, the automated driving system installed in the vehicle notifies the driver in step S10 that the end of the ODD is approaching. The driver, for example, prepares to switch driving control from automated driving to manual driving in response to this notification. If the switch of driving control from automated driving to manual driving is delayed beyond a predetermined time, the automated driving system issues an alarm to the driver (step S11).

[0109] In step S12, the automated driving system determines whether the handover of driving control from automated driving to manual driving was completed within a predetermined time after the notification of the ODD end point in step S10. If the automated driving system determines that the driver has completed the handover within the predetermined time (step S12, "OK"), it proceeds to step S13 to evaluate the completion of the handover.

[0110] On the other hand, if the automated driving system determines in step S12 that the handover of driving control from automated driving to manual driving has not been completed within a predetermined time (step S12, "NG"), it proceeds to step S14. In step S14, the automated driving system applies MRM to the control of the vehicle and performs evasive driving, for example, an emergency stop on the shoulder of the road.

[0111] Here, the concept of autonomous driving control for vehicles using existing technologies is that the level of autonomous driving that a vehicle can operate at is determined by the ODD, which is the design assumption range of the vehicle's onboard equipment. The driver is always required to follow and respond to all demanding situations according to the level of autonomous driving that the vehicle is capable of operating autonomously.

[0112] For example, suppose a particular highway allows Level 4 autonomous driving, and the vehicle's onboard equipment also allows for autonomous driving equivalent to Level 4. In this case, the driver can use and drive the vehicle at Level 4 autonomous driving in that section. When the autonomous driving system approaches a situation where the vehicle is about to leave the ODD section where Level 4 autonomous driving is permitted, it prompts the driver to return to manual driving (Figure 5, step S10), and if the response is delayed, it simply issues a warning (Figure 5, step S11). If the autonomous driving system fails to return to manual driving at the appropriate time despite issuing a warning in step S11, it is assumed that the system will transition to an emergency forced evasive steering maneuver, also known as MRM, within the ODD section where Level 4 autonomous driving is permitted (Figure 5, step S14), thereby preventing the vehicle from entering a section that the autonomous driving system cannot handle.

[0113] In existing vehicle control technologies, it is assumed that, depending on the performance limits of the autonomous driving system, the driver will engage in specific secondary tasks (NDRAs) other than driving, as long as the section of the road is permitted for Level 4 autonomous driving. On the other hand, when the autonomous driving system reaches its limits, it issues a forced request to the driver to switch from autonomous driving to manual driving. From the user's perspective, this means being forced to return from engaging in secondary tasks.

[0114] Thus, when using existing vehicle autonomous driving systems, the driver is forced into a subordinate relationship with the autonomous driving system. Consequently, for the driver, engaging in secondary tasks using the autonomous driving function was a stressful form of use or control.

[0115] Figure 6 is a flowchart illustrating a schematic example of the handover process from automated driving to manual driving according to this embodiment. Prior to the start of the process shown in the flowchart of Figure 6, it is assumed that the vehicle with the driver is traveling in an ODD section corresponding to automated driving level 4.

[0116] In step S20, the automated driving system notifies the driver in advance of the end of the ODD section. For example, the automated driving system notifies the driver of the end of the ODD earlier than the time it is estimated that will take the driver from the time they receive the notification until they resume manual driving.

[0117] In the next step, S21, an "agreement" regarding the handover start point is made between the autonomous driving system and the driver. This "agreement" refers to a series of actions in which the driver explicitly responds to notifications issued by the autonomous driving system. At this time, the autonomous driving system presents the driver with information indicating the point where the handover must be completed, and the risks of not completing the handover. It should be noted that this "agreement" is merely a sharing of information regarding the handover between the autonomous driving system and the driver, and does not impose any obligations on the driver; therefore, it should actually be called a "provisional agreement."

[0118] In this way, by establishing a provisional agreement regarding the handover start point through the driver's explicit response, the handover process can be imprinted on the driver's working memory. Working memory, also known as operational memory, is the memory capacity of the human brain that temporarily stores and processes information necessary for tasks and actions.

[0119] In the next step, S22, the automated driving system manages the process of handing over to manual driving by the driver. For example, the automated driving system monitors the driver's status and, based on the monitoring results and the vehicle's status at that time, determines in step S22 the margin of safety from the current point to the handover start point, and whether it is possible to extend the grace period until the handover starts. Depending on the determination result, the automated driving system takes further action, such as issuing notifications or transitioning to MRM.

[0120] The "margin of error" referred to here is the amount of time that can be secured that is longer than the time it would take for the vehicle to reach the handover completion limit point if it were traveling at a cruising speed estimated from the flow of surrounding vehicles on the road, compared to the time it would take for the driver to resume driving based on the driver status analysis detected through continuous passive monitoring. Furthermore, the "extension of the grace period before the start of the handover" mentioned above refers to extending the time it takes to reach the handover completion limit point by, for example, reducing the vehicle's speed, moving to the shoulder, temporarily moving to a vacant lot, or moving to a low-speed lane, without interfering with the flow of surrounding cruising vehicles.

[0121] In the next step, S23, the automated driving system determines whether the handover of driving control from automated driving to manual driving was completed within a predetermined time, for example, as set in the handover process management in step S22. If the automated driving system determines that the handover was not completed within the predetermined time, it has the vehicle move to a safe location and perform an emergency stop using MRM, similar to step S14 in Figure 5.

[0122] In the next step, S24, the automated driving system evaluates the driver's completion of the handover to manual driving. At this time, the automated driving system calculates an evaluation score according to the driver's actions during the handover. For example, the automated driving system adds points to the evaluation score for desirable actions during the handover, such as when the driver voluntarily initiates the handover process or when a provisional agreement is made. Also, if the driver abandons the handover in advance and chooses to take a break, this is considered a desirable action in terms of impact on social infrastructure, as it is equivalent to selecting a means to prevent interference with the driving of surrounding vehicles, and therefore points are added to the evaluation score. On the other hand, the automated driving system deducts points from the evaluation score according to the degree of impact for undesirable actions during the handover, such as when the handover process is initiated only after repeated warnings, when the handover process is initiated only when an urgent situation has occurred, or when the vehicle is controlled by MRM and the risk of the handover failing increases.

[0123] In the next step, S25, the autonomous driving system provides the driver with an incentive based on the evaluation score calculated in step S24. NteAn incentive or penalty may be imposed. For example, if an autonomous driving system adjusts the evaluation score based on a score of 0, and the evaluation score calculated in step S24 exceeds 0, an incentive may be imposed on the driver. Nte The system then assigns a penalty. On the other hand, if the evaluation score calculated in step S24 is lower than 0, the automated driving system imposes a penalty on the driver. The lower the evaluation score, the heavier the penalty. Possible penalties include restrictions on the driver's use of automated driving or restrictions on the driver's participation in secondary tasks.

[0124] In this way, drivers receive incentives based on their performance in the handover process. Nte By imposing a reward or penalty, it becomes possible to imprint the driver's working memory, including the risks, during the provisional contract in step S21 described above (step S26).

[0125] In the embodiments of this disclosure, the automated driving system continuously monitors and observes the driver's condition and evaluates the degree of the driver's ability to return to manual driving. To ensure that the return to manual driving can be performed properly and without delay, the automated driving system constantly provides the driver with advance notice of the need to return to manual driving while the vehicle is under autonomous steering control. Prior to the actual start of the return, the automated driving system enters into a "contract" with the driver regarding the determination of an appropriate start time for the return and manages the handover process for that start time.

[0126] In the embodiments of this disclosure, the aim is to provide a comfortable user experience of autonomous driving by implementing human-centered interactive control of autonomous driving to achieve a smooth and error-free return to manual driving. In other words, in the embodiments of this disclosure, instead of the autonomous driving system unilaterally notifying the driver of a Transition Demand, which is a request to return to manual driving, based solely on the vehicle's current status, the aim is for a cooperative handover control between the system and the driver, sharing prior knowledge of the handover with the driver and also engaging the driver's memory.

[0127] In other words, the automated driving system according to the embodiment of this disclosure notifies the driver of each handover start point earlier than the estimated time required for the driver to return to manual driving after receiving notification of the handover start point (step S20 in Figure 6). The driver then enters into a "provisional agreement" with the automated driving system regarding the actual handover start point (step S21 in Figure 6). Based on this provisional agreement, the automated driving system manages the handover process and performs budgeting and risk diversification of the handover sequence (step S22 in Figure 6). This enables the driver to perform appropriate preparation for the completion of secondary tasks and to grasp in advance the conditions necessary for manual driving (for example, understanding the surrounding environment necessary for driving control by manual driving).

[0128] In the embodiment of this disclosure, the "contract" (Figure 6, step S21) that the system enters into with the driver also plays a role in retaining in the driver's visual cortex memory the importance of the handover and an approximate sense of time, as will be described later, by devising the method of provision.

[0129] By establishing a prior agreement between the autonomous driving system and the driver regarding the transition from autonomous to manual driving, the autonomous driving system can ensure that the importance of the transition is reliably communicated to the driver. The autonomous driving system can reliably incorporate decision-making information related to the driver's actions into their working memory as "pre-emptive information" by presenting the driver in advance the points where the transition must be completed and the risks of an incomplete transition. As a result, unlike with existing technologies where the driver begins to understand the situation after receiving notification, certain important pre-emptive information is taken into memory in advance, preventing or at least reducing the possibility of making a mistake in deciding to initiate the transition due to inattention.

[0130] Furthermore, according to the embodiments of this disclosure, even if there is a risk of errors occurring and oversights in the decision to initiate a handover due to the driver's dementia or other cognitive impairment, it is possible to monitor the driver's repeated return habits each time and detect signs of such errors. Therefore, according to the automated driving system of the embodiment, the effect based on this prior "contract" (i.e., the expected execution of the return work) is reduced, which incidentally makes it possible to use it to identify signs of dementia or other cognitive impairment in the elderly and others.

[0131] <3-2. About the Human-Centered Design (HCD) according to the embodiment> The autonomous driving system according to this embodiment applies Human-Centered Design (HCD), a design philosophy centered on people (drivers, etc.), instead of Machine-Centered Design (MCD), a design philosophy centered on devices and systems that is commonly used in existing systems. In other words, the autonomous driving system according to this embodiment performs cooperative control that incorporates human behavioral characteristics into vehicle control.

[0132] <3-2-1. Overview of HCD according to the embodiment> First, an overview of the HCD according to the embodiment will be described. In existing MCDs, the automated driving system mechanically determines the ODD (Operational Design Unit) in which automated driving can be used, according to the performance of the onboard equipment installed in the vehicle, and uniquely permits the use of the automated driving function within that limited range. In this case, even if the user over-relies on the automated driving function, the control that the system provides to the user is limited to unilateral control instruction notifications, warnings, or MRM (Minimal Risk Maneuver) only when it is impossible to respond.

[0133] In contrast, the HCD (Human-Machine Interface) described in this disclosure controls whether or not users can use the autonomous driving function, ensuring that its use progresses within a socially acceptable range. Specifically, the control of whether or not to use the function takes into account characteristics derived from the driver's behavioral habits, and allows use if appropriate behavioral habits are present. On the other hand, for inappropriate behavioral habits (failure to respond to requests to return to manual driving, delays in returning to manual driving, deterioration of the quality of the return action, etc.), an HMI (Human-Machine Interface) is incorporated to encourage behavioral change in the user, and the range of available autonomous driving functions is proactively adjusted on an individual basis according to the degree of adaptation.

[0134] Implementing such HCD is difficult with simple function implementation; it requires multi-stage, hierarchical, multi-dimensional, and dynamic information feedback that encourages changes in human user behavior. In this disclosure, the concept of a "contract" is introduced, and the information feedback necessary to implement HCD is carried out through a contract between the system and the person (driver).

[0135] Specifically, it will be as follows: • A "contract" is concluded between the system and the driver before commencing use of each permitted section of the autonomous driving function. • Before reaching the end of the usage section covered by the contract, the system and the driver will confirm the "ancillary contract" which stipulates that the handover to manual driving will be completed quickly and safely without sudden stops or slowing down. • The driver's obligation to reconfirm changes in the usage status as time elapses. • Granting credit to drivers (driver credit) is a way to assess the individual effectiveness of the incidental obligations in the "contract" related to the repeated use of autonomous driving. • Based on the performance evaluation history regarding past obligations to return to manual driving, i.e., the driver's creditworthiness, the ODD (Operational Design Decision) for the range of autonomous driving permitted on recently used roads will be redefined, taking into account the driver's individual characteristics upon returning to manual driving.

[0136] The system provides this information to the driver as visual information, and changes after the start of each permitted section are also provided (as an ancillary contract) as visual information for confirming changes in the situation. For example, if the conditions for ending autonomous driving differ from the conditions at the start of use, the system provides visual feedback to the driver using visual information that depicts the factors that should be addressed when ending autonomous driving and the risks and consequences of not responding to the termination of autonomous driving. Because the information presented to the driver through the HMI reflects the information material for risk judgment in situations where the return action was disregarded, the driver's memory becomes clearer as they are affected by the psychological impact of disregarding the request to return to the vehicle.

[0137] Incidentally, memories retained in working memory deteriorate over time. In particular, when there is no immediate need to take over manual driving, if NDRA (Non-driving Related Activity) such as watching television continues, the idea of ​​handing over to manual driving ceases to be a primary concern. Therefore, the re-recognition of the necessity event is carried out by information processed in the subconscious mind from brain activity that occurs outside of conscious awareness. Consequently, HMIs that incorporate risk memories into the subconscious mind using subliminal techniques and revive interest in handing over driving are important.

[0138] Furthermore, the system may also prompt the driver to forcibly look ahead and confirm the direction of travel, request pointing and calling out to confirm the direction of travel, and evaluate the results.

[0139] To realize this HCD-based control, it is necessary to continuously reconstruct the memories required for recovery, taking into account individual differences and the working memory capacity of the situation in which the person is placed. The problem here is that human memory is not information that can be directly observed from the outside. Therefore, in this disclosure, the system realizes the continuous reconstruction of memories required for recovery through a "contract" made with the driver.

[0140] <3-2-2. Advantages of HCD in Autonomous Driving> Next, we will explain the advantages of the HCD related to this disclosure in autonomous driving.

[0141] The social introduction of autonomous driving technology will have a significant impact on users' long-term use and how they interact with it, depending on the implementation process. For the introduction of autonomous driving technology to be successful without negative social side effects, it is necessary to appropriately mitigate the potential drawbacks of autonomous driving. In other words, if the functions made possible by technological development are provided without limit, without considering human behavior and psychology, users may not necessarily utilize the technology within a socially acceptable range.

[0142] The existing concept of introducing autonomous driving into society involves gradually introducing autonomous driving functions in accordance with the level of technological achievement (such as the SAE's autonomous driving levels). In other words, it involves gradually expanding the use of functions that can be performed automatically as technological development progresses, thereby advancing social implementation. In other words, the existing concept of introducing autonomous driving into society refers to the idea of ​​gradually advancing the introduction of autonomous driving into society based on the results of technological development, that is, according to the performance achieved in machine development, at the performance levels of autonomous driving functions defined as autonomous driving levels 2 to 4 by the SAE, etc.

[0143] In contrast, the technology disclosed herein dynamically changes the autonomous driving functions provided and dynamically controls the degree of tolerance for autonomous driving control, in accordance with the driver's ability to adapt to the device using the autonomous driving function, that is, in accordance with whether the driver has the passive ability to appropriately utilize the technology.

[0144] In other words, in this disclosure, even if the mechanical and functional configuration of the autonomous driving system is exactly the same, the actual provision of autonomous driving functions will dynamically change the functions that the user (driver) can actually use, depending on whether the user (driver) possesses the behavioral adaptability to safely use the functions. This disclosure relates to technology that applies this HCD concept to the control and operation of an autonomous driving system.

[0145] <3-2-2-1. Regarding excessive dependence> Appropriate use of autonomous driving functions requires drivers to avoid excessive reliance on them. The simplest example of excessive reliance on autonomous driving functions is when, despite the design clearly assuming driver involvement in control, the driver neglects their duty to pay attention to the road ahead and monitor their surroundings. In this case, the driver's attention to the vehicle in front may decrease, and they may become inattentive to the distance between vehicles. That is, for example, when using an autonomous driving function limited to functions such as a lane keeping assist system, even though it is a limited auxiliary function, if there are no other vehicles or obstacles approaching in front of or behind the vehicle that could interfere with it, the driver may become reliant on this autonomous driving function. If the driver feels a sense of security from these assistance functions, it can lead to a decrease in driving attention, potentially resulting in delayed judgment or excessive evasive actions in emergency situations.

[0146] When situations involving decreased attention occur, drivers may experience delays in responding to emergencies, excessive evasive maneuvers, or even be unable to respond at all. In such cases, they may panic and perform emergency deceleration, which could lead to secondary damage such as rear-end collisions or traffic jams caused by the deceleration and panicked actions. As autonomous driving at Level 2 becomes capable of automatically controlling steering even in more complex situations, the frequency of the driver's direct involvement in steering decreases. Drivers may become complacent with these driver assistance functions and tend to neglect necessary attention, such as maintaining vigilance in preparation for situations where action is actually required.

[0147] The introduction of autonomous driving levels higher than Level 2 complicates this issue further, as drivers are no longer required to constantly pay attention to the road ahead. If the system determines that continuing to drive under automatic control exceeds the limits of the onboard equipment's situational awareness and response capabilities, and the vehicle is operating at Level 4 autonomous driving, the system must initiate procedures to abandon autonomous driving and appropriately hand over control to the driver. Alternatively, if a sudden emergency makes it difficult for the driver to take over, the system must automatically initiate risk minimization measures. In this case, if the driver does not promptly return to manual driving according to the system's instructions, or neglects to take measures such as returning to manual driving, it may delay the time it takes for the system to reach its limit, potentially leading to situations where the vehicle slows down against the speed of surrounding vehicles, initiates MRM (Measurement Monitoring), or takes other accident prevention measures. This is a prime example of over-reliance on autonomous driving.

[0148] Similarly, even with Level 2 and Level 3 autonomous driving, as the assistance becomes more sophisticated, drivers will experience how the system can handle situations appropriately and safely without frequent steering interventions from the driver under many driving conditions. Because a sustained decrease in attention while driving does not directly translate into a sense of risk, drivers become complacent with situations where the system handles situations. As a result, drivers may become accustomed to relying on autonomous driving, and their skepticism towards the system's imperfections may diminish.

[0149] In Level 2 and Level 3 autonomous driving, drivers are required to take immediate action in emergency situations, so a decrease in attention is unacceptable. In contrast, in Level 4 autonomous driving, this decrease in attention is a real possibility. Furthermore, while Level 2 and Level 3 autonomous driving require drivers to maintain continuous attention, in reality, for example from an ergonomic standpoint, there is no guarantee that drivers can always fulfill this duty of attention.

[0150] In other words, the design functions of a machine unilaterally force users to understand and utilize its individual design limitations in accordance with the performance achieved during development. When technology is introduced into society under the assumption that users will utilize it as expected in line with its design performance, the problem of excessive dependence on that technology remains. Generally, from a human psychological perspective, people tend to be skeptical of new and unknown technologies and unconsciously take precautions to deal with them. However, as the development and spread of autonomous driving progresses, and the functions of autonomous driving become more advanced and diverse, and anxiety and skepticism regarding its use gradually decrease, this problem of excessive dependence will become increasingly significant and potentially problematic.

[0151] This disclosure addresses this essential issue not simply as a matter of decreased awareness or attention, and addresses it with systems to prevent such decrease in user attention (warnings, alertness recovery, etc.), but rather concerns the technology necessary to introduce a series of mechanisms that enable users' behavioral psychology to naturally self-learn and adapt to the limits of autonomous driving performance. In other words, this disclosure provides a series of controls necessary to encourage behavioral improvement and modification that gradually changes the user's repeated usage behavior, and a mechanism that acts on the driver in a hierarchical structure to promote improvement in that usage behavior.

[0152] <3-2-2-2. About HCD> The key point of this disclosure is to change the relationship between the vehicle and the driver from the existing MCD to HCD, and to determine the system's operating domain based on how the person behaves. The determined operating domain then influences the benefits the user receives from using the vehicle, depending on the person's behavioral routines. The system is designed to create a situation that the user finds comfortable, and to weight the feedback loop so as not to unintentionally disrupt social activities (such as traffic congestion, rear-end collisions, or road closures). This disclosure concerns an HMI that is effective in maintaining a virtuous cycle between this weighted system control and the development of human behavioral habits.

[0153] In other words, users are required to move beyond simply relying on the functions provided when using existing autonomous driving vehicles and instead adopt a behavioral change towards collaborative use. To bring about this change in user behavior, a mechanism is needed to generate that change. This disclosure proposes a mechanism for generating this collaborative behavioral change and the technology for the overall operation of that mechanism.

[0154] For this form of HCD to be possible, user behavior change is essential, and HMI (Human-Machine Interface) is also necessary to bring about behavioral change. HCD is not simply a system that allows users to use functions as they please, but rather a whole system that prompts users to naturally take the necessary actions to comfortably use the functions. Human behavior can be redefined not as a design that allows use according to the animalistic instincts of humans, but as a design that incorporates a system that involves spontaneous actions and behavioral changes necessary to maintain social order in modern society (or to be able to follow the rules), and then the functional design required for that purpose.

[0155] Let me explain in more detail. First, when introducing autonomous driving into society, its usability will depend on various conditions, but it is a prerequisite that the system has the function to automatically control the steering of the vehicle.

[0156] The minimum requirement is for the vehicle to acquire information from external sources, supplement that information to understand the environment, plan its own driving, and then drive according to that plan. Furthermore, if it cannot be confirmed that the system can perform this series of processes under all conditions, then autonomous driving at levels 3 or 4, exceeding autonomous driving level 2, may be permitted, depending on whether or not a quick return to manual driving by the driver is required.

[0157] Furthermore, for example, in Level 4 autonomous driving, there is a possibility that the autonomous driving operation may end one or more times before the vehicle reaches the end point of its journey. In this case, the sequence of handing over to manual driving when autonomous driving ends will inevitably be included multiple times within the journey.

[0158] Here, the SAE has defined Level 5 autonomous driving, which is even more advanced than Level 4. Level 5 autonomous driving is operated in a closed environment, or with a fully functional LDM (Low-Resolution Motor) that provides higher-resolution, higher-refresh-rate information updates than the surroundings, achieved through significant infrastructure investment in acquiring environmental information. It is applied to applications such as robot taxis. Unless the vehicle is operated in a manner similar to a robot taxi at Level 5 autonomous driving, vehicles that enable Level 4 autonomous driving for general users will require drivers to switch from autonomous driving to manual driving at various points during their journey.

[0159] The requirements for autonomous driving functionality in a vehicle are determined by the limits of what can be achieved through the vehicle's design and development. In this case, it is possible to extend the limits of autonomous driving by investing more in equipment and infrastructure development, such as increasing the amount of information acquisition resources that can be allocated to optimal processing, increasing the amount of autonomous or externally acquired and retrieved resources, and allocating power and cost resources to calculations. On the other hand, although the frequency of requests for return to manual driving and the usable range of autonomous driving may differ, it is extremely difficult to completely eliminate situations where a return from autonomous driving to manual driving is required. Therefore, an HCD that enables appropriate driver intervention in response to these handover requests that occur when using the vehicle is needed as an alternative to existing MCDs.

[0160] When changing the system configuration from MCD, which is dependent on the performance of the equipment, to HCD, which emphasizes cooperation with humans, it is necessary to encourage appropriate use without becoming overly dependent on the user. To achieve this, the system needs to allow the driver to learn through use the balance between the benefits that users can gain from using autonomous driving and the losses and risks incurred in order to enjoy those benefits. A mechanism is needed that allows users to comfortably use the system while assuming the necessary obligations, thereby extracting the benefits within that balance.

[0161] <3-2-2-3. Benefits for Drivers> So, from the perspective of what the benefits will be for vehicle users, the desired benefits will be one or more combined, as shown below.

[0162] First, here are some examples of actions that can lead to benefits. (1) The objective is to purely achieve the movement from starting point A to destination point B. (2) To achieve comfortable travel between those two points. (3) Achieve travel on a lower budget. (4) Achieve travel in a shorter time. (5) Achieve travel within the scheduled time. (6) Achieve travel with less fatigue. (7) To accomplish the intended journey even when exhausted or unwell. (8) Get away from the scene immediately. (9) To achieve the purpose of transporting necessary goods. (10) Go for a drive outdoors in good weather or in a scenic location for recreation. (11) The employee may engage in tasks other than driving (secondary tasks, i.e., NDRA) as appropriate during travel.

[0163] Examples of NDRAs performed during travel, as mentioned in item (11) above, include the following: (11-1) Food and drink (11-2) Browsing using mobile devices, etc. (11-3) Texting emails (11-4) Conducting teleconferences (11-5) Taking time off from work, sorting and packing delivery items, etc. (11-6) To apply makeup or groom oneself. (11-7) Engaging in activities such as karaoke, watching movies, and watching sports broadcasts on television. (11-8) Operation of terminal devices such as smartphones, mobile phones, tablet computers, notebook computers, etc. (11-9) Enjoying the scenery while traveling (11-10) Check the contents of bags, etc., and search for lost items. (11-11) Conversation and interaction with other attendees, games such as crosswords. (11-12) Other e-sports (11-13) Benefits of using autonomous driving as a way to reduce the burden, limited to traffic congestion, etc. (11-14) First-line treatment for temporary physical discomfort (e.g., leg cramps) (11-15) Temporary vision loss (11-16) Use of eye drops, and support for temporary vision loss associated with eye drop application. (11-17) Managing asthma and epileptic seizures (11-18) If driving in a section where it is safe to drive at autonomous driving level 4, a temporary nap may be taken during that time. (11-19) Execution of undefined processing

[0164] Further examples of actions taken to gain benefits include the following: (12) Maximum continuous availability of autonomous driving functions (13) No loss of use occurs as a result of use. (14) Do not cause trouble to stakeholders.

[0165] If, during autonomous driving, it becomes necessary to switch from autonomous to manual driving, and the driver is unable to properly return to manual driving when the system issues a request to do so, as mentioned above, emergency deceleration or evasive stopping may become necessary as a Measuring and Responding Motor Vehicle (MRM). This could expose the negative aspects of autonomous driving, such as blocking pedestrian and logistics routes, causing congestion, and inducing rear-end collisions.

[0166] From the perspective of an individual user, the secondary losses resulting from these emergency controls, except in cases where their own vehicle is rear-ended, only affect following vehicles and are not considered to diminish the benefit of encouraging appropriate recovery actions. In other words, if the use of autonomous driving is simply left to the driver's good judgment from a human-centered design (HCD) perspective, there is a risk that social order will not be maintained.

[0167] Therefore, in order to maintain social order and encourage appropriate user behavior while introducing HCD control, a mechanism is needed to motivate drivers to return to their vehicles without delay when requested, while obtaining the benefits listed in items (1) to (14) above. However, moral motivation is merely an ideal, and even if drivers using autonomous driving are educated to behave morally, such as not ignoring or delaying requests to return to their vehicles, this does not necessarily guarantee effective and appropriate returns.

[0168] To elicit behavioral changes in drivers that encourage them to respond appropriately and promptly to system reset requests, the reset request must directly influence the driver's behavioral psychology. This is because drivers must accept the risk of disregarding the reset request in exchange for the benefits they receive. In other words, a mechanism is needed that provides some form of effective risk input to the driver, because their actions are determined by the balance between the benefits and the risks they bear.

[0169] Known examples of micro-level prompting of return actions in response to a return request include, for example, Japanese Patent Publication No. 2019-026247 and Japanese Patent Publication No. 2016-153960. Japanese Patent Publication No. 2019-026247 discloses a technology that blows cool air onto the driver to maintain the driver's alertness. Japanese Patent Publication No. 2016-153960 discloses a technology that uses an alarm to provide the driver with a gradual alerting notification. These known examples were not mechanisms that promoted macro-level behavioral changes in how the driver uses or utilizes the autonomous driving function.

[0170] For example, to gain the benefit of arriving quickly, one might consider the disadvantage of paying tolls when using highways, and therefore choose to use highways only when tolls are low, rather than when tolls are high. This is an example of a thoughtful balance. Also, if the NDRA (Non-Disruptive Driver Advisor) watches a sports broadcast while autonomous driving is in progress and continues watching without immediately interrupting after receiving a handover request, a penalty can be imposed on the driver as a disadvantage. In this case, the penalty could include a ban on viewing the broadcast for a certain period or for the same day, a forced stop of the vehicle in a designated parking space, or a ban on reusing autonomous driving for a certain period. The disadvantages for each user will vary.

[0171] <3-2-2-4. Regarding the driver's working memory and thinking during driving> In order to prevent the collapse of social infrastructure due to HCD control, regardless of the advantages or disadvantages for individual users, it ultimately comes down to human action, and a smooth and high-quality handover to manual operation is required from the time of the handover request to the point of return to normal operation, without reducing the basic cruising speed.

[0172] Here, we will explain the quality of the return (handover) action. Since there are individual differences in driver behavior, the system learns the target driver's normal handover action and estimates the time required for the driver to return based on the driver behavior learned from that action. The quality of the return action refers to the overall behavioral evaluation, which is indexed from behavioral quality evaluations such as whether the driver promptly performs the return action in response to the system's return request and completes the return action within the time limit, or whether, despite being notified that the return will be completed within the time limit, the driver does not take the return action that is learned and expected for a normal return, but instead takes actions that are not seen in normal learned return action, such as delaying the start of the return or taking slow actions.

[0173] In other words, from an HCD perspective, the control necessary for this user to recover needs to be considered from the following viewpoint.

[0174] It is unlikely that users can make decisions and respond based on an understanding of the invisible details of the ODD (Operational Design Decision) determined by an autonomous driving system, which is realized through highly sophisticated and meticulous design of equipment and vehicle performance. Therefore, autonomous driving systems need a mechanism to present benefits and risks to users as descriptive and tangible risks that can be intuitively grasped.

[0175] The human brain makes risk assessments based on limited information and finds and implements countermeasures to reduce risk within a limited time. In terms of human behavioral psychology, whether a person can take the necessary countermeasures at the right time depends on how they have learned the necessity and inevitability of those countermeasures from past experiences; it is dependent on experience and background and varies from person to person. On the other hand, as autonomous driving technology advances, it is expected that the system will be able to continuously handle a wider variety of situations automatically.

[0176] As autonomous driving systems become capable of handling a variety of situations, the need for the driver to intervene and switch back to manual driving becomes less and less, at least subjectively. Consequently, the driver's skepticism towards the system gradually diminishes, and they become less inclined to pay attention to the road ahead, check the sides and rear, or observe and confirm the vehicle following them, in preparation for sudden steering maneuvers when necessary.

[0177] Therefore, even if the system requests a sudden return to manual driving, once the driver leaves the loop of active steering, their thoughts shift to other matters of attention or interest. As a result, even if the system notifies the driver of the request to return to manual driving and the driver receives the notification, it takes a long time for the driver to start by understanding the situation that was interrupted, to gather the missing information, and to take action to actually avoid a handover accident. Furthermore, if the driver, as an NDRA, begins actions or behaviors that involve physically leaving the driver's seat, it will take even more time to move, including regaining consciousness.

[0178] When a person manually drives a car, they manage to deal with various events that occur along the way and the many events that arise as they happen, safely avoiding accidents and minimizing delays in progress. However, behind this seemingly effortless and safe driving, they are actually unconsciously checking a lot of information in advance, predicting the future effects of their actions based on that information, and gathering the necessary information beforehand to gain a certain level of confidence and prevent delays in making the necessary decisions to prevent accidents.

[0179] For example, even considering just the action of pressing the brake pedal, the information that the driver is aware of before pressing the brake pedal is: • How the vehicle's braking is affected by the degree to which the brake pedal is pressed. • Cargo on our own vehicle, • Information to determine whether the distance required for braking is longer due to the load of passengers. • Assess the risk of road slippage (wet, snow, etc.) and slow down before approaching the relevant section. • Prediction based on the behavior of the car in front. • Assessing the presence of following vehicles and the risk of sudden deceleration depending on the type of vehicle. • Fog conditions at the destination, • Whether there is a risk of delayed decision-making due to factors that obstruct visibility, such as backlighting. By combining various pieces of information, the final braking action is taken to prevent danger.

[0180] In other words, if the driver deviates significantly from the steering loop in autonomous driving, the relevant prior memory information (working memory) necessary for steering control will not be acquired, meaning the handover from autonomous driving to manual driving will begin without an understanding of the situation. Even if a sudden, mechanical request for a handover is issued to the driver, it is not guaranteed that the driver will be able to instantly acquire the prior decision-making information they have accumulated up to that point.

[0181] Therefore, if a driver is required to make a sudden decision or take action without adequately understanding the situation, they may fall into a state of panic, and in this case, they may be required to take action while in a state of panic. In other words, in order to enable HCD control that takes into account the human decision-making process, the system needs to balance the time required for the driver to recover their state of mind, posture, etc., with the ability to continue driving autonomously and to understand the road environment. It is necessary to have a mechanism that requests the driver to return to control while ensuring that there is always an option to secure a remaining grace period in which autonomous driving can continue.

[0182] In this situation, estimating a person's thought state is actually extremely difficult. For example, simply observing a driver from the outside might show that their gaze is directed forward and they appear to be paying attention, but in reality, they might be thinking about something else unrelated to driving. In such cases, the driver's thoughts (working memory) may be allocated to events completely unrelated to driving, and their working memory may lack the information necessary for making driving decisions.

[0183] The system needs to estimate the time (grace period) required for the driver to return to manual driving before the section where the autonomous driving function can be safely used ends. In the case of existing manual driving, for example, driver inattention to the road ahead can lead to overlooking hazards and potentially resulting in an accident. Therefore, to prevent situations where the driver inevitably neglects to pay attention to the road ahead, they should, in principle, not interrupt the continuous collection of information necessary for driving, even if they are temporarily engaged in tasks other than driving.

[0184] Therefore, in existing manual driving, the driver continuously and intermittently maintains visual attention, making it easy to quantify attention lapses such as drowsiness based on observational data by observing the decline in these behaviors. On the other hand, when using autonomous driving functions at level 1 or higher, the driver is not required to perform some steering tasks. As the autonomous driving function becomes more advanced beyond level 1, the need for driver intervention in driving gradually decreases. Consequently, the driver's information gathering and decision-making actions based on safe driving and steering decisions gradually decrease, and it becomes necessary to acquire missing additional information to understand the situation and take decision-making actions after receiving notification to return to full manual driving, which takes time.

[0185] There are two types of triggers that drive human behavior: triggers based on thought processes, and triggers that are driven by stimuli that lead to immediate action to avoid danger, even without a thought process. In this case, the latter type of behavior is a reflexive avoidance action, and because it is a risk avoidance action based on limited information performed in an unpredictable situation, it often occurs without appropriate and effective thought feedback.

[0186] In other words, when normal manual driving control is being continuously performed, the driver is constantly monitoring and checking the road ahead, so they usually do not make sudden steering maneuvers, suddenly press the brake pedal, or do any of these excessively. On the other hand, if the driver is distracted or inattentive and the vehicle is about to drift out of its lane, or if they are not aware of the braking of the vehicle in front and are only focused on the risk without being able to assess the situation, excessive sudden steering maneuvers or sudden braking may lead to unexpected accidents such as the vehicle overturning, rear-ending a following vehicle, or spinning out.

[0187] Factors contributing to this lack of appropriate behavioral control include insufficient information that enables the prediction of secondary damage to working memory necessary for suppressing the amount of behavior, and an overload of information requiring action. When information overload occurs, it can lead to panic in thinking, making it difficult to perform appropriate coping feedback actions such as controlling the degree of action, resulting in actions such as excessive steering to avoid the situation. Furthermore, human information gathering also has a function to filter out unnecessary information when continuously receiving information that is not relevant to behavioral judgment.

[0188] Therefore, even if the information is related to the handover, if it is not necessary for decision-making in normal use, and is continuously and mechanically provided to the driver without any changes, that information will occupy the driver's working memory. This becomes an obstacle to obtaining other potentially important information, and such unnecessary information is unconsciously recognized as noise by the brain, losing its importance and being filtered out. A good example of this filtering process that occurs before the brain processes information acquired from the outside world is the "cocktail party effect," known in psychology and other fields, where the conversation of certain people is easily heard even in noisy environments.

[0189] Based on this mechanism by which the human brain selects and utilizes information, a system is needed to provide the driver with a continuous stream of information about events that unfold and are updated as the vehicle travels, and to allocate this information to working memory in accordance with its importance, so that the driver can make decisions about relevant information that is important for taking over a new vehicle, based on the information provided by the system.

[0190] Here, the brain region known as working memory cannot be directly observed or visualized, and directly influencing working memory in an explicit form is extremely difficult with current technology. The priorities of a driver's thought processes vary greatly depending on the situation they find themselves in at any given time, and this working memory is not something that a system can directly store information about.

[0191] The system needs to take these human characteristics into account and provide the driver with unique information that serves as a priority factor for their actions, so that they can take over with ample time before the section where autonomous driving is available ends. Furthermore, it needs to teach the driver a measure of the impact of each piece of information that makes sense to them. In other words, simply providing information to the driver may result in that information being equivalent to noise. Therefore, if the provided information makes sense in predicting outcomes and poses a risk of disadvantage to the driver through learning, then that information is important and has a high priority in working memory.

[0192] <3-2-2-5. Regarding the "Contract" between the System and the Driver> Here, the initial exchange of notifications and confirmation actions between the system and the driver, where the driver receives information notified by the system and takes responsibility for addressing it, is considered a "contract" between the system and the driver. From this initial contract, the execution of the handover work based on that initial contract and the "degree of contract fulfillment" are analyzed based on observable information as the quality of the recovery transition (recovery action). The analyzed quality of the recovery action becomes "credit information" regarding the driver's execution of the contract. This credit information is used as a threshold for determining whether a higher-quality product than that contracted can be reused when switching to autonomous driving in the next or future travel segment. Furthermore, by providing feedback to the driver each time in the form of the events to be dealt with, their impact, and visual sensations, reinforcement learning progresses as an intuitive sensation for the driver.

[0193] In other words, this series of "contracts" is not simply a passive acceptance of information unilaterally notified to the driver by the system, but rather is carried out by responding to said notification. As a party to the contract, the driver perceives their response to the notification as an obligation to return to manual driving under that contract. Through a series of repetitive actions in accordance with this return obligation, the driver becomes able to control the HCD, which has been able to voluntarily participate in the use of the automated driving system.

[0194] The methods of presenting the individual pieces of information described in the embodiments of this disclosure describe only some representative means of achieving this HCD, and are not limited to these examples. In particular, how a driver stores the contractual terms of the "Contract" in their memory, how they recall their obligations over time, and how they perform them with high priority and without delay as needed varies from person to person and does not need to be limited.

[0195] Human-Centered Design (HCD) is not simply a control system that provides drivers with specific information. HCD is a more holistic design approach that incorporates considerations for fulfilling the functions associated with the proper use of a system, viewed from the perspective of human cognitive judgment and behavior. More specifically, HCD requires the construction of a system that incorporates mechanisms necessary for the development and growth of desirable cognitive behaviors.

[0196] Human behavioral psychology does not spontaneously and unconditionally develop into desirable behaviors. Rather, individuals develop their own unique behavioral psychology and cope with things by balancing the benefits, such as the desires they seek, with the disadvantages, such as punishments established as rules and norms by that society, and the risks they directly face regardless of those social norms. From this perspective, the impact on human behavioral psychology brought about by the driver assistance stage, one of the autonomous driving functions, is that the disadvantages or risks include a reduction in the sense of risk regarding direct driving errors and decreased attention due to fatigue, and an increase in unnecessary feelings of security.

[0197] However, the ultimate purpose of using driver assistance systems is to improve comfort and to enable accident prevention or avoidance, and in the worst-case scenario, reduce the risk of serious injury, even if a driver who normally uses the vehicle with sufficient attention experiences a momentary lapse in attention or overlooks important information.

[0198] Therefore, in the embodiments of this disclosure, in order to avoid excessive reliance by the driver on the driver assistance system, the system does not solely pursue comfort in all steering interventions it assists with. Instead, in cases of obvious over-reliance, the automated driving system performs self-avoidance measures, such as introducing uncomfortable controls for the driver or forcibly stopping the assistance function, but also imposes alternative risks that penalize the driver. This creates a mechanism where excessive reliance by the driver on the automated driving system can be prevented from directly leading to an accident, but does not result in complete risk avoidance, thereby fostering a virtuous cycle of driver-centered use.

[0199] Here, when the system incorporates autonomous driving functions beyond driver assistance, the concept of how the system is used changes significantly. In particular, the existence of a period in Level 4 autonomous driving where the driver is not required to be involved in driving control at all creates a situation where steering the vehicle is functionally completely risk-free for the driver.

[0200] MCD's view is that if all the conditions for Level 4 autonomous driving are met, then that would be fine. However, as already mentioned, excessive reliance on autonomous driving can lead to various negative consequences, such as traffic congestion. Therefore, from a societal perspective, the uncontrolled use of Level 4 autonomous driving is not a desirable situation. There are rules that must be followed when using the system with moderation, and the most representative rule is to use autonomous driving only within the scope of use where the conditions are met.

[0201] Based on the HCD concept, orderly use in accordance with social norms means that while the driver can participate in NDRA (Non-Disciplinary Routing) only in sections where the autonomous driving system permits Level 4 autonomous driving, there must be a mechanism in place that allows the driver to quickly learn and acquire the social norms of behavior when the need arises due to the predicted end of NDRA or changes in circumstances, and that reinforces these habits through daily use.

[0202] In other words, unless the driver's behavior changes to a swift and appropriate quality of return action after receiving a return request notification, they will not be able to enjoy the benefits of using Level 4 autonomous driving as a "reward (benefit)." Furthermore, the initial information of "risk" necessary for human judgment and thinking is information that is unconsciously provisionally stored in memory through the driver's approval of the "contract" presented to the driver by the system. Changes that may occur after the start of a section using Level 4 autonomous driving, and the interaction between the system and the driver via the HMI for reconfirmation, become "ancillary contracts" for reviewing conditions in response to changes that occur over time. In addition, each time the driver resumes driving a section using Level 4 autonomous driving, they are first shown information about the end point, and with agreement—that is, after being shown the necessity of returning, the approximate timing of returning, and the end request information—they begin the driving journey within the ODD section set and determined by the Level 4 autonomous driving system and the system.

[0203] In this disclosure, "contract" conceptually refers to any interaction between the system and the driver, not necessarily involving actual exchanges via physical documents. Through this interaction, the driver's memory is informed of the need for recovery, the risks of its impact, and the severity of the consequences of any violation, making it more memorable information depending on the importance of the action taken.

[0204] Drivers cannot unconditionally use Level 4 autonomous driving permitted by the system; rather, they can use it only if they have an obligation to switch from autonomous driving to manual driving as a condition. The quality of the driver's compliance with this condition will be used to assess their creditworthiness when using autonomous driving in the future. For example, if a driver's use of the autonomous driving function violates permitted usage, they will not receive any of the benefits of using autonomous driving, such as the NDRA (Non-Disruptive Driving Arrangement). Furthermore, in the case of a serious violation, penalties, restrictions on autonomous driving, and even restrictions on vehicle use may result in disadvantages. Due to these disadvantages, drivers develop cognitive sensitivity to advance warning information about risks along the way, and as reinforcement learning progresses, their sensitivity to advance warning information that contributes to maximizing the benefits without losing them will increase.

[0205] In other words, the automated driving control by HCD according to the embodiment of this disclosure is completely different from the conventional MCD, which simply incorporates a mechanism into the system to issue a warning when a handover point is imminent and forcibly restore the driver's awareness, or from the concept that the system periodically forces a return request in order to prevent the driver's awareness from leaving the driving steering loop.

[0206] This conventional MCD control concept is likely to be intuitively annoying to the driver. Consequently, some drivers may weaken the alarm's effectiveness to alleviate this annoyance, potentially leading to a situation where they become desensitized to the alarm and habitually immerse themselves in NDRA without paying much attention to it. This can result in drivers ignoring the system's alarms, or, if the alarm is, for example, a monotonous, repetitive buzzer sound, the driver's auditory filtering effect may cause it to become unimportant.

[0207] As described in detail above, when the system requests the driver to return to driving, it presents risk information that may affect the driver's near future as variable information in a multifaceted and variable manner, i.e., not uniform, and the driver actively reconfirms the "additional conditions" in response to this presentation. As a result, the risk information is distributed and stimulated in different parts of the driver's working memory, such as the auditory language center, visual language center, and visual cortex, so that the memory stimulation for the driver's return to driving is not monotonous.

[0208] As a result, even in cases of mental wandering or other distractions that deviate from the driving and steering task, forgetting the obligation to return to the task is suppressed. When starting autonomous driving in this Level 4 autonomous driving section, visual information about factors requiring a return to the task, as stipulated in the "contract," is presented. Furthermore, during the autonomous driving process in the Level 4 autonomous driving section, new updated information is presented in accordance with the driver's forgetfulness characteristics and updated information. This information presentation allows the driver to reassess the risks, thereby reactivating important memories in the driver's working memory.

[0209] Another important point based on HCD is that the contribution of stimuli such as notifications and warnings from the system to the driver, which are necessary for making decisions about driving actions, does not simply act as the intensity of the notification (such as the volume of the sound), but rather acts in accordance with the individual differences in the driver's sensitivity to stimuli that lead to risk.

[0210] In other words, in HCD, the strength of physical stimuli is not the important factor; rather, the brain has a mechanism that prioritizes processing information deemed important for near-future decisions, increasing sensitivity and postponing less important information. In this embodiment, the system uses artificial intelligence or similar methods to learn how a person has grown and developed a habit of presenting specific sets of information that are inherent to their individual characteristics, and then designs an HMI that promotes early decision-making using influential information.

[0211] For example, instead of limiting all information to visual information and narrowing the method of information presentation, information is presented to the driver by comprehensively stimulating them using multiple different types of information. Specifically, the driver could be stimulated by being exposed to a specific auditory sound, followed by a visual notification. The system then awards the driver credit points as a good driver if they respond quickly and accurately to this stimulus. Furthermore, instead of simply adding these credit points to a mechanical storage medium (memory, hard disk drive, etc.), the system also provides the driver with intuitive visual feedback via the HMI on the spot. This intuitively links the driver's obligation to perform an accurate and early return to driving at the scheduled time and under the circumstances based on their "contract," leading to psychological reinforcement learning in the driver, which optimizes their own responses.

[0212] In neurons, which are the optic nerves that control the triggers for judgment, numerous factors stimulate the synapses at a microscopic level. Memories that temporarily hold information requiring attention correspond to this alert state of waiting to fire. Sensitivity is increased to related information that carries risks in the near future, and when the necessary stimulus is received, the information necessary for judgment is stored in the working memory, which is a state of readiness to take action based on important items in memory. This is what is meant by incorporating related information into the working memory for behavioral judgment. Furthermore, if there are many and diverse stimulus paths, even if thoughts temporarily wander due to mind wandering, it acts as an anchor to maintain high priority, and the necessity is strongly maintained by the visual and auditory information of risk factors presented simultaneously and in parallel.

[0213] In other words, the act of the driver confirming the "contract" and "ancillary contracts" presented by the system constitutes a reconfirmation of information by the driver. Microscopically, this act of reconfirming information can be seen as activating the synaptic potential to a state just before the firing of a judgment. Because the judgment optic nerve is placed in this ready standby state, the driver's perceptual sensitivity to minor information increases as they approach the end of a section permitted as an ODD, such as autonomous driving level 4. As a result, even if the information the driver receives is incomplete, they are placed in a situation where their sense of necessity is heightened, and if their memory is insufficient, they will voluntarily try to supplement the information to mitigate the increase in risk, and as a result, aim to complete the "ancillary obligations" based on the initial "contract". If the driver feels anxious, this will be reflected in actions such as visually reconfirming the status screen regarding the contract.

[0214] <3-2-2-6. Operation of Autonomous Driving Level 4> Next, the concept of operation of Level 4 autonomous driving according to the embodiment of this disclosure will be described.

[0215] The challenge for the driver is determining what criteria they should use to promptly and appropriately abandon the continued use of Level 4 autonomous driving as the vehicle approaches the end of the section where Level 4 autonomous driving is permitted, and to take swift and appropriate action (return action). For the driver to interrupt the NDRA, which can be considered a benefit of using Level 4 autonomous driving, and transition to manual driving, relevant information for the transition to manual driving must be stored in the working memory that governs decision-making.

[0216] The trigger for storing relevant information in working memory is the "contract" made between the system and the driver when Level 4 autonomous driving is first used. The driver initially recognizes their responsibilities and obligations at the time of this "contract." However, if the point at which the actual driving section ends is still some time away, the driver needs to re-clarify their memory in order to fulfill their obligation to return to manual driving before reaching the point at which they can take action to return to manual driving. The presence or absence of triggering information and the importance of the risk greatly influence the success or failure of this process.

[0217] From an ergonomic perspective, stimuli linked to some reason lead to far more accurate decisions than simple recovery actions without any other underlying cause. Therefore, presenting different factors for transitioning to a new state, as described below, is useful for successful transitions.

[0218] Assuming the use of autonomous driving up to Level 4, if a driver initiates Level 4 autonomous driving on a well-maintained and managed road section, they are generally not required to immediately revert to manual driving within that section. In low-speed autonomous driving, such as Low Speed ​​Automated Driving, time can be secured for corrective actions such as slowing down or stopping the vehicle if the limits of autonomous driving are exceeded.

[0219] On the other hand, on public roads with heavy traffic of ordinary passenger cars, it is necessary to automatically handle the situation without disrupting the flow of traffic in the relevant section of road, while maintaining that flow. At this point, if automatic handling is deemed difficult, the system must choose to either switch to manual driving, ensure a smooth handover to the driver while maintaining a safe cruising speed for automatic driving, or, if the likelihood of this is low, to take refuge on the shoulder, service area, refuge parking pool space, or a public road where stopping or driving at a low speed is possible, while refuge options remain.

[0220] When a regular vehicle is operating at Level 4 autonomous driving on public roads, highways, or main roads, the factors that necessitate interrupting autonomous driving vary depending on the vehicle's characteristics, the road conditions, the environment, and the driver's ability to react. As previously mentioned, even when it becomes difficult to continue autonomous driving, the system is not guaranteed to switch to manual driving with 100% certainty. Furthermore, if an emergency stop or deceleration is performed on the spot, even if the vehicle itself is not significantly affected, there is a high probability of impact on following vehicles, and thus a significant social impact.

[0221] What is needed is a response control mechanism that, even if the driver is unable to take action to return to manual driving, allows the system to determine that a successful handover has been achieved while the vehicle is still in a remaining section of the road where there are still options to take evasive action that minimize disruption to following vehicles.

[0222] The following describes examples of situations that may make it difficult to continue using Level 4 autonomous driving.

[0223] Depending on individual vehicles, personal risk perception, capabilities, etc., driving at Level 4 of autonomous driving needs to be compatible with the actual use of roads, which is an orderly social infrastructure. Here, it is extremely difficult at present for the system to make judgments on appropriate use in an orderly road environment in the same way as human thinking ability. Or, leaving such functions to the system is an issue related to the very essence of what makes a person a person, and morally speaking, it is conceivable that ultimate autonomous driving will not be carried out. This is also a negation of a society where people are used by machines.

[0224] As an example, in a situation where passage is only possible through mutual concession on a narrow street, a person interrupts or intervenes in the steering of the autonomous driving once, selects the priority, resolves the intertwined situation, and then proceeds with the passage. As a simple example of this, when passing a narrow bridge with a single-lane width, by communicating with the oncoming vehicle in advance and alternately conceding passage, it is possible to prevent the progress of either vehicle from being completely obstructed, and each vehicle passes through the section.

[0225] Therefore, in practice, a control mechanism for maintaining social order can be artificially incorporated before that section. At this time, for example, on arterial roads, even when driving at Level 4 of autonomous driving, since it has no impact on other vehicles, it is preferable to proceed to selection and judgment processing such as abandoning the planned journey driving in any situation, handing over to manual driving by the driver, remote driving support, and prior avoidance without impact. In this case, appropriate judgment is required.

[0226] Examples of factors for giving up the continuous use of autonomous driving at Level 4 are shown below as each item. These items can be a factor alone or in combination of multiple items.

[0227] The first factor is a factor for giving up continuous driving at Level 4 of autonomous driving due to the availability or unavailability of road environment information that can be obtained in advance about the destination of the relevant road during use. Examples of this include the following items.

[0228] (20-1) Loss of fresh, updated section information from LDM due to malfunctions of section-operating vehicles that regularly collect information. (20-2) Information not acquired due to temporary congestion of the communication bandwidth used for the continuous updating of high-freshness LDM. (20-3) Restrictions on use of local map information due to expiration of subscription agreement (20-4) Following a lead vehicle with assistance from a paired lead vehicle, continuous data acquisition is impossible due to lead vehicle malfunction, etc. (20-5) Due to a decrease in the density of vehicles traveling in a section depending on the time of day, there is a lack of shadow probing data for road information from general vehicles. (20-6) Malfunction of the vehicle's communication equipment, malfunction of infrastructure communication (20-7) Lack of update information necessary for continuous autonomous driving due to cyberattacks on communications, or alteration due to fabrication.

[0229] A second factor is that the vehicle may abandon continuing to operate at Level 4 autonomous driving depending on information notified from the pre-update road environment information for the destination on the road currently being used. Examples of factors in this case include the following:

[0230] (21-1) Overcapacity of remote flight support controllers (21-2) Shortage of remote driver assistance operators (21-3) Intrusion of people or large animals onto the vehicle-only road, situations where animals escape from the cargo area of ​​a vehicle ahead and wander the road, and receiving emergency information about scattered debris from the lead vehicle for the section. (21-4) Receiving information that is difficult to predict, such as earthquakes, landslides, and tsunamis. (21-5) Receiving warning information for following vehicles obtained from voluntary danger reports from lead vehicles in a section. (21-6) Partial and unexpected freezing of road surfaces such as wet bridges and shaded areas in mountainous regions. (21-7) Traffic restrictions due to unexpected road construction or accident response. (21-8) Traffic control through human communication to facilitate passage through a designated section (traffic control for accident response, etc.) (21-9) When the vehicle continues to proceed in autonomous driving mode, there will be no areas where the vehicle can take refuge, such as on narrow road sections, single-lane bridges with alternating traffic, or tunnels, and the vehicle may enter restricted areas depending on the driver's condition. (21-10) Crossing the railway tracks in a level crossing section

[0231] A third factor is that the vehicle may have to abandon continued operation at Level 4 autonomous driving due to performance limitations of the sensing equipment installed in the vehicle or performance fluctuations over time. Examples of this include the following: In this case, continued operation may be abandoned depending on the start time of autonomous driving at Level 4 and changes in conditions during driving.

[0232] (22-1) Degradation of detection performance of millimeter-wave radar, LiDAR, cameras, etc. due to snow removal materials and dirt kicked up by the vehicle in front while driving. (22-2) Temporary degradation or limitation of sensing camera performance due to collisions with the windshield by insects, flying objects, etc., while driving at high speeds. (22-3) Noise generation and partial degradation of detection performance due to temperature rise during use (22-4) Localized window damage caused by pebbles kicked up by the car in front, or by flying objects from adjacent lanes, etc. (22-5) Headlight failures and other malfunctions in vehicles driving at night, and a decrease in the detection limit of the camera's field of view. (22-6) Fogging of the windshield due to misuse of the indoor air conditioning system, etc., and temporary deterioration of the detection performance of indoor sensing cameras. (22-7) Request for recovery of unknown cause based on self-diagnosis results by the vehicle system

[0233] A fourth factor is that changes in the condition of the cargo affecting the vehicle's movement during continuous autonomous driving, or other changes in vehicle dynamics, may cause the vehicle to abandon its continued operation at Level 4 autonomous driving. Examples of such cases include the following:

[0234] (23-1) Cargo shifting during normal driving (23-2) Tire deflation, tire burst (23-3) Riding on scattered objects, driving performance fluctuations due to abnormal roads, and abnormal noises inside the vehicle (23-4) Collapse of the load due to sudden braking for collision prevention, significant movement of passengers inside the vehicle, and resulting imbalance of the weight distribution (23-5) Detection of brake abnormalities during vehicle self-diagnosis while driving (23-6) Engine overheating, malfunction of control equipment

[0235] As a fifth factor, there may be cases where continuous driving at automated driving level 4 is abandoned due to driver abnormalities. Examples in this case are listed in the following sections.

[0236] (24-1) Unforeseen drowsiness, leaving the driver's seat of the driver, and inability to predict the return at the necessary time due to inability to detect the state (24-2) Sudden attacks of the driver (asthma, spasms or numbness of the feet, etc., allergic reactions due to sudden entry of pollen, etc. into the vehicle, heart attacks, sudden headaches, strokes, cerebral infarctions,... etc.) (24-3) Detection of abnormal behavior due to drug dependence (24-4) Interference with the driver state monitoring being carried out by the system (24-5) Ignoring or neglecting by the driver of the necessary response processing required by the system for the driver in the continuous use of automated driving (24-6) The driver's rest situation, activity, rest, and habit history information from the previous day, etc. <​​​​​Of the above, if the driver fails to take corrective action based on risk assessment for at least the first to fourth factors, according to the different conditions, the driver will receive prior feedback through an HMI capable of presenting sensory expressions, such as visual stimuli, regarding the degree of impact and the application of penalties for violations. As a result of this prior feedback from the HMI, the driver will temporarily retain the visual sensory stimulus, for example, because the visual stimulus is taken into working memory at least once. By providing the driver with stimuli related to this visual sensory stimulus, it is possible to refresh the memory and maintain the memory of the need for corrective action.

[0239] On the other hand, in the fifth case, when the driver themselves becomes incapacitated, it becomes extremely difficult to prompt early recovery actions using HCD. However, even in this case, the system can present the driver with a function to abandon autonomous driving early, and based on information such as possible escape points provided by this function, it can be used to request the driver to voluntarily abandon the driving or to request rescue. This approach allows for more proactive control than leaving the driver unattended without providing any information, allowing the vehicle to proceed to the handover limit point, and only activating MRM after reaching that point.

[0240] The examples given in each of the above items are representative of the treatment required on important arterial roads and other social infrastructure where problems may arise if a vehicle stops within the road lane. The above items do not apply to roads with extremely low traffic volume, or roads that are not arterial but have a wide width, where it is very unlikely that a vehicle will obstruct traffic even if it makes a sudden stop or comes to a complete stop using MRM.

[0241] In other words, if the driver continues driving at autonomous driving level 4, and the section of the road allows for emergency stopping or evacuation via MRM without disrupting social activities, the system can continue the planned driving route as scheduled, without considering the driver's ability to recover or regardless of the driver's condition. That is, even if the system encounters a situation that is difficult to handle at autonomous driving level 4, and the driver is unable to recover properly, making an emergency stop or evacuation of the vehicle at a designated location will not disrupt the flow of traffic on the road, which is a social infrastructure.

[0242] Thus, the conditions under which Level 4 autonomous driving is appropriate are not uniformly defined or fixed environments, but rather change proactively depending on various factors such as the vehicle's ability to respond to driver requests to resume operation, road conditions, usage environment, and vehicle condition.

[0243] <3-2-2-7. Effects of adopting HCD> Next, we will explain the effects of adopting HCD in place of existing MCD, as described above.

[0244] Firstly, the essence of the control of the HCD related to this disclosure is that the section in which the driver can use autonomous driving is variable, and the determination of the usable section depends not only on observable evaluation values ​​such as the driver's alertness at the observed time, but also on the driver's acquired credit information. Furthermore, in the HCD related to this disclosure, during the intermediate process of determining the usable section, the information detected by the system and its results are presented to the driver as near-future risk information, at least using visual representations.

[0245] Unlike simple return request notifications uniquely presented to the system, HCD presents information as a risk of the driver's own chosen action, prompting a thoughtful decision-making process that considers the balance between benefits and drawbacks. This allows information regarding importance in the driver's working memory to be incorporated and kept more up-to-date according to the importance of the handover. Furthermore, the behavioral evaluation of this thoughtful decision-making process is reflected in future usage conditions, and even current usage permission is determined based on past evaluation results to determine whether the benefits of using autonomous driving can be obtained. Therefore, with HCD, drivers can acquire a sense of responsibility for use and whether or not they can enjoy the benefits, which differs from the machine-based instructions given by MCD, through repeated use.

[0246] Furthermore, through repeated use, drivers can develop appropriate ways of interacting with the system, and if their behavior disrupts social activities, the system can impose penalties and discourage use. This helps drivers avoid detrimental behaviors that disrupt social activities, thereby curbing forms of use that are not socially acceptable.

[0247] Secondly, since autonomous driving can significantly reduce the burden of human intervention in driving and steering, it is expected that the occurrence of accidents in society will be greatly reduced, given that approximately 94% of accidents today are caused by human error, as vehicles will operate in autonomous driving mode instead of humans.

[0248] However, the introduction of autonomous driving as widely considered today is based on the premise that the driver will appropriately revert to manual driving in response to the system's requests and that the driver will be able to respond promptly in response to those requests. However, as the performance of autonomous driving systems improves, this premise may not hold true if it leads to excessive reliance on the system by drivers.

[0249] In this disclosure, the availability of autonomous driving functions is provided primarily on the basis of whether the driver can appropriately utilize them. Furthermore, the system incorporates an HMI (Human-Machine Interface) that encourages appropriate use. This prevents excessive reliance on the system by the driver and realizes control technology that allows the driver to subjectively take appropriate recovery actions.

[0250] In short, this disclosure relates to vehicle control that changes the method of controlling the use of autonomous driving of a vehicle from a conventional MCD (Multi-Controller Device) that unilaterally issues instructions from a device to a HCD (Human-Centered Control) that performs usage control according to human behavioral characteristics, and introduces an HMI (Human-Centered Interface) to realize this. In particular, when a driver uses the system's autonomous driving function, a "contract" is concluded between the driver and the system, which is a "confirmation act" that the driver performs without fail upon request at the completion of a "ritualistic" autonomous driving, and the validity of this contract is reconfirmed as appropriate during the course of autonomous driving use.

[0251] Such HCD-based systems cannot be realized simply by incorporating a single function into the system; they require reinforcement learning through repeated, complex use necessary for diverse drivers to develop usage habits. The embodiment is an HMI that interacts with the driver to advance this reinforcement learning and maintains appropriate early return to normal use in the long term. Note that the combination of execution means for providing feedback to the driver is not limited to the examples described in this specification.

[0252] Conventional autonomous driving systems had a predetermined permissible level of autonomous driving determined mechanically by the system based on the road environment conditions the equipment could handle, with the assumption that the driver would uniquely switch between modes based on their own judgment as needed. In contrast, this disclosure provides support for the development of the habit of drivers repeatedly initiating the recovery procedure promptly and appropriately in response to system requests, and provides the necessary HMI for this purpose. As a result, even if autonomous driving becomes widespread in society, the system will be able to broadly suppress the occurrence of emergency deceleration or stopping on the road due to delays in the driver's recovery process, and control actions that would lead to significant disruption of traffic if the vehicle stops, thereby preventing disruption to social activities.

[0253] Thirdly, the use of automated driving, which frequently involves handing over to manual driving, requires an understanding of the characteristics of the driver's working memory in order to safely continue driving operations.

[0254] Even if necessary action items are perceived and recognized and the information is initially taken into working memory, if a person's thoughts at the time or an imbalance in the autonomic nervous system cause their mind to wander to other events, the importance of even important matters may fade from memory, potentially leading to delayed action and critical consequences.

[0255] The amount of information that can be allocated to human thought is finite, and it is impossible to allocate complete attention to everything at the same time. Therefore, when information is learned through multiple different systems, and information is input into these different systems through different means such as visual and linguistic means, and the results of each can be depicted, it becomes easier to bring thoughts back to the necessary actions even when mind wandering occurs. In other words, HMIs that present information that links to the prediction of the consequences of a certain impact, rather than simply presenting monotonous information such as symbols, in a concrete sense of risk, are effective in clarifying the memory of the request to return.

[0256] When HCD is used for control, even information that has entered the working memory during such tasks may be forgotten over time due to a sudden shift in thought to other matters. Therefore, it is necessary to provide feedback to the driver to help them recall information, according to their current and unique forgetfulness. Accordingly, this disclosure evaluates the driver's inherent tendency to forget important matters and how well they can retain driving-related precautions at the time of use, and presents memory refresh information tailored to that driver.

[0257] Furthermore, in one embodiment of this disclosure, when remote support is used, it becomes possible to pre-confirm and predict locations where support will be unavailable. By providing drivers with information such as sufficient shoulder width and service areas (SAs) via the HMI, it becomes easier to choose a waiting location that will not obstruct other traffic if support is unavailable. This enables a system that can function even with a limited number of remote support operators, thus enabling the provision of practical and efficient remote support. It also enables the efficient operation of the infrastructure necessary for remote monitoring.

[0258] Fourth, regarding secondary tasks other than driving performed by the driver, i.e., NDRA, where the driver primarily uses visual information from an electronic terminal, attention can be improved through the following measures. Specifically, in terminal devices with a monitor screen, short-term information related to autonomous driving may be displayed in the display image area that is originally intended for NDRA, and visual information related to the need for a handover may also be displayed.

[0259] For example, visual information may be displayed for an extremely short period, aiming for a so-called subliminal effect where the viewer does not consciously notice it. In addition to information presentation that remains completely outside the viewer's awareness, information may also be presented for a longer duration than that aimed at a subliminal effect, and which the driver can clearly perceive.

[0260] In the case of information that does not necessarily go through verbal understanding, such as subliminal effects, it is more effective when a visual and intuitive sense of risk is conveyed to the driver as a consequence of ignoring a request for handover. For example, when a violation occurs, a visual depiction of being watched by a police motorcycle on the side of the road is more effective than written information about the penalty regulations; a depiction of being ordered to stop and be pursued for a traffic violation is more effective than static images of a police box; a depiction of a situation in which the driver is asked to confirm the violation; and a depiction of the risks incurred if the driver is unable to continue performing MRM (Monitoring of Riders).

[0261] Humans typically perceive visual information consciously and then interpret that information linguistically. On the other hand, academic research suggests that humans possess information transmission mechanisms that can influence the brain without any linguistic interpretation, and even without any conscious awareness remaining.

[0262] These short-term stimuli, such as subliminal stimuli, are expected to refresh and reactivate important information that has faded from working memory. Unlike the system's action to awaken people from fatigue or drowsiness, this effect revives memory information in the working memory necessary for conscious decision-making.

[0263] When people try to recall important information, they may understand the need to take action somewhere deep down, but they often forget and fail to remember it at the necessary time, only to recall it later.

[0264] This is because important information requiring attention is not prioritized and stored as important in working memory, and therefore cannot be retrieved from memory. For memory information to be effective in judgment, the amount of stimulation to that memory needs to be increased. Subliminal effects, like blindsight, influence behavioral judgment even without consciously perceiving it as visual information. However, since the main objective is not to prevent the display from appearing, subliminal effects are the ultimate example of short-term HMI, and it is possible to display information for a longer period than what would affect consciousness, or even to make it a stronger risk depiction by continuing to display it until the driver cancels it.

[0265] In other words, when using autonomous driving, the system will only allow continuous use of autonomous driving if a "contract" has been established between the system and the driver that stipulates that the system will respond to requests for manual control when circumstances change. In accordance with this "contract," the driver utilizes the autonomous driving function and takes advantage of the NDRA benefits while autonomous driving is in progress. To ensure that the driver does not forget to fulfill the "return obligation" associated with the "contract," some kind of reminder is effective. If an electronic device is being used, displaying this reminder on the screen of the electronic device is effective. In this case, it has the effect of making the driver "aware" of the "return obligation" associated with the "contract" without making them too bothered to view the screen related to NDRA when using the benefits.

[0266] <3-2-3. Specific Examples of HCDs According to the Embodiments> Next, the HCD according to the embodiment will be described in more detail. Unless otherwise specified below, the automated driving will be assumed to be automated driving according to the SAE Level 4.

[0267] <3-2-3-1. Example of Automated Driving Operation Applying the HCD According to the Embodiment> Using the flowcharts in Figures 7A to 7C, we will explain in more detail an example of the operation of autonomous driving with the HCD according to the embodiment. In Figures 7A to 7C, the symbols "A" to "F" indicate that the process will transition to the corresponding symbol in the flowchart of another figure within Figures 7A to 7C. Also, in the flowcharts in Figures 7A to 7C, the blocks that generally represent "documents" indicate that information is provided to the driver through visual or other stimuli.

[0268] Figure 7A is an example flowchart showing the flow from setting a travel itinerary to transitioning to autonomous driving mode according to the embodiment. The travel itinerary here refers to the vehicle's travel plan and includes information indicating the starting point and destination of the journey, as well as information indicating the travel route. Furthermore, "starting the travel itinerary" means beginning to drive according to the travel itinerary.

[0269] In step S100, the vehicle user (driver) sets the itinerary, including the destination. The set itinerary is input to the automatic driving control unit 10112 (see Figure 1). Based on the input itinerary, the automatic driving control unit 10112 acquires various information necessary to drive according to the itinerary, such as LDM. For example, in step S101, the automatic driving control unit 10112 acquires information such as LDM, the driver's characteristics for returning to manual driving, the weather in the areas included in the itinerary, and the cargo loaded in the vehicle. Of these, the driver's characteristics for returning to manual driving can be, for example, based on the evaluation of the driver's past actions to return to manual driving.

[0270] In the next step, S102, the automated driving control unit 10112 presents the driver with an overview of the entire journey. Specific examples will be described later, but the automated driving control unit 10112 generates display information that visualizes, for example, map information showing the entire travel route based on LDM, and information indicating sections within that route that can be driven at automated driving level 4. The automated driving control unit 10112 supplies the generated display information to the output unit 10106 via the output control unit 10105, causing, for example, a display device connected to the output unit 10106 to display an image according to the display information.

[0271] This display is a navigation display that shows the itinerary settings recommended by the system, i.e., the automatic driving control unit 10112. It should be noted that the overhead view referred to here does not necessarily have to be a three-dimensional overhead view based on a scale derived from physical distance; any means by which the driver can recognize the intervening point is acceptable, including a time-converted display, a stereoscopic display, or any other display format.

[0272] In the next step, S103, the automatic driving control unit 10112 asks the driver whether they agree to the itinerary recommended by the navigation display presented in step S102. For example, the automatic driving control unit 10112 determines whether they agree or not based on the driver's operation on the input unit 10101. However, the automatic driving control unit 10112 may also detect the driver's movements using a camera for in-vehicle imaging and determine whether they agree or not based on the detected movements, or it may make the determination based on the driver's voice.

[0273] If the automated driving control unit 10112 determines in step S103 that the driver does not agree to the recommended settings (step S103, "No"), it proceeds to step S104. In step S104, the automated driving control unit 10112 adds an alternative recommended route and presents the driver with the option to select this alternative route. The automated driving control unit 10112 then returns to step S102 and presents the driver with an overview of the entire journey using the alternative route.

[0274] On the other hand, if the automatic driving control unit 10112 determines in step S103 that the driver agrees to the recommended settings (step S103, "Yes"), it proceeds to step S105. At this time, the driver accepts and agrees to the itinerary proposed by the system, and the driver grasps the concept of the entire itinerary, and this fact is stored in the driver's working memory as memory information #1 (WM10).

[0275] In step S105, the driver starts driving the vehicle, and the journey begins. The automatic driving control unit 10112 updates the overview of the journey in accordance with the vehicle's movement at the start of the journey, and presents the updated overview to the driver (step S106). At this time, the automatic driving control unit 10112 controls the automatic driving for each section of the journey and calculates the automatic driving mode for each ODD corresponding to each section in a time series.

[0276] At this time, the driver can understand the current state of the journey by checking the updated overview presented by the automatic driving control unit 10112, and can understand the obligation to return to manual driving for the most recent selection. The information grasped is stored in the driver's working memory as memory information #2 (WM11).

[0277] In the next step, S107, the automatic driving control unit 10112 determines whether an ODD section in which automatic driving is permitted is approaching. If the automatic driving control unit 10112 determines that the section is not approaching (step S107, "No"), it moves the process to step S108, where it performs continuous monitoring of changes in the situation, updates various risk information based on the monitoring results, and returns the process to step S106.

[0278] On the other hand, if the automatic driving control unit 10112 determines that the section in question is approaching (step S107, "Yes"), it proceeds to step S109. In step S109, the automatic driving control unit 10112 presents the driver with a contract regarding the handling of the ODD section and determines whether the driver has agreed to this contract. This contract includes, for example, conditions for allowing automatic driving in the ODD section. The automatic driving control unit 10112 makes this determination based, for example, on whether the driver has performed any operation or action indicating agreement to the presented contract. If the automatic driving control unit 10112 determines that agreement has not been obtained (step S109, "No"), it returns to step S106.

[0279] If the automatic driving control unit 10112 determines in step S109 that an agreement to the contract has been reached (step S109, "Yes"), it permits the use of automatic driving within the ODD and proceeds to step S110. In step S110, when the automatic driving control unit 10112 enters an ODD section where automatic driving is permitted, it transitions the driving mode from manual driving mode to automatic driving mode.

[0280] When the driving mode switches to automated driving mode in step S110, the driver is obligated to return to manual driving in response to the selection made in step S109. By understanding the terms of the agreement (contract) with the system, the driver has agreed with the system regarding risk management at the end of the ODD section. Failure to fulfill the return obligation will result in a penalty for the driver. Information #3-1, #3-2, ... indicating each condition included in the agreed contract is stored in the driver's working memory as memory information #3 (WM12). This memory information #3 is also stored as an ancillary contract when using automated driving, providing information on how to handle any unexpected incidents that may occur during the journey (WM13).

[0281] When the driving mode is selected and transitioned to the automatic driving mode in step S110, the process proceeds to the flowchart shown in Figure 7B, according to the symbol "A".

[0282] Figure 7B is a flowchart showing an example of the processing flow in the automated driving mode according to the embodiment. Processing moves from step S110 in Figure 7A to step S120 in Figure 7B, where the automated driving control unit 10112 performs continuous monitoring of changes in conditions, updates various risk information based on the monitoring results, and moves the processing to step S121.

[0283] In step S121, the automatic driving control unit 10112 determines, based on the results of the situation monitoring in step S120, whether or not an event requiring driver intervention has occurred. If the automatic driving control unit 10112 determines in step S121 that the event has not occurred (step S121, "No"), it proceeds to step S122.

[0284] In step S122, the automatic driving control unit 10112 determines whether the end point of the ODD section in which automatic driving is permitted is approaching. If the automatic driving control unit 10112 determines that the end point of the section is not approaching (step S122, "No"), it returns to step S120. On the other hand, if the automatic driving control unit 10112 determines that the end point of the section is approaching (step S122, "Yes"), it moves the process to step S123.

[0285] The loop processing in steps S120 to S122 represents processing within an ODD section where autonomous driving at autonomous driving level 4 can be stably used.

[0286] In step S123, the automatic driving control unit 10112 notifies the driver that the handover point from automatic driving to manual driving is approaching. In the next step S124, the automatic driving control unit 10112 monitors the driver's actions related to the transition from automatic driving mode to manual driving mode, that is, the quality of the driver's handover action from automatic driving to manual driving, and adds or subtracts points from the driver's evaluation score according to the quality. The monitoring of the quality of the handover action and the calculation of the evaluation points added or subtracted for that quality will be described later.

[0287] In the next step, S125, the automatic driving control unit 10112 determines whether the entire journey set in step S100 in Figure 7A has been completed. If the automatic driving control unit 10112 determines that it has been completed (step S125, "Yes"), it terminates the series of processes shown in the flowcharts in Figures 7A to 7C. On the other hand, if the automatic driving control unit 10112 determines in step S125 that the entire journey has not been completed (step S125, "No"), it proceeds to step S106 in the flowchart of Figure 7A, according to the symbol "B".

[0288] If the automatic driving control unit 10112 determines in step S121 that an event requiring driver intervention has occurred (step S121, "Yes"), it proceeds to step S130 in the flowchart of Figure 7C, according to the symbol "D" in the figure.

[0289] Figure 7C is a flowchart illustrating an example of how to respond to an event occurring during autonomous driving at autonomous driving level 4 according to this embodiment. In step S130, the autonomous driving control unit 10112 notifies the driver of the occurrence of a new event. In the next step S131, the autonomous driving control unit 10112 determines the urgency of the new event. The autonomous driving control unit 10112 determines the urgency, for example, based on the distance between the location where the vehicle is currently traveling and the location where the new event occurred. This is essentially equivalent to determining the urgency based on the amount of time available for the vehicle to reach the location where the new event occurred.

[0290] The automatic driving control unit 10112 determines that the urgency of the new event is high (step S131, "high") if the distance to the location of the new event is below a predetermined level and the time margin is small, and proceeds to step S160. In step S160, the system starts MRM, and actions such as decelerating the vehicle and moving to a safe place such as the shoulder of the road are forcibly executed. When MRM starts in step S160, the series of processes shown in the flowcharts in Figures 7A to 7C are temporarily terminated.

[0291] Furthermore, if a situation is determined to be highly urgent in step S131 and proceeds to step S160, the driver's evaluation will be reduced, and this will be considered a point deduction item (1) as described later.

[0292] In step S131 described above, the automatic driving control unit 10112 determines that the urgency is moderate (step S131, "moderate") if the distance to the new event location is within a predetermined range (longer than the distance required for high urgency, and shorter than the distance required for low urgency) and there is sufficient time available, and proceeds to step S132.

[0293] In step S132, the automated driving control unit 10112 ensures sufficient time for deceleration beforehand, checks its own vehicle and surrounding conditions (such as the presence of following vehicles), and predicts the impact on the surroundings if the vehicle's speed decreases. The automated driving control unit 10112 also checks the driver's condition and observes whether the driver is capable of returning to manual driving in an emergency. Based on these observations, the automated driving control unit 10112 predicts the delay in the driver's action to return from automated driving to manual driving.

[0294] In the next step, S133, the automatic driving control unit 10112 determines, based on the prediction result in step S132, whether or not it is possible to extend the grace period before returning from automatic driving to manual driving. If the automatic driving control unit 10112 determines that it is possible to extend the grace period (step S133, "Yes"), it proceeds to step S140. On the other hand, if the automatic driving control unit 10112 determines that it is not possible to extend the grace period (step S133, "No"), it proceeds to step S134.

[0295] While I will omit a detailed explanation of the control system, if a new event requiring manual driving occurs during the journey due to a change in circumstances, the system might force a response within the limited time available by suddenly decelerating, which increases the risk of secondary damage such as rear-end collisions or traffic congestion, and is not necessarily safe.

[0296] Therefore, a decision-making process for countermeasures is necessary, and the driving plan is reviewed to determine whether slowing down in the relevant road section would have any impact on the road infrastructure. The usefulness of this decision-making process for reviewing the driving plan will be explained using a specific example.

[0297] As an example, consider a scenario where, while traveling along the route indicated in the itinerary at the maximum permissible speed for that section of road, the system determines that it is difficult to continue driving at Level 4 autonomous driving within the performance limits that the autonomous driving system can handle. In this case, if the road section is a sparsely populated double-lane road with no dense concentration of vehicles cruising at similar speeds around the vehicle, and is also a straight road, then it is likely that gradually slowing down the vehicle will not significantly impact road traffic, and thus slowing down may be the best course of action.

[0298] As another example, if regular monitoring beforehand reveals that the driver is unable to return to manual driving due to illness or other reasons, and furthermore, if the road section being driven has heavy traffic and many curves with poor visibility is approaching, it may be safer to slow down on a straight section of road beforehand.

[0299] In step S134, the automatic driving control unit 10112 urgently initiates MRM braking. For example, before initiating MRM, the automatic driving control unit 10112 issues a warning notification to inform those around the vehicle that MRM is about to start. The automatic driving control unit 10112 also instructs the driver to prepare a posture (stance) that is appropriate for MRM. After processing in step S134, the automatic driving control unit 10112 moves the process to step S160 and initiates MRM by the system.

[0300] Furthermore, the process of transitioning from step S134 to step S160 is considered a point deduction (2) described later, which will result in a reduction of the driver's evaluation.

[0301] In step S131 described above, the automatic driving control unit 10112 determines that the urgency is low if the distance to the new event occurrence point is greater than a predetermined value and there is sufficient time (step S131, "low"), and proceeds to step S140.

[0302] In step S140, the automated driving control unit 10112 adds the new event to the overview of the ODD section and updates the overview. In the next step S141, the automated driving control unit 10112 notifies the driver of the new event and observes the driver's response to this notification.

[0303] In the next step, S142, the automated driving control unit 10112 determines, based on the driver's response observed in step S141, whether the driver accepts the newly added event, that is, whether the agreement to the contract for the ODD section has been renewed. If the automated driving control unit 10112 determines that the driver has renewed the agreement (step S142, "Yes"), it proceeds to step S143.

[0304] In step S143, the automatic driving control unit 10112, upon recognition of the driver's good driving performance notification, evaluates the driver as an incentive. Nte The IV points are added. Then, the automatic driving control unit 10112 moves the process to step S122 in the flowchart of Figure 7B, according to the code "E".

[0305] The transition from step S143 to step S122 is perceived by the driver as an additional handover event, because the driver has voluntarily acknowledged and responded to the notification. This information acts upon working memory, and appropriate processing by the driver is expected. The same applies to the transition from step S149 to step S122, which will be described later.

[0306] The transition from step S142 to step S143 signifies that the driver has agreed to the contract presented by the system and consciously accepted the scheduled autonomous driving. Therefore, information regarding this contract is stored in the driver's working memory as memory information #4 (WM14).

[0307] On the other hand, if the automatic driving control unit 10112 determines in step S142 that the driver did not renew the agreement (step S142, "No"), it proceeds to step S144. In step S144, the automatic driving control unit 10112 observes whether the driver is able to accept notifications from the system based on the driver's status. In the next step S145, the automatic driving control unit 10112 determines, based on the observation results in step S144, the appropriateness of a forced return notification prompting the driver to forcibly return to manual driving. The automatic driving control unit 10112 also calculates an escape point that minimizes the impact of the return.

[0308] In the next step, S146, the automatic driving control unit 10112 determines whether there is a grace period from the current time until the time to return to manual driving. If the automatic driving control unit 10112 determines that there is a grace period (step S146, "Yes"), it returns to step S144. For example, if the driver is performing NDRA, which involves a significant absence from driving such as napping, there is a possibility that soft notifications such as displays or warning sounds may not be perceived by the driver. Therefore, the automatic driving control unit 10112 repeats the process from step S144 to step S146 until the grace period is exhausted.

[0309] If the automatic driving control unit 10112 determines in step S146 that there is no time to return to manual driving from the current point (step S146, "No"), it proceeds to step S147. In step S147, based on the determination result of the validity of the forced return notification in step S145, the automatic driving control unit 10112 adds a return point and notifies the driver of the return point in stages. For example, the automatic driving control unit 10112 issues preliminary warnings and notifications to the driver in stages.

[0310] When notifying a driver of an unexpected new event, if the driver is asleep when the event occurs, they will have no memory of the new handover point, the necessity of the handover, or the urgency of the situation. Therefore, to prevent the driver from panicking, unlike scheduled handovers, a certain amount of early notification and warning will be given to allow the driver time to think and understand the situation.

[0311] As mentioned above, this is because, in the case of a new event, the recovery action is performed earlier than intended because the driver's memory of the need to recover does not yet contain information about the recovery point, and the driver's work memory does not yet contain information prompting a return to manual driving. The limit of this judgment is the point at which, when MRM is activated, the RRR (Request Recovery Ratio) is high and there is a margin of time α or more that allows the driver to recover without causing disruption to traffic on main roads. For example, if the driver was napping, this is the time it takes to recover from the nap. If the quality of the driver's recovery from the nap is poor, the RRR is high, and the vehicle is approaching a section where there is a risk of causing traffic disruption, the automatic driving control unit 10112 will take preventative measures using MRM before that point.

[0312] RRR indicates the desired probability that the handover will be completed at the handover limit point when a request for the driver to return to manual driving is issued.

[0313] Let's explain RRR in more detail. Ideally, at the handover limit point, it is desirable that [1 / 1] of drivers successfully complete the handover. When indicating the success rate, RRR is defined as [1 / 1].

[0314] However, in reality, there are rare cases where the handover is unsuccessful. For example, if a certain road section allows for a level of failure where 5 out of 10,000 drivers fail the handover, the required RRR for that road section would be a ratio expressed as [1 - 0.0005 / 1].

[0315] This RRR is an index that represents the success target value for handover, defined for each lane of a road section, so that when a vehicle stops in that road section due to the activation of MRM, there will be no rear-end collisions or congestion caused by following vehicles, and the vehicle will not have to come to a sudden stop in the middle of a single-lane road. It is desirable that RRR be used in conjunction with LDM as a judgment factor that changes dynamically in response to changes in the situation over time.

[0316] As a specific example, in Japan, on road sections without shoulders or other escape routes, such as the Metropolitan Expressway, and even more so when the escape routes are already occupied by vehicles that arrived first, it is desirable to set the RRR to [1] in those sections. On the other hand, when there is space to pull over in an escape route, or when the driver's return is unavoidable near an expressway exit where it is possible to pull over onto an ordinary road, it is conceivable to set the return request rate to, for example, 0.95, as this allows for choices such as steering to the escape route as part of MRM, or exiting onto an ordinary road and stopping, while minimizing the impact on following vehicles. Furthermore, in road sections with extremely low traffic volume, if the impact of an emergency stop only affects the vehicle itself within that road section, the RRR may be [0].

[0317] Ideally, RRR (Random Routing) should be information that is constantly updated and provided to vehicles using autonomous driving as part of LDM (Landing Data Management) in order to minimize disruption to traffic caused by MRM (Multiple Reaction Management) to social infrastructure.

[0318] Once the processing in step S147 is complete, the automatic driving control unit 10112 moves the process to step S148. In step S148, the automatic driving control unit 10112 determines whether the preliminary warning and notification in step S147 were recognized by the driver.

[0319] In step S148, the automatic driving control unit 10112 detects a predetermined response from the driver to the alarm or notification (such as an operation on the input unit 10101 or a specific action), and if it determines that the alarm or notification has been acknowledged by the driver (step S148, "Yes"), it proceeds to step S149. In this case, because the driver responded promptly to the call notification, the process can proceed to the normal handover process.

[0320] In step S149, the automatic driving control unit 10112 evaluates the driver based on the quality of the driver's response, and provides an incentive. Nte The IV points are added or subtracted. Then, the automatic driving control unit 10112 proceeds to step S122 in the flowchart of Figure 7B, according to the code "E".

[0321] Furthermore, the transition from step S148 to step S149 means that the driver responded promptly to the call notification, allowing the process to proceed to the normal handover procedure. Therefore, the information that the driver has recognized this preliminary alarm and notification is stored in the driver's working memory as memory information #5 (WM15).

[0322] Here, the memory information #5 stored in working memory by WM15 and the memory information #4 stored in working memory by WM14 as described above are applied to the processing by WM13 in Figure 7A, as indicated by the symbol "C" in the figure.

[0323] If the automatic driving control unit 10112 determines in step S148 that the alarm or notification has not been recognized by the driver (step S148, "No"), it proceeds to step S150. In step S150, the automatic driving control unit 10112 determines whether there is sufficient time to wait for the driver to recognize its return to normal operation. If the automatic driving control unit 10112 determines that there is sufficient time to wait (step S150, "Yes"), it returns to step S132 in the figure according to the symbol "F".

[0324] On the other hand, if the automatic driving control unit 10112 determines in step S150 that there is no margin (step S150, "No"), it moves the process to step S160. This transition from step S150 to step S160 is a process in which the driver's return time runs out and the soft MRM is executed. In this case, the driver's evaluation is reduced according to the delay and negligence in the driver's return, and this becomes a point deduction target (3) described later.

[0325] Here, we will explain the meaning of the processes described in steps S144 to S148 above.

[0326] When Level 4 autonomous driving becomes available, in sections corresponding to that ODD (Operational Design Detachment) section, the driver will be able to engage in NDRA (Non-Directional Recreational Action) that is more detached from the driving and steering loop, allowing them to, for example, take a nap or move to the cargo area.

[0327] In particular, we consider the case where a minor incident occurs during an ODD (Autonomous Driving Level 4) driving section, especially when there is a significant gap between the vehicle and the driver, such as during a short nap. For example, this incident might occur on a straight section of road that includes a series of curves about ten minutes later. An example of such an incident is an insect strike, where an insect hits the windshield. Encountering an insect strike can cause the windshield to become dirty, potentially impairing forward visibility.

[0328] Here, even if the windshield is soiled by an insect strike, it is still safe to drive at Level 4 autonomous driving on straight sections of road. However, in the series of curves following the straight sections, the road becomes significantly uneven, and with a soiled windshield, it may become unsuitable for Level 4 autonomous driving. Therefore, the ODD (Operational Design Deck) may be revised in response to this change in conditions, and Level 4 autonomous driving may become difficult. In this case, the system, considering safety, will notify the driver of the transition from autonomous driving to manual driving earlier than the normal transition timing. The system will then observe the driver's confirmation response to this notification and take the actions described below.

[0329] In other words, similar to the concept of Level 3 autonomous driving, sections where Level 4 autonomous driving is permitted do not mean that all vehicles designed to operate at Level 4 autonomous driving can always operate at Level 4 autonomous driving. Rather, these are sections where Level 4 autonomous driving is possible only when the conditions are met. For the driver to respond appropriately to those conditions, the system must provide the driver with the necessary information to make a decision with sufficient time, and the driver must take the initiative to deal with the situation.

[0330] In such a situation, if the system has sufficient time before reaching a point where a new recovery is required due to a change in conditions, forcing the system to interrupt NDRA and requesting the driver to return to manual driving would be a pointless request for recovery from the perspective of the driver / user.

[0331] In reality, drivers often find the interruption of NDRA inconvenient, such as being in the cargo area or taking a nap, and there is no compelling reason for an immediate return to service. Therefore, checking for early changes in system conditions becomes nothing more than a tedious and unnecessary task for the driver, without any risk. Consequently, repeated unnecessary requests only increase the driver's sense of futility, leading to the filtering effect on notifications mentioned above, and gradually diminishing their importance.

[0332] In step S146, a determination is made to determine the timing for issuing notifications or warnings, taking into account a certain margin of safety, in order to prevent unnecessary premature checks and excessive risks, and to begin preparing the recovery procedure. Then, in step 148, a determination is made as to what to do if the driver's response is delayed, such as whether to start processing equivalent to the normal recovery procedure, depending on whether the notification or warning was recognized by the driver.

[0333] The processes described in steps S144 to S148 are for the purpose of achieving this type of control.

[0334] Furthermore, the criteria for the grace period in step S146 may be applied as parameterized criteria for general passenger cars, vehicles carrying heavy hazardous materials, and large ride-sharing vehicles, such as safety coefficients corresponding to the characteristics of the vehicle and target RRR values ​​required for the road section in question.

[0335] The information presented to the driver by the system via the information display unit 120, etc., is taken into the driver's working memory as information for risk assessment, and is retrieved from the working memory according to the driver's awareness of the importance of the handover, prompting the driver to make decisions about their actions.

[0336] <3-2-3-2. Evaluation of the driver's recovery actions> Here, we will explain in more detail the evaluation of the driver's recovery actions according to the embodiment. First, we will explain the items (1) to (3) for which points are deducted from the driver's evaluation score when the process moves to step S160, which starts MRM, with reference to Table 2.

[0337] [Table 2]

[0338] In the examples in Table 2, for item (1) where points are deducted, a single occurrence results in a deduction of [-1], and repeated occurrences within the same itinerary result in a deduction of [-2]. For item (2) where points are deducted, a single occurrence results in a deduction of [-4], and repeated occurrences within the same itinerary result in a deduction of [-4]. Furthermore, for item (3) where points are deducted, a single occurrence results in a deduction of [-5], and repeated occurrences within the same itinerary result in a deduction of [-5].

[0339] The penalty for point deduction (1) is a penalty for transitioning from step S131 to step S160, and is of an imminent nature. In this case, it is a response to an event that occurred immediately in front of the vehicle on the road it is traveling on without prior notice, and is not attributable to the driver's direct responsibility. However, if the start of MRM is foreseeable based on the situational assessment when using autonomous driving, a penalty will be applied (a third degree of penalty) to prevent repeated use of autonomous driving that is dependent on the system. In addition, for point deduction (1), a mechanism can be implemented where, for example, a temporary conditional penalty is flagged, and the penalty is canceled if it is not reapplied for a certain period of time.

[0340] Penalty item (2) is a penalty incurred when transitioning from step S134 to step S160, and is a penalty incurred in a situation where there is a small amount of time to spare. In this case, even if the system slows down the vehicle's speed to extend the time to reach the point where the handover is mandatory, the return to manual driving may be insufficient due to negligence or other reasons attributable to the driver, resulting in the initiation of MRM. In this case, since the driver is responsible for the initiation of MRM, a heavier penalty is imposed than for penalty item (1) mentioned above (second degree of penalty).

[0341] The penalty for point deduction (3) is for the transition from step S150 to step S160, and the degree of the deduction is such that there should have been sufficient time. In this case, the use should have had sufficient time, and the handover could have been resolved by returning to the vehicle early. The system will perform MRM in a software-based way that has less impact on the surroundings. However, in order to prevent negligent handover behavior by the driver and to encourage behavioral change to take prompt action, a heavier penalty will be imposed than the penalty for point deduction (2) mentioned above (the first degree of penalty).

[0342] Next, Table 3 will explain an example of the evaluation of a driver during a normal handover request from the system to the driver (also called a Request to Intervene or Transition Demand). The evaluation exemplified in Table 3 is performed, for example, in step S124 of Figure 7B, but it is not limited to this; evaluations according to Table 3 can also be performed at other handover timings or even at other timings.

[0343] [Table 3]

[0344] In Table 3, the first four rows show examples of cases where points are added to the driver's evaluation, row five shows examples where no points are added or deducted from the driver's evaluation, and rows six onwards show examples of cases where points are deducted from the driver's evaluation.

[0345] According to Table 3, examples of situations in which points are awarded include when the driver chooses to take a break or rest early, or when they choose a detour and abandon the handover at the designated point in advance, when the driver requests assistance from a lead vehicle, remote control, or remote operation early, and when the handover is initiated by a return warning sound or the driver's own situation check (autonomous generation of return sense). In each case, regardless of whether it is a one-time occurrence or a repeated occurrence within the journey, points are awarded [+0.2]. Additionally, if the driver recognizes and detects a prior notification of a return request, points are awarded [+0.1] regardless of whether it is a one-time occurrence or a repeated occurrence within the journey.

[0346] If the driver initiates the recovery procedure in response to the recovery notification, it will be considered a normal recovery procedure, and no points will be added or deducted.

[0347] On the other hand, as an example of when points are deducted, if a driver initiates a return to duty after receiving a return warning, a penalty of [-0.2] is imposed for a one-time occurrence, and double that amount for repeated occurrences within the same itinerary, as it is considered the driver to have disregarded the situation. This penalty is intended to prevent drivers from disregarding the situation or postponing high-priority tasks. If a driver fails to recognize or detect a prior notification of a return request (i.e., ignoring a notification that is not remembered), a penalty of [-0.5] is added as it is considered a malicious action, regardless of whether it is a one-time occurrence or repeated occurrences within the itinerary. If a driver initiates a return to duty after receiving a mandatory return request, a penalty of [-1.0] is imposed for a one-time occurrence, and double that amount for repeated occurrences within the same itinerary, as it is considered a lack of risk awareness.

[0348] Furthermore, on main roads, if the system performs pre-emptive deceleration to create a time buffer, resulting in a barely successful handover, a penalty of -2.0 is applied for a single occurrence, and 1.5 times that amount is applied for repeated occurrences within the same itinerary. Similarly, on main roads, if the driver is unable to handle the handover, i.e., the handover fails, and the system performs MRM, a penalty of -4.0 is applied for a single occurrence, and 1.5 times that amount is applied for repeated occurrences within the same itinerary. This penalty is intended to deter drivers from intentionally committing violations.

[0349] Furthermore, a penalty of -0.5 is applied in cases where, on low-speed non-main roads, the system generates a time buffer by slowing down in advance to achieve the handover, and on low-impact roads (such as roads with extremely low traffic volume), the driver is unable to handle the handover, i.e., the handover fails, and the system performs MRM, regardless of whether these are one-off occurrences or repeated occurrences within the same journey.

[0350] Furthermore, regarding the use of NDRA, if a driver initiates NDRA without confirming the application status of the ODD, a penalty of -2.0 will be imposed for a single occurrence, and double that penalty for repeated occurrences within the same itinerary. If NDRA is used outside of the ODD, this is a violation, and a penalty of -3.0 will be imposed for a single occurrence, and double that penalty for repeated occurrences within the same itinerary.

[0351] The system (automatic driving control unit 10112) accumulates the addition / deductive points shown in Table 3 for the same driver and uses this as the driver's evaluation value. The system accumulates the driver's addition / deductive points for, for example, all itineraries set and executed by the driver in the system, or itineraries executed within a predetermined period. In this way, by imposing usage-based penalties according to the driver's history, the evaluation results are reflected in the control system to prevent malicious use such as failing to respond despite a handover request issued by the system, or repeatedly performing recovery operations without regard for prompt NDRA interruption requests.

[0352] The system can penalize drivers who have a low rating (for example, a negative rating) for using autonomous driving.

[0353] One example of a penalty for drivers is the restriction on the use of autonomous driving. This restriction could include, for example, delaying the estimated arrival time at the destination, mandatory stops at service areas, locking the vehicle for a certain period of time, imposing restrictions on the maximum speed, restricting the use of autonomous driving mode (for the same day, week, or month), or restricting the sections in which autonomous driving mode can be used. These restrictions give drivers an intuitive sense of loss (risk), which encourages them to return to manual driving early or take appropriate action.

[0354] Another example of a penalty for the driver is the restriction on the use of secondary tasks (NDRAs) that the driver engages in while using autonomous driving. This restriction can give the driver an intuitive sense of the risks involved, encouraging them to return to manual driving earlier or take appropriate action.

[0355] Regarding restrictions on the use of this secondary task, one possible approach is to restrict the use of terminal devices used by drivers for secondary tasks. Restrictions on terminal devices could include obscuring the screen displayed on the device or encroaching on the screen with an arbitrary image. These measures could, for example, allow drivers to recognize risks through gradual advance warnings that appeal to their intuition. Another possibility is to swap the screen currently in use on the terminal device with the handover information window (swapping the child screen and the parent screen). This could, for example, prompt the driver to pay attention to the handover process.

[0356] Furthermore, measures such as forcibly freezing the screen of the terminal device and retroactively invalidating any actions performed by the driver using the terminal device can be considered. These measures, by forcibly interrupting the operation, can give the driver the feeling that any actions taken up to that point will be wasted, thereby encouraging greater attention to the handover of driving.

[0357] Control of these terminal devices can be implemented, for example, by installing application software on the terminal device to utilize the system according to the embodiment (such as displaying an overview of the itinerary, providing advance notification to the driver of the end of the ODD section, etc.), and implementing the functionality of the application software.

[0358] The values ​​for each addition and deduction explained using Tables 2 and 3 are examples only and are not limited to those examples. Similarly, the examples of how points are added and deducted are also examples only and are not limited to those examples.

[0359] <3-2-3-3. Overview of the itinerary applicable to the embodiment> Next, we will describe in more detail the overview display of the itinerary applicable to the embodiment.

[0360] Figure 8 is a schematic diagram illustrating an example of an overview display of a journey applicable to the embodiment. In Figure 8, the overview display 50 includes a short-range display section 51a, a medium-range display section 51b, and a long-range display section 51c. In Figure 8, the direction of travel of the vehicle is indicated from the lower end to the upper end. In Figure 8, the lower end represents the current position of the vehicle, but this is not limited to this example. The icon 52 representing the vehicle is for the purpose of making the journey easier to visualize and can be omitted.

[0361] In Figure 8, the short-range display unit 51a displays the section from the vehicle's current position to a predetermined first distance. The first distance is, for example, a distance of about 15 minutes in driving time from the vehicle. In the example in Figure 8, the short-range display unit 51a can establish a linear relationship between the vertical position on the screen and the actual distance.

[0362] The medium-range display section 51b is shaped to narrow in width according to the height on the screen, so that it converges at the point VP at infinity, starting from the upper end which is the width W1 of the short-range display section 51a. In the medium-range display section 51b, the relationship between the vertical position on the screen and the actual distance is made nonlinear, so that, for example, the change in the actual distance relative to the position on the screen can be made larger as you move upwards on the screen.

[0363] Here, in Figure 8, if the vertical position represents the time of arrival along the direction of travel, then the distance h from the point VP at infinity is shown. diff The reciprocal of this can be displayed in proportion to the time elapsed. In this way, by displaying a sense of perspective on the medium-distance display unit 51b, it becomes possible to efficiently display the arrival time on a small screen. By accurately displaying the degree of influence of each handover point, etc., through the display format of this overhead display 50, the driver can intuitively grasp the time at each destination.

[0364] On the other hand, the long-distance display unit 51c extends from a position with a width W2 in front of the point of infinity VP, while maintaining that width W2. Similar to the short-distance display unit 51a described above, the long-distance display unit 51c can establish a linear relationship between its vertical position on the screen and the actual distance.

[0365] Furthermore, it is assumed that the entire section shown in Figure 8 is a section where autonomous driving at Level 4 is possible. However, it is also assumed that within this section, there are sections where it is preferable for the driver to return to manual driving for reasons such as the road narrowing or the presence of a railway crossing. In such sections, the RRR (Return Request Probability) for the driver is expected to be high, and if the driver does not appropriately return to manual driving, it may cause social harm such as affecting following vehicles.

[0366] Therefore, information indicating sections where the RRR (Rapid Ratio) is high is displayed within the overhead view 50. For example, a warning sign 53 indicating a narrower road width is displayed for such sections. This warning sign 53 makes it possible to alert the driver. In addition, a recommended handover point can be set a predetermined distance before the section in question, and the recommended handover section 56 can be highlighted and displayed.

[0367] In this embodiment, to make it easier for the driver to switch from automatic driving to manual driving, the overhead view 50 shown in Figure 8 is modified to include section displays indicating the recommended driving mode for each section. The overhead view 50 with these section displays added will be explained using Figures 9A to 9C.

[0368] Figure 9A is a schematic diagram showing an example of an overhead view 50a in which each section is color-coded according to the embodiment. In Figure 9A, the overhead view 50a distinguishes between the section where automatic operation is possible 53a, the section where recovery posture is maintained 53b, and the section where return to operation is mandatory 53c by color coding.

[0369] The automated driving section 53a indicates a section where automated driving at Level 4 is possible, and is displayed in green, for example, to evoke a sense of safety and security. The return-to-manual driving section 53b is the section immediately before returning from automated driving to manual driving, and indicates a section where it is desirable for the driver to maintain a posture for returning to manual driving. The return-to-manual driving section 53b is displayed in yellow, for example, to draw the driver's attention. The mandatory return-to-manual driving section 53c indicates a section where manual driving by the driver is mandatory, and is displayed in red, for example, to indicate caution.

[0370] The aforementioned use of green, yellow, and red is merely an example, and the system is not limited to these color combinations. Furthermore, if each section is clearly distinguishable, a single color is acceptable instead of color coding.

[0371] By changing the display method according to the section and the distance from the vehicle, the driver can easily understand when to switch from autonomous driving back to manual driving.

[0372] Figure 9B is a schematic diagram showing an example of a circular overhead view display 50b according to the embodiment. In the example in Figure 9B, the top of the circular display represents the position of the vehicle, and the display shows the distance from the vehicle increasing in a clockwise (rightward) direction starting from that point. Furthermore, the width of the display is narrowed as the distance from the vehicle increases, thereby emphasizing the sense of distance.

[0373] The circular overhead view 50b, as described above, is suitable for use in narrow areas, such as the display screen of a wearable device like a wristwatch.

[0374] Figure 9C is a schematic diagram showing an example of an overhead view 50c including road information according to an embodiment. The overhead view 50c shown in Figure 9C is an example in which road information such as icons 54a corresponding to traffic signs and icons 54b indicating facilities has been added to the overhead view 50a shown in Figure 9A. Icon 54a indicates, for example, the locations and content that the driver should pay attention to in an autonomous vehicle, and in this example, it is a display that imitates traffic signs actually installed on roads. Icon 54b indicates facilities necessary when the vehicle is driving, and is displayed corresponding to locations such as gas stations, parking areas, and service areas.

[0375] Furthermore, in Figure 9C, sections where the time required to pass through varies greatly, such as congested areas, are indicated as section indicators 55a and 55b.

[0376] In this way, by using the overhead view 50c with added road information, as the driver's conscious memory progresses, risk information for each approach time is taken into the visual cortex, and at points of high importance, i.e., high risk, it becomes a stimulus that acts on the driver's working memory when making decisions about action. As a result, the driver can predict earlier when to switch from autonomous driving to manual driving, and can return to manual driving more smoothly compared to when only monotonous route displays are uniformly presented.

[0377] Furthermore, the overhead view 50c and the overhead view 50a shown in Figure 9A can be displayed, for example, on the screen of a terminal device used by a driver when using the automated driving system according to the embodiment. For example, the display of the overhead view 50a or overhead view 50c is controlled by an application program related to the information processing program according to the embodiment, which is installed on the terminal device, running on the CPU 10010. In this case, the overhead view 50a or overhead view 50c can be displayed, for example, at the right edge or left edge of the screen, in a state where its width is compressed. However, the overhead view 50a or overhead view 50c may also be displayed across two sides that share a vertex of the screen, or across three sides of the screen, or across the perimeter of the screen.

[0378] <3-2-4. Example of HCD control configuration according to the embodiment> Next, an example of the HCD control configuration according to the embodiment will be described in more detail. Figure 10 is an example of a functional block diagram to explain the function of HCD control in the automatic driving control unit 10112 according to the embodiment. Note that in Figure 10, the functions of the automatic driving control unit 10112 that realize HCD control are shown, and other functions are omitted as appropriate.

[0379] In Figure 10, the automated driving control unit 10112 includes the HMI 100, the driver return delay evaluation unit 101, the driving path pre-predictability acquisition range estimation unit 102, the remote support control / steering assistance feasibility monitoring unit 103, the driver behavior change achievement level estimation unit 104, the vehicle driving path performance information provision unit 105, the ODD application estimation unit 106, the automated driving use permission integrated control unit 107, and the driver behavior quality evaluation unit 108. Each of these units is implemented by running an information processing program according to the embodiment on the CPU 10010, and is configured, for example, as a module on the RAM 10012, which is the main memory.

[0380] The HMI100 provides an interface for the driver, and is connected to, for example, an information display unit 120, a terminal device 121, an in-vehicle light source 122, an audio device 123, and an actuator 124.

[0381] The information display unit 120 displays a predetermined information in accordance with commands from the HMI 100. The terminal device 121 may be a terminal device brought into the vehicle by the driver, or a terminal device pre-installed in the vehicle. The HMI 100 can communicate bidirectionally with the terminal device 121. The terminal device 121 can receive user operations and supply control signals to the HMI 100 according to the received user operations. The terminal device 121 also displays a predetermined screen on its display device in accordance with commands from the HMI 100. The in-vehicle light source 122 is a light source installed inside the vehicle, and its on / off state and light intensity are controlled by the HMI 100.

[0382] The sound device 123 includes a speaker or buzzer and a drive circuit to operate them. The sound device 123 emits sound in response to the control of the HMI 100. The sound device 123 may also include a microphone. The sound device 123 converts an analog sound signal based on the sound picked up by the microphone into a digital sound signal and supplies it to the HMI 100.

[0383] Actuator 124 drives predetermined parts inside the vehicle according to the control of HMI 100. For example, actuator 124 can apply vibrations such as haptic vibrations to the steering wheel. Another actuator 124 can also control the recline of the driver's seat according to the command of HMI 100.

[0384] Based on information from the road path prediction range estimation unit 102, remote support control / steering assistance compatibility monitoring unit 103, and ODD application estimation unit 106 (described later), the HMI 100 controls the operation of the information display unit 120, terminal device 121, in-vehicle light source 122, sound device 123, and actuator 124. This enables the driver to receive the following visual and auditory notifications.

[0385] • Prior notification via guidance sound In this case, it is preferable to use a sound that is easily noticeable to people but does not cause excessive stimulation, such as the in-flight chime sound in a passenger aircraft (for example, the "beep" sound when the seatbelt sign is turned on). This notification by guidance sound can be applied, for example, to prior notification of the return operation from autonomous driving to manual driving. For example, this notification by guidance sound could be used when the system presents the driver with an agreement to a "contract".

[0386] • Notification requesting a return to manual operation The HMI 100 can provide auditory notification through sounds emitted by the sound device 123 or visual notification through displays on the information display unit 120 when such a request is made. The HMI 100 may also provide tactile notification by driving the actuator 124 to apply haptic vibrations to the steering wheel when such a request is made. Furthermore, the HMI 100 can instruct the driver to point and call out to the road ahead.

[0387] Warnings and alarms The HMI 100 can provide warnings or alerts to the driver audibly, visually, or tactilely. For example, the HMI 100 can provide an audible warning by controlling the sound device 123 to emit a warning sound. In this case, the warning sound may be a louder sound compared to the guidance sound mentioned above. The HMI 100 can also provide a visual warning by controlling the information display unit 120 or the in-cabin light source 122 to cause red light to flash or warning lights to illuminate inside the cabin. Furthermore, the HMI 100 can provide a tactile warning by controlling the actuator 124 to strongly vibrate the seat in which the driver is seated.

[0388] Penalties The HMI100 can implement controls that result in penalties for the driver. For example, the HMI100 can implement controls that are considered to cause discomfort to the driver, such as visual cues, operational restrictions, mild pain to the driver, blowing cold air, or tilting the driver's seat forward. The HMI100 can also implement simulated controls, such as generating lateral swaying of the vehicle, performing uncomfortable acceleration and deceleration, and simulated lane departure, to impose penalties on the driver that directly encourage early recovery or have a delayed effect. Furthermore, the HMI100 can impose penalties tailored to the driver's knowledge, such as displaying information on fines, mandatory entry into service areas as a penalty and displaying the duration of the detention, notifying of the suspension of use due to autonomous driving penalties, and issuing warnings for future or repeated usage restrictions.

[0389] The driver return delay evaluation unit 101 evaluates the delay in the driver's return from autonomous driving to manual driving. For example, it is connected to a lifestyle log data information server 130, wearable device log data 131, face / upper body / eyeball cameras 132, biometric information index acquisition unit 134, in-vehicle localizer 135, and response evaluation input unit 136. The driver return delay evaluation unit 101 also acquires information indicating the individual driver's characteristics for returning to manual driving from a remote server dictionary 137.

[0390] Wearable device log data 131 is log data acquired from a wearable device when the driver is wearing that device. Wearable device log data 131 includes, for example, the driver's behavioral history and biometric information.

[0391] The face / upper body / eyeball camera 132 is a camera installed inside the vehicle to capture images of the driver's upper body, including the head. The face / upper body / eyeball camera 132 is installed inside the vehicle to capture images of the driver's facial expressions, subtle eye movements, and upper body movements. The face / upper body / eyeball camera 132 is not limited to this and may include multiple cameras that capture images of the face, eyes, and upper body, respectively. The body posture / head camera 133 is installed inside the vehicle and captures images of the driver's body posture, including the head. By analyzing the images captured by this body posture / head camera 133 in a time series, the driver's body posture and the position and orientation of the head can be tracked.

[0392] In this embodiment, the cameras are described as being separated into a face / upper body / eyeball camera 132 and a body posture / head camera 133 for convenience in terms of installation flexibility. However, this is not limited to this example, and a device integrating these cameras may also be used.

[0393] The biometric information acquisition unit 134 acquires the driver's biometric information based on the output of various sensors installed in the vehicle, for example. The biometric information to be acquired may include respiration, pulse, exhalation, body temperature distribution, and electrooculography. However, the biometric information acquisition unit 134 can also acquire some of the driver's biometric information from wearable devices worn by the driver.

[0394] The in-vehicle localizer 135 is a localizer installed inside the vehicle. The response evaluation input unit 136 receives the driver's response to requests, warnings, etc., presented to the driver by the HMI 100.

[0395] Furthermore, the information acquired by the wearable device log data 131, the face / upper body / eyeball camera 132, the biometric information index acquisition unit 134, the in-vehicle localizer 135, and the response evaluation input unit 136 is stored in the lifestyle log data information server 130 as the driver's lifestyle log.

[0396] Figure 11 is an example of a functional block diagram illustrating the functions of the driver return delay evaluation unit 101 according to the embodiment. In Figure 11, the driver return delay evaluation unit 101 includes a driver behavior response evaluation unit 1010, a correlation characteristic learning unit 1011, a conditional return distribution individual characteristics / situational awareness decline characteristic dictionary 1012, and a situational awareness decline transition prediction unit 1013.

[0397] The Conditional Return Distribution Individual Characteristics / Situation Awareness Reduction Characteristics Dictionary 1012 is a dictionary relating to the observable evaluation values ​​and situation awareness reduction characteristics of individual drivers. The Correlation Characteristics Learning Unit 1011 learns the correlation characteristics between the individual driver's observed evaluation values ​​and the return delay time distribution, based on information indicating the individual driver's return characteristics to manual driving obtained from the remote server dictionary 137 and the evaluation values ​​and situation awareness reduction characteristics obtained from the Conditional Return Distribution Individual Characteristics / Situation Awareness Reduction Characteristics Dictionary 1012.

[0398] In this embodiment, the remote remote server dictionary 137 is located on a remote remote server outside the vehicle, but this is not limited to this example. In other words, the reason for installing the remote remote server dictionary 137 on an external server is that, with the spread of commercial vehicles and shared cars, the characteristics of the driver are not necessarily tied to a specific vehicle, and this is one example of its use. The remote remote server dictionary 137 may also be installed in the vehicle being used.

[0399] The driver behavior response evaluation unit 1010 acquires various information about the driver from the HMI 100. For example, the driver behavior response evaluation unit 1010 acquires preliminary information from the HMI 100 based on lifestyle logs. In addition, the driver behavior response evaluation unit 1010 acquires the following information about the driver from the HMI 100 based on facial and body images acquired by various sensors (cameras). • Recognition information of facial expressions and body posture. • Information about the eyes. In this example, we obtain evaluations of local eye behavior, such as PERCLOS (percentage of time with eyes closed per unit of time) and saccades (rapid eye movements). • Posture and changes in posture. In this example, the quality of the recovery behavior is evaluated based on posture and changes in posture. • Position and posture when leaving the seat inside the vehicle. • Biometric information.

[0400] The driver behavior response evaluation unit 1010 evaluates the driver's behavioral responses based on the information obtained from the HMI 100 and the correlation characteristics obtained from the correlation characteristic learning unit 1011. The evaluation results are passed to the situation awareness decline trend prediction unit 1013. Based on these evaluation results, the situation awareness decline trend prediction unit 1013 predicts the trend regarding the decline in the driver's situation awareness.

[0401] Furthermore, if the lifestyle log data information server 130 is available, a portion of the lifestyle log data acquired from the lifestyle log data information server 130 can be input to the driver behavior response evaluation unit 1010. By utilizing the driver's lifestyle log data, the driver behavior response evaluation unit 1010 improves the accuracy of estimating the driver's level of alertness, such as insufficient sleep, accumulated fatigue, sleep apnea syndrome, and residual alcohol from drinking, thereby improving the accuracy of determining the driver's situation awareness ability. This enables safer control against sudden drowsiness such as microsleep.

[0402] Thus, the driver recovery delay evaluation unit 101 functions as an acquisition unit that acquires the driver's status based on information acquired from the lifestyle log data information server 130, wearable device log data 131, face / upper body / eyeball camera 132, biometric information index acquisition unit 134, response evaluation input unit 136, and remote remote server dictionary 137.

[0403] Returning to Figure 10, the explanation is as follows: the road pre-predictability acquisition range estimation unit 102 acquires the high freshness update LDM 140 and estimates the range of pre-predictability acquisition for the road based on the acquired high freshness update LDM 140. In other words, the road pre-predictability acquisition range estimation unit 102 acquires the range in the road where events can be predicted in advance, based on the high freshness update LDM 140.

[0404] Figure 12 is a schematic diagram illustrating a high-freshness update LDM 140 applicable to the embodiment. In each region, regionally distributed LDMs 1400a, 1400b, ..., 1400n are deployed. These regionally distributed LDMs 1400a, 1400b, ..., 1400n are updated as needed based on stationary sensors 1401, dedicated probe car information 1402, general vehicle ADAS information 1403, weather information 1404, and emergency notification information 1405 (such as falling hazardous materials), which correspond to each region.

[0405] The regionally distributed LDMs 1400a to 1400n are transmitted via 5G (fifth-generation communication), 4G (fourth-generation communication), or other communication methods. Each transmitted regionally distributed LDM 1400a to 1400n is received, for example, by the automatic driving control unit 10112, and aggregated to constitute the high-freshness update LDM 140. In addition, emergency notification information 1405 is distributed by broadcast 1406 and received by the automatic driving control unit 10112 either directly or included in the aforementioned 5G or 4G communication. The automatic driving control unit 10112 updates the high-freshness update LDM 140 based on the received emergency notification information 1405.

[0406] Returning to Figure 10, the explanation is as follows: The road pre-predictability acquisition range estimation unit 102 estimates the range in the road where events can be predicted in advance using the acquired high-freshness update LDM 140, based on the information and circumstances exemplified below.

[0407] Here, significant investment in environmental infrastructure and dedicated probe cars is possible in urban transportation centers, etc. On the other hand, in areas or times of use where the return on investment is small, there is a situation where the data relies mainly on sporadic data collected in shadow mode from ADAS information 1403 from general vehicles. The predictive ability of the driving route that can be provided by the highly up-to-date LDM 140 is living information that actively changes over time depending on the deployment of regionally distributed LDMs, as described later, and the allowable communication bandwidth of the communication network.

[0408] • Pre-trip information for the LDM140 service area, provided before the journey. • Risk information regarding a decrease in the replacement frequency of high-freshness replacement LDM140. The replacement frequency of high-freshness replacement LDM140 fluctuates over time, for example, due to the density of passing probing vehicles (e.g., dedicated probe cars). • Information gaps due to insufficient regional wireless communication bandwidth, etc. • Insufficient passage of probing vehicles to supplement information against reduced predictability due to bad weather. - Reduced predictability due to temporary information deficiencies in information acquired from lead vehicles / vehicle groups that supplement or replace information in travel itinerary sections where updated LDM is unavailable due to infrastructure development, etc.

[0409] The Remote Control and Steering Assistance Feasibility Monitoring Unit 103 monitors the feasibility of remote control and steering assistance based on information acquired from the Remote Control I / F 150. This monitoring by the Remote Control and Steering Assistance Feasibility Monitoring Unit 103 is intended for use in options such as regional traffic, platooning support, and limited bridging support.

[0410] Figure 13 is a schematic diagram illustrating the acquisition of information by the remote support control I / F 150, applicable to the embodiment. Remote operation commanders 1500a, ..., 1500n-1 collect information from standby steering operators 1501 and 1501n+1, and from the dedicated lead guidance contract vehicle 1502, respectively. In this example, remote operation commander 1500a also collects information from the lead guidance vehicles 1503m and 1503m+1. Remote operation commanders 1500a, ..., 1500n-1 each transmit the collected information to the remote support control I / F 150 using, for example, 5G communication. The communication method here is not limited to 5G; it may also be 4G. In the example shown in the figure, lead guidance vehicles 1503m and 1503m+1 directly transmit the collected information to the remote support control I / F 150.

[0411] Returning to Figure 10, the remote support control / steering support capability monitoring unit 103 performs the following processing based on the information obtained from the remote support control I / F 150. • Control support using remote control support services, such as when the driver has difficulty switching from autonomous driving to manual driving. This includes system-based control such as early evacuation actions and operator assignment control, which are performed on behalf of the driver. • Remote control of vehicle steering by a remote operator. This is a control command that is issued when an external remote control contract is in place and an operator can be assigned. • Monitoring of remote support execution and monitoring of fallback (degraded operation) in the event of malfunctions.

[0412] The driver behavior change achievement level estimation unit 104 estimates the level of change achieved in the driver's behavior change in relation to the system limits. Figure 14 is an example of a functional block diagram illustrating the function of the driver behavior change achievement level estimation unit 104 according to this embodiment. The driver behavior change achievement level estimation unit 104 includes an excellent recovery steering action evaluation point addition unit 1040 and a penalty action cumulative addition record unit 1041.

[0413] The excellent return steering action evaluation point addition unit 1040 adds evaluation points for excellent driving actions when returning to manual driving. The cumulative penalty action addition record unit 1041 deducts evaluation points according to violations of the system's request to return to manual driving or negligence in responding to the return request. Furthermore, the cumulative penalty action addition record unit 1041 adds cumulative evaluation points for actions that result in penalties. The driver behavior change achievement level estimation unit 104 can obtain evaluation points, for example, from the driver return delay evaluation unit 101.

[0414] Returning to Figure 10, the vehicle's travel route performance information provision unit 105 provides performance information regarding the vehicle's travel (passage) route to the LDM regional cloud (for example, the regionally distributed LDM 1400a to 1400n). Specifically, the vehicle's travel route performance information provision unit 105 provides the following information.

[0415] • Notification of fluctuations, discrepancies, and anomalies in pre-acquired map information, and risk information for following vehicles even after entering a section. If an abnormality or dangerous risk (falling object, accident, disaster, etc.) is detected while driving, the system will automatically or manually send emergency information from the vehicle. • Notification of characteristic events (event notifications by drivers / vehicle users), and information on suspected risks. • Information is provided through manual reporting by drivers, rather than automatic reporting. In this case, temporarily recorded road environment information from a few minutes prior to the report is provided. • Uploading probing requests from the server. For example, receiving an emergency report from a preceding vehicle regarding an unknown risk of falling objects, the system performs a detailed verification. To do this, it sends a detailed scanning request to the LDM cloud server managing the section and uploads the information obtained from a detailed scan equivalent to a normal environmental scan during driving, or from an enhanced environmental awareness scan with a faster refresh rate.

[0416] The vehicle's own driving history information provision unit 105 can, as an option, provide information to following vehicles and waiting vehicles based on the information it possesses, in cases where it is difficult to provide LDM.

[0417] For example, if the traffic volume passing through a section is low and it is difficult to provide constantly updated LDM with the uploaded information acquired, or if the infrastructure is not adequately equipped with a cloud full of LDM, then autonomous driving at Level 4 based on the infrastructure-based, highly up-to-date LDM140 cannot be expected.

[0418] In such cases, the system switches from autonomous driving to manual driving and pairs with the vehicle requiring assistance (e.g., a following vehicle or a waiting vehicle). Then, it provides the vehicle requiring assistance with environmental data acquired while the vehicle is driving at autonomous driving level 2 or lower, as data necessary for driving at autonomous driving level 4. Furthermore, it provides LDM (Least Drive Data) through pairing with the specific vehicle requiring assistance, and provides information when the following vehicle is driving at autonomous driving level 4.

[0419] For example, the vehicle's own driving history information provision unit 105 can provide this information in cooperation with the remote support control / steering support capability monitoring unit 103 mentioned above. For example, if the paired partner is a lead support vehicle, it can provide route guidance information to the following vehicles behind that partner. Combined with the high-freshness update LDM 140, the operation of these lead vehicles and remote support systems proves to be particularly useful when used in platoon transport including unmanned vehicles, and the operation may be applied to use where the driver is not actually in the vehicle.

[0420] The ODD application estimation unit 106 determines whether the vehicle is in a section (ODD section) where it can travel at each level of autonomous driving. The ODD application estimation unit 106 makes this determination based on the following information.

[0421] • Evaluation information on the driver's credit rating, reinstatement violations, penalty points, and other historical records. • Information evaluating the driver's understanding and proficiency regarding the need for return to work based on HCD (Human-Centered Design). • Information indicating the availability status of the high-freshness updated LDM140. • Information on the probability of a return request (RRR) based on LDM, such as the high-freshness update LDM140, and information indicating the locations that can be selected based on the evacuation options. • Information indicating the limitations of applying autonomous driving based on diagnostic results of vehicle-mounted equipment. • Information showing vehicle dynamics (characteristics of passenger load, cargo, and cargo shift risk).

[0422] Furthermore, the ODD application estimation unit 106 estimates ODD sections where unsupervised autonomous driving at autonomous driving level 4 is applicable, and ODD sections where autonomous driving equivalent to autonomous driving level 3 is available, depending on the update status of LDMs such as the high-freshness update LDM 140 and other updates. In addition, the ODD application estimation unit 106 reviews and updates the applicable ODD sections in accordance with newly acquired risk information, equipment contamination, changes in the driver's condition, etc., during the driving journey. At this time, the ODD application estimation unit 106 notifies the driver of the information update via the HMI 100 and evaluates the driver's understanding of the change in situation based on the driver's response to the notification.

[0423] The Automated Driving Permission Integrated Control Unit 107 provides integrated control over the permission to use automated driving. For example, the Automated Driving Permission Integrated Control Unit 107 provides integrated control over the permitted automated driving status for each driving section. The Automated Driving Permission Integrated Control Unit 107 also controls the execution of MRM (Measuring Memory Management). Furthermore, the Automated Driving Permission Integrated Control Unit 107 implements controls to impose penalties on the driver, such as forcibly interrupting the use of automated driving, in response to violations while using automated driving. Examples of violations include delays in the driver's response to requests from the system to return to manual driving, and repeated continuous use of automated driving at Automated Driving Level 3.

[0424] The driver behavior quality evaluation unit 108 evaluates the quality of the driver's behavior (behavioral quality) during autonomous driving and other similar activities.

[0425] The driver behavior quality evaluation unit 108 evaluates the quality of the driver's actions, for example, based on the driver's steering stability. The driver behavior quality evaluation unit 108 evaluates various driving-related items, such as the driver's steering, accelerator and brake operation, and turn signal operation. The driver behavior quality evaluation unit 108 also evaluates the driver's designated operations and actions, such as pointing and calling out, in response to the system's request to hand over to manual driving. It may also perform a posture recovery evaluation when the driver returns to a driving steering posture from an NDRA task where their posture has been compromised.

[0426] One piece of information that is difficult for the system to directly observe when implementing HCD control is understanding the driver's situation awareness, which is information from within the driver's brain. Therefore, in the HCD according to this embodiment, attention is paid to steering behavior when situation awareness is reduced. For example, in a state of insufficient situational awareness, i.e., a state of reduced situational awareness, intelligent feedback becomes insufficient, and steering due to excessive reflexes increases. In other words, in a state of reduced situational awareness, steering that would normally be performed smoothly often becomes excessive steering due to incorrect feedback. Focusing on this, steering during autonomous driving is compared with steering during normal manual driving to use as an evaluation index for the driver's situation awareness.

[0427] <3-3. Regarding Autonomous Driving Level 4 Applicable to the Embodiment> Here, we will describe the autonomous driving level 4 applicable to the embodiment.

[0428] <3-3-1. Basic structure> First, we will explain the basic structure of autonomous driving level 4. Figure 15 is a schematic diagram illustrating the basic structure of autonomous driving level 4 applicable to the embodiment.

[0429] In Figure 15, charts (a) to (g) each have position on the horizontal axis. Chart (a) shows the recovery time ΔT. drd Chart (b) shows an example of the relationship between position (or elapsed time to arrive calculated from vehicle speed), and the grace period ΔT.2lim Examples of the relationship between and position are shown. These recovery times ΔT drd and grace period ΔT 2lim This will be discussed later.

[0430] Chart (c) shows an example of a continuously updated (high-freshness updated) LDM data section. In this example, within a section where autonomous driving at Level 4 (labeled as Level 4 in the diagram) is possible, there is an LDM data section where the data has not been updated and has become outdated. This outdated LDM data section is designated as section 64 in Chart (b) as a section with insufficient LDM maintenance as previously announced, and is a section where manual driving is temporarily required. Section 63 in Chart (b) is a section where the provision of high-freshness updated LDM 140 cannot be maintained due to a decrease in passing vehicles or insufficient communication bandwidth due to excessive use of surrounding public communications. Manual driving is also required in this section.

[0431] Chart (d) shows examples of RRR and recovery success rates. Charts (e), (f), and (g) show examples of the presence or absence of lead vehicles, the availability of waiting areas, and the availability of control operators, respectively.

[0432] Although not illustrated, for example, in section 65b, there may be no assistance from the information shown in charts (e), (f), and (g), and a handover event may occur that the driver cannot handle when entering section 65b. In this case, if the vehicle stops in section 65b due to the MRM function, there is a high risk of serious violations, such as the closure of the road including section 65b, or the risk of the vehicle suddenly stopping and rear-ending a following vehicle at the exit of a tunnel with poor visibility (details will be discussed later).

[0433] The system communicates with the LDM on the cloud network via the regional infrastructure communication network to request new information, and from that cloud network, highly accurate status of the constantly updated LDM for the planned driving section is provided. Alternatively, from the leading vehicle, high-definition individual LDM information obtained in real-time situations such as V2V (Vehicle to Vehicle) is provided.

[0434] Based on the provided information, the system determines a margin time ΔT indicating a margin where it is presumed that the host vehicle can drive safely, according to the equipment status confirmed for use in the latest self-diagnosis status of the host vehicle. 2lim (Time to reach limit of MRM = Immediate far-ahead predictability range), and the recovery time ΔT that the system passively or actively detects, which is the time required for the driver to resume manual driving. drd (Time delay to resume driving = Notification to driving) and calculates them.

[0435] As shown in chart (a) of FIG. 15, the recovery time ΔT drd For example, assuming that the current position of the host vehicle is position P61, it indicates that the host vehicle resumes manual driving at a position advanced by a distance corresponding to the recovery time ΔT at position P61 from position P61. On the other hand, as shown in chart (b), the margin time ΔT drd Indicates the margin for resuming manual driving at a position retraced from position P62a by a distance corresponding to the margin time ΔT at position P62a when position P62a is a point where resuming manual driving is essential. 2lim For example, when position P62a is a point where resuming manual driving is essential, it indicates the margin for resuming manual driving at a position retraced from position P62a by a distance corresponding to the margin time ΔT at position P62a. 2lim Indicates the margin for resuming manual driving at a position retraced from position P62a by a distance corresponding to the margin time ΔT at position P62a.

[0436] These recovery times ΔT drd And the margin times ΔT 2lim Change according to the driving road environment and the driver's state, for example, as shown at positions P62a and P62b in chart (b).

[0437] The system uses these margin times ΔT 2limAnd, recovery time ΔT drd Compare this with the grace period ΔT 2lim And, recovery time ΔT drd We determine whether and satisfy the following relationship (1). ΔT 2lim >>ΔT drd …(1)

[0438] Grace period ΔT 2lim And, recovery time ΔT drd If the relationship in equation (1) is satisfied, then while the vehicle is in motion, the probability of the driver encountering a situation requiring immediate action is low, even when using the vehicle in Level 4 autonomous driving mode. Therefore, the risk is limited, and even if the driver is unable to switch back to manual driving in time, a fallback will occur as long as MRM does not drastically increase other traffic risks.

[0439] Here, the system determines, based on information obtained from LDM and other sources, the risk of causing traffic obstruction on the road if the vehicle performs an emergency stop using MRM in the driving section. Based on this determination, if there is a possibility of such obstruction, the system searches for possible detours, determines whether pairing with a lead vehicle is possible, and assesses the availability of the remote driving support controller, execution operator, and necessary communication lines before entering that section. Depending on whether or not evasive measures are provided in conjunction with the execution of MRM based on this determination, the system provides risk selection information to the user as detour or evasive options up to the limit of what autonomous driving can provide, prompting the user to make a decision, or the system prioritizes evasive action in advance. This is a configurable selectable option, and the system completes the process according to the decision.

[0440] In other words, for a vehicle traveling a section of road, the availability of Level 4 autonomous driving depends on one of the following: whether it can continue until it selects an alternative route, until it reaches the limit where it is remotely steered by a paired lead vehicle or remote operator, or until it reaches the limit of control where the system can drive without affecting following vehicles, i.e., without causing significant social impact. By presenting this information to the driver via the information display unit 120 as information that prompts the driver to make decisions regarding risk management, this information is taken into the driver's working memory, allowing the driver to recognize the situation early when approaching a point where action is required.

[0441] If the driver fails to perform the corrective action requested by the system at these limit points, thereby violating the contract, the system will impose hierarchical penalties on the driver that are more immediate and intuitive than secondary accidents, depending on the severity of the violation. Specifically, penalties will be imposed on the driver that directly act as disadvantages, rather than probabilistic possibilities that the driver may not be aware of, such as speed limits during continued driving, mandatory pit stops in parking lots, or unpleasant odors. This allows the system to discourage the misuse of autonomous driving or encourage drivers to change their behavior to avoid actively violating the system when using autonomous driving.

[0442] As explained using chart (b), the grace period ΔT 2lim This information changes as the vehicle travels, and it may not always be possible to obtain sufficiently long-term forecasts as initially planned. Even if all the data for the LDM140 provided by the infrastructure is received at the start of the journey, it may change over time, and acquiring the LDM140 each time could potentially strain communication bandwidth.

[0443] Therefore, the information acquired in advance by the vehicle's system includes confirmed information on sections where Level 4 autonomous driving is unavailable, and the grace period ΔT for each section where the service is scheduled. 2lim This is the prediction information. Here, the grace period ΔT 2limThis information is actually acquired as more accurate, recently updated information just before approaching each section. This information can be obtained by directly requesting it from the regional management server, by obtaining it via V2V from the lead vehicle, or by obtaining it from broadcasted information.

[0444] Chart (d) explains the RRR and recovery success rate. Sections with an RRR (Request Recovery Ratio) of 100% are sections where, if a vehicle were to stop or decelerate rapidly in that section, there is a very high probability that following vehicles would need to decelerate sharply. In these sections, to ensure safety, we request that the handover be completed in advance.

[0445] Examples of sections where a high RRR (Road Ratio Restriction) can be set include certain special limited sections such as one-way bridges where even with low traffic volume there is a possibility of stopping or complete closure of both directions; special roads such as the Metropolitan Expressway that do not have passing places for vehicles; sections where general vehicles need time to assess the situation, such as tunnel exits; and roundabouts and intersections. On the other hand, in sections where traffic volume is extremely low and even if a vehicle stops on the road, the possibility of obstructing the view or driving of following vehicles is extremely small, the RRR can be set to 0%.

[0446] In the example shown in the figure, in section 65a, the RRR is set to a value lower than 100%, while in section 65b, the RRR is set to 100%. This indicates that section 65b is a section where vehicle stopping or sudden deceleration is highly likely to have a significant impact on the driving of following vehicles. On the other hand, in section 65a, where the RRR is set to a value lower than 100%, it indicates that the impact of vehicle stopping or sudden deceleration on following vehicles is smaller compared to section 65b.

[0447] Chart (e) indicates whether there is a lead vehicle or not, which is a dedicated waiting vehicle or a mutually assisting volunteer support vehicle that guides and leads autonomous driving in sections where it is difficult to pass with the vehicle's own equipment and LDM. Chart (f) indicates the availability of waiting areas, for example, if a lead vehicle is present in Chart (e), which indicates the availability of waiting areas where the lead vehicle or the vehicle can wait until the lead vehicle arrives to assist with difficult sections. Chart (g) indicates the availability of control operators, which is the availability of controllers (whether they can respond) and whether actual pilot operators can be supplied. If the itinerary is planned with a system for receiving remote support, this will affect the rate of requests for the driver to return to the vehicle in the planned section.

[0448] These complex controls may not necessarily offer benefits to the average healthy person. On the other hand, when used as a public service, such as for groups with limited manual driving abilities (the elderly, children, etc.), autonomous driving is useful in providing service networks across a wide area of ​​society, especially in situations where securing drivers for mobile vehicles is difficult due to labor shortages.

[0449] If pairing that allows the vehicle to follow a lead vehicle, or pairing that enables remote driver assistance steering at cruising speed, can be ensured, then driving at Level 4 autonomous driving will be possible. On the other hand, if the driver is to take action alone, they are required to either complete the handover to manual driving or make a pre-stop using MRM before approaching the next section where RRR will reach 100% (shown as section 66 in the diagram).

[0450] <3-3-2. Regarding ODD in Autonomous Driving Level 4> Next, the ODD in autonomous driving level 4 according to the embodiment will be described. Figure 16 is a schematic diagram illustrating the ODD in autonomous driving level 4 according to the embodiment.

[0451] In Figure 16, the direction of travel of the vehicle is shown from left to right. The upper part of the figure shows an example of a section of road 70 that is provided as static information and is drivable with Level 4 autonomous driving. The lower part of Figure 16 schematically shows an example where, within the drivable section, a section 71 occurs where the lane width is restricted due to road construction or other reasons, making Level 4 autonomous driving difficult and resulting in a section 71 with a limited lane width. In Figure 16, section 71 is between point R and point S, and in this section 71, the driver must operate the vehicle manually. From point S at the end of section 71, in a section 72 of a predetermined length, the driver can switch from manual driving to autonomous driving.

[0452] Here, we will explain the ideal use of Level 4 autonomous driving. If the vehicle's equipment meets a certain level of performance, Level 4 autonomous driving can always be used on physical road sections based on the conditions confirmed before the start of the journey. On the other hand, because there is a possibility that situations may arise where Level 4 autonomous driving is not permitted once the journey has been decided and driving has started, or while driving after starting, continuously monitoring the vehicle's condition in preparation for abnormal situations diminishes the existence and significance of autonomous driving from the user's perspective.

[0453] Therefore, it is conceivable to introduce controls that minimize the use of MRM (Medical Risk Management), which is an emergency response measure, but limits its use to conditions that keep the negative social impact below a certain level.

[0454] Furthermore, it is possible to avoid the activation of MRM (Multiple Reaction Management) altogether. In this case, information should be provided accurately and intuitively, that is, in a manner that acts with appropriate priority on working memory, so that the driver can take proactive measures and gather necessary information before MRM occurs. The information provided to the driver could include, for example, vehicle dynamics characteristic displacement, self-diagnosis and status indication of onboard equipment, information on the predictability of the road ahead (including temporary degradation of sensing performance), and advance provision of information on the feasibility of evacuation (temporary capacity to accept vehicles due to capacity fluctuations).

[0455] Furthermore, even if drivers are involved with NDRA, a mechanism is needed to cultivate an understanding of usage priorities, enabling them to proactively respond to predictable priority actions and their usage patterns.

[0456] Next, we will discuss the more practical use of Level 4 autonomous driving. In a section where a vehicle is permitted to operate at Level 4 autonomous driving, the vehicle must autonomously determine its ODD (Operational Design Direction) on the spot and operate autonomously without driver intervention, as described below.

[0457] First, the system must be able to acquire in advance the high-freshness update LDM140 for the planned route. The acquired high-freshness update LDM140 contains information indicating the return success rate (RRR) for each road section along the planned route. The system calculates the estimated delay time required to return to manual driving by the driver's anticipated handover limit point as the delay time from notification of achieving this RRR to the return.

[0458] Furthermore, alternative options are presented in case the driver does not return to manual control before the time that accounts for this delay. Then, during actual driving, if any updated information from the road ahead indicates that the driver should return to manual control, the driver must take action to return to manual control without negligence, in accordance with that information.

[0459] Here, whether the driver takes the expected recovery action in response to the system's notification falls into the realm of human behavioral psychology, which the system cannot directly perceive.

[0460] However, people do not always take proactive measures to address problems simultaneously in all aspects. In other words, it is difficult to expect people to take proactive measures unless they develop an ethical framework based on social behavioral norms.

[0461] Here, assuming that a person has developed this autonomous coping mechanism, it is assumed that they will take action in response to a notification. The benefits of performing the person's secondary task, the benefits of the primary objective of travel, the disadvantages of not returning to driving when requested, and the disadvantages of failing to obtain the necessary prior information for returning to driving are projected as future outcomes of the chosen action, and the person makes a decision on a coping action within the range where the outcome can be intuitively depicted.

[0462] Furthermore, as a result, when actually taking action, information that is important for deciding on the pre-selected coping action is temporarily stored in working memory, that is, working memory which deteriorates over time.

[0463] Furthermore, from a psychological perspective, the delay time from notification of the need to return to manual driving to the actual completion of the return depends largely on factors such as how the prior information is provided, the driver's accurate perception of the importance of the notified content, the time elapsed between notification of the need for a new handover and notification of its implementation, whether or not the driver is distracted by non-driving activities, and individual differences in the driver's ability to retain important information in their working memory.

[0464] Now, let's explain the lower part of Figure 16. For example, if new handover information is generated by the high-freshness update LDM140 (step S70), this handover information is acquired by the system via the regional LDM cloud network, as shown in step S71. The system may also receive abnormality notification signals from lead vehicles, etc. (step S72). Based on the acquired handover information or abnormality notification signals, in step S73, the system notifies the driver of the information of the involvement points (locations) involved in the handover and the importance of dealing with the handover (presentation of a provisional contract).

[0465] The driver determines the importance of the notification in response to this notification (step S74) and agrees to and responds to the provisional contract. The system detects the driver's response (step S75). As a result, the provisional contract is concluded. The driver also stores information regarding the handover in working memory based on their agreement to this provisional contract (step S76).

[0466] For example, if the driver does not perform the handover action after a predetermined time has elapsed since the initial notification, the system sends a reminder notification to the driver and determines whether or not the driver is aware of the notification (step S77). Depending on whether or not the driver is aware, the response branches as shown, for example, at point P.

[0467] Here, the timing of the driver's return to manual driving varies depending on factors such as the driver's perception of the importance of the handover. For example, if the driver's response to the provisional contract is not detected at step S75, a request to return to manual driving is issued to the driver at point (location) Q1.

[0468] The system issues a confirmation notice (step S77), and depending on whether the driver acknowledges this notice, it issues a return request at point Q2, which is further ahead than point Q1 and closer to section 71, or at point Q3, which is even further ahead and close to section 71.

[0469] When a person engages in thinking activities (brain activity) that require conscious judgment, the brain unconsciously and temporarily takes in knowledge information that forms the basis of the thought into working memory, in order of importance. As the importance of this information decreases, it gradually fades from working memory.

[0470] During this time, for example, if the driver is engrossed in NDRA, suppose the system issues a request to switch from autonomous to manual driving. If the driver does not perceive the response as urgent, or does not have a sense of near-future risk that would result from overlooking the notification and failing to perform the handover, the sense of the need to respond, that is, the information stored in working memory, will fade. In addition, information necessary for the handover, such as surrounding monitoring information and the preconditions for the vehicle's operation (vehicle dynamics characteristics), will also fade from this working memory. For example, if surrounding monitoring information is deemed important, the information necessary for making that decision will be retained in working memory.

[0471] For example, suppose a vehicle is traveling along a route permitted for Level 4 autonomous driving, and information obtained from the high-freshness update LDM140 or V2V communication from a lead vehicle indicates that it is approaching a section where manual driving is required. Furthermore, if there is a narrow road section before that point where it is difficult to pull over, then, considering the maintenance of social order, it is necessary for the handover to be successful even earlier. The delay between notification and successful handover largely depends on how alert each driver is and how much prior information they have retained to make the decision.

[0472] If the driver is aware of the need, perceives the prior information with a sense of urgency, recognizes and responds to the information in the initial notification (i.e., the system detects the recognition in the form of a response), and its importance is retained in the driver's working memory as important, the time from notification to recovery can be shortened, and a notification just before the limit can suffice.

[0473] On the other hand, if the driver does not correctly recognize the notification and the system cannot detect the driver's recognition of the notification, the system will assume that the driver has not sufficiently retained the need for a handover in their working memory and will issue a return request at an earlier time (point Q1 in the lower part of Figure 16).

[0474] However, unlike the mechanical mechanisms of a system, working memory is a conceptual understanding of the brain's function that governs a person's thinking and judgment. The upper limit of what each individual can remember varies, and some people may quickly forget the priority of even important information due to their health condition or age.

[0475] This scenario assumes that the driver receives information about such changes well before reaching the relevant point R (for example, at point Q1), and that the time it takes from receiving this notification to reaching point R is sufficiently long, such as several tens of minutes. From an ergonomic perspective, when controlled by an HCD, people basically store information in their working memory based on its importance. In this case, if the person does not perceive the information as indicating immediate importance in the near future, it will be given lower priority than information such as NDRA, which is of higher importance at that moment.

[0476] In this case, the system indicates to the driver the urgency of the handover request and the penalty for failing to fulfill the request, and detects the driver's response. By providing a thoughtful response rather than a reflexive one, the driver can ensure that the information is properly stored in their working memory. As a means of observing a thoughtful response, the observation and evaluation of intentional gestures of the driver's aroused cognitive state using pointing and calling, as described in Japanese Patent Publication No. 2019-021229 and International Publication No. 19 / 017215, can be applied. Pointing and calling can play a very large role in confirmation and recognition because the gesture provides cognitive feedback. However, simpler cognitive response methods, such as the driver answering questions presented by the system, may also be used.

[0477] Here, drivers who receive a return-to-driving notification early are expected to have an insufficient response to the prior notification and to place less importance on the need to return to driving. Based on the driver's past delay history from notification to return, the required time is calculated at the necessary handover completion limit, based on a certain success rate of returning to manual driving. In this case, the time from the return-to-driving request notification to the completion of the return can be extended, while the quality of the return action taken quickly after notification is managed and indexed. Therefore, since low-quality return actions will result in penalty points, drivers are generally expected to take early return-to-driving action.

[0478] This prior notification by the driver and the appropriate action judgment in response to the notification are only possible if the prior information that leads to the judgment, presented by the information display unit 120, etc., is presented correctly and appropriately according to the risk, and is incorporated into the judgment memory.

[0479] Now, let's explain the use cases #1 to #5 shown in the lower part of Figure 16.

[0480] • Use case #1 Use case #1 is an example where a driving plan is created assuming that the permitted route for autonomous driving level 4, updated by the quasi-static LDM, is an executable route for autonomous driving level 4, without active monitoring and control through the continuous reception of the latest data such as the high-freshness update LDM140. In this case, depending on the driver's condition, the driver may not be able to complete the return to manual driving within the permissible limit of the driver handover period for new situations that require driver intervention and for which updated information has not been obtained from the quasi-static LDM. This results in emergency response by MRM, which may, in some cases, lead to obstruction of traffic for following vehicles or induce the risk of rear-end collisions.

[0481] This use case #1 can occur in various situations, such as when information updates are provided based on a subscription-based right to receive them, when the contract expires; when there are restrictions on the terms of use in importance-based billing, or when a remote support concierge service is canceled.

[0482] • Use case #2 Use case #2 involves receiving road environment information for the vehicle's route in advance from a high-freshness update LDM140 or similar device and notifying the driver accordingly. This scenario occurs when the driver does not accurately perceive the information and a response is not detected. In this case, it is uncertain whether the importance and timing of necessary interventions are stored in the driver's working memory, and there is a risk that the handover will not be completed safely and on time. Therefore, the driver will be notified early (point Q1 in Figure 16). As a result, the time the driver can dedicate to tasks other than driving (such as NDRA) is reduced. In particular, if the handover process is not carried out quickly after the return notification, and the return quality is poor, a penalty evaluation will be reduced, which will be a disadvantage for future use.

[0483] • Use case #3 Use case #3 differs from use case #2 described above in that the driver correctly recognizes the situation at the notification stage. In use case #3, the driver continues autonomous driving at Level 4 for a long time until they reach the actual location in question (Location Q2) after receiving early notification of these new events. In this case, it is uncertain whether the importance of the handover and the timing of its occurrence are stored in the driver's working memory at the time of notification. If a certain amount of time has passed since the initial notification in step S73, the memory may have faded. In this case, the system issues a confirmation notification to the driver (step S77) and observes the driver's response, which reveals that the driver's memory retention is weaker than in use case #4 described later, and prompts for an earlier notification.

[0484] In this case, since the driver has already responded to the change in situation in step S74, there is still some residual memory, and the time it takes to reach situation awareness is shorter than in use case #2 described above. Therefore, the notification will occur at an intermediate time between use case #2 and use case 43 described later.

[0485] • Use case #4 Use case #4 is an example where the driver receives a notification, understands its importance, responds to the reconfirmation notification (step S77), and retains that memory in working memory. In this case, even if there is a time gap before reaching the point in question, the driver can periodically check the situation as they approach the point (for example, by pointing and calling out to the road ahead or the notification screen) to refresh the memory in the driver's working memory, and their risk awareness increases as they approach. As a result, the system detects the driver's state and behavior in response to the reconfirmation, enabling the driver to accurately and appropriately return to the handover point even just before the notified handover point (point Q3) based on the remaining information in their working memory that has not deteriorated. This results in a high-quality return action and earns a positive evaluation score.

[0486] • Use case #5 Use case #5 is similar to use case #4 described above up to the point of cognition when receiving the next necessary handover information in working memory during autonomous driving. However, in use case #5, so-called mind wandering occurs, and as time passes and new information is retained in working memory, the driver's consciousness becomes increasingly diverted to other thoughts, and they break away from the driving control loop. In this case, the timing of reconfirmation to return to the necessary state varies greatly from person to person and depending on their condition at the time.

[0487] In use case #5, the system utilizes observable evaluation indicators such as the individual driver's state of alertness and health status, including autonomic nervous system dysfunction, to provide feedback to the driver in a continuous and intuitive manner, using both benefits and penalties. The system repeatedly presents the driver with appropriate preceding risks, information on options to avoid those risks, and near-future projections of the impact of risks if they are not avoided. This psychologically reinforces the driver's habit of returning to manual driving early and conducting necessary progress observations, and more reliably forms in their working memory the psychological need to understand the situation before reaching the handover point.

[0488] Use case #5 conceptually illustrates an example where the system provides variable notifications to the driver regarding the scope of their involvement in the NDRA as an ODD, based on the driver's performance, evaluation of behavioral characteristics acquired unconsciously through self-learning, derived from the presentation of information to the driver, the driver's individual health condition at any given time, and their repeated use of the system. This schematically shows that the scope can change significantly depending on the driver's behavioral changes.

[0489] As described above, based on the same physical environment, which is the section where autonomous driving at Level 4 is possible, the use cases will differ depending on the accessibility of the information, the risk information contained within that information, the weighting of importance, avoidance choices, timing information, and the response to that information.

[0490] In HCD-based autonomous driving, the system provides timely updated information, along with risk information that serves as a basis for human decision-making. Drivers repeatedly experience over the long term that they may be penalized or rewarded for good responses depending on their actions. While drivers enjoy the benefits of autonomous driving, excessive reliance can lead to an intuitively descriptive risk associated with its use, which can be realized by introducing the HCD according to the embodiment of this disclosure. As a result, drivers actively participate in switching from autonomous driving to manual driving in order to take advantage of the benefits of appropriate and comfortable NDRA, and the constant provision of risk information from the system allows them to confidently check information during autonomous driving while taking advantage of the benefits of using autonomous driving. Rather than forcing drivers to check information, the HMI that encourages proactive driver behavior, created by balancing the benefits of using NDRA with penalties, is what constitutes HCD-based control.

[0491] <3-3-3. Example of operation of autonomous driving level 4 according to the embodiment> Next, an example of operation of autonomous driving level 4 according to the embodiment will be described. Figures 17A and 17B are flowcharts illustrating an example of operation of autonomous driving level 4 according to the embodiment. In Figures 17A and 17B, the symbol "G" indicates that processing will transition to the corresponding symbol in Figures 17A and 17B.

[0492] In Figure 17A, in step S200, the automatic driving control unit 10112 acquires and stores various information such as LDM initial data 80, driver personal recovery characteristics dictionary 81, RRR 82, and vehicle dynamics characteristics 83. The automatic driving control unit 10112 also acquires updated LDM information (#1, #2, ...), updated diagnostic information, and other updated information (N).

[0493] In the next step S201, the automated driving control unit 10112 identifies the initial ODD and authorizes the settings for automated driving based on the information acquired in step S200. In the next step S202, the automated driving control unit 10112 presents the driver with the itinerary and requests the driver to select a route, etc. The automated driving control unit 10112 also requests the driver to select a route for which NDRA is permitted or not. In the next step S203, the driver starts driving the vehicle.

[0494] In the next step, S204, the automated driving control unit 10112 acquires the information described in step S200 and continuously updates the information in accordance with the driving after the start of the journey. The automated driving control unit 10112 also provides a visual display of driving obstruction information for each automated driving level, including the estimated arrival time (see Figures 8, 9A to 9C).

[0495] In the next step, S205, the automated driving control unit 10112 determines whether the vehicle has entered an ODD section for which automated driving at Level 4 is authorized. If the automated driving control unit 10112 determines that the vehicle has not entered the ODD section (step S205, "No"), it returns to step S204. On the other hand, if the automated driving control unit 10112 determines in step S205 that the vehicle has entered the ODD section (step S205, "Yes"), it moves the process to step S206.

[0496] In addition, in the flowcharts shown in Figures 17A and 17B, when the vehicle leaves a section where autonomous driving is possible and enters a section where it is determined that ODD (Autonomous Driving Device) is available, the same process is repeated from step S204 (not shown).

[0497] In step S206, the automatic driving control unit 10112 determines whether or not the driver has requested a switch to the automatic driving mode. If the automatic driving control unit 10112 determines that there is no such switch request (step S206, "No"), it returns to step S204. On the other hand, if the automatic driving control unit 10112 determines that there is such a switch request (step S206, "Yes"), it moves the process to step S207.

[0498] In step S207, the automated driving control unit 10112 determines the likelihood of the driver's ability to return to manual driving. Here, the automated driving control unit 10112, for example based on the driver's personal return characteristics dictionary 81, allows the driver to use the benefits of automated driving (such as NDRA) if the driver is a good automated driving user who actively performs the return action. On the other hand, the automated driving control unit 10112 prohibits or restricts the use of the automated driving function if the driver has a tendency towards drug dependence or sleep disorders, even if they have few penalty points or penalties. This determination process in step S207 allows many drivers to learn to avoid misuse in order to obtain the benefits of NDRA during automated driving.

[0499] The permitted use of a driving route does not always guarantee an environment where autonomous driving at Level 4 can be provided, even when autonomous driving is permitted. As mentioned above, under conditions where the driver's ability to recover is guaranteed, autonomous driving or advanced assistance may be permitted up to Level 3. In that case, the determination process in step S207 can be said to be a determination of the driver's ability to use autonomous driving at Level 3. The operation of autonomous driving Level 3 according to this embodiment will be described later.

[0500] If the automatic driving control unit 10112 determines that there is a reasonable expectation that the driver will be able to return to manual driving (step S207, "OK"), it proceeds to the flowchart in Figure 17B, according to the symbol "G" in the figure. On the other hand, if the automatic driving control unit 10112 determines that there is no reasonable expectation that the driver will be able to return to manual driving (step S207, "NG"), it proceeds to step S208.

[0501] Furthermore, when using autonomous driving in combination with remote driving assistance or lead vehicle assistance, it is not mandatory to determine the driver's ability to return to manual driving; a separate determination process is performed, and this process is not included in the example shown in this embodiment.

[0502] In step S208, the automated driving control unit 10112 presents the driver with a notice of denial of permission to use automated driving, along with the reasons. Possible reasons presented to the driver in this case include driver fatigue, drowsiness, a cumulative value of excessive reliance on past history violations exceeding a certain limit, or the driver's penalty history. Drivers who wish to benefit from automated driving are expected to improve their behavior through learning for improvement or by requesting permission for a limited number of good boosts (described later) after being presented with a notice of denial from the system.

[0503] After processing in step S208, the process returns to step S204. At this point, as the vehicle continues driving, it may be driving on a road section suitable for the use of the autonomous driving function, or the initial conditions may change to conditions under which use is permitted due to an improvement in the driver's awareness of the situation. Therefore, the system loops through the processing from step S204 to step S208 to continuously monitor the status of the driver and other factors.

[0504] The explanation then moves to the flowchart in Figure 17B. Following the symbol "G," that is, when the driver selects to drive in autonomous driving mode at autonomous driving level 4, in step S220, the autonomous driving control unit 10112 updates the latest information after the start of the journey. After entering an ODD section where autonomous driving at autonomous driving level 4 is permitted, the vehicle can continue to drive using autonomous driving at autonomous driving level 4, as long as the conditions do not change. In the flowchart in Figure 17B, step S220 shows the loop process that performs monitoring in the steady state, that is, the latest information update accompanying the driving.

[0505] If any change in conditions along the route is detected in step S220, or if the ODD is approaching its end point, the automatic driving control unit 10112 proceeds to step S221.

[0506] In step S221, the automatic driving control unit 10112 determines whether or not there is an update to the information related to the latest route, which is essential for continuous automatic driving. If the automatic driving control unit 10112 determines that there is no update to the information (step S221, "No"), it proceeds to step S226.

[0507] In step S226, the automated driving control unit 10112 starts executing the scheduled safe handover sequence as the end of the automated driving section (NDRA utilization section) approaches. The handover sequence is then executed.

[0508] Here, the automated driving control unit 10112 awards points to good drivers, such as drivers who are faithful to system-return requests, consistently do not disregard return requests, and check status changes during the journey. Furthermore, the automated driving control unit 10112 allows good drivers to select the next automated driving mode without complex verification and approval procedures such as multiple authentication. Good drivers also receive priority guidance on using Level 4 automated driving. In this way, good drivers can enjoy various benefits.

[0509] Essential to achieving HCD that enables drivers to take the best possible confirmation actions is the process by which the driver acquires the "memory" necessary for these confirmation judgments and decisions that lead to appropriate actions from the system, and the "quality" of the information that the system provides to the HMI. For example, as illustrated in Figure 9C, this involves working memory with information such as when, what, and what countermeasures should be taken to produce what effects.

[0510] Furthermore, the driver's return actions during the safe handover sequence in step S226 are evaluated for quality, acquired as return action data, and stored. This return action data influences the driver's evaluation score.

[0511] On the other hand, if the automatic driving control unit 10112 determines in step S221 that there is an update to the information (step S221, "Yes"), it proceeds to steps S222a, S222b, and S223.

[0512] In step S221, the branching when it is determined that there is an update to the information (step S221, "Yes") is intended to address situations where the acceptable ODD conditions at the time of entering the ODD section are unexpected, such as sudden deterioration of weather during travel, vehicle malfunction, or cargo shifting. When such unexpected events are discovered, countermeasures are required according to the grace period available to deal with the event. The flowchart in Figure 17B shows an example of a series of processes related to these countermeasures that can be applied to the embodiment. The quality of the driver's actions in dealing with abnormal situations is also something that the driver acquires through appropriate risk judgment, and is an important element that brings about appropriate behavioral change in the driver.

[0513] In step S222a, the automated driving control unit 10112 monitors the driver's state, including response decisions to notification recognition, and based on the monitoring results, calculates an estimated delay time required to return to manual driving and updates the existing estimate. In step S222b, the automated driving control unit 10112 updates the RRR information for the road and, based on this update, calculates a revision of the permissible limits for MRM use.

[0514] In step S223, the automatic driving control unit 10112 displays the revised ODD calculation based on the information determined to be updated in step S221, the updated and acquired information in steps S222a and S222b, and self-diagnosis information. The automatic driving control unit 10112 also confirms the driver's response to this display.

[0515] In the next step S224, the automatic driving control unit 10112 predicts the time to reach the end of the ODD section related to automatic driving level 4 T. L4ODDEND And the predicted time T for the delay in returning to manual operation. MDR The system calculates the predicted arrival time T. Then, in the next step S225, the automatic driving control unit 10112 calculates the predicted arrival time T. L4ODDEND and predicted time T MDR Toga, [T L4ODDEND >T MDR Determine whether the relationship [+α] is satisfied. Note that the value α is the margin time until the points required to start the handover.

[0516] The automatic driving control unit 10112, in step S225, [T L4ODDEND >T MDR If it is determined that the relationship [+α] is not satisfied (step S225, "No"), the process proceeds to step S226.

[0517] Meanwhile, the automatic driving control unit 10112, in step S225, [T L4ODDEND >T MDR If it is determined that the relationship [+α] is satisfied (step S225, "Yes"), the process moves to the next step S227.

[0518] In step S227, the automatic driving control unit 10112 determines, based on the monitoring results of the driver's state, whether the driver has largely strayed from driving. If the automatic driving control unit 10112 determines that the driver has not largely strayed from driving (step S227, "No"), the process returns to step S220. In this case, the driver has not largely strayed from driving and is in a state where a response to notifications from the system can be expected.

[0519] On the other hand, if the automatic driving control unit 10112 determines in step S227 that the driver has largely abandoned driving (step S227, "Yes"), it proceeds to step S228.

[0520] In step S228, the automatic driving control unit 10112 determines whether there is a section ahead where a high success rate of recovery is required. If the automatic driving control unit 10112 determines that there is no such section (step S228, "No"), the process returns to step S220. In this case, it means that even if the vehicle is brought to a sudden stop by means of MRM, for example, the impact on surrounding vehicles will be extremely small, and this situation will continue for a period of margin α.

[0521] On the other hand, if the automatic driving control unit 10112 determines that such a section exists (step S228, "Yes"), it proceeds to step S229.

[0522] In step S229, the automatic driving control unit 10112 determines whether there are any alternative routes, such as intermediate refuges or waiting pool areas, located along the route to the handover start point. If the automatic driving control unit 10112 determines that there are alternative routes (step S229, "Yes"), it returns to step S220.

[0523] On the other hand, if the automatic driving control unit 10112 determines that there is no way to avoid the problem (step S229, "No"), it proceeds to step S230. In this case, if the vehicle continues to drive, the means of escape will be cut off, and there is a high possibility that the MRM will be activated, which carries the risk of obstructing the passage of following vehicles or causing a rear-end collision.

[0524] In step S230, the automated driving control unit 10112 notifies the driver of the approaching point where a handover from automated driving to manual driving is required, and detects a response from the driver acknowledging this notification. In the next step S231, the automated driving control unit 10112 determines whether there is remaining time until the handover. If the automated driving control unit 10112 determines that there is no remaining time (step S231, "No"), it proceeds to the MRM execution sequence.

[0525] On the other hand, if the automatic driving control unit 10112 determines that there is sufficient remaining time (step S231, "Yes"), it proceeds to step S232 to attempt to return to manual driving within the allowable time for the driver. In this case, a success rate of RRR or higher cannot be expected. In the next step S233, the automatic driving control unit 10112 determines whether the handover attempted in step S232 was successful before the limit was reached. If the automatic driving control unit 10112 determines that the handover was successful (step S233, "Yes"), it considers that the use of the section of automatic driving that was entered has been completed. On the other hand, if the automatic driving control unit 10112 determines that the handover failed (step S233, "No"), it proceeds to the MRM execution sequence.

[0526] Furthermore, the driver's return actions during the transition of the MRM execution sequence from step S231 or step S233, and during the transition to the end of use of one section from step S233, are acquired and stored as return action data. This return action data affects the driver's evaluation score.

[0527] In the above description, the judgment processes in Figure 17B, such as steps S225 to S229, are shown to be performed sequentially in chronological order, but this is not limited to this example. For example, each process from steps S225 to S229 may be executed in parallel in chronological order to determine whether the driver can return to normal, and if at least one of the judgments from steps S225 to S229 results in a determination that recovery is not possible, the process may be directly transferred to the MRM execution sequence. It may also be possible to skip steps.

[0528] <3-4. Examples of HCD application to autonomous driving level 3> Next, we will describe an example of applying HCD to autonomous driving level 3 according to the embodiment.

[0529] Autonomous driving level 3 is defined as a mode in which the driver is always able to respond to abnormal situations. Therefore, in order for a vehicle to operate safely in autonomous driving level 3 without disrupting public order, the driver must always be aware of the road conditions beforehand and maintain a posture and position that allows them to quickly revert to manual driving while using the vehicle in autonomous driving level 3 mode. In other words, if the driver cannot be expected to meet these conditions, it is no longer appropriate to use the vehicle in autonomous driving level 3 mode. That is to say, in this case, considering the driver's condition, it is difficult to say that driving in autonomous driving level 3 mode is possible.

[0530] In other words, the ODD of autonomous driving level 3, as defined in at least the embodiment, is the operational design domain that can be used once these conditions are met. In this case, the limit of the operational design domain corresponding to autonomous driving level 3 is the range to which the driver is expected to be able to cope with situations that may arise even if driving continues under the driver's current condition.

[0531] In other words, the available ODD (Operational Design Degree) for Level 3 autonomous driving means that if the driver is away from steering for an extended period without intervention, it can lead to driver lapse in attention and a decrease in the driver's ability to continuously gather information about the surrounding environment necessary for driving, making it difficult to take appropriate action in response to an emergency handover from autonomous to manual driving.

[0532] When a driver is responsible for driving, they continuously gather information necessary to perceive, recognize, and judge the situation. This is because decision-making requires predictability of the near future associated with the chosen action, and in order to ensure a more reliable predictability, drivers continuously gather a lot of information while driving manually. The information gathered here includes, for example, not only the behavior of the vehicle in front, but also the cargo loaded in their own vehicle, road conditions further ahead of the vehicle in front, the presence or absence of congestion, and warning signs on road signs that indicate a section of road that cannot be obtained instantaneously after notification to return to manual driving.

[0533] The information that is normally necessary for these decisions and would be acquired unconsciously during manual driving gradually fades away or stops being updated in working memory once the Level 3 autonomous driving function is used and surrounding monitoring and attention are interrupted.

[0534] Therefore, in this embodiment, the ODD is determined by restricting driving at autonomous driving level 3 depending on how long the driver's unique cognitive characteristics and state allow them to continuously perceive the surrounding environment and the vehicle's status in a manner similar to the attention state of manual driving. The vehicle's ODD design is determined by taking into account the vehicle's environmental recognition performance, prior acquisition of driving route information, the vehicle's self-diagnosis results, as well as the driver's current state and future predictions.

[0535] From an HCD perspective, it is difficult for the driver to continuously grasp surrounding information in order to immediately take over steering without actually intervening in the steering (manual driving). In autonomous driving, it is thought that the driver's thoughts will be allocated to things other than driving for a large portion of the time spent driving, and there is a high risk that working memory will fade and become less accurate, as will the information necessary for safe steering, such as the expected surrounding environment and information about the vehicle's characteristics (changes, etc.) that are essential for short-term manual steering transitions, and even the driver's awareness that they are steering.

[0536] Therefore, the ODD that defines this HCD-based autonomous driving level 3 according to this embodiment is different from existing ODDs that are determined as part of the design based on the system's performance limits, road conditions, and prior road information. In other words, based on the driver's alertness and history of continuous awareness of the surrounding environment, a section is defined in which the driver is expected to be able to respond even if a request is made to switch from autonomous driving to manual driving. Within this section, the range further defined as part of the design based on the system's performance limits, road conditions, and prior road information is defined as the autonomous driving level 3 driving area that is permissible with the driver's current ability to understand and respond to the situation.

[0537] Here, we will explain the extension of Level 3 autonomous driving sections. The envisioned scenario is that the driver, depending on their own condition, will authorize the use of Level 3 autonomous driving for short periods when recovery is expected, assess the situation, and then apply to the system for an extension, resulting in intermittent use of Level 3 autonomous driving.

[0538] Figure 18A schematically illustrates how a driver, while driving their vehicle on Road 70, extends the section of autonomous driving at Level 3. In this example, the driver repeatedly performs short-term autonomous driving at Level 3 in sections where it is conditionally permitted, by applying for extensions. That is, the driver performs short-term autonomous driving at Level 3, applies for an extension when the period ends, and then performs another short-term autonomous driving at Level 3. In the example shown in the figure, the driver repeats this process.

[0539] In this scenario, the system allows extensions simply based on the driver's button press, for example. As a result, the driver's working memory does not process the attention necessary for continuous safety checks or the background information needed to understand the situation ahead for the handover. Consequently, the system may grant extensions while the predictive information in the driver's working memory is fading, increasing the risk of the system granting extensions prematurely.

[0540] For example, the system detects driver actions such as pointing and calling out to the road ahead, along with extension requests made via button presses. This allows for a "re-contract" agreement between the system and the driver regarding situational awareness and accountability, enabling the driver to re-imprint a sense of responsibility—that is, the need to return to work—into their working memory.

[0541] When Level 3 autonomous driving is permitted, the challenges lie in whether the driver maintains an appropriate readiness posture, attitude, and alertness during use, and whether the driver fulfills their duty of continuous attention to the road ahead. From an ergonomic perspective, if there are no penalties for neglecting these fundamental duties, and only the possibility of facing risks, less cautious drivers may not necessarily adhere to these obligations.

[0542] It is already a known fact that the use of autonomous driving systems will result in these violations, and this cannot be overlooked. In other words, if these state transitions are recorded and a feedback loop is formed in which they are not overlooked as matters subject to legal penalties, drivers will intuitively experience a "virtual pain" that is recorded and stored in their consciousness as a violation state that is subject to penalties, separate from the intervals in which they fulfill their duty of care to avoid the risk of accidents themselves.

[0543] The reason there are fewer speeding violations on roads where speed limit monitoring and enforcement are conducted is that drivers project the possibility of being caught as an immediate risk, leading to a psychologically driven behavioral decision. Even with the same penalties, the presence of enforcement—a more immediate and realistic form of monitoring—increases preventative psychology, thus producing the same effect on behavioral psychology.

[0544] Recording of violations in a driver's behavior can be done silently, that is, in shadow mode without the driver's awareness. However, even in this case, from an erg...

Claims

1. An acquisition unit that acquires the status of the vehicle driver, An automatic driving control unit that controls the autonomous driving of the vehicle, Equipped with, The automatic driving control unit, Based on the driver's status acquired by the acquisition unit, the system determines the return quality, which is the quality of the driver's actions when the vehicle's operation returns from automated driving to manual driving by the driver, and quantifies the determined return quality to perform driver monitoring on the driver. Depending on the results of the driver monitoring, the driver may be given an incentive or penalty. The aforementioned penalty includes restrictions on the use of secondary tasks that the driver engages in while using the autonomous driving system. Information processing device.

2. The automatic driving control unit, The system controls the vehicle to move to an escape route depending on the vehicle's status during automatic driving, and monitors the driver to return from automatic driving to manual driving before the vehicle moves to an escape route. The information processing apparatus according to claim 1.

3. The acquisition unit is, The driver's skeletal information and facial information are acquired. The automatic driving control unit, Based on the driver's skeletal information and facial information, the driver is monitored. The information processing apparatus according to claim 1.

4. The automatic driving control unit, The skeletal information and facial information acquired by the acquisition unit are parameterized, and the restoration quality is quantified based on the parameters generated by the parameterization. The information processing apparatus according to claim 3.

5. The automatic driving control unit, Based on the skeletal information, the driver's position within the vehicle is tracked, and the recovery quality is determined based on the tracked position. The information processing apparatus according to claim 3.

6. The automatic driving control unit, Based on the skeletal information, the movement of the driver's feet is detected, and the recovery quality is determined based on the detected foot movements. The information processing apparatus according to claim 3.

7. The automatic driving control unit, Based on the skeletal information and the facial information, the position and orientation of the driver's head are detected, and the recovery quality is determined based on the detected position and orientation of the head. The information processing apparatus according to claim 3.

8. The automatic driving control unit, Based on the facial information, the behavior of the driver's eyeballs is detected, and the recovery quality is determined based on the detected eyeball behavior. The information processing apparatus according to claim 3.

9. The automatic driving control unit, Based on the behavior of the eyeballs, the driver's state of alertness is estimated, and based on the estimated state of alertness, the recovery quality is determined. The information processing apparatus according to claim 8.

10. The automatic driving control unit, Based on the skeletal information, the seating position of the driver in the driver's seat of the vehicle is detected, and the return quality is determined based on the detected seating position. The information processing apparatus according to claim 3.

11. The automatic driving control unit, Based on the aforementioned skeletal information, the state of the driver's seat—whether it is in a non-driving position or a driving position—is detected as the seated state. The information processing apparatus according to claim 10.

12. The automatic driving control unit, Based on the aforementioned skeletal information, the system detects the response action to the notification issued to the driver, and determines the recovery quality based on the detected response action. The information processing apparatus according to claim 3.

13. The aforementioned skeletal information and the aforementioned facial information each include time information. The information processing apparatus according to claim 3.

14. The automatic driving control unit, The numerically defined recovery quality is weighted according to each evaluation item used to evaluate the recovery quality, and the numerically defined and weighted recovery quality for each evaluation item is summed up to obtain an overall evaluation value of the driver's recovery quality. The information processing apparatus according to claim 1.

15. Executed by the processor, The acquisition process involves obtaining the status of the vehicle driver, An autonomous driving control process for controlling the autonomous driving of the vehicle, Includes, The aforementioned automatic driving control process is: Based on the driver's status acquired in the acquisition process, the return quality, which is the quality of the driver's actions when the vehicle's operation returns from automated driving to manual driving by the driver, is determined, and the determined return quality is quantified to perform driver monitoring of the driver. Depending on the results of the driver monitoring, the driver may be given an incentive or penalty. The aforementioned penalty includes restrictions on the use of secondary tasks that the driver engages in while using the autonomous driving system. Information processing methods.

16. On the computer, The acquisition process involves obtaining the status of the vehicle driver, An autonomous driving control process for controlling the autonomous driving of the vehicle, An information processing program for executing, The aforementioned automatic driving control process is: Based on the driver's status acquired in the acquisition process, the return quality, which is the quality of the driver's actions when the vehicle's operation returns from automated driving to manual driving by the driver, is determined, and the determined return quality is quantified to perform driver monitoring of the driver. Depending on the results of the driver monitoring, the driver may be given an incentive or penalty. The aforementioned penalty includes restrictions on the use of secondary tasks that the driver engages in while using the autonomous driving system. Information processing program.

Images