Information processing apparatus, information processing method, and information processing program

Abstract

An information processing apparatus according to the present disclosure includes an autonomous driving control unit (10112) that issues a notification to a driver of a vehicle regarding a condition for enabling the vehicle to be autonomously driven, based on a running design domain set for the vehicle. In addition, an information processing method according to the present disclosure includes an autonomous driving control step of issuing a notification to a driver of a vehicle regarding a condition for enabling the vehicle to be autonomously driven, based on a running design domain set for the vehicle, the steps being performed by a processor. Furthermore, an information processing program according to the present disclosure causes a computer to perform an autonomous driving control step of issuing a notification to a driver of a vehicle regarding a condition for enabling the vehicle to be autonomously driven, based on a running design domain set for the vehicle.

Description

Technical Field

[0001] This disclosure relates to information processing equipment, information processing methods, and information processing procedures. Background Technology

[0002] In recent years, autonomous driving technology, which uses vehicle control systems (information processing systems) to control vehicles, has seen significant development. However, even with the widespread adoption of such autonomous driving technology, many technical challenges remain until vehicles operating under autonomous control can travel at speeds comparable to those of manually controlled vehicles. Therefore, research has been conducted on limiting autonomous driving to driving ranges where, for example, good road environment maintenance and preliminary monitoring information about a constant road environment can be obtained, and on attempting to coordinate autonomous driving within these limited ranges using this preliminary monitoring information.

[0003] In this scenario, depending on conditions such as the actual maintenance conditions of the road infrastructure, a mixed zone is expected. Specifically, this includes a mixture of autonomous driving permitted zones (road sections where autonomous control via the vehicle control system is allowed) and manual driving zones (road sections where autonomous driving is not permitted). That is, there may be situations where the vehicle control system performs fully autonomous and continuous autonomous driving, and there may also be situations where driving control needs to be switched from autonomous driving as described above to manual driving where the driver performs driving controls such as steering.

[0004] Patent document 1 describes a technology related to the control switching from automatic driving to manual driving.

[0005] List of references

[0006] Patent documents

[0007] Patent Document 1: JP 2018-180594 A Summary of the Invention

[0008] Technical issues

[0009] If the switch from automatic to manual driving is performed when the driver is not adequately prepared for manual driving, it could lead to social hazards such as causing accidents to follow vehicles.

[0010] This disclosure aims to provide an information processing device, information processing method, and information processing program capable of appropriately performing the switch from autonomous driving to manual control driving.

[0011] Technical solutions to the problem

[0012] To address the aforementioned problems, an information processing device according to one aspect of this disclosure has an automatic driving control unit that, based on an operation design domain set for the vehicle, notifies the driver of the conditions that enable the vehicle to drive automatically. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating a construction example of the schematic functions of a vehicle control system applicable to embodiments of the present disclosure.

[0014] Figure 2 This is a block diagram illustrating the construction of an example information processing device including an autonomous driving control unit suitable for an embodiment.

[0015] Figure 3 This is a diagram illustrating the situation from the user's perspective, where the various levels of autonomous driving defined in SAE are observed as usage states.

[0016] Figure 4 This is a schematic diagram used to illustrate the application of Level 3 autonomous driving.

[0017] Figure 5 This is a flowchart that schematically illustrates an example of the switching process from autonomous driving to manual control driving according to existing technology.

[0018] Figure 6 This is a flowchart schematically illustrating an example of the switching process from autonomous driving to manual control driving according to an embodiment.

[0019] Figure 7A This is a flowchart illustrating an example of the process of switching from driving route setting to autonomous driving mode according to an embodiment.

[0020] Figure 7B This is a flowchart illustrating an example of the processing flow in an autonomous driving mode according to an embodiment.

[0021] Figure 7C This is a flowchart illustrating an example of a response to an event occurring during autonomous driving at Level 4, according to an embodiment.

[0022] Figure 8 This is a schematic diagram illustrating an example of an overhead view of a driving route applicable to an embodiment.

[0023] Figure 9A This is a schematic diagram illustrating an example of a color-coded top-down view of various parts according to an embodiment.

[0024] Figure 9B This is a schematic diagram showing an example of a ring-shaped structure viewed from above according to an embodiment.

[0025] Figure 9C This is a schematic diagram illustrating an example of an overhead display including road information according to an embodiment.

[0026] Figure 10 This is a functional block diagram illustrating an example of the control function of the HCD in the autonomous driving control unit according to an embodiment.

[0027] Figure 11 This is a functional block diagram illustrating an example of the function of the driver recovery delay evaluation unit according to an embodiment.

[0028] Figure 12 This is a schematic diagram illustrating the high-precision LDM update applicable to the embodiments.

[0029] Figure 13 This is a schematic diagram illustrating the acquisition of information by a remote auxiliary control I / F applicable to the embodiments.

[0030] Figure 14 This is a functional block diagram illustrating an example of how a driver behavior change enables the level estimation unit to function according to an embodiment.

[0031] Figure 15 This is a schematic diagram illustrating the basic structure of Level 4 autonomous driving applicable to the embodiments.

[0032] Figure 16 This is a schematic diagram illustrating an ODD at Level 4 of autonomous driving according to an embodiment.

[0033] Figure 17A This is a flowchart illustrating an example of an applicable example of autonomous driving level 4 according to an embodiment.

[0034] Figure 17B This is a flowchart illustrating an example of an applicable example of autonomous driving level 4 according to an embodiment.

[0035] Figure 18A This is a diagram schematically illustrating the state of a driver extending the autonomous driving level 3 interval while driving their own vehicle on road 7.

[0036] Figure 18B This is a flowchart illustrating an example of processing within the available range of conditional autonomous driving level 3 according to an embodiment.

[0037] Figure 19A This is a flowchart illustrating an example of an autonomous driving processing flow applicable to an embodiment, focusing on the ODD.

[0038] Figure 19B This is a flowchart illustrating in more detail an example of the ODD setup process applicable to the embodiments.

[0039] Figure 20 This is a schematic diagram used to illustrate more specifically an example of setting an ODD range applicable to the embodiments.

[0040] Figure 21 This is a functional block diagram illustrating an example of the functionality of a driver behavior evaluation unit applicable to a DMS according to an embodiment.

[0041] Figure 22 This is a functional block diagram illustrating an example of the functionality of the learning unit in the offline setup applicable to the embodiments.

[0042] Figure 23A This is a schematic diagram illustrating the generation of a 3D head model applicable to the embodiments.

[0043] Figure 23B This is a schematic diagram illustrating the generation of a body model applicable to the embodiments.

[0044] Figure 24 This is a schematic diagram illustrating a method for determining a driver's alertness applicable to an embodiment.

[0045] Figure 25 This is a flowchart illustrating an example of a process for evaluating the quality of behavior according to an embodiment. Detailed Implementation

[0046] Embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. In each of the following embodiments, the same parts are indicated by the same reference numerals, and therefore repeated descriptions thereof will be omitted.

[0047] In the following description, embodiments of the present disclosure will be described in the following order.

[0048] 0. Overview of this Disclosure

[0049] 1. Construction applicable to embodiments of this disclosure

[0050] 2. Overview of SAE-based Automated Driving Levels

[0051] 3. Embodiments according to this disclosure

[0052] 3-1. Overview of the Implementation Examples

[0053] 3-2. HCD (Human Centered Design) according to the embodiment

[0054] 3-2-1. Overview of HCD according to the embodiment

[0055] 3-2-2. Advantages of HCD in Autonomous Driving

[0056] 3-2-2-1. Over-reliance

[0057] 3-2-2-2.HCD

[0058] 3-2-2-3. Benefits for drivers

[0059] 3-2-2-4. Driver's working memory and thinking patterns during driving

[0060] 3-2-2-5. The "Transaction" Between the System and the Driver

[0061] 3-2-2-6. Application of Level 4 Automated Driving

[0062] 3-2-2-7. The effect of using HCD

[0063] 3-2-3. Specific Examples of HCD According to the Embodiments

[0064] 3-2-3-1. Example of autonomous driving using HCD according to the embodiment 3-2-3-2. Evaluation of driver recovery behavior

[0065] 3-2-3-3. Aerial view of the driving route applicable to the embodiment

[0066] 3-2-4. Example of HCD control structure according to the embodiment

[0067] 3-3. Automated driving level 4 applicable to the embodiments

[0068] 3-3-1. Basic Structure

[0069] 3-3-2. ODD for Level 4 Automated Driving

[0070] 3-3-3. Applicable example of autonomous driving level 4 according to the embodiments

[0071] 3-4. Examples of applying HCD to Level 3 autonomous driving

[0072] 3-5. Determinants of ODD

[0073] 3-6. DMS (Driver Monitoring System) according to the embodiment

[0074] 3-6-1. Overview of the DMS according to the embodiment

[0075] 3-6-2. A more detailed description of the DMS according to the embodiment

[0076] 3-6-3. Quantification of Quality of Action (QoA) according to the embodiment

[0077] 3-6-4. Applicable Construction of DMS According to Embodiments

[0078] 3-6-5. Specific Examples of Behavioral Quality Assessment According to the Embodiments

[0079] 3-6-6. Summary of DMS according to the embodiments

[0080] <<0. Overview of this Disclosure>>

[0081] This disclosure relates to the processing performed when switching from an automated driving system (which is a system that allows the vehicle to drive autonomously) to manual driving (where the driver performs operations such as steering the vehicle) in the event of a switch from automated driving to manual driving (where the driver performs operations such as steering the vehicle). More specifically, this disclosure provides a mechanism that naturally assists the driver in self-learning through repeated use of automated driving, thereby enabling a smooth handover of driving from the vehicle to the driver.

[0082] In other words, autonomous driving functions, initially acquired only through theoretical descriptions or materials, are physically unknown. This can cause drivers to feel anxious or even skeptical about systems they have no practical experience with. Meanwhile, in normal behavioral decisions, when people take a risky action to obtain something, they make selective decisions to balance that risk.

[0083] Therefore, even drivers who have started using autonomous driving still feel uneasy while driving in autonomous mode. To mitigate this sense of risk, drivers maintain a certain level of focus even while using autonomous driving functions. Thus, for drivers, some of the awareness required when using autonomous driving functions is maintained rather than completely disappears.

[0084] Here, as the event handling performance of autonomous driving systems gradually improves and the anxiety about risks is reduced through repeated use of autonomous driving, users of autonomous driving will alleviate their unease about over-reliance on it. In particular, advanced autonomous driving features to be introduced in recent years are required to include functions such as: under satisfactory conditions, assuming the driver is required to switch from autonomous driving to manual control, the ability to take measures to avoid accidents even if the driver has difficulty returning to manual control; and the ability to minimize the impact of accidents even if they are unavoidable.

[0085] With the realization of such advanced autonomous driving capabilities, drivers' anxiety about the risks of over-reliance on autonomous driving gradually diminishes. Consequently, drivers may be unprepared for requests to take over manual control. Therefore, in situations where drivers find it difficult to take necessary actions within time constraints, measures such as emergency braking of autonomous driving have been studied as safety measures to minimize the impact of risks.

[0086] However, the frequent deceleration or emergency stopping of vehicles in every road environment can lead to situations that impede the movement of other vehicles, such as sudden deceleration of following vehicles, stopping in poor visibility conditions, and congestion on narrow roads (e.g., bridges with narrow driving zones). These effects occur in the form of impacts that the driver cannot directly see. The impact of these situations may reduce the efficiency of main road environments for social activities. In other words, existing autonomous driving control mechanisms do not include a means to reflect these social impacts as a sense of risk in the driver's behavioral judgments when using autonomous driving functions.

[0087] The purpose of this disclosure is to provide a mechanism that enables drivers to incorporate the aforementioned social impacts as a sense of risk into their behavioral judgments when using autonomous driving functions.

[0088] <<1. Construction Applicable to Embodiments of this Disclosure>>

[0089] First, the construction of the various embodiments applicable to this disclosure will be described.

[0090] Figure 1 This is a block diagram illustrating a schematic example of the construction of a vehicle control system 10100, which is an example of a mobile body control system applicable to embodiments of this disclosure.

[0091] In the following text, when distinguishing a vehicle having the vehicle control system 10100 from other vehicles, the vehicle is referred to as this vehicle.

[0092] The vehicle control system 10100 includes an input unit 10101, a data acquisition unit 10102, a communication unit 10103, an on-board device 10104, an output control unit 10105, an output unit 10106, a drive system control unit 10107, a drive system 10108, a body system control unit 10109, a body system 10110, a storage unit 10111, and an automatic driving control unit 10112.

[0093] The input unit 10101, data acquisition unit 10102, communication unit 10103, output control unit 10105, drive system control unit 10107, body system control unit 10109, storage unit 10111, and automatic driving control unit 10112 are interconnected via communication network 10121. Communication network 10121 may include, for example, an in-vehicle communication network and bus conforming to any standard such as CAN (controller area network), LIN (local interconnect network), LAN (local area network), or FlexRay (registered trademark). Note that the units of the vehicle control system 10100 may also be directly connected without using communication network 10121.

[0094] Note that in the following text, when the various units of the vehicle control system 10100 communicate via the communication network 10121, the description of the communication network 10121 will be omitted. For example, when the input unit 10101 and the automatic driving control unit 10112 communicate with each other via the communication network 10121, it will be simply described as the input unit 10101 and the automatic driving control unit 10112 communicating with each other.

[0095] Input unit 10101 includes devices for passengers to input various data and instructions. For example, input unit 10101 includes operating devices such as touch panels, buttons, switches, and levers, as well as operating devices such as microphones and cameras that can input data via methods other than manual operation, such as voice and gestures. Furthermore, input unit 10101 may also be a remote control device utilizing infrared or other radio waves, or an external connection device such as a mobile device or wearable device compatible with the operation of vehicle control system 10100. Input unit 10101 generates input signals based on data or instructions input by the passenger (e.g., driver) and provides the generated input signals to various units of vehicle control system 10100.

[0096] The data acquisition unit 10102 includes devices such as various sensors, which acquire data for processing by the vehicle control system 10100 and provide the acquired data to each unit of the vehicle control system 10100.

[0097] For example, the data acquisition unit 10102 includes various sensors for detecting the state of the vehicle, etc. Specifically, for example, the data acquisition unit 10102 includes: a gyroscope sensor; an accelerometer sensor; an inertial measurement unit (IMU) and sensors for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine speed, the motor speed, or the wheel speed, etc.

[0098] Furthermore, for example, the data acquisition unit 10102 includes various sensors for detecting information outside the vehicle. Specifically, for example, the data acquisition unit 10102 includes imaging devices such as a ToF (time of flight) camera, a stereo camera, a single-lens reflex camera, an infrared camera, and other cameras. Additionally, for example, the data acquisition unit 10102 includes environmental sensors for detecting weather, etc., and ambient information detection sensors for detecting objects around the vehicle. Environmental sensors include, for example, rain sensors, fog sensors, sunlight sensors, and snow sensors. Examples of ambient information detection sensors include ultrasonic sensors, radar, LiDAR (light detection and ranging or laser imaging detection and ranging), and sonar.

[0099] Furthermore, for example, the data acquisition unit 10102 includes various sensors for detecting the current position of the vehicle. Specifically, for example, the data acquisition unit 10102 includes devices such as a GNSS receiver for receiving GNSS signals from GNSS (global navigation satellite system) satellites.

[0100] Furthermore, the data acquisition unit 10102 includes various sensors, for example, for detecting information inside the vehicle. Specifically, for example, the data acquisition unit 10102 includes devices such as a camera device for photographing the driver, a biosensor for detecting the driver's biometric information, and a microphone for collecting sounds inside the vehicle. In this case, the camera device is preferably capable of photographing the driver's head, upper body, waist, lower body, and feet. Multiple camera devices can also be provided to photograph various parts of the body. The biosensor is, for example, disposed on the seat surface or steering wheel, and detects the biometric information of the occupant sitting in the seat or the driver holding the steering wheel.

[0101] The communication unit 10103 communicates with the vehicle-mounted device 10104, various external devices, servers, and base stations, transmitting data supplied from each unit of the vehicle control system 10100 and providing received data to each unit of the vehicle control system 10100. There are no particular restrictions on the communication protocols supported by the communication unit 10103, and the communication unit 10103 can support multiple types of communication protocols.

[0102] For example, the communication unit 10103 can wirelessly communicate with the vehicle-mounted device 10104 via wireless LAN, Bluetooth (registered trademark), NFC (near field communication), or wireless USB (WUSB). Alternatively, for example, the communication unit 10103 can wiredly communicate with the vehicle-mounted device 10104 via a connection terminal (if necessary, with a cable) such as USB (universal serial bus), HDMI (high-definition multimedia interface) (registered trademark), or MHL (mobile high-definition link).

[0103] Furthermore, for example, communication unit 10103 communicates with devices (e.g., application servers or control servers) residing on external networks (e.g., the Internet, cloud networks, or carrier-specific networks) via a base station or access point. Furthermore, for example, communication unit 10103 uses P2P (peer-to-peer) technology to communicate with terminals located near the vehicle (e.g., terminals in pedestrians or shops, or MTC (machine-type communication) terminals). Furthermore, for example, communication unit 10103 performs V2X communication, such as vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication. Furthermore, for example, communication unit 10103 includes a beacon receiver and receives radio waves or electromagnetic waves transmitted from radio stations or the like installed on the road to obtain information such as current location, congestion, traffic restrictions, and estimated travel time.

[0104] The in-vehicle device 10104 includes devices such as mobile or wearable devices owned by passengers, information devices carried or installed in the vehicle, and navigation devices for searching routes to a specific destination.

[0105] Output control unit 10105 controls the output of various information to the occupants of the vehicle or to the exterior of the vehicle. For example, output control unit 10105 generates an output signal including at least one of visual information (e.g., image data) or auditory information (e.g., sound data) and provides the output signal to output unit 10106, thereby controlling the output of visual and auditory information from output unit 10106. Specifically, for example, output control unit 10105 combines image data captured by different camera devices of data acquisition unit 10102 to generate images such as bird's-eye view images or panoramic images, and provides an output signal including the generated images to output unit 10106. Furthermore, for example, output control unit 10105 generates sound data including warning sounds or warning messages for dangers such as collisions, contact, or entry into dangerous areas, and provides an output signal including the generated sound data to output unit 10106.

[0106] Output unit 10106 includes devices capable of outputting visual or auditory information to occupants of the vehicle or to the outside of the vehicle. For example, output unit 10106 includes devices such as display devices, dashboards, head-up displays (HUDs), audio speakers, headphones, wearable devices (e.g., glasses-type displays worn by occupants), projectors, and lights. The display devices included in output unit 10106 can be devices that display visual information within the driver's field of vision, in addition to ordinary displays, such as head-up displays, projection displays, or devices with AR (augmented reality) display capabilities.

[0107] The drive system control unit 10107 controls the drive system 10108 by generating various control signals and providing them to the drive system 10108. Additionally, the drive system control unit 10107 provides control signals to other units besides the drive system 10108 as needed, and notifies the drive system 10108 of its control status, etc.

[0108] The drive system 10108 includes various devices related to the drive system of the vehicle. For example, the drive system 10108 includes the following devices: a drive force generating device such as an internal combustion engine or a drive motor for generating drive force; a drive force transmission mechanism for transmitting drive force to the wheels; a steering mechanism for adjusting the steering angle; a braking device for generating braking force; an anti-lock braking system (ABS); electronic stability control (ESC); and an electric power steering device.

[0109] The body system control unit 10109 controls the body system 10110 by generating various control signals and providing them to the body system 10110. Additionally, the body system control unit 10109 provides control signals to other units besides the body system 10110 as needed, and notifies the body system 10110 of its control status, etc.

[0110] The body system 10110 includes various devices of the body system installed on the vehicle body. For example, the body system 10110 includes devices such as keyless entry system, smart key system, power window device, power seat, steering wheel, air conditioning and various lights (e.g., headlights, taillights, brake lights, turn signals and fog lights).

[0111] Storage unit 10111 includes a storage medium for storing data and a controller for controlling the reading of data from and writing of data to the storage medium. The storage medium included in storage unit 10111 can be implemented using one or more devices such as magnetic storage devices, semiconductor storage devices, optical storage devices, and magneto-optical storage devices, including ROM (read-only memory), RAM (random access memory), and HDD (hard disk drive). Storage unit 10111 stores various programs and data used by the various units of vehicle control system 10100. For example, storage unit 10111 stores map data, examples of which include three-dimensional high-precision maps (e.g., dynamic maps), global maps with lower precision than high-precision maps but covering a wide area, and local maps including information about the surroundings of the vehicle.

[0112] Incidentally, one of the mappings stored in storage unit 10111 is a Local Dynamic Map (LDM). Conceptually, based on the rate of data change, the LDM comprises four levels: static data (Type 1), transient static data (Type 2), transient dynamic data (Type 3), and dynamic data (Type 4).

[0113] exist Figure 1In this system, the autonomous driving control unit 10112 includes a detection unit 10131, a self-position estimation unit 10132, a situation analysis unit 10133, a planning unit 10134, and an operation control unit 10135. The detection unit 10131, self-position estimation unit 10132, situation analysis unit 10133, planning unit 10134, and operation control unit 10135 are implemented through predetermined programs running on a CPU (central processing unit). However, the implementation is not limited to this; some or all of the detection unit 10131, self-position estimation unit 10132, situation analysis unit 10133, planning unit 10134, and operation control unit 10135 can be implemented through hardware circuits that cooperate with each other.

[0114] The autonomous driving control unit 10112 performs controls related to autonomous driving, such as autonomous driving or driver assistance. Specifically, the autonomous driving control unit 10112 performs cooperative control for the purpose of realizing advanced driver assistance system (ADAS) functions, such as collision avoidance or mitigation, following based on inter-vehicle distance, maintaining vehicle speed, collision warning, and lane departure warning. Furthermore, for example, the autonomous driving control unit 10112 performs cooperative control for the purpose of autonomous driving, enabling the vehicle to drive itself without driver intervention.

[0115] The detection unit 10131 detects various types of information required for controlling autonomous driving. The detection unit 10131 includes an external information detection unit 10141, an internal information detection unit 10141, and a vehicle status detection unit 10143.

[0116] The external information detection unit 10141 detects information outside the vehicle based on data or signals from various units of the vehicle control system 10100. For example, the external information detection unit 10141 performs detection processing, recognition processing, and tracking processing of objects around the vehicle, as well as distance detection processing to those objects. Examples of detected objects include vehicles, pedestrians, obstacles, buildings, roads, traffic lights, traffic signs, and road signs. Furthermore, the external information detection unit 10141 performs detection processing of the environment surrounding the vehicle. The detected environment includes weather, temperature, humidity, brightness, and road surface conditions.

[0117] The external information detection unit 10141 provides data representing the results of the detection processing to the following units: for example, the self-position estimation unit 10132; units in the situation analysis unit 10133, such as the map analysis unit 10151, the traffic rule recognition unit 10152 and the situation recognition unit 10153; and the emergency avoidance unit 10171 in the operation control unit 10135.

[0118] The in-vehicle information detection unit 10141 performs in-vehicle information detection processing based on data or signals from various units of the vehicle control system 10100. For example, the in-vehicle information detection unit 10141 performs processes such as driver authentication and identification processing, driver state detection processing, occupant detection processing, and in-vehicle environment detection processing. Examples of detected driver states include physical condition, alertness, concentration, fatigue level, and gaze direction. Examples of detected in-vehicle environment conditions include temperature, humidity, brightness, and odor. The in-vehicle information detection unit 10141 provides data representing the results of the detection processing to the situation recognition unit 10153 of the situation analysis unit 10133 and the emergency avoidance unit 10171 of the operation control unit 10135, etc.

[0119] The vehicle status detection unit 10143 performs vehicle status detection processing based on data or signals from various units of the vehicle control system 10100. Examples of the detected vehicle status include speed, acceleration, steering angle, presence and details of any abnormalities, driving operation status, position and tilt of the power seat, door lock status, and the status of other on-board equipment. The vehicle status detection unit 10143 provides data representing the results of the detection processing to the status recognition unit 10153 of the status analysis unit 10133 and the emergency avoidance unit 10171 of the operation control unit 10135, etc.

[0120] The self-position estimation unit 10132 performs estimation processing of the vehicle's position and attitude based on data or signals from various units of the vehicle control system 10100 (e.g., the external information detection unit 10141 and the situation recognition unit 10153 of the situation analysis unit 10133). Furthermore, the self-position estimation unit 10132 generates a local map (hereinafter referred to as the self-position estimation map) as needed for estimating its own position. The self-position estimation map is, for example, a high-precision map using techniques such as simultaneous localization and mapping (SLAM). The self-position estimation unit 10132 provides data representing the results of the estimation processing to units in the situation analysis unit 10133, such as the map analysis unit 10151, the traffic rule recognition unit 10152, and the situation recognition unit 10153. In addition, the self-position estimation unit 10132 stores the self-position estimation map in the storage unit 10111.

[0121] The situation analysis unit 10133 performs the analysis of the vehicle and its surrounding conditions. The situation analysis unit 10133 includes a map analysis unit 10151, a traffic rule recognition unit 10152, a situation recognition unit 10153, and a situation prediction unit 10154.

[0122] Map analysis unit 10151 uses data or signals from various units of vehicle control system 10100 (e.g., self-position estimation unit 10132 and external information detection unit 10141) as needed to perform analysis and processing of various maps stored in storage unit 10111, and constructs a map including information required for autonomous driving processing. Map analysis unit 10151 provides the constructed map to traffic rule recognition unit 10152, situation recognition unit 10153, situation prediction unit 10154, and units in planning unit 10134 (e.g., route planning unit 10161, action planning unit 10162, and operation planning unit 10163).

[0123] The traffic rule recognition unit 10152 performs traffic rule recognition processing around the vehicle based on data or signals from various units of the vehicle control system 10100 (e.g., self-position estimation unit 10132, external information detection unit 10141, and map analysis unit 10151). This recognition processing identifies information such as the location and status of signals around the vehicle, details of traffic rules around the vehicle, and lanes the vehicle is permitted to travel in. The traffic rule recognition unit 10152 provides data representing the results of the recognition processing to the situation prediction unit 10154, etc.

[0124] The situation recognition unit 10153 performs situation recognition processing related to the vehicle based on data or signals from various units of the vehicle control system 10100 (e.g., self-position estimation unit 10132, external information detection unit 10141, internal information detection unit 10141, vehicle state detection unit 10143, and map analysis unit 10151). For example, the situation recognition unit 10153 performs situation recognition processing on conditions such as the condition of the vehicle itself, the condition around the vehicle, and the condition of the driver. Additionally, the situation recognition unit 10153 generates a local map (hereinafter referred to as a situation recognition map) as needed for recognizing the condition around the vehicle. The situation recognition map is, for example, an occupancy grid map.

[0125] Examples of the vehicle's condition to be identified include its position, attitude, and motion (e.g., speed, acceleration, and direction of motion), as well as the presence and details of any abnormalities. Examples of the vehicle's surroundings to be identified include the type and position of stationary objects, the type, position, and motion (e.g., speed, acceleration, and direction of motion) of moving objects, the composition and condition of the surrounding roads, and conditions such as the surrounding weather, temperature, humidity, and brightness. Examples of the driver's condition to be identified include the driver's physical condition, level of alertness, level of concentration, level of fatigue, eye movement, and driving actions.

[0126] The situation identification unit 10153 provides data representing the result of the identification process (including a situation identification map if necessary) to its own position estimation unit 10132 and situation prediction unit 10154, etc. In addition, the situation identification unit 10153 stores the situation identification map in the storage unit 10111.

[0127] The situation prediction unit 10154 performs predictive processing on situations related to the vehicle based on data or signals from various units of the vehicle control system 10100 (e.g., map analysis unit 10151, traffic rule recognition unit 10152, and situation recognition unit 10153). For example, the situation prediction unit 10154 performs predictive processing on situations such as the situation of the vehicle itself, the situation around the vehicle, and the situation of the driver.

[0128] Examples of conditions to be predicted for the vehicle include the vehicle's behavior, the occurrence of anomalies, and the drivable distance. Examples of conditions to be predicted for the vehicle's surroundings include the behavior of moving objects around the vehicle, changes in signal status, and changes in the environment (e.g., weather). Examples of conditions to be predicted for the driver include the driver's behavior and physical condition.

[0129] The condition prediction unit 10154 provides data representing the result of the prediction process, along with data from the traffic rule recognition unit 10152 and the condition recognition unit 10153, to the units of the planning unit 10134 (e.g., route planning unit 10161, motion planning unit 10162, and operation planning unit 10163, etc.).

[0130] The planning unit 10134 includes a route planning unit 10161, a motion planning unit 10162, and an operation planning unit 10163.

[0131] The route planning unit 10161 plans a route (driving route) to the destination based on data or signals from various units of the vehicle control system 10100 (e.g., map analysis unit 10151 and situation prediction unit 10154). For example, the route planning unit 10161 sets a route from the current location to the specified destination based on a global map. Additionally, the route planning unit 10161 appropriately modifies the route based on conditions such as traffic congestion, traffic accidents, traffic restrictions, and construction, as well as conditions such as the driver's physical condition. The route planning unit 10161 provides data representing the planned route to the motion planning unit 10162, etc.

[0132] The motion planning unit 10162 plans the vehicle's actions based on data or signals from various units of the vehicle control system 10100 (e.g., map analysis unit 10151 and situation prediction unit 10154) to ensure safe travel on the route planned by the route planning unit 10161 within a planned timeframe. For example, the motion planning unit 10162 executes plans including starting, stopping, driving direction (e.g., forward, reverse, left turn, right turn, and steering), driving lane, driving speed, and overtaking. The motion planning unit 10162 provides data representing the planned actions of the vehicle to the operation planning unit 10163, etc.

[0133] The operation planning unit 10163 plans the vehicle's operations based on data or signals from various units of the vehicle control system 10100 (e.g., map analysis unit 10151 and situation prediction unit 10154) to implement the actions planned by the action planning unit 10162. For example, the operation planning unit 10163 plans acceleration, deceleration, and driving trajectory. The operation planning unit 10163 provides data representing the planned vehicle operations to various units of the operation control unit 10135 (e.g., acceleration / deceleration control unit 10172 and steering control unit 10173).

[0134] The operation control unit 10135 controls the operation of the vehicle. The operation control unit 10135 includes an emergency avoidance unit 10171, an acceleration / deceleration control unit 10172, and a steering control unit 10173.

[0135] Based on the detection results of the external information detection unit 10141, the internal information detection unit 10142, and the vehicle status detection unit 10143, the emergency avoidance unit 10171 performs emergency response procedures for detected emergencies such as collisions, contact incidents, entry into dangerous areas, driver abnormalities, or vehicle abnormalities. When an emergency is detected, the emergency avoidance unit 10171 plans the vehicle's actions to avoid emergencies such as emergency braking or sharp turns. The emergency avoidance unit 10171 provides data indicating the planned vehicle actions to the acceleration / deceleration control unit 10172 and the steering control unit 10173, etc.

[0136] Acceleration / deceleration control unit 10172 performs acceleration / deceleration control to achieve the vehicle's operation planned by operation planning unit 10163 or emergency avoidance unit 10171. For example, acceleration / deceleration control unit 10172 calculates control target values ​​for the drive force generating device or braking device to achieve planned acceleration, deceleration, or emergency braking, and provides control commands representing the calculated control target values ​​to drive system control unit 10107.

[0137] The steering control unit 10173 performs steering control to achieve the vehicle's operation planned by the operation planning unit 10163 or the emergency avoidance unit 10171. For example, the steering control unit 10173 calculates a control target value for the steering mechanism to achieve the driving trajectory or sharp turn planned by the operation planning unit 10163 or the emergency avoidance unit 10171, and provides a control command representing the calculated control target value to the drive system control unit 10107.

[0138] Figure 2 It shows including Figure 1 A block diagram illustrating the construction of an example information processing device for an autonomous driving control unit 10112.

[0139] exist Figure 2 In the information processing device 10000, there are CPU 10010, ROM (read only memory) 10011, RAM (random access memory) 10012, storage device 10013, input / output I / F 10014 and control I / F 10015, which are communicatively connected to each other via bus 10020.

[0140] Storage device 10013 is a non-volatile data storage medium, and suitable examples include hard disk drives and flash memory. CPU 10010 controls the operation of information processing device 10000 by using RAM 10012 as working memory, based on programs stored in storage device 10013 and ROM 10011.

[0141] Input / output I / F 10014 is an interface for controlling the input / output of data to information processing device 10000. Control I / F 10015 is an interface between information processing device 10000 and the controlled device. For example, input / output I / F 10014 and control I / F 10015 are connected to communication network 10121.

[0142] For example, by having the CPU 10010 execute the information processing program according to the embodiment, the aforementioned units, namely the detection unit 10131, the self-position estimation unit 10132, the situation analysis unit 10133, the planning unit 10134, and the operation control unit 10135, are respectively formed as modules in the main storage area of, for example, RAM 10012.

[0143] For example, the information processing program is pre-installed in the information processing device 10000 when it is installed in and transported in a vehicle. The installation time is not limited to this, and the information processing program can also be installed in the information processing device 10000 after it has been installed in and transported. Furthermore, the information processing program can also be provided via the input / output I / F 10014 through the communication unit 10103 to communicate with external devices (servers, etc.) and is installed in the information processing device 10000.

[0144] <<2. Overview of SAE-based Automated Driving Levels>>

[0145] Next, we will describe the autonomous driving of the vehicle applied to the examples. For autonomous driving of vehicles, the SAE (Society of Automotive Engineers) defines levels of autonomous driving. Table 1 shows the levels of autonomous driving defined by SAE.

[0146] Table 1

[0147]

[0148] The following explanation will provide a basic reference to the SAE-defined levels of autonomous driving shown in Table 1. However, the survey of autonomous driving levels shown in Table 1 did not thoroughly examine the problems and effectiveness of autonomous driving technology when it becomes widely available. Therefore, in the following explanation, some points need not be explained according to the SAE definition, taking into account these issues.

[0149] As shown in Table 1, according to SAE, the levels of automated driving requiring human intervention are divided into five levels, for example, from Level 0 to Level 4. Incidentally, although SAE also defines Level 5, which assumes fully automated driving without human intervention, this disclosure considers Level 5 to be outside the scope because the driver is not involved in steering.

[0150] Level 0 of autonomous driving is manual control driving (direct driving by the driver with the steering wheel) without the assistance of a vehicle control system, in which the driver performs all driving tasks and constantly performs monitoring related to safe driving (e.g., avoidance maneuvers).

[0151] Level 1 autonomous driving is manual control driving (direct driving with the steering wheel) that includes driver assistance systems such as automatic braking, ACC (adaptive cruise control), and LKAS (lane keeping assistant system). In this mode, the driver performs all driving tasks except for the single function being assisted and also performs monitoring related to safe driving.

[0152] Level 2 Automated Driving, also known as "partial automation or condition-specific automated driving," involves the vehicle control system performing sub-tasks related to vehicle control in the longitudinal (forward / backward) and lateral (left / right) directions under specific conditions. For example, in Level 2, the vehicle control system coordinates steering and acceleration / deceleration (e.g., the coordinated operation of ACC and LKAS). Nevertheless, even in Level 2, the driver remains the primary agent for performing driving tasks, and the driver is also the primary agent for monitoring safe driving.

[0153] Level 3 autonomous driving, also known as "conditional autonomous driving," is a system where the vehicle control system is capable of performing all driving tasks within a limited area. In Level 3, the vehicle control system is the primary agent for performing driving tasks, and it is also the primary agent for monitoring safe driving.

[0154] Level 3 of automated driving as defined by SAE does not specifically define which secondary tasks a driver is actually capable of performing. Note that "secondary tasks" refer to actions performed by the driver during driving that are not related to driving, also known as non-driving related activities (NDRA).

[0155] More specifically, during Level 3 autonomous driving, the driver can perform secondary tasks such as actions and movements other than steering, such as operating a mobile device, conducting conference calls, watching videos, playing games, thinking, and conversing with other passengers. On the other hand, within the SAE definition of Level 3 autonomous driving, the driver is expected to respond appropriately to requests from the vehicle control system due to system malfunctions or deteriorating driving conditions, performing actions such as driving maneuvers. Therefore, in Level 3 autonomous driving, to ensure safe driving, even when performing the aforementioned secondary tasks, the driver is expected to always be ready to immediately revert to manual control.

[0156] Level 4 autonomous driving, also known as "fully automated driving under certain conditions," involves the vehicle control system performing all driving tasks within a limited area. In Level 4, the vehicle control system is both the primary agent for executing driving tasks and the primary agent for monitoring safe driving.

[0157] However, unlike Level 3 of autonomous driving described above, in Level 4 of autonomous driving, the driver is not expected to perform responsive actions such as driving operations (manual driving) in response to requests from the vehicle control system due to system malfunctions or other reasons. Therefore, in Level 4 of autonomous driving, the driver can perform secondary tasks as described above and can take a nap, for example, if necessary.

[0158] As mentioned above, within the range of Automated Driving Level 0 to Automated Driving Level 2, the vehicle operates in a manual control driving mode, in which the driver actively performs all or part of the driving tasks. Therefore, in these three levels of automated driving, the driver is not allowed to engage in secondary tasks, which are behaviors other than manual control driving and related actions, such as tasks that would reduce the driver's attention or impair the driver's attention to the road ahead while driving.

[0159] On the other hand, in Level 3 of autonomous driving, the vehicle operates in an autonomous driving mode, in which the vehicle control system actively performs all driving tasks. However, as mentioned above, there may be situations where the driver performs driving operations in Level 3 of autonomous driving. Therefore, in Level 3 of autonomous driving, when the driver is allowed to perform secondary tasks, the driver is required to be in a ready state to resume manual control of driving from the secondary task.

[0160] On the other hand, in Level 4 of autonomous driving, the vehicle also operates in an autonomous driving mode, in which the vehicle control system performs all driving tasks. However, even within areas where Level 4 of autonomous driving is intended to be applied, there may be sections where it is not feasible due to factors such as the actual maintenance conditions of road infrastructure. For example, suppose such sections are set at Level 2 or lower, thus requiring the driver to actively perform driving tasks. Therefore, even during the route planning phase or after the route has started and autonomous driving based on Level 4 is in use, a transition request to Level 2 or lower, as described above, may occur if conditions deviate from permitted conditions. Therefore, in the event of such changes in conditions, even if not planned at the start of the route plan, the driver is required to be ready to resume manual control of driving from secondary tasks depending on the situation.

[0161] Here, the actual usable scope of each autonomous driving level, as permitted by different levels of automation, is called the ODD (Operation Design Domain). More specifically, the ODD is the driving environment condition for the autonomous driving system to operate. The autonomous driving system can only function properly, enabling the vehicle to drive autonomously, when all the conditions indicated by the ODD are met. When the conditions indicated by the ODD are no longer met during driving, the vehicle's driving control needs to be switched from autonomous driving to manual control. Note that the conditions indicated by the ODD typically vary depending on the autonomous driving system, the degradation and contamination of equipment such as sensors, and the performance changes at the time caused by the self-diagnostic results of the devices used to control the autonomous driving system.

[0162] Figure 3 This is a schematic diagram illustrating the various levels of autonomous driving in SAE as viewed from the user's perspective.

[0163] Level 0 autonomous driving applies to environments such as private roads, ordinary roads, and highways—general social road infrastructure. Level 1 autonomous driving applies to roads and environments equipped with driver assistance systems, such as sections of highways and expressways. At Level 1, existing manually controlled vehicles require the application of driver assistance systems such as ACC and LKAS, which are based on the vehicle control system. In this case, because the driver assistance systems do not provide comprehensive support, decreased driver attention can lead to intuitive risks.

[0164] Furthermore, Level 2 Automated Driving is applicable to fixed roads such as highways where there are sufficient devices and environments for driver assistance. Within the Level 2 Automated Driving zone, in addition to acceleration / deceleration in the driving direction via driving control such as ACC, lateral control is generally permitted to allow for lane keeping, enabling the vehicle to follow lane markings implemented by LKAS (Lane Control Alerts). Therefore, continuous driver attention is still required in this zone. On the other hand, Level 2 Automated Driving allows continued driving unless obstructed. Therefore, the driver's sense of risk may decrease when driver assistance is overused. Thus, Level 2 Automated Driving is considered a level of automated driving that requires preventative measures to address driver inattention.

[0165] As mentioned above, in these ranges of autonomous driving levels 0 to 2, the vehicle's movement is controlled manually by the driver.

[0166] On the other hand, as mentioned above, the ranges for Level 3 and Level 4 autonomous driving are the ranges within which the vehicle's autonomous driving system can control the vehicle autonomously. Specifically, the environment suitable for Level 4 autonomous driving can be achieved, for example, by continuously updating each type of information in the LDM to obtain the range that ensures road predictability.

[0167] In contrast, Level 3 of Automated Driving applies to environments that are essentially capable of Level 4 Automated Driving but, for some reason, do not meet the ODD conditions corresponding to Level 4. For example, this could be a region where only transient static data is available in the LDM, or a region where Level 4 driving conditions cannot be continuously obtained due to deterioration or inadequacy in the system's environmental response performance. Examples of regions where Level 4 Automated Driving conditions are unavailable include temporary construction zones, flooded zones, complex intersection zones, zones lacking LDM, zones with temporary communication band gaps, and zones with risk reports from preceding vehicles.

[0168] Furthermore, environments suitable for Level 3 autonomous driving can include areas that are functionally permissible under Level 4 autonomous driving control, but for some reason, the application of Level 4 autonomous driving has been cancelled. Examples of such areas include those with a risk of causing traffic congestion, such as those where vehicles stop due to reasons such as MRM (Minimum Risk Maneuver) or emergency evacuation deceleration, construction zones, and railway crossings, thus obstructing the smooth flow of traffic infrastructure. Additionally, environments suitable for Level 3 autonomous driving can include areas where, by fixed or actively configured rules, passage without driver intervention is prohibited (due to penalties for violations of preventative procedures).

[0169] Note that, as Figure 3 As indicated by the solid arrows, from the user's perspective, even when driving at high speeds under conditions of smooth traffic flow and no issues such as traffic congestion, average speed will decrease due to congestion, even in areas where only Level 2 autonomous driving is permitted. Furthermore, when conditions temporarily allow Level 3 autonomous driving, autonomous driving functions such as ACSF (Automatically Commanded Steering Function) can be used to study future operations. For example, even in areas where autonomous driving was not initially assumed, such as congested sections on highways where Level 2 autonomous driving is applied, Level 3 or Level 4 autonomous driving can be used. Specifically, when Level 4 autonomous driving is permitted, NDRA (Non-Disruptive Response Alert) can be safely executed. In this case, it is necessary to be able to predict the end of congestion and manage the return to manual control.

[0170] This scenario assumes a situation where, as a form of autonomous driving, the vehicle transitions from a Level 2 autonomous driving zone to a Level 4 autonomous driving zone, and the vehicle's control switches from manual driving by the driver to autonomous driving by the vehicle's control system. In this case, the driver does not need to concentrate on driving, thus reducing attention span. That is, switching from manual driving to autonomous driving can be considered a form of use that leads to a decrease in the driver's sustained attention span.

[0171] Furthermore, considering the transition from Level 4 to Level 2 (return to manual control), the area for performing Level 4 autonomous driving can be defined as a usage zone where driver attention is reduced and the time budget until recovery needs to be planned based on driver monitoring information from the previous stable state. In the prior art, at Level 4, notifications of reverting to Level 2 or lower autonomous driving (i.e., reverting to manual control) are always sent to the driver.

[0172] The function corresponding to Level 3 of Automated Driving can be considered as a connecting role to avoid the separation between the areas of automated driving based on Level 4 and the areas of manual control driving based on Level 2 or lower. That is, the use of Level 3 automated driving involves the driver maintaining attention to the vehicle and expecting to recover within a short time (e.g., a few seconds). In Level 3, detecting a decrease in driver attention through the DMS (driver monitoring system) and maintaining driver attention is a necessary requirement for using Level 3 or lower automated driving levels.

[0173] Figure 4 This is a schematic diagram used to illustrate the application of Level 3 autonomous driving. Figure 4 The map shows the vehicle traveling in the direction indicated by the arrows (counterclockwise) from the starting point ST to the destination EP along the route TP (the filled part in the map).

[0174] exist Figure 4 In this diagram, intervals RA1, RA2, and RA3 represent intervals corresponding to, for example, levels 0 to 2 of automated driving, where manual control is essential. For instance, when a vehicle enters intervals RA1 to RA3 in an automated driving mode at level 4, the vehicle's automated driving system needs to transfer driving control from automated driving to manual control, such as by a driver. On the other hand, intervals RB1 to RB5 represent intervals where automated driving is permitted under attentive monitoring during the return to manual control. Intervals RB1 to RB5 are, for example, intervals corresponding to level 3 of automated driving.

[0175] In order to enter the zones RA1, RA2, and RA3 that require manual driving control, the driver needs to be prepared in the posture, etc., to switch from automatic driving to manual driving. Therefore, the zones RB1, RB4, and RB5 corresponding to automatic driving level 3 are respectively set on the entry side of zones RA1, RA2, and RA3.

[0176] On the other hand, sections RB2 and RB3 are, for example, sections that are functionally passable under Level 4 autonomous driving control, but are set to disable Level 4 autonomous driving for some reason. Section RB2 is, for example, a temporary construction section or a flooded section, while section RB3 is, for example, a section where attention to the vehicle's movement is required due to a sharp turn.

[0177] In this way, when a situation arises during Level 4 autonomous driving that requires a return to manual control, if there is sufficient time between the system issuing the return request and the vehicle arriving at the designated handover location, the system will first pass through a Level 3 autonomous driving zone, or a state where the driver's driving ability is awakened and they are able to pay attention to their surroundings, and their driving skills are restored. Because Level 3 autonomous driving requires the driver to maintain awareness of the situation without direct driver involvement, it may cause the driver to remain in a state of prolonged waiting and concentration without actual steering wheel operation, potentially leading to discomfort.

[0178] <<3. Embodiments according to this disclosure>>

[0179] Next, embodiments according to this disclosure will be described. In the following description, unless otherwise specified, ODD refers to the ODD corresponding to Automated Driving Level 4, which includes the conditions of Automated Driving Level 4. The driving range that satisfies the conditions represented by the ODD is simply referred to as the ODD range.

[0180] <3-1. Overview of the Implementation>

[0181] First, an overview of this embodiment will be explained by comparing it with the prior art. Figure 5 This is a flowchart schematically illustrating an example of the switching process from autonomous to manual driving according to existing technology. Figure 5 Before the flowchart processing begins, it is assumed that the vehicle, including the driver, is traveling in the ODD zone corresponding to Automated Driving Level 4.

[0182] As the vehicle approaches the end of the ODD (Operational Distance Registry) section, the automated driving system installed on the vehicle notifies the driver in step S10 that the end of the ODD is approaching. In response to this notification, the driver prepares to, for example, transfer driving control from automated driving to manual driving. If the handover of driving control from automated driving to manual driving is delayed beyond a predetermined time, the automated driving system issues an alert to the driver (step S11).

[0183] In step S12, the autonomous driving system determines whether the handover of driving control from autonomous driving to manual driving will be performed within a predetermined time after the notification issued as the ODD end pre-notification point in step S10. If it has been determined that the driver has completed the handover within the predetermined time (step S12, "OK"), the autonomous driving system proceeds to step S13 and evaluates the completion of the handover.

[0184] Conversely, if it is determined in step S12 that the handover of driving control from automated driving to manual driving has not been completed within the predetermined time (step S12, "NG"), the automated driving system proceeds to step S14. In step S14, the automated driving system applies MRM to the control of the vehicle and controls the vehicle to perform evacuation driving, such as emergency stopping at the roadside.

[0185] Here, in the concept of vehicle autonomous driving control according to existing technology, the level that allows the vehicle to operate within the SAE's autonomous driving level category is defined as ODD, which is the scope of the design concept for the vehicle's onboard equipment. Depending on the level of autonomous driving the vehicle is capable of, the driver is required to handle all requesting situations with constant consistency.

[0186] For example, suppose a specific highway allows autonomous driving at Level 4, and the vehicle's onboard automation capabilities allow for autonomous driving corresponding to Level 4. In this case, the driver can drive through this section using the vehicle's Level 4 automation. When the vehicle approaches a deviating zone (where the vehicle can drive at Level 4), the autonomous driving system prompts the driver to revert to manual control. Figure 5 (Step S10). If the response is delayed, the autonomous driving system will simply issue a warning ( Figure 5 (Step S11). Although the autonomous driving system issues a warning in step S11, if it does not revert to manual control at the appropriate time, the autonomous driving system should switch to emergency evasive maneuvering within the ODD range where the vehicle can drive at autonomous driving level 4. This control is called MRM (Maintenance Management System). Figure 5 (Step S14) thereby preventing the vehicle from entering the area where the system cannot handle autonomous driving.

[0187] In this type of vehicle control according to existing technology, based on the performance limits of the autonomous driving system, within the range where the autonomous driving system is allowed to operate at Level 4, the driver should engage in specific secondary tasks (NDRA) other than driving. On the other hand, when the autonomous driving system reaches its processing limits, it will issue a request to the driver to forcibly resume manual control of driving. From the user's perspective, this request forces the user to forcibly resume participation in secondary tasks.

[0188] Thus, when using a vehicle's autonomous driving system based on existing technology, the driver is forced into a subordinate relationship with the system. Consequently, for the driver, using autonomous driving functions for secondary tasks is a stressful mode of utilization or control.

[0189] Figure 6 This is a flowchart schematically illustrating an example of the handover process from autonomous driving to manual driving according to an embodiment. In accordance with... Figure 6 Before the flowchart processing begins, it is assumed that the vehicle the driver is riding in is traveling in the ODD zone corresponding to Automated Driving Level 4.

[0190] In step S20, the autonomous driving system notifies the driver in advance of the end of the ODD interval. For example, the autonomous driving system notifies the driver of the end of the ODD earlier than the time required for the driver to resume manual control of driving after receiving the notification.

[0191] In the next step, S21, a "transaction" related to the handover point is exchanged between the autonomous driving system and the driver. Here, "transaction" refers to a series of processes in which the driver explicitly responds to notifications issued by the autonomous driving system. At this point, the autonomous driving system presents the driver with information indicating the location where the handover must be completed and the risks that will arise from failure to complete the handover. Note that this "transaction" involves sharing information about the handover between the autonomous driving system and the driver, and does not require the driver to assume any obligations; therefore, it should actually be called a "temporary transaction."

[0192] In this way, through the driver's explicit response, a temporary transaction regarding the handover starting point is exchanged between the autonomous driving system and the driver, and the handover work is imprinted in the driver's working memory. Working memory, also known as working storage, refers to the brain's ability to temporarily store and process information needed for work or operations.

[0193] In the next step, S22, the autonomous driving system manages the handover process from the driver to manual control. For example, the autonomous driving system monitors the driver's state and, based on the monitoring results and the vehicle's state at that point in time, determines in step S22 the remaining time from the current point in time (current geographic location) to the handover start point, and whether to extend the grace period before the handover begins. Based on the determination, the autonomous driving system executes controls such as providing further notifications and transferring to the MRM (Management Manager).

[0194] The "margin" here refers to the time, determined by continuous passive monitoring and analysis of the driver's state, that ensures a longer time than is required to reach the handover completion threshold when the vehicle is traveling at a cruising speed estimated based on the surrounding traffic flow on the road it is traveling at, compared to the estimated time needed for the driver to resume driving. Furthermore, the aforementioned "extended grace time before the start of the handover" refers to extending the time before reaching the handover completion threshold without disrupting the flow of surrounding cruising vehicles, by means of, for example, reducing the vehicle's speed or moving to the roadside, a temporary evacuation area, or a low-speed lane.

[0195] In the next step S23, the autonomous driving system determines whether the handover of drive control from autonomous driving to manual driving will occur within a predetermined time, for example, as set in the handover step management in step S22. If the autonomous driving system determines that the handover has not been completed within the predetermined time, it will... Figure 5 The same as step S14 in the previous step, the autonomous driving system controls the vehicle to perform evasive maneuvers and performs emergency stops through MRM.

[0196] In the next step, S24, the autonomous driving system evaluates the driver's successful handover to manual control. At this time, the autonomous driving system calculates a score based on the driver's responses related to the handover. For example, the system adds points for actions deemed optimal for handover, such as when the driver voluntarily initiates the handover or performs a temporary transaction. Additionally, situations where the driver prematurely abandons the handover and chooses to rest are considered as choosing a means to prevent obstruction of surrounding vehicles, and are deemed appropriate considering the impact on social infrastructure, thus earning additional points. On the other hand, in cases where handover is initiated in response to repeated warnings or after an emergency, the vehicle is controlled via the MRM (Management Management System), and for actions deemed undesirable for handover, such as an increased risk of handover failure, the autonomous driving system deducts points from the score based on the degree of impact.

[0197] In the next step, S25, the autonomous driving system rewards or penalizes the driver based on the score calculated in step S24. For example, if the autonomous driving system adds or subtracts points from the score, for instance, using a score of 0 as a baseline, and the score calculated in step S24 is greater than 0, the system rewards the driver. Conversely, if the score calculated in step S24 is less than 0, the system penalizes the driver. The lower the score, the more severe the penalty. Examples of penalties include restrictions on the driver's use of autonomous driving and restrictions on the driver performing secondary tasks.

[0198] Thus, by rewarding or punishing drivers based on their evaluation of the handover process, a risk-inclusive marker can be added to the driver's work memory during the temporary transaction in step S21 above (step S26).

[0199] In embodiments of this disclosure, the autonomous driving system continuously monitors the driver's state and assesses the likelihood of the driver reverting to manual control. While the vehicle is under autonomous driving control, the system continuously presents the driver with advance notice of the necessary resumption of manual control, enabling the driver to resume manual control appropriately and without delay. Prior to the actual resumption start time, the system exchanges a "transaction" with the driver regarding the determination of an appropriate resumption start time and manages the handover process for that start time.

[0200] In this way, by implementing human-centered interactive control of autonomous driving, a smooth and reliable return to manual driving is achieved. Embodiments of this disclosure aim to enable comfortable use of autonomous driving. That is, embodiments of this disclosure do not merely unilaterally notify the driver of a transition demand—a request to return to manual driving—based solely on the vehicle's state to facilitate the driver's return. Instead, they aim to achieve handover control through collaboration between the system and the driver, sharing prior knowledge of the handover and influencing the driver's memory.

[0201] That is, the autonomous driving system according to embodiments of this disclosure notifies the driver of each handover start point at a time earlier than the predicted time required for the driver to resume manual control driving after receiving notification of the handover start point. Figure 6 Step S20 in the process. Then, the driver enters into a "provisional transaction" with the autonomous driving system regarding the actual handover starting point ( Figure 6 Step S21 in the process. The autonomous driving system manages the handover steps based on temporary transactions and performs budget and risk allocation for the handover sequence. Figure 6(Step S22 in the text). Therefore, the driver is able to make appropriate preparations for completing the secondary task and gain prior knowledge of the conditions required for manual driving control (e.g., knowledge of the surrounding environment required for driving control through manual driving control).

[0202] In embodiments of this disclosure, the “transaction” exchanged between the system and the driver ( Figure 6 Step S21) also has the following function: by designing the provided method, the driver's visual memory retains the importance of the handover and a general sense of time, as described below.

[0203] By establishing an initial agreement between the autonomous driving system and the driver regarding the handover from autonomous to manual control, the system reliably conveys the importance of the handover to the driver. By presenting the driver in advance with the location of the necessary handover and the risks that may arise if the handover is not completed, the system stores judgment information related to the driver's actions as "pre-notification information" in its working memory. Therefore, unlike existing technologies where the driver only begins to understand the situation after receiving notification, this prevents or at least reduces unintentional misjudgments of the handover commencement due to pre-loading certain non-negligible pre-notification information into memory.

[0204] Furthermore, according to embodiments of this disclosure, even in cases where there is a risk of neglect due to errors occurring at the handover initiation determination caused by the driver's dementia or neurocognitive impairment, it is possible to monitor the driver's recurring recovery habits and detect signs of error each time they occur. Therefore, the autonomous driving system according to embodiments may potentially gain the additional benefit of detecting symptoms such as dementia and neurocognitive impairment in the elderly by reducing the impact based on previous "transactions" (i.e., the expected execution of recovery work).

[0205] <3-2. Human-Centered Design (HCD) according to the embodiments>

[0206] The autonomous driving system according to the embodiment applies a human-centered design concept (HCD) to replace the device-centered or system-centered design concept (MCD) commonly used in existing systems. In other words, the autonomous driving system according to the embodiment performs coordinated control that incorporates human behavioral characteristics into vehicle control.

[0207] <3-2-1. Overview of HCD according to the embodiment>

[0208] First, an overview of the HCD according to the embodiment will be explained. In existing MCDs, the autonomous driving system mechanically determines the ODDs (Operational Designations) that can be used for autonomous driving based on the performance of the vehicle's onboard devices, and uniformly allows the use of autonomous driving functions within a limited scope. In this case, even if the user over-relies on the autonomous driving functions, the system's control over the user is limited to unilateral control instructions, notifications, and alarms, or only falls within the scope of the Minimum Risk Maneuver (MRM) when dealing with problems is impractical.

[0209] In contrast, the HCD disclosed herein controls whether a user can use the autonomous driving function during autonomous driving to promote use within a socially acceptable range. That is, it performs usability control taking into account characteristics derived from the driver's behavioral habits, thereby allowing the function to be used when the user has appropriate behavioral habits. On the other hand, for inappropriate behavioral habits (e.g., not responding to requests to return to manual driving, delaying the action to return to manual driving, or reducing the quality of the return action), an HMI (Human-Machine Interface) is employed to prompt the user to change their behavior. Based on their adaptability, the availability of the autonomous driving function is proactively applied to the individual.

[0210] Implementing such HCD is difficult simply by introducing a single function; it requires multi-layered, multi-dimensional, and dynamic information feedback that encourages behavioral change during use. This disclosure employs the concept of "transactions" and uses transactions between the system and the person (driver) to facilitate information feedback for implementing HCD.

[0211] Specifically, the transactions and feedback are conducted as follows.

[0212] • When the autonomous driving function is first used in each permitted zone, the system and the driver exchange “trades”.

[0213] • Before reaching the end of the usage range associated with the transaction, the system and the driver confirm the “associated transaction”, which means completing a quick and safe handover of manual control driving without the need for emergency stops or deceleration.

[0214] • The driver shall fulfill the ancillary obligation related to reconfirming the changed status within the elapsed period of use.

[0215] • Grant credit to drivers (driver credit), which serves as an assessment of an individual's validity in the "transactions" involving repeated use of autonomous driving.

[0216] • The ODD of the permitted range of autonomous driving on recently used roads is redefined based on the assessment history of past performance of the obligation to resume manual driving, i.e., driver credit, and this redefinition takes into account the individual characteristics of the driver's resumption.

[0217] The system provides these items as visual information to the driver, and also provides visual information on changes after the start of each permitted interval (as additional transactions) to confirm changes in circumstances. For example, if the conditions at the end of the autonomous driving period differ from those at the start, the system will provide visual feedback to the driver using visual information describing the factors that would lead to the termination of autonomous driving and the potential risks that might occur if the end of the autonomous driving period is not handled correctly. After the information is presented to the driver via the HMI, the information, as risk assessment material corresponding to the situation where a recovery action is ignored, is retained in memory. As a psychological influence on behavioral judgment, the driver's memory is enhanced because of the impact of the situation where the recovery request is ignored.

[0218] Simultaneously, stored memories retained as working memory gradually fade over time. Specifically, when there is no urgent need for manual driving, if non-driving-related activities (NDRA) such as watching television broadcasts continue, the handover to manual driving ceases to be a primary concern. Therefore, processing information about brain activity occurring outside of conscious awareness in the subconscious becomes crucial for re-establishing the necessary event. That is, in this sense, the importance of HMI (Hardware Mindfulness) increases, as it introduces risk memories into the subconscious level of the brain through subconscious methods and other means, thereby restoring interest in the handover of driving.

[0219] In addition, the system can ask the driver to make a pointing vocalization call to confirm the direction of travel, thereby prompting the driver to look forward and confirm, and the result can be evaluated.

[0220] To achieve HCD-based control, continuous reconstruction of the necessary memories is required, taking into account individual differences and the current working memory storage capacity. The problem here is that human memory differs from information that can be directly observed externally. Therefore, in this disclosure, the system achieves continuous reconstruction of the necessary memories through a "transaction" exchanged with the driver.

[0221] <3-2-2. Advantages of HCD in Autonomous Driving>

[0222] Next, the advantages of HCD in autonomous driving according to this disclosure will be explained.

[0223] Depending on the introduction process, the introduction of autonomous driving technology into society has a significant impact on long-term users and their interactions. To successfully introduce autonomous driving technology without causing adverse social side effects, it is necessary to appropriately mitigate its negative impacts to facilitate its adoption. That is, if the functions developed through technological advancements are provided indefinitely without considering human behavior and psychology, users may not necessarily use the technology within a socially acceptable range.

[0224] The concept of introducing autonomous driving into society based on existing technology involves progressively expanding the use of autonomous driving functions according to the level of technological realization (e.g., SAE autonomous driving levels). In other words, as technological development progresses, the scope of automated functions is gradually expanded, starting with those that can be automated, and then gradually introduced into society. In other words, the concept of introducing autonomous driving into society based on existing technology represents the idea of ​​progressively introducing autonomous driving into society based on the achievements of technological development, specifically, based on the performance of the developed and implemented machines, and on the performance stages of autonomous driving functions defined by SAE and other organizations as autonomous driving levels 2 to 4.

[0225] In contrast, in the technology according to this disclosure, the provided autonomous driving function is dynamically changed and the permissible deviation of the autonomous driving is dynamically controlled based on the driver's adaptability to the device, i.e., based on whether the driver has the acceptance to properly use the technology.

[0226] That is, in this disclosure, even if the mechanical and functional construction of the autonomous driving system is completely identical, the autonomous driving functions to be provided in practice are such that the autonomous driving functions actually available to the user are dynamically changed based on whether the driver, as the user, has sufficient behavioral compatibility to safely use the functions. This disclosure relates to techniques for applying such an HCD concept to the control and operation of autonomous driving systems.

[0227] <3-2-2-1. Over-reliance>

[0228] To use autonomous driving functions appropriately, drivers need to avoid over-reliance on them. The simplest example of over-reliance is when, despite being clearly designed to be involved in control, the driver neglects to pay attention to what's ahead or fails to fulfill their duty to monitor their surroundings. In this situation, the driver's attention to the vehicle ahead may decrease, potentially leading to a lack of awareness of inter-vehicle distances. That is, for example, when using autonomous driving functions limited to lane-keeping assist, even though these are limited aids, the driver may become reliant on them if there are no other vehicles or obstacles interfering with the vehicle's movement in front or behind. When drivers feel comfortable with these aids, their attention to driving may decrease, potentially leading to delayed judgment or excessive evasive behavior in emergency situations.

[0229] In situations involving decreased attention, drivers may experience delays in responding to emergencies, excessive evasive maneuvers, or an inability to handle emergencies. In such cases, drivers might panic and attempt to decelerate suddenly, which could cause secondary damage such as rear-end collisions or traffic congestion. As Level 2 autonomous driving, it can perform autonomous control even in more complex and multifaceted situations, requiring less driver involvement. Drivers feel secure with these driver assistance features and, while requiring continued attention to situations necessitating intervention, can still easily reduce their focus.

[0230] Introducing a level of automation higher than Level 2 eliminates the need for the driver to constantly monitor the road, thus complicating the problem. If the system determines that continuing in automatic control would exceed the limits of the onboard equipment's situational awareness (which is dangerous), then when the vehicle is traveling at Level 4, the system needs to relinquish automatic control and initiate a procedure to allow the driver to take over appropriately. Alternatively, if a sudden event requiring emergency handling makes it difficult for the driver to take over, a risk minimization procedure needs to be automatically activated. In this case, if the driver is unwilling to quickly revert to manual control as instructed by the system and is unwilling to take measures such as reverting to manual control, then to delay the system reaching its limits, the system may take preventative measures, such as reducing the vehicle's speed relative to surrounding vehicles or activating MRM (Mission Management). These are good examples of over-reliance on automatic control.

[0231] Similarly, in Level 2 or Level 3 autonomous driving scenarios, with increasingly advanced assistance, drivers can directly perceive the system's appropriate and successful handling of events under many driving conditions, even without frequent steering intervention. Because drivers can continue driving even when their attention is continuously reduced and their sense of risk is not directly related, they may take the system's handling of events for granted. Over time, drivers become accustomed to relying on autonomous driving, which may lead them to be less suspicious of the system's shortcomings.

[0232] In Level 2 or Level 3 autonomous driving, driver attentiveness is not permitted because immediate action is required in unusual situations. However, at Level 4, the possibility of attentiveness lapses exists. Furthermore, while Levels 2 and 3 require continuous driver attention, there is no guarantee that drivers will always fulfill their attentiveness obligations in terms of, for example, ergonomics.

[0233] In other words, the current situation is one where users are unilaterally forced to correctly understand and use a single design constraint based on the performance achievable by the machine's designed functions. Assuming users can use the technology as expected based on its designed performance, the introduction of this technology into society retains the problem of over-reliance on it. Typically, the human brain is skeptical of newly developed, unknown technologies and unconsciously takes precautions against them. However, with the advancement and popularization of autonomous driving, the progress and diversification of autonomous driving functions gradually reduce the unease and skepticism during use, thus exacerbating and making the problem of over-reliance more serious.

[0234] This disclosure does not address the fundamental problem by simply viewing it as a matter of decreased awareness and attention, and by implementing systems to prevent such declines (warnings, recovery of awareness, etc.). Instead, it relates to the technology of introducing a series of mechanisms necessary for naturally learning and adapting to the user's behavioral psychology based on the limitations of autonomous driving performance. Specifically, this disclosure provides a mechanism that executes a series of controls on the user to induce improved and changed behavior. These controls gradually alter the user's repetitive actions and, to further promote improved behavior, employ a layered mechanism for the driver.

[0235] <3-2-2-2.HCD>

[0236] The key point of this disclosure is to shift the relationship between the vehicle and the driver from the existing MCD (Motor Vehicle Control) to HCD (Handheld Control), and to determine the system's operating area based on human behavior. This determined operating area, based on human action routines, influences the benefits the user gains from using the vehicle. When the user feels comfortable, the system weights its feedback loops to avoid unintentional harm to social activities (traffic congestion, rear-end collisions, road blockages, etc.). This disclosure relates to an HMI (Hardware Management Interface) that effectively enables the system to maintain weighted system control and the cultivation of human behavioral habits in a virtuous cycle.

[0237] That is, it requires users to change their behavior, whereby the usage pattern of vehicles equipped with existing autonomous driving functions shifts away from excessive reliance on the provided functions and towards collaborative use. To generate this change in usage behavior, a mechanism for generating this change is needed. This disclosure proposes mechanisms for the various elements that generate this behavioral change in collaborative use, as well as techniques related to the overall application of these mechanisms.

[0238] For HCD to be usable in various ways, changes in user behavior are essential, and an HMI (Human Interface Management) is also needed to generate these changes. HCD is not simply a mechanism that allows users to use functions to satisfy their own desires; rather, it is a whole mechanism developed to naturally prompt users to take responsive actions in order to make them comfortable using the functions. Considering human behavior, the desired design is not one that allows the system to satisfy instinctive, animalistic desires, but rather one that can be redefined as a design that incorporates the mechanisms necessary to comply with (or allow compliance with) the rules required to maintain social order in modern society, and where functional design can be considered as a design to achieve these mechanisms.

[0239] This will be explained in more detail. First, when autonomous driving is introduced into society, its availability varies depending on the conditions and is determined on the premise that the system has the function of performing automatic vehicle steering control.

[0240] The vehicle needs to have at least the following functions: acquiring information from the outside, supplementing information with the acquired information while understanding the environment, executing the vehicle's driving plan, and driving along the generated plan. Based on this, when the system cannot confirm its ability to perform a series of processes under all conditions, depending on whether the driver is required to quickly return to manual control driving or whether a request for resumption is not required, the system allows the vehicle to perform autonomous driving beyond Level 2, such as Level 3 or Level 4.

[0241] Furthermore, for example, in Level 4 autonomous driving, one or more instances of autonomous driving termination may occur before the vehicle reaches the end of the driving route. In such cases, the driving route will necessarily contain multiple handover sequences from the termination of autonomous driving to manual control.

[0242] Here, Level 5 Automated Driving, which has higher capabilities than Level 4, is defined by SAE. Level 5 is applied after a Level 5 Driver Module (LDM) is ready to update information with higher clarity and refresh rates than its surroundings, either by initiating in a closed environment or by making significant infrastructure investments in acquiring environmental and environmental information. Examples of applications for this level include situations such as robotaxis. Unless operating like a robotaxi in Level 5, users acting as drivers in vehicles capable of Level 4 Automated Driving will be required to revert from automated driving to manual control during the journey.

[0243] The necessary conditions for autonomous driving functions in vehicles are identified as the implementation limits that can be addressed through vehicle design and development. In this context, the limits of autonomous driving can be expanded through equipment construction and infrastructure development, albeit at high cost, such as the amount of information acquisition resources that can be allocated for optimal processing, the amount of resources available for autonomous or external acquisition and revenue generation, and the power and cost resources that can be allocated for computing. On the other hand, although the frequency of reversal requests and the available range of autonomous driving differ, it is difficult to completely eliminate situations requiring a return to manual control from autonomous driving. Therefore, for these handover requests occurring while using the vehicle, a driver-controlled driver (HCD) system is needed to replace the existing driver-controlled driver (MCD) system, enabling appropriate driver intervention.

[0244] When the system's structure is changed from a MCD (Multi-Device Control) model that relies on device performance to a HCD model that emphasizes coordination with the human, the system should guide the user to use it appropriately without over-reliance. To this end, a mechanism is needed where the system enables the driver to self-learn the balance between the benefits of using autonomous driving and the losses or risks incurred in enjoying those benefits, allowing the user to benefit from this balance while comfortably using the system and fulfilling necessary obligations.

[0245] <3-2-2-3. Benefits for Drivers>

[0246] From the perspective of finding candidate benefits for vehicle users, the desired benefits are shown below, represented as one or more complex items.

[0247] First, examples of actions used to gain benefits will be given below.

[0248] (1) Simply complete the movement from the starting point A to the target point B.

[0249] (2) Move between two points through comfortable movement.

[0250] (3) Complete the sport with a lower budget.

[0251] (4) Complete the exercise in a shorter time.

[0252] (5) Complete the exercise within the scheduled time.

[0253] (6) Complete the exercise with less fatigue.

[0254] (7) Even when fatigued or in poor physical condition, the expected exercise can be completed.

[0255] (8) Leave the scene no matter what.

[0256] (9) To achieve the purpose of transporting necessary goods.

[0257] (10) Enjoy outdoor driving or scenic driving in good weather as a form of recreation.

[0258] (11) Able to perform work other than driving while on the move (secondary task, i.e., NDRA).

[0259] Examples of NDRAs performed during movement in the above item (11) include the following.

[0260] (11-1) Diet

[0261] (11-2) Browse using devices such as mobile terminals

[0262] (11-3) Sending text messages using email

[0263] (11-4) Implementation of Teleconference

[0264] (11-5) Execution of leaving one's seat and delivering packages, etc.

[0265] (11-6) Makeup and Hair Styling

[0266] (11-7) Enjoy karaoke, watching movies, and watching sports broadcasts, etc.

[0267] (11-8) Operation of terminal devices such as smartphones, mobile phones, tablets, and laptops

[0268] (11-9) Enjoying the scenery while moving

[0269] (11-10) Check the inner pockets, etc., and look for lost items.

[0270] (11-11) Talk or interact with other passengers, or play crossword puzzles.

[0271] (11-12) Other eSports

[0272] (11-13) Using autonomous driving only during periods of traffic congestion as a benefit to alleviate burden

[0273] (11-14) Main measures for temporary poor physical condition (e.g., leg cramps)

[0274] (11-15) Temporary vision loss

[0275] (11-16) Use and support of eye drops: Temporary visual impairment associated with the use of eye drops

[0276] (11-17) Response to asthma or epileptic seizures

[0277] (11-18) Take a short break while driving in areas where it is safe to drive at Level 4 autonomous driving.

[0278] (11-19) Perform undefined processing

[0279] Further examples of assumptions about actions taken to gain benefits are as follows:

[0280] (12) Maximize the continuous use of autonomous driving functions

[0281] (13) Avoid losses related to use

[0282] (14) Avoid causing trouble for any relevant parties.

[0283] During autonomous driving, when it's necessary to revert to manual control, if the driver cannot properly revert to manual control when the system issues a recovery request, as mentioned above, emergency deceleration or reverse braking will be required as part of the Management Responsibility Response (MRM). Therefore, this could potentially expose negative effects of autonomous driving, such as highway congestion and traffic jams on personal / goods delivery routes, and rear-end collisions.

[0284] From the user's perspective, secondary losses arising from emergency control, aside from the scenario where the vehicle is hit in a rear-end collision, only affect the vehicle behind and do not diminish the benefit of prompting appropriate corrective action. In other words, if the use of autonomous driving is simply left to the driver's common sense from an HCD perspective, maintaining social order may become impossible.

[0285] Therefore, in order to maintain social order and promote appropriate user actions while introducing HCD control, for example, while achieving the benefits listed in (1) to (14) above, a mechanism is needed to incentivize drivers to perform restorations related to requested restorations without delay. However, moral incentives are only an idea, and even if drivers using autonomous driving are educated and expected to act ethically without ignoring restoration requests or taking measures to prevent delays, appropriate effectiveness of restorations is not always guaranteed.

[0286] To encourage drivers to respond appropriately and promptly to recovery requests from the system, a mechanism is needed where drivers take risks rather than reap benefits when ignoring recovery requests, and where the recovery requests directly influence the driver's psychological behavior to maintain balance. That is, a mechanism is needed to provide drivers with effective risk-related input in some form. This is because the balance between advantages, benefits, and current risks determines the action.

[0287] For example, as previous examples of prompting recovery action in response to a recovery request from a microscopic perspective, JP 2019-026247 A and JP 2016-153960 A are known. JP 2019-026247 A discloses a technique of blowing cold air onto the driver to keep the driver awake. JP 2016-153960 A discloses a technique of using alarms to periodically provide wake-up notifications to the driver. These previous examples are not mechanisms that encourage drivers to change their behavior regarding the use or utilization of autonomous driving functions from a macroscopic perspective.

[0288] For example, to gain the benefit of rapid arrival, and considering the balance with the responsibility of paying highway usage fees, the example of using highways when fees are low and not using them when fees are high is a conceptually achieved balance. Furthermore, if a driver watches sports broadcasts designated as NDRA (National Disturbance Administration) during autonomous driving and continues watching even after receiving a handover request, the driver can be penalized as a demerit. Examples of such penalties in this case could include issuing injunctions regarding the right to watch broadcasts during a specific time period or on the same day, prohibiting repeated use of autonomous driving during a specific time period, including forced stopping of the vehicle in evacuation zones, etc.—that is, imposing various penalties on the user individually.

[0289] <3-2-2-4. Driver's Working Memory and Thinking During Driving>

[0290] Here, to prevent the social infrastructure from collapsing due to HCD control, regardless of each user's strengths and weaknesses, the final focus will be on human action. Therefore, within the timeframe from the handover request to the recovery limit point, a smooth and high-quality handover to manual control is required from the driver, without reducing the basic cruising speed.

[0291] Here, the quality of the recovery (handover) action will be explained. Driver actions vary individually for each driver. Therefore, the system learns the normal handover actions of the target driver and estimates the time required for the driver to perform the recovery based on the driver actions learned in action learning. The quality of the recovery action refers to: using overall action evaluation as an indicator of the quality of actions such as, for example, the driver responding quickly to a recovery request from the system and completing the recovery operation within a time limit, or, despite being notified to complete the recovery within a time limit, the driver not taking the recovery action expected in learning to perform the recovery normally, but instead taking actions included in the normally learned recovery action, such as delaying the start of the recovery or taking slow actions.

[0292] That is, it is necessary to define the controls required for user recovery from the perspective of HCD, as described below.

[0293] It cannot be guaranteed that users will take action-judgment-based responses after understanding the details of the unseen ODD determined by an autonomous driving system with advanced and precise design (including the performance of the equipment and vehicle). Therefore, autonomous driving systems must have mechanisms to present users with describable and specific risks, allowing users to intuitively grasp the benefits and risks as intuition or feeling.

[0294] The human brain assesses risk based on limited information, identifies risk-mitigating behaviors within a limited timeframe, and then takes action. In behavioral psychology, whether a person can take necessary actions at the right time depends on experience and history—that is, on how the necessity or inevitability of remedial action has been learned from past experiences, and this varies from person to person. On the other hand, with the development of autonomous driving, it is expected that vehicles using autonomous driving will be able to continuously cope with a wider range of situations.

[0295] As autonomous driving systems adapt to various situations, the need for drivers to intervene, at least intuitively, when switching from autonomous to manual control is reduced. This gradually decreases driver skepticism towards the system. Consequently, when necessary, drivers no longer prepare for instantaneous steering maneuvers, gradually ceasing their focus on the front of the vehicle, checking the sides / rear of the vehicle, or observing and confirming the vehicle in front.

[0296] Therefore, once a driver leaves the active driving cycle, even if the driver receives a sudden request from the system to resume manual driving, their mind will shift to things other than driving and matters of interest. In this state, even if the system notifies the driver of the request to resume manual driving and the driver receives the notification, it will take a considerable amount of time for the driver to acquire insufficient information, grasp the situation, and take action to avoid an actual handover accident, as they begin to understand the temporary interruption. Furthermore, when a driver begins actions or behaviors as an NDRA (Disengagement and Responsibility Assistance) while physically away from the driver's seat, including movement back to a state of heightened awareness, even more time will be required.

[0297] When a person manually drives a vehicle, they take measures to deal with various events that occur along the route, and each event, in turn, without any accidents, performing the driving task while avoiding traffic disruptions. However, behind this seemingly smooth driving operation lies the fact that the driver unconsciously reviews a large amount of information beforehand. In order to predict the future impact associated with the operation based on this information, the driver explores the information needed to judge each situation in advance, gaining a certain degree of security, and taking measures to prevent delays in the action judgments required to prevent accidents.

[0298] For example, even in a single operation of pressing the brake pedal, the driver will have the following information before pressing the brake pedal and then combining a large amount of information to execute the final braking action, in order to prevent a potentially dangerous situation:

[0299] • How to apply braking force to the vehicle by pressing the brake pedal;

[0300] • Goods carried by the vehicle itself;

[0301] • Information used to determine whether occupant load increases braking distance;

[0302] • Information on road surface slippage risks (wet roads, snow-covered roads, etc.) is used to determine whether a vehicle will slow down in advance before reaching the corresponding section;

[0303] • Make predictions based on the behavior of the vehicle in front;

[0304] • Determine the risk of sudden deceleration due to the presence and type of vehicle behind;

[0305] • Fog conditions in front of the vehicle;

[0306] • Determine if there is a risk of delay due to visibility-impeding factors such as backlighting.

[0307] In other words, if the driver deviates significantly from the steering loop during automated driving, the system will begin to hand over control of the vehicle from automated driving to manual control without having acquired the necessary pre-stored information (working memory) required for steering control, i.e., without a grasp of the situation. Even if the handover request is mechanically and abruptly issued to the driver, the driver may not be able to immediately obtain prior judgment information.

[0308] Therefore, if a driver is asked to make a brief decision-making action when unprepared for the situation, the driver may begin to panic. In this situation, the driver may be forced to take action while in a state of panic. That is, in order to control HCD while taking into account the human judgment process, the system needs to provide a mechanism that balances the time required to recover to the driver's state (e.g., thought and posture) with the system's ability to achieve continuous autonomous driving and master the road environment, thereby issuing a recovery request from the driver with a choice of a remaining grace period that always ensures continuous autonomous driving.

[0309] At this point, estimating a person's mental state is actually extremely difficult. For example, there might be a situation where, from an external perspective, the driver's gaze appears to be directed forward, seemingly focused, but the driver's mind is actually preoccupied with things unrelated to driving. In this case, the driver's thoughts (working memory) may be directed to events completely different from driving, and the working memory may be in a state of lacking the information needed to make driving decisions.

[0310] The system needs to estimate the time (grace period) required for the driver to normally revert to manual control before the end of the safe period for using the automated driving function. In existing manual control driving scenarios, for example, driver inattention while looking ahead can lead to overlooking hazards, directly causing accidents. Therefore, to prevent drivers from neglecting to pay attention to what's ahead, the system essentially ensures that the collection of information necessary for driving is not continuously interrupted, even if the driver is temporarily engaged in tasks other than driving.

[0311] Therefore, during existing manual control driving, the driver constantly engages in intermittent visual attention. Thus, based on observational data obtained by noticing the reduction in these actions, distraction can easily manifest as symptoms such as drowsiness while driving. On the other hand, by utilizing Level 1 or higher autonomous driving functions, some of the actions performed by the driver are eliminated. Therefore, the higher the level of autonomous driving, such as Level 1 or higher, the less the driver needs to intervene. This gradually reduces the information gathering and judgment actions the driver takes based on safe driving steering decisions. Therefore, upon receiving a notification to revert to fully manual control driving, the driver needs to acquire the missing additional information to grasp the situation and make judgment actions, which takes time.

[0312] Triggers for human action include conceptual step-based triggers and stimulus triggers that immediately reflect in action to avoid danger, even if conceptual steps are lacking. Here, the latter is a reflexive avoidance action. This corresponds to risk-avoidance behavior based on limited information utilizing unexpected information, which is typically a feedback behavior useful to actions that cannot function properly or effectively.

[0313] In other words, when a driver maintains normal manual control, they are constantly monitoring and confirming the conditions ahead. Therefore, the driver typically does not suddenly operate the steering wheel, suddenly press the brake pedal, or over-operate the steering wheel or brake pedal. On the other hand, there may be situations where the vehicle is about to leave its lane due to distracted driving or carelessness, or the driver is so distracted by the risk that they fail to notice situations such as the braking of the vehicle in front. These situations can lead to accidents such as rollovers, rear-end collisions, and skidding due to sudden actions (unnecessary oversteering and sudden braking, etc.).

[0314] Examples of conceivable factors contributing to a lack of proper motor control include insufficient information to predict secondary impairment of working memory required to inhibit motor activity, and an overabundance of information to process. Excessive information can cause the brain to panic, making it difficult to appropriately execute reactive feedback actions, such as controlling the degree of responsiveness, leading to actions like over-direction for avoidance. Furthermore, human information gathering also includes the function of filtering out unnecessary information when continuously receiving information that is not needed for motor judgments.

[0315] Therefore, even information related to handover, if unilaterally and mechanically provided to the driver by the system during normal use, resulting in information that is neither necessary nor changing, will lead to this information occupying the driver's working memory. This becomes an obstacle to acquiring other potentially important information. Consequently, this unnecessary information is unconsciously identified by the brain as noise, becomes unimportant in terms of weight, and is then filtered out. A good example of filtering is that a portion of the external information a person receives physically is filtered out before it enters the brain for thought; this is known in fields such as psychology as the "cocktail party effect," which is the ability to clearly hear conversations of specific people even in noisy environments.

[0316] Based on this mechanism of the human brain's selective use of information, there needs to be a mechanism in which the driver is provided with information about events that occur in various ways during the journey, are updated and appear on new routes, as continuously approaching information; and a mechanism that allocates information to working memory based on the importance and relevance of the information, with the allocated information being based on information provided by the system to determine how to identify relevant information that is crucial for the driver's handover.

[0317] Here, the brain regions known as working memory cannot be directly observed or visualized. Therefore, directly operating on working memory in this technique, in terms of its explicit implementation, is extremely difficult. The priorities of a driver's brain activity vary greatly depending on the driver's current situation. Furthermore, working memory does not allow the system to directly retain information.

[0318] Considering these human characteristics, in order to ensure that the driver has sufficient time leeway to hand over control during autonomous driving before the end of the autonomous driving period, the system needs to provide the driver with unique information on the priority factors for changing actions, and needs to train the driver to learn a measure of the significant impact of each piece of information on the driver. That is, simply providing the driver with information can be considered equivalent to noise to the driver. Therefore, information is defined as having high importance and high priority in working memory when it is determined to be important in outcome prediction and when there is a risk of learning into a driver's shortcomings.

[0319] <3-2-2-5. "Transactions" between the system and the driver>

[0320] Here, the first exchange of notifications and confirmations between the system and the driver, where the driver receives system notifications and is responsible for receiving them, is considered a "transaction" between the system and the driver. Starting from the initial transaction, the execution of the handover work based on the initial transaction and the "degree of completion of the transaction" are analyzed, based on observable information, as the quality of the recovery transition (recovery operation). The quality of the analyzed recovery operation is the "credibility information" of the driver's execution of the transaction. Credibility information is used as a threshold to determine whether a high-quality, defect-free product from the transaction was used when the autonomous driving switch is reused in the next driving route or subsequent driving route segments. Then, credibility information is used in each feedback to the driver to process events, effects, and visual sensations, thereby promoting reinforcement learning in the driver's intuitive senses.

[0321] That is, instead of simply receiving information unilaterally from the system to the driver, the driver executes a series of "transactions" by responding to the notifications. As the subject of the event, the driver views responding to the notifications as an obligation to resume manual control of driving within the transaction. Through a series of repetitive actions corresponding to the obligation to resume driving, the driver is able to control the HCD, thereby successfully and voluntarily participating in the use of the automated driving system.

[0322] The various information presentation methods described in the embodiments of this disclosure are merely representative examples of the various means that can be used to implement the HCD, and therefore the method is not limited to the examples described. In particular, there are individual differences in how a driver stores the transaction items of a “transaction” in their memory, how a driver remembers the obligation over a period of time, and how a driver fulfills the obligation with high priority and without delay at the necessary time, and therefore, there is no need to limit the method.

[0323] HCD does not represent simple control that provides drivers with simple, specific information. HCD considers human cognitive judgment and behavior; it is a design that takes into account the broader perspective of achieving functionality relevant to the use of the system design. More specifically, HCD requires establishing a system that includes the mechanisms necessary for the development and growth of ideal cognitive behavior.

[0324] In behavioral psychology, behavioral development is not spontaneous or unconditionally progressive. That is, individuals develop behaviorally according to the norms and rules required of them as members of a family, community, or society. This development involves a balance between the benefits of satisfying desires, the drawbacks of social punishments imposed by rules and norms, and the risks directly suffered by individuals unrelated to social norms. From this perspective, the impact of driver assistance levels (as a disadvantage or risk) on human behavioral psychology, as a function of autonomous driving, would be a reduction in the perceived risk of decreased attention due to direct driving errors such as steering errors or fatigue, and an unnecessary increase in the sense of security.

[0325] However, the ultimate goal of using driver assistance systems is to improve comfort and prevent or avoid accidents even when drivers using the vehicle with expected attention have reduced or ignored important information, and even in the worst case, reduce the risk of aggravating the accident.

[0326] Therefore, in embodiments of this disclosure, to avoid excessive driver reliance on driver assistance systems, a mechanism is provided that does not solely prioritize comfort in all steering maneuvers receiving system assistance and intervention. Instead, a mechanism is provided to assign alternative risks to apparent over-reliance when performing self-avoidance processing of the automated driving system. These alternative risks serve as penalties for the driver, such as introducing uncomfortable controls and forcibly disabling the assistance function. By employing a mechanism provided in this way, situations where excessive driver reliance on automated driving systems directly leads to accidents but fails to fully achieve risk avoidance can be avoided, thereby establishing a psychological reaction-operational cycle for the driver.

[0327] Here, the system's operational concept changes significantly when it incorporates additional autonomous driving functions beyond driver assistance. Specifically, because there are periods of Level 4 autonomous driving where the driver does not need to participate in driving control at all, there will be situations where steering is functionally risk-free for the driver.

[0328] From MCD's perspective, it is sufficient to meet all the conditions for a vehicle to operate at Level 4 of autonomous driving. However, as mentioned above, over-reliance on autonomous driving can lead to negative consequences such as traffic congestion. Therefore, as a social mechanism, uncontrolled use of Level 4 autonomous driving is not ideal. There are rules that must be followed regarding the use of the system, and a typical rule is to use autonomous driving only within the limits of satisfactory conditions.

[0329] Based on the HCD perspective, in order to achieve use that conforms to orderly social norms, a mechanism is needed in which the driver uses Level 4 autonomous driving, which includes NDRA, only within the range where the autonomous driving system allows use at Level 4. When the use is predicted to end or when it becomes necessary due to changes in the situation, the driver can quickly learn the behavior defined by social norms and can develop reinforcement learning habits in daily use.

[0330] In other words, if there is no improvement in the quality of the driver's behavior in transitioning to a rapid and appropriate recovery after a recovery request notification, the driver cannot enjoy the benefits of using Level 4 autonomous driving as a "reward (benefit)". Furthermore, the initial information about the "risks" required for a human decision is information unconsciously and temporarily stored in the driver's memory through their approval of the "transactions" presented to them by the system. Changes that may occur after the start of a Level 4 autonomous driving interval, or interactions between the system and the driver via the HMI for reconfirmation, will be "incidental transactions" involving a review of conditions that have changed over time. Additionally, whenever the driver resumes driving within a Level 4 autonomous driving interval, the driver checks information related to the end point and, with consent—that is, after checking the necessity and timing of the resumption and the end request information—begins traversing the route within the ODD interval determined by the system as Level 4 autonomous driving.

[0331] The term "transaction" in this disclosure can conceptually refer to any interaction performed between the system and the driver, not limited to the exchange of physical documents. Through this interaction, the necessity of recovery, its impact risks, and the severity of the consequences of the violation are instilled in the driver's memory, thus making the stored information difficult to forget, depending on the importance of the response action.

[0332] Drivers cannot unconditionally use the Level 4 autonomous driving system, which is permitted by the system. Instead, they must adhere to conditions, including the obligation to revert from autonomous driving to manual control. The quality of the driver's compliance with these conditions will be a credit factor in their future use of autonomous driving. For example, if a driver's use of autonomous driving functions violates the permitted scope, they will not receive the benefits of enforcement permits such as the NDRA (National Disclosure Agreement) that are supposed to facilitate the use of autonomous driving. Furthermore, disadvantages include penalties, restrictions on autonomous driving, and even restrictions on vehicle use, for example, in the event of serious violations. Due to these disadvantages, drivers gradually become more sensitive to warning information about risk predictions during their journey. As reinforcement learning progresses, drivers become more sensitive to warning information that maximizes benefits without sacrificing advantages.

[0333] That is, the automatic driving control of the HCD according to the embodiments of this disclosure is completely different from the concept of the conventional MCD. In the conventional MCD, the system incorporates a mechanism that issues an alarm and forcibly restores the driver's consciousness when a handover point is about to arrive, or periodically forcibly requests restoration in order to prevent the driver's consciousness from leaving the steering loop.

[0334] The concept associated with traditional MCD controls is considered to be intuitively annoying to drivers. Therefore, to eliminate this annoyance, some drivers may downplay the alarm's functionality, and may gradually become desensitized to it, developing a habit of immersing themselves in the NDRA and paying little attention to the alarm. This can also lead to situations where drivers ignore system alarms; that is, when the alarm sound is a repetitive, monotonous buzzer sound, the driver will not pay much attention to the sound due to auditory filtering.

[0335] As described in detail above, when a recovery request is presented to the driver, the system presents risk information that has recently affected the driver as appropriately varied information in a multidimensional and variable manner (i.e., different ways). The driver then responds to this prompt by actively reconfirming the "attached conditions." Through this action, the risk information is dispersed to stimulate different areas of the driver's working memory, such as the auditory-language center, visual-language center, and visual cortex, thus making the memory stimulation for the driver's recovery less monotonous.

[0336] As a result, even when awareness and driving are separated, such as during periods of inattention, the degree of forgetting the obligation to resume driving is suppressed. When autonomous driving is initiated in Level 4, information about factors associated with the "transaction" related to the requirement to resume driving according to the obligation is visually displayed. Furthermore, during intermediate steps of autonomous driving in Level 4, new updated information is presented based on the driver's tendency to forget. This information prompts the driver to reassess the risks, thereby reactivating important memories stored in the driver's working memory.

[0337] Another important aspect of HCD is that the contribution of notifications or alarms from the system to the driver's cognitive judgment of driving behavior does not simply function as the intensity of the notification (loudness of the sound, etc.), but rather according to the differences in sensitivity to stimuli that lead to inherent risks to the driver.

[0338] In other words, HCD utilizes a brain mechanism where the intensity of physical stimuli is not important; information that is more important in the immediate judgment is processed first with higher sensitivity and higher priority, while less important information is processed later with lower priority. In an embodiment, the system uses machine learning, such as artificial intelligence, to perform machine learning on how people are presented with specific sets of information, and designs an HMI that encourages the use of influential information for early judgments, which is elevated and fixed as a human characteristic.

[0339] For example, instead of narrowing down the presentation of information by limiting all information to visual information, the system presents information to the driver by applying a comprehensive stimulus using multiple different types of information. Specifically, one could conceive of stimulating the driver by letting them hear a specific sound as an auditory stimulus, followed by a visual notification. When the driver responds quickly and accurately to the stimulus, the system awards the driver credit points as a high-performing driver. Furthermore, awarding credit points to high-performing drivers is not limited to simply adding stored points to mechanical storage media (memory, hard drive, etc.), but involves providing the driver with intuitive visual feedback on-site via an HMI. This intuitively connects to the driver's obligation to make appropriate advance responses based on predetermined timing and circumstances, promoting the optimization of the driver's own reaction behavior, thereby reinforcing the driver's psychological learning.

[0340] In the neurons of the optic nerve, which trigger judgments, synapses, from a microscopic perspective, are stimulated by many key factors. The memory that temporarily stores information requiring attention corresponds to an alert state of ignition standby. To increase sensitivity to relevant information and act quickly upon receiving information necessary for judgment as a necessary stimulus, the storage state of relevant information with recent risks in working memory, known as the memory for behavioral judgment, corresponds to a processing standby state of important matters based on memory. Through numerous and multiple stimulus pathways, an anchoring effect that maintains high priority even during temporary mental drift due to inattention can be achieved, thereby enhancing the retention of necessity through the simultaneous presentation of visual and auditory information indicating risk factors.

[0341] In other words, the driver's act of confirming the "transaction" or "accompanying transaction" presented to them by the system is a reconfirmation of information. From a microscopic perspective, this reconfirmation can be viewed as being performed by activating synaptic potentials in a state prior to the triggering of a judgment. The optic nerve, responsible for judgment, is placed in this ready state, thereby increasing the driver's perceptual sensitivity to subtle information discovered near the end of the ODD (e.g., Level 4 of autonomous driving). This makes the driver more aware of the necessity even when the information is incomplete. Therefore, when memory is insufficient, the driver will voluntarily attempt to supplement the information to mitigate the increase in risk. Thus, the driver aims to complete the "accompanying obligation" based on the initial "transaction." When the driver feels uneasy, for example, this unease is reflected in the act of visually reconfirming the state image related to the transaction.

[0342] <3-2-2-6. Application of Autonomous Driving Level 4>

[0343] Next, the concept of applying autonomous driving level 4 according to embodiments of this disclosure will be explained.

[0344] As the vehicle approaches the end of the usable range of Level 4 Automated Driving, the following issue arises: how to enable the driver to abandon continued use of Level 4 Automated Driving as early as possible and take timely and appropriate measures (recovery actions). In order to interrupt the recovery action that is considered an advantage of the driver's use of Level 4 Automated Driving and switch to manual driving, relevant information related to the transition to recovery action needs to be stored in the working memory of the control decision.

[0345] The opportunity to store relevant information in working memory arises from the "transaction" between the system and the driver when initiating Level 4 autonomous driving. Upon starting this "transaction," the driver temporarily recognizes their responsibilities and obligations. However, if the end of the actual driving interval is not far off in time, the memory needs to be refreshed in order to fulfill the obligation to resume manual driving before reaching the boundary point for taking resuming manual control. In this situation, the presence or absence of information serving as the opportunity and the significance of the risk will have a significant impact on the success or failure of the resumption.

[0346] From an ergonomic perspective, stimuli associated with a specific cause overwhelmingly promote more accurate judgment compared to simple recovery movements without any other reason. Therefore, presenting transfer factors in different ways is useful for successful recovery.

[0347] Assuming autonomous driving reaches Level 4, under satisfactory maintenance and management, when a driver begins using Level 4 autonomous driving in an extended section of road, there is essentially no need to immediately revert to manual control within that section. In the case of low-speed autonomous driving (referred to as low-speed autonomous driving), when autonomous driving exceeds its processing limits, processing time can be gained by slowing down or stopping the vehicle.

[0348] On the other hand, on ordinary roads with a lot of regular passenger traffic, automatic processing must be carried out along this flow without disrupting the flow of traffic in the corresponding road sections. At this point, in stages where automatic processing is anticipated to be difficult, the system needs to choose whether to hand over control to manual driving, thus completing a smooth handover to the driver at a safe cruise speed for autonomous driving. When the estimated success rate is low, the system needs to make a choice, for example, while retaining avoidance as a candidate, whether to take evasive action towards the roadside, service areas, parking spaces, or ordinary roads where parking or low-speed driving is permitted.

[0349] When a regular vehicle is driving at Level 4 of automated driving on roads such as ordinary roads, highways, or main roads, various factors can trigger the interruption of automated driving, such as vehicle characteristics, road conditions, environment, and the driver's driving ability. Here, at points where automated driving becomes difficult to continue, the system is not always able to transfer control to manual driving with 100% probability. Furthermore, as mentioned above, in the event of an emergency stop or deceleration, even if the vehicle itself is not significantly negatively affected, it is very likely to impact following vehicles, i.e., generate significant social impact.

[0350] Therefore, it is necessary that even if the driver has difficulty taking action to return to manual control, the system can determine that the handover has been successfully completed and begin to process control when the vehicle is in a driving range where it has the option to avoid obstacles such as following vehicles.

[0351] The following will illustrate examples of events that may make it difficult to continue using Level 4 autonomous driving.

[0352] Due to factors such as individual vehicles, individual risk awareness and capabilities, Level 4 autonomous driving needs to be compatible with the orderly use of roads as social infrastructure. Here, the system currently struggles to determine suitable uses for an orderly road environment in a manner similar to human cognitive abilities. In another sense, entrusting such a heavy workload to the system may be a matter of basic human aptitude, leading to ethical decisions that ultimately prevent autonomous driving from making appropriate decisions. This is also seen as a rejection of a society where humans are manipulated by machines.

[0353] As an example, when a vehicle can pass through a narrow street after yielding, a human can temporarily interrupt or intervene in the steering of the autonomous driving system to break the entanglement and allow it to pass through the street. As a simple case, when a vehicle is crossing a narrow, single-lane bridge, it communicates in advance with oncoming vehicles and yields to each other, preventing both vehicles from being completely blocked, thus allowing each vehicle to pass through the section.

[0354] In practice, it's sufficient to intentionally incorporate control mechanisms to maintain social order before reaching this point. For example, on a main road, to ensure that even when a vehicle is operating at Level 4 of autonomous driving, it won't affect other vehicles, a decision-making process is preferable. This could involve deciding whether to abandon the planned route, allow the driver to take over manual control, implement remote driving assistance, or preemptively avoid impacting other vehicles. Appropriate decisions are necessary in this situation.

[0355] The following will illustrate examples of factors for abandoning continued use of Level 4 autonomous driving, presented as a list of items. These items, individually or in combination, may be the factors mentioned.

[0356] The first factor is the decision to abandon continuing at Level 4 autonomous driving due to the successful / failed acquisition of prior road environment information related to the destination of the currently used road. Examples of the first factor include the following.

[0357] (20-1) Due to reasons such as malfunctions of vehicles operating in the area where information is collected periodically, there is a lack of very new or updated LDM area information.

[0358] (20-2) Information acquisition failure was caused by major congestion in the communication frequency band used for continuously updating high-freshness LDM.

[0359] (20-3) Local map information based on subscription agreements is subject to usage restrictions due to the expiration of the agreement.

[0360] (20-4) During the process of the leader vehicle following the vehicle with the assistance of the paired leader, continuous data could not be obtained due to defects in the leader vehicle.

[0361] (20-5) Due to the reduced density of vehicles traveling within a specific time zone, there is insufficient shadow detection data from road information of ordinary vehicles.

[0362] (20-6) Communication equipment malfunction in this vehicle, or communication infrastructure malfunction.

[0363] (20-7) Due to information manipulation caused by communication network attacks or data forgery, there is insufficient updated information required for continuous autonomous driving.

[0364] As a second factor, there may be situations where continued driving at Level 4 of autonomous driving is abandoned based on information from updated preliminary road environment information regarding the destination from the currently used road. Examples of factors in this case include the following.

[0365] (21-1) The remote driving assistance controller is handling matters beyond its capabilities.

[0366] (21-2) Insufficient number of remote driving assistance operators

[0367] (21-3) Reception of information such as people and large animals entering the vehicle lane, animals escaping from the luggage rack of a vehicle ahead and walking on the lane, and the scattering of objects falling from a vehicle ahead in an emergency.

[0368] (21-4) Receive information about unpredictable anomalies such as earthquakes, cliff collapses, or tsunamis.

[0369] (21-5) Receive rear vehicle alert information obtained from proactive hazard reports from vehicles ahead in the section.

[0370] (21-6) Localized and accidental icing of road surfaces, such as on damp bridges or in shady areas of mountains.

[0371] (21-7) Traffic restrictions due to unexpected road construction or post-accident procedures

[0372] (21-8) Through human communication, through traffic control, through sections (such as traffic control for accident handling, etc.)

[0373] (21-9) When encountering narrow road sections, alternating one-way bridges, or tunnels during continued autonomous driving, resulting in no areas to avoid, the system enters prohibited sections based on the driver's condition.

[0374] (21-10) Crossing railway crossing sections

[0375] As a third factor, there may be situations where continued driving at Level 4 of autonomous driving is abandoned due to performance limitations of the sensors installed on the vehicle or changes in performance over time. Examples of such situations include the following: In this case, continued driving is abandoned based on the starting point / start time of autonomous driving at Level 4 or changes in conditions during the driving process.

[0376] (22-1) The detection performance of equipment such as millimeter-wave radar, LiDAR, and cameras is reduced due to de-icing agents or dirt kicked up by vehicles ahead.

[0377] (22-2) Preliminary degradation or limitation of sensor camera performance caused by collisions with the windshield by insects or flying objects during high-speed driving.

[0378] (22-3) During use, the noise and detection performance deteriorate due to the increase in temperature.

[0379] (22-4) Partial window breakage caused by pebbles kicked up by the vehicle in front or objects flying from the adjacent lane.

[0380] (22-5) When driving at night, damage to vehicle headlights and other components, as well as a reduction in the camera's field of view detection limit, can occur.

[0381] (22-6) Due to misoperation of indoor air conditioning, the windshield fogged up, and the detection performance of the indoor sensor camera initially declined.

[0382] (22-7) Recovery request for unknown causes based on the self-diagnostic results of the vehicle system

[0383] As a fourth factor, the following situations may occur: The continuation of Level 4 autonomous driving may be abandoned due to changes in the condition of the cargo affecting the vehicle's movement or other vehicle dynamics during continued autonomous driving. Examples of this situation include the following.

[0384] (23-1) Load collapse during normal driving

[0385] (23-2) Tire leak or blowout

[0386] (23-3) Passengers experiencing scattering, changes in driving performance due to road abnormalities, and unusual noises occurring in the vehicle.

[0387] (23-4) Load collapse caused by emergency braking to prevent a collision, significant passenger movement within the vehicle, and loss of load balance related to the movement.

[0388] (23-5) Detection of braking abnormalities in the self-diagnosis of a moving vehicle

[0389] (23-6) Engine overheating, control equipment malfunction

[0390] As a fifth factor, the following situations may occur: The driver abandons Level 4 autonomous driving due to abnormal behavior. Examples of this situation include the following.

[0391] (24-1) Necessary recovery cannot be predicted due to the driver unexpectedly falling asleep, not being in their seat, or status detection failure.

[0392] (24-2) Sudden onset of symptoms in the driver (asthma, leg cramps or numbness, allergic reaction caused by pollen or other substances suddenly entering the vehicle, heart attack, sudden headache, stroke, cerebral infarction, etc.)

[0393] (24-3) Detecting abnormal behavior caused by drug addition

[0394] (24-4) Interference with driver status monitoring performed by the system

[0395] (24-5) During the continued use of autonomous driving, the driver ignores or disregards the system's necessary response to the driver's requests.

[0396] (24-6) Information about the driver's rest, activities the previous day, rest schedule, and routine history.

[0397] Each of the first through fifth factors mentioned above may involve a combination of conditions and contributions. Continuing to operate at Level 4 autonomous driving under these combinations requires avoiding situations including vehicle deceleration or emergency braking, reduced efficiency of social traffic infrastructure, or significant disruptions.

[0398] Regarding at least the first through fourth factors mentioned above, when a driver fails to take corrective action based on different conditions due to risk assessment, the driver will receive initial feedback information through the HMI, such as the degree of impact or the applicable penalty for the violation. This HMI can present sensory performance, such as the use of visual stimuli. For example, through the initial feedback from the HMI, the driver's visual stimulus is captured in working memory at least once, and the stimulus is temporarily retained as a visual sensory stimulus. By providing the driver with a stimulus related to this visual sensory stimulus, the memory can be refreshed and the memory required for corrective action can be maintained.

[0399] On the other hand, as a fifth factor, when the driver experiences an abnormality, it will be extremely difficult for HCD to prompt the driver to take early recovery action. However, even in this situation, by presenting the driver with a function to disengage from autonomous driving as early as possible, the system can utilize this application and, based on information obtained through this function regarding avoidance locations, issue a proactive disengagement request or rescue request to the driver. Compared to a system that does not provide any information to the driver and allows the vehicle to continue driving to the handover limit, only activating MRM after the vehicle reaches the handover limit, this application can provide control with an initial processing margin.

[0400] Note that the examples shown above are representative examples of the processing required by major arterial roads and other social infrastructures, where problems may arise when the vehicle stops in the driving zone of the road. For roads with very low traffic volume, or even wide roads that are not arterial roads, the likelihood of traffic congestion is low even if the vehicle stops suddenly or completely due to MRM, and the processing will not be subject to the limitations of the above.

[0401] In other words, when a driver continues driving at Level 4 of automated driving, if the area is one where emergency stopping or evacuation can be performed via MRM without hindering social activities, the system can continue driving according to the predetermined plan, regardless of the feasibility of actions related to driver recovery or the driver's condition. In other words, even if the system encounters a situation that is difficult to handle at Level 4 of automated driving and the driver cannot properly recover to bring the vehicle to an emergency stop or evacuate, it will not cause traffic congestion on roads, which are part of social infrastructure.

[0402] In this way, the appropriate circumstances for using Level 4 autonomous driving are not uniformly determined or fixed environments. Due to various factors, such as the processing capabilities related to requesting driver recovery, road conditions, usage environment, and vehicle status, these circumstances will actively change.

[0403] <3-2-2-7. The Effect of Using HCD>

[0404] Next, we will explain the effects of replacing the existing MCD with an HCD, as described above.

[0405] First, the essence of the HCD control according to this disclosure is that the available range of the driver's automated driving is variable, and the determination of the available range depends not only on observable assessment values ​​such as the driver's alertness at the observation time, but also on the acquired driver credit information. Furthermore, in the HCD according to this disclosure, during the intermediate process of determining the available range, at least visual representations are used to present the information detected by the system and its results as recent risk information to the driver.

[0406] Unlike simple recovery request notifications uniformly presented in the system, information is presented as the risk of the driver's own chosen action, thus achieving a thoughtful selection operation that balances advantages and disadvantages. Through this operation, information related to its importance in the driver's working memory is acquired, and its freshness is maintained based on the importance of the transfer. Furthermore, the evaluation of this thoughtful selection operation will be reflected in future usage conditions, and the current usage permission will also be determined based on past evaluation results to determine whether the benefits of autonomous driving can be obtained. Therefore, in HCD, the driver can gain a sense of responsibility and enjoyment of benefits through repeated use, different from the machine instructions in MCD.

[0407] Furthermore, through repeated use, drivers learn appropriate methods. Additionally, if a driver's behavior interferes with social activities, the system penalizes them and inhibits further use. Therefore, it can help drivers avoid undesirable behaviors that become obstacles, thereby suppressing socially unacceptable forms of use.

[0408] Secondly, autonomous driving can significantly reduce the human intervention load in driving and steering. Therefore, in today's social environment, it is estimated that about 94% of accidents are caused by human factors. When autonomous driving mode replaces human drivers, the occurrence of social accidents is expected to be greatly reduced.

[0409] However, with the widespread consideration of introducing autonomous driving today, research has been conducted on the premise that the driver appropriately reverts to manual control in response to the system's requests and that the driver can respond quickly to such requests. However, this premise may not hold true if improving the performance of autonomous driving systems could lead to excessive driver dependence on them.

[0410] In this disclosure, the availability of autonomous driving functions is essentially based on the successful establishment of appropriate driver use suitability. Furthermore, an HMI (Human-Machine Interface) that incentivizes appropriate use is included in this mechanism. This enables control techniques designed to prevent excessive driver reliance on the system and allow the driver to subjectively take appropriate corrective actions.

[0411] That is, this disclosure relates to vehicle control in which, by introducing a modified HMI, the method of controlling the use of autonomous driving has been changed from a unilateral MCD that issues commands from a conventional device to an HCD that performs use control based on human behavioral characteristics. Specifically, when the driver uses the system's autonomous driving function, the driver exchanges a "transaction" with the system. This "transaction" is "ceremonial" and represents a "confirmation action" taken by the driver without neglecting to complete the autonomous driving request, and the validity of this transaction is appropriately reconfirmed during the use of autonomous driving.

[0412] Such an HCD-based system cannot be achieved by simply incorporating certain functions into the system; instead, it requires reinforcement learning through repeated, complex uses necessary for drivers to form usage habits. The embodiments describe an HMI that motivates the driver to perform reinforcement learning and maintain appropriate early recovery as a sense of use over a long period. Note that the combination of means of performing feedback actions on the driver is not limited to the examples described in this specification.

[0413] Traditional autonomous driving systems mechanically determine the permissible level of automation based on the road conditions the device can handle, requiring the driver to make timely and appropriate adjustments based on their own judgment. In contrast, this disclosure provides development support for drivers to develop the habit of appropriately initiating recovery procedures without delay through repeated responses to system requests, and includes the necessary HMI (Hardware Management Interface) for providing this support. Therefore, even with the widespread adoption of autonomous driving, the system will have the following effects: significantly reducing emergency deceleration and stopping on the road due to delayed driver recovery operations, further suppressing significant traffic disruptions when the vehicle is stopped, and preventing obstruction of social activities.

[0414] Third, in order to use automated driving systems that involve frequent handovers to manual control, it is necessary to understand the characteristics of the driver's working memory in order to safely continue driving operations.

[0415] Even if a person perceives and recognizes the necessary processing events and temporarily stores the information in working memory, loss of consciousness may occur due to states of mental impairment, autonomic nervous system imbalance, or other factors. This can lead to inattentiveness towards other events. Consequently, the importance of important matters may fade from memory, resulting in delays in problem-solving and potentially serious consequences.

[0416] The amount of information allocated to a person's mind is limited, and it's impossible to consider everything comprehensively at the same time. Therefore, when multiple different systems learn information, using different means (e.g., visual or linguistic) to process information input from different systems, if each outcome can be described, it's easier to bring the mind back to the necessary processing items even if there are lapses in concentration. That is, HMI will be effective in clarifying the memory of recovery requests. In HMI, the necessity of transfer is not presented in a dry form of information such as simple symbols, but specifically as a sense of risk, as information presented as a prediction of the outcome leading to a certain impact.

[0417] When HCD is used for control, information that entered working memory during work may be lost due to a shift in thinking while attending to other matters, and information acquired as essential may be forgotten over time. Therefore, it is necessary to provide memory-refreshing feedback to the driver based on their current level of forgetfulness. This disclosure evaluates the driver's "forgetfulness" regarding important matters as a daily characteristic and the "freshness of the driver's memory of driving-related precautions during driving," and then presents memory-refreshing information suitable for that driver.

[0418] Furthermore, as an embodiment of this disclosure, locations where assistance is unavailable can be identified and predicted in advance when using remote assistance. Thus, by providing the driver with information such as sufficient shoulder area and service area (SA) via the HMI, it becomes easy to select a standby location that does not obstruct traffic elsewhere when assistance is not received, and a mechanism can be established even with a small number of remote assistance operators, thereby providing practical and effective remote assistance. In addition, the infrastructure required for remote monitoring can be operated efficiently.

[0419] Fourth, when the driver is performing secondary tasks other than driving, namely NDRA (Non-Driving Assistance), which primarily involves visual information from electronic terminals, attention can be enhanced through the following measures. Specifically, in terminal devices with monitoring screens, short-term information related to autonomous driving can be presented in the display image area originally designed for NDRA, so as to display visual information relevant to the necessity of handover.

[0420] For example, visual information can be displayed for a very short period of time to achieve what is known as a subconscious effect, which the viewer cannot consciously notice. In addition to cues that remain entirely outside of conscious awareness, there can also be cues that are longer and more clearly conscious for the driver than those targeting a subconscious effect.

[0421] In cases where information is not necessarily understood through language (e.g., subconscious effects), the sense of risk, as a visual and intuitive feeling, acts more effectively on the driver as a risk that occurs when a handover request is ignored. For example, descriptions such as: a visual description of the user being monitored by a roadside police motorcycle in the event of a violation, rather than a description of written information including penalty regulations; a description of the user receiving a tracking stop order as a violation control measure, rather than static information including an image of a police booth; a description of the situation requiring confirmation of the violation; and a description of the risk the user faces in the event that MRM cannot continue.

[0422] People consciously capture visual information and interpret it verbally. On the other hand, academics believe that humans possess an information transmission mechanism that can function in the brain without any verbal interpretation or conscious intervention.

[0423] These short-term stimuli, such as those caused by subconscious effects, are expected to refresh and reactivate important information that has almost disappeared from working memory. Unlike the effects of the system in waking up from a decline in consciousness caused by fatigue or drowsiness, this effect has the capacity to restore stored information to the working memory required for conscious judgment.

[0424] When people recall important information, there are instances where they realize the necessity of responding, forget the necessity of responding, and fail to recall important information at the crucial moment, only remembering it later.

[0425] This is because important information that needs to be processed is not prioritized for storage in working memory and cannot be retrieved from memory. To make stored information function effectively in judgment, it is necessary to increase the amount of stimulation to the memory. Similar to blindsight, subconscious effects, such as the involuntary perception of information as visual, influence behavioral judgment even when it is not consciously perceived as visual information. However, since suppressing the display is not the primary objective, subconscious effects are ultimately an example of short-term HMI, and can be used for longer displays that act on consciousness, or, moreover, can continue to be displayed until the driver cancels the display, thus functioning as a risk description with greater intensity.

[0426] That is, when using autonomous driving, if a handover request to manual control occurs between the system and the driver due to a change in circumstances, the system can continue using autonomous driving as long as the "transaction" corresponding to the handover request is valid. To ensure that the driver performs the "responsibility to resume" attached to the "transaction" without fault when utilizing the autonomous driving function based on the "transaction" and taking advantage of the benefits of NDRA during autonomous driving, it is effective to use some kind of stimulus reminder. When using an electronic terminal, displaying this stimulus on the application screen of the electronic terminal is effective. In this case, the following effect exists: the driver is made "noticed" of the "responsibility to resume" attached to the "transaction" without unduly hindering the user's browsing operations on screens related to NDRA benefits.

[0427] <3-2-3. Specific Examples of HCD According to the Embodiments>

[0428] The HCD according to the embodiments will now be described in more detail. In the following, unless otherwise specified, it is assumed that autonomous driving is Level 4 autonomous driving as defined by SAE.

[0429] <3-2-3-1. Example of Automated Driving Using HCD According to an Embodiment>

[0430] refer to Figures 7A to 7C The flowchart below will illustrate in more detail an application example of autonomous driving using HCD according to an embodiment. Figures 7A to 7C In the attached figures, reference numerals "A" through "F" indicate that the processing is transferred to... Figures 7A to 7C The corresponding processing of the reference numerals in the flowcharts of other figures in the document. Figures 7A to 7C In flowcharts, boxes that typically represent "documents" indicate that information is being provided to the driver through visual stimuli or other means.

[0431] Figure 7A This is a flowchart illustrating an example of the process from setting a driving route to switching to autonomous driving mode according to an embodiment. Note that the driving route described herein represents the vehicle's driving plan and includes information indicating the starting point and destination of the journey, as well as information indicating the driving route itself. Furthermore, starting the driving route means beginning to drive according to the driving route.

[0432] Step S100 sets the driving route, including the destination, as set by the vehicle user (driver). The set driving route is input into the automatic driving control unit 10112 (see reference). Figure 1 The autonomous driving control unit 10112 acquires various information necessary for driving according to the input driving route, such as the LDM (Local Driver Management Module). For example, in step S101, the autonomous driving control unit 10112 acquires information such as the LDM, the driver's ability to revert to manual control driving, the weather conditions in the areas included in the driving route, and the cargo loaded on the vehicle. Among these characteristics, for example, characteristics based on an evaluation of the driver's past actions of reverting to manual control driving can be applied as characteristics for the driver to revert to manual control driving.

[0433] In the next step S102, the autonomous driving control unit 10112 presents a bird's-eye view of the entire driving route to the driver. Although specific examples will be described below, the autonomous driving control unit 10112 may generate, for example, map information that visualizes the entire driving route based on the LDM (Level 4 Autonomous Driving Model) and information indicating the sections of the driving route where the vehicle can drive at Level 4 autonomous driving. For example, the autonomous driving control unit 10112 provides the generated display information to the output unit 10106 via the output control unit 10105, and causes the display device connected to the output unit 10106 to display an image based on the display information.

[0434] This display is a navigation display showing the driving route set by the system (i.e., the automatic driving control unit 10112). Note that the overhead view display mentioned here does not need to be a three-dimensional overhead view with an adjustment scale based on physical distance, and can be a time-transition display, a stereoscopic display, or other display, as long as the driver can identify the intervention point.

[0435] In the next step S103, the autonomous driving control unit 10112 asks the driver whether they agree to the driving route recommended by the navigation display presented in step S102. For example, the autonomous driving control unit 10112 determines whether to agree based on the driver's operation on the input unit 10101. The determination method is not limited to this, and the autonomous driving control unit 10112 can use a camera for capturing images of the vehicle's interior to detect the driver's movements and determine whether the driver agrees based on the detected movements, or it can determine this based on the driver's voice.

[0436] If, in step S103, it is determined that the driver disagrees with the recommended settings (step S103, "No"), the autonomous driving control unit 10112 proceeds to step S104. In step S104, the autonomous driving control unit 10112 adds another recommended driving route and provides the driver with the option to select the other driving route. Subsequently, the autonomous driving control unit 10112 returns to the processing of step S102 and presents the driver with a bird's-eye view of the entire driving route of the other route.

[0437] Conversely, when it is determined in step S103 that the driver agrees to the recommended settings (step S103, "Yes"), the automatic driving control unit 10112 proceeds to step S105. At this time, when the driver accepts and agrees to the driving route suggested by the system, the driver grasps the concept of the entire driving route, and stores this fact as stored information #1 (WM10) in the driver's working memory.

[0438] Step S105: The driver begins driving the vehicle to start the driving route. The automatic driving control unit 10112 updates the bird's-eye view of the driving route along the vehicle's route from the start of the driving route, and presents the updated bird's-eye view to the driver (step S106). At this time, the automatic driving control unit 10112 controls the automatic driving of each section in the driving route, and calculates the automatic driving mode of each ODD corresponding to each section in chronological order.

[0439] At this point, by confirming the updated bird's-eye view presented by the automatic driving control unit 10112, the driver is able to grasp the status of the current driving route and the obligation to revert to manual control driving based on the latest choice. The information obtained is stored in the driver's working memory (WM11) as storage information #2.

[0440] In the next step S107, the autonomous driving control unit 10112 determines whether it is approaching an ODD (Operation Distance Domain) zone that allows autonomous driving. If it has been determined that it is not approaching the zone (step S107, "No"), the autonomous driving control unit 10112 proceeds to step S108, continues to monitor changes in the situation, updates various risk information based on the monitoring results, and returns to step S106.

[0441] Conversely, if it has been determined that the driver is close to the ODD (step S107, "Yes"), the autonomous driving control unit 10112 proceeds to step S109. In step S109, the autonomous driving control unit 10112 presents a transaction related to the response to the ODD to the driver and determines whether the driver has consented to the agreement. For example, the transaction includes various conditions that allow autonomous driving within the ODD. The autonomous driving control unit 10112 makes this determination, for example, based on whether the driver has performed any operation or action indicating consent to the presented transaction. If it has been determined that consent has not been obtained (step S109, "No"), the autonomous driving control unit 10112 returns to step S106.

[0442] If it has been determined that consent to the transaction has been obtained in step S109 (step S109, "Yes"), then the automatic driving control unit 10112 allows automatic driving in the ODD and proceeds to step S110. In step S110, when the vehicle enters the ODD area where automatic driving is permitted, the automatic driving control unit 10112 switches the driving mode from manual control driving mode to automatic driving mode.

[0443] When the driving mode is switched to automatic driving mode in step S110, the driver is forced to return to manual control driving in response to the selection in step S109. Understanding the agreement with the system (details of the transaction) indicates that the driver and the system have reached an agreement on risk handling at the end of the ODD (Operational Distance Term). Violations of the obligation to resume driving involve penalties for the driver. Information #3-1, #3-2, ... indicating the conditions included in the agreed transaction is stored as storage information #3 in the driver's working memory (WM12). Additionally, stored information #3 is stored as related transactions when using automatic driving, as information related to countermeasures in the event of a newly occurring unplanned accident along the driving route (WM13).

[0444] When a driving mode is selected and switched to automatic driving mode in step S110, the process proceeds to step A in the attached diagram. Figure 7B The flowchart shown illustrates the processing.

[0445] Figure 7BThis is a flowchart illustrating an example of a processing flow in autonomous driving mode according to an embodiment. The processing starts from... Figure 7A Step S110 is performed. Figure 7B Step S120. In step S120, the autonomous driving control unit 10112 performs continuous monitoring of status changes, updates various risk information based on the monitoring results, and proceeds to step S121.

[0446] In step S121, the autonomous driving control unit 10112 determines whether an event requiring driver intervention is occurring based on the status monitoring result in step S120. If it is determined in step S121 that no event has occurred (step S121, "No"), the autonomous driving control unit 10112 proceeds to step S122.

[0447] In step S122, the autonomous driving control unit 10112 determines whether it is approaching the end point of an ODD (Operational Domain Controller) segment that allows autonomous driving. If it is determined that it is not approaching the end point of the segment (step S122, "No"), the autonomous driving control unit 10112 returns to the processing in step S120. Conversely, if it is determined that it is approaching the end point of the segment (step S122, "Yes"), the autonomous driving control unit 10112 proceeds to the processing in step S123.

[0448] The loop processing in steps S120 to S122 shows the processing in the ODD range that can be stably applied to autonomous driving level 4.

[0449] In step S123, the automatic driving control unit 10112 notifies the driver that the handover point for transferring driving from automatic to manual control is approaching. In the next step S124, the automatic driving control unit 10112 monitors the driver's actions related to the transition from automatic to manual driving mode, i.e., the quality of the driver's handover operation from automatic to manual control, and adds / subtracts a score from the driver's evaluation score based on the monitored quality. The monitoring of the handover operation's quality and the calculation of the quality evaluation adjustment score will be explained below.

[0450] In the next step S125, the automatic driving control unit 10112 determines that... Figure 7A The system checks whether the entire driving route set in step S100 has ended. If it determines that the entire route has ended (step S125, "Yes"), then the automatic driving control unit 10112 terminates. Figures 7A to 7C The flowchart in the process involves a series of processes. Conversely, if it is determined in step S125 that the entire driving route has not ended (step S125, "No"), then the automatic driving control unit 10112 proceeds after symbol "B". Figure 7AThe processing of step S106 in the flowchart.

[0451] If it is determined in step S121 above that an event requiring driver intervention has occurred (step S121, "Yes"), then the automatic driving control unit 10112 will proceed after reference numeral "D" in the figure. Figure 7C The processing of step S130 in the flowchart.

[0452] Figure 7C This is a flowchart illustrating an example of a response to an event occurring during autonomous driving at Level 4, according to an embodiment. In step S130, the autonomous driving control unit 10112 notifies the driver of a new event. In the following step S131, the autonomous driving control unit 10112 determines the urgency of the new event. For example, the autonomous driving control unit 10112 determines the urgency based on the distance between the location where the vehicle is currently traveling and the location where the new event occurred. This is largely synonymous with determining urgency based on the time margin before the vehicle reaches the location where the new event occurred.

[0453] When the distance to the location of a new event is less than or equal to a predetermined distance and the time margin is small, the autonomous driving control unit 10112 determines that the urgency of the new event is high (step S131, "high") and proceeds to step S160. In step S160, the system initiates MRM and forcibly executes deceleration of the vehicle and movement to an evasive location such as the roadside. When MRM is initiated in step S160, the system temporarily terminates the process based on... Figures 7A to 7C A series of processes in the flowchart.

[0454] Additionally, the driver's evaluation is set to a lighter deduction in step S131 and the processing in step S160 is the deduction object (1) described below.

[0455] In step S131 above, when it is determined that the distance to the location of the new event is within a predetermined range (the distance is longer than the case where the urgency is high and shorter than the case where the urgency is low) and there is a certain amount of time margin, the autonomous driving control unit 10112 determines that the urgency is at a medium level (step S131, "medium") and proceeds to step S132.

[0456] In step S132, after obtaining the deceleration time in advance, the automatic driving control unit 10112 confirms the vehicle and surrounding conditions (e.g., whether there are vehicles behind), and predicts the impact on the surrounding environment when the vehicle reduces its speed. Additionally, the automatic driving control unit 10112 checks the driver's condition and observes whether the driver can quickly switch back to manual control. Based on this observation, the automatic driving control unit 10112 predicts the delay in the driver's return to manual control.

[0457] In the next step S133, the autonomous driving control unit 10112, based on the prediction result in step S132, determines whether the grace time for the action of resuming manual control driving from autonomous driving can be extended. If it is determined that the grace time can be extended (step S133, "Yes"), the autonomous driving control unit 10112 proceeds to step S140. Conversely, if it is determined that the grace time cannot be extended (step S133, "No"), the autonomous driving control unit 10112 proceeds to step S134.

[0458] Although the details of the control are not specified, when a new accident or event occurs on the driving route due to changes in the driving conditions, requiring manual control of the driving, in order to forcibly handle the emergency within a short time margin, extending the arrival time by using emergency deceleration or other methods will increase the risk of secondary injuries such as rear-end collisions or traffic congestion, and is not necessarily safe.

[0459] Therefore, it is necessary to assess the handling strategy and re-evaluate the driving plan, including examining whether the vehicle's deceleration within the road section impacts road infrastructure. Specific examples will be used to illustrate the usefulness of this judgmental approach of re-evaluating the driving plan.

[0460] The example to be examined is the following situation: While traveling along a route indicated by the driving trajectory at the maximum permissible speed within the road section, the system determines that Level 4 autonomous driving is difficult to continue at speeds within the performance limits of the autonomous driving system. In this case, if the road area is a two-lane road with sparse traffic (i.e., traffic is light) and the road is straight, slow deceleration is considered to have a minor impact on road traffic. In this situation, deceleration can be determined as the optimal choice.

[0461] As another example, when prior monitoring has revealed that a driver is having difficulty returning to manual control due to poor physical condition, and when a busy section of road with many curves and poor visibility is approaching the driver's current section of road, it may be safer to slow down in advance on a straight section of road.

[0462] In step S134, the automatic driving control unit 10112 urgently initiates the braking of the MRM. For example, when MRM is initiated, the automatic driving control unit 10112 issues a warning notification in advance to inform the surrounding area of ​​the vehicle of the initiation of MRM. Furthermore, the automatic driving control unit 10112 instructs the driver to prepare a posture (gesture) for responding to MRM. After the processing of step S134, the automatic driving control unit 10112 proceeds to step S160 to initiate the system's MRM.

[0463] In addition, the transition process from step S134 to step S160 is for deducting points from the driver's evaluation, and the deduction object (2) will be described below.

[0464] In step S131 above, when the distance to the location of the new event is greater than or equal to the predetermined distance and there is sufficient time margin, the automatic driving control unit 10112 determines that the urgency is low (step S131, "low") and proceeds to step S140.

[0465] In step S140, the autonomous driving control unit 10112 adds the new event to the bird's-eye view of the ODD region and updates the bird's-eye view. In the next step S141, the autonomous driving control unit 10112 notifies the driver of the new event and observes the driver's response to the notification.

[0466] In the next step S142, based on the driver's response observed in step S141, the autonomous driving control unit 10112 determines whether the driver accepts the added new event, that is, whether the agreement regarding the transaction in the ODD interval has been updated. If it is determined that the driver has updated the agreement (step S142, "Yes"), the autonomous driving control unit 10112 proceeds to step S143.

[0467] In step S143, the automatic driving control unit 10112 adds incentive points as an evaluation of the driver based on the driver's perception of the positive notification. Subsequently, the automatic driving control unit 10112 proceeds after the reference numeral "E". Figure 7B The processing of step S122 in the flowchart.

[0468] The transition from step S143 to step S122 is considered by the driver as an addition of a handover event because the driver has already made an intentional cognitive response to the notification. This is information that operates on working memory and is expected to be processed appropriately by the driver. The same applies to the processing from step S149 to step S122, as described below.

[0469] Note that the transition from step S142 to step S143 signifies that the driver has consented to the transaction presented by the system and consciously permitted the autonomous driving plan. Therefore, information related to this transaction is stored in the driver's working memory as stored information #4 (WM14).

[0470] Conversely, if it is determined in step S142 that the driver has not updated the agreement (step S142, "No"), the automatic driving control unit 10112 proceeds to step S144. In step S144, based on the driver's state, the automatic driving control unit 10112 observes whether the driver has accepted a notification from the system. In the following step S145, based on the observation results in step S144, the automatic driving control unit 10112 determines the validity of a forced resumption notification that prompts the driver to forcibly return to manual control driving. Additionally, the automatic driving control unit 10112 calculates the refuge location with the least impact upon resumption.

[0471] In the next step S146, the automatic driving control unit 10112 determines whether there is a grace period from the current time until the moment when driving should resume manual control. If a grace period is determined to exist (step S146, "Yes"), the automatic driving control unit 10112 returns to the processing in step S144. For example, if the driver is performing NDRA (Non-Driving Assist) which deviates significantly from driving (e.g., taking a nap), the driver may not recognize mild notifications such as displays or warning sounds. Therefore, the automatic driving control unit 10112 repeats the processing from steps S144 to S146 until the grace period ends.

[0472] If, in step S146, it is determined that there is no grace period from the current time point to the point of resuming manual control of driving (step S146, "No"), then the automatic driving control unit 10112 proceeds to step S147. In step S147, based on the determination result regarding the validity of the forced resumption notification in step S145, the automatic driving control unit 10112 adds a resumption point and progressively notifies the driver of the resumption point. For example, the automatic driving control unit 10112 progressively issues initial warnings and notifications to the driver.

[0473] When a new, unplanned event is communicated to the driver, if the driver is in a state such as asleep at the time of the event, the driver will have no memory of the events that constitute the new handover point, the necessity of the handover, or the urgency of the handover. Therefore, to prevent the driver from panicking, unlike planned handovers, it provides some advance notice and alerts to give the driver time to think and understand the situation.

[0474] This is because, as mentioned above, for recovery operations that occur earlier than initially anticipated due to new events, the recovery point information is not stored in the driver's memory of the necessity of recovery, and the information prompting a return to manual control driving is also not yet stored in the driver's working memory. The threshold for this judgment is the point where the request recovery ratio (RRR) is high when MRM is activated; this point is one with a time margin α greater than the threshold that ensures the driver can recover without obstructing traffic flow on the main road. For example, this corresponds to the time before the driver recovers from a nap. Here, if the quality of the driver's recovery from a nap is poor, the RRR is high, and the vehicle is approaching a section with a risk of causing traffic congestion, the automatic driving control unit 10112 performs preliminary measures via MRM before reaching that section.

[0475] RRR represents the expected probability of completing the handover at the handover limit point when the driver is asked to return to manual control.

[0476] RRR will be explained in more detail. Ideally, the driver at [1 / 1] is expected to complete the handover normally at the handover limit point. When representing the success rate, RRR is defined as [1 / 1].

[0477] However, in rare cases, the handover is not successful in practice. For example, on a certain road segment, if allowing 5 out of 10,000 drivers to fail is not possible, the required RRR for that segment is a ratio expressed as [1-0.0005 / 1].

[0478] RRR is an indicator representing the target value for successful handover defined for each road segment. It aims to ensure that, among various dynamic information defined as LDM (Local Road Management) in terms of physical information, for each lane of a road segment, when a vehicle stops within the segment due to the activation of MRM (Movement Management), it will not cause rear-end collisions or traffic congestion, and the vehicle will not need to make an emergency stop on a one-way street. The goal is for RRR to serve as a judgment factor that accompanies LDM and dynamically changes according to time-related conditions.

[0479] As a specific example of RRR in Japan, in sections of road without shoulders or other refuge areas, such as metropolitan expressways, especially when refuge areas are already filled by vehicles that have arrived earlier, RRR is preferably set to [1]. On the other hand, when there is refuge space at refuge areas, or when it is unavoidable for the driver to recover before they can evacuate to the exit of a highway to the general road, as part of MRM, the driver can choose to steer to these refuge areas, get off onto the general road, and stop, while minimizing the impact on other following vehicles. Therefore, it is conceivable to set the recovery request rate to, for example, around 0.95. Furthermore, in sections of road with very low traffic volume, if the impact of an emergency stop in that section only involves the vehicle itself, then RRR can also be [0].

[0480] Basically, in order to mitigate the traffic congestion caused by MRM to social infrastructure, RRR, as part of LDM, should ideally provide and keep up-to-date with information to vehicles using autonomous driving.

[0481] After completing step S147, the autonomous driving control unit 10112 proceeds to step S148. In step S148, the autonomous driving control unit 10112 determines whether the driver has recognized the preliminary warning and notification in step S147.

[0482] In step S148, if a predetermined response from the driver to the alarm and notification (operation on input unit 10101, specific action, etc.) is detected and it is determined that the driver has recognized the alarm and notification (step S148, "Yes"), then the automatic driving control unit 10112 proceeds to step S149. In this case, since the driver responded to the call notification in advance, normal handover processing can proceed.

[0483] In step S149, based on the driver's handling quality, the automatic driving control unit 10112 increases or decreases incentive points as the driver's evaluation. Subsequently, the automatic driving control unit 10112 proceeds after reference numeral "E". Figure 7B The processing of step S122 in the flowchart.

[0484] The transition from step S148 to step S149 indicates that the driver has responded to the call notification in advance, thus enabling the driver to transition to normal handover processing. Therefore, information indicating that the driver has recognized the initial alarm or notification is stored in the driver's working memory as stored information #5 (WM15).

[0485] Here, the aforementioned stored information #5 stored in working memory via WM15 and stored information #4 stored in working memory via WM14 are shown by reference numeral "C" in the figure, applicable to... Figure 7AThe processing is done through WM13.

[0486] If it is determined in step S148 that the driver has not recognized the alarm and notification (step S148, "No"), then the automatic driving control unit 10112 proceeds to step S150. In step S150, the automatic driving control unit 10112 determines whether there is sufficient leeway for waiting for the driver to regain awareness. If it is determined that there is sufficient leeway for standby (step S150, "Yes"), then the automatic driving control unit 10112 returns to the processing of step S132 in the figure according to the reference numeral "F".

[0487] Conversely, if it is determined in step S150 that there is no margin (step S150, "No"), the automatic driving control unit 10112 proceeds to step S160. The transition from step S150 to step S160 is a process performed due to the driver's recovery timeout, including the execution of software MRM. This situation corresponds to the following deduction object (3), in which points are deducted from the driver's evaluation based on the driver's delay and negligence in recovery.

[0488] Here, the details of the processing in steps S144 to S148 above will be explained.

[0489] When Level 4 autonomous driving is available, the driver can engage in NDRA (Deviation from Driving Direction) within the zone corresponding to the ODD (Operational Departure Distance) zone, and can, for example, take a short nap or move to the cargo compartment.

[0490] Specifically, here is a hypothetical scenario: In situations with significant deviation from the loop (e.g., during a nap), within an ODD (Operational Distance Controller) zone of Level 4 Autopilot, specifically within a zone comprising, for example, a straight road followed by several consecutive curves after more than 10 minutes of driving on that straight road, a minor accident occurs in the straight section. An example of such an accident is an insect impact on the windshield. An insect impact can soil the windshield, causing problems with forward visibility.

[0491] Here, even if insect impacts soil the windshield, it is safe to drive at Level 4 autonomous driving on straight sections of road. However, in the subsequent curved sections following the straight sections, the road curves significantly. Therefore, with a dirty windshield, the vehicle will be in a state unsuitable for Level 4 autonomous driving. This may lead to a reassessment of the Operating Design Domain (ODD) based on this change in conditions, and Level 4 autonomous driving may become difficult. In this situation, for safety reasons, the system first notifies the driver of the transition from autonomous to manual driving earlier than the normal handover time. Furthermore, the system needs to observe the driver's response to this notification and take the measures described below.

[0492] In other words, similar to the concept of Level 3 Automated Driving, the areas permitted for Level 4 Automated Driving can be defined as zones where vehicles meeting the necessary conditions are allowed to operate at Level 4 Automated Driving, rather than zones where all vehicles designed to include Level 4 Automated Driving capabilities can always operate at Level 4 Automated Driving. For drivers to be able to appropriately handle situations based on these conditions, the system must provide drivers with sufficient time to make judgments and must allow drivers to proactively manage situations.

[0493] At this point, given that there is sufficient time before the system reaches the point where it needs to be restored due to changes in conditions, from the driver's perspective, the system's command to the driver to forcibly interrupt NDRA and the system's request to the driver to return to manual control driving are useless restoration requests.

[0494] In reality, it's troublesome for the driver to interrupt NDRA since they've already moved to the loading platform or are napping. Furthermore, because immediate recovery isn't required, confirming early changes in system conditions is simply a risk-free, unnecessary, and tedious task for the driver. Therefore, unnecessary repeated requests only increase the driver's sense of futility and exacerbate the filtering effect of the aforementioned notifications, causing the importance of the process to be gradually overlooked.

[0495] To prevent unnecessary early detection and excessive risk, step S146 determines the appropriate timing and time margin for issuing a notification or alarm to prepare for recovery steps. Depending on whether the driver has recognized the notification or alarm, step S148 determines actions to take in the event of a delay in the driver's response, such as whether to initiate a process equivalent to a normal recovery procedure.

[0496] The processes described in steps S144 to S148 are for the purpose of achieving this control.

[0497] Note that the reference values ​​for parameterized general passenger vehicles, heavy vehicles and vehicles carrying dangerous goods, large shared vehicles, such as the safety factor corresponding to the characteristics of the vehicle, the RRR target value obtained in the road section, etc., can be based on the grace time reference in step S146.

[0498] Information presented to the driver by the system via the information display unit 120 is stored in the driver's working memory as information for risk assessment, and is retrieved from the working memory according to the driver's understanding of the importance of the handover, so as to prompt the driver to make a behavioral judgment.

[0499] <3-2-3-2. Evaluation of the driver's recovery behavior>

[0500] Here, the evaluation of the driver's recovery behavior according to the embodiment will be explained in more detail. The deduction objects (1) to (3) will be explained with reference to Table 2, that is, the objects on which the driver's evaluation value is deducted when the process proceeds to the step S160 of starting the above-described MRM.

[0501] Table 2

[0502] Bonus / Deduction when MRM occurs

[0503]

[0504] In the examples in Table 2, for deduction object (1), the deduction for a single occurrence is [-1], and the deduction for repeated occurrences on the same route is [-2]. For deduction object (2), the deduction for a single occurrence is [-4], and the deduction for repeated occurrences on the same route is [-4]. For deduction object (3), the deduction for a single occurrence is [-5], and the deduction for repeated occurrences on the same route is [-5].

[0505] The penalty object (1) is the penalty applied when processing proceeds from step S131 to step S160, indicating a penalty under urgent circumstances. This situation involves a response to an event that occurs directly in front of the vehicle on the road in which the vehicle is traveling without prior notice, and does not involve the driver's direct responsibility. However, when the start of MRM is predicted by using situation judgment during automatic driving, a penalty (third-level penalty) is applied in order to avoid repeated use of system-dependent automatic driving. In addition, in the penalty object (1), for example, a mechanism can be used such that a penalty with a temporary condition flag can be cancelled if it is not repeatedly applied within a certain period of time.

[0506] Point deduction object (2) refers to the point deduction that occurs when processing proceeds from step S134 to step S160, indicating a point deduction level with a small amount of extra time. In this case, even if the system reduces the vehicle's speed and extends the arrival time to the point where handover is required, the manual control of driving recovery measures is insufficient due to driver negligence, and the MRM is activated. In this case, since the driver is responsible for activating the MRM, the point deduction (second-level point deduction) is more severe than in the case of point deduction object (1).

[0507] Points deduction object (3) refers to the points deducted when processing proceeds from step S150 to step S160, indicating a deduction level that would have been in cases where sufficient additional time was originally available. This situation corresponds to situations where a significant amount of additional time should have been available, where the handover should have been completed with early recovery. As a system, MRM is implemented using software and in a manner with minimal impact on the surrounding environment. However, to prevent negligent handovers and encourage drivers to make behavioral changes and take swift action, the points deducted (first-level deduction) are more severe than those deducted object (2) described above.

[0508] Next, referring to Table 3, we will illustrate an example of driver evaluation when the system issues a normal handover request (also known as an intervention request and transition request) to the driver. For example, in Figure 7B The evaluation shown in Table 3 is performed in step S124, but is not limited to this. The evaluation according to Table 3 can be performed at other handover times or even at times different from the handover time.

[0509] Table 3

[0510] Bonus / minus points for restoring quality when an intervention request is issued

[0511]

[0512]

[0513]

[0514] In Table 3, the first to fourth rows are examples of driver evaluations that add points, the fifth row is examples of driver evaluations that do not add or deduct points, and the sixth row and subsequent rows are examples of driver evaluations that deduct points.

[0515] According to Table 3, as examples of cases where bonus points are awarded, bonus points [+0.2] are awarded when the driver chooses to stop or rest in advance, or chooses an alternate route and abandons the handover at the handover point in advance; when the driver requests assistance from the vehicle ahead, remote control, or remote operation in advance; and when the handover begins through a recovery notification sound or the driver's self-confirmation of the situation (autonomous generation of the recovery sensation), regardless of whether it occurs once or repeatedly within the driving route. Additionally, bonus points [+0.1] are awarded when there is a cognitive test indicating that the driver has received prior notification of the recovery request, regardless of whether it occurs once or repeatedly within the driving route.

[0516] If the driver responds to the resumption notice and begins the resumption operation, the resumption operation is considered a normal resumption operation and no points will be added or deducted.

[0517] On the other hand, as an example of a situation where points are deducted, if the driver initiates recovery in response to a recovery alert, this situation is considered as the driver ignoring the situation. In this case, a deduction of [-0.2] is made for a single occurrence, and a deduction of x2 (double the deduction) is made for repeated occurrences within the same driving route. This deduction is intended to prevent drivers from ignoring situations or delaying high-priority handling. If the driver's prior notification of the recovery request has not been detected (notifications not entered into memory are not processed), whether it is a single occurrence or repeated occurrences within the driving route, it is considered a serious offense and a deduction of [-0.5]. If the driver initiates recovery in response to a mandatory recovery request, a deduction of [-1.0] is made for a single occurrence, and a deduction of x2 (double the deduction) is made for repeated occurrences within the same driving route, which is judged as insufficient risk perception.

[0518] Additionally, on main roads, if a handover is barely completed after the system pre-emptively slows down and generates a time grace period, a penalty of -2.0 points is deducted for a single occurrence, while repeated occurrences on the same route incur a penalty of x1.5 points. Similarly, on main roads, if a driver's handover processing fails—that is, the driver fails to hand over the vehicle and the system executes MRM—a penalty of -4.0 points is deducted for a single occurrence, while repeated occurrences on the same route incur a penalty of x1.5 points. This penalty is intended to deter intentional violations by drivers.

[0519] In addition, on low-speed non-arterial roads, points will be deducted [-0.5] if the system generates a time grace period by slowing down in advance and completes the handover, or if the driver's handover processing fails, i.e., if the driver's handover fails on a low-impact road (e.g., a road with very little traffic) and the system executes MRM, whether it occurs once or repeatedly within the same driving route.

[0520] Furthermore, regarding the use of NDRA, if a driver initiates NDRA without confirming the application of ODD, a penalty of [-2.0] points will be deducted for a single offense, and doubled for repeated offenses within the same route. If a driver uses NDRA outside of ODD, this is considered a violation, and a penalty of [-3.0] points will be deducted for a single offense, and doubled for repeated offenses within the same route.

[0521] The system (autonomous driving control unit 10112) accumulates the points shown in Table 3 for the same driver and uses this accumulated value as the driver's evaluation value. For example, the system accumulates the driver's acceleration / deceleration records for all driving routes set and executed by the driver in the system or driving routes executed within a predetermined time period. Thus, by imposing status-based penalties according to the driver's track record, this evaluation result is reflected in controls to prevent malicious use, such as ignoring timely, unprocessed NDRA interruption requests even after a handover request has been issued by the system, and carelessly repeating recovery operations.

[0522] When a driver's rating is low (e.g., a negative rating), the system can penalize the driver for using autonomous driving.

[0523] As an example of penalties for drivers, the use of automated driving is restricted. Examples of restrictions on automated driving use include: handling estimated arrival times for delayed destinations, mandatory stops at service areas, driving lockouts for certain periods, speed limits, restrictions on the use of automated driving mode (for the same day, week, or month), and restrictions on the area where automated driving mode can be used. The effect of these restrictions is to make drivers visually aware of the consequences (risks) and encourage them to revert to manual control or take appropriate action as soon as possible.

[0524] As another example of driver penalty, the use of secondary driving actions (NDRA) during autonomous driving is restricted. This restriction gives the driver an intuitive sense of potential loss (risk) and encourages them to return to manual control and take appropriate action as early as possible.

[0525] One hypothetical example of restrictions on secondary task usage is the restriction on the terminal device used by the driver during secondary tasks. One hypothetical scenario for restricting terminal device usage is to fill the screen displayed on that terminal device and to erode that screen using arbitrary images. Through these operations, for example, by gradually and intuitively prompting the driver to recognize the risk, it is possible to make the driver aware of the risk. Furthermore, it is conceivable to exchange the screen the terminal device is using with the handover information window (an exchange between a sub-screen and the main screen). For example, this could prompt the driver to pay attention to the handover of driving.

[0526] In addition, other methods can be considered, such as forcibly locking the terminal device's screen or invalidating the retroactive effect of operations performed by the driver using the terminal device. Because of the forced interruption of operation, a forced NDRA might make the driver feel a sense of wasted effort, thereby further prompting the driver to pay attention to the handover of driving.

[0527] Control of the terminal device can be achieved, for example, by utilizing the functions of application software installed in the terminal device using the system according to the embodiment (e.g., bird's-eye view display of the driving route, advance notification of the end of the ODD section to the driver, etc.).

[0528] Note that the addition / subtraction values ​​described using Tables 2 and 3 are examples and are not limited to those examples. Furthermore, the individual examples of addition / subtraction are also examples and are not limited to those examples.

[0529] <3-2-3-3. Top-down view of the driving route applicable to the embodiment>

[0530] Next, a top-down view of the driving route applicable to the embodiments will be described in more detail.

[0531] Figure 8 This is a schematic diagram illustrating an example of an overhead view of a driving route applicable to an embodiment. Figure 8 In the above view display 50, there are a near-view display unit 51a, a medium-view display unit 51b, and a far-view display unit 51c. Figure 8 This shows the direction of travel of the vehicle from the lower end of the diagram towards the upper end. Figure 8 The lower part indicates the current position of the vehicle, but the vehicle's position is not limited to this example. Furthermore, the icon 52 representing the vehicle is for ease of displaying the driving route and can be omitted from display.

[0532] exist Figure 8 In the foreground, the proximity display unit 51a displays the distance from the vehicle's current position to a predetermined first distance. The first distance is, for example, approximately 15 minutes' travel time from the current location. Figure 8 In the example, in the near-field display unit 51a, the vertical position on the screen can be linearly related to the actual distance.

[0533] The shape of the intermediate-distance display unit 51b is such that its width is reduced according to the vertical direction on the screen, so that it converges at an infinity point VP at the upper end of the width W1 of the near-distance display unit 51a. In the intermediate-distance display unit 51b, the position in the vertical direction on the screen has a non-linear relationship with the actual distance, and for example, the change in the actual distance relative to the position on the screen can increase as it moves upward on the screen.

[0534] Here, in Figure 8 In the context of distance display, if the position in the longitudinal direction is set to the arrival time along the travel direction, then the distance h from the infinity point VP can be displayed. diff The reciprocal of the arrival time is displayed proportionally to the travel time. Thus, by displaying the intermediate-distance display section 51b using a perspective method, the arrival time can be effectively presented on a narrow screen. By accurately displaying the impact of each handover point, etc., using the overhead display 50, the driver can intuitively grasp the time at each arrival point.

[0535] On the other hand, the far-distance display unit 51c extends from a position of width W2 in front of the infinity point VP while maintaining the width W2. Similar to the near-distance display unit 51a described above, the far-distance display unit 51c can make the vertical position on the screen linearly related to the actual distance.

[0536] in addition, Figure 8 All the sections shown should be those where autonomous driving at Level 4 is feasible. Here, it is assumed that there are sections where the driver would prefer to revert to manual control due to factors such as narrowing road width or railway crossings. Such sections are considered to have a high request-to-resume (RRR) for the driver. Therefore, if the driver does not properly revert to manual control, it could cause adverse social consequences, such as affecting following vehicles in that section.

[0537] Therefore, the overhead display 50 shows information indicating high RRR intervals. For example, for such intervals, a warning display 53 indicating a narrowing of the road width is displayed. The warning display 53 can attract the driver's attention. Additionally, a recommended handover start point can be provided at a predetermined distance in front of the driver from the interval, and the interval 56 recommended for handover can be highlighted.

[0538] In the embodiment, to Figure 8 The overhead view shown in section 50 adds a section display indicating the recommended driving mode for each section, making it easier for the driver to revert from automatic driving to manual control. (See reference...) Figures 9A to 9C The description adds an overview view of the range 50.

[0539] Figure 9A This is a schematic diagram illustrating an example of a top-down display 50a showing color-coded intervals according to an embodiment. Figure 9A In the overhead view 50a, color coding is used to distinguish between the autonomous driving permitted zone 53a, the posture maintenance recovery zone 53b, and the driving recovery necessary zone 53c.

[0540] The Automated Driving Permitted Zone 53a indicates the zone where Automated Driving Level 4 is permitted, and is displayed in green, for example, in association with safety and peace of mind. The Resumption of Posture Maintenance Zone 53b is the zone immediately preceding the return to manual control from Automated Driving, and indicates the zone where it is desired that the driver maintain the posture required for manual control. The Resumption of Posture Maintenance Zone 53b is displayed in yellow, for example, to draw the driver's attention. The Driving Resumption Required Zone 53c indicates the zone where the driver needs to manually control the vehicle, and is displayed in red, for example, to indicate a warning.

[0541] Note that the color coding using green, yellow, and red described above is an example and is not limited to this color combination. Additionally, a single color can be used without color coding, as long as the different zones can be clearly distinguished.

[0542] In this way, by changing the display method and distance from the vehicle according to the interval, the driver can easily determine when to switch from automatic driving to manual control.

[0543] Figure 9B This is a schematic diagram illustrating an example of a ring-shaped, top-view display of 50b according to an embodiment. Figure 9B In the example, the top of the circular display is set to the position of the vehicle, and the distance from the vehicle increases clockwise (right-handed) from the starting position. Furthermore, the sense of distance is emphasized by narrowing the display width as the distance from the vehicle increases.

[0544] As described above, the overhead display 50b, arranged in a circular pattern, is suitable for displaying in narrow areas such as the display screen of wearable devices like watches.

[0545] Figure 9C This is a schematic diagram illustrating an example of an overhead display 50c including road information according to an embodiment. Figure 9C The overhead view 50c shown adds road information such as icons 54a corresponding to traffic signs or icons 54b representing facilities to the display. Figure 9A The example shown is an overhead view of 50a. Icon 54a represents locations and things the driver should pay attention to in an autonomous vehicle; in this example, icon 54a is displayed as a simulation of traffic signs actually installed on the road. For example, icon 54b represents facilities needed for vehicle operation, corresponding to locations such as gas stations, parking areas, or service areas.

[0546] also, Figure 9C It also displays congestion zones, such as zones 55a and 55b, for which the transit time varies significantly.

[0547] Thus, by using the overhead display 50c with added road information, as the aforementioned conscious memory develops, risk information at various proximate times is acquired in the driver's visual field. Furthermore, at locations of high importance—i.e., high-risk locations—the sense of risk stimulates the driver's working memory during behavioral judgment. Therefore, the driver can predict the timing of reverting from automated driving to manual control earlier and can more smoothly transition to manual control compared to simply presenting a monotonous route display.

[0548] Aerial view of 50c and Figure 9A The overhead view display 50a shown can be displayed on the screen of a terminal device, for example, when a driver uses the autonomous driving system according to the embodiment. For example, an application related to the information processing program according to the embodiment, installed on the terminal device, runs on the CPU 10010 to control the display of the overhead view display 50a or the overhead view display 50c. At this time, it is conceivable that the overhead view display 50a or the overhead view display 50c is displayed, for example, in a compressed state in the width direction at the right or left end of the screen. Alternatively, the overhead view display 50a or the overhead view display 50c can be displayed on both sides of a shared vertex of the screen, or on three sides of the screen or around the perimeter of the screen.

[0549] <3-2-4. Example of HCD control structure according to the embodiment>

[0550] Next, an example of the control structure of the HCD according to the embodiment will be described in more detail. Figure 10 This is a functional block diagram illustrating an example of the control function of the HCD in the automatic driving control unit 10112 according to an embodiment. Note that... Figure 10 The functions of the automatic driving control unit 10112 used to implement HCD are highlighted, while other functions are omitted as appropriate.

[0551] exist Figure 10 In this embodiment, the autonomous driving control unit 10112 includes an HMI 100, a driver recovery delay evaluation unit 101, a driving route pre-predictive acquisition range estimation unit 102, a remote assistance control / steering assistance capability monitoring unit 103, a driver behavior change implementation level estimation unit 104, a vehicle driving route performance information providing unit 105, an ODD application estimation unit 106, an autonomous driving license integrated control unit 107, and a driver behavior quality assessment unit 108. When the information processing program according to the embodiment runs on the CPU 10010, each of these units is configured and implemented as a module, for example, on the RAM 10012, which serves as the main storage device.

[0552] The HMI 100 is implemented as a driver-oriented interface and is connected to, for example, an information display unit 120, a terminal device 121, an in-vehicle light source 122, a sound device 123, and an actuator 124.

[0553] The information display unit 120 executes a predetermined display based on commands from the HMI 100. The terminal device 121 can be a terminal device brought into the vehicle by the driver, or a terminal device pre-installed in the vehicle. The HMI 100 is capable of bidirectional communication with the terminal device 121. The terminal device 121 can receive user operations and provide control signals corresponding to the received user operations to the HMI 100. Furthermore, the terminal device 121 displays a predetermined image on a display device included in the terminal device 121 based on commands from the HMI 100. The in-vehicle light source 122 is a light source installed inside the vehicle, and is controlled by the HMI 100, including power-on / off control and light intensity control.

[0554] The sound device 123 includes a speaker, a buzzer, and drive circuitry for driving the speaker and buzzer. The sound device 123 emits sound according to the control of the HMI 100. Furthermore, the sound device 123 may include a microphone. The sound device 123 converts analog sound signals collected by the microphone into digital sound signals and provides the digital sound signals to the HMI 100.

[0555] Actuator 124 drives predetermined parts within the vehicle under the control of HMI 100. For example, actuator 124 applies vibrations such as tactile vibrations to the steering. In addition, another actuator 124 is capable of controlling the tilt of the driver's seat according to commands from HMI 100.

[0556] The HMI 100 controls the operation of the information display unit 120, terminal device 121, in-vehicle light source 122, sound device 123, and actuator 124 based on information from the driving route pre-prediction acquisition range estimation unit 102, remote assistance control / steering assist capability monitoring unit 103, and ODD application estimation unit 106 described below. Using this configuration, the driver can receive the following visual and auditory notifications.

[0557] • Prior notification via guided voice

[0558] In this case, a guiding sound is preferably used that is easily noticed but does not cause excessive stimulation, such as the in-flight beep in a passenger aircraft (e.g., a gentle beep reminding passengers to fasten their seatbelts). For example, guiding sound notifications can be used to provide advance notice of the resumption of driving from autopilot to manual control. For example, guiding sound notifications can be used when the system presents an agreement regarding a "transaction" to the driver.

[0559] • Notification requesting a return to manual driving control

[0560] Upon request, the HMI 100 can provide auditory notification via sound emitted by the sound device 123, or visual notification via display on the information display unit 120. Additionally, upon request, the HMI 100 can actuate the actuator 124 to provide tactile vibration to the steering wheels, thus providing tactile notification. Furthermore, the HMI 100 can instruct the driver to make a pointing vocalization call towards the road ahead.

[0561] Warnings and Alerts

[0562] HMI 100 can issue auditory, visual, or tactile warnings or alarms to the driver. For example, HMI 100 can control the sound device 123 to emit a warning sound to provide an audible warning. In this case, it is conceivable that the warning sound is a more stimulating sound compared to the aforementioned guidance sound. HMI 100 can also control the information display unit 120 and the interior light source 122 to provide visual warnings through flashing red lights, interior warning lights, etc. In addition, HMI 100 can control the actuator 124 to cause the driver's seat to vibrate strongly, thereby providing a tactile warning.

[0563] • Penalties

[0564] HMI 100 can perform controls that impose penalties on the driver. For example, HMI 100 can perform controls deemed to cause discomfort to the driver, such as visual restrictions, operational restrictions, applying mild pain or cold air to the driver, and moving the driver's seat forward. Additionally, HMI 100 can perform spurious controls, such as lateral vehicle sway, uncomfortable acceleration / deceleration, and suspected lane departure, and can penalize the driver to directly prompt an early recovery or to have the effect later rather than immediately. Furthermore, HMI 100 can impose penalties based on the driver's knowledge, such as presenting fine information, mandatory entry into a service area, the current restriction period, notification of prohibition from using autonomous driving, and warnings about restrictions on future or repeated use.

[0565] The driver recovery delay evaluation unit 101 evaluates the delay in a driver's recovery from automated driving to manual control, and is connected to, for example, a life record data information server 130, wearable device data recording 131, a face / upper body / eye camera 132, a biometric indicator acquisition unit 134, an in-vehicle locator 135, and a response evaluation input unit 136. Additionally, the driver recovery delay evaluation unit 101 obtains information representing the individual driver's recovery characteristics to manual control from a remote server dictionary 137.

[0566] Wearable device recorded data 131 is recorded data acquired from the wearable device while the driver is wearing it. Wearable device recorded data 131 includes, for example, the driver's behavioral history and biometric information.

[0567] The face / upper body / eye camera 132 is a camera installed inside the vehicle to image the upper body, including the driver's head. The face / upper body / eye camera 132 is installed inside the vehicle to image the driver's facial expressions, subtle eye movements, and upper body behavior. The face / upper body / eye camera 132 is not limited to this and may include multiple cameras for separately imager of the face, eyes, and upper body. The body posture / head camera 133 is a camera installed inside the vehicle to capture body posture, including the driver's head. By analyzing the images acquired by the body posture / head camera 133 in chronological order, the driver's body posture and the position and orientation of the head can be tracked.

[0568] Note that in this embodiment, for ease of installation, the face / upper body / eye camera 132 and the body posture / head camera 133 are described separately, but this is not limited to this example, and devices that integrate these cameras can also be used.

[0569] The biometrics acquisition unit 134 acquires the driver's biometrics, for example, based on the outputs of various sensors installed in the vehicle. Examples of biometrics to be acquired include respiration, pulse, exhalation, body temperature distribution, and electrooculogram. The invention is not limited thereto, and the biometrics acquisition unit 134 can also acquire a portion of the driver's biometrics from a wearable device worn by the driver.

[0570] The in-vehicle locator 135 is a locator installed inside the vehicle. The response evaluation input unit 136 receives from the driver responses to requests or warnings presented to the driver by the HMI 100.

[0571] Information acquired by wearable device data 131, face / upper body / eye camera 132, biometric information acquisition unit 134, in-vehicle locator 135, and response evaluation input unit 136 is accumulated in the life record data information server 130 as the driver's life log.

[0572] Figure 11 This is a functional block diagram illustrating an example of the function of the driver recovery delay evaluation unit 101 according to an embodiment. Figure 11 In the driver recovery delay evaluation unit 101, there are driver behavior response evaluation unit 1010, relevant feature learning unit 1011, condition-based recovery distribution personal feature / situational cognitive impairment feature dictionary 1012, and situational cognitive impairment transition prediction unit 1013.

[0573] The condition-based recovery distribution personal characteristic / situation awareness impairment characteristic dictionary 1012 is a dictionary related to the driver's individual observable assessment value and characteristics of decreased situation awareness. The related characteristic learning unit 1011 learns the correlation characteristics between the driver's individual observed assessment value and the recovery delay time distribution based on information representing characteristics related to the driver's recovery towards manual control driving obtained from the remote server dictionary 137, and the assessment value and situation awareness impairment characteristics obtained from the condition-based recovery distribution personal characteristic / situation awareness impairment characteristic dictionary 1012.

[0574] In this embodiment, the remote server dictionary 137 is located on a remote server outside the vehicle, but its location is not limited to this example. That is, the reason for installing the remote server dictionary 137 on an external server is to use an application that, due to the prevalence of commercial vehicles and car-sharing, the characteristics of the driver are not necessarily associated with the inherent vehicle. Therefore, the remote server dictionary 137 can be installed in the vehicle to be used.

[0575] The driver behavior response evaluation unit 1010 acquires various information about the driver from the HMI 100. For example, the driver behavior response evaluation unit 1010 acquires preliminary information from the life log of the HMI 100. In addition, the driver behavior response evaluation unit 1010 acquires information about the driver from the HMI 100, such as the following, based on facial and body images acquired by various sensors (cameras).

[0576] • Recognition information of facial expressions and body posture.

[0577] • Information about the eyes. This example involves obtaining evaluations of localized eye behaviors, such as the percentage of time the eyelids are closed (PERCLOS) or saccades (rapid eye movements).

[0578] • Posture and posture changes. In this example, the quality of recovery behavior is evaluated based on posture and posture changes.

[0579] • The position and posture of the passenger in the carriage.

[0580] Bioinformatics

[0581] The driver behavior response evaluation unit 1010 evaluates the driver's behavior response based on information obtained from the HMI 100 and relevant characteristics obtained from the relevant characteristics learning unit 1011. The evaluation results are then passed to the situation awareness decline prediction unit 1013. Based on the evaluation results, the situation awareness decline prediction unit 1013 predicts changes related to the driver's decreased situation awareness.

[0582] When the life record data information server 130 is available, a portion of the life log data obtained from the life record data information server 130 can be input into the driver behavior response evaluation unit 1010. By using the driver's life log data, the driver behavior response evaluation unit 1010 can improve the estimation accuracy of alertness based on preliminary driver state information such as insufficient sleep time, accumulation of overwork, sleep apnea syndrome, and residual alcohol from drinking, and can also improve the accuracy of judging the driver's conditional awareness, thereby enabling safer control of sudden drowsiness such as microsleep.

[0583] Thus, the driver recovery delay evaluation unit 101 functions as a monitoring unit, which monitors the driver's state based on information obtained from the life record data information server 130, wearable device recorded data 131, face / upper body / eye camera 132, biometric indicator acquisition unit 134, in-vehicle locator 135 and response evaluation input unit 136.

[0584] Return to Figure 10 As explained, the route prediction acquisition range estimation unit 102 acquires a high-freshness update LDM 140 and estimates the route prediction acquisition range based on the acquired high-freshness update LDM 140. That is, based on the high-freshness update LDM 140, the route prediction acquisition range estimation unit 102 acquires the range on the route where events can be predicted in advance.

[0585] Figure 12This is a schematic diagram illustrating the high-freshness update LDM 140 applicable to the embodiment. In each region, regionally distributed LDMs 1400a, 1400b, ..., 1400n are arranged. Based on information from installed sensors 1401, dedicated detection vehicle information 1402, general vehicle ADAS information 1403, weather information 1404, and emergency report information 1405 (e.g., the fall of hazardous materials), corresponding to each region, these regionally distributed LDMs 1400a, 1400b, ..., 1400n are updated as needed.

[0586] Regionally distributed LDMs 1400a to 1400n are transmitted via 5G, 4G, or other communication methods. The transmitted regionally distributed LDMs 1400a to 1400n are received, for example, by the autonomous driving control unit 10112 and aggregated to form a high-freshness update LDM 140. Furthermore, emergency report information 1405 is transmitted via broadcast 1406 and is directly received by the autonomous driving control unit 10112, or included in the aforementioned 5G or 4G communications. The autonomous driving control unit 10112 updates the high-freshness update LDM 140 based on the received emergency report information 1405.

[0587] Return to Figure 10 The description states that the route prediction acquisition range estimation unit 102 estimates the range of events that can be predicted in advance by acquiring high-freshness LDM 140 in the route based on the following information and conditions.

[0588] Here, in the central part of urban traffic, significant investments can be made in environmental infrastructure and dedicated detection vehicles. On the other hand, in areas or time periods where the investment effect is smaller, there are also cases where sporadic data collected mainly from ADAS information 1403 of ordinary vehicles in shadow mode is relied upon. Therefore, the predictive ability of the driving route that the highly up-to-date LDM 140 can provide depends on real-time information that actively changes over time based on factors such as the regional distribution of LDM deployment and the allowable communication bandwidth of the communication network.

[0589] • Regarding the provision of highly up-to-date LDM 140 information, prior to the commencement of the driving route.

[0590] • Risk information indicating a decrease in the update frequency of high-freshness updated LDM 140. The update frequency of high-freshness updated LDM 140 varies over time, which is attributed to the traffic / presence density of detection vehicles (e.g., dedicated detection vehicles).

[0591] • Information loss due to lack of regional radio communication frequency bands, etc.

[0592] • In response to reduced predictability due to severe weather, additional detection vehicles are used to supplement information.

[0593] • The predictive accuracy is reduced due to temporary information shortages in the information obtained from the lead vehicles / vehicle groups in the route sections that are not available due to infrastructure construction, etc., which are replaced by updated LDMs.

[0594] The remote assist control / steering assist capability monitoring unit 103 monitors the availability of remote assist control and the ability to handle steering assist based on information obtained from the remote assist control I / F 150. It is assumed that the monitoring performed by the remote assist control / steering assist capability monitoring unit 103 is used for options such as area traffic, platooning support, or limited connectivity support.

[0595] Figure 13 This is a schematic diagram illustrating information acquired by the remote assistance control I / F 150 applicable to an embodiment. Remote operation commanders 1500a, ..., 1500n-1 collect information from standby turning operators 1501, standby turning operators 1501n+1, and dedicated pilot guide contract vehicle 1502, respectively. In this example, remote operation commander 1500a also collects information from pilot guide vehicles 1503m and 1503m+1. Remote operation commanders 1500a, ..., 1500n-1 transmit the collected information to the remote assistance control I / F 150 using, for example, 5G communication. The communication scheme here is not limited to 5G; 4G could also be used. In the example shown, pilot guide vehicles 1503m and 1503m+1 transmit the collected information directly to the remote assistance control I / F 150.

[0596] Return to Figure 10 The remote assistance control / steering assist capability monitoring unit 103 performs the following processing based on the information obtained from the remote assistance control I / F150.

[0597] • Utilize remote control support services for control support, such as when a driver has difficulty resuming manual control from autonomous driving. This includes systems taking over driver control, such as early evasive maneuvers and operator-assigned control.

[0598] • Remote operator control of vehicle steering. This is the control command given when an external remote control transaction is completed and operator assignment is possible.

[0599] • Monitor the execution of remote assistance and monitor reversal in case of failure (fail-safe driving).

[0600] The driver behavior change achievement level estimation unit 104 estimates the driver's behavior change achievement level relative to system constraints. Figure 14 This is a functional block diagram illustrating an example of the function of the driver behavior change implementation level estimation unit 104 according to an embodiment. The driver behavior change implementation level estimation unit 104 includes a good recovery steering behavior score addition unit 1040 and a penalty behavior accumulation addition recording unit 1041.

[0601] The Excellent Steering Resumption Behavior Score Addition Unit 1040 adds an evaluation value to the Excellent Driving Operation score when resuming manual control driving. The Penalty Behavior Accumulation Addition Record Unit 1041 deducts points from the evaluation value based on violations such as requests to resume manual control driving or failures to respond to such requests. Furthermore, the Penalty Behavior Accumulation Addition Record Unit 1041 accumulates the evaluation value to the penalized behavior. The Driver Behavior Change Achievement Level Estimation Unit 104 can obtain evaluation values ​​from, for example, the Driver Resumption Delay Evaluation Unit 101.

[0602] Return to Figure 10 As explained, the vehicle route performance information providing unit 105 provides operational information related to the vehicle's route to the LDM regional cloud (e.g., regionally distributed LDMs 1400a to 1400n). Specifically, the vehicle route performance information providing unit 105 provides the following information, etc.

[0603] • Reporting of changes / differences in pre-acquired map information, reports of abnormal information, and risk information for following vehicles after entering the area.

[0604] • When an abnormality / dangerous risk (falling object, accident, disaster, etc.) is detected during the driving process, an emergency message is sent from the vehicle automatically or manually.

[0605] • Notifications of characteristic events (event notifications for drivers / vehicle users) and suspicious risk information.

[0606] • Information is provided not through automatic reporting, but through manual reporting by the driver. In this case, the notification includes a temporary record of road condition information starting from a point in time prior to the report.

[0607] • Upload detection requests from the server. For example, upon receiving an emergency report of a falling object risk with unknown details from the vehicle ahead, conduct detailed verification. To do this, send a detailed scan request to the LDM cloud server in the managed area and upload information obtained through environmental identification-enhanced scanning equivalent to environmental scanning during normal driving or through detailed scanning at a higher refresh rate.

[0608] Alternatively, in cases where LDM is difficult to provide, the vehicle route performance information providing unit 105 can further provide information to following vehicles and backup vehicles based on the information available to the vehicle.

[0609] For example, in situations where traffic volume through the area is low and it is difficult to provide continuously updated LDMs in the uploaded information, or when the infrastructure's LDM cloud is not adequately prepared, it is difficult to expect the infrastructure to drive autonomously at Level 4 based on highly up-to-date LDM 140 updates.

[0610] In this scenario, the driver switches from automated driving to manual control and pairs the vehicle requiring assistance (e.g., a following vehicle or a backup vehicle). Then, environmental data acquired when the vehicle is driving at Level 2 or lower automation is provided to the vehicle requiring assistance as the data needed to drive at Level 4 automation. Furthermore, LDM (Local Driver Management) is provided through pairing with the specific vehicle being assisted, and information about the following vehicle at Level 4 automation is also provided.

[0611] For example, the vehicle route performance information providing unit 105 can cooperate with the aforementioned remote assistance control / steering assistance capability monitoring unit 103 to provide this information. For example, when the paired object is a lead support vehicle, this information can be provided as road guidance information to the following vehicle of the paired object. It has been further found that the operation of the lead vehicle, remote assistance, etc., combined with the high freshness update LDM 140 is particularly useful when using inline transportation including driverless vehicles, and this operation can be applied to use where the driver does not ride in the actual vehicle.

[0612] ODD application estimation unit 106 determines whether the vehicle is within the range (ODD range) in which the vehicle can drive at each level of autonomous driving. ODD application estimation unit 106 makes this determination based on the following information.

[0613] • Evaluation information related to a driver's historical records, such as good credit ratings, reinstatement of violations, demerit points, and penalties.

[0614] • HCD-based assessment of a driver's understanding of the necessity of recovery and their proficiency.

[0615] • This indicates the acquisition status of the high-freshness update LDM 140.

[0616] • Information based on the request recovery rate (RRR) of LDMs such as high freshness update LDM 140 and information indicating the locations that can be selected by the backoff option.

[0617] • This indicates information that limits the application of autonomous driving based on diagnostic results obtained from onboard devices.

[0618] • Information indicating vehicle dynamics (risk characteristics of collapse when carrying passengers, luggage, and cargo).

[0619] Furthermore, the ODD application estimation unit 106 estimates the ODD ranges applicable to unmonitored autonomous driving at Level 4 and the ODD ranges applicable to autonomous driving at Level 3, based on other update states such as the high-freshness updated LDM 140. In addition, the ODD application estimation unit 106 reviews and updates the applicable ODD ranges based on newly acquired risk information, equipment contamination, and changes in driver status during the driving journey. At this time, the ODD application estimation unit 106 notifies the driver of the information update via the HMI 100 and evaluates the driver's understanding of the changes in situation based on the driver's response to the notification.

[0620] The autonomous driving license integrated control unit 107 comprehensively controls the use of autonomous driving licenses. For example, the autonomous driving license integrated control unit 107 comprehensively controls the autonomous driving license status for each driving section. Furthermore, the autonomous driving license integrated control unit 107 controls the execution of the Management Responsibility Regulator (MRM). In addition, the autonomous driving license integrated control unit 107 controls violations during autonomous driving use to impose penalties or punishments on the driver, such as forced termination of use. Examples of violations include delays in the driver's response to requests to resume manual driving from the system, and repeated continuous use of autonomous driving at Level 3.

[0621] The driver behavior quality assessment unit 108 evaluates the quality of driver behavior (behavior quality) during processes such as autonomous driving.

[0622] The driver behavior quality assessment unit 108 evaluates driver behavior quality, for example, based on the driver's steering stability. The driver behavior quality assessment unit 108 evaluates driving-related aspects such as the driver's steering operations, accelerator and brake operations, and turn signal operations. Additionally, in response to a handover request from the system to manual driving control, the driver behavior quality assessment unit 108 evaluates specified operations or actions by the driver, such as pointing and making a vocal call. Furthermore, when the driver returns from an NDRA task performed in a relaxed posture to a steering posture used for driving, a posture recovery evaluation can be performed.

[0623] One type of information that is difficult for the system to directly observe when implementing HCD control is the assessment of situation awareness, which is related to the driver's brain activation. Therefore, in the HCD according to the embodiment, attention is paid to steering behavior in states of reduced situation awareness. For example, in steering behavior under conditions of insufficient situation awareness, i.e., decreased situation awareness, intelligent feedback becomes inadequate, leading to increased steering due to overreaction. That is, focusing on states of reduced situation awareness, which typically result in steering that should be smooth under normal conditions becoming incorrectly responsive oversteering, the steering during autonomous driving is compared to steering during normal manual driving, and the result is used as an evaluation metric for the driver's situation awareness.

[0624] 3-3. Automated driving level 4 applicable to the embodiments

[0625] Here, the autonomous driving level 4 applicable to the embodiments will be described.

[0626] <3-3-1. Basic Structure>

[0627] First, the basic structure of Level 4 autonomous driving will be explained. Figure 15 This is a schematic diagram illustrating the basic structure of Level 4 autonomous driving applicable to the embodiments.

[0628] exist Figure 15 In each of the charts (a) to (g), the horizontal axis represents the respective location. Chart (a) shows the recovery time ΔT. drd An example of the relationship between location (or arrival time calculated based on vehicle speed). Figure (b) illustrates the grace period ΔT. 2lim An example of the relationship with location. The recovery time ΔT will be explained below. drd and grace time ΔT 2lim .

[0629] Figure (c) illustrates an example of a continuously updated (high-freshness updated) LDM data range. In this example, the drivable range in autonomous driving at Level 4 (described as Level 4 in the figure) includes, for example, LDM data ranges that have not been updated and have reduced freshness. As shown in range 64 in Figure (b), this LDM data range with reduced freshness is an area where LDM maintenance is notified in advance and is considered to require temporary manual control of driving. Note that range 63 in Figure (b) is an area where the provision of high-freshness updated LDM 140 cannot be maintained due to factors such as reduced number of passing vehicles and insufficient communication bandwidth caused by overuse of surrounding public communications. This range is also considered to require manual control of driving.

[0630] Chart (d) shows an example of RRR and recovery success rate. Charts (e), (f), and (g) show examples of the presence or absence of vehicles ahead, the idle status of the waiting point, and the idle status of the control operator, respectively.

[0631] Although the illustrations are omitted, situations may arise where, for example, in section 65b, there is no supporting information as indicated by diagrams (e), (f), and (g), and a handover event occurs that is beyond the driver's control upon entering section 65b. In such cases, if the vehicle stops in section 65b using the MRM function, it could potentially lead to serious violations, such as a risk of rear-end collisions due to emergency stopping at locations including road closures that include section 65b, or at tunnel exits with low visibility (described in detail below).

[0632] The system communicates with the LDM on the cloud network via a regional infrastructure communication network to request new information, and then receives continuously updated status information of the LDM within a predetermined driving range from the cloud network. Alternatively, the vehicle ahead provides individually up-to-date LDM information obtained one by one through methods such as vehicle-to-vehicle (V2V) communication.

[0633] Based on this information, the system determines the grace period ΔT, which indicates the period during which it is safe to drive the vehicle. 2lim (Time to reach MRM limit = Predictable distance directly ahead) and the recovery time ΔT required for the driver to resume manual control driving, as detected passively or actively by the system. drd (Time delay for resuming driving = driving notification). This determination is based on the maintenance status confirmed in the latest self-diagnostic report for this vehicle.

[0634] like Figure 15 As shown in Figure (a), for example, if the current position of this vehicle is position P61, then the recovery time ΔT drd This represents the recovery time ΔT between moving from position P61 and returning to position P61. drd At the corresponding distance, manual control driving is resumed. On the other hand, as shown in Figure (b), if position P62a is the location where manual control driving must be resumed, then the grace time ΔT is... 2lim This represents the grace time ΔT between moving back from position P62a and position P62a. 2lim At the corresponding distance, the grace period for manual driving is restored.

[0635] For example, as shown at positions P62a and P62b in Figure (b), the recovery time ΔT drd and grace time ΔT 2limIt varies depending on the road conditions and the driver's condition.

[0636] The system will grant a grace period ΔT 2lim With recovery time ΔT drd Compare and determine the grace period ΔT. 2lim and recovery time ΔT drd Does it satisfy the relationship in equation (1)?

[0637] ΔT 2lim >>ΔT drd (1)

[0638] When the grace period ΔT 2lim and recovery time ΔT drd When the relationship in equation (1) is satisfied, the probability of the driver encountering a situation requiring immediate action while the vehicle is in motion, even if the driver is using the vehicle in autonomous driving mode 4, is very low. This results in limited risk, and therefore, even if the driver cannot return to manual control in time, an emergency plan will be triggered unless the MRM rapidly increases other traffic risks.

[0639] Here, based on information obtained from LDM and other sources, the system determines the risk of traffic congestion caused by the vehicle using MRM to perform emergency stops within the driving zone. Based on the assessment, if such a possibility exists, the system performs the following operations before entering the zone: for example, searching for permissible detour routes; determining whether to pair with a lead vehicle; and assessing the availability of remote driving assistance controllers, operators, and necessary communication lines. Based on the assessment, the system provides risk selection information to the user, either by offering avoidance measures associated with MRM execution, through the available limits of autonomous driving. The system allows for optional settings, such as whether to prompt the user for a decision or whether to prioritize avoidance options in advance, and completes the corresponding processing.

[0640] That is, for vehicles traveling within the zone, the availability of Level 4 autonomous driving is determined by considering the selection of detour routes, the continuous limit points where paired lead vehicles or remote operators perform remote steering, and the control limit points at which the system can drive without affecting following vehicles (i.e., without causing significant social impact). This information is presented by the information display unit 120 as prompts for the driver to make risk management decisions, and this information is stored in the driver's working memory. Therefore, the driver can recognize the situation as early as possible when approaching a location requiring action.

[0641] When drivers ignore system-requested recovery actions and violate transactions at these extreme locations, they will be subject to more general and direct penalties based on the violation, rather than secondary incidents. That is, drivers will receive penalties with direct adverse effects, such as speed limits while continuing, mandatory parking in a parking lot, or exposure to odors, rather than random possibilities unknown to the driver. This penalty allows the system to prompt drivers to change their behavior by taking actions such as prohibiting unauthorized use of autonomous driving or refraining from actively violating regulations while using autonomous driving.

[0642] Here, as shown in Figure (b), the grace period ΔT 2lim This information changes as the vehicle travels, and there may be instances where forecasts cannot be obtained for a sufficiently long period as initially planned. Even if data for all route segments has been received at the start of the route, the data in the high-freshness LDM 140 update provided by the infrastructure can change over time. Therefore, obtaining the high-freshness LDM 140 update each time can likely put pressure on communication frequency bands, etc.

[0643] Therefore, the information obtained in advance by the vehicle's system includes confirmation information of the sections where Level 4 autonomous driving is not permitted, and the grace time ΔT for each service-reserved section. 2lim The prediction information. Here, the grace time ΔT 2lim Before actually approaching each area, more precise, recently updated information is obtained. This information can be obtained by directly requesting the area management server, by obtaining it from the lead vehicle via V2V communication, or by obtaining it from broadcast information.

[0644] The RRR and recovery success rate in chart (d) will be explained. An RRR (Request Recovery Ratio) value of 100% represents a range where, if a vehicle stops or rapidly decelerates significantly within this range, following vehicles are highly likely to inevitably decelerate sharply. Within this range, early handover is required to ensure safety.

[0645] Examples of sections requiring a high RRR setting include sections that are specially defined sections, such as one-way bridges where traffic volume may prevent stopping at least halfway or cause complete congestion in both directions; special roads without vehicle refuge areas, such as the Metropolitan Expressway; sections where general vehicles need time to grasp the situation, such as tunnel exits; roundabouts and intersections. Conversely, in cases where traffic volume is very low and stopping on the road is highly unlikely to obstruct the view or movement of following vehicles, an RRR of 0% can be set.

[0646] In the example diagram, RRR is set below 100% in interval 65a, while RRR is set to 100% in interval 65b. This indicates that interval 65b is the interval where a vehicle's stopping or sudden deceleration is highly likely to significantly affect the movement of following vehicles. On the other hand, interval 65a, where RRR is set below 100%, indicates that a vehicle's stopping or emergency braking has a smaller impact on following vehicles than in interval 65b.

[0647] The presence or absence of a lead vehicle in Figure (e) indicates whether there is mutual volunteer support using a dedicated backup vehicle or a regular vehicle guiding and supporting autonomous driving in sections where only the vehicle is equipped and the LDM (Leadership Development Vehicle) is difficult to traverse. The idle status of waiting areas in Figure (f) indicates, for example, whether there are waiting areas where the lead vehicle or the vehicle itself waited before arriving in difficult sections when assistance from the lead vehicle is obtained, provided that a lead vehicle is identified as present in Figure (e). The idle status of control operators in Figure (g) indicates the availability (capacity) of the controller and the availability of actual operating operators. If a mechanism for receiving remote assistance is used to schedule the driving route, it will affect the driver's recovery request rate within the predetermined section.

[0648] These complex controls may not be readily apparent to the average healthy person using them. On the other hand, as an advantage of autonomous driving, this function is useful for providing services to a wide social network when used for public services such as those for groups with limited manual driving abilities (the elderly, children, etc.), where it is difficult to find the necessary drivers due to manpower shortages.

[0649] If the pairing for following the lead vehicle or for remote driving assistance steering at cruising speed is reliably executed, the vehicle can operate at Level 4 of autonomous driving. On the other hand, if the driver handles the situation themselves, actions such as completing the handover to manual driving or using MRM to stop in advance are required before approaching the next RRR of 100% (shown as interval 66 in the diagram).

[0650] <3-3-2. ODD for Level 4 Automated Driving>

[0651] Next, the ODD of autonomous driving level 4 according to the embodiment will be described. Figure 16 This is a schematic diagram illustrating the ODD of autonomous driving level 4 according to an embodiment.

[0652] Figure 16 The diagram shows the direction the vehicle travels from left to right. The diagram above shows an example of a section on road 70 where Level 4 autonomous driving is permitted, provided as static information. Figure 16The diagram below schematically illustrates an example of a situation within a vehicle-accessible area where lane width is limited due to road construction, making autonomous driving at Level 4 difficult (section 71). Figure 16 In this section, section 71 is the range from point R to point S, and within section 71, the driver needs to manually control the vehicle. In section 72, which has a predetermined length and begins at point S at the end of section 71, the driver can switch from manual to automatic driving.

[0653] This section will explain the ideal use of Level 4 Automated Driving. When a vehicle is equipped with a performance level exceeding a certain standard, Level 4 Automated Driving can be consistently used on physical road sections, provided the driving route is confirmed before it begins. On the other hand, there may be situations where Level 4 Automated Driving is not permitted at the start of driving after the driving route has been determined, or during driving after it begins. Therefore, by performing continuous state monitoring while preparing for abnormal situations, the presence and significance of Automated Driving will be reduced from the user's perspective.

[0654] Therefore, it is conceivable to introduce a control that minimizes the use of emergency measures known as MRM by maintaining their use below a certain level of negative social impact.

[0655] Alternatively, it is conceivable to avoid MRM activation. In this case, information is provided accurately and intuitively to enable the driver to take proactive measures and acquire the necessary information before MRM activation, i.e., to act on working memory with appropriate priority. Examples of information to be provided to the driver could include vehicle dynamics displacement, onboard device self-diagnostics and condition alerts, predictive information about the road ahead (including temporary reductions in sensing performance), and advance information on the appropriateness of evacuation / retreat (in temporarily acceptable amounts due to capacity fluctuations).

[0656] In addition, there needs to be a mechanism to cultivate a sense of priority in use, so that even when drivers participate in NDRA, they can take predictable priority measures and autonomous actions as their form of response.

[0657] Next, we will explain a more practical application of Level 4 Automated Driving. In the area where the vehicle can operate at Level 4 Automated Driving, in order for the vehicle to autonomously determine its ODD as Level 4 Automated Driving and to perform automated driving without a driver, the following conditions must be met.

[0658] First, the system must be able to acquire a high-freshness update LDM 140 in advance during the journey along the assumed driving route. The acquired high-freshness update LDM 140 includes information representing the recovery success rate (or requested recovery rate: RRR) for each segment along the driving route. The system calculates an estimated delay time before the driver can resume manual control driving before the driver's assumed handover limit point, as the delay time from the notification for achieving that RRR to the resumption.

[0659] Furthermore, prior to the timeframe considering the delay, the system provides an avoidance option if the driver fails to resume driving. During actual driving, based on updated information from the driving route, after receiving information indicating the expectation for the driver to return to manual control, the system prompts the driver to promptly take the actual resumption action.

[0660] Here, whether the driver responds to the notification from the system and takes the expected recovery action corresponds to the area determined by the human behavioral psychological pattern that the system cannot directly perceive.

[0661] However, people will not necessarily take autonomous actions in all aspects simultaneously. That is, based on social norms, it is difficult to expect people to take such autonomous measures unless they have developed an ethical mind.

[0662] Here, we first assume that a person, under the premise of highly developed autonomous sensory processing, performs processing actions based on notifications. The benefits of performing secondary tasks, the benefits of movement as the primary purpose, the disadvantages of not performing recovery when a driving recovery request is received, and the disadvantages of ignoring the prior information required for recovery are projected onto the future as consequences of choosing actions, and the choice of coping behavior is made within the scope of being able to intuitively describe the consequences.

[0663] Furthermore, as a result, when actual measures are taken, key information for judging the pre-selected response is preferentially stored in working memory, that is, in working memory that decays over time.

[0664] As a matter of human behavior and psychology, the delay time from the notification of the necessity to resume manual driving to the actual completion of the resumption largely depends on the following conditions: such as how prior information is provided; the driver's correct perception of the importance of the notification; the time elapsed from the new handover of the necessity notification to the implementation of the notification in the memory of its importance decay; whether the driver's attention has been focused on matters other than driving; and individual differences in the driver's ability to retain important matters in working memory.

[0665] Here, it will be explained Figure 16The lower half of the diagram. For example, if new handover information is generated by updating LDM 140 with high freshness (step S70), the handover information is obtained by the system via the regional LDM cloud network as shown in step S71. The system can also receive anomaly notification signals from the lead vehicle, etc. (step S72). In step S73, based on the obtained handover information or anomaly notification signal, the system informs the driver of the information of the participating points (locations) involved in the handover and the importance of handling the handover (prompt for temporary transactions).

[0666] The driver recognizes the importance of the notification (step S74) and agrees to and responds to the temporary transaction. The system detects the driver's response (step S75). Therefore, the temporary transaction is exchanged. In addition, based on the driver's agreement to the temporary transaction, the driver stores the information about the handover in working memory (step S76).

[0667] For example, if the driver fails to perform the handover operation within a predetermined time after the first notification, the system sends a reconfirmation notification to the driver and determines whether the driver is aware of the notification (step S77). Depending on whether the driver is aware of the notification, the process branches as shown at point P.

[0668] Here, the timing for the driver to revert to manual control varies depending on conditions such as the driver's perception of the importance of the handover. For example, if no response to the temporary transaction is detected in step S75, the system requests the driver to revert to manual control at location Q1.

[0669] The system sends a notification to the driver for reconfirmation (step S77), and depending on whether the driver recognizes the notification, sends a recovery request at location Q2, which is closer to section 71 after location Q1, or at location Q3, which is farther from location Q1 but closer to section 71.

[0670] When a person engages in thinking activities (brain activity) that require conscious judgment, the brain unconsciously captures knowledge and information that forms the basis of thinking and temporarily stores them in working memory according to their importance. As the importance of the information decreases, the information acquired in working memory gradually disappears from working memory.

[0671] During this period, suppose, for example, while the driver is immersed in NDRA (Non-Driving Assist), a handover request to switch from automated driving to manual control is issued from the system. When the driver does not feel urgency during the process or has little sense of the near-term risk of acting intuitively due to the driver's negligence in ignoring the notification and failing to perform the handover, the driver's sense of the necessity of the action—that is, the retention of working memory—gradually diminishes. Furthermore, information required at the time of handover, such as environmental monitoring information and preconditions for vehicle operation (vehicle dynamics), is diluted in working memory. For example, when environmental monitoring information is deemed important, the information required to determine its importance is retained in working memory.

[0672] For example, consider the hypothetical scenario where, while the vehicle is traveling along a route allowing Level 4 autonomous driving, information such as the high-freshness LDM 140 update and prior information about the route ahead obtained from the lead vehicle via V2V communication indicates that, over time, an event occurs where the vehicle approaches a section of the road requiring manual control. Furthermore, in cases where there is a narrow section of road ahead that is difficult to navigate, a successful handover before reaching that section is required to maintain social order. The delay from notification to successful handover largely depends on the driver's level of alertness and the extent to which they retain the necessary prior information in their memory.

[0673] When the driver recognizes the necessity, captures prior information with a sense of urgency, identifies the information through the initial notification and responds (i.e., the system detects perception in the form of a response), and the importance of the information is significant enough to be retained in the driver's working memory, the time from notification to recovery can be shortened, and the notification only needs to be given slightly earlier than the critical point.

[0674] On the other hand, if the driver does not correctly recognize the notification and the system cannot detect the driver's recognition of the notification, the system determines that the driver has not adequately retained the necessity of the handover in working memory, and at an earlier time ( Figure 16 A recovery request is sent to the lower half of the location Q1).

[0675] However, unlike the mechanical mechanisms of a system, working memory is a conceptual representation of the brain functions that control human thinking and judgment. Therefore, everyone has a different upper limit to their memory capacity, and people may even quickly forget the priority of important information due to factors such as health conditions and aging.

[0676] Suppose that the time from when the driver receives this information associated with the change sufficiently before arriving at the corresponding location R (e.g., location Q1) to when the driver responds to the notification and arrives at the corresponding location R will be sufficiently long, such as tens of minutes. From an ergonomic perspective, if this is controlled via HCD, the person essentially stores information in working memory according to its importance. In this case, if information cannot be perceived as having recent importance, its priority is reduced compared to information such as NDRA, which has high importance at that point in time.

[0677] In this scenario, the system indicates to the driver the urgency of the handover request and the penalty for ignoring it, and then monitors the driver's response. By responding based on understanding rather than a reflexive response, the driver can inject information into their working memory in a more reliable form. As a means of checking the response after understanding, an observational evaluation of the driver's conscious cognitive state can be applied using intentional gestures pointing to a vocal call, as shown in JP 2019-021229 A and WO 19 / 017215A. Since there is cognitive feedback in the gestures pointing to the vocal call, it has the potential to serve a significant role in confirmation and identification. The application is not limited to this, and simpler cognitive response methods can be used, where the driver answers questions posed by the system.

[0678] Here, it can be presumed that drivers receiving early recovery notifications do not respond adequately to previous notifications and have a diminished sense of the necessity and importance of recovery. Based on the driver's learning history regarding the time delay from the last notification to recovery, the required time is calculated at the necessary handover completion critical point, based on a certain success rate of recovery to manual driving. In this case, although the time from recovery request notification to recovery completion can be extended, the quality of recovery behavior—that is, the quality of rapid recovery from notification—is managed and quantified. Therefore, low-quality recovery behavior is penalized and penalized, thus creating an expectation of early recovery behavior in the driver's mind.

[0679] The driver's accurate behavioral judgment in response to the previous notification or notice can only be executed when the prior information prompting the information display unit 120, etc., leading to the judgment is appropriately and correctly implemented based on the risk and stored in the judgment memory.

[0680] Here, it will be explained Figure 16 Usage of the lower half, #1 to #5.

[0681] Usage #1

[0682] Use Case #1 is an example of this scenario: Suppose that, without active monitoring and control through continuous reception of the latest data, such as the high-freshness LDM 140, a permitted route for Level 4 autonomous driving, updated via transient static LDM, is assumed to be an executable route for Level 4 autonomous driving, and a driving plan is formulated. In this case, depending on the driver's state, for new situations requiring driver intervention, for which no updated information has been obtained from the transient static LDM, the return to manual control cannot be completed within the driver handover permission limit. This will trigger emergency measures by the MRM, which in some cases could impede the passage of following vehicles or cause a rear-end collision.

[0683] Use Case #1 can occur under various circumstances, such as when information updates are made through a transaction involving the right to receive them, i.e., a subscription, or when the contract expires; when receiving information fails due to conditions imposed on high-importance charges; and when there is a termination of the contract regarding the existence of remote assisted concierge services.

[0684] Usage #2

[0685] Use Case #2 is an example of this: Driving environment information along the route is received in advance from a high-freshness update LDM140, etc., and the driver is notified. This information is correctly perceived by the driver, but no response from the driver is detected. In this case, it is difficult to determine whether the importance and timing of necessary intervention are stored in the driver's working memory, and there is a risk that the handover cannot be completed safely and in a timely manner. Therefore, the driver is notified in advance (…). Figure 16 Location Q1). This allows the driver time to participate in tasks other than driving (e.g., NDRA). In all cases, if the handover is not performed promptly from the start of the recovery notice and the recovery quality is poor, penalty evaluations will result in demerit points, thus becoming a disadvantage for future use.

[0686] Usage #3

[0687] Unlike use case #2 described above, use case #3 is an exemplary case where the driver correctly recognizes the event during the notification phase. Here, in use case #3, the Level 4 autonomous driving continues for a considerable period from the early receipt of the notification of the new event to the actual arrival at the corresponding location (location Q2). In this situation, it is difficult to determine whether the importance and timing of the handover are stored in the driver's working memory at the time of receiving the notification. If a certain period has elapsed since the first notification in step S73, the memory may have faded. In this case, the system issues a reconfirmation notification to the driver (step S77) and checks the driver's response. Therefore, it can be found that the driver's residual memory is less than in use case #4 described below, and the system issues an earlier notification.

[0688] At this point, in step S74, the driver makes a cognitive response to the change in situation. Therefore, there is a small amount of residual memory, and the time required for situation cognition is shorter than that in use case #2 above. Therefore, notification is given during the intermediate time between use case #2 above and use case 43 below.

[0689] Usage #4

[0690] Use Case #4 is an exemplary scenario where the driver understands the importance of the notification upon receiving it, responds to the reconfirmation notification (step S77), and the driver's memory is retained as working memory. In this case, even with a long time before arriving at the location, the driver can refresh their memory as working memory by appropriately checking the situation near the location (e.g., calling out to directions on the road ahead or the notification screen), and risk awareness increases as the distance to the location decreases. Therefore, the system can perform behavioral detection of the driver's state and reconfirmation, and even immediately before arriving at the notified handover location (location Q3), the driver can accurately and appropriately perform handover recovery based on the undiminished residual information in working memory. This enables high-quality recovery behavior, rated as excellent.

[0691] Usage #5

[0692] Use Case #5 is the same as Use Case #4 described above, prior to the recognition of receiving the next information needed to transfer during autopilot operation for working memory. However, in Use Case #5, the driver's consciousness gradually drifts away from the cycle of driving operations due to the retention of new information in working memory and the passage of time caused by a state known as inattentiveness. In this case, the timing of reconfirming and resuming driving varies greatly from person to person, depending on the state at the time.

[0693] In use case #5, the system utilizes observable evaluation metrics, such as each driver's health status including autonomic nervous system dysfunction based on their state of consciousness, and provides feedback to the driver in a continuous and intuitive manner, activating benefits and penalties. The system repeatedly presents the driver with information such as appropriate prior risks, options for risk avoidance, or recently plotted information about the extent of risk impact when not avoided. This facilitates mental reinforcement learning for the driver, establishing a habit of early recovery and observation of the progress required for manual control driving, thereby more reliably forming the idea of ​​being in control of the situation before reaching the handover point as working memory.

[0694] Note that use case #5 conceptually illustrates an example where, based on information prompts to the driver, the driver's occasional health status, and repeated use of the system, and an evaluation of behavioral characteristics unconsciously acquired through self-learning by the driver, the system is variably notified of the scope of NDRA participation as ODD, depending on the driver's performance. It also schematically shows that the scope can vary considerably depending on changes in the driver's behavior.

[0695] As mentioned above, based on the drivable range of Level 4 autonomous driving as the same physical environment, usage will vary depending on the accessibility of information, the risk information included in the information, the weighting and avoidance choices based on importance, the time information, and the processing (response) of the information.

[0696] In HCD-based autonomous driving, drivers repeatedly experience a long cycle including: adding risk information as a basis for human behavioral judgment criteria to the timely updated information provided by the system; imposing penalties based on responses during use; and receiving benefits for good responses. By introducing HCD according to embodiments of this disclosure, the benefits of autonomous driving can be enjoyed while presenting intuitive risks to state usage resulting from the driver's over-reliance on autonomous driving. This configuration allows the driver to participate in the recovery from active autonomous driving to manual control driving to appropriately utilize the advantages of Comfort NDRA while continuously receiving risk information from the system. Therefore, the driver can confidently confirm information during autonomous driving while taking advantage of its benefits. This HMI, controlled by HCD, is not a mandatory confirmation requirement for the driver but rather prompts the driver to take proactive action by balancing the benefits and penalties of using NDRA.

[0697] <3-3-3. Applicable Example of Autonomous Driving Level 4 according to the Embodiments>

[0698] Next, an example of the application of autonomous driving level 4 according to the embodiments will be described. Figure 17A and Figure 17BThis is a flowchart illustrating an example of an applicable example of autonomous driving level 4 according to an embodiment. Figure 17A and Figure 17B In the attached diagram, the label "G" indicates a process transition to... Figure 17A and Figure 17B The corresponding figure labels in the diagram.

[0699] exist Figure 17A In step S200, the autonomous driving control unit 10112 acquires and maintains various information such as LDM initial data 80, driver personal recovery characteristic dictionary 81, RRR 82, and vehicle dynamic characteristics 83. Furthermore, the autonomous driving control unit 10112 also acquires updated LDM information (#1, #2, ...), updated diagnostic information, and other updated information (N), etc.

[0700] In the next step S201, the autonomous driving control unit 10112 identifies the initial ODD and authorizes the autonomous driving settings based on the information obtained in step S200. In the next step S202, the autonomous driving control unit 10112 presents the driving route to the driver and requests the driver to select a route, etc. Furthermore, the autonomous driving control unit 10112 requests the driver to select a route that conforms to the specified NDRA (National Driving Regulation Authority) specifications. In the next step S203, the driver begins driving the vehicle.

[0701] In the next step S204, the autonomous driving control unit 10112 acquires the information described in step S200 and updates the constant information accompanying the driving route after it begins. Additionally, the autonomous driving control unit 10112 performs visual display including arrival times such as driving obstacle information for each autonomous driving level (see reference). Figure 8 , Figures 9A to 9C ).

[0702] In the next step S205, the autonomous driving control unit 10112 determines whether the vehicle has entered the permitted ODD (Operational Domain) zone for Level 4 autonomous driving. If it is determined that the vehicle has not entered the ODD zone (step S205, "No"), the autonomous driving control unit 10112 returns to the processing in step S204. Conversely, if it is determined in step S205 that the vehicle has entered the ODD zone (step S205, "Yes"), the autonomous driving control unit 10112 proceeds to the processing in step S206.

[0703] Note that in Figure 17A and Figure 17B In the flowchart shown, when the vehicle leaves the area where it was previously able to perform autonomous driving and enters the area where it is determined whether a new ODD can be used, the same process is repeated from step S204 (not shown).

[0704] In step S206, the autonomous driving control unit 10112 determines whether the driver requests to switch to autonomous driving mode. If no switching request is determined (step S206, "No"), the autonomous driving control unit 10112 returns to the processing in step S204. Conversely, if a switching request is determined (step S206, "Yes"), the autonomous driving control unit 10112 proceeds to the processing in step S207.

[0705] In step S207, the autonomous driving control unit 10112 determines the likelihood of the driver reverting to manual control. Here, for example, based on the driver's personal recovery characteristic dictionary 81, when the driver is a high-quality autonomous driving user actively performing recovery operations, the autonomous driving control unit 10112 allows the driver to utilize the advantages gained through autonomous driving (NDRA, etc.). On the other hand, when the driver may have drug dependence or sleep disorders, even if the driver has few demerit points or penalties, the autonomous driving control unit 10112 prohibits or restricts the use of autonomous driving functions. Through the judgment process in step S207, many drivers learn to avoid being prohibited from using the system in order to gain the benefits of NDRA during autonomous driving.

[0706] Even if autonomous driving is permitted, the environment allowed by the driving route may not necessarily be an environment that can consistently provide the use of Level 4 autonomous driving. As mentioned above, even if the driver has the ability to recover, there are still conditions that allow the use of autonomous driving or advanced assistance systems prior to Level 3. In this case, it can be said that the determination process in step S207 is a determination by the driver regarding the use of Level 3 autonomous driving. The application of Level 3 autonomous driving according to the embodiment will be described below.

[0707] If it is determined that the driver is estimated to have the ability to resume manual control driving (step S207, "OK"), the automatic driving control unit 10112 performs the operation according to the reference numeral "G" in the figure. Figure 17B The flowchart in the process is processed. Conversely, if it is determined that the driver is estimated to be incapable of recovery (step S207, "NG"), the automatic driving control unit 10112 proceeds to the process of step S208.

[0708] Note that in the autonomous driving usage mode, when autonomous driving is used in combination with remote driving assistance, pilot-guided vehicle assistance, etc., determining the driver's ability to resume manual driving is not mandatory. In this case, other decision-making processes are performed, which are not included in the processes described in the examples of this embodiment.

[0709] In step S208, the autonomous driving control unit 10112 presents the driver with a notification that autonomous driving is not permitted, along with the reason. Examples of conceivable reasons presented to the driver in this case include driver fatigue and drowsiness, or a driver's history of penalties for violations resulting from over-reliance exceeding a predetermined value. Once the reason for the disallowed notification has been presented to the system, the driver, who expects to benefit from autonomous driving, anticipates behavioral improvement learning through methods such as improvement learning and a limited number of requests for improved permission (described below).

[0710] After the processing in step S208, the process returns to step S204. Here, there are situations where the vehicle is traveling on a road segment suitable for the use of the autonomous driving function during continued driving; or the initial conditions are replaced with conditions that allow use by enhancing the driver's awareness of the process. Therefore, the system cyclically performs the processing from steps S204 to S208 and continuously monitors the state of the driver, etc.

[0711] conduct Figure 17B The flowchart is explained below. Following the reference numeral "G," i.e., when the driver selects to drive at Level 4 autonomous driving, the autonomous driving control unit 10112 updates the latest information since the start of the driving route in step S220. After entering an ODD zone that allows Level 4 autonomous driving, the vehicle can continue to drive at Level 4 autonomous driving as long as the conditions remain unchanged. Figure 17B In the flowchart, step S220 shows the state being monitored in a steady state, i.e., the cyclical process of updating the latest messages in relation to driving.

[0712] When a change in certain conditions along the route is detected in step S220, or when the vehicle approaches the end point of the ODD, the autonomous driving control unit 10112 performs the processing of step S221.

[0713] In step S221, the autonomous driving control unit 10112 determines whether the information related to the latest route, which is essential for continuous autonomous driving, has been updated. If the information has not been updated (step S221, "No"), the autonomous driving control unit 10112 proceeds to step S226.

[0714] In step S226, the autonomous driving control unit 10112 begins to execute a predetermined safe handover sequence according to the approaching end of the autonomous driving interval (NDRA usage interval). Then, the handover sequence is executed.

[0715] Here, the autonomous driving control unit 10112 adds an evaluation value to a skilled driver, such as one who faithfully responds to the system's recovery requests, continuously monitors the recovery requests, and acknowledges changes in the driving route's status. Furthermore, the autonomous driving control unit 10112 allows skilled drivers to choose to switch back to autonomous driving mode next without undergoing complex checks and approval procedures such as multi-certification. Skilled drivers also have priority access to the autonomous driving usage guide for Level 4 autonomous driving. In this way, skilled drivers can enjoy various benefits.

[0716] In achieving HCD where the driver takes optimal confirmation action, it is essential to have the process by which the driver retrieves these confirmation judgments from the system and the necessary "memory" to trigger the correct action, as well as the "quality" of the information provided by the system to the HMI. For example, this information corresponds to instructions. Figure 9C Information such as when, what, and what actions to take as countermeasures will have an impact on working memory is information that plays a role in working memory.

[0717] Note that the recovery behavior of the driver in the safe handover sequence in step S226 is evaluated to obtain the recovery behavior data and stored as recovery behavior data. This recovery behavior data affects the driver's score.

[0718] Conversely, if the determination information is updated in step S221 (step S221, "Yes"), the autonomous driving control unit 10112 performs the processing of steps S222a, S222b and S223.

[0719] The disagreement in step S221 regarding the determination that the information has been updated (step S221, "Yes") is that the ODD allowance conditions when entering the ODD zone are assumed to be the handling of emergencies such as sudden weather deterioration, vehicle malfunction, or cargo collapse during travel. In the event of such an emergency, measures corresponding to the grace period required to handle the event need to be taken. Figure 17B The flowchart illustrates an example of a series of processes related to the measure applicable to the embodiments. The quality of a driver's response to an event in an abnormal situation is also acquired by the driver through their appropriate risk assessment and is an important factor in the driver's appropriate behavioral changes.

[0720] In step S222a, the autonomous driving control unit 10112 monitors the driver's state, including the response judgment to notification recognition, obtains an estimate of the delay time required to return to manual control driving based on the monitoring results, and updates the existing estimate. In step S222b, the autonomous driving control unit 10112 updates the road's RRR information and performs a recalculation of the allowable limits for use of the MRM based on the update.

[0721] In step S223, the autonomous driving control unit 10112 displays the re-checked ODD calculation results based on the information determined to be updated in step S221, the information updated and acquired in steps S222a and S222b, self-diagnostic information, etc. Furthermore, the autonomous driving control unit 10112 confirms the driver's response to this display.

[0722] In the next step S224, the autonomous driving control unit 10112 calculates the predicted arrival time T up to the end of the ODD interval associated with autonomous driving level 4. L4ODDEND And the predicted time T for the delay in resuming manual driving control MDR In the next step S225, the autonomous driving control unit 10112 determines the calculated predicted arrival time T. L4ODDEND and prediction time T MDR Does [T] satisfy? L4ODDEND >T MDR The relationship is +α]. Note that the value α is the remaining time at the necessary location for the handover to begin.

[0723] If it is determined in step S225 that [T] is not satisfied L4ODDEND >T MDR If the relationship between +α] is not found (step S225, "No"), then the automatic driving control unit 10112 will proceed to step S226.

[0724] Conversely, if it is determined in step S225 that [T] is satisfied L4ODDEND >T MDR If the relationship between +α] is established (step S225, "Yes"), then the automatic driving control unit 10112 proceeds to the next step S227.

[0725] In step S227, the automatic driving control unit 10112 determines whether the driver has seriously deviated from the driving position based on the monitoring results of the driver's state. If it is determined that the driver has not seriously deviated from the driving position (step S227, "No"), the automatic driving control unit 10112 returns to the processing in step S220. In this case, the driver has not seriously deviated from the driving position and can expect a response to the notification from the system.

[0726] On the other hand, if it is determined in step S227 that the driver has seriously deviated from the driving position (step S227, "Yes"), the automatic driving control unit 10112 performs the processing in step S228.

[0727] In step S228, the autonomous driving control unit 10112 determines whether there is an interval requiring a high recovery success rate. If it is determined that the interval does not exist (step S228, "No"), the autonomous driving control unit 10112 returns to the processing in step S220. In this case, for example, this operation means that during the time period of the margin α, events such as emergency braking of the vehicle via MRM will have minimal impact on surrounding vehicles.

[0728] Conversely, if it is determined that the interval exists (step S228, "Yes"), the automatic driving control unit 10112 performs the processing of step S229.

[0729] In step S229, the autonomous driving control unit 10112 determines whether there are any avoidance measures (e.g., intermediate refuge areas or waiting areas) on the way to the handover start point. If it is determined that there are avoidance measures (step S229, "Yes"), the autonomous driving control unit 10112 returns to the processing in step S220.

[0730] Conversely, if it is determined that there is no evasive action (step S229, "No"), the automatic driving control unit 10112 proceeds to step S230. In this situation, if the vehicle continues to drive, it is likely to lose evasive action and activate MRM, thereby posing a risk of causing traffic obstruction or rear-end collision to following vehicles.

[0731] In step S230, the autonomous driving control unit 10112 notifies the driver of the approaching handover point where the driver must return to manual control, and detects the driver's response to accepting the notification. In the next step S231, the autonomous driving control unit 10112 determines whether there is remaining time before the handover. If it is determined that there is no remaining time (step S231, "No"), the autonomous driving control unit 10112 processes the MRM execution sequence.

[0732] Conversely, if it is determined that there is remaining time (step S231, "Yes"), the autonomous driving control unit 10112 proceeds to step S232 and attempts to allow the driver to resume manual control driving within the allowed time. In this case, a success rate higher than RRR cannot be expected. In the next step S233, the autonomous driving control unit 10112 determines whether the handover attempted in step S232 was successful before the limit. If the handover is determined to be successful (step S233, "Yes"), the autonomous driving control unit 10112 determines that the use of an autonomous driving usage range that the vehicle has entered has been completed. Conversely, if the handover is determined to be unsuccessful (step S233, "No"), the autonomous driving control unit 10112 proceeds to the MRM execution sequence.

[0733] Incidentally, the driver's recovery behavior is obtained from the transition to the MRM execution sequence from step S231 or step S233, and from the transition to the end of an interval from step S233, and is stored as recovery behavior data. This recovery behavior affects the driver's rating.

[0734] Note that in the above explanation, Figure 17B The various decision processes in the sequence, such as the decision processes in steps S225 to S229, are executed sequentially in chronological order, but this is not limited to this example. For instance, the processes in steps S225 to S229 can be executed in parallel in chronological order to determine whether the driver is capable of recovery, and when at least one of the decisions in steps S225 to S229 indicates that recovery is difficult to achieve, the process can be directly transferred to the MRM execution sequence. This means jumping to that sequence.

[0735] <3-4. Application Examples of HCD for Level 3 Automated Driving>

[0736] Next, an application example of HCD for Level 3 autonomous driving will be described according to the embodiments.

[0737] Level 3 autonomous driving is defined as a mode in which the driver is always able to respond to anomalies. Therefore, for a vehicle to operate safely at Level 3 without disrupting public order, the driver needs to anticipate the road conditions and be prepared to revert to a manual driving posture and demeanor. This ensures that the driver can always quickly revert to manual control when using a vehicle in Level 3 autonomous driving mode. In other words, if the driver cannot be expected to be in these conditions, then the driver is no longer suitable for using the vehicle at Level 3. That is, in such situations, considering the driver's condition, it is difficult to say that driving at Level 3 is permissible.

[0738] That is, at least in the definition of autonomous driving level 3 according to the embodiment, the ODD of autonomous driving level 3 is an operational design domain that can be used when these conditions are met. In this case, in the ODD of autonomous driving level 3, the range up to the point where the driver is expected to be able to manage the situation while the driver continues to drive in the current state is the limit of the operational design domain corresponding to autonomous driving level 3.

[0739] In other words, in an ODD capable of using Level 3 autonomous driving, if the driver is no longer engaged in long-term driving tasks (i.e., a state of not driving), the driver's attention becomes distracted, and their ability to gather continuous information about the surrounding environment required for driving decreases. Therefore, it is difficult to respond to emergency handovers from autonomous driving to manual control.

[0740] While in control of the vehicle, drivers continue to gather the information necessary to perceive, identify, and judge situations. This is because behavioral judgment requires short-term predictability related to chosen actions and operations, and to ensure this predictability is as reliable as possible, drivers continuously collect a large amount of information during sustained manual control driving. This information includes not only the behavior of vehicles ahead, but also much information that is not immediately available after a request to return to manual control, such as the vehicle's cargo, road conditions further ahead of the vehicles ahead, any road congestion, and the intervals indicated by road signs.

[0741] Starting from the stage where the driver interrupts their attention to surrounding surveillance when using Level 3 autonomous driving, the information that would otherwise be needed for these judgments and is unconsciously acquired while manually controlling the vehicle gradually disappears from or stops being updated in working memory.

[0742] Therefore, in this embodiment, the ODD (Operational Disclosure) is determined by limiting the driving at Level 3 of autonomous driving, based on the driver's unique cognitive characteristics and state, and the duration of their sustained near-manual control of the surrounding environment and vehicle status. The vehicle's ODD design considers not only the vehicle's environmental recognition performance, the acquisition of prior information about the driving route, and the vehicle's self-diagnostic results, but also the driver's current state and future predictive estimates.

[0743] From the perspective of HCD (Hardware Controlled Driving), when the driver is not actively involved in actual steering (manual control), it is difficult for the driver to continuously grasp surrounding information and immediately take over steering. In autonomous driving, it is believed that for most of the time while driving, the mind is allocated to things other than driving. Therefore, working memory gradually fades (e.g., the estimated surrounding environment and vehicle characteristics (including changes) information necessary for short-term handover of manual control and essential for safe steering) and even the risk of the driver losing consciousness during steering maneuvers increases.

[0744] Therefore, the ODD for Level 3 Automated Driving defined according to the embodiment differs from existing ODDs that are designed based on system performance limits, road maintenance conditions, and prior information about the road. That is, based on the driver's level of alertness and history of continuous awareness of surrounding conditions, the ODD determines the range of actions the driver can anticipate when receiving a handover request from automated driving to manual control. Within this range, the design settings, taking into account system performance limits, road maintenance conditions, and prior information about the road, are determined as the drivable area of ​​Level 3 Automated Driving allowed by the driver's current situational awareness and handling capabilities.

[0745] Here, we will explain the extension of the interval for Level 3 autonomous driving. There exists a situation where, based on the driver's own condition, assuming the driver is allowed to use Level 3 autonomous driving for a short period of time in that condition, and after assessing the situation, the driver requests an extension from the system, and uses Level 3 autonomous driving intermittently.

[0746] Figure 18A This diagram schematically illustrates a scenario where a driver extends the Level 3 autonomous driving interval while driving the vehicle on road 70. The example shows a situation where, within an interval where conditional use of Level 3 autonomous driving is permitted, the driver repeatedly executes Level 3 autonomous driving within a short period by extending the request. That is, the driver executes Level 3 autonomous driving for a short time, requests an extension at the end of that period, and then further executes Level 3 autonomous driving for a short period. In the example shown in the diagram, the driver repeats this behavior.

[0747] Here, because the system allows extensions only through simple driver requests (e.g., button operation), the driver does not store in their working memory the attention required to confirm actual continued safety and the prerequisite information needed to grasp the situation ahead of the vehicle for handover. This increases the likelihood that the system will allow an extension when the predictive information has disappeared from the driver's working memory.

[0748] For example, the system detects confirmation behaviors such as the driver verbally pointing to the road ahead and extended requests such as button operations. Therefore, a "second transaction" can be made between the system and the driver regarding the understanding of the situation and the resulting responsibility, allowing the driver to reintegrate the sense of responsibility (i.e., the memory of reviving necessity) into their working memory.

[0749] If Level 3 autonomous driving is permitted, questions arise regarding whether the driver is in an appropriate state of alertness, posture, and consciousness during operation, and whether the driver has fulfilled their duty of continuous attention to the vehicle ahead. From an ergonomic perspective, there are no penalties for drivers neglecting their basic responsibilities; therefore, for careless drivers, these responsibilities may not necessarily be fulfilled simply because of potential risk.

[0750] The use of autonomous driving involving these violations is already known to society, and it is not permissible to ignore this situation without taking any action. That is, if these state transitions are recorded and form a feedback loop that cannot be ignored as a matter of legal punishment, then the driver will intuitively receive the "virtual pain" recorded and stored in the driver's consciousness as a violation state. This "virtual pain" is in addition to the intervals of fulfilling the duty of care to avoid the risk of the accident itself, making the punishment more intuitive.

[0751] The reason why speeding cases are rare on roads with speed monitoring or traffic enforcement is that drivers' experience with traffic enforcement, as a behavioral judgment, is seen as a near-term risk projected onto the driver's mind. Even if the same penalties are introduced, monitoring violations that are "more common and more realistic" will increase preventative awareness, thus achieving the same effect from a behavioral psychology perspective.

[0752] Violations can also be silently recorded in the driver's behavior, that is, recorded in a shadow mode without the driver's awareness. However, in the same scenario, from an ergonomic perspective, it is desirable to intentionally perform sensory recording such as visual, auditory, tactile, and olfactory...

Claims

1. An information processing device, comprising: An autonomous driving control unit, based on an operating design domain set for the vehicle, notifies the driver of the conditions that enable the vehicle to drive autonomously. When an unexpected event occurs while the vehicle is in motion, the autonomous driving control unit determines, based on the urgency of the unexpected event, whether to use autonomous driving or manual control to perform evasive maneuvers. Based on the result of the determination and the driver's behavior according to the result of the determination, the autonomous driving control unit deducts points from the driver's evaluation at any one of a first level, a second level lower than the first level, and a third level lower than the second level.

2. The information processing device according to claim 1, wherein The autonomous driving control unit receives the driver's approval response to the notification.

3. The information processing device according to claim 1, wherein The autonomous driving control unit issues the notification before the driver sits in the driver's seat of the vehicle and begins driving the vehicle.

4. The information processing device according to claim 1, wherein The autonomous driving control unit issues the notification during the manual control driving section before the vehicle enters the section that meets the conditions.

5. The information processing device according to claim 1, wherein, The autonomous driving control unit issues the notification based on the importance of the handover from autonomous driving to manual driving.

6. The information processing device according to any one of claims 1 to 5 further includes a monitoring unit, wherein the monitoring unit performs driver monitoring to monitor the driver's state. wherein The autonomous driving control unit evaluates the recovery from the driver's autonomous driving state to a state where manual control of the driver can be performed, based on the driver state obtained through the driver monitoring.

7. The information processing device according to claim 6, wherein When the evaluation of the recovery status is below a specified level, the autonomous driving control unit imposes a penalty on the driver.

8. The information processing device according to claim 7, wherein As a penalty, the autonomous driving control unit imposes restrictions on the driver's non-driving behaviors during the autonomous driving process.

9. The information processing device according to claim 7, wherein As a form of punishment, the autonomous driving control unit provides unpleasant stimulation to the driver's body during the autonomous driving process.

10. The information processing device according to claim 7, wherein As a penalty, the autonomous driving control unit imposes restrictions on the driver's use of the autonomous driving system.

11. The information processing device according to claim 6, wherein The autonomous driving control unit changes the conditions based on the evaluation.

12. The information processing device according to claim 1, wherein If it is determined that the evasive maneuver will be initiated by manual control, and there is a predetermined margin of safety before the manual control of the evasive maneuver begins, then the automatic driving control unit notifies the driver of the occurrence of the unexpected event, and If the driver's response to the notification cannot be obtained, and the margin before the manual control of the evasive maneuver begins is below a predetermined level, the automatic driving control unit initiates the evasive maneuver by using automatic driving and deducts points from the driver's evaluation to the first degree.

13. The information processing device according to claim 1, wherein If it is determined that the evasive maneuver is initiated by manual control of the driver, and it is determined that the grace period before the evasive maneuver begins cannot be extended, then the automatic driving control unit initiates the evasive maneuver by using automatic driving and deducts points from the driver's evaluation at the second level.

14. The information processing device according to claim 1, wherein If it is determined that the avoidance maneuver is initiated by using autonomous driving, the autonomous driving control unit deducts points from the driver's evaluation at the third level.

15. An information processing device, comprising: An autonomous driving control unit that controls the autonomous driving of the vehicle based on an operating design domain set for the vehicle; and The monitoring unit performs driver monitoring of the vehicle's driver. The autonomous driving control unit performs the following: Based on the driver state obtained through the driver monitoring, the recovery from the driver's autonomous driving state to a state where the driver can manually control the driving is evaluated, and based on the evaluation, the conditions under which the vehicle can perform autonomous driving based on the operating design domain are changed. as well as When an unexpected event occurs while the vehicle is in motion, the driver determines whether to use autonomous driving or manual control to avoid the event, based on the urgency of the event. Based on the result of the determination and the driver's behavior according to the result of the determination, the driver's evaluation score is deducted to any one of a first degree, a second degree lower than the first degree, and a third degree lower than the second degree.

16. An information processing method comprising steps executed by a processor, the steps including: The autonomous driving control steps, based on the operating design domain set for the vehicle, notify the driver of the conditions that enable the vehicle to drive autonomously. When an unexpected event occurs while the vehicle is in motion, the driver determines whether to use autonomous driving or manual control to avoid the event, based on the urgency of the event. Then, based on the result of this determination and the driver's behavior according to the determination, the driver's evaluation score is deducted to the degree of any one of a first level, a second level lower than the first level, and a third level lower than the second level.

17. A computer-readable storage medium having stored thereon an information processing program configured to cause a computer to perform automatic driving control steps, to notify the driver of the vehicle of conditions enabling the vehicle to drive automatically based on an operating design domain set for the vehicle, and, when an unexpected event occurs while the vehicle is in motion, to determine, based on the urgency of the unexpected event, whether to use automatic driving or manual control to perform evasive maneuvers, and, based on the result of the determination and the driver's behavior according to the result of the determination, to deduct points from the driver's evaluation to any one of a first degree, a second degree lower than the first degree, and a third degree lower than the second degree.

18. An information processing method comprising steps performed by a processor, the steps including: The autonomous driving control steps are based on the operating design domain set for the vehicle to control the vehicle's autonomous driving. and The monitoring steps include monitoring the driver of the vehicle. Among them, the autonomous driving control steps Based on the driver's state monitored through the monitoring steps, the recovery from the driver's autonomous driving state to a state where manual driving is possible is evaluated, and based on the evaluation, the conditions under which the vehicle can perform autonomous driving based on the operational design domain are changed; and... When an unexpected event occurs while the vehicle is in motion, the driver determines whether to use autonomous driving or manual control to avoid the event, based on the urgency of the event. Based on the result of the determination and the driver's behavior according to the result of the determination, the driver's evaluation score is deducted to any one of a first degree, a second degree lower than the first degree, and a third degree lower than the second degree.

19. A computer-readable storage medium having an information processing program stored thereon, the information processing program being used to cause a computer to perform... an autonomous driving control step of controlling autonomous driving of the vehicle based on the operation design domain set for the vehicle; and The monitoring steps include monitoring the driver of the vehicle. in, The autonomous driving control steps Based on the driver's state monitored through the monitoring steps, the recovery from the driver's autonomous driving state to a state where the driver can manually control the driving is evaluated, and based on the evaluation, the conditions under which the vehicle can perform autonomous driving based on the operating design domain are changed. and When an unexpected event occurs while the vehicle is in motion, the driver determines whether to use autonomous driving or manual control to avoid the event, based on the urgency of the event. Based on the result of the determination and the driver's behavior according to the result of the determination, the driver's evaluation score is deducted to any one of a first degree, a second degree lower than the first degree, and a third degree lower than the second degree.

Images