Processing system and information presentation device

Abstract

A processing system (50) comprises at least one processor (51b). The processing system (50) executes a process for performing a presentation to a driver of a vehicle (1). A processor (51b) executes evaluation of driving performed by the driver, using rules set according to a safety model for automated driving. On the basis of the evaluation, the processor (51b) executes outputting information pertaining to teaching for following the rules, so that the information can be presented to the driver.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Patent Application No. 2022-83974 filed in Japan on May 23, 2022, and the contents of the original application are incorporated by reference in their entirety. [Technical Field]

[0002] The disclosure of this specification relates to a technique for evaluating or teaching driving in a mobile vehicle. [Background technology]

[0003] The technology disclosed in Patent Document 1 evaluates the driving characteristics of a driver. Specifically, the evaluation of the driving characteristics includes evaluating compliance with traffic rules based on the speed, position, and map information of the vehicle driven by the driver, and evaluating the speed according to the position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 150425 Summary of the Invention

[0005] In recent years, the development of autonomous driving technology for mobile vehicles has progressed rapidly, and autonomous vehicles are on the verge of being driven on public roads. In this case, an environment is expected in which driver-driven mobile vehicles and mobile vehicles that are driven automatically according to rules defined by an autonomous driving safety model coexist on the road. In this environment, if a driver does not drive in accordance with the rules, the driver's driving may be evaluated relatively unfavorably.

[0006] One of the purposes of the disclosure of this specification is to provide a processing system and an information presentation device that improve the appropriateness of driving by a driver.

[0007] The processing system disclosed herein includes at least one processor, and executes a process for providing a presentation to a driver of a vehicle, the processing system comprising: The processor Evaluating the driving by the driver using rules prescribed by the safety model of automated driving; In the evaluation or separately from the evaluation, detecting a deviation of the driver from the rules of driving; and outputting information on instructions for following the rules based on the evaluation so that the information can be presented to the driver. death, When outputting, the information is output according to the degree of deviation. do. Another disclosed processing system includes at least one processor, and executes a process for providing a presentation to a driver of a vehicle, the process comprising: The processor Evaluating the driving by the driver using rules prescribed by the safety model of automated driving; outputting, based on the evaluation, information relating to lessons for following the rules so as to be presentable to the driver; Recognizing the driver's state; Extracting a causal relationship between a driver's state and a potential danger in driving by the driver; Classification of potential hazards according to their causal relationships; The instruction is based on the classification of the generating factor. Another disclosed processing system includes at least one processor, and executes a process for providing a presentation to a driver of a vehicle, the process comprising: The processor Evaluating the driving by the driver using rules prescribed by the safety model of automated driving; outputting, based on the evaluation, information relating to lessons for following the rules so as to be presentable to the driver; predicting scenarios that the moving body is expected to encounter as a result of being driven by a driver, in which the moving body will fall into an unsafe state; The instruction is a teaching for the mobile body to follow the rules in a scenario where the mobile body falls into an unsafe state. Another disclosed processing system is A processing system including at least one processor and configured to execute a process for providing a presentation to a driver of a vehicle, The processor Evaluating the driving by the driver using rules prescribed by the safety model of automated driving; outputting, based on the evaluation, information relating to lessons for following the rules so as to be presentable to the driver; and determining a presentation mode of the presentation content for implementing the instruction based on the result of the evaluation of the driving by the driver.

[0008] According to this aspect, information regarding instructions to the driver is output so that it can be presented to the driver. The rules that serve as the basis for instructions to the driver are specified by the safety model of automated driving. By having the driver refer to these instructions, it is possible to prevent the driver's driving from being evaluated unfavorably in a relative evaluation of the automated driving vehicle. Therefore, it is possible to increase the validity of the driver's driving.

[0009] The information presentation device disclosed herein is an information presentation device that presents information to a user, a communication interface configured to be able to communicate with a processing system that executes processing related to the mobile body, and configured to be able to acquire, from the processing system, information regarding instructions for a driver of the mobile body to follow rules defined by the safety model of autonomous driving; a user interface configured to be able to present presentation content relating to instructions for following the rules based on the information; 、 The presented content includes a combination of visual information showing a scenario that the vehicle will encounter while driving by the driver and audio information giving advice on how to improve driving in the scenario. .

[0010] According to this aspect, the user interface presents presentation content based on information regarding instructions to the driver, which is acquired from the communication interface. The rules that serve as the basis for the instructions to the driver are defined by an automated driving safety model. By having the driver refer to these instructions, it is possible to prevent the driver's driving from being evaluated unfavorably in a relative evaluation of the automated driving vehicle. Therefore, it is possible to increase the appropriateness of the driver's driving. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram showing a schematic configuration of the driving system. [Figure 2] FIG. 1 is a block diagram showing the configuration of the technology level of the driving system. [Figure 3] FIG. 2 is a block diagram showing the functional level configuration of the driving system. [Figure 4]FIG. 2 is a block diagram showing a configuration for realizing an evaluation function and a teaching function. [Figure 5] FIG. 1 illustrates a scenario related to driving evaluation. [Figure 6] FIG. 1 illustrates a scenario related to driving evaluation. [Figure 7] FIG. 1 illustrates a scenario related to driving evaluation. [Figure 8] FIG. 1 is a block diagram showing a configuration for realizing content generation and presentation. [Figure 9] FIG. 1 is a block diagram showing a configuration for realizing content generation and presentation. [Figure 10] FIG. 1 is a block diagram showing a configuration for realizing content generation and presentation. [Figure 11] FIG. 10 is a diagram illustrating presented content. [Figure 12] FIG. 10 is a diagram illustrating presented content. [Figure 13] FIG. 10 is a diagram illustrating presented content. [Figure 14] 10 is a flowchart illustrating an evaluation process and a teaching process. [Figure 15] 10 is a flowchart illustrating a process of estimating a risk level. [Figure 16] 10 is a flowchart illustrating a process of presenting information to a driver. [Figure 17] 10 is a flowchart illustrating a presentation process while the driver is driving. [Figure 18] 10 is a flowchart illustrating a presentation process after a driver finishes driving. [Figure 19] FIG. 10 is a diagram illustrating prediction of a driver's state. [Figure 20] 10 is a flowchart illustrating a process of estimating a risk level. [Figure 21] FIG. 10 is a diagram illustrating the estimation of a causal relationship. [Figure 22] 10 is a flowchart illustrating a process of estimating a risk level. [Figure 23] 10 is a flowchart illustrating a process of presenting information to a driver. [Figure 24]10 is a flowchart illustrating a process of presenting information to a driver. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, several embodiments will be described with reference to the drawings. Note that corresponding components in each embodiment are given the same reference numerals, and redundant description may be omitted. When only a portion of the configuration is described in each embodiment, the configuration of another embodiment described previously can be applied to the remaining portion of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of several embodiments can also be partially combined together even if not explicitly stated, as long as there is no particular problem with the combination.

[0013] In the following embodiments, the contents of “Safety First for Automated Driving” by Aptiv, Audi, Baidu, BMW, Continental, Daimler, FCA, here, Infineon, Intel, and Volkswagen, Tech.Rep., 2019; “On a formal model of safe and scalable self-driving cars,” by S. Shalev-Shwartz, S. Shammah, and A. Shashua, arXiv:1708.06374, 2017; and “The Safety Force Field” Technical report by David Nister, Hon-Leung Lee, Julia Ng, and Yizhou Wang, 2019, are incorporated by reference in their entirety.

[0014] (First embodiment) The driving system 2 of the first embodiment shown in FIG. 1 realizes functions related to driving a moving object. Part or all of the driving system 2 is mounted on the moving object. The moving object that is the target of processing by the driving system 2 is a vehicle 1. This vehicle 1 can be called the host vehicle and corresponds to a host moving object. The vehicle 1 may be configured to be able to communicate with other vehicles directly or indirectly via a communication infrastructure. The other vehicles correspond to target moving objects.

[0015] The vehicle 1 may be a road user capable of performing manual driving, such as a car or truck. The vehicle 1 may also be capable of performing automated driving. Driving is classified into levels according to the extent to which the driver performs all dynamic driving tasks (DDTs). Automated driving levels are specified, for example, in SAE J3016. At levels 0 to 2, the driver performs some or all of the DDTs. Levels 0 to 2 may be classified as so-called manual driving. Level 0 indicates that driving is not automated. Level 1 indicates that the driver is assisted by the driving system 2. Level 2 indicates that driving is partially automated.

[0016] At levels 3 and above, the driving system 2 performs all of the DDT while engaged. Levels 3 to 5 may be classified as so-called automated driving. A system capable of driving at levels 3 and above may be called an automated driving system. Level 3 indicates that driving is conditionally automated. Level 4 indicates that driving is highly automated. Level 5 indicates that driving is fully automated.

[0017] Furthermore, a driving system 2 that is unable to perform driving at level 3 or above but can perform driving at least at level 1 or 2 may be referred to as a driving assistance system. In the following, unless there are particular circumstances to specify the maximum achievable level of autonomous driving, the autonomous driving system or driving assistance system will be referred to simply as driving system 2 and the explanation will continue.

[0018] <Sense-Plan-Act Model> The architecture of the driving system 2 is selected so as to enable an efficient SOTIF (safety of the intended functionality) process. For example, the architecture of the driving system 2 may be configured based on the sense-plan-act model. The sense-plan-act model includes a sense element, a plan element, and an act element as main system elements. The sense element, plan element, and act element interact with each other. Here, sense may be replaced with perception, plan with judgment, and act with control, respectively.

[0019] As shown in Fig. 1, in such a driving system 2, at the vehicle level, vehicle level functions 3 are implemented based on a Vehicle Level Safety Strategy (VSLL). At the function level (in other words, from a functional perspective), a perception function, a decision-making function, and a control function are implemented. At the technical level (reduced to a technical perspective), at least a plurality of sensors 40 corresponding to the perception function, at least one processing system 50 corresponding to the decision-making function, and a plurality of motion actuators 60 corresponding to the control function are implemented.

[0020] In detail, a recognition unit 10 may be constructed in the driving system 2 as a functional block realizing a recognition function, mainly consisting of a plurality of sensors 40, a processing system that processes information detected by the plurality of sensors 40, and a processing system that generates an environmental model based on information from the plurality of sensors 40. A judgment unit 20 may be constructed in the driving system 2 as a functional block realizing a judgment function, mainly consisting of a processing system 50. A control unit 30 may be constructed in the driving system 2 as a functional block realizing a control function, mainly consisting of a plurality of movement actuators 60 and at least one processing system that outputs operation signals for the plurality of movement actuators 60.

[0021] Here, the recognition unit 10 may be realized in the form of a recognition system 10a as a subsystem provided so as to be distinguishable from the determination unit 20 and the control unit 30. The determination unit 20 may be realized in the form of a determination system 20a as a subsystem provided so as to be distinguishable from the recognition unit 10 and the control unit 30. The control unit 30 may be realized in the form of a control system 30a as a subsystem provided so as to be distinguishable from the recognition unit 10 and the determination unit 20. The recognition system 10a, the determination system 20a, and the control system 30a may constitute components independent of each other.

[0022] Furthermore, a plurality of HMI (Human Machine Interface) devices 70 may be mounted on the vehicle 1. A portion of the plurality of HMI devices 70 that realizes an operation input function by an occupant may be part of the recognition unit 10. A portion of the plurality of HMI devices 70 that realizes an information presentation function may be part of the control unit 30. On the other hand, the function realized by the HMI device 70 may be positioned as a function independent of the recognition function, the judgment function, and the control function.

[0023] The recognition unit 10 is responsible for recognition functions, including localization (e.g., location estimation) of the vehicle 1, other vehicles, and other road users. The recognition unit 10 detects the external environment, internal environment, vehicle state, and the state of the driving system 2 of the vehicle 1. The recognition unit 10 combines the detected information to generate an environmental model. The judgment unit 20 applies the objective and driving policy to the environmental model generated by the recognition unit 10 to derive a control action. The control unit 30 executes the control action derived by the recognition unit 10.

[0024] <Technical level system configuration> An example of a detailed configuration of the driving system 2 at a technical level will be described using FIG. 2. The configuration at the technical level may refer to a physical architecture. The driving system 2 includes a plurality of sensors 40, a plurality of motion actuators 60, a plurality of HMI devices 70, and at least one processing system. These components can communicate with each other via one or both of wireless and wired connections. These components may also be able to communicate with each other via an in-vehicle network such as CAN (registered trademark).

[0025] The multiple sensors 40 include one or more external environment sensors 41. The multiple sensors 40 may include at least one of one or more internal environment sensors 42, one or more communication systems 43, and a map database (DB) 44. When the sensor 40 is interpreted narrowly to refer to the external environment sensor 41, the internal environment sensor 42, the communication system 43, and the map database 44 may be positioned as components separate from the sensor 40 corresponding to the technical level of the recognition function.

[0026] The external environment sensor 41 may detect targets present in the external environment of the vehicle 1. Examples of the target detection type external environment sensor 41 include a camera 41a, a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging) 41b, a laser radar, a millimeter wave radar, an ultrasonic sonar, and an imaging radar. As a typical example of sensor installation, the vehicle 1 may be equipped with a plurality of cameras 41a (for example, eleven cameras 41a) configured to monitor the front, front-side, side, rear-side, and rear directions of the vehicle 1, respectively.

[0027] As another example of installation, the vehicle 1 may be equipped with a plurality of cameras 41a (e.g., four cameras 41a) configured to monitor the front, sides, and rear of the vehicle 1, a plurality of millimeter-wave radars (e.g., five millimeter-wave radars) configured to monitor the front, front-side, sides, and rear of the vehicle 1, and a LiDAR 41b configured to monitor the front of the vehicle 1.

[0028] Furthermore, the external environment sensor 41 may detect atmospheric conditions and weather conditions in the external environment of the vehicle 1. The condition detection type external environment sensor 41 is, for example, an outside air temperature sensor, a temperature sensor, a raindrop sensor, or the like.

[0029] The internal environment sensor 42 may detect a specific physical quantity related to vehicle motion (hereinafter referred to as a motion physical quantity) in the internal environment of the vehicle 1. Examples of the motion physical quantity detection type internal environment sensor 42 include a speed sensor 42c, an acceleration sensor, and a gyro sensor. The internal environment sensor 42 may detect the state of an occupant (e.g., the driver state) in the internal environment of the vehicle 1. Examples of the occupant detection type internal environment sensor 42 include an actuator sensor, a sensor and system for monitoring the driver (hereinafter referred to as a driver monitor 42a), a biological sensor, a pulse wave sensor 42b, a seating sensor, and a vehicle equipment sensor. Here, examples of the actuator sensor in particular include an accelerator sensor, a brake sensor, a steering sensor, and the like that detect the state of driver operation of a motion actuator 60 related to the motion control of the vehicle 1.

[0030] The communication system 43 acquires communication data usable in the driving system 2 via wireless communication. The communication system 43 may receive positioning signals from artificial satellites of a global navigation satellite system (GNSS) present in the external environment of the vehicle 1. The communication device of the communication system 43 is, for example, a GNSS receiver.

[0031] The communication system 43 may transmit and receive communication signals to and from an external system 96 that exists in the external environment of the vehicle 1. Examples of V2X type communication devices in the communication system 43 include a dedicated short range communications (DSRC) communication device, a cellular V2X (C-V2X) communication device, etc. Examples of communication with the external system 96 that exists in the external environment of the vehicle 1 include communication with a system of another vehicle (V2V), communication with infrastructure equipment such as a communication device installed in a traffic light (V2I), communication with a mobile terminal of a pedestrian (V2P), and communication with a network such as a cloud server (V2N).

[0032] Furthermore, the communication system 43 may transmit and receive communication signals to and from the internal environment of the vehicle 1, for example, a mobile terminal 91 such as a smartphone brought into the vehicle. The terminal communication type communication device in the communication system 43 is, for example, a Bluetooth (registered trademark) communication device, a Wi-Fi (registered trademark) communication device, an infrared communication device, etc.

[0033] The map DB 44 is a database that stores map data that can be used by the driving system 2. The map DB 44 includes at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium. The map DB 44 may include a database for a navigation unit that navigates the driving route to the destination of the vehicle 1. The map DB 44 may include a database of high-precision maps with a high level of accuracy that are primarily used for automated driving systems. The map DB 44 may also include a database of parking lot maps that include detailed parking lot information, such as parking space information, that is used for automated parking or parking assistance.

[0034] The map DB 44 suitable for the driving system 2 may acquire and store the latest map data by communicating with a map server via, for example, a V2X-type communication system 43. The map data is data representing the external environment of the vehicle 1, and is converted into two-dimensional or three-dimensional data. The map data may include, for example, marking data representing at least one of the position coordinates, shape, road surface condition, and standard driving route of a road structure. The marking data included in the map data may include marking data representing at least one of the position coordinates and shape of landmarks, such as road signs, road markings, and lane markings. The marking data included in the map data may represent, for example, traffic signs, arrow markings, lane markings, stop lines, directional signs, landmark beacons, business signs, changes in road line patterns, and the like. The map data may also include structure data representing at least one of the position coordinates and shape of buildings and traffic lights facing the road. The marking data included in the map data may represent landmarks such as street lights, road edges, reflectors, balls, and the like.

[0035] The motion actuator 60 can control vehicle motion based on an input control signal. The drive-type motion actuator 60 is, for example, a power train including at least one of an internal combustion engine, a drive motor, etc. The braking-type motion actuator 60 is, for example, a brake actuator. The steering-type motion actuator 60 is, for example, a steering.

[0036] At least one of the HMI devices 70 may be an operation input device that can input operations by occupants including the driver of the vehicle 1 in order to transmit their will or intentions to the driving system 2. Examples of the operation input type HMI device 70 include an accelerator pedal, a brake pedal, a shift lever, a steering wheel, a turn signal lever, a mechanical switch, and a touch panel of a navigation unit. Of these, the accelerator pedal controls a powertrain as a motion actuator 60. The brake pedal controls a brake actuator as a motion actuator 60. The steering wheel controls a steering actuator as a motion actuator 60.

[0037] At least one of the HMI devices 70 may be an information presentation device equipped with a user interface 70b that presents information such as visual information, auditory information, and cutaneous information to occupants including the driver of the vehicle 1. Examples of the visual information presentation type HMI device 70 include a graphic meter, a combination meter, a navigation unit, a CID (center information display), a HUD (head-up display), and an illumination unit. Examples of the auditory information presentation type HMI device 70 include a speaker and a buzzer. Examples of the cutaneous information presentation type HMI device 70 include a steering wheel vibration unit, a driver's seat vibration unit, a steering wheel reaction force unit, an accelerator pedal reaction force unit, a brake pedal reaction force unit, an air conditioning unit, and the like.

[0038] Furthermore, the HMI device 70 may realize an HMI function linked to a mobile terminal 91 such as a smartphone by communicating with the terminal 91 via the communication system 43. For example, the HMI device 70 may present information acquired from a smartphone to occupants including the driver. Alternatively, for example, operation input to a smartphone may be used as an alternative means for the HMI device 70. Furthermore, the mobile terminal 91 that can communicate with the driving system 2 via the communication system 43 may function as the HMI device 70 itself.

[0039] As described above, the HMI device 70 may include a communication interface 70a and a user interface 70b. For example, when presenting visual information, the user interface 70b may include a device for presenting visual information, such as a display for displaying images or an emitting light. The user interface 70b may further include a circuit for controlling the device. The communication interface 70a may include at least one of a circuit and a terminal for communicating with other devices or systems via an in-vehicle network.

[0040] At least one processing system 50 is provided. For example, the processing system 50 may be an integrated processing system that integrally executes processing related to the recognition function, processing related to the judgment function, and processing related to the control function. In this case, the integrated processing system 50 may further execute processing related to the HMI function, or a processing system dedicated to the HMI function may be provided separately. For example, the processing system dedicated to the HMI function may be an integrated cockpit system that integrally executes processing related to each HMI device.

[0041] For example, the processing system 50 may be configured to have at least one processing unit corresponding to processing related to the recognition function, at least one processing unit corresponding to processing related to the judgment function, and at least one processing unit corresponding to processing related to the control function.

[0042] The processing system 50 has an interface to the outside and is connected via a communication means to at least one type of element related to processing by the processing system 50. The communication means is, for example, at least one type of element selected from a LAN (Local Area Network), a CAN (registered trademark), a wire harness, an internal bus, and a wireless communication circuit. The elements related to processing by the processing system 50 include the sensor 40, the motion actuator 60, and the HMI device 70.

[0043] The processing system 50 includes at least one dedicated computer 51. The processing system 50 may combine multiple dedicated computers 51 to realize functions such as recognition functions, judgment functions, control functions, and HMI functions.

[0044] For example, the dedicated computer 51 constituting the processing system 50 may be an integration ECU that integrates the driving functions of the vehicle 1. The dedicated computer 51 constituting the processing system 50 may be a determination ECU that determines the DDT. The dedicated computer 51 constituting the processing system 50 may be a monitoring ECU that monitors the driving of the vehicle 1. The dedicated computer 51 constituting the processing system 50 may be an evaluation ECU that evaluates the driving of the vehicle 1. The dedicated computer 51 constituting the processing system 50 may be a navigation ECU that navigates the driving route of the vehicle 1.

[0045] Furthermore, the dedicated computer 51 constituting the processing system 50 may be a locator ECU that estimates the position of the vehicle 1. The dedicated computer 51 constituting the processing system 50 may be an image processing ECU that processes image data detected by the external environment sensor 41. The dedicated computer 51 constituting the processing system 50 may be an HCU (HMI Control Unit) that controls the HMI device 70 in an integrated manner.

[0046] The dedicated computer 51 constituting the processing system 50 may have at least one memory 51a and one processor 51b. The memory 51a may be at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores programs and data that can be read by the processor 51b. The memory 51a may further include a rewritable volatile storage medium, such as a random access memory (RAM). The processor 51b includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), or a reduced instruction set computer (RISC)-CPU.

[0047] The dedicated computer 51 constituting the processing system 50 may be an SoC (System on a Chip) that integrates memory, a processor, and an interface on a single chip, or may have an SoC as a component of the dedicated computer 51.

[0048] Furthermore, the processing system 50 may include at least one database for executing the dynamic driving task. The database may be configured to include at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, and an interface for accessing the storage medium. The database may be a scenario DB 53 that stores a scenario structure in a database. Note that the scenario DB 53 does not need to be provided in the driving system 2, and may be configured, for example, in an external system 96 so that it can be accessed from the processing system 50 of the vehicle 1 via the communication system 43.

[0049] The scenario DB 53 may include at least one of functional scenarios, logical scenarios, and concrete scenarios. A functional scenario defines the highest level of qualitative scenario structure. A logical scenario is a scenario in which a quantitative parameter range is assigned to a structured functional scenario. A concrete scenario defines the boundary of safety judgment that distinguishes between safe and unsafe states.

[0050] The processing system 50 may also include at least one recording device 55 that records at least one of recognition information, judgment information, and control information of the driving system 2. The recording device 55 may include at least one memory 55a and an interface 55b for writing data to the memory 55a. The memory 55a may be at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium.

[0051] At least one of the memories 55a may be mounted on the board in a form that is not easily detachable or replaceable, and in this form, for example, an eMMC (embedded multi media card) using a flash memory may be used. At least one of the memories 55a may be detachable and replaceable from the recording device 55, and in this form, for example, an SD card may be used.

[0052] The recording device 55 may have a function of selecting information to be recorded from the recognition information, judgment information, and control information. In this case, the recording device 55 may have a dedicated computer 55c. In the dedicated computer 55c provided in the recording device 55, a processor may temporarily store information in RAM or the like. The processor may select information to be non-temporarily recorded from the temporarily stored information and save the selected information in the memory 51a.

[0053] The mobile terminal 91, which can communicate with the processing system 50 via the communication system 43, may be, for example, a smartphone or a tablet terminal. The mobile terminal 91 may include, for example, a dedicated computer 92, a user interface 94, and a communication interface 93.

[0054] The dedicated computer 92 constituting the mobile terminal 91 may have at least one memory 92a and one processor 92b. The memory 92a may be at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores programs and data that can be read by the processor 92b. The memory 92a may further include a rewritable volatile storage medium, such as a random access memory (RAM). The processor 92b includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), or a reduced instruction set computer (RISC)-CPU.

[0055] The user interface 94 may include a display and a speaker. The display may be a display capable of displaying color images, such as a liquid crystal display or an OLED display. The display and speaker can present information to the user under the control of the dedicated computer 92.

[0056] The communication interface 93 transmits and receives communication signals to and from an external device or system, and may include at least one type of communication device such as a cellular V2X (C-V2X) communication device, a Bluetooth (registered trademark) communication device, a Wi-Fi (registered trademark) communication device, or an infrared communication device.

[0057] The external system 96, which can communicate with the processing system 50 via the communication system 43, may be, for example, a cloud server or a remote center. The external system 96 may include at least one dedicated computer 97 and at least one driving information DB 98.

[0058] The dedicated computer 97 constituting the external system 96 may have at least one memory 97a and one processor 97b. The memory 97a may be at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores programs and data readable by the processor 97b. The memory 97a may further include a rewritable volatile storage medium, such as a random access memory (RAM). The processor 97b includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), or a reduced instruction set computer (RISC)-CPU.

[0059] The driving information DB 98 is a database that records and accumulates information about the driving of a plurality of vehicles including the vehicle 1. The driving information DB 98 has a large-capacity storage area, and may be configured to include at least one type of non-transient tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores data that can be read by the processor 97b, and an interface for accessing the storage medium.

[0060] <Functional level system configuration> Next, an example of a detailed configuration of the driving system 2 at the functional level will be described with reference to Fig. 3. The functional level configuration may refer to a logical architecture. The recognition unit 10 may include an external recognition unit 11, a self-location recognition unit 12, a fusion unit 13, and an internal recognition unit 14 as sub-blocks that further classify the recognition functions.

[0061] The external recognition unit 11 individually processes the detection data detected by each external environment sensor 41, and realizes the function of recognizing objects such as targets and other road users. The detection data may be detection data provided by, for example, millimeter-wave radar, sonar, LiDAR 41b, etc. The external recognition unit 11 may generate relative position data including the direction, size, and distance of the object relative to the vehicle 1 from the raw data detected by the external environment sensors 41.

[0062] Furthermore, the detection data may be image data provided by, for example, the camera 41a, the LiDAR 41b, etc. The external recognition unit 11 processes the image data and extracts objects reflected within the angle of view of the image. The object extraction may include estimation of the direction, size, and distance of the object relative to the vehicle 1. The object extraction may also include classifying the object using, for example, semantic segmentation.

[0063] The self-location recognition unit 12 performs localization of the vehicle 1. The self-location recognition unit 12 acquires global position data of the vehicle 1 from a communication system 43 (e.g., a GNSS receiver). In addition, the self-location recognition unit 12 may acquire at least one of the position information of targets extracted by the external recognition unit 11 and the position information of targets extracted by the fusion unit 13. The self-location recognition unit 12 also acquires map information from a map DB 44. The self-location recognition unit 12 integrates this information to estimate the position of the vehicle 1 on the map.

[0064] The fusion unit 13 fuses the external recognition information of each external environment sensor 41 processed by the external recognition unit 11, the localization information processed by the self-position recognition unit 12, and the V2X information acquired by V2X.

[0065] The fusion unit 13 fuses object information of other road users and the like individually recognized by each external environment sensor 41, and identifies the type and relative position of the object around the vehicle 1. The fusion unit 13 fuses road target information individually recognized by each external environment sensor 41, and identifies the static structure of the road around the vehicle 1. The static structure of the road includes, for example, curve curvature, number of lanes, free space, and the like.

[0066] Next, the fusion unit 13 fuses the types of objects around the vehicle 1, their relative positions, the static structure of the road, the localization information, and the V2X information to generate an environment model. The environment model can be provided to the determination unit 20. The environment model may be a model specialized for modeling the external environment.

[0067] The environmental model may also be a comprehensive model that is realized by adding acquired information and that combines information such as the internal environment, the vehicle state, and the state of the driving system 2. For example, the fusion unit 13 may acquire traffic rules such as the Road Traffic Act and reflect them in the environmental model.

[0068] The internal recognition unit 14 processes the detection data detected by each internal environment sensor 42 and realizes a function of recognizing the vehicle state. The vehicle state may include the state of the physical quantities of motion of the vehicle 1 detected by the speed sensor 42c, acceleration sensor, gyro sensor, etc. The vehicle state may also include at least one of the states of the occupants including the driver, the operation state of the driver with respect to the motion actuator 60, and the switch state of the HMI device 70.

[0069] The determination unit 20 may include an environment determination unit 21, an operation planning unit 22, and a mode management unit 23 as sub-blocks that further classify the determination functions.

[0070] The environment determination unit 21 acquires the environment model generated by the fusion unit 13 and the vehicle state recognized by the internal recognition unit 14, and makes a determination about the environment based on these. Specifically, the environment determination unit 21 may interpret the environment model and estimate the current situation of the vehicle 1. The situation here may be an operational situation. The environment determination unit 21 may interpret the environment model and predict the behavior of other road users. The environment determination unit 21 may interpret the environment model and predict the trajectories of objects such as other road users. The environment determination unit 21 may also interpret the environment model and predict potential hazards.

[0071] The environment determination unit 21 may also interpret the environment model and make a determination regarding the scenario in which the vehicle 1 is currently located. The determination regarding the scenario may involve selecting at least one scenario in which the vehicle 1 is currently located from a catalog of scenarios constructed in the scenario DB 53.

[0072] Furthermore, the environment judgment unit 21 may estimate the driver's intention based on at least one of the predicted behavior, the predicted object trajectory, the predicted potential hazard, and the judgment regarding the scenario, and the vehicle state provided by the internal recognition unit 14.

[0073] The driving planning unit 22 plans the driving of the vehicle 1 based on at least one of the estimated information of the vehicle 1's position on a map by the self-position recognition unit 12, the judgment information and driver intention estimation information by the environment judgment unit 21, and the function constraint information by the mode management unit 23.

[0074] The driving planner 22 realizes a route planning function, a behavior planning function, and a trajectory planning function. The route planning function is a function of planning at least one of a route to a destination and a mid-range lane plan based on estimated information of the position of the vehicle 1 on a map. The route planning function may further include a function of determining at least one of a lane change request and a deceleration request based on the mid-range lane plan. Here, the route planning function may be a mission / route planning function in a strategic function, and may be a function of outputting a mission plan and a route plan.

[0075] The behavior planning function is a function that plans the behavior of the vehicle 1 based on at least one of the route to the destination planned by the route planning function, the mid-distance lane plan, the lane change request and the deceleration request, the judgment information and the driver's intention estimation information by the environment judgment unit 21, and the function constraint information by the mode management unit 23. The behavior planning function may include a function that generates conditions related to the state transition of the vehicle 1. The conditions related to the state transition of the vehicle 1 may correspond to triggering conditions. The behavior planning function may include a function that determines the state transition of the application that realizes the DDT and further the state transition of the driving behavior based on the conditions. The behavior planning function may include a function that determines longitudinal constraints on the path of the vehicle 1 and lateral constraints on the path of the vehicle 1 based on the state transition information. The behavior planning function may be a tactical behavior plan in the DDT function and may output a tactical behavior.

[0076] The trajectory planning function is a function that plans the driving trajectory of the vehicle 1 based on the judgment information by the environment judgment unit 21, longitudinal constraints on the path of the vehicle 1, and lateral constraints on the path of the vehicle 1. The trajectory planning function may include a function that generates a path plan. The path plan may include a speed plan, or the speed plan may be generated as a plan independent of the path plan. The trajectory planning function may include a function that generates multiple path plans and selects an optimal path plan from the multiple path plans, or a function that switches between path plans. The trajectory planning function may further include a function that generates backup data of the generated path plan. The trajectory planning function may be a trajectory planning function in the DDT function, and may output a trajectory plan.

[0077] The mode management unit 23 monitors the driving system 2 and sets constraints on driving-related functions. The mode management unit 23 may manage the autonomous driving mode, for example, the autonomous driving level. The management of the autonomous driving level may include management of switching between manual driving and autonomous driving, i.e., management of the transfer of authority between the driver and the driving system 2, in other words, management of takeover. The mode management unit 23 may monitor the status of subsystems related to the driving system 2 and determine system malfunctions (e.g., errors, unstable operating states, system failures, and malfunctions). The mode management unit 23 may determine a mode based on the driver's intention based on driver's intention estimation information generated by the internal recognition unit 14. The mode management unit 23 may set constraints on driving-related functions based on at least one of the system malfunction determination result, the mode determination result, the vehicle state determined by the internal recognition unit 14, the sensor abnormality (or sensor failure) signal output from the sensor 40, the application state transition information and trajectory plan determined by the driving planner 22, etc.

[0078] Furthermore, the mode management unit 23 may have a comprehensive function of determining, in addition to constraints on the driving functions, longitudinal constraints on the path of the vehicle 1 and lateral constraints on the path of the vehicle 1. In this case, the operation planning unit 22 plans the behavior and the trajectory in accordance with the constraints determined by the mode management unit 23.

[0079] The control unit 30 may include a motion control unit 31 and an HMI output unit 71 as sub-blocks that further classify the control functions. The motion control unit 31 controls the motion of the vehicle 1 based on the trajectory plan (e.g., a path plan and a speed plan) acquired from the driving plan unit 22. Specifically, the motion control unit 31 generates accelerator request information, shift request information, brake request information, and steering request information according to the trajectory plan, and outputs them to the motion actuator 60.

[0080] Here, the motion control unit 31 can directly obtain the vehicle state recognized by the recognition unit 10 (particularly the internal recognition unit 14), such as at least one of the current speed, acceleration, and yaw rate of the vehicle 1, from the recognition unit 10 and reflect this in the motion control of the vehicle 1.

[0081] The HMI output unit 71 outputs information related to the HMI based on at least one of the judgment information and driver's intention estimation information from the environment judgment unit 21, the application state transition information and trajectory plan from the driving planner 22, and function constraint information from the mode manager 23. The HMI output unit 71 may manage vehicle interactions. The HMI output unit 71 may generate a notification request based on the management state of the vehicle interactions and control the information presentation function of the HMI device 70. Furthermore, the HMI output unit 71 may generate control requests for wipers, a sensor washing device, headlights, and an air conditioner based on the management state of the vehicle interactions and control these devices.

[0082] <Safety model and its rules> The driving system 2 may be configured to incorporate assumptions about the reasonably foreseeable behavior of other road users, which are taken into account in a safety model for automated driving. The safety model may correspond to, for example, a safety-related model or a formal model. As the safety model, for example, the Responsibility-Sensitive Safety (RSS) model or the Safety Force Field (SFF) model may be adopted, but other models, more generalized models, or composite models combining multiple models may also be adopted.

[0083] For example, the RSS model adopts five rules (five principles). The first rule is, "Do not hit someone from behind." The second rule is, "Do not cut-in recklessly." The third rule is, "Right-of-way is given, not taken." The fourth rule is, "Be careful of area with limited visibility." The fifth rule is, "If you can avoid an accident without causing another one, you must do it." These rules may correspond to the rules prescribed by the safety model of autonomous driving.

[0084] Based on the five rules, especially the first and second rules, a safety envelope can be defined. The safety envelope may refer to the longitudinal and lateral safety distances themselves relative to other road users, or may refer to the conditions or concepts for calculating these safety distances. The longitudinal and lateral safety distances may be calculated taking into account reasonably foreseeable assumptions about other road users.

[0085] The longitudinal safe distance may be the distance at which a rear-end collision will not occur when a leading vehicle, traveling at a predetermined speed, brakes at maximum speed and stops, and a following vehicle accelerates with a predetermined response time and maximum acceleration, and then brakes at minimum deceleration and stops.Also, the longitudinal safe distance may be the distance at which a head-on collision will not occur when two vehicles, traveling toward each other at their respective speeds, accelerate with a predetermined response time and maximum acceleration, and then brakes at minimum deceleration and stops.

[0086] The lateral safety distance may be the minimum distance that will prevent a collision even if two vehicles are traveling side by side at a lateral speed, accelerate at a predetermined reaction time and maximum acceleration, and then decelerate laterally at a maximum deceleration.

[0087] For example, the SFF model adopts one core principle: "All actors are required to apply safety control actions that contribute at least as much as the safety procedures to improving the safety potential." This principle may correspond to the rules prescribed by the safety model for automated driving.

[0088] Here, the amount of space-time between the two deceleration schedules, the safety procedure schedule and the maximum braking schedule, is defined as the claimed set. The safety potential can be defined as a measure of the overlap between the claimed sets of two vehicles. The SFF can be defined as the negative slope of the safety potential.

[0089] <Driving evaluation and instruction> The driving system 2 of this embodiment has a function to evaluate driving (hereinafter referred to as evaluation function) and a function to provide instruction (hereinafter referred to as instruction function) to a driver who performs manual driving and a driver who performs manual driving with driving assistance. The driver who performs manual driving may be a driver who drives the vehicle 1 in an autonomous driving level 0 state. The driver who performs manual driving with driving assistance may be a driver who drives the vehicle 1 in an autonomous driving level 1 or 2 state. The driving system 2 can present instructions to the driver via the HMI device 70 for following the rules stipulated by the safety model of the autonomous driving model.

[0090] For example, in the processing system 50, the evaluation function and the teaching function may be realized by further constructing functional blocks such as an information acquisition unit 72, a driver estimation unit 73, a driving behavior information generation unit 74, and a risk level estimation unit 75 as shown in Fig. 4 using a dedicated computer 51. When at least some of the functions realized by the information acquisition unit 72, the driver estimation unit 73, the driving behavior information generation unit 74, and the risk level estimation unit 75 overlap with the functions of the environment determination unit 21, the driving plan unit 22, and the mode management unit 23, the overlapping functional blocks may take on those functions.

[0091] The information acquisition unit 72 acquires information necessary for realizing the teaching function. The information necessary for realizing the teaching function may be, for example, various information on the vehicle state, the driver state, and the external environment. This information may be acquired directly from detection data detected by the sensors 40, such as the speed sensor 42c and the communication system 43, or may be acquired from an environmental model generated based on this detection data.

[0092] The driver estimation unit 73 performs estimation regarding the driver using the information acquired by the information acquisition unit 72. The estimation regarding the driver may be at least one of estimation of the current driver state, estimation of the future driver state, and estimation of the current driver's intention.

[0093] The estimation of the driver state may include estimating whether the driver state is positive or negative. The estimation of whether the driver state is positive or negative may be performed based on the driver's facial expression and heart rate.

[0094] For example, the information acquired by the information acquisition unit 72 may be input as input parameters to a trained neural network constructed in the processing system 50, thereby obtaining an analysis result indicating whether the driver's state is positive or negative. Specifically, an image of the driver's face captured by the driver monitor 42a and heart rate data of the driver detected by the pulse wave sensor 42b are input as input parameters to the neural network. Then, based on the analysis result output from the neural network, it may be estimated whether the driver's state is positive or negative. The analysis result may, for example, represent a numerical value from 0 to 100 indicating each emotion of the driver. For example, if the index of the driver's "happy" emotion is high, the driver's state is estimated to be positive. Alternatively, if the index of the driver's "sad" emotion is high, the driver's state is estimated to be negative.

[0095] The driving behavior information generation unit 74 detects the driving behavior of the driver and generates information about the driving behavior. Here, generating information about the driving behavior may simply mean extracting the behavior of the vehicle 1 as a result of the driving behavior of the driver. Here, generating information about the driving behavior may further include associating the behavior of the vehicle 1 with the external environment. Associating the behavior of the vehicle 1 with the external environment may be generating information that associates the external environment with the behavior of the vehicle 1. Information that associates the external environment with the behavior of the vehicle 1 is, for example, information that the vehicle 1 proceeded through an intersection despite the traffic light displaying a stop signal, information that the vehicle 1 went straight through the intersection from a right-turn-only lane, etc.

[0096] Furthermore, generating information about driving behavior may include associating information that associates the external environment with the behavior of the vehicle 1 with rules defined by a safety model for autonomous driving.

[0097] The risk estimation unit 75 estimates the risk of driving by the driver. The risk estimation here may be an example of an evaluation of the driving by the driver. The risk here may indicate, for example, the possibility of interference or collision with other road users. For example, when the RSS model is adopted as a safety model for autonomous driving, the risk may be replaced with a responsibility value indicating the degree of accident responsibility that the vehicle 1 bears toward other road users, or may be a concept equivalent to the responsibility value.

[0098] The risk estimation may include evaluating the driver's driving using rules defined by a safety model for automated driving. The evaluation using the rules defined by the safety model for automated driving may include determining whether the vehicle 1 violates the rules. This determination may be performed under the assumption that the manually driven vehicle 1 is autonomous. For example, this determination may include determining whether the vehicle 1 violates a safety envelope. For example, if an RSS model is adopted as the safety model for automated driving, this determination may include determining whether the distance between the vehicle 1 and other road users, such as other vehicles, is below a safe distance.

[0099] The evaluation using the rules defined by the autonomous driving safety model may include an evaluation based on safety evaluation criteria set based on the rules. Here, the safety evaluation criteria may include at least one of the following indicators: the possibility of a collision with a surrounding object, the proportion of blind spots on the road while traveling, and the probability of collision avoidance when a collision avoidance action is taken. Whether the safety evaluation criteria are met may be determined based on a predetermined threshold set for each indicator.

[0100] The risk estimation may include detecting a deviation of the driver's driving from a rule. The deviation may indicate the degree of violation of the rule. For example, if the driver's driving does not violate the rule, the deviation may be set to 0. The detection of the deviation may be included in the evaluation using the rules defined by the automated driving safety model, or may be performed separately after the evaluation. When the deviation is calculated based on a safety evaluation standard, the deviation may be the difference between a threshold and a value that quantifies the evaluation of the violation calculated in the evaluation of the actual driver's driving behavior described above. When the deviation is calculated based on a safety evaluation standard, the deviation may be calculated based on the difference between a safety evaluation value and a threshold. The deviation may be calculated as a composite or comprehensive parameter for multiple rules or safety evaluation standards.

[0101] The risk estimation may also include an evaluation of the time to collision with other road users. The time to collision is an index that indicates how much time remains until a collision occurs between vehicle 1 and other road users if the current relative speed is maintained.

[0102] The estimation of the risk level may include an evaluation of the driver's state, which may include a determination based on the estimation result of whether the driver's state estimated by the driver estimation unit 73 is positive or negative.

[0103] The risk level may be estimated by any one of the above evaluations or judgments, or by a combination of the above evaluations or judgments. The risk level may be classified and estimated into three levels: low risk, medium risk, and high risk. The risk level may be classified and estimated into two levels or multiple levels of four or more. The risk level may be represented by a continuous value from 0 to 100.

[0104] For example, consider a scenario in which the inter-vehicle distance between vehicle 1 and preceding vehicle OV1 is smaller than the safe distance, as shown in Fig. 5. In this scenario, the collision probability is greater than a predetermined threshold value based on the regulations. In this case, the risk estimation unit 75 may estimate that the driver's driving is high risk.

[0105] Also, consider a scenario in which the vehicle 1 is speeding at an obstructed intersection with poor visibility, as shown in Fig. 6. In this scenario, it is estimated that the driver does not anticipate that another road user OV2 (e.g., a safety-related object) will appear from the obstructed area OA. In this case, the risk level estimation unit 75 may estimate that the driver's driving is high risk.

[0106] Also, for example, as shown in FIG. 7, consider a scenario in which vehicle 1 is traveling in the left lane L1 of a two-lane road, and another vehicle OV3 traveling ahead of vehicle 1 suddenly drops a load OB1. In this scenario, it is assumed that another vehicle OV4 traveling in the right lane L2 is present to the right of vehicle 1. If vehicle 1 then attempts to change lanes into right lane L2, the scenario becomes a composite scenario that combines the load drop scenario and the cut-in scenario. In this scenario, the collision avoidance probability is smaller than a predetermined threshold. In this case, the risk estimation unit 75 may estimate the driver's driving to be high risk.

[0107] Then, the process of outputting information for presenting instructions to the driver for following the rules, in other words, information necessary for presentation (hereinafter referred to as necessary presentation information) to at least one of the HMI device 70, the mobile terminal 91, and the external system 96 may be realized, for example, by the HMI output unit 71.

[0108] The presentation-required information may be at least one of, for example, an estimation result of an estimation regarding the driver, driving behavior information, and an estimation result of a risk level. As will be described in detail later, when content to be presented to the driver is generated by the sender of the presentation-required information, the presentation-required information may be the content itself to be presented to the driver.

[0109] At least one type of data among the estimation result of the driver's estimation, the driving behavior information, and the estimation result of the risk level may be stored in the recording device 55 of the processing system 50. This data may be stored in the driving information DB 98 of the external system 96 by transmitting and receiving information via the communication system 43. The stored data may be used for determining whether to implement instruction. The stored data may be used for generating presentation content, which will be described later. The stored data may be used for verification after an accident occurs.

[0110] When an evaluation is made that violates the rules, the HMI output unit 71 may output the required presentation information to at least one of the HMI device 70, the mobile terminal 91, and the external system 96. On the other hand, when no violation of the rules is confirmed, the required presentation information does not need to be output, but may be output as reference information or for accumulating statistical data.

[0111] When an evaluation is made that a violation of the rules is made, the HMI output unit 71 may determine the timing of presentation according to at least one of the degree of danger, the degree of deviation, the responsibility value, and the urgency. The timing of presentation may be selected from during driving by the driver and after driving by the driver has ended. Presentation content optimized for both during driving by the driver and after driving by the driver has ended may be presented.

[0112] While the driver is driving, more detailed timings may be selectable, such as immediately, or when a predetermined condition is met while driving (for example, when the vehicle stops at an intersection).After the driver has finished driving, more detailed timings may be selectable, such as during autonomous driving after a takeover from manual driving to autonomous driving at levels 3 to 5, or after arriving at a destination.

[0113] At least one of the processing system 50 (e.g., HMI output unit 71) of the vehicle 1, which is the sender of the information to be presented, and the HMI device 70, mobile terminal 91, and external system 96, which are the receivers of the information to be presented, may have the function of generating content to be presented to the driver.

[0114] The presentation content here may be visual information presentation content that presents visual information such as still image content or video content. The presentation content may be auditory information presentation content that presents auditory information such as audio content. The presentation content may be cutaneous sensory information content that presents cutaneous sensory information. Furthermore, the presentation content may be content that combines visual information and auditory information. The presentation content may be generated according to generation rules based on at least one of the rules of the safety model and the safety evaluation criteria. The content of the presentation content may be determined taking into consideration the driver's driving habits and the results of comparing current driving with usual driving (e.g., past driving).

[0115] The presentation content may be generated by selecting one content from a plurality of pre-prepared contents based on the driver state estimation result, driving behavior information, and risk estimation result as the required presentation information. This selection may be performed under conditions that comply with the above-mentioned generation rules. The selected content may be partially changeable based on the detailed content of the driving behavior information.

[0116] Furthermore, the presentation content may be generated by a trained neural network that has learned the above-mentioned generation rules. Specifically, the driver state estimation result, driving behavior information, and risk estimation result as the information required for presentation are input as input parameters to the neural network, and the presentation content is output from the neural network. At least one of the detection data of the external environment sensor 41, the environmental model, and the vehicle state may further be added to the input parameters.

[0117] 8 shows an example in which a presentation content generation unit 76a is provided as a functional block constructed by a dedicated computer 51 in the processing system 50, and the presentation content is generated by the presentation content generation unit 76a. In this example, the presentation content generation unit 76a generates the presentation content based on the estimation results of the driver's estimation, driving behavior information, and risk estimation results recorded in the recording device 55. The generated content data may be directly transmitted to the HMI device 70 and the mobile terminal 91 that instruct the driver. Alternatively, the generated content data may be transmitted to an external system 96, stored in a driving information DB 98, and then downloaded to the mobile terminal 91, thereby providing the mobile terminal 91 that instructs the driver.

[0118] 9 shows another example in which a presentation content generation unit 76b is provided in a mobile terminal 91 as a functional block realized by a dedicated computer 92. In this example, the driver state estimation result, driving behavior information, and risk estimation result as information required to be presented, as well as a presentation command, are output from the HMI output unit 71 of the processing system 50 to the mobile terminal 91. In response to this, the presentation content generation unit 76b of the mobile terminal 91 generates the presentation content. This configuration may be realized by downloading and installing a program that executes the content generation process by the presentation content generation unit 76b from a network or an external system 96, together with an application that performs the teaching.

[0119] As another example, Fig. 10 shows an example in which a presentation content generation unit 76c is provided in an external system 96 as a functional block realized by a dedicated computer 97. In this example, the driver state estimation result, driving behavior information, and risk estimation result as presentation-required information are output from the HMI output unit 71 of the processing system 50 to the external system 96, and in response to this, the presentation content generation unit 76c of the external system 96 generates presentation content. The presentation-required information including the generated content data may be recorded in a driving information DB 98. Meanwhile, when the mobile terminal 91 receives a presentation command from the HMI output unit 71, it may download content data from the external system 96 and provide instructions to the driver.

[0120] As an example of real-time instruction, an instruction may be given by combining content displayed on the HUD and audio from the speaker (see FIGS. 11 and 12). For example, FIG. 11 illustrates an instruction mode in a case where a pedestrian P1 is about to cross in front of the vehicle 1 from the front right of the vehicle 1 and it is estimated that the driver is not taking the pedestrian P1 into consideration when driving. In this case, the HUD displays a virtual image of an instruction image IM1 informing the driver of the presence of the pedestrian P1 in a portion of the displayable area of ​​the windshield WS of the vehicle 1 that is closest to the pedestrian P1. At the same time, the speaker issues an instruction audio message, such as "Pay attention to the pedestrian ahead on the right," instructing the driver to take the pedestrian P1 into consideration when driving.

[0121] 12 shows an example of a teaching mode when the inter-vehicle distance between the vehicle 1 and the preceding vehicle OV5 is smaller than the safe distance. In this case, the HUD displays a virtual image of a teaching image IM2 using multiple horizontal lines to make the driver aware of the inter-vehicle distance in a portion of the displayable area of ​​the windshield WS of the vehicle 1 that is visible behind the preceding vehicle OV5. At the same time, the speaker issues a teaching voice that instructs the driver to take the inter-vehicle distance into consideration, such as "Please keep a sufficient inter-vehicle distance from the vehicle ahead."

[0122] As an example of providing instruction after driving, instruction may be provided using content that combines video display and audio on the mobile terminal 91, as shown in Fig. 13. The content here can be said to be a combination of visual information that shows a scenario that the vehicle 1 will encounter while driving by the driver, and audio information that gives advice on how to improve driving in that scenario.

[0123] Specifically, the speaker of the mobile terminal 91 emits instructional audio suggesting to the driver how to correct bad driving habits, such as, "We will show you a video of a scene that almost led to an accident. You have a habit of driving too fast in blind spots. In places with poor visibility, drive slowly and be prepared to deal with pedestrians or cyclists suddenly appearing in front of you." At the same time, the display of the mobile terminal 91 displays an instructional video illustrating a scenario that almost led to an accident.

[0124] It is preferable that the visual information presentation content used for the instruction is generated in a manner that takes into consideration the privacy of other road users. For example, when the visual information content is generated using information based on the detection data of the sensor 40, the content may be generated so that the personal information of other road users is difficult to identify. For example, a video in which the faces of pedestrians captured by the camera 41a are blurred may be generated as the content.

[0125] When teaching is performed by the mobile terminal 91, the driver may install an application having a program that realizes a teaching function in advance on the mobile terminal 91, and the teaching may be performed. The teaching may be started by the driver operating the application. The teaching may also be started automatically in response to the timing at which a driver teaching command is received.

[0126] As another example of providing instruction after driving, instruction may be provided in the form of a report using visual information presentation content via a meter, CID, HUD, mobile terminal 91, etc., or audio information presentation content via a speaker.

[0127] Specifically, the driver may be presented with a report such as, "You have a habit of drifting outward when turning, which could result in a collision with a vehicle in the adjacent lane. Slow down before entering the curve and reduce your speed before turning. One factor that contributes to this is that you are driving with one hand and are unable to steer smoothly, so keep both hands on the steering wheel when driving."

[0128] In addition, a report such as, "Today, your following distances tended to be shorter than usual. You may not be able to respond to the sudden deceleration of the vehicle in front, which could result in a collision. Please try to maintain a sufficient following distance when driving." may be presented to the driver.

[0129] As described above, the upper limit of the information amount of the presented content intended to be given to the driver while driving may be set to be smaller than the upper limit of the information amount of the presented content intended to be given to the driver after driving. Furthermore, the upper limit of the playback time of the presented content intended to be given to the driver while driving may be set to be smaller than the upper limit of the playback time of the presented content intended to be given to the driver after driving. In other words, the instruction given to the driver while driving may be shorter than the instruction given to the driver after driving, and may be realized in a manner in which only the main points are notified.

[0130] Furthermore, at least one of the amount of information and the timing of presentation of the presented content is adjusted depending on the result of the risk estimation. For example, when the risk is estimated to be high, the timing of presentation may be set to while the driver is driving, and the amount of information of the presented content may be set to be smaller than when the risk is estimated to be low.

[0131] <Processing flow> Next, an example of a processing method for realizing the evaluation function and the teaching function will be described using the flowchart in Fig. 14. The series of processes shown in steps S11 to S16 is executed by the driving system 2 at predetermined time intervals or based on a predetermined trigger. As a specific example, the series of processes may be executed at predetermined time intervals when the autonomous driving mode is managed at autonomous driving level 0. As another specific example, the series of processes may be executed at predetermined time intervals when the autonomous driving mode is managed at autonomous driving levels 0 to 2.

[0132] As will be described in detail later, part of the series of processes may be executed by at least one of the external system 96 and the mobile terminal 91. The series of processes may be executed according to a computer program stored in a memory.

[0133] First, in S11, the information acquisition unit 72 acquires information necessary to realize the teaching function. After the process of S11, the process proceeds to S12.

[0134] In S12, the driver estimation unit 73 performs estimation regarding the driver using the information acquired in S11. After the process of S12, the process proceeds to S13.

[0135] In S13, the driving behavior information generation unit 74 generates information on the driving behavior of the driver using the information acquired in S11. After the process of S13, the process proceeds to S14. Note that the order of the process of S12 and the process of S13 may be reversed, and for example, the processes may be executed in parallel using two separate processors.

[0136] In S14, the risk estimation unit 75 estimates the risk using the estimation in S12 and the information on the driving behavior in S 13. After the process of S14, the process proceeds to S15.

[0137] In S15, the HMI output unit 71 outputs the presentation-required information to at least one of the HMI device 70, the mobile terminal 91, and the external system 96. The output of the presentation-required information to the mobile terminal 91 or the external system 96 is essentially the transmission of the presentation-required information via the communication system 43. After processing S15, the process proceeds to S16.

[0138] In S16, at least one of the HMI device 70 and the mobile terminal 91 that have already acquired the presentation content generated by the HMI device 70 and the required presentation information, or the HMI device 70 and the mobile terminal 91 that have acquired the required presentation information and generated the presentation content from them, will instruct the driver. S16 ends the series of processes.

[0139] Next, an example of the processing method for estimating the risk level in S14 will be described in detail with reference to the flowchart of FIG.

[0140] In S101, the risk estimation unit 75 determines whether the driver's driving violates the safety envelope based on the driving behavior information. If a positive determination is made in S101, the process proceeds to S102. If a negative determination is made in S101, the process proceeds to S105.

[0141] In S102, the risk estimation unit 75 detects the degree of deviation from the driving rules by the driver and determines whether the degree of deviation is smaller than a predetermined judgment reference value. The judgment reference value may be a fixed value set in advance. Note that if the degree of deviation cannot be expressed as a quantitative value and is difficult to compare with the judgment reference value, a negative judgment may be made. If a positive judgment is made in S102, the process proceeds to S103. If a negative judgment is made in S103, the process proceeds to S107.

[0142] In S103, the risk estimation unit 75 determines whether the margin of time is longer than a predetermined judgment reference value. The judgment reference value may be a fixed value set in advance. If a positive judgment is made in S103, the process proceeds to S104. If a negative judgment is made in S103, the process proceeds to S107. Note that if the content of the judgment in S103 substantially overlaps with the content of the judgment in S101, the process of S103 may be omitted.

[0143] In S104, the risk estimation unit 75 determines whether the driver's state is negative based on the estimation result of the driver estimation unit 73. If a positive determination is made in S104, the process proceeds to S107. If a negative determination is made in S104, the process proceeds to S106.

[0144] In S105, the risk estimation unit 75 estimates that the driving by the driver is low risk. After S105, the series of processes ends.

[0145] In S106, the risk estimation unit 75 estimates that the driving by the driver is medium risk. After S106, the series of processes ends.

[0146] In S107, the risk estimation unit 75 estimates that the driving by the driver is highly risky. After S107, the series of processes ends.

[0147] Next, an example of a processing method for exchanging information and presenting it to the driver in S15 and S16 will be described in detail with reference to the flowchart of FIG.

[0148] In S111, the HMI output unit 71 determines whether the risk of the driver's driving is estimated to be medium or higher, i.e., medium or high. If a positive determination is made in S111, the process proceeds to S112. If a negative determination is made in S111, the process ends.

[0149] In S112, the HMI output unit 71 determines whether the driving by the driver is estimated to be highly dangerous. If a positive determination is made in S112, the process proceeds to S113. If a negative determination is made in S112, the process proceeds to S115.

[0150] In S113, the HMI output unit 71 and the HMI device 70 perform a presentation process while the driver is driving. In this example, if the driver's driving is estimated to be highly dangerous, the HMI output unit 71 selects to provide instruction to the driver while he is driving. Based on the output of the information required for presentation and the presentation command by the HMI output unit 71, the HMI device 70 presents the information to the driver, i.e., provides instruction. After processing S113, the process proceeds to S114.

[0151] In S114, information such as the presentation-required information and presentation history information of the presented content is saved. This information may be stored in the recording device 55 as information for the vehicle 1 alone. This information may also be stored in the driving information DB 98 in the external system 96 in an aggregated form together with information for multiple vehicles. After processing S114, the process proceeds to S116.

[0152] In S115, the required presentation information is saved. This information may be stored in the recording device 55 as information for the vehicle 1 alone. This information may also be stored in the driving information DB 98 in a form where information for multiple vehicles is aggregated together. After processing S115, the process proceeds to S116.

[0153] In S116, the HMI output unit 71 determines whether or not the driver has finished driving. If a positive determination is made in S116, the process proceeds to S117. If a negative determination is made in S116, the process of S116 is performed again, for example, after a predetermined time has elapsed.

[0154] In S117, the HMI output unit 71 and at least one of the HMI device 70 and the mobile terminal 91 perform a presentation process after the driver has finished driving. In this example, if the risk level of the driver's driving is estimated to be medium or higher, the HMI output unit 71 selects to perform instruction after the driver has finished driving. For example, based on the output of the presentation-required information and presentation command by the HMI output unit 71, at least one of the HMI device 70 and the mobile terminal 91 may present, i.e., provide instruction, to the driver. Also, for example, at least one of the HMI device 70 and the mobile terminal 91 may acquire and refer to the information saved in S115 and S116 and present, i.e., provide instruction, to the driver. A series of processes ends with S117.

[0155] In this way, for the same driving behavior of the driver, the presentation process while the driver is driving (see S113) and the presentation process after the driver has finished driving (S117) may be performed overlappingly. In this way, for the same driving behavior of the driver, multiple teachings may be performed by changing at least one of the device that performs the teaching, the amount of information, and the presentation timing. By providing multiple teachings with changing the presentation mode, it is possible to reduce the annoyance felt by the driver and increase the appropriateness of the driving by the driver.

[0156] Next, an example of a processing method for implementing the notification while the driver is driving in S113 will be described in more detail with reference to the flowchart of FIG.

[0157] In S121, if the HMI output unit 71 has presented the same or similar content in the past, it determines whether a predetermined time has passed since the last time the content was presented. The predetermined time may be, for example, one minute, ten minutes, or one hour. If a positive determination is made in S121, or if the same or similar content has not been presented in the past, the process proceeds to S122. If a negative determination is made in S121, the process ends.

[0158] In S122, the HMI device 70 receives a presentation command from the HMI output unit 71 and performs a teaching that combines the HUD and voice, as described with reference to Figures 11 and 12. The series of processes ends with S122.

[0159] That is, when the driver's driving is estimated to be highly dangerous, the instruction during driving may be unconditionally implemented, but the instruction may be omitted under predetermined conditions as in S121 and S122. By preventing the situation where the same or similar content is taught multiple times in a short period of time, the annoyance felt by the driver can be reduced.

[0160] Next, an example of a processing method for implementing the presentation after the driver has finished driving in S117 will be described in more detail with reference to the flowchart of FIG.

[0161] In S131, the processing system 50 (for example, the HMI output unit 71) reads out the information saved in S115 and S116 from the storage location. This reading may be achieved by transmitting and receiving information. After the processing of S131, the process proceeds to S132.

[0162] In S132, the HMI output unit 71 determines whether the driving behavior of the driver in question is a behavior that is repeatedly performed. If a positive determination is made in S132, the process proceeds to S133. If a negative determination is made in S132, the process proceeds to S134.

[0163] In S133, the HMI output unit 71 determines whether the driving behavior of the driver in question is a behavior that is repeatedly performed. If a positive determination is made in S132, the process proceeds to S133. If a negative determination is made in S132, the process proceeds to S134.

[0164] In S133, the HMI output unit 71 determines whether the driving behavior of the driver in question is unsafe compared to the driver's usual driving behavior. If a positive determination is made in S133, the process proceeds to S134. If a negative determination is made in S133, the process ends.

[0165] In S134, at least one of the HMI device 70 and the mobile terminal 91 that received the presentation command from the HMI output unit 71 performs teaching by video or teaching by report as described with reference to Fig. 13. The series of processes ends with S134.

[0166] That is, when the driving by the driver is estimated to be medium or high risk, post-driving instruction may be unconditionally implemented, but instruction may be omitted under predetermined conditions as in S131 to 134. By preventing the situation where the driver is taught something that he or she has already understood, the annoyance felt by the driver can be reduced.

[0167] (Action and effect) The effects of the first embodiment described above will be explained below.

[0168] According to the processing system 50 of the first embodiment, information regarding instructions to the driver that can be presented to the driver is output. The rules that serve as the basis for instructions to the driver are defined by the safety model of automated driving. By having the driver refer to these instructions, it is possible to prevent the driver's driving from being evaluated unfavorably in a relative evaluation of an automated driving vehicle. Therefore, it is possible to increase the validity of the driver's driving.

[0169] According to the HMI device 70 and the mobile terminal 91 of the first embodiment, the user interfaces 70b and 94 present presentation content based on information regarding instructions to the driver acquired from the communication interfaces 70a and 93. The rules that serve as the basis for instructions to the driver are defined by the safety model of automated driving. By having the driver refer to these instructions, it is possible to prevent the driver's driving from being evaluated unfavorably in a relative evaluation of the automated driving vehicle. Therefore, it is possible to increase the appropriateness of the driver's driving.

[0170] According to the first embodiment, since the teaching is performed according to the degree of deviation of the driver's driving from the rules, it is possible to optimize the teaching so that the driver can easily follow the rules. Therefore, it is possible to improve the appropriateness of the driver's driving.

[0171] According to the first embodiment, the presentation mode of the presentation content for implementing the instruction is determined. This determination is based on the result of the evaluation of the driver's driving, so it is possible to optimize the instruction so that the driver can easily follow the rule. Therefore, the appropriateness of the driver's driving can be improved.

[0172] According to the first embodiment, the presentation mode based on the results of the evaluation of the driver's driving includes the concept of information amount, so that it is possible to provide instructions for following rules while reducing the annoyance felt by the driver.

[0173] According to the first embodiment, the presentation mode based on the results of the evaluation of the driver's driving includes the concept of presentation timing, so that instruction to follow rules can be given at a timing that is likely to promote the driver's understanding.

[0174] According to the first embodiment, while the driver is driving, the same or similar presentation content is presented at intervals of at least a predetermined time, thereby making it possible to instruct the driver to follow the rules while reducing the annoyance felt by the driver.

[0175] According to the first embodiment, the presented content is a combination of visual information showing a scenario that the vehicle 1 will encounter while driving by the driver and auditory information giving advice on how to improve driving in the scenario. The scenario presented as visual information helps the driver to quickly understand the situation they are encountering. At the same time, the advice given as auditory information can increase the persuasiveness of the instruction. Therefore, it is possible to provide an instruction that makes it easier for the driver to follow the rules.

[0176] According to the first embodiment, when the user interfaces 70b and 94 present presentation content, they use information read from the external system 96. This prevents the HMI device 70 or the mobile terminal 91 from continuing to hold information from the time the driver starts driving until teaching is given, so teaching can be given while saving hardware resources installed in the HMI device 70 or the mobile terminal 91.

[0177] (Second embodiment) 19 and 20, the second embodiment is a modification of the first embodiment. The second embodiment will be described, focusing on the differences from the first embodiment.

[0178] In the second embodiment, the risk estimation unit 75 predicts scenarios that the vehicle 1 may encounter before arriving at the destination, and estimates the risk based on the scenarios. The risk estimation unit 75 may predict scenes instead of scenarios. Specifically, the risk estimation unit 75 predicts the route that the driver will take while driving the vehicle 1, based on road information and destination information acquired by the map DB 44 and V2X. Furthermore, the risk estimation unit 75 predicts a scenario in which the vehicle 1 will fall into an unsafe state, based on road information related to the predicted route.

[0179] The scenario resulting in an unsafe state may refer to a so-called hazardous situation or a scenario that is likely to result in a hazardous situation. The scenario resulting in an unsafe state may refer to a scenario that is likely to cause the driver to deviate from the rules defined by the safety model. The scenario that the risk estimation unit 75 can predict corresponds to a known dangerous scenario.

[0180] The risk estimation unit 75 may extract scenarios in which the vehicle 1 will fall into an unsafe state by determining the similarity between the scenario predicted to be encountered by the vehicle 1 and dangerous scenarios among the concrete scenarios stored in the scenario DB 53.

[0181] The prediction of unsafe situations in a scenario may be performed under assumptions about the reasonably foreseeable behavior of other road users. This assumption may be based on consideration of the rules specified by the safety model. For example, if the information about the other vehicle predicted in the scenario indicates that the other vehicle is a vehicle equipped with an RSS model, the behavior of the other vehicle may be assumed based on the rules of the RSS model.

[0182] The scenario here may include the mental state of the driver (for example, at least one of the driver's intention and emotion) as a factor for determining an unsafe state. For example, as shown in Fig. 19, if it is predicted that the vehicle 1 will enter a traffic jam in five minutes, an irritated state may be predicted as the mental state that the driver is likely to fall into. Of the scenarios that the vehicle 1 is likely to encounter after entering the traffic jam, a scenario in which a correlation between the irritated state and an unsafe state is recognized may be extracted as a scenario in which the vehicle 1 will fall into an unsafe state.

[0183] Furthermore, for example, if it is predicted that the vehicle 1 will be traveling on an unfamiliar road in 10 minutes, a state of tension may be predicted as the mental state that the driver is likely to fall into. Of the scenarios that the vehicle 1 is likely to encounter while traveling on an unfamiliar road, a scenario in which a correlation between a state of tension and an unsafe state is recognized may be extracted as a scenario in which the vehicle 1 will fall into an unsafe state.

[0184] An example of a processing method for estimating the risk level in the second embodiment will be described in detail with reference to the flowchart of FIG.

[0185] In S201, the risk estimation unit 75 predicts a scenario in which the vehicle 1 falls into an unsafe state. After the processing of S201, the process proceeds to S202.

[0186] In S202, the risk estimation unit 75 determines whether a scenario resulting in an unsafe state is predicted. If a positive determination is made in S202, the process proceeds to S204. If a negative determination is made in S202, the process proceeds to S203.

[0187] In S203, the risk estimation unit 75 estimates that the driving by the driver is low risk. After S203, the series of processes ends.

[0188] In S204, the risk estimation unit 75 estimates that the driving by the driver is high risk. A series of processes are estimated through S204.

[0189] In this flow, the risk level is classified into two levels, but the risk level may be classified into three or more levels or into a continuous number depending on the predicted scenario. Based on the estimated risk level, instructions regarding route changes, instructions regarding the driver's mental state, etc. may be given.

[0190] According to the second embodiment described above, the scenarios that are the subject of instruction to enable the vehicle 1 to comply with the rules are scenarios that are predicted to be encountered by the vehicle 1 when driven by the driver, and are scenarios that are predicted to result in an unsafe state for the vehicle 1. By the driver referring to this instruction, it becomes possible to prepare in advance to avoid falling into an unsafe state when the instructed scenario is encountered, and therefore the effect of suppressing the driver's driving from receiving an unfavorable evaluation is dramatically increased.

[0191] According to the second embodiment, when it is predicted that the driver will deviate from the driving rules, the presented content is presented at a timing earlier than the predicted timing. By referring to this instruction, the driver can prepare in advance to avoid the driver's driving deviating from the rules, thereby dramatically increasing the effect of suppressing the driver's driving from receiving an unfavorable evaluation.

[0192] (Third embodiment) 21 and 22, the third embodiment is a modification of the first embodiment. The third embodiment will be described, focusing on the differences from the first embodiment.

[0193] In the third embodiment, the risk estimation unit 75 estimates a causal relationship between the driver's state and driving behavior, and estimates the risk based on the causal relationship. Specifically, the risk estimation unit 75 refers to the value of each parameter in the driving behavior information. Based on the value of each parameter, the risk estimation unit 75 estimates a causal relationship between the driver's driving behavior and the cause that led the driver to perform the target driving behavior. The target driving behavior may be a risky driving behavior (hereinafter referred to as a risky behavior).

[0194] 21, for example, assume that data is obtained showing that the average inter-vehicle distance d between vehicle 1 and other vehicles ahead on a road with a speed limit of 60 km / h in the driving behavior of the driver of vehicle 1 is 45 m when the driver's mental state is normal, but 30 m when the driver is in an irritated state. In this case, the risk level estimation unit 75 estimates a causal relationship between the driver state of "irritated state" and the driving behavior of "closing the inter-vehicle distance" for this driver.

[0195] For example, suppose data is obtained showing that the reaction time t of the driver of vehicle 1 to the behavior of other road users, such as obstacles, during driving behavior is 0.1 seconds under normal conditions, but 0.8 seconds when the driver is drowsy. In this case, the risk estimation unit 75 estimates a causal relationship between the driver state of "drowsy state" and the driving behavior of "delayed avoidance behavior" for this driver.

[0196] For example, suppose data is obtained showing that the number of pedestrians recognized by the driver of vehicle 1 during driving behavior is four under normal conditions, but two under stress. In this case, the risk estimation unit 75 estimates a causal relationship between the driver state of "stress" and the driving behavior of "increased oversight of pedestrians."

[0197] When the current driver state is a state that causes the risky behavior identified in the estimation of the causal relationship, the risk level estimation unit 75 may estimate the risk level to be higher than when the current driver state is not a cause of the risky behavior.

[0198] An example of a processing method for estimating the risk level in the third embodiment will be described in detail with reference to the flowchart of FIG.

[0199] In S300, the risk estimation unit 75 estimates the causal relationship between the driver's state and the driving behavior of the driver. After the process of S300, the process proceeds to S301.

[0200] In S301, the risk estimation unit 75 determines whether the driver's driving violates the safety envelope based on the driving behavior information. If a positive determination is made in S301, the process proceeds to S302. If a negative determination is made in S301, the process proceeds to S305.

[0201] In S302, the risk estimation unit 75 detects the degree of deviation of the driver's driving from the rules and determines whether the degree of deviation is smaller than a predetermined judgment reference value. Note that if the degree of deviation cannot be expressed as a quantitative value and is difficult to compare with the judgment reference value, a negative judgment may be made. If a positive judgment is made in S302, the process proceeds to S303. If a negative judgment is made in S303, the process proceeds to S307.

[0202] In S303, the risk estimation unit 75 determines whether the margin time is longer than a predetermined reference value. If a positive determination is made in S303, the process proceeds to S304. If a negative determination is made in S303, the process proceeds to S307. Note that if the determination made in S303 substantially overlaps with the determination made in S301, the process of S303 may be omitted.

[0203] In S304, the risk estimation unit 75 determines whether the driver's current state is a state that could cause risky behavior, based on the estimation of the causal relationship in S300. If a positive determination is made in S304, the process proceeds to S307. If a negative determination is made in S304, the process proceeds to S306.

[0204] In S305, the risk estimation unit 75 estimates that the driving by the driver is low risk. After S305, the series of processes ends.

[0205] In S306, the risk estimation unit 75 estimates that the driving by the driver is medium risk. After S306, the series of processes ends.

[0206] In S307, the risk estimation unit 75 estimates that the driving by the driver is high risk. After S307, the series of processes ends.

[0207] The causal relationship between the driver's state and the driver's driving behavior used to estimate the risk level may not be a causal relationship specific to a particular driver who drives vehicle 1, but may be a causal relationship recognized by ordinary drivers.

[0208] According to the third embodiment described above, factors that cause potential danger are classified according to the causal relationship between the driver's state and the potential danger in driving by the driver. Since the instruction is based on the classification of the factors, it is possible to improve the persuasiveness of the instruction.

[0209] (Fourth embodiment) 23 and 24, the fourth embodiment is a modification of the first embodiment. The fourth embodiment will be described, focusing on the differences from the first embodiment.

[0210] The teaching function in the fourth embodiment is specialized for teaching the driver while he is driving. If the estimation result by the risk level estimation unit 75 is a high risk, teaching the driver while he is driving is implemented in the same manner as in the first embodiment. If the estimation result by the risk level estimation unit 75 is a medium risk, it is determined whether or not to implement teaching the driver while he is driving, depending on a comparison between driving by the current driver and driving by the driver in the past (hereinafter referred to as past driving).

[0211] An example of a processing method for presenting information to the driver will now be described in detail with reference to the flowchart of FIG.

[0212] In S411, the HMI output unit 71 determines whether the risk of the driver's driving is estimated to be medium or higher, i.e., medium or high. If a positive determination is made in S411, the process proceeds to S412. If a negative determination is made in S411, the process ends.

[0213] In S412, the HMI output unit 71 determines whether the driving by the driver is estimated to be highly dangerous. If a positive determination is made in S412, the process proceeds to S413. If a negative determination is made in S412, the process proceeds to S414.

[0214] In S413, the HMI output unit 71 and the HMI device 70 perform a presentation process while the driver is driving. The presentation process may be similar to S121 and S122 shown in Fig. 17. After the process of S413, the process proceeds to S414.

[0215] In S414, the required presentation information is saved. This information may be stored in the recording device 55 as information for the vehicle 1 alone. This information may also be stored in the driving information DB 98 in the external system 96 in a form where information for multiple vehicles is aggregated. The series of processes ends with S414.

[0216] In S415, the HMI output unit 71 and the HMI device 70 perform a presentation process based on the comparison result of the past driving. After S415, the series of processes ends.

[0217] Next, an example of a processing method for presenting the results of comparison with past travel in S415 will be described in more detail with reference to the flowchart of FIG.

[0218] In S421, the processing system 50 (for example, the HMI output unit 71) reads out from the storage location the past driving behavior information among the required presentation information stored in S414. The past driving behavior information includes information about past travels. This reading may be achieved by transmitting and receiving information. After processing S421, the process proceeds to S422.

[0219] In S422, the processing system 50 (e.g., the HMI output unit 71) compares the current driving by the driver with the information about past driving acquired in S421. The processing system 50 (e.g., the HMI output unit 71) determines whether the current driving is likely to lead to dangerous driving in the future, compared with normal (i.e., past) driving. If a positive determination is made in S422, the process proceeds to S423. If a negative determination is made in S422, the process proceeds to S424.

[0220] In S423, the HMI output unit 71 and the HMI device 70 perform a presentation process while the driver is driving. The presentation process may be similar to S121 and S122 shown in Fig. 17. After the process of S423, the process proceeds to S424.

[0221] In S424, the required presentation information is saved, as in S414. The series of processes ends with S424.

[0222] According to the fourth embodiment described above, the presentation mode of the presented content is determined based on a comparison between the current driving behavior and the past driving behavior of the driver, and therefore, it is possible to provide appropriate instruction according to the driver's state, changes in driving ability over time, etc.

[0223] (Other embodiments) Although multiple embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0224] In a first modification, a processing system that executes the processes of the risk estimation unit 75 and the HMI output unit 71 among the evaluation function and teaching function may be a separate system separate from the driving system 2. This processing system may or may not be mounted on the vehicle 1. This processing system may be provided in the HMI device 70 or the mobile terminal 91, or may be provided as an external system 96 such as a remote center.

[0225] As a second modification, the processing system that executes the processes of the risk estimation unit 75 and the HMI output unit 71 among the evaluation function and teaching function may be applied to a manually driven vehicle that cannot perform autonomous driving.

[0226] As a third modification, the processing system that executes the processes of the risk estimation unit 75 and the HMI output unit 71 among the evaluation function and the teaching function may be applied to a vehicle that does not have a V2X function. In this case, teaching may be performed solely by the on-board HMI device 70.

[0227] The controller and methods described herein may be implemented by a special-purpose computer comprising a processor programmed to perform one or more functions embodied in a computer program. Alternatively, the apparatus and methods described herein may be implemented by special-purpose hardware logic circuitry. Alternatively, the apparatus and methods described herein may be implemented by one or more special-purpose computers comprising a processor executing a computer program in combination with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium.

[0228] (Terminology explanation) The following describes terms related to the present disclosure, which are included in the embodiments of the present disclosure.

[0229] A road user may be a person using a road, including sidewalks and other adjacent spaces. Road users may include pedestrians, cyclists, other VRUs, and vehicles (e.g., human-driven cars, vehicles equipped with automated driving systems). A road user may be a road user on or adjacent to an active road for the purpose of traveling from one place to another.

[0230] A dynamic driving task (DDT) may be a real-time operational and tactical function for operating a vehicle in traffic.

[0231] An automated driving system may be a collection of hardware and software capable of executing the entire DDT on a continuous basis, whether or not it is limited to a specific operational design domain.

[0232] SOTIF (safety of the intended functionality) may be the absence of undue risk due to insufficient functionality of the intended functionality or its implementation.

[0233] A driving policy may be a strategy and rules that define control behavior at the vehicle level.

[0234] A scenario may be a depiction of the temporal relationships between several scenes in a sequence of scenes, including the goals and values ​​in a particular situation influenced by actions and events. A scenario may be a depiction of a continuous time series of activities that integrates a subject vehicle, all of its external environments, and their interactions in the process of performing a particular driving task.

[0235] A triggering condition may be a specific condition of a scenario that acts as a catalyst for subsequent system responses that contribute to unsafe behavior, failure to prevent, detect, and mitigate reasonably foreseeable indirect misuse.

[0236] Takeover may be the transfer of the driving task between the automated driving system and the driver.

[0237] Safety-related models may be representations of safety-related aspects of driving behavior based on assumptions about the reasonably foreseeable behavior of other road users. Safety-related models may be on-board or off-board safety verification or analysis devices, mathematical models, more conceptual sets of rules, sets of scenario-based behaviors, or a combination of these.

[0238] A formal model may be a model expressed in a formal notation used for system performance verification.

[0239] A safety envelope may be a set of limits and conditions within which an (automated) driving system is designed to operate, subject to constraints or controls, in order to maintain operation within an acceptable level of risk. A safety envelope may be a general concept that can be used to accommodate all principles to which a driving policy can adhere, according to which an ego-vehicle operated by an (automated) driving system may have one or more boundaries around it.

[0240] Response time may be the time it takes a road user in a given scenario to perceive a particular stimulus and begin to execute a response (braking, steering, accelerating, stopping, etc.).

[0241] A hazardous situation may be an increased risk for a potential breach of the safety envelope and may represent an increased level of risk present in the DDT.

[0242] (Disclosure of technical ideas) This specification discloses multiple technical ideas described in the following multiple clauses. Some clauses may be written in a multiple dependent form, where the subsequent clause alternatively refers to the preceding clause. These multiple dependent clauses define multiple technical ideas.

[0243] <Technical philosophy 1> A processing system including at least one processor (51b) for executing a process for presenting a vehicle (1) to a driver, The processor: Evaluating the driving by the driver using rules defined by a safety model for automated driving; and outputting, based on the evaluation, information regarding instructions for following the rules so as to be presentable to the driver.

[0244] <Technical philosophy 2> The processor further detects a deviation of the driving by the driver from the rule; The processing system according to Technical Idea 1, wherein the output is performed according to the magnitude of the deviation.

[0245] <Technical philosophy 3> The processor: Recognizing the driver's state; Extracting a causal relationship between the driver's state and a potential danger in driving by the driver; Classifying potential risk factors according to the causal relationships; and The processing system according to Technical Idea 1 or 2, wherein in outputting the instruction, the instruction is output according to the classification of the occurrence factor.

[0246] <Technical philosophy 4> The processor further executes predicting a scenario that the moving object is predicted to encounter due to driving by the driver, in which the moving object falls into an unsafe state; A processing system described in any one of technical ideas 1 to 3, wherein the instruction is an instruction for the mobile body to follow the rule in a scenario in which the mobile body falls into an unsafe state.

[0247] <Technical philosophy 5> The processing system described in any one of technical ideas 1 to 4 further includes determining the presentation manner of the presentation content for implementing the instruction based on the results of the evaluation of the driver's driving.

[0248] <Technical philosophy 6> The processing system according to Technical Idea 5, wherein the presentation format of the presented content includes an amount of information of the presented content.

[0249] <Technical philosophy 7> The processing system according to Technical Idea 5 or 6, wherein the presentation mode of the presented content includes a presentation timing of the presented content.

[0250] <Technical philosophy 8> The processing system according to Technical Idea 7, wherein when the presentation timing occurs while the driver is driving, the same or similar presentation content is presented at a time interval of at least a predetermined time.

[0251] <Technical philosophy 9> The processing system described in Technical Idea 7 or 8, wherein the presented content is presented at the presentation timing before the predicted occurrence timing when it is predicted that the driver's driving will deviate from the rules.

[0252] <Technical Thought 10> The processing system according to any one of technical ideas 5 to 9, wherein the presentation mode of the presentation content is determined based on a comparison between current driving and past driving by the driver.

[0253] <Technical Thought 11> The processing system according to any one of Technical Ideas 1 to 10, wherein the outputting includes outputting the information when an evaluation that violates the rule is made.

[0254] <Technical Thought 12> An information presentation device that presents information to a user, a communication interface (70a, 93) configured to be able to communicate with a processing system (50) that executes processing related to the moving body (1), and configured to be able to acquire, from the processing system, information regarding instructions for the driver of the moving body to follow rules defined by the safety model of automated driving; and a user interface (70b, 94) configured to be able to present presentation content relating to instructions for following the rules based on the information.

[0255] <Technical Thought 13> The information presentation device described in Technical Idea 12, wherein the presented content includes content that combines visual information indicating a scenario that the moving body will encounter while driving by the driver and auditory information that provides advice on improving driving in the scenario.

[0256] <Technical Thought 14> the communication interface is configured to be able to communicate with an external system (96) provided outside the mobile body; The information presentation device according to Technical Idea 12 or 13, wherein the user interface is configured to be able to present the presentation content using information read from the external system.

[0257] <Technical Thought 15> A recording device for recording information about a driver of a moving object (1), At least one storage medium (55a) driving behavior by the driver; A recording device that records the driving behavior in association with the results of a comparison between the driving behavior and rules defined by the automated driving safety model or standards based on the rules.

[0258] <Technical Thought 16> The recording device according to Technical Idea 14 further associates and records the estimated result of the driver state of the moving body in the storage medium.

[0259] <Technical Thought 17> A processing method for performing a process for presenting information to a driver of a moving object (1), comprising: At least one processor (51b) Evaluating the driving by the driver using rules defined by a safety model for automated driving; and if an evaluation is made that violates the rule, outputting information regarding instructions for complying with the rule so that the information can be presented to the driver.

[0260] <Technical Thought 18> A storage medium configured to be readable by at least one processor (51b), the processor, Evaluating the driving of a driver of a moving object using rules defined by a safety model for automated driving; and outputting, when an evaluation that violates the rule is made, information regarding instructions for complying with the rule so that the information can be presented to the driver.

[0261] <Technical Thought 19> At least one processor (51b) Evaluating the driving of a driver of a moving object using rules defined by a safety model for automated driving; and when an evaluation that violates the rule is made, outputting information regarding instructions for complying with the rule so that the information can be presented to the driver.

[0262] <Technical Thought 20> An information presentation method for presenting driving instructions to a driver, comprising: acquiring information used to evaluate the driving of the driver from at least one of an external environment or an internal environment of the vehicle using a sensor; Calculating, by at least one processor, a deviation of the driving by the driver from rules defined by at least one safety model of an automated driving RSS (Responsibility-Sensitive Safety) model or an SFF (Safety Force Field) model, which rules are stored in at least one recording medium, based on the acquired information; determining whether the calculated deviation exceeds a predetermined threshold; When it is determined that the deviation degree exceeds the threshold value, a signal for causing an information presentation device to present an instruction for the driver to follow the rule is outputted to the information presentation device; an information presentation method in which, upon receiving the signal, the information presentation device presents the instruction to the driver;

[0263] <Technical Thought 21> An information presentation system that presents driving instructions to a driver, a sensor (40) provided in the vehicle (1) for acquiring information from at least one of an external environment or an internal environment of the vehicle for use in evaluating the driving of the driver; an on-board processing system (50) having at least one processor (51a) and at least one recording medium (51b); an information presentation device (70) provided in the vehicle for presenting the instruction to the driver; The at least one recording medium stores rules defined by at least one safety model of an autonomous driving RSS (Responsibility-Sensitive Safety) model or an SFF (Safety Force Field) model; The at least one processor: calculating a deviation of the driving by the driver from the rules based on the information acquired by the sensor; determining whether the calculated deviation exceeds a predetermined threshold; When it is determined that the deviation degree exceeds the threshold value, a signal for instructing the information presentation device to present an instruction for the driver to follow the rule is output to the information presentation device; The information presentation system is configured such that the information presentation device receives the signal and presents the instruction to the driver.

Claims

1. A processing system including at least one processor (51b) for executing a process for presenting a vehicle (1) to a driver, The processor: Evaluating the driving by the driver using rules defined by a safety model for automated driving; In the evaluation or separately from the evaluation, detecting a deviation of the driving by the driver from the rule; and outputting, based on the evaluation, information relating to instructions for following the rules so as to be presentable to the driver; In the outputting, the processing system outputs the information according to the magnitude of the deviation.

2. The processor: Recognizing the driver's state; Extracting a causal relationship between the driver's state and a potential danger in driving by the driver; Classifying potential risk factors according to the causal relationships; and The processing system according to claim 1 , wherein the instruction is based on a classification of the occurrence factor.

3. A processing system including at least one processor (51b) for executing a process for presenting a vehicle (1) to a driver, The processor: Evaluating the driving by the driver using rules defined by a safety model for automated driving; outputting, based on the evaluation, information regarding instructions for complying with the rule so as to be presentable to the driver; and Recognizing the driver's state; Extracting a causal relationship between the driver's state and a potential danger in driving by the driver; Classifying potential risk factors according to the causal relationships; The instruction is the instruction according to a classification of the occurrence factor.

4. The processor further executes predicting a scenario that the moving object is predicted to encounter due to driving by the driver, in which the moving object falls into an unsafe state; The processing system according to claim 1 , wherein the instruction is an instruction for the mobile body to follow the rule in a scenario in which the mobile body falls into an unsafe state.

5. A processing system including at least one processor (51b) for executing a process for presenting a vehicle (1) to a driver, The processor: Evaluating the driving by the driver using rules defined by a safety model for automated driving; outputting, based on the evaluation, information regarding instructions for complying with the rule so as to be presentable to the driver; and predicting a scenario that the moving body is expected to encounter due to driving by the driver, in which the moving body will fall into an unsafe state; The instruction is an instruction for the mobile body to follow the rule in a scenario in which the mobile body falls into an unsafe state.

6. The processing system according to claim 1 , further comprising: determining a presentation manner of presentation content for implementing the instruction based on a result of evaluation of the driving by the driver.

7. A processing system including at least one processor (51b) for executing a process for presenting a vehicle (1) to a driver, The processor: Evaluating the driving by the driver using rules defined by a safety model for automated driving; outputting, based on the evaluation, information regarding instructions for complying with the rule so as to be presentable to the driver; and and determining a presentation manner of presentation content for implementing the instruction based on a result of the evaluation of the driving by the driver.

8. The processing system according to claim 6 , wherein the presentation format of the presented content includes an amount of information of the presented content.

9. The processing system according to claim 6 , wherein the presentation mode of the presented content includes a presentation timing of the presented content.

10. The processing system according to claim 9 , wherein when the presentation timing is while the driver is driving, the same or similar presentation content is presented at a time interval of at least a predetermined time.

11. The processing system according to claim 9 , wherein, when it is predicted that the driver will deviate from the rules in his driving, the presented content is presented at the presentation timing prior to the predicted timing of the deviation.

12. The processing system according to claim 6 or 7, wherein the presentation manner of the presentation content is determined based on a comparison between current driving and past driving by the driver.

13. The processing system according to claim 1 , wherein the outputting includes outputting the information when an evaluation that violates the rule is made.

14. An information presentation device that presents information to a user, a communication interface (70a, 93) configured to be able to communicate with a processing system (50) that executes processing related to the moving body (1), and configured to be able to acquire, from the processing system, information regarding instructions for the driver of the moving body to follow rules defined by the safety model of autonomous driving; a user interface (70b, 94) configured to be able to present presentation content relating to instructions for following the rules based on the information; The information presentation device includes content that combines visual information indicating a scenario that the moving body will encounter while driving by the driver and auditory information that provides advice on improving driving in the scenario.

15. The communication interface is configured to be able to communicate with an external system (96) provided outside the mobile object; The information presentation device according to claim 14 , wherein the user interface is configured to be able to present the presentation content using information read from the external system.

Images