Processing system and information presentation device
By using autonomous driving safety models in the driver's driving system for driving evaluation and teaching, the problem of insufficient driving proper driving in the autonomous driving environment is solved, and the driver's awareness of following rules and driving safety is improved. Effect.
Patent Information
- Application Number
- CN202380033319.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-23
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-26
AI Technical Summary
In the context of the development of autonomous driving technology, drivers may be subject to adverse evaluation when driving in a hybrid environment, and the prior art is difficult to effectively improve the driver's driving appropriateness.
A processing system and information presentation device are designed to evaluate the driver's driving using the rules specified in the safety model of autonomous driving, and present teaching information to the driver through the user interface to improve the driver's awareness of following the rules and driving. Properity.
Through the teaching of the autonomous driving safety model, it can suppress the driver's driving being subject to adverse evaluation, improve the driver's driving properness, ensure that the driver follows safety rules, and reduce the risk of accidents.
Smart Images

Figure CN119032035A_ABST
Abstract
Description
[0001] Cross-references of related applications
[0002] This application is based on patent application No. 2022-083974 filed in Japan on May 23, 2022, and the contents of the basic application are cited by reference in their entirety. Technical Field
[0003] The disclosure of this specification relates to a technology for evaluating or teaching driving of a mobile object. Background Art
[0004] The technology disclosed in Patent Document 1 evaluates the driving characteristics of the driver. Specifically, the evaluation of the driving characteristics includes evaluating compliance with traffic regulations based on the speed, position, and map information of the vehicle driven by the driver, and evaluating the speed according to the position.
[0005] Prior art literature
[0006] Patent Literature
[0007] Patent Document 1: International Publication No. 2019 / 150425 Summary of the invention
[0008] In recent years, the development of autonomous driving technology in mobile objects has been progressing rapidly, and it is intended to realize the driving of mobile objects that perform autonomous driving on roads. In this case, it is assumed that there is an environment where mobile objects driven by a driver and mobile objects that perform autonomous driving according to the rules specified by the safety model of autonomous driving coexist on the road. In this environment, if the driver does not drive in consideration of the rules, the driver's driving may be evaluated relatively unfavorably.
[0009] One of the objects disclosed in this specification is to provide a processing system and an information presentation device for improving the appropriateness of a driver's driving.
[0010] The processing system disclosed herein is a processing system including at least one processor, and executing processing for presenting information to a driver of a mobile object.
[0011] The processor performs the following processing:
[0012] evaluating the driver's driving using rules specified by the safety model of the automated driving; and
[0013] Based on the evaluation, information related to the teaching for compliance with the rules is output in a form that can be presented to the driver.
[0014] According to such a method, information related to the teaching to the driver that can be presented to the driver is output. The safety model of the autonomous driving specifies the rules that serve as the basis for the teaching to the driver. By having the driver refer to the teaching, it is possible to suppress the driver's driving from being unfavorably evaluated in the relative evaluation of the autonomously driven moving body. Therefore, the appropriateness of the driver's driving can be improved.
[0015] The information presentation device disclosed herein is an information presentation device that presents information to a user and comprises:
[0016] a communication interface configured to communicate with a processing system that executes processing related to the mobile body and to obtain from the processing system information related to teaching for causing a driver of the mobile body to follow rules defined by a safety model for autonomous driving; and
[0017] The user interface is configured to be able to present presentation content related to teaching for compliance with the rule based on the information.
[0018] According to such a method, the user interface presents the presentation content based on the information related to the teaching to the driver acquired from the communication interface. The rule serving as the basis for the teaching to the driver is defined by the safety model of the autonomous driving. By having the driver refer to the teaching, it is possible to suppress the driver's driving from being unfavorably evaluated in the relative evaluation of the autonomously driven moving body. Therefore, the appropriateness of the driver's driving can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 This is a block diagram showing a schematic configuration of a driving system.
[0020] Figure 2 This is a block diagram showing the technical structure of the driving system.
[0021] Figure 3 It is a block diagram showing the functional structure of the driving system.
[0022] Figure 4 This is a block diagram showing a structure for realizing an evaluation function and a teaching function.
[0023] Figure 5 This is a diagram showing scenes related to driving evaluation.
[0024] Figure 6 This is a diagram showing scenes related to driving evaluation.
[0025] Figure 7 This is a diagram showing scenes related to driving evaluation.
[0026] Figure 8 It is a block diagram showing the structure that enables the generation and presentation of content.
[0027] Fig. 9 It is a block diagram showing the structure that enables the generation and presentation of content.
[0028] Fig.10 It is a block diagram showing the structure that enables the generation and presentation of content.
[0029] Fig.11 is a diagram illustrating the contents of the presentation.
[0030] Fig.12 is a diagram illustrating the contents of the presentation.
[0031] Fig.13 is a diagram illustrating the contents of the presentation.
[0032] Fig.14 This is a flowchart for explaining the evaluation process and the teaching process.
[0033] Fig.15 This is a flowchart illustrating the process of estimating the risk level.
[0034] Fig.16 is a flowchart illustrating the presentation process to the driver.
[0035] Fig.17 is a flowchart illustrating the presentation process during driving of the driver.
[0036] Fig.18 This is a flowchart illustrating the presentation process after the driver finishes driving.
[0037] Fig.19 It is a diagram for explaining the prediction of the driver's state.
[0038] Fig. 20 This is a flowchart illustrating the process of estimating the risk level.
[0039] Fig.21 is a graph illustrating the estimation of causal relationships.
[0040] Fig. 22 This is a flowchart illustrating the process of estimating the risk level.
[0041] Fig.23 is a flowchart illustrating the presentation process to the driver.
[0042] Fig.24 is a flowchart illustrating the presentation process to the driver. DETAILED DESCRIPTION
[0043] Hereinafter, a plurality of embodiments are described based on the figures. In addition, in each embodiment, repeated descriptions are sometimes omitted by attaching the same symbols to corresponding structural elements. In the case where only a part of the structure is described in each embodiment, the structure of other embodiments described first can be applied to the other parts of the structure. In addition, not only the combination of the structures explicitly described in the description of each embodiment, but also the structures of a plurality of embodiments can be partially combined with each other even if not explicitly described, as long as no particular obstacles are caused in the combination.
[0044] In the following multiple embodiments, the contents of "CSafety First for Automated Driving," Tech. Rep., 2019. by Aptiv, Audi, Baidu, BMW, Continental, Daimler, FCA, here, Infineon, Intel, and Volkswagen, "On a formal model of safe and scalable self-driving cars," arXiv: 1708.06374, 2017. by S. Shalev-Shwartz, S. Shammah, and A. Shashua, and "The Safety Force Field" Technical report, 2019. by David Nister, Hon-Leung Lee, Julia Ng, and Yizhou Wang are cited as a whole by reference.
[0045] (First Embodiment)
[0046] Figure 1 The driving system 2 of the first embodiment shown implements functions related to driving of a mobile body. A part or all of the driving system 2 is mounted on the mobile body. The mobile body that is processed by the driving system 2 is a vehicle 1. The vehicle 1 can be called the main vehicle, which is equivalent to the main mobile body. The vehicle 1 can also be configured to be able to communicate with other vehicles directly or indirectly via a communication infrastructure. The other vehicles are equivalent to the target mobile body.
[0047] Vehicle 1 may also be a road user such as a car or a truck that can perform manual driving. Vehicle 1 may also be capable of automatic driving. Driving is graded according to the extent performed by the driver in all dynamic driving tasks (DDT). For example, the automatic driving levels are specified in SAE J3016. In the case of levels 0 to 2, the driver performs part or all of the DDT. Levels 0 to 2 can also be classified as so-called manual driving. Level 0 means that driving is not automated. Level 1 means that the driving system 2 assists the driver. Level 2 means that driving is partially automated.
[0048] In the case of level 3 or above, the entire DDT is performed by the driving system 2 during the running-in period. Levels 3 to 5 can also be classified as so-called automated driving. A system that can perform driving at level 3 or above can be called an automated driving system. Level 3 means that driving is conditionally automated. Level 4 means that driving is highly automated. Level 5 means that driving is fully automated.
[0049] In addition, the driving system 2 that cannot perform driving at level 3 or higher but can perform driving at least one of level 1 and level 2 can be referred to as a driving assistance system. Hereinafter, unless the maximum achievable autonomous driving level is particularly specified, the autonomous driving system or driving assistance system is simply referred to as the driving system 2 for continued description.
[0050] <Perception-Planning-Action Model>
[0051] The architecture of the driving system 2 is selected so as to be able to implement an efficient SOTIF (safety of the intended functionality) process. For example, the architecture of the driving system 2 can also be constructed based on a sense-plan-act model. The sense-plan-act model has a sense (recognition) element, a planning (planning) element, and an action (action) element as main system elements. The sense element, the planning element, and the action element interact with each other. Here, sense can be replaced by recognition, planning can be replaced by judgment, and action can be replaced by control.
[0052] like Figure 1As shown, in such a driving system 2, at the vehicle level, a vehicle level function 3 is installed based on a vehicle level safety strategy (Vehicle Level Safety Strategy: VSLL). At the functional level (in other words, from a functional point of view), an identification function, a judgment function, and a control function are installed. At the technical level (in other words, from a technical point of view), at least a plurality of sensors 40 corresponding to the identification function, at least one processing system 50 corresponding to the judgment function, and a plurality of motion actuators 60 corresponding to the control function are installed.
[0053] Specifically, the recognition unit 10 as a functional block implementing the recognition function can be constructed in the driving system 2 with multiple sensors 40, a processing system that processes the detection information of the multiple sensors 40, and a processing system that generates an environment model based on the information of the multiple sensors 40 as the main body. The judgment unit 20 as a functional block implementing the judgment function can be constructed in the driving system 2 with the processing system 50 as the main body. The control unit 30 as a functional block implementing the control function can be constructed in the driving system 2 with multiple motion actuators 60 and at least one processing system that outputs the motion signals of the multiple motion actuators 60 as the main body.
[0054] Here, the recognition unit 10 may be implemented as a recognition system 10a which is a subsystem provided distinguishably from the judgment unit 20 and the control unit 30. The judgment unit 20 may be implemented as a judgment system 20a which is a subsystem provided distinguishably from the recognition unit 10 and the control unit 30. The control unit 30 may be implemented as a control system 30a which is a subsystem provided distinguishably from the recognition unit 10 and the judgment unit 20. The recognition system 10a, the judgment system 20a, and the control system 30a may constitute components independent of each other.
[0055] Furthermore, a plurality of HMI (Human Machine Interface) devices 70 may be mounted on the vehicle 1. The portion of the plurality of HMI devices 70 that implements the operation input function of the occupant may also be a part of the recognition unit 10. The portion of the plurality of HMI devices 70 that implements the information presentation function may also be a part of the control unit 30. On the other hand, the function implemented by the HMI device 70 may also be positioned as a function independent of the recognition function, the judgment function, and the control function.
[0056] The recognition unit 10 is responsible for the recognition function including the positioning (e.g., estimation of the position) of the vehicle 1, other vehicles, and other road users. The recognition unit 10 detects the external environment, internal environment, vehicle state, and state of the driving system 2 of the vehicle 1. The recognition unit 10 fuses the detected information to generate an environmental model. The judgment unit 20 applies the purpose and driving policy to the environmental model generated by the recognition unit 10 to derive control actions. The control unit 30 executes the control actions derived by the recognition unit 10.
[0057] <Technical System Structure>
[0058] use Figure 2 An example of a detailed structure of the driving system 2 at the technical level is described below. The structure at the technical level may also refer to a physical architecture. The driving system 2 includes a plurality of sensors 40, a plurality of motion actuators 60, a plurality of HMI devices 70, and at least one processing system. These structural elements can communicate with each other through one or both of a wireless connection and a wired connection. These structural elements can also communicate with each other through, for example, an in-vehicle network based on CAN (registered trademark) or the like.
[0059] The plurality of sensors 40 include one or more external environment sensors 41. The plurality of sensors 40 may also include at least one of one or more internal environment sensors 42, one or more communication systems 43, and a map DB (database) 44. In the case where the sensor 40 is interpreted narrowly as representing the external environment sensor 41, the internal environment sensor 42, the communication system 43, and the map DB 44 may also be positioned as structural elements of the sensor 40 that are different from the technical level of the recognition function.
[0060] The external environment sensor 41 may also detect an object in the external environment of the vehicle 1. Examples of the object detection type external environment sensor 41 include a camera 41a, a LiDAR (Light Detection and Ranging / Laserimaging Detection and Ranging: Light Detection and Ranging / Laser Imaging Detection and Ranging) 41b, a laser radar, a millimeter wave radar, an ultrasonic sonar, an imaging radar, etc. As a typical example of sensor installation, a plurality of cameras 41a (e.g., 11 cameras 41a) configured to monitor the front, front side, side, rear side, and rear directions of the vehicle 1 may be installed on the vehicle 1.
[0061] As other installation examples, multiple cameras 41a (for example, four cameras 41a) configured to respectively monitor the front, sides and rear of the vehicle 1, multiple millimeter-wave radars (for example, five millimeter-wave radars) configured to respectively monitor the front, front sides, sides and rear of the vehicle 1, and a LiDAR 41b configured to monitor the front of the vehicle 1 can also be installed on the vehicle 1.
[0062] Furthermore, the external environment sensor 41 may detect the state of the atmosphere or the state of the climate in the external environment of the vehicle 1. The state detection type external environment sensor 41 is, for example, an external air temperature sensor, a temperature sensor, a raindrop sensor, or the like.
[0063] The internal environment sensor 42 may also detect a specific physical quantity related to the vehicle motion in the internal environment of the vehicle 1 (hereinafter referred to as the motion physical quantity). Examples of the motion physical quantity detection type of the internal environment sensor 42 include a speed sensor 42c, an acceleration sensor, a gyroscope sensor, and the like. The internal environment sensor 42 may also detect the state of an occupant in the internal environment of the vehicle 1 (for example, the driver state). Examples of the occupant detection type of the internal environment sensor 42 include an actuator sensor, a sensor for monitoring the driver and its system (hereinafter referred to as the driver monitor 42a), a biosensor, a pulse sensor 42b, a seat sensor, and a vehicle equipment sensor. Here, in particular, as an actuator sensor, there are acceleration sensors, brake sensors, steering sensors, and the like that detect the driver's operating state of the motion actuator 60 associated with the motion control of the vehicle 1.
[0064] The communication system 43 acquires communication data that can be used in the driving system 2 through wireless communication. The communication system 43 may also receive positioning signals from artificial satellites of the GNSS (global navigation satellite system) existing in the external environment of the vehicle 1. The communication device of the communication system 43 is, for example, a GNSS receiver.
[0065] The communication system 43 may also send and receive communication signals with an external system 96 existing in the external environment of the vehicle 1. The V2X type communication device in the communication system 43 is, for example, a DSRC (dedicated short range communications) communicator, a cellular V2X (C-V2X) communicator, etc. Examples of communication with the external system 96 existing in the external environment of the vehicle 1 include communication with systems of other vehicles (V2V), communication with infrastructure equipment such as a communicator set in a traffic light (V2I), communication with a mobile terminal of a pedestrian (V2P), and communication with a network such as a cloud server (V2N).
[0066] Furthermore, the communication system 43 can also send and receive communication signals with a mobile terminal 91 such as a smartphone brought into the interior environment of the vehicle 1. The terminal communication type communication device in the communication system 43 is, for example, a Bluetooth (registered trademark) communication device, a Wi-Fi (registered trademark) communication device, an infrared communication device, etc.
[0067] The map DB 44 is a database storing map data that can be used in the driving system 2. The map DB 44 is configured to include at least one non-transitory tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium. The map DB 44 may also include a database of a navigation unit that navigates the driving route to the destination of the vehicle 1. The map DB 44 may also include a database of high-precision maps with a high level of accuracy that are mainly used for the purpose of an automatic driving system. The map DB 44 may also include a database of a parking lot map that includes detailed parking lot information, such as parking box information, used for the purpose of automatic parking or parking assistance.
[0068] The map DB 44 suitable for the driving system 2 can also acquire and store the latest map data by communicating with the map server via the V2X type communication system 43, for example. The map data is digitized into two or three dimensions as data representing the external environment of the vehicle 1. The map data can also include marking data representing at least one of the position coordinates, shape, road surface state and standard runway of the road structure, for example. The marking data included in the map data can also include marking data representing at least one of the position coordinates and shape of the object marks, such as road signs, road signs, and dividing lines. The marking data included in the map data can also represent the object marks, such as traffic signs, arrow marks, lane marks, stop lines, direction signs, landmark beacons, commercial signs, line pattern changes of the road, etc. The map data can also include structural data representing at least one of the position coordinates and shapes of buildings and traffic lights facing the road, for example. The marking data included in the map data can also represent the object marks, such as street lights, edges of roads, reflectors, balls, etc.
[0069] The motion actuator 60 can control the vehicle motion based on the input control signal. The driving type motion actuator 60 is a powertrain including at least one of an internal combustion engine, a drive motor, etc. The braking type motion actuator 60 is, for example, a brake actuator. The steering type motion actuator 60 is, for example, a steering gear.
[0070] At least one of the HMI devices 70 may be an operation input device for transmitting the meaning or intention of the occupants including the driver of the vehicle 1 to the driving system 2 and capable of inputting the operation of the occupants including the driver. The operation input type HMI device 70 is, for example, an accelerator pedal, a brake pedal, a gear lever, a steering wheel, a turn signal lever, a mechanical switch, a touch panel of a navigation unit, etc. Among them, the accelerator pedal controls the power transmission system as the motion actuator 60. The brake pedal controls the brake actuator as the motion actuator 60. The steering wheel controls the steering actuator as the motion actuator 60.
[0071] At least one of the HMI devices 70 may be an information presentation device having a user interface 70b for presenting visual information, auditory information, skin sensory information, and the like to the occupants including the driver of the vehicle 1. Examples of the HMI device 70 of the visual information presentation type are a graphic instrument, a combination instrument, a navigation unit, a CID (center information display), a HUD (head-up display), a lighting unit, and the like. Examples of the HMI device 70 of the auditory information presentation type are a speaker, a buzzer, and the like. Examples of the HMI device 70 of the skin sensory information presentation type are a vibration unit of a steering wheel, a vibration unit of a driver's seat, a reaction force unit of a steering wheel, a reaction force unit of an accelerator pedal, a reaction force unit of a brake pedal, an air conditioning unit, and the like.
[0072] In addition, the HMI device 70 may communicate with a mobile terminal 91 such as a smartphone through the communication system 43, thereby realizing an HMI function in cooperation with the terminal 91. For example, the HMI device 70 may present information acquired from a smartphone to passengers including the driver. In addition, for example, operation input to a smartphone may be provided as an alternative to the HMI device 70. In addition, the mobile terminal 91 capable of communicating with the driving system 2 through the communication system 43 may function as the HMI device 70 itself.
[0073] As described above, the HMI device 70 may include a communication interface 70a and a user interface 70b. The user interface 70b may include a display that displays an image, a light that emits light, or other devices that present visual information, for example, when presenting visual information. The user interface 70b may also include a circuit for controlling the device. The communication interface 70a may also include at least one of a circuit and a terminal for communicating with other devices or systems via an in-vehicle network.
[0074] At least one processing system 50 is provided. For example, the processing system 50 may be a comprehensive processing system that comprehensively performs processing related to the recognition function, processing related to the judgment function, and processing related to the control function. In this case, the comprehensive processing system 50 may also perform processing related to the HMI function, or a processing system dedicated to the HMI function may be separately provided. For example, the processing system dedicated to the HMI function may also be a comprehensive cockpit system that comprehensively performs processing related to each HMI device.
[0075] In addition, for example, the processing system 50 may also have a structure that includes at least one processing unit corresponding to processing related to the identification function, at least one processing unit corresponding to processing related to the judgment function, and at least one processing unit corresponding to processing related to the control function.
[0076] The processing system 50 has an interface to the outside, and is connected to at least one element related to the processing of the processing system 50 via a communication unit. The communication unit is, for example, at least one of a LAN (Local Area Network), a CAN (registered trademark), a harness, an internal bus, and a wireless communication circuit. Elements related to the processing of the processing system 50 are the sensor 40, the motion actuator 60, and the HMI device 70.
[0077] The processing system 50 is configured to include at least one dedicated computer 51. The processing system 50 may also combine a plurality of dedicated computers 51 to realize functions such as recognition function, judgment function, control function, HMI function, etc.
[0078] For example, the dedicated computer 51 constituting the processing system 50 may be an integrated ECU that integrates the driving functions of the vehicle 1. The dedicated computer 51 constituting the processing system 50 may be a determination ECU that determines DDT. The dedicated computer 51 constituting the processing system 50 may be a monitoring ECU that monitors the driving of the vehicle 1. The dedicated computer 51 constituting the processing system 50 may be an evaluation ECU that evaluates the driving of the vehicle 1. The dedicated computer 51 constituting the processing system 50 may be a navigation ECU that navigates the driving route of the vehicle 1.
[0079] In addition, the dedicated computer 51 constituting the processing system 50 may be a locator ECU that estimates the position of the vehicle 1. The dedicated computer 51 constituting the processing system 50 may be an image processing ECU that processes image data detected by the external environment sensor 41. The dedicated computer 51 constituting the processing system 50 may be an HCU (HMI Control Unit) that comprehensively controls the HMI device 70.
[0080] The dedicated computer 51 constituting the processing system 50 may also include at least one memory 51a and processor 51b. The memory 51a may be at least one non-temporary tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium that non-temporarily stores programs and data that can be read by the processor 51b. In addition, as the memory 51a, a rewritable volatile storage medium such as a RAM (Random Access Memory) may also be provided. The processor 51b includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU as a core.
[0081] The dedicated computer 51 constituting the processing system 50 may be a SoC (System on a Chip) in which a memory, a processor, and an interface are integrated into one chip, or the dedicated computer 51 may include a SoC as a component.
[0082] Furthermore, the processing system 50 may also include at least one database for performing dynamic driving tasks. The database is configured to include at least one non-temporary tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium, and an interface for accessing the storage medium. The database may also be a scenario DB 53 in which a scenario structure is databased. In addition, the scenario DB 53 may not be provided in the driving system 2, and may be constructed in the external system 96 to be accessible from the processing system 50 of the vehicle 1 through the communication system 43, for example.
[0083] The scenario DB 53 may also include at least one of a functional scenario, a logical scenario, and a concrete scenario. The functional scenario defines the top-level qualitative scenario structure. The logical scenario is a scenario that gives a quantitative parameter range to the constructed functional scenario. The concrete scenario defines the boundary of the safety judgment that distinguishes the safe state from the unsafe state.
[0084] In addition, the processing system 50 may also include at least one recording device 55 for recording at least one of the identification information, the determination information, and the control information of the driving system 2. The recording device 55 may also include at least one memory 55a and an interface 55b for writing data to the memory 55a. The memory 55a may also be at least one non-transitory tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium.
[0085] At least one of the memories 55a may be mounted on the substrate in a manner that is not easily removable and replaceable, and in this manner, for example, an eMMC (embedded Multi Media Card) using a flash memory may be used. At least one of the memories 55a may be removable and replaceable relative to the recording device 55, and in this manner, for example, an SD card may be used.
[0086] The recording device 55 may also have a function of selecting information to be recorded from the identification information, the judgment information, and the control information. In this case, the recording device 55 may also have a dedicated computer 55c. In the dedicated computer 55c provided in the recording device 55, the processor may also temporarily store the information in a RAM or the like. The processor may also select information to be recorded non-temporarily from the temporarily stored information, and save the selected information in the memory 51a.
[0087] The mobile terminal 91 capable of communicating with the processing system 50 via the communication system 43 may be, for example, a smartphone or a tablet terminal. The mobile terminal 91 may be, for example, a structure including a dedicated computer 92, a user interface 94, and a communication interface 93.
[0088] The dedicated computer 92 constituting the mobile terminal 91 may also include at least one memory 92a and processor 92b. The memory 92a may be at least one non-temporary tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium that non-temporarily stores programs and data that can be read by the processor 92b. In addition, as the memory 92a, a rewritable volatile storage medium such as a RAM (Random Access Memory) may also be provided. The processor 92b includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU as a core.
[0089] The user interface 94 may be a structure including a display and a speaker. The display may be a display capable of displaying color images, such as a liquid crystal display, an OLED display, etc. The display and the speaker may present information to the user based on the control of the dedicated computer 92.
[0090] The communication interface 93 sends and receives communication signals with an external device or system. The communication interface 93 may include at least one of a cellular V2X (C-V2X) communicator, a Bluetooth (registered trademark) communicator, a Wi-Fi (registered trademark) communicator, an infrared communicator, and the like.
[0091] The external system 96 capable of communicating with the processing system 50 via the communication system 43 may be, for example, a cloud server, a remote center, etc. The external system 96 may be a structure including at least one dedicated computer 97 and at least one driving information DB 98 .
[0092] The dedicated computer 97 constituting the external system 96 may also include at least one memory 97a and processor 97b. The memory 97a may be at least one non-temporary tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium that non-temporarily stores programs and data that can be read by the processor 97b. In addition, as the memory 97a, a rewritable volatile storage medium such as a RAM (Random Access Memory) may also be provided. The processor 97b includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU as a core.
[0093] The driving information DB 98 is a database that records and accumulates information related to driving of a plurality of vehicles including the vehicle 1. The driving information DB 98 may also be configured to include at least one non-temporary tangible storage medium such as a semiconductor memory, a magnetic medium, and an optical medium that has a large-capacity storage area and non-temporarily stores data that can be read by the processor 97b, and an interface for accessing the storage medium.
[0094] <System structure at the functional level>
[0095] Next, use Figure 3 An example of a detailed structure of the driving system 2 at the functional level is described below. The structure at the functional level may also refer to a logical architecture. The recognition unit 10 may include an external recognition unit 11, a self-position recognition unit 12, a fusion unit 13, and an internal recognition unit 14 as sub-blocks into which the recognition functions are further classified.
[0096] The external recognition unit 11 realizes the function of processing the detection data detected by each external environment sensor 41 individually and identifying objects such as objects, other road users, etc. The detection data can be, for example, detection data provided by millimeter wave radar, sonar, LiDAR 41b, etc. The external recognition unit 11 can also generate relative position data including the direction, size and distance of the object relative to the vehicle 1 based on the raw data detected by the external environment sensor 41.
[0097] In addition, the detection data may be, for example, image data provided by the camera 41a, LiDAR 41b, etc. The external recognition unit 11 processes the image data and extracts the object within the field of view of the image. The extraction of the object may also include the direction of the object relative to the vehicle 1, the estimation of the size and the distance. In addition, the extraction of the object may include, for example, the classification of the object using semantic segmentation.
[0098] The own position recognition unit 12 implements positioning of the vehicle 1. The own position recognition unit 12 obtains global position data of the vehicle 1 from the communication system 43 (e.g., a GNSS receiver). In addition, the own position recognition unit 12 may also obtain at least one of the position information of the object extracted by the external recognition unit 11 and the position information of the object extracted by the fusion unit 13. In addition, the own position recognition unit 12 obtains map information from the map DB 44. The own position recognition unit 12 combines this information to estimate the position of the vehicle 1 on the map.
[0099] The fusion unit 13 fuses the external recognition information of each external environment sensor 41 processed by the external recognition unit 11, the positioning information processed by the own position recognition unit 12, and the V2X information obtained through V2X.
[0100] The fusion unit 13 fuses the information of other road users and other objects individually identified by each external environment sensor 41, and determines the type and relative position of the objects around the vehicle 1. The fusion unit 13 fuses the information of the object marks on the road individually identified by each external environment sensor 41, and determines the static structure of the road around the vehicle 1. The static structure of the road includes, for example, the curvature of the curve, the number of lanes, and the vacant space.
[0101] Next, the fusion unit 13 fuses the types and relative positions of objects around the vehicle 1, the static structure of the road, the positioning information, and the V2X information to generate an environment model. The environment model can be provided to the determination unit 20. The environment model can be a model dedicated to the external environment.
[0102] The environment model may also be a comprehensive model that integrates information such as the internal environment, vehicle state, and state of the driving system 2 by adding information to be acquired. For example, the fusion unit 13 may acquire traffic regulations such as the Road Traffic Act and reflect them in the environment model.
[0103] The interior recognition unit 14 realizes the function of processing the detection data detected by each interior environment sensor 42 and recognizing the vehicle state. The vehicle state may also include the state of the motion physical quantity of the vehicle 1 detected by the speed sensor 42c, the acceleration sensor, the gyro sensor, etc. In addition, the vehicle state may also include at least one of the state of the passengers including the driver, the operation state of the driver on the motion actuator 60, and the switch state of the HMI device 70.
[0104] The determination unit 20 may include an environment determination unit 21 , a driving plan unit 22 , and a mode management unit 23 as sub-blocks into which the determination functions are further classified.
[0105] The environment judgment unit 21 obtains the environment model generated by the fusion unit 13 and the vehicle state recognized by the internal recognition unit 14, and implements judgment on the environment based on them. Specifically, the environment judgment unit 21 can also interpret the environment model to estimate the current situation of the vehicle 1. The situation here can also be a driving situation (operational situation). The environment judgment unit 21 can also interpret the environment model to predict the actions of other road users. The environment judgment unit 21 can also interpret the environment model to predict the trajectory of objects such as other road users. In addition, the environment judgment unit 21 can also interpret the environment model to predict potential dangers.
[0106] In addition, the environment determination unit 21 may interpret the environment model and perform determination related to the scene that the vehicle 1 is currently in. The determination related to the scene may also be to select at least one scene that the vehicle 1 is currently in from the directory of scenes constructed in the scene DB 53 .
[0107] Furthermore, the environment judgment unit 21 may estimate the driver's intention based on at least one of the predicted actions, the predicted trajectory of the object, the predicted potential danger, the scene-related judgment, and the vehicle state provided by the internal recognition unit 14 .
[0108] The driving planning unit 22 plans driving of the vehicle 1 based on at least one of the estimated position information of the vehicle 1 on the map by the own position recognition unit 12, the judgment information and driver intention estimation information by the environment judgment unit 21, and the function restriction information by the mode management unit 23.
[0109] The driving planning unit 22 implements a route planning function, a behavior planning function, and a trajectory planning function. The route planning function is a function of planning at least one of a route to a destination and a lane plan at a medium distance based on estimated information of the position of the vehicle 1 on a map. The route planning function may also include a function of determining at least one of a lane change request and a deceleration request based on the lane plan at a medium distance. Here, the route planning function may be a task / route planning function in a strategic function, or may be a function of outputting a task plan and a route plan.
[0110] The behavior planning function is a function of planning the behavior of the vehicle 1 based on at least one of the route to the destination planned by the route planning function, the lane plan at the medium distance, the lane change request and the deceleration request, the judgment information of the environment judgment unit 21 and the driver's intention estimation information, and the function restriction information of the mode management unit 23. The behavior planning function may also include a function of generating conditions related to the state transition of the vehicle 1. The conditions related to the state transition of the vehicle 1 may also be equivalent to a triggering condition. The behavior planning function may also include a function of determining the state transition of the application of the DDT and the state transition of the driving action based on the condition. The behavior planning function may also include a function of determining the longitudinal restrictions related to the path of the vehicle 1 and the lateral restrictions related to the path of the vehicle 1 based on the information of these state transitions. The behavior planning function may be a tactical behavior plan in the DDT function, or a function of outputting tactical behavior.
[0111] The trajectory planning function is a function of planning the driving trajectory of the vehicle 1 based on the judgment information of the environment judgment unit 21, the longitudinal restrictions related to the path of the vehicle 1, and the lateral restrictions related to the path of the vehicle 1. The trajectory planning function may also include a function of generating a path plan. The path planning may also include speed planning, or speed planning may be generated as a plan independent of the path planning. The trajectory planning function may also include a function of generating multiple path plans and selecting the best path plan from multiple path plans, or a function of switching path plans. The trajectory planning function may also include a function of generating backup data of the generated path plan. The trajectory planning function may be a trajectory planning function in the DDT function, or a function of outputting a trajectory plan.
[0112] The mode management unit 23 monitors the driving system 2 and sets restrictions on driving-related functions. The mode management unit 23 may also manage the mode of automatic driving, such as the state of the automatic driving level. The management of the automatic driving level may also include the switching between manual driving and automatic driving, that is, the transfer of authority between the driver and the driving system 2, in other words, the management of takeover. The mode management unit 23 may also monitor the state of the subsystem related to the driving system 2 and determine the abnormality of the system (such as error, unstable state of action, system failure, fault). The mode management unit 23 may also determine the mode based on the driver's intention based on the driver's intention estimation information generated by the internal recognition unit 14. The mode management unit 23 may also set restrictions on driving-related functions based on at least one of the determination result of the abnormality of the system, the determination result of the mode, the vehicle state obtained by the internal recognition unit 14, the sensor abnormality (or sensor failure) signal output from the sensor 40, the state transition information of the application obtained by the driving planning unit 22, and the trajectory plan.
[0113] In addition, the mode management unit 23 may also have a function of determining longitudinal restrictions on the path of the vehicle 1 and lateral restrictions on the path of the vehicle 1 in addition to determining restrictions on driving-related functions. In this case, the driving planning unit 22 plans the behavior and the trajectory according to the restrictions determined by the mode management unit 23.
[0114] The control unit 30 may include a motion control unit 31 and an HMI output unit 71 as sub-blocks into which the control functions are further classified. The motion control unit 31 controls the motion of the vehicle 1 based on the trajectory plan (e.g., path planning and speed planning) obtained from the driving plan unit 22. Specifically, the motion control unit 31 generates acceleration request information, shift request information, brake request information, and steering request information corresponding to the trajectory plan, and outputs them to the motion actuator 60.
[0115] Here, the motion control unit 31 can directly obtain the vehicle state recognized by the recognition unit 10 (especially the internal recognition unit 14 ), for example, at least one of the current speed, acceleration and yaw rate of the vehicle 1 , from the recognition unit 10 and reflect it in the motion control of the vehicle 1 .
[0116] The HMI output unit 71 outputs information related to the HMI based on at least one of the judgment information and driver intention estimation information of the environment judgment unit 21, the state transition information and trajectory plan of the application of the driving plan unit 22, and the restriction information of the function of the mode management unit 23. The HMI output unit 71 can also manage vehicle interaction. The HMI output unit 71 can generate a notification request based on the management status of the vehicle interaction and control the information presentation function in the HMI device 70. In addition, the HMI output unit 71 can also generate a control request for the wiper, sensor cleaning device, headlight, and air conditioning device based on the management status of the vehicle interaction and control these devices.
[0117] <Security model and its rules>
[0118] The driving system 2 may be constructed by incorporating assumptions about reasonably foreseeable behaviors of other road users that are considered in the safety model of the autonomous driving. The safety model may be, for example, a safety association model or a formal model. As the safety model, for example, an RSS (Responsibility-Sensitive Safety) model or an SFF (Safety Force Field) model may be used, or other models, a more generalized model, or a composite model that combines multiple models may be used.
[0119] For example, five rules (five principles) are adopted in the RSS model. The first rule is "Do not hit someone from behind." The second rule is "Do not cut-in recklessly." The third rule is "Right-of-way is given, not taken." The fourth rule is "Be careful of area with limited visibility." The fifth rule is "If you can avoid an accident without causing another one, you must do it." These rules can also be equivalent to the rules specified by the safety model of autonomous driving.
[0120] The safety envelope can be defined based on five rules, in particular the first rule and the second rule. The safety envelope can refer to the longitudinal safety distance and the lateral safety distance relative to other road users themselves, or it can refer to the conditions or concepts used to calculate these safety distances. The longitudinal safety distance and the lateral safety distance can be calculated taking into account the reasonably foreseeable assumptions of other road users.
[0121] The longitudinal safety distance may be set to the following distance: when the leading vehicle is traveling at a specified speed and brakes at the maximum speed limit to stop, the following vehicle will not rear-end the vehicle even if it accelerates with a specified reaction time and maximum acceleration and then brakes at the minimum deceleration to stop. In addition, the longitudinal safety distance may be set to the following distance: when two vehicles are traveling at their respective speeds in the opposite direction, they accelerate with a specified reaction time and maximum acceleration and then brakes at the minimum deceleration to stop, and there will be no head-on collision.
[0122] The lateral safety distance may be set to a minimum distance at which two vehicles do not collide even if they accelerate with a predetermined reaction time and maximum acceleration and then decelerate with maximum deceleration in the lateral direction while traveling adjacent to each other at respective lateral speeds.
[0123] In addition, for example, a core principle is adopted in the SFF model. The principle is that "all relevant persons (actors) are required to apply safety control actions that contribute at least to the same degree as the safety program used to improve safety potential." This principle can also be equivalent to the rules specified by the safety model of autonomous driving.
[0124] Here, the size of the space between the two deceleration schedules of the safety program schedule and the maximum braking schedule is defined as the clamed set. The safety potential can be defined as an indicator of the overlap between the clamed sets of two vehicles. The SFF can be defined as the negative gradient of the safety potential.
[0125] <Driving evaluation and instruction>
[0126] The driving system 2 of the present embodiment has a function of evaluating the driving of a driver who performs manual driving and a driver who receives driving assistance and performs manual driving (hereinafter, the evaluation function) and a function of teaching (hereinafter, the teaching function). The driver who performs manual driving may be a driver who drives the vehicle 1 in a state of autonomous driving level 0. The driver who receives driving assistance and performs manual driving may be a driver who drives the vehicle 1 in a state of autonomous driving level 1 or 2. The driving system 2 can present teaching to the driver through the HMI device 70 for following the rules specified by the safety model of the autonomous driving model.
[0127] For example, in the processing system 50, by further constructing a dedicated computer 51 as shown in FIG. Figure 4 The functional blocks shown in the figure, such as the information acquisition unit 72, the driver estimation unit 73, the driving action information generation unit 74, and the risk estimation unit 75, realize the evaluation function and the teaching function. In the case where at least a part of the functions realized by the information acquisition unit 72, the driver estimation unit 73, the driving action information generation unit 74, and the risk estimation unit 75 overlap with the functions of the environment judgment unit 21, the driving plan unit 22, and the mode management unit 23, the overlapping functional blocks may also assume the functions.
[0128] The information acquisition unit 72 acquires information required for the implementation of the teaching function. The information required for the implementation of the teaching function may be, for example, various information on the vehicle state, the driver state, and the external environment. Such information acquisition may be direct information acquisition of detection data detected by sensors 40 such as the speed sensor 42c and the communication system 43, or information acquisition from an environment model generated based on such detection data.
[0129] The driver estimation unit 73 performs estimation on the driver using the information acquired by the information acquisition unit 72. The estimation on the driver may be at least one of estimation of the current driver state, estimation of the future driver state, and estimation of the current driver's intention.
[0130] The estimation of the driver's state may also include an estimation of whether the driver's state is positive or negative. The estimation of whether the driver's state is positive or negative may be implemented based on the driver's facial expression and heart rate.
[0131] For example, the information acquired by the information acquisition unit 72 can be input as an input parameter to the learned neural network constructed in the processing system 50 to obtain an analysis result of whether the driver's state is positive or negative. Specifically, the image of the driver's face taken by the driver monitor 42a and the driver's heart rate data detected by the pulse sensor 42b are input as input parameters to the neural network. Then, it is possible to estimate whether the driver's state is positive or negative based on the analysis result output from the neural network. The analysis result can also be, for example, a result representing a numerical value of 0 to 100 regarding an index representing each emotion of the driver. For example, when the index of the driver's "happy" emotion is a high index, the driver's state is estimated to be positive. In addition, for example, when the index of the driver's "sad" emotion is a high index, the driver's state is estimated to be negative.
[0132] The driving action information generating unit 74 detects the driving action of the driver and generates information related to the driving action. The generation of the information related to the driving action here may also refer to only extracting the behavior of the vehicle 1 as a result of the driving action of the driver. The generation of the information related to the driving action here may also include associating the behavior of the vehicle 1 with the external environment. The association of the behavior of the vehicle 1 with the external environment may be the generation of information in which the external environment is associated with the behavior of the vehicle 1. The information in which the external environment is associated with the behavior of the vehicle 1 is, for example, information that the vehicle 1 is traveling at an intersection while the traffic light is displaying a stop signal, information that the vehicle 1 is traveling straight at the intersection from the right turn lane, and the like.
[0133] Furthermore, the generation of the information on the driving action may include further associating the information in which the external environment and the behavior of the vehicle 1 are associated with the rules defined by the safety model of the autonomous driving.
[0134] The risk estimation unit 75 estimates the risk of the driver's driving. The risk estimation here may be an example of an evaluation of the driver's driving. The risk here may also indicate, for example, the possibility of interference or collision with other road users. For example, when the RSS model is used as a safety model for autonomous driving, the risk may be replaced by a responsibility value indicating the degree of accident responsibility that the vehicle 1 bears for other road users, or may be a concept equivalent to the responsibility value.
[0135] The estimation of the risk level may also include evaluating the driver's driving using the rules specified by the safety model of autonomous driving. The evaluation using the rules specified by the safety model of autonomous driving may also include a determination as to whether the vehicle 1 violates the rules. The determination may also be implemented under the assumption that the vehicle 1 in manual driving is autonomous driving. For example, the determination may also include a determination as to whether the vehicle 1 violates the safety envelope. For example, when the RSS model is used as the safety model of autonomous driving, it may also include a determination as to whether the distance between the vehicle 1 and other road users such as other vehicles is below the safe distance.
[0136] The evaluation using the rules defined by the safety model of autonomous driving may also include an evaluation based on a safety evaluation criterion set according to the rules. Here, the safety evaluation criterion may also include at least one indicator of the possibility of collision with surrounding objects, the blind spot ratio of the road being driven, and the collision avoidance probability when a collision avoidance action is performed. The determination of whether the safety evaluation criterion is met may also be determined based on a predetermined threshold value set for each indicator.
[0137] The estimation of the degree of danger may also include detecting the degree of deviation of the driver's driving from the rules. The degree of deviation may indicate the degree of violation of the rules. For example, when the driver's driving does not violate the rules, the degree of deviation may be set to 0. The detection of the degree of deviation may be included in the evaluation using the rules specified by the safety model of autonomous driving, or may be implemented separately after the evaluation. In the case of calculating the degree of deviation based on the safety evaluation standard, the degree of deviation may be the difference between the value obtained by quantifying the evaluation value at the time of violation calculated in the evaluation of the actual driver's driving action described above and the threshold value itself. In the case of calculating the degree of deviation based on the safety evaluation standard, the degree of deviation may also be calculated based on the difference between the safety evaluation value and the threshold value. The degree of deviation may also be calculated as a composite or comprehensive parameter for multiple rules or safety evaluation standards.
[0138] The estimation of the risk level may also include an evaluation of the margin time to collision with other road users. The margin time to collision is an indicator indicating how long it will take for the vehicle 1 to collide with other road users if the current relative speed is maintained.
[0139] The estimation of the risk level may include the evaluation of the driver's state. The evaluation of the driver's state may include the determination based on the estimation result of whether the driver's state is positive or negative estimated by the driver estimation unit 73 .
[0140] The risk level can be estimated by any one of the above-mentioned evaluation and determination methods, or by a combination of the above-mentioned evaluation and determination methods. The risk level can also be estimated by classifying it into three levels: low risk, medium risk, and high risk. The risk level can also be estimated by classifying it into two levels or four or more levels. The risk level can also be expressed as a continuous value from 0 to 100.
[0141] For example, consider Figure 5 As shown in the figure, the distance between the vehicle 1 and the preceding other vehicle OV1 is less than the safe distance. In this scenario, the possibility of collision is greater than the threshold value specified by the rule. In this case, the risk estimation unit 75 may also estimate that the driver's driving is highly risky.
[0142] In addition, for example, consider Figure 6 As shown in FIG. 1 , a vehicle 1 is speeding at a blocked intersection with poor visibility. In this scenario, it is estimated that the driver did not expect other road users OV2 (e.g., safety-related objects) to appear from the blocked area OA. In this case, the risk estimation unit 75 may also estimate the driver's driving as highly risky.
[0143] In addition, for example, consider Figure 7As shown in the figure, a vehicle 1 is traveling in the left lane L1 of a single-sided two-lane road, and another vehicle OV3 traveling in front of the lane of the vehicle 1 suddenly drops a cargo OB1. In this scenario, another other vehicle OV4 traveling in the right lane L2 is set to the right side of the vehicle 1. Here, if the vehicle 1 wants to change lanes further to the right lane L2, the scenario becomes a composite scenario composed of a cargo drop scenario and a queue-jumping scenario. In this scenario, the collision avoidance probability is less than a predetermined threshold. In this case, the risk estimation unit 75 may also estimate the driver's driving as having a high risk.
[0144] Then, for example, the HMI output unit 71 can implement processing for outputting information for presenting instructions to the driver for following rules, in other words, presenting required information (hereinafter referred to as "presenting required information") to at least one of the HMI device 70, the mobile terminal 91, and the external system 96.
[0145] The information required for presentation may be, for example, at least one of an estimated result of an estimation related to the driver, driving action information, and an estimated result of the degree of danger. As described in detail later, in the case where the presentation content to the driver is generated by the sending side of the information required for presentation, the information required for presentation may also be the presentation content itself presented to the driver.
[0146] At least one of the estimation result, driving action information, and the estimation result of the risk level of the estimation related to the driver can be stored in the recording device 55 of the processing system 50. The data can be stored in the driving information DB 98 of the external system 96 by sending and receiving information via the communication system 43. The stored data can also be used for determination for implementing teaching. The stored data can also be used for generation of presentation content described later. The stored data can also be used for verification after an accident occurs.
[0147] When an evaluation of rule violation is made, the HMI output unit 71 may output the information required for presentation to at least one of the HMI device 70, the mobile terminal 91, and the external system 96. On the other hand, when no rule violation is confirmed, the information required for presentation may not be output, and on the other hand, the information required for presentation may be output as reference information or for the purpose of accumulating statistical data.
[0148] In the case of an evaluation of violation of rules, the HMI output unit 71 may also determine the timing of presentation based on at least one of the degree of danger, degree of deviation, responsibility value, and urgency. The timing of presentation may also be selected from during the driver's driving and after the driver's driving. The optimized presentation content may also be presented during the driver's driving and after the driver's driving.
[0149] For the driver's driving, it is also possible to select further subdivided timings such as immediately, when a specified condition is met during driving (for example, when the car temporarily stops at an intersection), etc. For the driver's driving, it is also possible to select further subdivided timings such as during the automatic driving period after taking over from manual driving to automatic driving levels 3 to 5, or after reaching the destination, etc.
[0150] The processing system 50 (e.g., HMI output unit 71) of the vehicle 1 as the sending side for presenting the required information and at least one of the HMI device 70, mobile terminal 91, and external system 96 as the receiving side for presenting the required information may also have the function of generating presentation content to the driver.
[0151] The presentation content here may be visual information presentation content such as still image content and dynamic image content that presents visual information. The presentation content may be auditory information presentation content such as sound content that presents auditory information. The presentation content may be skin sensory information content that presents skin sensory information. Furthermore, the presentation content may be content that combines visual information and auditory information. Regarding the generation of the presentation content, it may be generated according to a generation rule based on at least one of a safety model and a safety evaluation benchmark. Regarding the content of the presentation content, it may be determined by considering the driver's driving habits and the comparison results between the current driving and the usual driving (e.g., past driving).
[0152] The generation of the presentation content may also be achieved by selecting one content from a plurality of pre-prepared contents based on the estimated result of the driver's state, the driving action information, and the estimated result of the risk level as the information required for presentation. The selection may be implemented based on the conditions according to the above-mentioned generation rule. It may also be possible to partially change the selected content based on the detailed content of the driving action information.
[0153] In addition, the generation of presentation content can also be generated by using a learned neural network that has learned the above-mentioned generation rules. Specifically, the estimated result of the driver's state, driving behavior information, and the estimated result of the risk level, which are the information required for presentation, are input as input parameters to the neural network, and the presentation content is output from the neural network. The input parameters can also further include at least one of the detection data of the external environment sensor 41, the environment model, and the vehicle state.
[0154] Figure 8, an example is shown in which a presentation content generating unit 76a, which is a functional block constructed by a dedicated computer 51, is provided in a processing system 50, and presentation content is generated by the presentation content generating unit 76a. In this example, the presentation content generating unit 76a generates presentation content based on the estimation result of the estimation related to the driver, the driving action information, and the estimation result of the degree of danger recorded in the recording device 55. Then, the generated content data can be directly sent to the HMI device 70 and the mobile terminal 91 for implementing the teaching to the driver. On the other hand, the generated content data can also be provided to the mobile terminal 91 for implementing the teaching to the driver by being sent to the external system 96 and stored in the driving information DB 98 and then downloaded to the mobile terminal 91.
[0155] As another example, Fig. 9 9 shows an example in which a presentation content generation unit 76b, which is a functional block implemented using a dedicated computer 92, is provided in a mobile terminal 91. In this example, the estimated result of the driver's state, the driving action information, and the estimated result of the risk level as the information required for presentation, and the presentation instruction are output from the HMI output unit 71 of the processing system 50 to the mobile terminal 91. Accordingly, the presentation content generation unit 76b of the mobile terminal 91 generates presentation content. This structure can also be implemented by downloading and installing a program that executes the content generation processing of the presentation content generation unit 76b together with an application that implements teaching from a network or an external system 96.
[0156] As another example, Fig.10 9 shows an example in which a presentation content generation unit 76c, which is a functional block implemented using a dedicated computer 97, is provided in an external system 96. In this example, the estimated result of the driver's state, the driving action information, and the estimated result of the risk level as the information required for presentation are output from the HMI output unit 71 of the processing system 50 to the external system 96, and accordingly, the presentation content generation unit 76c of the external system 96 generates presentation content. The information required for presentation including the generated content data can be recorded in the driving information DB 98. On the other hand, the mobile terminal 91 can also download the content data from the external system 96 and implement teaching to the driver when receiving a presentation instruction from the HMI output unit 71.
[0157] As an example of real-time teaching, teaching based on the contents of the combination of HUD display and speaker sound can also be performed (see Fig.11 , Fig.12 ).For example Fig.112 shows a teaching method in which a pedestrian P1 intends to cross the front of the vehicle 1 from the right front of the vehicle 1 and the driver is estimated to drive without considering the pedestrian P1. In this case, the HUD displays a teaching image IM1 that teaches the existence of the pedestrian P1 as a virtual image in the portion of the displayable area of the windshield WS of the vehicle 1 closest to the pedestrian P1. At the same time, the speaker emits a teaching voice such as "Pay attention to the pedestrian in front of the right" to teach the driver to drive with consideration for the pedestrian P1.
[0158] In addition, for example, Fig.12 2 shows a teaching method when the distance between the vehicle 1 and the other vehicle OV5 ahead is less than the safe distance. In this case, the HUD displays a teaching image IM2 in a virtual image in the displayable area of the windshield WS of the vehicle 1 that is visually confirmed to be behind the other vehicle OV5 ahead, using a plurality of horizontal lines to make the driver aware of the distance. At the same time, the speaker emits a teaching voice such as "Please leave a gap with the vehicle ahead" to teach the driver to drive with consideration for the distance.
[0159] As an example of teaching after driving, the following can also be implemented: Fig.13 The teaching based on the content combining the display of dynamic images and sound using the mobile terminal 91 shown here can be said to be the content combining the visual information showing the scene encountered by the driver and the auditory information suggesting the improvement of driving in the scene.
[0160] Specifically, the speaker of the mobile terminal 91 emits a teaching voice such as "Use dynamic images to show scenes that may lead to accidents. It is customary to exceed the speed limit in places with blind spots. Please drive slowly in places with poor visibility to deal with pedestrians / bicycles that suddenly rush out." to advise the driver to correct the driver's bad driving habits. At the same time, the display of the mobile terminal 91 displays a teaching dynamic image that illustrates scenes that may lead to accidents.
[0161] The visual information presentation content used in teaching is preferably generated in a manner that takes into account the privacy of other road users. For example, when the visual information content is generated using information based on the detection data of the sensor 40, the content can be generated in a manner that makes it difficult to identify the personal information of other road users. For example, a dynamic image in which the face of a pedestrian reflected by the camera 41a is blurred can be generated as content.
[0162] When teaching is performed by the mobile terminal 91, the teaching can be performed by the driver installing an application having a program for implementing the teaching function in advance in the mobile terminal 91. The teaching can also be started by the driver operating the application. The teaching can also be started automatically according to the timing of receiving the driver's teaching instruction.
[0163] As another example of teaching after driving, teaching may be performed by reporting the contents of visual information presented using an instrument, CID, HUD, mobile terminal 91, etc., or the contents of auditory information presented using a speaker.
[0164] Specifically, a report such as "You tend to bulge outward when turning, which may cause a collision with vehicles in adjacent lanes. Please slow down before entering the curve and then turn after slowing down. One of the reasons is that you cannot operate the steering wheel smoothly due to one-handed driving, so please drive with both hands on the steering wheel." can be presented to the driver.
[0165] In addition, a report such as "The distance between vehicles is becoming narrower today than usual. The driver cannot cope with the sudden deceleration of the vehicle in front, and there is a risk of collision. Please drive with a sufficient distance between vehicles." can be presented to the driver.
[0166] As described above, the upper limit of the amount of information of the presentation content assuming the driver's teaching during driving can be set smaller than the upper limit of the amount of information of the presentation content assuming the driver's teaching after driving. Furthermore, the upper limit of the reproduction time of the presentation content assuming the driver's teaching during driving can be set smaller than the upper limit of the reproduction time of the presentation content assuming the driver's teaching after driving. That is, the teaching during driving can be implemented in a shorter manner than the teaching after driving, and only the key points can be notified.
[0167] In addition, at least one of the amount of information and the timing of presentation of the content to be presented is adjusted according to the estimated result of the risk level. For example, in the case where the risk level is estimated to be high, the timing of presentation can be set during the driver's driving, and the amount of information of the content to be presented can be set smaller than in the case where the risk level is estimated to be lower.
[0168] <Processing Flow>
[0169] Next, use Fig.14 The flowchart of is used to illustrate an example of a processing method for realizing the evaluation function and the teaching function. The series of processes shown in steps S11 to S16 are executed by the driving system 2 at predetermined intervals or based on a predetermined trigger. As a specific example, when the autonomous driving mode is managed as autonomous driving level 0, a series of processes may also be executed at predetermined intervals. As another specific example, when the autonomous driving mode is managed as autonomous driving levels 0 to 2, a series of processes may also be executed at predetermined intervals.
[0170] As described in detail later, a part of the series of processes may be executed by at least one of the external system 96 and the mobile terminal 91. The series of processes may be executed in accordance with a computer program stored in a memory.
[0171] In the first S11, the information acquisition unit 72 acquires information necessary for realizing the teaching function. After the processing of S11, the process moves to S12.
[0172] In S12, the driver estimation unit 73 estimates the driver using the information acquired in S11. After the processing in S12, the process moves to S13.
[0173] In S13, the driving action information generating unit 74 generates information on the driver's driving action using the information acquired in S11. After the processing in S13, the process moves to S14. In addition, the order of the processing in S12 and the processing in S13 may be reversed, for example, two different processors may be used to perform the processing simultaneously in parallel.
[0174] In S14, the risk level estimation unit 75 estimates the risk level using the estimation in S12 and the information on the driving action in S13. After the processing in S14, the process moves to S15.
[0175] In S15, the HMI output unit 71 outputs the presentation information to at least one of the HMI device 70, the mobile terminal 91, and the external system 96. The output of the presentation information to the mobile terminal 91 or the external system 96 is substantially the transmission of the presentation information through the communication system 43. After the processing of S15, the process moves to S16.
[0176] In S16, at least one of the HMI device 70 and the mobile terminal 91 that have acquired the presentation content generated by the HMI device 70 as the presentation required information, or the HMI device 70 and the mobile terminal 91 that have acquired the presentation required information and generated the presentation content based on the information, implements teaching to the driver. A series of processes ends with S16.
[0177] Next, use Fig.15 An example of a method of processing the estimated risk level in S14 will be described in detail with reference to the flowchart of FIG.
[0178] In S101, the risk estimation unit 75 determines whether the driver's driving violates the safety envelope based on the information of the driving action. If an affirmative determination is made in S101, the process proceeds to S102. If a negative determination is made in S101, the process proceeds to S105.
[0179] In S102, the risk estimation unit 75 detects the deviation of the driver's driving from the rules and determines whether the deviation is less than a predetermined judgment reference value. The judgment reference value may be a fixed value set in advance. In addition, in the case where the deviation cannot be expressed as a quantitative value and is difficult to compare with the judgment reference value, a negative judgment may be made. In the case where a positive judgment is made in S102, the process moves to S103. In the case where a negative judgment is made in S103, the process moves to S107.
[0180] In S103, the risk estimation unit 75 determines whether the margin time is longer than a predetermined judgment reference value. The judgment reference value may be a predetermined fixed value. In the case of a positive determination in S103, the process proceeds to S104. In the case of a negative determination in S103, the process proceeds to S107. In addition, in the case where the determination content of S103 substantially overlaps with the determination content of S101, the process of S103 may be omitted.
[0181] In S104, the risk estimation unit 75 determines whether the driver state is negative based on the estimation result of the driver estimation unit 73. If an affirmative determination is made in S104, the process proceeds to S107. If a negative determination is made in S104, the process proceeds to S106.
[0182] In S105, the risk level estimation unit 75 estimates that the driver's driving is of low risk. A series of processing ends with S105.
[0183] In S106, the risk level estimation unit 75 estimates that the driver's driving is of medium risk level. A series of processing ends with S106.
[0184] In S107, the risk level estimation unit 75 estimates that the driver's driving is of high risk. A series of processing ends with S107.
[0185] Next, use Fig.16 An example of a processing method for implementing the exchange of information of S15 and S16 and presentation to the driver is described in detail with reference to the flowchart of FIG.
[0186] In S111, the HMI output unit 71 determines whether the risk level of the driver's driving is estimated to be medium or higher, that is, medium or high risk. If a positive determination is made in S111, the process moves to S112. If a negative determination is made in S111, the series of processes ends.
[0187] In S112, the HMI output unit 71 determines whether the driver's driving is estimated to be highly dangerous. If an affirmative determination is made in S112, the process proceeds to S113. If a negative determination is made in S112, the process proceeds to S115.
[0188] In S113, the HMI output unit 71 and the HMI device 70 perform a presentation process during the driver's driving. In this example, when the driver's driving is estimated to be highly dangerous, the HMI output unit 71 selects to perform a teaching process during the driver's driving. Based on the output of the presentation required information and the presentation instruction by the HMI output unit 71, the HMI device 70 performs presentation, i.e., teaching, to the driver. After the processing of S113, the process moves to S114.
[0189] In S114, information such as the information required for presentation and presentation history information of the presentation content is saved. This information may be stored in the recording device 55 as information of the vehicle 1 alone. This information may also be stored in the driving information DB 98 in the external system 96 in a manner of being collected together with information of a plurality of vehicles. After the processing of S114, the process moves to S116.
[0190] In S115, the information required for presentation is saved. This information may be stored in the recording device 55 as information of the vehicle 1 alone. This information may also be stored in the driving information DB 98 in a manner of being collected together with information of a plurality of vehicles. After the processing of S115, the process moves to S116.
[0191] In S116, the HMI output unit 71 determines whether the driver has finished driving. If an affirmative determination is made in S116, the process proceeds to S117. If a negative determination is made in S116, S116 is performed again after, for example, a predetermined time has passed.
[0192] In S117, at least one of the HMI output unit 71, the HMI device 70, and the mobile terminal 91 implements the presentation process after the driver has finished driving. In this example, when the risk level of the driver's driving is estimated to be medium or above, the HMI output unit 71 chooses to implement the teaching of the driver after the driver has finished driving. For example, based on the output of the presentation required information and the presentation instruction by the HMI output unit 71, at least one of the HMI device 70 and the mobile terminal 91 can present, i.e., teach, to the driver. In addition, for example, at least one of the HMI device 70 and the mobile terminal 91 can obtain and refer to the information stored in S115 and S116 to present, i.e., teach, to the driver. A series of processes ends with S117.
[0193] In this way, sometimes for the same driving action of the driver, the presentation process during the driver's driving (refer to S113) and the presentation process after the driver's driving (S117) are repeatedly implemented. In this way, for the same driving action of the driver, multiple teachings can also be implemented on the basis of changing at least one of the device for implementing teaching, the amount of information, and the timing of presentation. Through multiple teachings accompanied by changes in the presentation method, the driver's driving appropriateness can be improved while reducing the trouble felt by the driver.
[0194] Next, use Fig.17 An example of a method for processing the driver's driving presentation during implementation S113 is described in more detail with reference to a flowchart of FIG.
[0195] In S121, if the HMI output unit 71 has presented the same or similar content in the past, it is determined whether a prescribed time has passed since the last presentation. The prescribed time may be, for example, 1 minute, 10 minutes, or 1 hour. If a positive determination is made in S121, or if the same or similar content has not been presented in the past, the process proceeds to S122. If a negative determination is made in S121, the series of processes is terminated.
[0196] In S122, the HMI device 70 that has received the presentation instruction from the HMI output unit 71 implements the Fig.11 , Fig.12 The teaching is performed by combining HUD and sound as described above. A series of processing ends at S122.
[0197] That is, when the driver's driving is estimated to be highly dangerous, the teaching during driving can be implemented unconditionally, but the teaching can be omitted under specified conditions as in S121 and S122. By suppressing the situation of teaching the same or similar content multiple times in a short period of time, the trouble felt by the driver can be reduced.
[0198] Next, use Fig.18 An example of a method for processing the presentation after the driver has finished driving in S117 will be described in more detail with reference to the flowchart of FIG.
[0199] In S131, the processing system 50 (for example, the HMI output unit 71) reads the information stored in S115 and S116 from the storage destination. This reading may also be realized by sending and receiving information. After the processing of S131, the process moves to S132.
[0200] In S132, the HMI output unit 71 determines whether the target driver's driving action is a repetitive action. If an affirmative determination is made in S132, the process proceeds to S133. If a negative determination is made in S132, the process proceeds to S134.
[0201] In S133, the HMI output unit 71 determines whether the target driver's driving action is a repetitive action. If an affirmative determination is made in S132, the process proceeds to S133. If a negative determination is made in S132, the process proceeds to S134.
[0202] In S133, the HMI output unit 71 determines whether the driving behavior of the target driver is unsafe compared to the driver's usual driving behavior. If a positive determination is made in S133, the process moves to S134. If a negative determination is made in S133, the series of processes ends.
[0203] In S134, at least one of the HMI device 70 and the mobile terminal 91 that has received the presentation instruction from the HMI output unit 71 implements the Fig.13 The teaching by the moving image or the teaching by the report as described above is completed with S134.
[0204] That is, when the driver's driving is estimated to be of medium or high risk, teaching after driving can be performed unconditionally, but teaching can be omitted under specified conditions as in S131 to S134. By suppressing teaching of content that the driver already understands, the trouble felt by the driver can be reduced.
[0205] (Effect)
[0206] The effects of the first embodiment described above will be described below.
[0207] According to the processing system 50 of the first embodiment, information related to the teaching to the driver that can be presented to the driver is output. The rule that serves as the basis for the teaching to the driver is defined by the safety model of the autonomous driving. By having the driver refer to the teaching, it is possible to suppress the driver's driving from being evaluated unfavorably in the relative evaluation of the moving body being autonomously driven. Therefore, the appropriateness of the driver's driving can be improved.
[0208] According to the HMI device 70 and the mobile terminal 91 of the first embodiment, the user interface 70b, 94 presents the presentation content based on the information related to the teaching to the driver obtained from the communication interface 70a, 93. The rule serving as the basis for the teaching to the driver is defined by the safety model of the autonomous driving. By having the driver refer to the teaching, it is possible to suppress the driver's driving from being unfavorably evaluated in the relative evaluation of the moving body being autonomously driven. Therefore, the appropriateness of the driver's driving can be improved.
[0209] According to the first embodiment, since teaching is performed according to the degree of deviation of the driver's driving from the rules, the teaching can be optimized so that the driver can easily follow the rules. Therefore, the appropriateness of the driver's driving can be improved.
[0210] According to the first embodiment, the presentation method of the presentation content for implementing the teaching is determined based on the result of the evaluation of the driver's driving, so that the teaching can be optimized in a way that makes it easy for the driver to follow the rules. Therefore, the appropriateness of the driver's driving can be improved.
[0211] According to the first embodiment, the presentation method of the result based on the evaluation of the driver's driving includes the concept of information amount, so it is possible to implement teaching for the driver to follow the rules while reducing the trouble felt by the driver.
[0212] According to the first embodiment, the presentation method of the result based on the evaluation of the driver's driving includes the concept of presentation timing, so it is possible to implement teaching for the driver to follow the rules at a timing that facilitates the driver's understanding.
[0213] According to the first embodiment, while the driver is driving, the same or similar presentation contents are presented at time intervals of a predetermined time or longer, so that it is possible to implement teaching for the driver to follow the rules while reducing the trouble felt by the driver.
[0214] According to the first embodiment, a presentation content is presented that combines visual information representing a scene encountered by the vehicle 1 due to the driver's driving and auditory information that suggests improvements in driving in the scene. The scene presented with visual information facilitates the driver's understanding of the encountered situation in a short time. At the same time, the persuasiveness of the teaching can be increased by the suggestions shown with auditory information. Therefore, it is possible to achieve teaching that makes it easy for the driver to follow the rules.
[0215] According to the first embodiment, when presenting content on the user interface 70b, 94, information read from the external system 96 is used. It is possible to suppress the HMI device 70 or the mobile terminal 91 from continuously retaining information from the time when the driver drives until teaching, so that teaching can be implemented while saving hardware resources mounted on the HMI device 70 or the mobile terminal 91.
[0216] (Second Embodiment)
[0217] like Fig.19 , Fig. 20 As shown, the second embodiment is a modified example of the first embodiment. The second embodiment will be described mainly focusing on the points that are different from the first embodiment.
[0218] In the second embodiment, the danger level estimation unit 75 predicts a scene that the vehicle 1 may encounter before reaching the destination, and estimates the danger level based on the scene. The danger level estimation unit 75 may also predict a scene instead of a scene. Specifically, the danger level estimation unit 75 predicts the path that the vehicle 1 passes through by the driver based on the road information and destination information acquired through the map DB 44 and V2X. Furthermore, the danger level estimation unit 75 predicts a scene in which the vehicle 1 falls into an unsafe state based on the road information related to the predicted path.
[0219] The scenario of falling into an unsafe state may also refer to a so-called hazardous situation or a scenario with a high possibility of falling into a hazardous situation. The scenario of falling into an unsafe state may also refer to a scenario with a high possibility that the driver deviates from the rules prescribed by the safety model. The scenario that can be predicted by the risk estimation unit 75 is equivalent to a known dangerous scenario.
[0220] The risk level estimation unit 75 may extract a scenario in which the vehicle 1 falls into an unsafe state by determining the similarity between the scenario predicted to be encountered by the vehicle 1 and dangerous scenarios among the concrete scenarios accumulated in the scenario DB 53 .
[0221] The prediction of unsafe conditions in the scene can be implemented under the assumption of reasonably foreseeable behavior of other road users. This assumption can be based on the consideration of rules specified by the safety model. For example, if the information of other vehicles predicted in the scene shows that the other vehicle is a vehicle equipped with an RSS model, the behavior of the other vehicle can be assumed based on the rules of the RSS model.
[0222] The scenario here may include the driver's mental state (such as at least one of the driver's intention and emotion) as a factor for determining an unsafe state. Fig.19 When it is predicted that the vehicle 1 will enter a traffic jam in 5 minutes, as shown in the figure, an anxious state can also be predicted as a mental state of the driver with a high probability of being trapped. Among the scenes that the vehicle 1 is likely to encounter after entering a traffic jam, scenes in which a correlation between an anxious state and an unsafe state is confirmed can be extracted as scenes in which the vehicle 1 falls into an unsafe state.
[0223] In addition, for example, when it is predicted that the vehicle 1 will travel on an unfamiliar road in 10 minutes, a nervous state may be predicted as a mental state of the driver that is likely to be trapped. Among the scenes that the vehicle 1 is likely to encounter while traveling on an unfamiliar road, scenes in which a correlation between a nervous state and an unsafe state is confirmed may be extracted as scenes in which the vehicle 1 is trapped in an unsafe state.
[0224] use Fig. 20An example of a method of estimating the risk level in the second embodiment will be described in detail with reference to a flowchart.
[0225] In S201, the risk level estimation unit 75 predicts a scenario in which the vehicle 1 falls into an unsafe state. After the processing in S201, the process moves to S202.
[0226] In S202, the risk estimation unit 75 determines whether a scenario that leads to an unsafe state is predicted. If an affirmative determination is made in S202, the process proceeds to S204. If a negative determination is made in S202, the process proceeds to S203.
[0227] In S203, the risk level estimation unit 75 estimates that the driver's driving is of low risk. A series of processing ends with S203.
[0228] In S204, the risk level estimation unit 75 estimates that the driver's driving is of high risk. A series of processes are estimated in S204.
[0229] In addition, in this process, the risk level is classified into two levels, but the risk level can also be classified into three or more levels or classified as continuous numerical values according to the predicted scenario. Then, based on such an estimation of the risk level, teaching about route changes, teaching about the driver's mental state, etc. can be implemented.
[0230] According to the second embodiment described above, the scene that is the object of teaching for making the vehicle 1 follow the rules is a scene that is predicted to be encountered by the vehicle 1 due to the driver's driving, and is a scene that is predicted to put the vehicle 1 into an unsafe state. By having the driver refer to the teaching, it is possible to prepare in advance to avoid being put into an unsafe state when encountering the taught scene, so the effect of suppressing the driver's driving from being evaluated unfavorably is significantly improved.
[0231] According to the second embodiment, when it is predicted that the driver's driving deviates from the rules, the presentation content is presented at a presentation timing before the predicted occurrence timing. By having the driver refer to this teaching, it is possible to prepare in advance to avoid the driver's driving deviating from the rules, thereby significantly improving the effect of suppressing the driver's driving from being unfavorably evaluated.
[0232] (Third Embodiment)
[0233] like Fig.21 , Fig. 22 As shown, the third embodiment is a modified example of the first embodiment. The third embodiment will be described mainly focusing on the points different from the first embodiment.
[0234] In the third embodiment, the risk estimation unit 75 estimates the causal relationship between the driver's state and the driving action, and estimates the risk based on the causal relationship. Specifically, the risk estimation unit 75 refers to the value of each parameter in the driving action information. The risk estimation unit 75 estimates the causal relationship between the driver's driving action and the reason why the driver made the driving action as the object based on the value of each parameter. The driving action as the object may be a dangerous driving action (hereinafter, dangerous action).
[0235] Assume that the following data is obtained: Fig.21 As shown, for example, in the driving behavior of the driver of the vehicle 1, the average distance d between the vehicle 1 and other vehicles ahead on a road with a speed limit of 60 km / h is 45 m when the driver's mental state is normal, whereas it is 30 m when the driver is in an anxious state. In this case, the risk estimation unit 75 estimates the causal relationship between the driver's state of "anxious state" and the driving behavior of "shortening the distance between vehicles" for the driver.
[0236] In addition, for example, assume that the following data is obtained: the reaction time t of the driver of the vehicle 1 to the behavior of other road users such as obstacles in the driving action is 0.1s in normal times, while it is 0.8s when drowsiness occurs. In this case, the risk estimation unit 75 estimates the causal relationship between the driver state such as "drowsiness occurs" and the driving action such as "avoidance action delay" for the driver.
[0237] For example, assume that the following data is obtained: the number of pedestrians recognized n in the driving action of the driver of the vehicle 1 is 4 in normal times, while it is 2 in the stress state. In this case, the risk estimation unit 75 estimates the causal relationship between the driver state such as "stress state" and the driving action such as "increase in the number of pedestrians missed".
[0238] Then, when the current driver state is a state that is identified as a cause of the dangerous action in the estimation of the causal relationship, the risk level estimation unit 75 may estimate the risk level to be higher than when the current driver state is not such a state.
[0239] use Fig. 22 An example of a method of estimating a risk level in the third embodiment will be described in detail with reference to a flowchart.
[0240] In S300, the risk level estimation unit 75 estimates the causal relationship between the driver's state and the driver's driving action. After the processing of S300, the process moves to S301.
[0241] In S301, the risk estimation unit 75 determines whether the driver's driving violates the safety envelope based on the information of the driving action. If an affirmative determination is made in S301, the process proceeds to S302. If a negative determination is made in S301, the process proceeds to S305.
[0242] In S302, the risk estimation unit 75 detects the degree of deviation of the driver's driving from the rules and determines whether the deviation is less than a predetermined judgment reference value. In addition, in the case where the deviation cannot be expressed as a quantitative value and is difficult to compare with the judgment reference value, a negative judgment may be made. In the case of a positive judgment in S302, the process moves to S303. In the case of a negative judgment in S303, the process moves to S307.
[0243] In S303, the risk estimation unit 75 determines whether the margin time is longer than a predetermined judgment reference value. If a positive determination is made in S303, the process moves to S304. If a negative determination is made in S303, the process moves to S307. In addition, if the determination content of S303 substantially overlaps with the determination content of S301, the process of S303 may be omitted.
[0244] In S304, the risk estimation unit 75 determines whether the current driver state is a state that causes dangerous behavior based on the estimation of the causal relationship in S300. If an affirmative determination is made in S304, the process proceeds to S307. If a negative determination is made in S304, the process proceeds to S306.
[0245] In S305, the risk level estimation unit 75 estimates that the driver's driving is of low risk. A series of processes are terminated in S305.
[0246] In S306, the risk level estimation unit 75 estimates that the driver's driving is of medium risk level. A series of processing ends with S306.
[0247] In S307, the risk level estimation unit 75 estimates that the driver's driving is of high risk. A series of processing ends with S307.
[0248] Furthermore, the causal relationship between the driver's state and the driver's driving action used in estimating the risk level may not be a causal relationship specific to a specific driver driving the vehicle 1 but may be a causal relationship recognized by general drivers.
[0249] According to the third embodiment described above, the factors causing potential danger are classified according to the causal relationship between the driver's state and the potential danger during the driver's driving. The teaching is provided according to the classification of the factors causing potential danger, so the persuasiveness of the teaching can be improved.
[0250] (Fourth Embodiment)
[0251] like Fig.23 , Fig.24 As shown, the fourth embodiment is a modified example of the first embodiment. The fourth embodiment will be described mainly focusing on the points different from the first embodiment.
[0252] The teaching function in the fourth embodiment is a method dedicated to teaching the driver during driving. When the estimation result of the danger level estimation unit 75 is that the danger level is high, the same teaching of the driver during driving as in the first embodiment is implemented. When the estimation result of the danger level estimation unit 75 is that the danger level is medium, it is determined whether to implement the teaching of the driver during driving based on the comparison of the driving based on the current driver and the driving based on the driving based on the past driver (hereinafter, past driving).
[0253] Here, use Fig.23 An example of a method for implementing a process for presenting information to a driver is described in detail with reference to a flowchart of FIG.
[0254] In S411, the HMI output unit 71 determines whether the risk level of the driver's driving is estimated to be medium or higher, that is, medium or high risk. If a positive determination is made in S411, the process moves to S412. If a negative determination is made in S411, the series of processes ends.
[0255] In S412, the HMI output unit 71 determines whether the driver's driving is estimated to be high-risk. If an affirmative determination is made in S412, the process proceeds to S413. If a negative determination is made in S412, the process proceeds to S414.
[0256] In S413, the HMI output unit 71 and the HMI device 70 perform a presentation process during the driver's driving. The presentation process may be performed in conjunction with Fig.17 The same is true for S121 and S122 shown. After the processing of S413, the process moves to S414.
[0257] In S414, the information required for presentation is saved. This information may be stored in the recording device 55 as information of the vehicle 1 alone. This information may also be stored in the driving information DB 98 in the external system 96 in a manner of being collected together with information of a plurality of vehicles. A series of processing ends with S414.
[0258] In S415, the HMI output unit 71 and the HMI device 70 perform a presentation process based on the comparison result of the past travel. A series of processes ends with S415.
[0259] Next, use Fig.24An example of a processing method for implementing S415 of presenting the comparison result based on past travel is described in more detail with reference to the flowchart of FIG.
[0260] In S421, the processing system 50 (for example, the HMI output unit 71) reads the past driving action information from the storage destination among the information required for presentation stored in S414. The past driving action information includes information related to past driving. This reading can also be achieved by sending and receiving information. After the processing of S421, it transfers to S422.
[0261] In S422, the processing system 50 (e.g., the HMI output unit 71) compares the driving based on the current driver's driving with the information about the past driving acquired in S421. The processing system 50 (e.g., the HMI output unit 71) determines whether the current driving is more likely to lead to dangerous driving in the future than the usual (i.e., past) driving. If a positive determination is made in S422, the process moves to S423. If a negative determination is made in S422, the process moves to S424.
[0262] In S423, the HMI output unit 71 and the HMI device 70 perform a presentation process during the driver's driving. The presentation process may be performed in conjunction with Fig.17 The same is true for S121 and S122 shown. After the processing of S423, the process moves to S424.
[0263] In S424, similarly to S414, information required for presentation is stored, and a series of processing ends with S424.
[0264] According to the fourth embodiment described above, the presentation method of the presentation content is determined based on the comparison between the driver's current driving and past driving, so that appropriate teaching can be performed according to the driver's state, the change in driving ability over time, etc.
[0265] (Other embodiments)
[0266] Although a plurality of embodiments have been described above, the present disclosure is not limited to these embodiments and can be applied to various embodiments and combinations within a scope not departing from the gist of the present disclosure.
[0267] As a modification example 1, the processing system that performs the processing of the risk estimation unit 75 and the HMI output unit 71 in the evaluation function and the teaching function may be another system separate from the driving system 2. The processing system may be mounted on the vehicle 1 or may not be mounted on the vehicle 1. The processing system may be provided in the HMI device 70 or the mobile terminal 91, or may be provided in an external system 96 such as a remote center.
[0268] As a second modification, the processing system that executes the processing of the risk level estimation unit 75 and the HMI output unit 71 in the evaluation function and the teaching function can also be applied to a manually driven vehicle that cannot execute automatic driving.
[0269] As a third modification, the processing system that executes the processing of the risk estimation unit 75 and the HMI output unit 71 in the evaluation function and the teaching function can also be applied to a vehicle that does not have a V2X function. In this case, the teaching can be performed exclusively by the vehicle-mounted HMI device 70.
[0270] The control unit and method described in the present disclosure may also be implemented by a special-purpose computer that is programmed to execute one or more functions embodied by a computer program. Alternatively, the device and method described in the present disclosure may also be implemented by a dedicated hardware logic circuit. Alternatively, the device and method described in the present disclosure may also be implemented by one or more special-purpose computers that are composed of a combination of a processor that executes a computer program and one or more hardware logic circuits. In addition, the computer program may also be stored as an instruction executed by a computer in a non-temporary tangible recording medium that can be read by a computer.
[0271] (Explanation of terms)
[0272] The following describes terms related to the present disclosure. This description is included in the embodiments of the present disclosure.
[0273] A road user may be a person who uses a road including a sidewalk and other adjacent spaces. A road user may also include pedestrians, cyclists, other VRUs, and vehicles (e.g., cars driven by people, vehicles equipped with autonomous driving systems). A road user may be a road user who is on or adjacent to an active road for the purpose of moving from one location to another.
[0274] A dynamic driving task (DDT) may be a real-time operational and tactical function used to operate a vehicle in traffic.
[0275] An automated driving system may be an integrated hardware and software that can continuously perform overall DDT regardless of whether it is limited to a specific operating design area.
[0276] SOTIF (safety of the intended functionality) can mean the absence of unreasonable risks due to inadequacy of the intended functionality or its installed functionality.
[0277] A driving policy is a strategy and rules that define control actions at the vehicle level.
[0278] A scenario can be a description of the temporal relationship between several scenes within a series of scenes, including goals and values under specific conditions affected by actions and events. A scenario can be a description of a continuous time sequence of activities that integrates the vehicle as the subject, all its external environment, and their interactions in the process of performing a specific driving task.
[0279] A triggering condition may be a specific condition of a scenario that functions as a trigger for a subsequent system response that results in a failure to prevent, detect, or mitigate a dangerous behavior or reasonably foreseeable indirect misuse.
[0280] Takeover may be the handover of driving tasks between the automated driving system and the driver.
[0281] Safety-related models can be representations of safety-related aspects of driving behavior based on assumptions about the reasonably foreseeable behavior of other road users. Safety-related models can also be on-board or off-board safety validation or safety analysis devices, mathematical models, more conceptual sets of rules, sets of scenario-based behaviors, or a combination thereof.
[0282] The formal model may be a model expressed in a formal expression used in system performance verification.
[0283] A safety envelope may be a set of restrictions and conditions designed for the (autonomous) driving system to act as an object of restriction or control in order to maintain operation within a tolerable level of risk. A safety envelope may be a general concept that can be used to address all principles that a driving strategy can be based on, according to which the host vehicle acting by the (autonomous) driving system can have one or more boundaries around it.
[0284] Reaction time can be the time it takes for a road user to perceive a specific stimulus and start to execute a response (braking, steering, accelerating, stopping, etc.) in a given scenario.
[0285] A hazardous situation can be an increased risk for a potential violation of the safety envelope or can represent an increased level of risk that exists in the DDT.
[0286] (Disclosure of technical ideas)
[0287] This specification discloses multiple technical ideas recorded in the following multiple items. Some items are sometimes recorded in a multiple dependent form (a multiple dependent form) in which the previous items are selectively cited in the subsequent items. These items recorded in multiple dependent forms define multiple technical ideas.
[0288] <Technical Thought 1>
[0289] A processing system comprises at least one processor (51b) for executing processing for presenting information to a driver of a mobile body (1).
[0290] The processor performs the following processing:
[0291] evaluating the driver's driving using rules specified by a safety model for automated driving; and
[0292] Based on the evaluation, information related to the teaching for compliance with the rule is output in a form that can be presented to the driver.
[0293] <Technical Thoughts 2>
[0294] The processing system according to technical idea 1, wherein:
[0295] The processor further performs the following processing: detecting the degree of deviation of the driver's driving relative to the rule,
[0296] In the output process, the information is output according to the magnitude of the deviation.
[0297] <Technical Thought 3>
[0298] The processing system according to technical idea 1 or 2, wherein:
[0299] The processor also performs the following processing:
[0300] identifying a status of the driver;
[0301] extracting a causal relationship between the driver's state and potential danger during the driver's driving; and
[0302] Classify the factors causing potential dangers according to the causal relationship.
[0303] In the process of outputting the teaching, the teaching corresponding to the classification of the generation factor is output.
[0304] <Technical Thought 4>
[0305] The processing system according to any one of technical concepts 1 to 3, wherein:
[0306] The processor further performs the following processing: predicting a scenario that is predicted to be encountered by the mobile body due to the driving of the driver and in which the mobile body falls into an unsafe state,
[0307] The teaching is a teaching for causing the moving body to follow the rule in a scenario where the moving body falls into an unsafe state.
[0308] <Technical Thought 5>
[0309] The processing system according to any one of technical concepts 1 to 4, wherein:
[0310] The method further includes a process of determining a presentation method of presentation content for implementing the teaching based on a result of the evaluation of the driver's driving.
[0311] <Technical Thought 6>
[0312] The processing system according to technical idea 5, wherein:
[0313] The presentation manner of the presented content includes the information amount of the presented content.
[0314] <Technical Thought 7>
[0315] The processing system according to technical idea 5 or 6, wherein:
[0316] The presentation method of the presentation content includes the presentation timing of the presentation content.
[0317] <Technical Thought 8>
[0318] The processing system according to technical idea 7, wherein:
[0319] When the presentation timing is during driving of the driver, the same or similar presentation content is presented at a time interval equal to or longer than a predetermined time.
[0320] <Technical Thought 9>
[0321] The processing system according to technical idea 7 or 8, wherein:
[0322] When it is predicted that the driving of the driver deviates from the rule, the presentation content is presented at the presentation timing before the predicted occurrence timing.
[0323] <Technical Thought 10>
[0324] The processing system according to any one of technical concepts 5 to 9, wherein:
[0325] The presentation method of the presentation content is determined based on a comparison between the driver's current driving and past driving.
[0326] <Technical Thought 11>
[0327] The processing system according to any one of technical concepts 1 to 10, wherein:
[0328] In the output process, the information is output when it is evaluated that the rule is violated.
[0329] <Technical Thoughts 12>
[0330] An information presentation device, for presenting information to a user, comprising:
[0331] A communication interface (70a, 93) configured to communicate with a processing system (50) that performs processing related to a mobile body (1), and to obtain from the processing system information related to teaching for causing a driver of the mobile body to follow rules defined by a safety model for autonomous driving; and
[0332] The user interface (70b, 94) is configured to be able to present content related to teaching for following the rule based on the information.
[0333] <Technical Thought 13>
[0334] The information presentation device according to technical idea 12, wherein:
[0335] The presentation content includes content that combines visual information representing a scene encountered by the mobile body due to driving by the driver and auditory information suggesting improvements in driving in the scene.
[0336] <Technical Thought 14>
[0337] The information presentation device according to technical idea 12 or 13, wherein:
[0338] The communication interface is configured to be able to communicate with an external system (96) provided outside the mobile body.
[0339] The user interface is configured to be able to present the presentation content using the information read out from the external system.
[0340] <Technical Thought 15>
[0341] A recording device for recording information related to a driver of a mobile object (1).
[0342] In at least one storage medium (55a), the driving action of the driver and the comparison result of the driving action with the rule specified by the safety model of the autonomous driving or the benchmark based on the rule are recorded in association.
[0343] <Technical Thoughts 16>
[0344] The recording device according to technical idea 14, wherein:
[0345] In the storage medium, an estimation result regarding the driver state of the mobile object is further recorded in association.
[0346] <Technical Thoughts 17>
[0347] A processing method for performing a process for presenting information to a driver of a mobile body (1),
[0348] At least one processor (51b) is caused to perform the following processing:
[0349] evaluating the driver's driving using rules specified by a safety model for automated driving; and
[0350] When an assessment is made that the rule is violated, information related to instruction for following the rule is output in a form that can be presented to the driver.
[0351] <Technical Thoughts 18>
[0352] A storage medium is configured to be readable by at least one processor (51b), and stores a program for causing the processor to execute the following processing:
[0353] evaluating the driving of the driver of the mobile object using rules defined by the safety model of the autonomous driving; and
[0354] When an assessment is made that the rule is violated, information related to instruction for following the rule is output in a form that can be presented to the driver.
[0355] <Technical Thoughts 19>
[0356] A program causes at least one processor (51b) to execute the following processing:
[0357] evaluating the driving of the driver of the mobile object using rules defined by the safety model of the autonomous driving; and
[0358] When an assessment is made that the rule is violated, information related to instruction for following the rule is output in a form that can be presented to the driver.
[0359] <Technical Thoughts 20>
[0360] An information presentation method presents driving-related instruction to a driver, comprising:
[0361] Acquiring information for evaluating the driver's driving from at least one of an external environment and an internal environment of the vehicle using a sensor;
[0362] At least one processor calculates, based on the acquired information, a degree of deviation of the driver's driving relative to a rule specified by at least one safety model of an RSS (Responsibility-Sensitive Safety) model and an SFF (Safety Force Field) model of autonomous driving and stored in at least one recording medium;
[0363] Determining, by at least one processor, whether the calculated deviation exceeds a specified threshold;
[0364] When it is determined by at least one processor that the degree of deviation exceeds the threshold, outputting a signal for presenting a teaching for the driver to follow the rule to an information presentation device; and
[0365] The information presentation device receives the signal to present the teaching to the driver.
[0366] <Technical Thoughts 21>
[0367] An information presentation system presents driving-related instructions to a driver, comprising:
[0368] A sensor (40) is provided in the vehicle (1) and acquires information for evaluating the driving of the driver from at least one of the external environment and the internal environment of the vehicle;
[0369] An on-vehicle processing system (50) having at least one processor (51a) and at least one recording medium (51b); and
[0370] An information presentation device (70) is arranged in the vehicle and presents the teaching to the driver.
[0371] The at least one recording medium stores rules defined by at least one safety model of an RSS (Responsibility-Sensitive Safety) model and an SFF (Safety Force Field) model for autonomous driving,
[0372] The at least one processor calculates the degree of deviation of the driver's driving from the rule based on the information acquired by the sensor,
[0373] The at least one processor determines whether the calculated deviation exceeds a specified threshold,
[0374] When the at least one processor determines that the deviation exceeds the threshold, the at least one processor outputs a signal to the information presentation device for presenting a teaching for the driver to follow the rule.
[0375] The information presentation device is configured to receive the signal and present the teaching to the driver.
Claims
1. A processing system comprising at least one processor (51b) for executing processing for presenting information to a driver of a mobile body (1), characterized in that: The processor performs the following processing: evaluating the driver's driving using rules specified by a safety model for automated driving; and Based on the evaluation, information related to the teaching for compliance with the rule is output in a form that can be presented to the driver.
2. The processing system according to claim 1, characterized in that The processor further performs the following processing: detecting the degree of deviation of the driver's driving relative to the rule, In the output process, the information is output according to the magnitude of the deviation.
3. The processing system according to claim 1 or 2, characterized in that: The processor also performs the following processing: identifying a status of the driver; Extracting the causal relationship between the driver's state and the potential danger during the driver's driving; as well as Classify the factors causing potential dangers according to the causal relationship. In the process of outputting the teaching, the teaching corresponding to the classification of the generation factor is output.
4. The processing system according to claim 1 or 2, characterized in that: The processor further performs the following processing: predicting a scenario that is predicted to be encountered by the mobile body due to the driving of the driver and in which the mobile body falls into an unsafe state, The teaching is a teaching for causing the moving body to follow the rule in a scenario where the moving body falls into an unsafe state.
5. The processing system according to claim 1, characterized in that The method further includes a process of determining a presentation method of presentation content for implementing the teaching based on a result of the evaluation of the driver's driving.
6. The processing system according to claim 5, characterized in that The presentation manner of the presented content includes the information amount of the presented content.
7. The processing system according to claim 5, characterized in that The presentation method of the presentation content includes the presentation timing of the presentation content.
8. The processing system according to claim 7, characterized in that When the presentation timing is during driving of the driver, the same or similar presentation content is presented at a time interval equal to or longer than a predetermined time.
9. The processing system according to claim 7 or 8, characterized in that: When it is predicted that the driving of the driver deviates from the rule, the presentation content is presented at the presentation timing before the predicted occurrence timing.
10. The processing system according to any one of claims 5 to 8, characterized in that The presentation method of the presentation content is determined based on a comparison between the driver's current driving and past driving.
11. The processing system according to claim 1 or 2, characterized in that: In the output process, the information is output when it is evaluated that the rule is violated.
12. An information presentation device for presenting information to a user, characterized in that: have: a communication interface (70a, 93) configured to communicate with a processing system (50) that performs processing related to a mobile body (1), and to obtain from the processing system information related to teaching for causing a driver of the mobile body to follow rules defined by a safety model for autonomous driving; as well as The user interface (70b, 94) is configured to be able to present content related to teaching for following the rule based on the information.
13. The information presentation device according to claim 12, characterized in that: The presentation content includes content that combines visual information representing a scene encountered by the mobile body due to driving by the driver and auditory information suggesting improvements in driving in the scene.
14. The information presentation device according to claim 12 or 13, characterized in that: The communication interface is configured to be able to communicate with an external system (96) provided outside the mobile body. The user interface is configured to be able to present the presentation content using the information read out from the external system.
