Operation support system, operation support program, operation support method

Abstract

To provide an operation support system securing security and safety in interaction between a host vehicle and other road users.SOLUTION: A processor of an operation support system which supports an operation of a host vehicle is configured to acquire virtual animation data Dm virtually representing the host vehicle as if the host vehicle is viewed from other road users existing in the surrounding of the host vehicle, and control the host vehicle in an operation behavior according to the virtual animation data Dm.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present disclosure relates to a driving assistance technology that assists in driving a host vehicle. [Background technology]

[0002] The technology disclosed in Patent Document 1 determines the driving behavior of the host vehicle according to the collision risk with other road users present in the external world of the host vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2021 / 0009121 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology disclosed in Patent Document 1, although a collision risk is determined using a mathematical model between the host vehicle and other road users, the mathematical model is constructed on the premise of the motion trajectory of the other road users predicted from the viewpoint of the host vehicle. Therefore, the driving behavior of the host vehicle is not necessarily safe and secure from the viewpoint of the other road users, and there is a concern that the mathematical model may invite interactions between the host vehicle and other road users that are not anticipated.

[0005] An object of the present disclosure is to provide a driving assistance system that ensures safety and security in interactions between a host vehicle and other road users. Another object of the present disclosure is to provide a driving assistance program that ensures safety and security in interactions between a host vehicle and other road users. Yet another object of the present disclosure is to provide a driving assistance method that ensures safety and security in interactions between a host vehicle and other road users. [Means for solving the problem]

[0006] The technical means of the present disclosure for solving the problems will be described below. Note that the claims and the reference characters in parentheses in this section indicate the corresponding relationship with the specific means described in the embodiments described later in detail, and do not limit the technical scope of the present disclosure.

[0007] A first aspect of the present disclosure is A driving assistance system having a processor (12) for assisting driving of a host vehicle (2), The processor Acquiring virtual video data (Dm) representing the host vehicle virtually as seen from the viewpoint of another road user (3) present in the external world of the host vehicle; and controlling the host vehicle to perform a driving behavior in accordance with the virtual video data.

[0008] A second aspect of the present disclosure is A driving assistance program stored in a storage medium (10) for assisting driving of a host vehicle (2), the driving assistance program including instructions to be executed by a processor (12), Acquiring virtual video data (Dm) representing the host vehicle virtually as seen from the viewpoint of another road user (3) present in the external world of the host vehicle; and controlling the host vehicle to perform a driving behavior in accordance with the virtual video data.

[0009] A third aspect of the present disclosure is A driving assistance method executed by a processor (12) for assisting driving of a host vehicle (2), comprising: Acquiring virtual video data (Dm) representing the host vehicle virtually as seen from the viewpoint of another road user (3) present in the external world of the host vehicle; and controlling the host vehicle to perform driving actions according to the virtual video data.

[0010] In these first to third aspects, virtual video data is acquired that virtually represents the host vehicle as seen from the viewpoint of another road user present in the external world of the host vehicle. Thus, according to the first to third aspects, the host vehicle is controlled to drive in accordance with the virtual video data from the viewpoint of the other road user. This allows the viewpoint of the other road user to be directly reflected in the driving behavior of the host vehicle, making it possible to ensure safety and security in the interaction between the host vehicle and the other road user. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a physical configuration of a driving assistance system according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing a traveling environment of a host vehicle to which the first embodiment is applied. [Diagram 3] 1 is a block diagram showing a functional configuration of a driving assistance system according to a first embodiment. [Figure 4] FIG. 2 is a schematic diagram showing a display state of virtual video data according to the first embodiment. [Diagram 5] 5 is a characteristics table showing display conditions for displaying virtual moving image data according to the first embodiment. [Figure 6] FIG. 4 is a schematic diagram for explaining arbitration of operation control patterns according to the first embodiment. [Figure 7] FIG. 4 is a schematic diagram for explaining arbitration of operation control patterns according to the first embodiment. [Figure 8] FIG. 4 is a schematic diagram for explaining arbitration of operation control patterns according to the first embodiment. [Figure 9] FIG. 4 is a schematic diagram for explaining arbitration of operation control patterns according to the first embodiment. [Figure 10] FIG. 4 is a schematic diagram for explaining arbitration of operation control patterns according to the first embodiment. [Figure 11] FIG. 4 is a schematic diagram for explaining arbitration of operation control patterns according to the first embodiment. [Figure 12]FIG. 4 is a schematic diagram for explaining arbitration of operation control patterns according to the first embodiment. [Figure 13] 4 is a flowchart showing a driving assistance process according to the first embodiment. [Figure 14] FIG. 11 is a block diagram showing a functional configuration of a driving assistance system according to a second embodiment. [Figure 15] FIG. 11 is a schematic diagram for explaining arbitration of an operation control pattern and an operation control parameter according to a second embodiment. [Figure 16] FIG. 11 is a schematic diagram for explaining arbitration of an operation control pattern and an operation control parameter according to a second embodiment. [Figure 17] FIG. 11 is a schematic diagram for explaining arbitration of an operation control pattern and an operation control parameter according to a second embodiment. [Figure 18] FIG. 11 is a schematic diagram for explaining arbitration of an operation control pattern and an operation control parameter according to a second embodiment. [Figure 19] FIG. 11 is a schematic diagram for explaining arbitration of an operation control pattern and an operation control parameter according to a second embodiment. [Figure 20] FIG. 11 is a schematic diagram for explaining arbitration of an operation control pattern and an operation control parameter according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. In addition, by assigning the same reference numerals to corresponding components in each embodiment, duplicated descriptions may be omitted. In addition, when only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other parts of the configuration. Furthermore, in addition to the combination of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments can be partially combined together even if not explicitly stated, as long as there is no particular problem with the combination.

[0013] The driving assistance system 1 of the first embodiment shown in FIG. 1 assists the driving of a host vehicle 2. At least a part of the driving assistance system 1 is mounted on the host vehicle 2. The host vehicle 2 to which the driving assistance system 1 is applied may be capable of achieving a level of automated driving, as defined in, for example, SAE J3016, at which there is a manual driving assistance task that assists the operator in performing manual driving operations, together with an automated driving task. Such a host vehicle 2 may be a road user, such as a car or a truck, and may be referred to as an ego-vehicle. As described above, the driving assistance system 1, as the operator of the host vehicle 2, targets a driver who is on board the host vehicle 2 and can perform manual driving operations as a driving assistance target.

[0014] As shown in FIG. 2, in the driving environment in which the host vehicle 2 drives, a traffic scene in which other road users 3 other than the host vehicle 2 exist is assumed. The other road users 3 include non-vulnerable road users and vulnerable road users according to their vulnerability. A non-vulnerable road user is at least one type of moving body on which a human occupants rides, such as a car, a truck, a motorcycle, and a bicycle. A vulnerable user is, for example, a pedestrian. Such other road users 3 may be in either a stopped state or a moving state in the assumed traffic scene.

[0015] 1, a host vehicle 2 is equipped with an actuator system 4, a sensor system 5, a communication system 6, a map database (DB) 7, and an information presentation system 8, together with at least a part of a driving assistance system 1. However, FIG 1 representatively shows an example in which all of the driving assistance system 1 implemented in the form of a processing circuit (e.g., a processing ECU, etc.) or a semiconductor device (e.g., a semiconductor chip, etc.) is equipped in the host vehicle 2.

[0016] The actuator system 4 is configured to be capable of controlling the driving behavior of the host vehicle 2 based on a control command given from the driving assistance system 1. The actuator system 4 may be at least one type of power train actuator, such as an internal combustion engine or a motor-generator motor. The actuator system 4 may be at least one type of braking actuator, such as a brake unit. The actuator system 4 may be at least one type of steering actuator, such as a power steering unit.

[0017] The sensor system 5 senses the external and internal environments of the host vehicle 2 to obtain sensing information that can be used in the driving assistance system 1. For this purpose, the sensor system 5 includes an external sensor 50 and an internal sensor 52.

[0018] The external sensor 50 senses targets including other road users 3 as targets existing in the external world of the host vehicle 2. The target sensing type external sensor 50 is at least one type of sensor among, for example, an image sensor (i.e., an on-board camera), a LiDAR (light detection and ranging / laser imaging detection and ranging), a laser sensor, a millimeter wave sensor, and a sonar sensor. The target sensing type external sensor 50 may be implemented by combining a plurality of types of sensors so as to be capable of sensing the front, side, and rear directions of the host vehicle 2.

[0019] The internal sensor 52 senses a specific motion physical quantity related to vehicle motion in the internal world of the host vehicle 2. The internal sensor 52 of the motion sensing type is at least one of, for example, a speed sensor, an acceleration sensor, a gyro sensor, etc. The internal sensor 52 may sense the operation or state of an occupant including a driver as an occupant riding in the internal world of the host vehicle 2. The internal sensor 52 of the occupant sensing type is at least one of, for example, a steering angle sensor, a steering torque sensor, a brake pedal sensor, an accelerator pedal sensor, a shift sensor, an occupant camera, a steering switch, a biological sensor, a seating sensor, an in-vehicle device switch, etc.

[0020] The communication system 6 acquires communication information available in the driving assistance system 1 by wireless communication. The communication system 6 may receive a positioning signal from a satellite of a global navigation satellite system (GNSS) existing in the external world of the host vehicle 2. The positioning type communication system 6 is, for example, a GNSS receiver. The communication system 6 may transmit and receive a communication signal between the host vehicle 2 and a V2X system existing in the external world of the host vehicle 2. The V2X communication type communication system 6 is, for example, at least one of a dedicated short range communications (DSRC) communication device and a cellular V2X (C-V2X) communication device. The communication system 6 may transmit and receive a communication signal between the host vehicle 2 and a mobile terminal existing in the internal world of the host vehicle 2. The terminal communication type communication system 6 is, for example, at least one of a Bluetooth (registered trademark) device, a Wi-Fi (registered trademark) device, and an infrared communication device.

[0021] The map DB 7 stores map information available in the driving assistance system 1. The map DB 7 includes at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium. The map DB 7 may be a DB of a locator that estimates the self-position of the host vehicle 2. The map DB may be a DB of a navigation unit that navigates the driving route of the host vehicle 2. The map DB 7 may be constructed by combining multiple types of DBs.

[0022] The map DB 7 acquires and stores the latest map information, for example, by V2X communication with an external center via the communication system 6. The map information is converted into two-dimensional or three-dimensional data as information representing the external environment in which the host vehicle 2 is traveling. Digital data of a high-precision map may be adopted as the three-dimensional map information.

[0023] The map information includes road information representing at least one of the following: the position, the shape, and the road surface condition of a road structure. The road information included in the map information may include still images of the road. The road information included in the map information may include a plurality of computer graphics (CG) images of the road drawn in two or three dimensions from a plurality of viewpoints. The map information may include structure information representing at least one of the following: the position, the shape, and the like of buildings and traffic lights facing the road. The map information may include marking information representing at least one of the following: the position, the shape, and the like of signs and dividing lines attached to the road.

[0024] The information presentation system 8 presents notification information to occupants including the driver of the host vehicle 2. The information presentation system 8 presents the notification information by stimulating the vision of the occupants in the host vehicle 2. The information presentation system 8 of the visual information presentation type is at least one of an in-vehicle monitor, a HUD (head-up display), a combination meter, a navigation unit, an illumination unit, and the like.

[0025] The information presentation system 8 may present the notification information by stimulating the hearing of the occupant. The information presentation system 8 of the auditory information presentation type is, for example, at least one of a speaker, a buzzer, a vibration unit, etc. The information presentation system 8 may present the notification information by stimulating the cutaneous sense of the occupant. The information presentation system 8 of the cutaneous sense information presentation type is, for example, at least one of a vibration unit, a reaction force unit, an air conditioning unit, etc.

[0026] The driving assistance system 1 is connected to an actuator system 4, a sensor system 5, a communication system 6, a map DB 7, and an information presentation system 8 via at least one of, for example, a LAN (local area network), a wire harness, an internal bus, and a wireless communication line. The driving assistance system 1 is configured to include at least one dedicated computer.

[0027] The dedicated computer constituting the driving assistance system 1 may be an integrated ECU (electronic control unit) that integrates the driving control of the host vehicle 2. The dedicated computer constituting the driving assistance system 1 may be a sensing ECU that processes sensing information in the driving control of the host vehicle 2. The dedicated computer constituting the driving assistance system 1 may be a recognition ECU that recognizes the outside world in the driving control of the host vehicle 2. The dedicated computer constituting the driving assistance system 1 may be a locator ECU that estimates the self-position of the host vehicle 2.

[0028] The dedicated computer constituting the driving assistance system 1 may be a planning ECU that plans driving control of the host vehicle 2. The dedicated computer constituting the driving assistance system 1 may be a navigation ECU that navigates a driving route in the driving control of the host vehicle 2. The dedicated computer constituting the driving assistance system 1 may be an actuator ECU that controls the actuator system 4 as the driving control of the host vehicle 2.

[0029] The dedicated computer constituting the driving assistance system 1 may be an information management ECU that controls the information presentation system 8 as driving control of the host vehicle 2. The dedicated computer constituting the driving assistance system 1 may be at least one external computer that constitutes, for example, an external center or a mobile terminal capable of communicating via the communication system 6.

[0030] The dedicated computer constituting the driving assistance system 1 has at least one memory 10 and one processor 12. The memory 10 is at least one type of non-transitory tangible storage medium, such as a semiconductor memory, a magnetic medium, or an optical medium, that non-temporarily stores computer-readable programs and data. The processor 12 includes at least one type of core, such as a central processing unit (CPU), a graphics processing unit (GPU), or a reduced instruction set computer (RISC)-CPU.

[0031] The processor 12 executes a plurality of instructions included in a driving assistance program stored as software in the memory 10. In this way, the driving assistance system 1 constructs a plurality of functional blocks for performing driving assistance processing for the host vehicle 2. The functional blocks thus constructed by the driving assistance system 1 include a recognition block 100, a planning block 120, and a control block 140 shown in FIG.

[0032] The recognition block 100 acquires sensing information from the sensor system 5. The recognition block 100 acquires communication information from the communication system 6. The recognition block 100 acquires map information from the map DB 7. The planning block 120 acquires control data of the host vehicle 2 related to past driving behavior from the control block 140. The recognition block 100 processes these acquired information individually and then fuses them to generate recognition data that recognizes the external and internal environments of the host vehicle 2. Therefore, the recognition data generated by the recognition block 100 represents the state of the external and internal environments recognized for each traffic scene in which the host vehicle 2 travels.

[0033] Specifically, the recognition block 100 generates recognition data by recognizing targets including other road users 3, obstacles, and structures in the external world of the host vehicle 2. The recognition data of a type related to a target may represent at least one type of motion physical quantity, such as a separation distance, a motion direction, a relative speed, and a relative acceleration. The recognition data of a type related to a target may represent a classification of targets clustered based on such motion physical quantities.

[0034] The recognition block 100 may generate the recognition data by recognizing markings associated with the road on which the host vehicle 2 travels. The recognition data for a type of marking may represent at least one type of marking state, such as a sign, a lane marking, a traffic light, etc. The recognition data for the marking may represent a traffic rule recognized from the marking state.

[0035] The recognition block 100 may generate recognition data by recognizing roads on which the host vehicle 2 will currently and in the future travel. The recognition data of a type related to a road may represent at least one type of road condition, such as a road surface, a lane, a road edge, and a free space.

[0036] The recognition block 100 may generate recognition data by localization that recognizes the self-state including the self-position of the host vehicle 2. The recognition data of the type related to the self-state may further include at least one type of static performance data of the external sensor 50 (particularly the image sensor) in the host vehicle 2, such as mounting coordinates, sensing range, sensing resolution, and sensing accuracy. The recognition block 100 may generate update information of map information related to the road on which the host vehicle 2 is traveling at the same time as the recognition data of the type related to the self-state, and feed back the update information to the map DB 7.

[0037] The recognition block 100 may generate recognition data by recognizing a manual driving operation of a driver with respect to the host vehicle 2. The recognition data of a type related to manual driving steering may represent at least one type of operation parameter corresponding to the driver operation, such as a steering angle, a steering torque, a brake pedal operation amount, an accelerator pedal operation amount, and a shift operation position.

[0038] The recognition block 100 identifies the notable user 3A from among the other road users 3 shown in Fig. 2 based on at least the recognition data of the target and the vehicle's own state among the multiple types of recognition data as described above. At this time, the other road users 3 existing in an interaction area Ia within a set distance range in the traveling direction of the host vehicle 2 are identified as the notable user 3A. The interaction area Ia refers to an area where an interaction between the host vehicle 2 and the other road user 3 is expected to occur due to the host vehicle 2 and the other road user 3 approaching each other relatively.

[0039] By being present in such an interaction area Ia, the noted user 3A is identified as an other road user 3 expected to come close relative to the host vehicle 2. Thus, the relative approach between the host vehicle 2 and the other road user 3 is recognized based on, for example, position coordinates and speed vectors as state values ​​of the host vehicle 2 and the other road user 3. At this time, it is preferable to also use performance data of the external sensor 50 (particularly the mounting coordinates of the image sensor) in recognizing the relative approach as state values ​​of the host vehicle 2 represented by the recognition data of its own state.

[0040] At least one of the other road users 3 that are present in the interaction area Ia and are in a relatively close relationship with the host vehicle 2 is selected as the noted user 3A. It is preferable that the other road user 3 that is present at the closest distance from the host vehicle 2 in the interaction area Ia is selected as the noted user 3A. However, if there are multiple other road users 3 in the interaction area Ia, the other road user 3 that is present at the closest distance to the host vehicle 2 may be selected as the noted user 3A from among other road users 3 (e.g., pedestrians) that have a high degree of vulnerability. Therefore, in the following description, the closest distance is understood to include a concept that takes into account the vulnerability of the other road users 3, as well as a concept that is not taken into consideration.

[0041] A plurality of other road users 3 that are present in the interaction area Ia and are in a relatively close relationship with the host vehicle 2 may be selected as the attention user 3A. However, the attention user 3A that is the display target of the virtual video data Dm in the recognition block 100 described in detail later may be limited to a single other road user 3 that is present at the closest distance. On the other hand, as long as at least the other road user 3 at the closest distance corresponds to the attention user 3A that is the interaction prediction target and the arbitration target in the planning block 120 described in detail later, some or all of the other road users 3 that are not at the closest distance may also correspond to the attention user 3A.

[0042] The recognition block 100 shown in Fig. 3 acquires, as a type of recognition data, virtual video data Dm that virtually represents the host vehicle 2 as seen from the viewpoint of the other road user 3 identified as the noted user 3A. At this time, the virtual video data Dm is generated so as to be reflected in the driving behavior of the host vehicle 2 by being provided for interaction prediction and driving control in the planning block 120, which will be described in detail later. Therefore, as shown in Fig. 4, the virtual video data Dm that represents the host vehicle 2 is generated periodically and continuously for each video frame based on the latest recognition data regarding at least targets and the vehicle's own state.

[0043] To generate the virtual video data Dm, a plurality of still images of the host vehicle 2 captured from a plurality of viewpoints and stored in the memory 10 may be read from the memory 10 as vehicle image portions Dh representing the host vehicle 2 as shown in Fig. 4. To generate the virtual video data Dm, a plurality of CG images of the host vehicle 2 rendered in two or three dimensions from a plurality of viewpoints and stored in the memory 10 may be read from the memory 10 as vehicle image portions Dh.

[0044] To generate the virtual video data Dm, among the road information included in the map information, a plurality of still images of the road photographed from a plurality of viewpoints and stored in the map DB7 may be read from the map DB7 as a background image portion Db that forms the background of the vehicle image portion Dh as shown in Fig. 4. To generate the virtual video data Dm, among the road information included in the map information, a plurality of CG images of the road drawn two-dimensionally or three-dimensionally from a plurality of viewpoints and stored in the map DB7 may be read from the map DB7 as the background image portion Db. The data of the background image portion Db may be stored in the memory 10 instead of the map DB7, and thus may be read from the memory 10.

[0045] In generating the virtual video data Dm, a periodic process may be repeated in which the vehicle image portion Dh and the background image portion Db are subjected to viewpoint transformation using a viewpoint transformation algorithm, such as projective transformation or affine transformation, stored in the memory 10 to generate image data for one frame of the virtual video data Dm. Each time such a periodic process is performed, the viewpoint transformation algorithm is substituted with state values ​​related to the host vehicle 2 and the noted user 3A, such as position coordinates and a velocity vector, and / or viewpoint coordinates calculated from the state values ​​of the noted user 3A relative to the host vehicle 2. At this time, the viewpoint transformation algorithm may also be substituted with performance data (particularly the mounting coordinates of the image sensor) of the external sensor 50 that sensed the noted user 3A, among the recognition data of the self state used to identify the noted user 3A, as state values ​​of the host vehicle 2.

[0046] In generating the virtual video data Dm, a periodic process may be repeated in which the vehicle image portion Dh and the background image portion Db are subjected to viewpoint conversion to generate image data for one frame of the virtual video data Dm by passing the vehicle image portion Dh and the background image portion Db through a machine learning model such as neural radiance fields (NeRF) constructed in the memory 10. Each time such periodic process is performed, the machine learning model receives input of state values ​​related to the host vehicle 2 and the noted user 3A, such as position coordinates and a velocity vector, and / or viewpoint coordinates calculated from the state values ​​of the noted user 3A relative to the host vehicle 2. At this time, the machine learning model may also receive input of performance data (particularly the mounting coordinates of the image sensor) of the external sensor 50 that sensed the noted user 3A, among the recognition data of the self state used to identify the noted user 3A, as the state values ​​of the host vehicle 2.

[0047] The recognition block 100 controls the visual information presentation type information presentation system 8 to continuously display the generated virtual video data Dm frame by frame to the driver in the host vehicle 2. The display of the virtual video data Dm may be performed constantly while the host vehicle 2 is running, but in particular in the driving assistance system 1, it is permitted and / or performed on the condition of the driver or an event related to the driver. Here, permission includes not only the case where it is performed in response to the satisfaction of a corresponding condition, but also the case where it is suspended until at least one other condition is satisfied. In the following description, the condition where the display of the virtual video data Dm is permitted and / or performed is referred to as a display condition C.

[0048] 5, the display condition C of the virtual video data Dm may be condition C1, which is set when the occupant sensing type internal sensor 52 receives a display request from the driver of the host vehicle 2. The display condition C may be condition C2, which is set when the driver is in a hands-off state in which the driver is permitted to take his / her hands off the steering wheel of the host vehicle 2. The display condition C may be condition C3, which is set when the driver is in an eyes-off state in which the driver is permitted to take his / her hands off the steering wheel of the host vehicle 2 and also to take his / her eyes off the field of view ahead of the host vehicle 2.

[0049] The display condition C may be a condition C4 that is set when the host vehicle 2 is in an automatic parking processing state. The display condition C may be a condition C5 that is set when the host vehicle 2 is in an automatic low-speed driving state within an approved speed range (for example, 10 km / h or less). The display condition C may be a composite condition in which the display of the virtual video data Dm is performed by satisfying multiple simultaneous approvals among the above conditions C1, C2, C3, C4, and C5.

[0050] When the display condition C is satisfied, the recognition block 100 may control the information presentation system 8 of an auditory information presentation type to issue, for example, a warning sound or a voice warning to the driver in the host vehicle 2. When the display condition C is satisfied, the recognition block 100 may control the information presentation system 8 of a cutaneous sensation information presentation type to issue, for example, vibration, reaction force, or conditioned air to the driver in the host vehicle 2.

[0051] 3 acquires recognition data from the recognition block 100. The planning block 120 acquires control data of the host vehicle 2 related to past driving behavior from the control block 140. Based on the acquired data, the planning block 120 generates prediction data Dp (see FIGS. 6 to 12 described in detail later) that predicts interactions between the host vehicle 2 and other road users 3, including the noted user 3A.

[0052] At this time, the generation of the predicted data Dp is performed in order to control the driving behavior of the host vehicle 2 by the downstream control block 140 in accordance with the virtual video data Dm. Therefore, the virtual video data Dm included in the recognition data from the recognition block 100 is provided to the planning block 120 for predicting an interaction between the host vehicle 2 and the attention user 3A.

[0053] In predicting the interaction with the noted user 3A, the driving trajectory Th of the host vehicle 2 along with the time transition of the relative relationship with the noted user 3A is predicted as shown in Figures 6 to 12, and prediction data Dp representing the result of the prediction is generated. Note that in Figures 6 to 12, the multiple black and white circles on the driving trajectory Th are a schematic representation of the driving control points in each control cycle in the driving assistance system 1, thinned out from the actual points.

[0054] The driving trajectory Th predicted when generating the prediction data Dp specifies the position coordinates for each control cycle of the trajectory that the host vehicle 2 will follow in the future. The driving trajectory Th may be predicted to further specify at least one of, for example, speed, acceleration, yaw angle, and yaw rate as a motion physical quantity assumed for each control cycle on the trajectory. For such a prediction of the driving trajectory Th, at least one of, for example, a result of the planning block 120 planning the future driving route of the host vehicle 2 based on the acquired data, and recognition data regarding the driver's manual driving operation, etc., may be used together with the virtual video data Dm.

[0055] As for the relative relationship between the host vehicle 2 and the noted user 3A, it is preferable that the time transition of a physical quantity used in the arbitration process in the planning block 120 described later is predicted, and the time transition is included in the predicted data Dp in association with the driving trajectory Th. The interaction prediction including such a driving trajectory Th and the relative relationship is performed based on at least one of the following grasped from the virtual video data Dm: the area change over time of the vehicle image portion Dh (see FIG. 12 described later in detail), the behavior of the vehicle image portion Dh (i.e., the behavior of the host vehicle 2), and the attitude of the vehicle image portion Dh (i.e., the attitude of the host vehicle 2).

[0056] The planning block 120 may adjust the autonomous driving level of the host vehicle 2 based on the acquired data and the predicted data Dp described above. The adjustment of the autonomous driving level may include a handover in which the driving task is transferred between the driving assistance system 1 and the driver due to a transition of the driving mode between the autonomous driving task and the manual driving assistance task. Such a handover is realized at least one of the following timings, for example: a handover request timing from the driver, a timing of moving forward and backward from an operational design domain (ODD) of the autonomous driving, and a timing required for a minimum risk manoeuvre (MRM).

[0057] The planning block 120 executes a control plan for controlling the driving behavior of the host vehicle 2 based on the above-mentioned acquired data and predicted data Dp. At this time, a control command is generated that is sent to the actuator system 4 to control a driving task corresponding to the autonomous driving level among the autonomous driving task and the manual driving assistance task in the host vehicle 2. Examples of the driving task corresponding to the autonomous driving level include adaptive cruise control (ACC), autonomous emergency braking (AEB), and lane keeping assist (LKA).

[0058] The planning block 120 plans a control command to control the driving behavior of the host vehicle 2 in accordance with the interaction predicted based on the virtual video data Dm between the host vehicle 2 and the noted user 3A. The control command is generated to adapt a driving control pattern for controlling the driving behavior to be given to the host vehicle 2 to predicted data Dp of the interaction predicted based on the virtual video data Dm and including the driving trajectory Th. In particular, the control command generated by the planning block 120 of the first embodiment includes a driving control pattern P that selectively reflects the self-perspective of the host vehicle 2 and the other-perspective of the noted user 3A as shown in Figs. 3, 6 to 12.

[0059] Specifically, as the driving control pattern P in Figures 6 to 12, the planning block 120 arbitrates between the self-control pattern Ph from the self-perspective of the host vehicle 2 and the other-control pattern Po from the other-perspective of the noted user 3A. Note that, although Figures 6 to 12 in the first embodiment representatively illustrate traffic scenes in which the noted user 3A is stopped, the noted user 3A may of course be in a moving state.

[0060] 6 to 8, when the prediction data Dp includes an approach distance Lo from the driving trajectory Th of the host vehicle 2 approaching the noted user 3A to the noted user 3A, the planning block 120 may adjust the driving control pattern P to suit the approach distance Lo. At this time, when the prediction data Dp also includes a time to collision (TTC) of the host vehicle 2 with the noted user 3A, the planning block 120 determines the approach distance Lo in a trajectory section ΔT on the driving trajectory Th where the TTC is expected to be equal to or less than a threshold time.

[0061] Such distance judgment is performed from an operation control point before the TTC threshold time to an operation control point within the trajectory section ΔT. In particular, the distance judgment is preferably performed from an operation control point at or just before the TTC threshold time (examples of Figs. 6 to 8) to an operation control point within the trajectory section ΔT. Note that the white circles in Figs. 6 to 8 are schematic illustrations of operation control points on the operation trajectory Th in the trajectory section ΔT where the TTC is equal to or less than the threshold time.

[0062] If the approach distance Lo is estimated to be equal to or greater than the threshold distance shown by the dashed line in Fig. 6 in the orbital section ΔT where the TTC is equal to or less than the threshold time as a result of the distance judgment, the operation control pattern P is adjusted to the automatic control pattern Ph that gives the approach distance Lo as is. Here, the operation control pattern P is adjusted to the automatic control pattern Ph even at operation control points other than the one where the distance judgment is performed.

[0063] On the other hand, when it is determined that a safety margin is required from the other-self viewpoint because the approach distance Lo in the trajectory section ΔT is less than the threshold distance shown by the dashed line in Fig. 7, the driving control pattern P is adjusted to the other-self control pattern Po that increases the approach distance Lo more than the self-control pattern Ph, as shown in Fig. 8. Thus, Fig. 8 shows a schematic diagram of the approach distance Lo on the driving trajectory Th according to the self-control pattern Ph before the adjustment, and the approach distance Lo expected on the driving trajectory Th according to the other-self control pattern Po after the adjustment.

[0064] 6 to 8 is defined as the closest distance (i.e., the smallest distance) among the horizontal distances from the driving trajectory Th to the center point of the noted user 3A in the trajectory section ΔT. The threshold time for determining the trajectory section ΔT in the arbitration for conforming to the approach distance Lo may be set to, for example, 2 seconds as a time for prioritizing the other-self viewpoint with respect to the TTC assumed by assuming that the closest position on the driving trajectory Th to the noted user 3A is the position of the noted user 3A.

[0065] The threshold distance of the approach distance Lo, which determines whether or not arbitration is required to prioritize the other-side control pattern Po over the self-side control pattern Ph, may be set to a fixed value such as 1 m or 1.5 m. The threshold distance of the approach distance Lo may be variably set according to the vulnerability of the noted user 3A, who is the other road user 3, and in this case, the threshold distance of the approach distance Lo may be set to a longer distance value as the vulnerability increases (i.e., as the vulnerability increases), such as a pedestrian operating a stroller. The other-side control pattern Po in the case where the other-side control pattern Po is prioritized over the self-side control pattern Ph may be specified including a driving control parameter for variably adjusting the approach distance Lo. The driving control parameter for variably adjusting the approach distance Lo may be at least one of, for example, a yaw angle parameter and a yaw rate parameter, which are set in the host vehicle 2.

[0066] 9 to 11, when the prediction data Dp includes the time transition of the relative approaching speed Vo of the host vehicle 2 approaching the noted user 3A, the planning block 120 may adjust the driving control pattern P to suit the relative approaching speed Vo. In this case, when the prediction data Dp also includes the TTC of the host vehicle 2 with respect to the noted user 3A, the planning block 120 determines the relative approaching speed Vo in the trajectory section ΔT on the driving trajectory Th where the TTC is expected to be equal to or less than the threshold time.

[0067] Such speed judgment is performed from an operation control point before the TTC threshold time to an operation control point within the trajectory section ΔT. In particular, the speed judgment is preferably performed from an operation control point at or just before the TTC threshold time (examples of Figs. 9 to 11) to an operation control point within the trajectory section ΔT. Note that the white circles in Figs. 9 to 11 and Fig. 12, which will be described in detail later, also show schematic illustrations of operation control points on the operation trajectory Th in the trajectory section ΔT where the TTC is equal to or less than the threshold time.

[0068] In the case of Fig. 9, where the relative approach speed Vo is assumed to be equal to or lower than the threshold speed in the orbit section ΔT where the TTC is equal to or lower than the threshold time as a result of the speed judgment, the operation control pattern P is adjusted to the automatic control pattern Ph that gives the relative approach speed Vo as is. Here, the operation control pattern P is adjusted to the automatic control pattern Ph even at operation control points other than the one where the speed judgment is performed.

[0069] On the other hand, in the case of Fig. 10 where it is determined that a safe speed from the other's point of view is necessary due to the relative approach speed Vo exceeding the threshold speed in the trajectory section ΔT, the driving control pattern P is adjusted to the other's control pattern Po that reduces the relative approach speed Vo below the self-control pattern Ph as shown in Fig. 11. Fig. 11 shows a schematic diagram of the relative approach speed Vo on the driving trajectory Th according to the self-control pattern Ph before the adjustment, as well as the relative approach speed Vo expected on the driving trajectory Th according to the other's control pattern Po after the adjustment.

[0070] The relative approach speed Vo shown in Figures 9 to 11 is defined by assuming that the closest approach position on the driving trajectory Th to the noted user 3A is the location of the noted user 3A. Therefore, in Figures 9 to 11 which show the noted user 3A in a stopped state as described above, the relative approach speed Vo is substantially equivalent to the traveling speed of the host vehicle 2, and is therefore illustrated with respect to each driving control point in the trajectory section ΔT. In the arbitration to adapt the relative approach speed Vo, the threshold time of the TTC which determines the trajectory section ΔT may be set in accordance with the arbitration to adapt the approach distance Lo described above.

[0071] The threshold speed of the relative approaching speed Vo, which determines whether or not arbitration is required to prioritize the other-side control pattern Po over the self-side control pattern Ph, may be set to a fixed value such as 30 km / h based on the safety concept of the zone 30. The threshold speed of the relative approaching speed Vo may be variably set according to the vulnerability of the noted user 3A who is the other road user 3, and in this case, the threshold speed of the relative approaching speed Vo may be set to a lower speed value as the vulnerability increases (i.e., as the vulnerability increases), for example, for a pedestrian operating a stroller. The other-side control pattern Po when prioritized over the self-side control pattern Ph may be specified including a driving control parameter for variably adjusting the relative approaching speed Vo. The driving control parameter for variably adjusting the relative approaching speed Vo may be at least one of a speed parameter, an acceleration parameter, etc., which is set in the host vehicle 2.

[0072] The arbitration for adapting the relative approaching speed Vo in the first embodiment may be combined with the arbitration for adapting the approaching distance Lo in the first embodiment described above. In the arbitration for adapting the relative approaching speed Vo, the time transition of the relative approaching speed Vo may be predicted and included in the predicted data Dp based on the time change occurring in the pixel area of ​​the vehicle image portion Dh in the virtual video data Dm up to the TTC threshold time as shown in Fig. 12. Note that Fig. 12 shows a schematic diagram in which the pixel area of ​​the vehicle image portion Dh that changes over time in the virtual video data Dm is replaced by the area of ​​a rectangular figure attached to each driving control point.

[0073] In the arbitration for adapting to the relative approach speed Vo, the time change occurring in the pixel area of ​​the vehicle image portion Dh in the virtual video data Dm up to the TTC threshold time as shown in Fig. 12 may be included in the predicted data Dp instead of the time transition of the relative approach speed Vo. In this case, the event in which the time rate of change occurring in the pixel area of ​​the vehicle image portion Dh at or near the TTC threshold time exceeds the threshold rate may be assumed to be equivalent to the event in which the relative approach speed Vo in the trajectory section ΔT exceeds the threshold speed, and the other-side control pattern Po may be preferentially arbitrated.

[0074] The control block 140 shown in Fig. 3 obtains a control command including the reconciled driving control pattern P from the planning block 120. The control block 140 controls the driving behavior of the host vehicle 2 according to the control command from the planning block 120. At this time, the control block 140 may obtain recognition data from the recognition block 100 and use it to control the driving behavior. As a result, the host vehicle 2 is controlled to a motion behavior that matches the prediction data Dp that predicts the interaction according to the virtual video data Dm. The control data that indicates the result of the driving behavior controlled in this way is fed back from the control block 140 to the recognition block 100 and the planning block 120.

[0075] (Driving assistance flow) In the first embodiment, a driving assistance flow that realizes a driving assistance method for assisting driving of a host vehicle 2 is repeatedly executed according to Fig. 13 by cooperation of multiple blocks 100, 120, and 140. Note that in the following description, each "S" in the driving assistance flow refers to multiple steps that are executed by multiple commands included in a driving assistance program.

[0076] In S10, the recognition block 100 determines whether or not a noted user 3A approaching the host vehicle 2 relatively in the interaction area Ia has been identified among other road users 3 present in the external world of the host vehicle 2. If a negative determination is made in S10, the current execution of the driving assistance flow ends. On the other hand, if a positive determination is made in S10, the driving assistance flow proceeds to S20.

[0077] In S20, the recognition block 100 generates virtual video data Dm that virtually represents the host vehicle 2 as seen from the viewpoint of the noted user 3A identified in S10. After completion of S20, in the driving assistance flow, the display sequence of S30, S40 and the control sequence of S50, S60, S70 are executed in parallel.

[0078] In S30 of the display sequence, the recognition block 100 judges whether or not the display condition C for displaying the virtual video data Dm generated in S20 is satisfied. At this time, the satisfaction of the display condition C means that at least one of the above-mentioned conditions C1, C2, C3, C4, and C5 is satisfied. If a negative judgment is made in S30, the current execution of the driving assistance flow is terminated. On the other hand, if a positive judgment is made in S30, the driving assistance flow proceeds to S40.

[0079] In S40 of the display sequence, the recognition block 100 causes the information presentation system 8 to display the virtual video data Dm for which the display condition C is satisfied, to the driver in the host vehicle 2. Upon completion of S40, the current execution of the display sequence in the driving assistance flow ends.

[0080] In a control sequence S50 parallel to the display sequence, the planning block 120 generates predicted data Dp of an interaction predicted between the host vehicle 2 and the attention user 3A based on the virtual video data Dm generated in S20. Then, in a control sequence S60, the planning block 120 plans a control command for controlling the driving behavior of the host vehicle 2 in accordance with the predicted data Dp generated in S50 according to the virtual video data Dm. Furthermore, in a control sequence S70, the control block 140 controls the host vehicle 2 to perform a driving behavior in accordance with the control command planned in S60, i.e., a driving behavior in accordance with the virtual video data Dm.

[0081] Completion of S70 ends the current execution of the control sequence in the driving assistance flow. Therefore, if the other-self control pattern Po is preferentially arbitrated by the current execution of the control sequence, and the condition of the priority arbitration is not satisfied in the next execution of the control sequence or later, the driving control pattern P is mediated back to the self control pattern Ph. Such a priority arbitration condition of the other-self control pattern Po may be unsatisfied by an automated driving task following the other-self control pattern Po, or may be unsatisfied by the driver's manual driving operation under the manual driving assistance task following the other-self control pattern Po.

[0082] (Action and effect) The effects of the first embodiment described above will now be described.

[0083] In the first embodiment, virtual video data Dm is acquired that virtually represents the host vehicle 2 as seen from the viewpoint of an other road user 3 (specifically, a user of interest 3A) that exists in the external world of the host vehicle 2. Thus, according to the first embodiment, the host vehicle 2 is controlled to perform driving actions according to the virtual video data Dm from the viewpoint of the other road user 3. This allows the viewpoint of the other road user 3 to be directly reflected in the driving actions of the host vehicle 2, making it possible to ensure safety and security in the interaction between the host vehicle 2 and the other road user 3.

[0084] According to the first embodiment, the driving behavior of the host vehicle 2 is controlled in accordance with the interaction predicted between the host vehicle 2 and the other road user 3 based on the virtual video data Dm. This makes it possible to reflect even the viewpoint of the other road user 3 that is the subject of the prediction in the prediction of the interaction with the other road user 3 that affects the driving behavior of the host vehicle 2. Therefore, it is possible to enhance the effect of ensuring safety and security in the interaction between the host vehicle 2 and the other road user 3.

[0085] According to the first embodiment, a driving control pattern P for controlling the driving behavior of the host vehicle 2 is adjusted to match the predicted data Dp of the interaction predicted based on the virtual video data Dm. At this time, as the driving control pattern P, a self-control pattern Ph based on the self-perspective of the host vehicle 2 is adjusted with an other-control pattern Po based on the other-perspective of the other road user 3. According to this, it can be said that the adjustment of the other-control pattern Po against the self-control pattern Ph based on the self-perspective of the host vehicle 2 can be realized according to the interaction prediction that reflects the other-perspective of the other road user 3 from the virtual video data Dm. Therefore, it is possible to increase the reliability of ensuring safety and security in the interaction between the host vehicle 2 and the other road user 3.

[0086] According to the first embodiment, the driving control pattern P is adjusted to the other-side control pattern Po, which has priority over the self control pattern Ph, in accordance with the predicted data Dp of the trajectory section ΔT in which the collision margin time of the host vehicle 2 with the other road user 3 is expected to be equal to or less than the threshold time on the driving trajectory Th. According to this, in an approaching scene in which the collision margin time of the host vehicle 2 with the other road user 3 becomes short, the other-side perspective of the other road user 3 can be prioritized by adjusting to the other-side control pattern Po. Therefore, it is possible to timely ensure safety and security in the interaction between the host vehicle 2 and the other road user 3, particularly safety and security from the other-side perspective of the other road user 3.

[0087] According to the first embodiment, the driving control pattern P may be adjusted to the other-side control pattern Po in accordance with the predicted data Dp in which the approach distance Lo from the driving trajectory Th (specifically, the trajectory section ΔT) of the host vehicle 2 approaching the other road user 3 to the other road user 3 is less than the threshold distance. According to this, in an approach scene in which the other-side perspective of the other road user 3 is prioritized over the driving trajectory Th of the host vehicle 2 according to the self control pattern Ph, the approach distance Lo can be enlarged to be greater than the self control pattern Ph by mediation to the other-side control pattern Po. Therefore, it is possible to timely ensure safety and security in the interaction between the host vehicle 2 and the other road user 3, particularly safety and security from the other-side perspective of the other road user 3. Furthermore, when the threshold distance of such an approach distance Lo is variably set according to the vulnerability of the other road user 3, it is possible to timely ensure safety and security in accordance with the other road user 3 with a high vulnerability.

[0088] According to the first embodiment, the driving control pattern P may be adjusted to the other-side control pattern Po in accordance with the prediction data Dp in which the relative approaching speed Vo of the host vehicle 2 approaching the other road user 3 exceeds the threshold speed. According to this, in an approaching scene in which the other-side perspective of the other road user 3 is prioritized over the relative approaching speed Vo of the host vehicle 2 according to the self-control pattern Ph, the relative approaching speed Vo can be lowered below the self-control pattern Ph by adjusting to the other-side control pattern Po. Therefore, it is possible to timely ensure safety and security in the interaction between the host vehicle 2 and the other road user 3, particularly safety and security from the other-side perspective of the other road user 3. Furthermore, when the threshold speed of such a relative approaching speed Vo is variably set according to the vulnerability of the other road user 3, it is possible to timely ensure safety and security in accordance with the other road user 3 with a high vulnerability.

[0089] Second Embodiment As shown in FIG. 14, the second embodiment is a modification of the first embodiment.

[0090] In the second embodiment, the control command generated by the planning block 2120 includes a driving control parameter ρ that is set in the host vehicle 2 so as to weight the self viewpoint of the host vehicle 2 and the other-person viewpoint of the noted user 3A to mediate the driving control pattern P. As such a driving control parameter ρ, the planning block 2120 mediates between a self control parameter ρh that defines the self control pattern Ph and an other-person control parameter ρo that defines the other-person control pattern Po.

[0091] Specifically, the planning block 2120 adjusts the weighting of the self control parameter ρh and the other-self control parameter ρo in accordance with the predicted interaction data Dp predicted based on the virtual video data Dm reflecting the other-self viewpoint of the noted user 3A, who is the other road user 3. Therefore, the planning block 2120 sets the driving control parameter ρ according to the following equations 1 to 4 using the weight ωh of the self control parameter ρh and the weight ωo of the other-self control parameter ρo shown in Figs. 15 to 20. Such a setting is equivalent to determining the driving control pattern P by weighting adjustment between the self control pattern Ph defined by the self control parameter ρh and the other-self control pattern Po defined by the other-self control parameter ρo. Note that Figs. 15 to 20 of the second embodiment also representatively show traffic scenes in which the noted user 3A is stopped, but the noted user 3A may of course be in a moving state.

number

number

number

number

[0092] 15 to 17, when the prediction data Dp includes the approach distance Lo from the driving trajectory Th of the host vehicle 2 approaching the noted user 3A to the noted user 3A, the planning block 2120 may adjust the weights ωh, ωo related to the driving control parameter ρ to the approach distance Lo. In this case, when the prediction data Dp also includes the TTC of the host vehicle 2 with respect to the noted user 3A, the planning block 2120 determines the approach distance Lo in the trajectory section ΔT on the driving trajectory Th where the TTC is expected to be equal to or less than the threshold time.

[0093] As a result of such distance judgment, when the approach distance Lo is assumed to be equal to or greater than the threshold distance shown by the dashed line in Fig. 15 in the orbital section ΔT where the TTC is equal to or less than the threshold time, the weight ωo of the other-self control parameter ρo is adjusted lower than the weight ωh of the self control parameter ρh. Here, even at driving control points other than the point where the distance judgment is performed, the weight ωo of the other-self control parameter ρo is adjusted lower than the weight ωh of the self control parameter ρh.

[0094] On the other hand, when it is determined that a safety margin is required from the other-self's viewpoint because the approach distance Lo in the trajectory section ΔT is less than the threshold distance shown by the dashed line in Fig. 16, the weight ωo of the other-self control parameter ρo is adjusted to be higher than the weight ωh of the self control parameter ρh, as shown in Fig. 17. Thus, Fig. 17 shows a schematic diagram of the approach distance Lo on the driving trajectory Th according to the weighting relationship Fh:ωh>ωo before arbitration, as well as the approach distance Lo expected on the driving trajectory Th according to the weighting relationship Fo:ωh<ωo after arbitration.

[0095] In the second embodiment, the driving control point for performing distance judgment, the approach distance Lo, the threshold time of the TTC, the threshold distance of the approach distance Lo, and the driving control parameter ρ for variably adjusting the approach distance Lo are similar to those in the first embodiment. In the second embodiment, the self-control parameter ρh specifies the self-control pattern Ph that gives the approach distance Lo as is, and the other-self-control parameter ρo specifies the other-self-control pattern Po that enlarges the approach distance Lo more than the self-control pattern Ph.

[0096] Therefore, the arbitration in which the weight ωo of the other-self control parameter ρo is set to 0 in the case of Fig. 15 in the second embodiment is equivalent to the arbitration in the case of Fig. 6 in the first embodiment. On the other hand, the arbitration in which the weight ωh of the self-control parameter ρh is set to 0 in the cases of Figs. 16 and 17 in the second embodiment is equivalent to the arbitration in the cases of Figs. 7 and 8 in the first embodiment.

[0097] 18 to 20, when the prediction data Dp includes the time transition of the relative approaching speed Vo of the host vehicle 2 approaching the noted user 3A, the planning block 2120 may adjust the weights ωh, ωo related to the driving control parameter ρ to match the relative approaching speed Vo. In this case, when the prediction data Dp also includes the TTC of the host vehicle 2 with respect to the noted user 3A, the planning block 2120 determines the relative approaching speed Vo in the trajectory section ΔT on the driving trajectory Th where the TTC is expected to be equal to or less than the threshold time.

[0098] As a result of such speed judgment, in the orbit section ΔT where the TTC is equal to or less than the threshold time, in the case of Fig. 18 where the relative approach speed Vo is assumed to be equal to or less than the threshold speed, the weight ωo of the other-self control parameter ρo is adjusted to be lower than the weight ωh of the self control parameter ρh. Here, even at the driving control points other than the driving control points where the speed judgment is performed, the weight ωo of the other-self control parameter ρo is adjusted to be lower than the weight ωh of the self control parameter ρh.

[0099] On the other hand, in the case of Fig. 19 where it is determined that a safe and secure speed from the other's point of view is necessary due to the relative approach speed Vo exceeding the threshold speed in the trajectory section ΔT, the weight ωo of the other's control parameter ρo is adjusted to be higher than the weight ωh of the self-control parameter ρh as shown in Fig. 20. Thus, Fig. 20 shows a schematic diagram of the relative approach speed Vo on the driving trajectory Th according to the weighting relationship Fh:ωh>ωo before arbitration, as well as the relative approach speed Vo expected on the driving trajectory Th according to the weighting relationship Fo:ωh<ωo after arbitration.

[0100] In the second embodiment, the driving control point for speed judgment, the threshold time of the TTC, the threshold speed of the relative approaching speed Vo, the driving control parameter ρ for variably adjusting the relative approaching speed Vo, and the vehicle image portion Dh are used in accordance with the first embodiment. In the second embodiment, the self control parameter ρh specifies the self control pattern Ph that provides the relative approaching speed Vo as is, and the other-self control parameter ρo specifies the other-self control pattern Po that reduces the relative approaching speed Vo to a value lower than the self control pattern Ph.

[0101] Therefore, the arbitration in which the weight ωo of the other-self control parameter ρo is set to 0 in the case of Fig. 18 in the second embodiment is equivalent to the arbitration in the case of Fig. 9 in the first embodiment. On the other hand, the arbitration in which the weight ωh of the self control parameter ρh is set to 0 in the cases of Figs. 19 and 20 in the second embodiment is equivalent to the arbitration in the cases of Figs. 10 and 11 in the first embodiment. Such arbitration to adapt to the relative approach speed Vo in the second embodiment may be combined with the arbitration to adapt to the approach distance Lo in the second embodiment described above.

[0102] (Action and effect) The effects unique to the second embodiment described above will be described below.

[0103] According to the second embodiment, the driving control parameter ρ set in the host vehicle 2 is adjusted to provide a driving control pattern P that matches the predicted data Dp of the interaction predicted based on the virtual video data Dm from the other-road user 3's other-viewpoint. At this time, as the driving control parameter ρ, the self-control parameter ρh that defines the self-control pattern Ph from the self-viewpoint of the host vehicle 2 is adjusted with the other-road control parameter ρo that defines the other-road control pattern Po from the other-road user 3's other-viewpoint. According to this, it can be said that the adjustment of the other-road control parameter ρo with respect to the self-control parameter ρh from the self-viewpoint of the host vehicle 2 is realized according to the interaction prediction that reflects the other-road user 3's other-viewpoint from the virtual video data Dm. Therefore, it is possible to ensure safety and security in the interaction between the host vehicle 2 and the other road user 3, especially safety and security from the other-road user 3's other-viewpoint.

[0104] According to the second embodiment, the weighting of the self control parameter ρh and the other-side control parameter ρo is adjusted in accordance with the predicted data Dp of the interaction predicted based on the virtual video data Dm from the other-side perspective of the other road user 3. According to this, it can be said that the weighting adjustment of the other-side control parameter ρo with respect to the self control parameter ρh from the self-side perspective of the host vehicle 2 is realized according to the interaction prediction that reflects the other-side perspective of the other road user 3 from the virtual video data Dm. Therefore, it is possible to ensure safety and security in the interaction between the host vehicle 2 and the other road user 3, particularly safety and security from the other-side perspective of the other road user 3, by appropriate weighting adjustment.

[0105] According to the second embodiment, the weight ωo of the other-side control parameter ρo is adjusted to be higher than the weight ωh of the self-control parameter ρh in accordance with the predicted data Dp of the trajectory section ΔT in which the collision time margin of the host vehicle 2 with the other road user 3 is expected to be equal to or shorter than the threshold time on the driving trajectory Th. According to this, in the host vehicle 2 in the approaching scene in which the collision time margin with the other road user 3 is short, the reflection degree of the other-side control parameter ρo on the driving behavior can be increased in accordance with the fine weight arbitration compared to the self-control parameter ρh. Therefore, it is possible to timely and appropriately ensure safety and security in the interaction between the host vehicle 2 and the other road user 3, particularly safety and security from the other road user 3's other-side perspective.

[0106] According to the second embodiment, in the trajectory section ΔT where the collision allowance time is equal to or less than the threshold time, the weight ωo of the other-side control parameter ρo may be adjusted to be higher than the weight ωh of the self-control parameter ρh in accordance with the predicted data Dp where the approach distance Lo from the driving trajectory Th is less than the threshold distance. According to this, in the host vehicle 2 in the approach scene where the collision allowance time and the approach distance Lo are both short with respect to the other road user 3, the reflection degree of the other-side control parameter ρo on the driving behavior can be increased according to the fine weight adjustment. Therefore, it is possible to timely and appropriately ensure safety and security in the interaction between the host vehicle 2 and the other road user 3, particularly safety and security from the other road user 3's other-side perspective. Furthermore, in the case where the threshold distance of the approach distance Lo is variably set according to the vulnerability of the other road user 3 in accordance with the first embodiment, it is possible to timely ensure safety and security in accordance with the other road user 3 with high vulnerability.

[0107] According to the second embodiment, in the trajectory section ΔT where the collision time is equal to or less than the threshold time, the weight ωo of the other-side control parameter ρo may be adjusted to be higher than the weight ωh of the self-control parameter ρh in accordance with the prediction data Dp where the relative approach speed Vo exceeds the threshold speed. According to this, in the host vehicle 2 in the approach scene where the collision time to the other road user 3 is shortened while the relative approach speed Vo increases, the reflection degree of the other-side control parameter ρo on the driving behavior can be increased in accordance with the fine weight adjustment. Therefore, it is possible to timely and appropriately ensure safety and security in the interaction between the host vehicle 2 and the other road user 3, particularly safety and security from the other road user 3's other-side perspective. Furthermore, in the case where the threshold speed of the relative approach speed Vo is variably set according to the vulnerability of the other road user 3 in accordance with the first embodiment, it is possible to timely ensure safety and security in accordance with the other road user 3 with high vulnerability.

[0108] (Other embodiments) Although several embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope not departing from the gist of the present disclosure.

[0109] In a modified example, the dedicated computer constituting the driving assistance system 1 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), CPLD (Complex Programmable Logic Device), etc. In addition, such a digital circuit may have a memory that stores a program.

[0110] In a modified example, the target user 3A to be displayed in the virtual video data Dm may include not only the other road user 3 at the closest distance, but also some or all of the other road users 3 in the interaction area Ia who are farther away from the host vehicle 2 than the closest distance. In a modified example, the display sequence for displaying the virtual video data Dm by the recognition block 100 may be omitted.

[0111] In a modified example, the recognition block 100 may predict an interaction between the host vehicle 2 and another road user 3 in advance based on recognition data other than the virtual video data Dm, prior to prediction based on the virtual video data Dm in the planning block 120. In this case, the attention user 3A may be identified according to at least one of the following types of predicted interaction: probability, collision risk, etc.

[0112] In a modified example, the operator who manually drives the host vehicle 2 to which the driving assistance system 1 is applied may be a remote operator who remotely controls the driving of the host vehicle 2 from an external center. In this case, the virtual video data Dm may be displayed to the remote operator.

[0113] In a modified example, the arbitration between the self-control pattern Ph and the other-self control pattern Po may be performed in accordance with prediction data Dp that indicates the level of risk from the other-self viewpoint in the interaction area Ia, in addition to the approach distance Lo and the relative approach speed Vo. Specifically, the level of risk may be expressed by the type of road. In this case, for example, the self-control pattern Ph may be prioritized in accordance with a main road, while the other-self control pattern Po may be prioritized in accordance with a community road. Alternatively, the self-control pattern Ph may be prioritized in accordance with a road structure with a sidewalk, while the other-self control pattern Po may be prioritized in accordance with a road structure without a sidewalk.

[0114] In a modified example, the weighting of the self-control parameter ρh and the other-control parameter ρo may be adjusted to suit the predicted data Dp representing the level of risk from the other-perspective in the interaction area Ia. Specifically, the level of risk may be expressed by the type of road. In this case, for example, the weighting ωo of the other-perspective may be adjusted to be lower than the weighting ωh to suit a main road, while the weighting ωo of the other-perspective may be adjusted to be higher than the weighting ωh to suit a community road. Alternatively, the weighting ωo of the other-perspective may be adjusted to be lower than the weighting ωh to suit a road structure with a sidewalk, while the weighting ωo of the other-perspective may be adjusted to be higher than the weighting ωh to suit a road structure without a sidewalk.

[0115] (Additional remarks) This specification discloses the following technical ideas and their combinations. Note that the reference characters in parentheses in the appended remarks indicate the corresponding relationship with the specific means described in the above detailed embodiment, and do not limit the technical scope of this disclosure.

[0116] (Technical thought 1) A driving assistance system having a processor (12) for assisting driving of a host vehicle (2), The processor, acquiring virtual video data (Dm) representing the host vehicle virtually as seen from the viewpoint of another road user (3) present in an external world of the host vehicle; and controlling the host vehicle to perform a driving behavior in accordance with the virtual video data.

[0117] (Technical thought 2) The control of the host vehicle includes: A driving assistance system as described in technical idea 1, which includes controlling the driving behavior of the host vehicle in accordance with a predicted interaction between the host vehicle and the other road user based on the virtual video data.

[0118] (Technical Thought 3) The control of the host vehicle includes: A driving assistance system as described in technical idea 2, which includes reconciling a self-control pattern based on the host vehicle's own perspective with an other-control pattern based on the other-viewpoint of the other road user as a driving control pattern that controls the driving behavior of the host vehicle in accordance with predicted data (Dp) of the interaction predicted based on the virtual video data.

[0119] (Technical Thought 4) The control of the host vehicle includes: A driving assistance system according to technical idea 3, which includes adjusting the driving control pattern to the other-self control pattern, which has priority over the self-control pattern, in accordance with the prediction data for a trajectory section in which the collision margin time of the host vehicle against the other road user is expected to be equal to or less than a threshold time on the driving trajectory.

[0120] (Technical Thought 5) The control of the host vehicle includes: A driving assistance system as described in technical idea 3 or 4, which includes adjusting the driving control pattern to the other-road control pattern that increases the approach distance more than the self-control pattern by adapting it to the prediction data in which the approach distance from the driving trajectory of the host vehicle approaching the other road user to the other road user is less than a threshold distance.

[0121] (Technical Thought 6) The control of the host vehicle includes: A driving assistance system as described in any one of technical ideas 3 to 5, which includes adjusting the driving control pattern to the other-road control pattern that lowers the relative approaching speed of the host vehicle approaching the other road user more than the self-control pattern in accordance with the prediction data in which the relative approaching speed of the host vehicle approaching the other road user exceeds a threshold speed.

[0122] (Technical Thought 7) The control of the host vehicle includes: A driving assistance system as described in any one of technical ideas 3 to 6, which includes reconciling a self-control parameter that defines the self-control pattern with an other-control parameter that defines the other-control pattern as a driving control parameter that is set in the host vehicle to provide the driving control pattern that conforms to the prediction data.

[0123] (Technical Thought 8) The control of the host vehicle includes: A driving assistance system according to Technical Idea 7, which includes adjusting the weighting of the self-control parameters and the other-self-control parameters in accordance with the prediction data.

[0124] (Technical Thought 9) The control of the host vehicle includes: A driving assistance system according to technical idea 8, which includes adjusting the weight of the other-self-control parameter to be higher than the weight of the self-control parameter by adapting it to the prediction data for a trajectory section in which the collision margin time of the host vehicle against the other road user is expected to be equal to or less than a threshold time on the driving trajectory.

[0125] (Technical Thought 10) The control of the host vehicle includes: A driving assistance system as described in technical idea 9, which includes adjusting the weight of the other-self-control parameter to be higher than the weight of the self-control parameter in accordance with the prediction data in which the approach distance from the driving trajectory of the host vehicle approaching the other road user to the other road user is less than the threshold distance in a trajectory section in which the collision margin time of the host vehicle with the other road user is expected to be less than a threshold time on the driving trajectory.

[0126] (Technical Thought 11) The control of the host vehicle includes: A driving assistance system as described in Technical Idea 9 or 10, which includes adjusting the weight of the other-self-control parameter to be higher than the weight of the self-control parameter in accordance with the prediction data in which the relative approach speed of the host vehicle approaching the other road user exceeds the threshold speed in a trajectory section in which the collision margin time of the host vehicle with the other road user is expected to be below a threshold time on the driving trajectory.

[0127] (Technical Thought 12) The control of the host vehicle includes: A driving assistance system according to Technical Idea 5 or 10, which includes variably setting the threshold distance depending on the vulnerability of the other road user.

[0128] (Technical Thought 13) The control of the host vehicle includes: The driving assistance system according to Technical Idea 6 or 11, which includes variably setting the threshold speed depending on the vulnerability of the other road user.

[0129] The above technical concepts 1 to 13 may be understood as the respective technical concepts of a program and a method. [Explanation of symbols]

[0130] 1: Driving assistance system, 2: Host vehicle, 3: Other road users, 10: Memory, 12: Processor, Dm: Virtual video data, Dp: Prediction data

Claims

1. A driving assistance system having a processor (12) for assisting driving of a host vehicle (2), comprising: The processor, acquiring virtual video data (Dm) representing the host vehicle virtually as seen from the viewpoint of another road user (3) present in an external world of the host vehicle; and controlling the host vehicle to perform a driving behavior in accordance with the virtual video data.

2. The control of the host vehicle includes: The driving assistance system of claim 1 , further comprising controlling the driving behavior of the host vehicle in accordance with a predicted interaction between the host vehicle and the other road user based on the virtual video data.

3. The control of the host vehicle includes: The driving assistance system of claim 2, further comprising: reconciling a self-control pattern based on the host vehicle's own viewpoint with a different-control pattern based on the different road user's different viewpoint as a driving control pattern for controlling the driving behavior of the host vehicle in accordance with predicted data (Dp) of the interaction predicted based on the virtual video data.

4. The control of the host vehicle includes: The driving assistance system of claim 3, further comprising: adjusting the driving control pattern to the other-self control pattern, which has priority over the self-control pattern, in accordance with the prediction data for a trajectory section in which the collision margin time of the host vehicle against the other road user is expected to be equal to or less than a threshold time on the driving trajectory.

5. The control of the host vehicle includes: The driving assistance system of claim 3, further comprising: adjusting the driving control pattern to the other-self control pattern, which increases the approach distance more than the self control pattern, in accordance with the prediction data in which the approach distance from the driving trajectory of the host vehicle approaching the other road user to the other road user is less than a threshold distance.

6. The control of the host vehicle includes: The driving assistance system of claim 3, further comprising: adjusting the driving control pattern to the other-road user control pattern, which lowers the relative approaching speed of the host vehicle approaching the other road user more than the self-control pattern, in accordance with the prediction data in which the relative approaching speed of the host vehicle approaching the other road user exceeds a threshold speed.

7. The control of the host vehicle includes: The driving assistance system according to claim 3, further comprising: arbitrating a self-control parameter that defines the self-control pattern with an other-control parameter that defines the other-control pattern as a driving control parameter that is set in the host vehicle to provide the driving control pattern that conforms to the prediction data.

8. The control of the host vehicle includes: The driving assistance system according to claim 7 , further comprising adjusting weighting of the self-control parameter and the other-control parameter in accordance with the prediction data.

9. The control of the host vehicle includes:

9. The driving assistance system of claim 8, further comprising: adjusting the weight of the other-self-control parameter to be higher than the weight of the self-control parameter in accordance with the prediction data for a trajectory section in which the collision time margin of the host vehicle against the other road user is expected to be equal to or less than a threshold time on the driving trajectory.

10. The control of the host vehicle includes: The driving assistance system of claim 9, further comprising: adjusting the weight of the other-self-control parameter to be higher than the weight of the self-control parameter in accordance with the prediction data in which the approach distance from the driving trajectory of the host vehicle approaching the other road user to the other road user is less than the threshold distance in a trajectory section in which the collision margin time of the host vehicle with the other road user is expected to be less than a threshold time on the driving trajectory.

11. The control of the host vehicle includes: The driving assistance system of claim 9, further comprising: adjusting the weight of the other-self-control parameter to be higher than the weight of the self-control parameter in accordance with the prediction data in which the relative approach speed of the host vehicle approaching the other road user exceeds the threshold speed in a trajectory section in which the collision margin time of the host vehicle with the other road user is expected to be equal to or less than a threshold time on the driving trajectory.

12. The control of the host vehicle includes: The driving assistance system according to claim 5 or 10, further comprising:variably setting the threshold distance depending on a vulnerability level of the other road user.

13. The control of the host vehicle includes: The driving assistance system according to claim 6 or 11, further comprising:variably setting the threshold speed depending on a vulnerability level of the other road user.

14. A driving assistance program stored in a storage medium (10) for assisting driving of a host vehicle (2), the driving assistance program including instructions to be executed by a processor (12), acquiring virtual video data (Dm) representing the host vehicle virtually as seen from the viewpoint of another road user (3) present in an external world of the host vehicle; and controlling the host vehicle to perform a driving action corresponding to the virtual video data.

15. A driving assistance method executed by a processor (12) for assisting driving of a host vehicle (2), comprising: acquiring virtual video data (Dm) representing the host vehicle virtually as seen from the viewpoint of another road user (3) present in an external world of the host vehicle; and controlling the host vehicle to perform a driving action corresponding to the virtual video data.

Images