Notification method and notification system

Abstract

A notification method according to the present invention is a notification method for a notification system in which: an image including the surrounding conditions of a vehicle is obtained by a sensor of the vehicle; at least one of a future trajectory and speed of the vehicle is predicted from the surrounding conditions being acquired; and the driver is notified of the trajectory and speed by image display or voice guidance. When at least one of a future trajectory and speed of the vehicle is predicted, the presence probability of the vehicle at each location where the vehicle may be present after a predetermined set time is calculated, and at least one of the future trajectory and speed of the vehicle is predicted on the basis of the presence probability being calculated.

Description

Notification method and notification system

[0001] The present invention relates to a notification method and a notification system.

[0002] Patent Literature 1 discloses a vehicle control system that allows occupants to easily grasp the timing of changes in vehicle behavior. This vehicle control system is configured to display an image indicating the occurrence of an event that involves a change in vehicle behavior, which is executed by an autonomous driving control unit, on a display unit in a first display mode at a first timing before the change in vehicle behavior. Also, at a second timing after the first timing but before the change in vehicle behavior, the system is configured to display the image on the display unit in a second display mode that emphasizes the change in vehicle behavior more than the first display mode.

[0003] Japanese Patent Application Laid-Open No. 2018-203011

[0004] In the above system, the change in vehicle behavior (route) during autonomous driving is displayed on the display unit as an image, but since only one route is proposed, the reliability of route selection for autonomous driving was not necessarily high. The present invention has been made to solve the above problem, and aims to provide a notification method and notification system that can increase the reliability of route selection during autonomous driving by considering multiple routes.

[0005] The notification method of the present invention is a notification method in a notification system that acquires an image including the situation around the vehicle using a vehicle sensor, predicts at least one of the vehicle's future trajectory and speed from the acquired situation around the vehicle, and notifies the driver of the trajectory and speed by image display or audio guidance.When predicting at least one of the vehicle's future trajectory and speed, the method calculates the probability of the vehicle's presence at each position where the vehicle may be located after a predetermined set time, and predicts at least one of the vehicle's future trajectory and speed based on the probability of presence.

[0006] According to the present invention, in automated driving, the reliability of route selection can be increased by considering multiple routes.

[0007] It is a block diagram showing a schematic configuration of a system according to an embodiment of the present invention. It is a flowchart showing processing in a notification system. It is a flowchart showing processing of a fusion calculation unit. It is a diagram showing an example of a future trajectory of a vehicle. It is a diagram showing an example of an image displayed on a display unit.

[0008] A notification system according to an embodiment of the present invention will be described below with reference to the drawings. The vehicle system is a notification system that predicts at least one of a future trajectory and a future speed of an autonomously driven vehicle and notifies the driver of the predicted trajectory and / or speed by visual display or audio guidance.

[0009] 1 is a block diagram of the notification system. This notification system includes an on-vehicle sensor 1, an information processing unit 2, a vehicle position detection device 3, a navigation device 4, a speaker 5, and a display unit 6. The notification system will be described in detail below.

[0010] Examples of the on-board sensor 1 include a front camera that captures images in front of the vehicle, side cameras that capture images of the left and right sides of the vehicle, a rear camera that captures images behind the vehicle, a front radar that detects obstacles in front of the vehicle, a rear radar that detects obstacles behind the vehicle, a side radar that detects obstacles on the left and right sides of the vehicle, and a vehicle speed sensor that detects the vehicle speed. Note that the on-board sensor 1 may be one of the above-mentioned multiple sensors, or a combination of two or more types of sensors. The detection results of the on-board sensor 1 are output to the information processing unit 2 at predetermined time intervals.

[0011] The information processing unit 2 is configured as a computer for predicting the vehicle's future trajectory and speed based on the vehicle's surrounding conditions acquired by on-board sensors. The information processing unit 2 has, for example, a control unit including a CPU, RAM, ROM, etc., and a storage unit configured with an auxiliary storage device such as an HDD or SSD. A program for notifying the driver is stored in the storage unit. The area in which the program is stored is not particularly limited, and the program can be stored in, for example, ROM. The program loaded into the RAM by the control unit is then interpreted and executed by the CPU. As a result, the information processing unit 2 functions as a computer including an image feature extraction unit 21, a trajectory generation unit 22, a fusion calculation unit 23, and a large-scale language model 24. These functional configurations will be described later.

[0012] The vehicle position detection device 3 is a positioning system for detecting the current position of the vehicle, and is not particularly limited and may be a known system. The vehicle position detection device 3 calculates the current position of the vehicle from radio waves received from a satellite for the GPS (Global Positioning System), for example.

[0013] The navigation device 4 is a device that refers to map information and calculates a driving route from the current position of the vehicle detected by the vehicle position detection device 3 to a destination set by the occupants (including the driver). The navigation device 4 searches for a driving route for the vehicle to reach the destination from the current position using road information, facility information, etc. from the map information. The driving route includes at least information on the route the vehicle will travel, the driving lanes, and the vehicle's driving direction, and is displayed, for example, linearly. There may be multiple driving routes depending on the search conditions. The driving route calculated by the navigation device 4 is output to the information processing unit 2, as will be described later.

[0014] The speaker 5 can be an audio speaker provided in the vehicle, and in this notification system, as will be described later, the predicted future trajectory and speed of the vehicle can be output from the speaker.

[0015] The display unit 6 is a device for providing necessary information to the vehicle occupants, and is, for example, a liquid crystal display provided on the instrument panel, a projector such as a head-up display (HUD), etc. The display unit 6 may also include an input device for the vehicle occupants to input instructions, questions, etc. to the notification system.

[0016] 2. Processing by the Information Processing Unit Next, processing by the functional configuration of the information processing unit 2 described above will be described with reference to the flow charts of Fig. 2 and Fig. 3. Fig. 2 is a flow chart showing notification processing in the notification system, and Fig. 3 is a flow chart showing processing by the fusion calculation unit.

[0017] 2, the on-board sensor 1 acquires images of the surroundings of the vehicle (e.g., images of the front of the vehicle including the sides) at predetermined time intervals (step S11). The image feature extraction unit 21 extracts feature vectors from the acquired images (step S12). The configuration of the image feature extraction unit 21 is not particularly limited, but for example, a visual encoder of CLIP (Contrastive Language Image Pre-training) can be used.

[0018] <2-2. Trajectory Generation Unit> The on-board sensor 1 acquires images of the surroundings of the vehicle at predetermined time intervals, and based on these images, the trajectory generation unit 22 calculates the future trajectory of the vehicle. The trajectory generation unit 22 uses the navigation device 4 to generate a driving route that the vehicle will travel by autonomous driving control from its current position to its destination.

[0019] Furthermore, the trajectory generation unit 22 obtains images of the surroundings and past driving history, and generates a future trajectory of the vehicle (step S13). At this time, the trajectory generation unit 22 generates at least one position (route) where the vehicle may be located after a predetermined time. Specifically, the trajectory of the vehicle estimated at every predetermined time is generated on a frame. For example, the trajectory Pi on the i-th frame is expressed as a Cartesian coordinate system in the future trajectory generated by the vehicle, with multiple coordinates Pi=((x1, y1), ..., (x N , y N)); i = (1, ..., T). This represents one trajectory. T is the total number of frames, and N is the number of coordinates in the future trajectory generated by the vehicle. The method for generating multiple trajectories is not particularly limited. For example, the trajectory generation unit 22 can be configured using a learning model that has been trained to output multiple trajectory candidates when past driving history, surrounding images, obstacles included in the images, etc. are input.

[0020] <2-3. Fusion Calculation Unit> As shown below, the fusion calculation unit 23 performs preprocessing of data to be input to the large-scale language model 24 (described later). The following processing will be explained with reference to FIG.

[0021] 3, first, the above-mentioned Cartesian coordinates are converted into polar coordinates Pi = ((r1, θ1), ..., (rN, θN)); i = (1, ..., T) (step S21). r of the vehicle indicates acceleration / deceleration, and θ indicates the steering angle. The reason for performing polar coordinate conversion is that separating the acceleration / deceleration information and the direction information is more suitable for learning to generate action descriptions, as they are directly linked to the accelerator and steering, which are the behavior of an autonomous vehicle.

[0022] <2-3-2. Standardization and Normalization> The following process corresponds to step S22. First, as shown in the following equation (1), each point on the trajectory is standardized by subtracting the mean value μ from the polar coordinate Pi and dividing by the standard deviation σ, as follows: Here, the mean is set to 0.

[0023] Next, as shown in the following formula (2), the absolute value of the minimum value after standardization is added to the standardized data, and the result is divided by the sum of the absolute values ​​of the maximum and minimum values ​​after standardization. This normalizes the polar coordinates to be between 0 and 1.

[0024] <2-3-3. Calculation of Presence Probability> The presence probability is the probability that a vehicle will be present at each possible position where the vehicle will be present after a predetermined set time. For example, FIG. 4 shows a future vehicle trajectory. At points close to the vehicle (e.g., times t1 and t2), the route is roughly determined, but at points farther from the vehicle (e.g., time tn), multiple routes are assumed. At least one trajectory at time tn shown in FIG. 4 corresponds to the i-th frame described above. Here, multiple presence probabilities are calculated for each point on the trajectory.

[0025] First, the values ​​of each point on the trajectory normalized as described above are used as the average value, and the existence probability is calculated so that the standard deviation based on previously collected data (e.g., past driving data) and the existence probability of multiple routes assumed at each point on the trajectory are inversely proportional to each other. The existence probability of multiple routes is generated to follow a normal distribution (e.g., preferably 2 sigma or more, more preferably 3σ or more, and particularly preferably 4σ or more). Here, the existence probabilities of multiple routes are normalized so that the sum of the existence probabilities is 1 (step S23). For example, the sum of the existence probabilities of the multiple points circled at time tn in FIG. 4 is 1.

[0026] As described above, the trajectory generation unit 22 generates at least one future route, but may present only one route with the highest probability of existence. In this case, as described above, the trajectory generation unit 22 calculates the probability of existence including routes other than the route with the highest probability of existence using data collected in the past.

[0027] The presence probability can also be recalculated based on newly acquired images, for example, when the surrounding situation changes, such as when another vehicle suddenly enters in front of the vehicle.

[0028] <2-3-4. Embedding> Next, the following processing is performed as an example of preprocessing (processing to standardize the data format) for assigning existence probabilities to the trajectory data described above and fusing it with the feature vector generated by the image feature amount extraction unit 21. From the trajectory data normalized as described above in Section 2-3-2, an all-zero vector of the same length as the hidden states of the feature vector generated by the image feature amount extraction unit 21 is generated, and the index of the generated vector is defined as the position where the vehicle is present using the normal distribution obtained in Section 2-3-3 above, and the existence probability at that position is filled in the index (step S24).

[0029] As will be described later, from among the multiple routes generated, the route with the highest probability of existence is selected as the trajectory.

[0030] <2-3-5. Fusion> Here, the feature vector generated by the image feature extraction unit 21 is fused with the data on the embedded trajectory to generate fused data to be input to the large-scale language model (step S25). This fusion is performed by the input data generation model.

[0031] The input data generation model, which receives trajectory information and image information, is trained to generate fusion data that associates an image associated with the feature vector with a corresponding future trajectory when a feature vector and a future trajectory are input. That is, the model generates fusion data that associates a generated trajectory with an image containing the trajectory. The input data generation model also includes an attention mechanism that weights features included in the feature vector that the vehicle should pay attention to. For example, when an image associated with the feature vector contains an object that has a high influence on trajectory generation (e.g., an obstacle close to the trajectory, a traffic light, or other noteworthy feature), the attention mechanism can be configured to weight these features more highly. The higher the influence of a feature, the higher the weighting. The influence level can be determined, for example, based on the proximity to the trajectory and the type of obstacle (e.g., a vehicle, a person, etc.).

[0032] This input data generation model can have, for example, three cross-attention layers (attention mechanisms). The first two cross-attention layers input the embedded trajectory described above as query (Q), and the key (K) and value (V) input feature vectors obtained from the image feature extraction unit 21. The third cross-attention layer fuses the acceleration / deceleration information / image feature fusion feature and the orientation information / image feature fusion feature output from each layer. This generates data to be input to a large-scale language model.

[0033] <2-4. Large-scale Language Models> The large-scale language models 24 are trained to output reasons for predicting at least one of the vehicle's future trajectory and speed from at least one path of the vehicle associated with the above-mentioned presence probability and the images acquired by the on-board sensor 1.

[0034] Specifically, as described above, when the fusion data output from the input data generation model is input, a character string indicating the following reasons is output (step S15). The reasons include reasons related to at least one of the selected route and speed. "Drive at a constant speed because there is a sufficient distance between you and the vehicle ahead." "Drive slowly because there is a vehicle stopped ahead on the left and it is difficult to see that area." "Drive closer to the right lane because there is a vehicle stopped ahead on the left."

[0035] The method for inputting fusion data into the large-scale language model 24 is not particularly limited, but for example, the data can be input into the large-scale language model 24 via a learning model that bridges the gap between images and text, such as a Q-former (Query Transformer).

[0036] The reason is output from the speaker 5 or displayed on the display unit 6 as shown in FIG. 5 (step S16). As shown in FIG. 5, the display unit 6 displays an image of the area ahead of the vehicle, the route and lane with the highest probability of existence superimposed on the image, and the above-mentioned reason. In this example, a vehicle is parked ahead on the left side, so the attention mechanism described above assigns a high weight to the feature vector of this vehicle as an obstacle. Therefore, the reason for the selected route takes this obstacle into consideration. The selected route can also be colored; for example, different colors can be used to color the route depending on acceleration and deceleration. In this case, the higher the acceleration, the darker the color. This allows the driver to visually see how much the vehicle will accelerate or decelerate along the route.

[0037] <3. Features> According to this embodiment, the following effects can be obtained. (1) Since the route with the highest existence probability is selected from multiple routes, it is possible to select an appropriate route from multiple options. At this time, since the existence probability is calculated to follow a normal distribution of 2σ or more, it is possible to cover 95% or more of possible routes. Therefore, it is possible to select a route with a higher degree of reliability.

[0038] (2) The reason for selecting a route is output, giving the driver a sense of security. In addition, since candidate routes are associated with their existence probability, the reason why a route other than the one with the highest existence probability was not selected can also be output as needed. Therefore, a more reliable reason can be shown to the driver.

[0039] (3) The input data generation model is equipped with an attention mechanism and is configured to weight features according to the objects shown in the acquired image. Therefore, when an obstacle such as the stopped vehicle described above is extracted, the weight is increased, and the output reason also reflects the obstacle. Therefore, the accuracy of the basis of the reason can be improved.

[0040] (4) By using the above-described fusion calculation unit 23, the dimensions of the acquired image and trajectory can be made the same when they are input. In other words, different types of data, namely, image and trajectory, can be appropriately input to a large-scale language model.

[0041] 4. Modifications Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the present invention. For example, the following modifications are possible. Furthermore, the gist of the following modifications can be combined with each other as appropriate, and can also be combined with the above embodiment.

[0042] (1) Although the input data generation model of the fusion calculation unit 23 is provided with an attention mechanism, the attention mechanism is not necessarily required.

[0043] (2) In the above embodiment, the data input to the large-scale language model is generated by a fusion calculation unit, but this is not limited to this. In the present invention, an image of the vehicle's surroundings and a route are input to the large-scale language model. However, the method of inputting the above data is not particularly limited as long as the large-scale language model outputs the reasons for route and speed selection.

[0044] (3) The large-scale language model 24 does not need to indicate the reasons for all trajectory selections, but may be configured to indicate the reasons for at least some of the trajectory selections (e.g., important reasons, reasons that were unlikely to be assumed, etc.). Furthermore, the large-scale language model 24 can be queried about the reasons for selecting the trajectory. For example, if the reasons for selecting the trajectory are not indicated, and an occupant such as a driver inquires about the reasons for selecting the trajectory by voice input or text input, the large-scale language model 24 can respond to the question by voice or text and indicate the reasons for selecting the trajectory.

[0045] (4) In the above embodiment, the route and speed reasons are notified using a large-scale language model, but this is not necessarily required and may be provided as needed. In other words, the notification system and notification method of the present invention are only required to be configured to select the vehicle's future trajectory from at least multiple routes based on the probability of existence. Furthermore, various learning models other than a large-scale language model may also be used.

[0046] 1: In-vehicle sensor (sensor) 2: Information processing unit 21: Image feature extraction unit 24: Large-scale language model 5: Speaker 6: Display unit

Claims

1. A notification method in a notification system that acquires an image including the situation around the vehicle using a vehicle sensor, predicts at least one of the vehicle's future trajectory and speed from the acquired image, and notifies the driver of the vehicle of at least one of the trajectory and speed by image display or audio guidance, wherein when predicting at least one of the vehicle's future trajectory and speed, the notification method calculates the vehicle's presence probability at each position where the vehicle may be located after a predetermined set time, and predicts at least one of the vehicle's future trajectory and speed based on the presence probability.

2. The notification method according to claim 1, wherein the future trajectory of the vehicle includes at least one candidate route, the existence probability is calculated for each candidate route, and the route with the highest existence probability is selected.

3. The notification method according to claim 1 or 2, wherein, when predicting at least one of the future trajectory and speed of the vehicle, a reason for predicting at least one of the future trajectory and speed of the vehicle is estimated, and the reason is notified to the driver by the image display or the voice guidance.

4. The notification method described in claim 3, wherein the reason for notifying the driver is estimated using a large-scale language model, and the large-scale language model is trained to output a reason for predicting at least one of the vehicle's future trajectory and speed from at least one path of the vehicle associated with the presence probability and the image acquired by the sensor.

5. The notification method according to claim 4, wherein the reasons include a reason why no route candidate other than the one with the highest probability of existence was selected when notifying the driver.

6. The notification method described in claim 4 or 5, further comprising: an image feature extraction unit that generates a feature vector from an image acquired by the sensor; and an input data generation model that generates data to be input to the large-scale language model, wherein the input data generation model is trained to associate the image related to the feature vector with the corresponding future trajectory when the feature vector and the future trajectory are input, and the input data generation model weights features included in the feature vector that the vehicle should pay attention to.

7. The notification method described in claim 6, wherein the input data generation model is configured to weight the feature more highly the influence of the part contained in the image related to the feature vector on the trajectory generation.

8. The notification method according to claim 2, wherein the presence probability is recalculated based on an image including the surrounding situation that is newly acquired after the presence probability is calculated.

9. A notification system comprising: a sensor that acquires the situation around a vehicle; and a prediction unit that predicts at least one of the vehicle's future trajectory and speed from the acquired situation around the vehicle, wherein the system is configured to notify the driver of the vehicle of at least one of the trajectory and speed by image display or audio guidance, wherein the prediction unit, when predicting at least one of the vehicle's future trajectory and speed, calculates the probability of the vehicle being present at each position where the vehicle may be located after a predetermined set time, and predicts at least one of the vehicle's future trajectory and speed based on the probability of presence.

Images