Method and apparatus for predicting future trajectories of surrounding vehicles for autonomous driving

The deep neural network structure addresses mode blurring in vehicle trajectory prediction by generating and updating encodings, improving risk assessment in autonomous driving systems.

KR103044747B1Active Publication Date: 2026-11-11ELECTRONICS & TELECOMM RES INST
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
KR1020230141326
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2023-07-19
Filing Date
2023-10-20
Publication Date
2026-11-11
Estimated Expiration
2043-10-20

AI Technical Summary

Technical Problem

Existing artificial neural networks for predicting the future trajectories of surrounding vehicles in autonomous driving systems suffer from mode blurring, where the predicted trajectories are distributed between adjacent lane centerlines, making it difficult to assess risk levels accurately.

Method used

A deep artificial neural network structure that generates past trajectory encodings and local path encodings, updates these encodings based on vehicle interactions, and predicts future trajectories using a combination of LSTM, self-attention, and generative models like VAE and GAN to minimize mode blurring.

Benefits of technology

The proposed method accurately predicts future vehicle trajectories by minimizing mode blurring, enhancing the accuracy of risk assessment in autonomous driving systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112023115506940-PAT00051_ABST
    Figure 112023115506940-PAT00051_ABST
Patent Text Reader

Abstract

The present invention relates to a method and apparatus for predicting the future trajectory of surrounding vehicles for autonomous driving. The method for predicting the future trajectory of surrounding vehicles according to the present invention comprises the steps of generating a past trajectory of surrounding vehicles, extracting a local path where surrounding vehicles can travel in a precision map, encoding the past trajectory and the local path, and generating a future trajectory of surrounding vehicles using a deep artificial neural network based on the encoded past trajectory and the updated local path encoding.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a method and apparatus for predicting the future trajectory of a vehicle surrounding an autonomous vehicle using an artificial neural network. Background Technology

[0002] A typical Autonomous Driving System (ADS) drives a vehicle through the processes of perception, judgment, and control.

[0003] In the recognition process, the autonomous driving system utilizes data acquired from sensors such as cameras and LiDAR to locate static and dynamic objects around the vehicle and track their positions. Additionally, the autonomous driving system recognizes lanes and surrounding buildings, compares them with a high-definition (HD) map, and predicts the position and attitude of the autonomous vehicle (hereinafter referred to as an 'autonomous vehicle' or 'autonomous car').

[0004] In the decision-making process, the autonomous driving system generates multiple paths aligned with the driving intent from the results of perception, assesses the risk of each path, and determines a single path. Finally, in the control process, the autonomous driving system controls the vehicle's steering angle and speed to enable the car to move along the path generated during the decision-making process.

[0005] Predicting the future movements of surrounding moving objects is essential for assessing risk by path in the decision-making process. For example, when changing lanes, an autonomous driving system must determine in advance whether a vehicle is present in the intended lane and whether that vehicle will cause a collision with the autonomous vehicle in the future; to achieve this, predicting the future movements of such vehicles is crucial.

[0007] With the advancement of Deep Neural Networks (DNNs), many techniques for predicting the future trajectory of moving objects using DNNs have been proposed. To predict the future trajectory of moving objects more accurately, DNNs are designed to satisfy the following conditions (see Fig. 1).

[0008] Condition (1). When predicting future trajectories, use high-precision maps or driving environment images.

[0009] Condition (2). Consider the interaction between moving objects when predicting future trajectories

[0010] Condition (3). Resolving ambiguity in moving object movements by predicting multiple future trajectories per object.

[0012] Condition (1) is applied to reflect that vehicles move mainly along lanes and people move along paths such as sidewalks. Condition (2) is applied to reflect that the movement of an object is influenced by the movement of other objects in the surroundings. Condition (3) is applied to reflect that the future position of an object follows a multimodal distribution due to the ambiguity of the object's movement intention.

[0014] Meanwhile, many artificial neural networks have been proposed to satisfy condition (3), but as shown in FIG. 2, the problem of the predicted trajectory being distributed in the area between adjacent lane centerlines frequently occurs. In the present invention, this phenomenon is named mode blur. Lane centerlines show a very high correlation with the mode of the vehicle's future trajectory distribution. The phenomenon of the future trajectory being distributed between two adjacent lane centerlines can be seen as occurring because the mode of the future trajectory distribution is blurred.

[0016] Figures 3a and 3b are reference diagrams for explaining the mode blurring phenomenon. When comparing the future trajectory distribution in Figure 3a with the future trajectory distribution in Figure 3b, it can be said that the mode blurring phenomenon is more severe in Figure 3b compared to Figure 3a. Although the modes of distribution (ma1, ma2, ma3, mb1, mb2, mb3) are formed at the lane centerlines (r1, r2, r3) in both Figure 3a and Figure 3b, the probability of future trajectories appearing between the lane centerlines is greater in the distribution of Figure 3b compared to the distribution of Figure 3a. As the mode blurring in the future trajectory distribution of surrounding vehicles intensifies, there is a problem in that it becomes difficult to assess the risk level for each path of the autonomous vehicle. The problem to be solved

[0018] The present invention aims to provide a deep artificial neural network structure capable of resolving mode blurring, a method for learning the same, and an apparatus for an artificial neural network for predicting the future trajectories of surrounding vehicles of an autonomous vehicle. More specifically, the invention aims to provide a structure of an artificial neural network for predicting multiple future trajectories of surrounding vehicles from past location records of surrounding vehicles and a precision map, and a method and apparatus for effectively learning said artificial neural network.

[0019] The objectives of the present invention are not limited to those mentioned above, and other unmentioned objectives will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0020] A method for predicting the future trajectory of a surrounding vehicle of an autonomous vehicle according to an embodiment of the present invention comprises: a step of generating a past trajectory of the surrounding vehicle based on the location of one or more surrounding vehicles; a step of extracting a local path that the surrounding vehicle can drive on a precision map based on the current location of the surrounding vehicle; a step of encoding the past trajectory and the local path to generate a past trajectory encoding and a local path encoding; a step of selecting a target vehicle among the surrounding vehicles and updating the local path encoding of the target vehicle by reflecting the correlation between the target vehicle and the local path of the target vehicle and the correlation between the target vehicle and the surrounding vehicles of the target vehicle; and a step of generating a future trajectory of the target vehicle based on the past trajectory encoding and the updated local path encoding.

[0021] A device for predicting the future trajectory of surrounding vehicles of an autonomous vehicle according to one embodiment of the present invention includes a memory for storing computer-readable commands and at least one processor implemented to execute said commands.

[0022] The above at least one processor is configured to generate a past trajectory of a surrounding vehicle based on the location of one or more surrounding vehicles by executing the above commands, extract a local path that the surrounding vehicle can drive on a precision map based on the current location of the surrounding vehicle, generate a past trajectory encoding and a local path encoding by encoding the past trajectory and the local path, select a target vehicle among the surrounding vehicles, update the local path encoding of the target vehicle by reflecting the correlation between the target vehicle and the local path of the target vehicle and the correlation between the target vehicle and the surrounding vehicles of the target vehicle, and generate a future trajectory of the target vehicle based on the past trajectory encoding and the updated local path encoding. Effects of the invention

[0023] According to one embodiment of the present invention, the future trajectory of a moving object around an autonomous vehicle can be predicted more accurately through the learning of a deep artificial neural network that minimizes mode blurring.

[0024] Figure 4 is a diagram showing an example of predicting the future trajectory of an object according to the present invention. In Figure 4, the gray circle and the black circle represent the past location and the actual future location, respectively. The rainbow-colored squares represent the locations of the future trajectories generated by the proposed invention. Finally, the red solid line represents the local paths that the object can choose. As seen in Figure 4, it can be seen that the probability distribution of the future trajectory reflects the shape of the local paths well.

[0025] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0026] Figure 1 is a diagram illustrating the conditions of a DNN for accurately predicting the future trajectory of a moving object. Figure 2 is a diagram illustrating the problem of the predicted trajectory being distributed in the area between adjacent lane centerlines. Figures 3a and 3b are reference diagrams for explaining mode blurring. FIG. 4 is a diagram showing an example of predicting the future trajectory of an object according to the present invention. FIG. 5 is a flowchart illustrating a method for predicting the future trajectory of surrounding vehicles of an autonomous vehicle according to an embodiment of the present invention. Figure 6 is a reference diagram to explain the process of extracting a local path from a precision map. Figure 7 is a flowchart illustrating the local path generation step of surrounding vehicles of an autonomous vehicle. Figure 8 is an example diagram of the local path generation of surrounding vehicles of an autonomous vehicle. Figure 9 is an example diagram showing the tangent vector of the lane and the heading vector of the vehicle. Figure 10 is a flowchart illustrating the steps for generating the future trajectory of surrounding vehicles of an autonomous vehicle. Figures 11a and 11b are reference diagrams regarding the update of local path encoding. FIGS. 12a to 12c are reference diagrams for explaining the calculation process required for data encoding and generating future trajectories for each vehicle. FIG. 13 is a block diagram showing a computer system for implementing a future trajectory prediction method according to an embodiment of the present invention. Specific details for implementing the invention

[0027] The present invention relates to a method and apparatus for training an artificial neural network to predict the future trajectories of surrounding vehicles of an autonomous vehicle. More specifically, the invention proposes a structure of an artificial neural network that predicts multiple future trajectories of surrounding vehicles from past location records of surrounding vehicles and a high-precision map, and introduces various techniques for effectively training the said artificial neural network.

[0028] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Meanwhile, the terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. The terms "comprises" and / or "comprising" as used in this specification do not exclude the presence or addition of one or more other components, steps, actions, and / or elements in addition to the mentioned components, steps, actions, and / or elements.

[0029] In describing the present invention, detailed descriptions of related prior art are omitted if it is determined that such descriptions may unnecessarily obscure the essence of the invention.

[0030] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate an overall understanding in describing the present invention, the same reference numerals will be used for the same means regardless of the drawing number.

[0032] FIG. 5 is a flowchart illustrating a method for predicting the future trajectory of surrounding vehicles of an autonomous vehicle according to an embodiment of the present invention.

[0033] The method for predicting the future trajectory of surrounding vehicles of an autonomous vehicle, illustrated in FIG. 5 (hereinafter referred to as the 'future trajectory prediction method'), consists of steps S100 to S400. The future trajectory prediction method illustrated in FIG. 5 is according to one embodiment, and the steps of the future trajectory prediction method according to the present invention are not limited to the embodiment illustrated in FIG. 5 and may be added, changed, or deleted as necessary.

[0034] It is assumed that the future trajectory prediction method illustrated in FIG. 5 is performed by a future trajectory prediction device (1000, hereinafter referred to as the 'future trajectory prediction device') for a vehicle surrounding an autonomous vehicle, according to one embodiment of the present invention. The future trajectory prediction device (1000) may be mounted on the autonomous vehicle or installed outside the autonomous vehicle. Furthermore, while this embodiment deals with a method and device for predicting the future trajectory of a vehicle surrounding an autonomous vehicle, it is understood that the present invention can be applied to any moving object in addition to vehicles surrounding the autonomous vehicle. However, for the sake of convenience of explanation, it is assumed that the future trajectory of a vehicle surrounding the autonomous vehicle (hereinafter referred to as a 'surrounding vehicle' or 'vehicle') is predicted.

[0035] The future trajectory prediction device (1000) predicts the future trajectory of a vehicle surrounding an autonomous vehicle through the learning and inference of a deep artificial neural network (DNN). In this embodiment, the deep artificial neural network (DNN) is composed of four modules, including a data receiving module, a local path generation module, a data encoding module, and a future trajectory generation module, and each of the four modules corresponds to steps S100 to S400 of FIG. 5. The future trajectory prediction device (1000) executes each module of the deep artificial neural network to perform steps S100 to S400.

[0036] Step S100 is a step of receiving location data for each vehicle. In this step, the future trajectory prediction device (1000) receives location data and attitude data of vehicles surrounding the autonomous vehicle and stores and manages them for each vehicle. The future trajectory prediction device (1000) receives location data and attitude data of vehicles surrounding the autonomous vehicle through sensors mounted on the autonomous vehicle or through communication with the outside (e.g., other vehicles or computer systems).

[0037] Step S200 is a step for generating a drivable area path for each vehicle. In this step, the future trajectory prediction device (1000) returns an area path (or lane centerline information) that the vehicle can move from a precision map using the vehicle's current position and attitude information.

[0038] Step S300 is a data encoding step. In this step, the future trajectory prediction device (1000) encodes the vehicle's location data and local path.

[0039] Finally, the S400 step is the vehicle-specific future trajectory generation step. In this step, the future trajectory prediction device (1000) predicts a future trajectory for each vehicle for which data has been received.

[0040] Although the future trajectory prediction method according to one embodiment of the present invention illustrated in FIG. 5 is depicted as being performed sequentially, the future trajectory prediction device (1000) can perform each step of the future trajectory prediction method simultaneously through parallel processing.

[0041] Below, each step of the above-mentioned future trajectory prediction method will be explained in detail.

[0042] In step S100, the future trajectory prediction device (1000) detects or receives position data and attitude data for one or more surrounding vehicles in real time, and stores and manages them for each vehicle. The attitude data includes heading angle data of the surrounding vehicles.

[0043] The future trajectory prediction device (1000) generates the past trajectory of a vehicle based on the vehicle's position data. Vehicle A that the future trajectory prediction device (1000) can obtain at the current time t. i The past of T obs The movement trajectory for seconds is X i =[ x (t-Hobs) , … , x t It is indicated by ]. Here x t =[x t ,y t ] is A at time t i It is the location of H obs =T obs * Sampling Rate (Hz). If a total of N vehicles are detected at the current time t, then [X1, … , X N You can obtain ]

[0044] In step S200, the future trajectory prediction device (1000) extracts a drivable regional path for each vehicle from a precision map based on the vehicle's position and heading angle at the current time.

[0045] The future trajectory prediction device (1000) predicts vehicle A at the current time t. i M local paths (or lane centerlines) L that a vehicle can select using the position and heading angle i 1:M Extract from the precision map. Here, L i n = [L0, … , L P ] refers to the nth local path among M local paths, and has a 2D coordinate value L p =[x p , y p It consists of a series of ]. FIG. 6 shows an example thereof. FIG. 6 shows a target vehicle (31) and lane centerlines (32a, 32b, 32c) around the target vehicle (31).

[0046] The lane centerline of a high-precision map consists of consecutive segments, and each segment is defined by two coordinate values. Each segment has a fixed start and end point and possesses a vector direction defined by these two coordinate values. Segments within the same lane centerline are connected to each other. If another segment B connected to segment A is ahead of A in the direction of travel, B is called the preceding segment of A, and conversely, A is called the following segment of B.

[0048] FIG. 7 is a flowchart illustrating the step of generating a local path for surrounding vehicles of an autonomous vehicle. As previously described, step S200 is a step of finding a local path that the vehicle can select from the vehicle's current position and heading angle, and step S200 can be subdivided as shown in the flowchart of FIG. 7. As illustrated in FIG. 7, in one embodiment of the present invention, step S200 includes steps S210 through S270.

[0050] FIG. 8 is an example diagram of the generation of a local path for surrounding vehicles of an autonomous vehicle. In FIG. 8, a straight line connecting two consecutive points is a segment. Each point has a coordinate value. First, the future trajectory prediction device (1000) finds all lane centerline segments within a radius K meters centered on the autonomous vehicle in a precision map (S210, search for lane centerline segments around the vehicle). Radius K is a value set in advance. Next, the future trajectory prediction device (1000) removes all segments where the angle difference with the vehicle's heading angle is greater than or equal to a specific value (S220, remove reverse segments). Next, the future trajectory prediction device (1000) combines segments that are connected to each other among the remaining segments into a single group (S230, merge forward segments). Next, the future trajectory prediction device (1000) utilizes the segment connection information of the precision map to find a segment that is connected prior to a segment within the segment group and adds it to that group to generate a local path (S240, expand lane centerline segments). At this time, the segment expansion ensures that the sum of the lengths of all segments within a group does not exceed a specific value. Next, the future trajectory prediction device (1000) resamples all connected segments within a group into specific length units and removes coordinate values ​​exceeding a specific distance so that all lane centerlines have the same length (S250, lane centerline resampling and length limiting). Then, duplicate centerlines are removed (S260, removal of duplicate lane centerlines).

[0051] Finally, the future trajectory prediction device (1000) adds a fake centerline with all coordinate values ​​of 0 if the number of finally selected lane centerlines (or local paths) is less than a specific value M. Conversely, if it is greater than M, it randomly removes centerlines so that the number of local paths for the vehicle becomes M (S270, adjustment of the number of local paths).

[0053] In step S300, the future trajectory prediction device (1000) encodes the vehicle's location data and local path. The future trajectory prediction device (1000) encodes the vehicle's past trajectory {X i}, correct answer future trajectory {Y i}, and local path {L i 1:M Data with continuous position coordinate values, as shown in}, is encoded through an LSTM (Long Short-Term Memory) network to generate past trajectory encoding, ground truth future trajectory encoding, and local path encoding. Encoding the ground truth future trajectory is necessary when training a deep artificial neural network.

[0054] Hereinafter, a method for a future trajectory prediction device (1000) to encode the trajectory and local path of a vehicle is described.

[0055] First, the future trajectory prediction device (1000) describes how to encode the trajectory of a vehicle. The future trajectory prediction device (1000) encodes the trajectory X of the vehicle i Preprocessing is performed before encoding. The future trajectory prediction device (1000) each position coordinate x t =[x t , y t Vehicle speed v in ] t (Unit m / s) and heading h t After calculating (unit radians), to the coordinate vector x t =[x t , y t , v t , h t Add as in ]. And the future trajectory prediction device (1000) preprocesses X using LSTM. i Encodes. The result of the encoding is It is defined as follows. The future trajectory prediction device (1000) uses the correct future trajectory Y for learning. iIn the case of encoding, similar to the vehicle trajectory encoding, preprocessing is performed and an LSTM-based encoding method is adopted. The result of the encoding is Defined as.

[0056] Next, the future trajectory prediction device (1000) describes how to encode a local path. The future trajectory prediction device (1000) encodes the nth local path L for the i-th vehicle. i n = [L0, ..., L P Preprocessing is performed before encoding ]. The future trajectory prediction device (1000) performs preprocessing on each coordinate L that constitutes the path. p =[x p , y p tangent vector Vp=[vx p , vy p ] and the direction of the tangent vector vd p After calculating (unit radians), Lp=[x in the coordinate vector p , y p , vx p , vy p , vd p Add as in ]. And the future trajectory prediction device (1000) is L, which has been preprocessed using LSTM. i n Encodes. The result of the encoding is Defined as.

[0057] The reason for performing preprocessing of the vehicle trajectory and lane centerline in this way is that the vehicle generally maintains its orientation parallel to the tangent vector of the lane centerline. Figure 9 shows an example of this.

[0059] In step S400, the future trajectory prediction device (1000) predicts the future trajectory of a vehicle surrounding the autonomous vehicle. The future trajectory prediction device (1000) selects a target vehicle among the vehicles surrounding the autonomous vehicle to predict the future trajectory of, updates the local path encoding of the target vehicle by reflecting the correlation between the target vehicle and its local path and the correlation between the target vehicle and the vehicles surrounding the target vehicle, and generates the future trajectory of the target vehicle based on the past trajectory encoding of the target vehicle and the updated local path encoding of the target vehicle.

[0060] In the present invention, the prediction of the future trajectory of surrounding vehicles recognized by the autonomous vehicle is performed independently. Therefore, step S400 can be performed simultaneously for each vehicle through parallel processing. Hereinafter, in specifically describing step S400, the i-th vehicle A i It is assumed that future trajectory prediction is performed using as the target vehicle. Therefore, the data required for future trajectory prediction is 1) the encoding of the target vehicle's past trajectory. , 2) Encoding the future trajectory of the target vehicle (for training) , 3) Encoding of the past trajectories of surrounding vehicles of the target vehicle , 4) Local path encoding am.

[0062] FIG. 10 is a flowchart illustrating the steps for generating the future trajectory of surrounding vehicles of an autonomous vehicle. As illustrated in FIG. 10, step S400 includes steps S410 through S430.

[0063] Step S410 is the local path encoding update step.

[0064] The future trajectory prediction device (1000) updates the local path encoding of the target vehicle by reflecting the interaction between the target vehicle and the local path, and the interaction between the target vehicle and the surrounding vehicles of the target vehicle, in the local path encoding of the target vehicle. The future trajectory prediction device (1000) predicts the future trajectory using each local path, or predicts the probability of moving to a specific local path, and the local path encoding to increase accuracy. Updates. For example, the encoding of the nth local path of the i-th vehicle. is updated as in mathematical formula 1.

[0066]

[0068] Here, [;] is the concatenation operation. Vehicle A i and local path L i 1:M It is a vector that reflects the interaction between them. Next is the nth path L i n It is a vector showing the representative movement of surrounding vehicles at a specific distance τ m or less from. First, α n It is calculated as in mathematical formula 2.

[0070]

[0072] Here is past trajectory encoding and local path encoding It is a weight representing the correlation between It satisfies. For example, A i In a situation where A changes from the nth lane to the mth lane, w m The value of becomes a value close to 1, and the remainder {w k} k≠m The values ​​will be close to 0. Therefore Local path encoding satisfying and updated by Equation 1 When used for future trajectory prediction, it becomes possible to accurately represent movements such as lane changes. The future trajectory prediction device (1000) and w through the attention operation between m You can obtain.

[0073] Next, the future trajectory prediction device (1000) is To find , first L i n All A existing at a distance less than or equal to a predetermined τ from i Select surrounding vehicles separately. Encode the past trajectories of the selected surrounding vehicles Let's say. Here, N i is a set whose elements are the indices of selected surrounding vehicles. Now, the future trajectory prediction device (1000) is A i encoding of past trajectory and The past trajectory encodings of surrounding vehicles are updated using [the method]. At this time, the update can be defined as self-attention, etc. That is, the future trajectory prediction device (1000) can update the past trajectory encodings of surrounding vehicles of the target vehicle using a self-attention technique. Using the updated past trajectory encodings, β n Finally, it is defined as in mathematical formula 3.

[0075]

[0077] Therefore, local path encoding updated by Equation 1 It includes information about surrounding vehicles along the nth local path, and consequently, when predicting the future trajectory, it reflects the interaction with surrounding vehicles, thereby increasing the accuracy of the future trajectory prediction.

[0078] FIGS. 11a to 11b are each α n and β nThis is a diagram illustrating the method for calculating . Fig. 11a shows α reflecting the weights of each local path. n Fig. 12b illustrates the process of calculating the past trajectory encoding of vehicles (62a, 62b) within a specific distance from the lane (61) and β after updating. n It shows the process of calculating.

[0080] Returning to Fig. 10, the steps S420 and below are explained.

[0081] Step S420 is a local path selection probability calculation step. In this step, the future trajectory prediction device (1000) uses the updated local path encoding to calculate the target vehicle A i The probability of selecting each local path is calculated. The future trajectory prediction device (1000) generates a driving environment context vector representing the local path through a concatenation operation based on the target vehicle's past trajectory encoding and the updated local path encoding. The driving environment context vector representing the nth local path It can be represented as follows. The future trajectory prediction device (1000) uses a Multi-Layer Perceptron (MLP) based on the driving environment context vector to determine the local path selection probability P as shown in Equation 4. i It can calculate.

[0082]

[0084] Here, P i The value of the nth element of is A i This represents the probability of selecting the nth local path.

[0086] Step S430 is a step for generating a future trajectory of a target vehicle. The future trajectory prediction device (1000) generates a future trajectory of a target vehicle by reflecting local path selection probabilities based on the past trajectory encoding of the target vehicle and the updated local path encoding of the target vehicle.

[0087] The future trajectory prediction device (1000) independently generates future trajectories from each local path. At this time, the future trajectory prediction device (1000) determines the number of future trajectories generated based on each local path, and the probability P that the local path is selected. i It is determined in proportion to (n). For example, A i Let's assume that 100 future trajectories are generated for . Also, P i (1)=0.9, P i (2)=0.1, P i (2)=...=P i Let us assume that (M)=0. In this case, the future trajectory prediction device (1000) has a local path L i 1 The driving environment context vector c corresponding to i 1 Generate 90 future trajectories from and local path L i 2 The driving environment context vector c corresponding to i 2 Generates 10 future trajectories from. And does not generate future trajectories from the remaining local paths.

[0089] The future trajectory prediction device (1000) has a driving environment context vector c i n When generating future trajectories using [this], methods that utilize latent variables, such as VAE (Variational AutoEncoder) or GAN (Generative Adversarial Networks), can be used.

[0090] The future trajectory prediction device (1000) can be calculated as follows when using VAE. First, two artificial neural network MLPs post and MLP prior Defines. Usually an MLP post is a recognizer network, MLP prioris called the prior network. Each network is the mean vector of a Gaussian probability distribution as shown in Equations 5 and 6. and standard deviation vector vector Creates.

[0092]

[0093]

[0095] MLP post is used during the training of deep artificial neural networks, and as shown in Equation 5, not only the driving environment context vector but also the correct answer future trajectory encoding ( It also takes ) as input and generates the mean and standard deviation vectors of the Gaussian probability distribution. MLP prior It is used when predicting future trajectories, and as shown in Equation 6, it receives a driving environment context vector as input and generates the mean vector and standard deviation vector of a Gaussian probability distribution.

[0096] Each mean and standard deviation vector is a Gaussian random vector and It is used to generate. An arbitrary random vector z is c i n A together with i It is used to generate a future trajectory that travels the nth local path, which can be implemented as shown in Equation 7. That is, the future trajectory prediction device (1000) generates a future trajectory of a target vehicle using an MLP decoder based on a Gaussian random vector and a driving environment context vector.

[0098]

[0100] For each local path, the future trajectory prediction device (1000) extracts a Gaussian random vector from a Gaussian probability distribution defined by the mean and standard deviation vectors generated by Equation 5 or Equation 6. At this time, the future trajectory prediction device (1000) can determine the number of random vectors to be extracted in proportion to the probability of selecting a local path. That is, the future trajectory prediction device (1000) generates a Gaussian random vector by reflecting the probability of selecting a local path based on the mean vector and standard deviation vector of the Gaussian probability distribution. Since future trajectories are generated as many times as the number of random vectors, the number of future trajectories generated based on each local path corresponds to the probability P that the corresponding local path will be selected. i It is determined in proportion to (n).

[0102] The future trajectory prediction device (1000) generates different random vectors and then generates a future trajectory using mathematical formula 7, thereby generating the mode of the probability distribution of the predicted future trajectory (local path L). i n The same effect as sampling a future trajectory from the mode of the probability distribution corresponding to it can be obtained.

[0103] Additionally, in the learning process of the artificial neural network proposed in this invention, z post Generates a future trajectory using, and during the testing process, z prior Generate a future trajectory using .

[0105] FIGS. 12a to 12c are reference diagrams for explaining the computational processes required for data encoding and generating future trajectories for each vehicle. FIGS. 12a to 12c summarize the entire process for generating future trajectories.

[0107] 'Feature Extraction (FE)' in Fig. 12a corresponds to data encoding. 'Scene Context Extraction (SCE)' through local path encoding updates It shows the process of updating. For reference, the updated according to mathematical formula 1 [α n ;β n It is identical to ]. Fig. 12b shows the process of calculating local path selection probabilities. And Fig. 12c is c n = It shows the process of generating a future trajectory from and a random variable z.

[0109] Below, a method for training a deep artificial neural network for future trajectory prediction using a future trajectory prediction device (1000) is described.

[0110] The future trajectory prediction device (1000) trains the deep artificial neural network so that the Evidence Lower Bound (ELBO) is minimized. Equation 8 represents the ELBO loss function.

[0111]

[0112] Here, β is an arbitrary constant and KL(||) represents KL divergence. Q and P are Gaussian distributions defined as the outputs (μ,σ) of the recognizer network and the prior network, respectively.

[0114] In addition, the future trajectory prediction device (1000) learns the probability that a vehicle selects a specific local path, and the future trajectory prediction device (1000) learns the local path selection probability such that a binary cross entropy loss function like Equation 9 is minimized.

[0116]

[0118] Here, P i is vehicle A i α is defined by mathematical formula 4 as the predicted value of the probability of selecting each local path. is the correct one-hot vector as Ai is a vector where the local path index of the actual path traveled at a future point in time is 1 and the remaining indices are 0. For example, A i If A moves to the 2nd regional path at a future point in time, This becomes.

[0119] Therefore, the loss that the future trajectory prediction device (1000) must ultimately minimize for the learning of the deep artificial neural network is defined as Equation 10.

[0121]

[0122] Here, λ is an arbitrary constant.

[0124] The aforementioned future trajectory prediction method has been described with reference to the flowchart presented in the drawings. For simplicity of explanation, the method has been illustrated and described in a series of blocks; however, the present invention is not limited to the order of said blocks, and some blocks may occur in a different order or simultaneously with other blocks as illustrated and described herein, and various other branches, flow paths, and sequences of blocks may be implemented to achieve the same or similar results. Furthermore, not all illustrated blocks may be required for the implementation of the method described herein.

[0126] Meanwhile, in the description with reference to FIGS. 5 to 12c, each step may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Also, some steps may be omitted as necessary, and the order between steps may be changed.

[0127] FIG. 13 is a block diagram showing a computer system for implementing a future trajectory prediction method according to an embodiment of the present invention. A future trajectory prediction device (1000) can be implemented in the form of the computer system of FIG. 13.

[0128] Referring to FIG. 13, a computer system (1000) may include at least one of a processor (1010), memory (1030), input interface device (1050), output interface device (1060), and storage device (1040) communicating via a bus (1070). The computer system (1000) may also further include a communication device (1020) coupled to a network. The processor (1010) may be a central processing unit (CPU) or a semiconductor device that executes instructions stored in memory (1030) or storage device (1040). Memory (1030) and storage device (1040) may include various forms of volatile or non-volatile storage media. For example, memory may include read-only memory (ROM) and random access memory (RAM). In the embodiments of this description, memory may be located inside or outside the processor, and memory may be connected to the processor through various known means. Memory is a volatile or non-volatile storage medium of various forms, and for example, memory may include read-only memory (ROM) or random access memory (RAM).

[0129] Accordingly, embodiments of the present invention may be implemented as a method implemented on a computer or as a non-transient computer-readable medium storing computer-executable instructions. In one embodiment, when executed by a processor, the computer-readable instructions may perform a method according to at least one aspect of the present description.

[0130] The communication device (1020) can transmit or receive wired or wireless signals.

[0131] In addition, the method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable medium.

[0132] The above computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for embodiments of the present invention, or they may be known and available to a person skilled in the art of computer software. The computer-readable recording medium may include a hardware device configured to store and execute program instructions. For example, the computer-readable recording medium may be magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROM; RAM; flash memory, etc. The program instructions may include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer through an interpreter, etc.

[0133] Meanwhile, the contents of FIGS. 5 to 12c may be applied to FIG. 13. For example, the processor (1010) performs the operations and operations presented as being performed by the future trajectory prediction device (1000) in FIGS. 5 to 12c, and the memory (1030) and storage device (1040) store data and information collected from the outside or generated through operations.

[0134] In addition, the content of FIG. 13 can be applied to the content of FIG. 5 to FIG. 12c.

[0136] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols

[0138] 1000: Future Trajectory Prediction Device 1010: Processor 1020: Communication device 1030: Memory 1040: Storage device 1050: Input interface device 1060: Output interface device 1070: Bus

Claims

Claim 1 A method for predicting the future trajectory of surrounding vehicles of an autonomous vehicle comprises: a step of generating a past trajectory of said surrounding vehicles based on the locations of one or more surrounding vehicles; a step of extracting a local path drivable by said surrounding vehicles from a precision map based on the current locations of said surrounding vehicles; a step of encoding said past trajectory and said local path to generate a past trajectory encoding and a local path encoding; a step of selecting a target vehicle among said surrounding vehicles and updating the local path encoding of said target vehicle by reflecting the correlation between said target vehicle and said target vehicle's local path and the correlation between said target vehicle and said target vehicle's surrounding vehicles. A method for predicting the future trajectory of surrounding vehicles of an autonomous vehicle, comprising the step of generating a future trajectory of the target vehicle based on the past trajectory encoding and the updated local path encoding, wherein the step of generating the future trajectory comprises: concatenating the past trajectory encoding and the updated local path encoding to generate a driving environment context vector for each local path; inputting the driving environment context vector into a pre-trained network composed of a first MLP to generate a mean vector and a standard deviation vector of a Gaussian probability distribution; inputting the driving environment context vector into a pre-trained second MLP to calculate a local path selection probability; for each local path, generating a random vector proportional to the local path selection probability using the mean vector and the standard deviation vector; and inputting the random vector and the driving environment context vector into a decoder composed of a third MLP to generate the future trajectory. Claim 2 A method for predicting the future trajectory of a surrounding vehicle of an autonomous vehicle, wherein, in claim 1, the step of extracting the local path is to extract the local path from the precision map based on the current position of the surrounding vehicle and the heading angle of the surrounding vehicle. Claim 3 A method for predicting the future trajectory of surrounding vehicles of an autonomous vehicle, wherein, in claim 1, the step of generating the past trajectory encoding and the local path encoding comprises encoding the past trajectory and the local path using an LSTM to generate the past trajectory encoding and the local path encoding. Claim 4 A method for predicting the future trajectory of surrounding vehicles of an autonomous vehicle, wherein the step of extracting the local path comprises: a step of searching for lane centerline segments within a predetermined radius centered on the autonomous vehicle in the precision map; a step of removing segments among the searched segments for which the angle difference with the heading angle is greater than or equal to a specific value; a step of merging segments connected to each other among the remaining segments into a group; and a step of searching for a preceding segment of a segment belonging to a group and adding it to the group to generate the local path. Claim 5 delete Claim 6 delete Claim 7 Memory that stores computer-readable commands; The system comprises at least one processor implemented to execute the above commands, wherein the at least one processor, by executing the above commands, generates a past trajectory of a surrounding vehicle based on the location of one or more surrounding vehicles, extracts a local path drivable by the surrounding vehicle on a precision map based on the current location of the surrounding vehicle, encodes the past trajectory and the local path to generate a past trajectory encoding and a local path encoding, selects a target vehicle among the surrounding vehicles, updates the local path encoding of the target vehicle by reflecting the correlation between the target vehicle and the local path of the target vehicle and the correlation between the target vehicle and the surrounding vehicles of the target vehicle, and generates a future trajectory of the target vehicle based on the past trajectory encoding and the updated local path encoding, wherein in the process of generating the future trajectory, the at least one processor concatenates the past trajectory encoding and the updated local path encoding to generate a driving environment context vector for each local path, and inputs the driving environment context vector into a pre-network composed of a pre-trained first MLP to a Gaussian probability distribution of A future trajectory prediction device for surrounding vehicles of an autonomous vehicle, configured to generate an average vector and a standard deviation vector, input the driving environment context vector into a pre-trained second MLP to calculate a local path selection probability, generate a random vector proportional to the local path selection probability using the average vector and the standard deviation vector for each local path, and input the random vector and the driving environment context vector into a decoder composed of a third MLP to generate the future trajectory. Claim 8 A future trajectory prediction device for a surrounding vehicle of an autonomous vehicle according to claim 7, wherein at least one processor is configured to extract a local path from a precision map based on the current position of the surrounding vehicle and the heading angle of the surrounding vehicle. Claim 9 A future trajectory prediction device for surrounding vehicles of an autonomous vehicle, wherein, in claim 7, the at least one processor is configured to generate the past trajectory encoding and the local path encoding by encoding the past trajectory and the local path using an LSTM. Claim 10 A future trajectory prediction device for surrounding vehicles of an autonomous vehicle according to claim 8, wherein at least one processor is configured to search for lane centerline segments within a predetermined radius centered on the autonomous vehicle in the precision map, remove segments among the searched segments whose angle difference with the heading angle is greater than or equal to a specific value, merge segments connected to each other among the remaining segments into a group, and search for preceding segments of segments belonging to the group and add them to the group to generate the local path. Claim 11 delete Claim 12 delete

Citation Information

Patent Citations

  • Drive supporting system using decision for intent of a moving object based on map

    KR1020170083233A

  • Systems and methods for trajectory prediction

    KR1020210074193A