A learning method for artificial neural networks to predict future trajectories of multiple moving objects for autonomous driving
The multi-object future trajectory prediction device uses a shared information generation module and type-specific modules to predict future trajectories for diverse objects, addressing inefficiencies in conventional DNNs by integrating LSTM, CNN, and VAE/MLP, thereby improving autonomous driving systems' accuracy and resource utilization.
Patent Information
- Application Number
- JP2023065693
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-06-28
- Filing Date
- 2023-04-13
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-04-13
AI Technical Summary
Conventional deep neural networks (DNNs) for autonomous driving systems are inefficient as they require separate networks for different types of objects, lacking the ability to share resources and effectively predict future trajectories of various moving objects around autonomous vehicles.
A multi-object future trajectory prediction device using a shared information generation module and future trajectory prediction module, which generates driving environment feature maps and predicts future trajectories based on past movement trajectories and type-specific modules, incorporating LSTM, CNN, and VAE/MLP for accurate trajectory prediction.
Enables the prediction of future trajectories for various types of objects, including vehicles and pedestrians, without the need for separate networks, enhancing the efficiency and accuracy of autonomous driving systems.
Smart Images

Figure 0007590488000011 
Figure 0007590488000012 
Figure 0007590488000013
Abstract
Description
[Technical field]
[0001] The present invention relates to a method for training an artificial neural network for predicting future trajectories of multiple moving objects around an autonomous vehicle, and more particularly to a method for effectively training the artificial neural network, which proposes a structure for an artificial neural network that predicts multiple future trajectories for each moving object from past position records and a high-resolution map of multiple moving objects. [Background technology]
[0002] A typical autonomous driving system (ADS) enables a vehicle to drive autonomously through the processes of recognition, judgment, and control.
[0003] In the recognition process, the autonomous driving system uses data acquired from sensors such as cameras and lidars to find static or dynamic objects around the vehicle and track their positions. The autonomous driving system also recognizes lanes and surrounding buildings, compares them with a high definition map (HD map), and then predicts the position and attitude of the autonomous vehicle (hereinafter referred to as the autonomous vehicle).
[0004] In the judgment process, the autonomous driving system generates multiple routes that match the driving intention from the recognition results, judges the risk level of each route, and decides on one route.
[0005] Finally, in the control process, the autonomous driving system controls the steering angle and speed of the vehicle so that the car moves along the path generated in the judgment process.
[0006] In the process of determining the risk level for each route in the autonomous driving system's decision-making process, it is essential to predict the future movement of surrounding moving objects. For example, when changing lanes, the autonomous driving system must determine in advance whether there is a vehicle in the lane to which the autonomous driving system is about to move and whether the vehicle will collide with the autonomous driving system in the future, and for this reason, it is very important to predict the future movement of the vehicle.
[0007] With the development of deep neural networks (DNNs), many techniques for predicting the future trajectory of moving objects using DNNs have been proposed. For more accurate future trajectory prediction, DNNs are designed to satisfy the following conditions (see Figure 1). (1) Use of high-resolution maps or images of the driving environment when predicting future trajectories (2) Consider interactions between moving objects when predicting future trajectories (3) Disambiguating the motion of moving objects by predicting multiple future trajectories for each object
[0008] Condition (1) is to reflect the situation where vehicles mainly move along lanes and people move along paths such as roads, and condition (2) is to reflect the fact that the movement of an object is influenced by the movement of surrounding objects. Finally, condition (3) is to reflect the point that the future position of an object follows a multi-modal distribution due to the ambiguity of the object's movement intention.
[0009] Meanwhile, there are various types of objects (vehicles, pedestrians, cyclists, etc.) around an autonomous vehicle, and the autonomous driving system must be able to predict the future trajectory of the object without any restrictions on the type. However, conventional DNNs have been proposed considering only certain types of objects, so when used in an autonomous driving system, a separate DNN must be used for each type of object. However, this DNN operation method has the problem that it is very inefficient because it is impossible to share resources between different DNNs. Summary of the Invention [Problem to be solved by the invention]
[0010] The present invention aims to propose a deep neural network (DNN) structure for predicting future trajectories of multiple objects and to present a method for effectively training the deep neural network.
[0011] The objects of the present invention are not limited to those mentioned above, and other objects not mentioned will become clearly understood by those skilled in the art from the following description. [Means for solving the problem]
[0012] To achieve the above object, a multiple object future trajectory prediction device according to one embodiment of the present invention includes: a shared information generation module that collects position information of one or more objects around an autonomous vehicle at a predetermined time, generates past movement trajectories for the one or more objects based on the position information, and generates a driving environment feature map for the autonomous vehicle based on road information around the autonomous vehicle and the past movement trajectories; and a future trajectory prediction module that generates future trajectories for the one or more objects based on the past movement trajectories and the driving environment feature map.
[0013] In one embodiment of the present invention, the shared information generation module can collect type information of the one or more objects, and the multi-object future trajectory prediction device includes a plurality of future trajectory prediction modules corresponding to each type that the type information may have.
[0014] In one embodiment of the present invention, the shared information generation module may include an object position data receiving unit that collects position information of the one or more objects and generates past movement trajectories for the one or more objects based on the position information, a driving environment context information generating unit that generates a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectory, and a driving environment feature map generating unit that inputs the driving environment context information image to a first convolutional neural network to generate the driving environment feature map.
[0015] In one embodiment of the present invention, the future trajectory prediction module may include an object past trajectory information extraction unit that generates a motion feature vector based on the past movement trajectory using a long short-term memory (LSTM), an object centered context information extraction unit that generates an object environment feature vector based on the driving environment feature map using a second convolutional neural network, and a future trajectory generation unit that generates the future trajectory based on the motion feature vector and the object environment feature vector using a variational auto-encoder (VAE) and MLP.
[0016] In one embodiment of the present invention, the driving environment context information generation unit may extract the road information including a lane centerline from a high-definition map, and generate the driving environment context information image in a manner of displaying the road information and the past movement trajectory on a 2D image.
[0017] In one embodiment of the present invention, the driving environment context information generation unit extracts the road information including a lane centerline from a high-resolution map, generates a road image based on the road information, generates a past movement trajectory image based on the past movement trajectory, and combines the road image and the past movement trajectory image in a channel direction to generate the driving environment context information image.
[0018] In one embodiment of the present invention, the object-centered context information extraction unit may generate a lattice template in which a plurality of position points are arranged in a lattice pattern, move all of the position points included in the lattice template to a coordinate system centered on the position and heading direction of a specific object, extract feature vectors from positions in the driving environment feature map corresponding to all of the moved position points to generate an agent feature map, and input the agent feature map to a second convolutional neural network to generate the object environment feature vector.
[0019] In one embodiment of the present invention, the object-centric context information extractor can set at least one of a horizontal interval and a vertical interval between position points included in the grid template based on a type of the specific object.
[0020] According to an embodiment of the present invention, a method for training an artificial neural network for predicting future trajectories of various objects includes a learning data generating step of generating past movement trajectories of one or more objects based on position information of the one or more objects within a predetermined distance range around an autonomous vehicle at a specific time point, generating a driving environment context information image for the autonomous vehicle by displaying road information around the autonomous vehicle and the past movement trajectories in a 2D image, and generating a correct future trajectory of the one or more objects based on position information of the one or more objects at a specific time point after the specific time point; inputting the past movement trajectories, the driving environment context information image, and the correct future trajectory into a deep neural network (DNN) to generate a future trajectory of the object, calculating a value of a loss function based on a difference between the future trajectory of the object and the correct future trajectory; and training the DNN to reduce the value of the loss function.
[0021] In an embodiment of the present invention, the learning data generating step may increase the driving environment context information image by at least one of inversion, rotation, and color change, or a combination thereof.
[0022] In one embodiment of the present invention, the loss function may be an evidence lower bound (ELBO) loss.
[0023] A method for predicting future trajectories of multiple objects according to an embodiment of the present invention includes the steps of collecting position information of one or more objects around an autonomous vehicle at a predetermined time, and generating past movement trajectories for the one or more objects based on the position information; generating a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectories; inputting the driving environment context information image into a first convolutional neural network to generate a driving environment feature map; generating a motion feature vector using a long short-term memory (LSTM) based on the past movement trajectories; generating an object environment feature vector using a second convolutional neural network based on the driving environment feature map; and generating future trajectories for the one or more objects using a variational auto-encoder (VAE) and an MLP based on the motion feature vector and the object environment feature vector.
[0024] The method for predicting future trajectories of multiple objects may further include a step of transforming the past trajectories into a coordinate system centered on each object, in which the step of generating a motion feature vector includes generating a motion feature vector using an LSTM based on the past trajectories transformed into the coordinate system centered on the object.
[0025] In one embodiment of the present invention, the step of generating the driving environment context information image may include extracting the road information including a lane centerline from a high-definition map, and generating the driving environment context information image in a manner of displaying the road information and the past movement trajectory on a 2D image.
[0026] In one embodiment of the present invention, the step of generating the driving environment context information image may include extracting road information including a lane centerline from a high-definition map, generating a road image based on the road information, generating a past movement trajectory image based on the past movement trajectory, and combining the road image and the past movement trajectory image in a channel direction to generate the driving environment context information image.
[0027] In one embodiment of the present invention, the step of generating the object environment feature vector may include generating a lattice template in which a plurality of position points are arranged in a lattice pattern, moving all of the position points included in the lattice template to a coordinate system centered on the position and heading direction of a specific object, extracting feature vectors from positions in the driving environment feature map corresponding to all of the moved position points to generate an agent feature map, and inputting the agent feature map to the second convolutional neural network to generate the object environment feature vector.
[0028] In one embodiment of the present invention, the step of generating the object environment feature vector may include setting at least one of the horizontal spacing and vertical spacing between position points included in the grid template based on the type of the specific object. Effect of the Invention
[0029] According to an embodiment of the present invention, future trajectories of various types of objects can be predicted regardless of the type of the object.
[0030] FIG. 2 is an example diagram showing prediction of future trajectories of a vehicle and a person in the same driving environment by the present invention. FIG. 2(a) shows the result of prediction of the future trajectory of a vehicle, and FIG. 2(b) shows the result of prediction of the future trajectory of a pedestrian. In FIG. 2, the large and small circles indicate the past trajectories of a vehicle and a pedestrian, respectively. The solid lines attached to the circles indicate the future trajectories of each object. As is clear from FIG. 2, the present invention can predict the future trajectories of various types of objects well.
[0031] The effects obtained from the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those having ordinary skill in the art to which the present invention pertains from the following description. [Brief description of the drawings]
[0032] [Figure 1] Diagram of design criteria for a deep neural network to predict the future trajectory of a moving object. [Diagram 2] An example diagram showing predicted future trajectories of a vehicle and a human in the same driving environment. [Diagram 3] 1 is a block diagram showing the configuration of a multi-object future trajectory prediction device according to an embodiment of the present invention; [Figure 4] 1 is a block diagram showing a detailed configuration of a multi-object future trajectory prediction device according to an embodiment of the present invention; [Figure 5A] 2D imagery of roadway centerlines and crosswalks. [Figure 5B] A 2D image of the object's past movement trajectory. [Figure 6] FIG. 13 is a diagram showing the process of extracting an agent feature map for a specific object from a driving environment feature map using a lattice template. [Figure 7] 13 is an example of a grid template and center points according to object type. [Figure 8] FIG. 1 shows the structure of a DNN for generating future trajectories of objects according to the present invention. [Figure 9]13 is a diagram showing a case where an arbitrary angle is added to a driving environment context information image to generate a new driving environment context information image. [Figure 10] 2 is a flow chart illustrating a method for training an artificial neural network to predict future trajectories of multiple objects according to one embodiment of the present invention. [Figure 11] 4 is a flowchart illustrating a method for predicting future trajectories of multiple objects according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0033] The advantages and features of the present invention, and the method of achieving the same, will become clear from the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be realized in various different forms, and the embodiments are provided merely to complete the disclosure of the present invention and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Meanwhile, the terms used in this specification are for the purpose of describing the embodiments, and are not intended to limit the present invention. In this specification, the singular form includes the plural form unless otherwise stated in the text. The terms "comprises" and / or "comprising" used in the specification should be interpreted as meaning that the components, steps, operations and / or elements referred to do not exclude the presence or addition of one or more other components, steps, operations and / or elements. In this specification, "movement" also includes "stopping". For example, even if an object is stopped, a "movement trajectory" of the object, which is the position sequence of the object over time, can exist.
[0034] In describing the present invention, if it is determined that a detailed description of such known techniques may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.
[0035] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, the same reference numerals will be used to refer to the same means regardless of the drawing numbers in order to facilitate overall understanding.
[0036] FIG. 3 is a block diagram showing the configuration of a multi-object future trajectory prediction device according to an embodiment of the present invention.
[0037] The multiple object future trajectory prediction device 100 according to an embodiment of the present invention is a device for generating future trajectories of objects by prediction based on object, road, and traffic condition information around an autonomous vehicle, and supports an autonomous driving system or is included in the autonomous driving system. The multiple object future trajectory prediction device 100 includes a shared information generation module 110 and a future trajectory prediction module 120, and may further include a learning module 130. The future trajectory prediction module 120 is composed of a plurality of modules according to the type of object. For example, if there are M types of objects, M future trajectory prediction modules 120-1, 120-2, ..., 120-M are included in the multiple object future trajectory prediction device 100 as the future trajectory prediction module 120.
[0038] The shared information generation module 110 generates past movement trajectories of objects based on position and attitude information (object information) of moving objects around the autonomous vehicle, and generates a driving environment feature map (scene context feature map) for the autonomous vehicle based on road / traffic information (e.g., lane information) around the autonomous vehicle and the past movement trajectories. The shared information generation module 110 can receive position and attitude information (e.g., heading angle) of moving objects around the autonomous vehicle from an object detection and tracking module (3D object detection&tracking module) of the autonomous vehicle, and generate past movement trajectories for multiple moving objects. For example, when the learning module 130 trains an artificial neural network related to prediction of future trajectories included in the multi-object future trajectory prediction device 100, the shared information generation module 110 acquires position and attitude information (e.g., 5 seconds) of moving objects from the object detection and tracking module of the autonomous vehicle in advance, and generates past movement trajectories (X trajectory) based on a part of the information (e.g., 2 seconds). i ), and based on a portion of the remaining time (e.g., 3 seconds), a correct future trajectory (Y) can be generated and transmitted to the future trajectory prediction module 120.
[0039] Here, it goes without saying that the position and attitude information of the moving object or the object movement trajectory data received from the object detection and tracking module of the autonomous vehicle may be manually corrected by a person or corrected by a pre-set algorithm.
[0040] The shared information generation module 110 can generate a driving environment context information image based on road / traffic information within a range of a predetermined distance from the position of the autonomous vehicle and the past movement trajectory of a moving object within the predetermined distance. The "driving environment context information" is information on roads and traffic conditions and objects around the autonomous vehicle while driving, and includes lanes, road signs, traffic signals, as well as types and movement trajectories of moving objects around the autonomous vehicle. The "driving environment context information image" refers to a representation of the "driving environment context information" in a 2D image. The shared information generation module 110 inputs the driving environment context information image into an artificial neural network to generate a driving environment feature map. Therefore, the "driving environment feature map" can be said to be a feature map in which the driving environment context information image is encoded.
[0041] The future trajectory prediction module 120 generates a future trajectory of an object based on the past movement trajectory of the object and the driving environment feature map. The future trajectory prediction module 120 generates a motion feature vector by encoding the past movement trajectory of the object, and generates an object environment feature vector based on the driving environment feature map. The "motion feature vector" is a vector in which information on the past movement trajectory of the object is encoded, and the "object environment feature vector" is a vector in which information on the road and traffic conditions around the object and the types and movement trajectories of other objects is encoded. Then, the future trajectory prediction module 120 generates a future trajectory of the object based on the motion feature vector, the object environment feature vector, and the random noise vector.
[0042] The learning module 130 trains the artificial neural networks included in the shared information generation module 110 and the future trajectory prediction module 120. The learning module 130 controls the shared information generation module 110 and the future trajectory prediction module 120 to progress the learning, and can increase the learning data as necessary.
[0043] FIG. 4 is a block diagram showing a detailed configuration of a multi-object future trajectory prediction device according to an embodiment of the present invention.
[0044] The shared information generation module 110 generates a driving environment feature map (scene context feature map, F) shared by various objects around the autonomous vehicle. The future trajectories of the objects are predicted by extracting an object-centric driving environment feature map from the shared information F. The future trajectory prediction module 120-K predicts the future trajectory of the objects by extracting the object-centric driving environment feature map from the shared information F. k If there are a total of M types of objects to be processed by the autonomous driving system, there will be a total of M future trajectory prediction modules.
[0045] The shared information generation module 110 includes an object position data receiving unit 111, a driving environment context information generating unit 112, and a driving environment feature map generating unit 113, and may further include a high-resolution map database 114. The functions of each component of the shared information generation module 110 will be described in detail below.
[0046] The object position data receiving unit 111 receives in real time the type, position and attitude information (hereinafter referred to as object information) of the moving objects around the autonomous vehicle detected in the recognition process, and stores and manages them for each object. i Past T obs The movement trajectory in seconds is X i =[x t-Hobs ,x t ] where x t= [x, y] is object A at time t i is the position of the object, which is generally expressed in the global coordinate system. obs =T obs *Sampling Rate (Hz). If a total of N objects are detected at the current time t, [X1, ,X N ] can be obtained. The object position data receiving unit 111 transmits the object movement trajectory information to the driving environment context information generating unit 112 and the future trajectory prediction module 120. If the future trajectory prediction module 120 is composed of a plurality of modules 120-1, 120-2, ..., 120-M according to the type of object, the object position data receiving unit 111 transmits the object movement trajectory information to the future trajectory prediction module that matches the type of object included in the object information. For example, if a specific future trajectory prediction module 120-K is a module that corresponds to "pedestrian" among the object types, the object position data receiving unit 111 transmits the object movement trajectory information whose object type is "pedestrian" to the future trajectory prediction module 120-K.
[0047] The driving environment context information generation unit 112 generates all lane information and past movement trajectories of objects [X1, . . . , X N ] is drawn on a 2D image of size H*W to generate a driving environment context information image (I).
[0048] FIG. 5A is an example of a 2D image of a lane centerline and a crosswalk. In order to obtain such an image, the driving environment context information generator 112 first obtains all lane centerline segments within a predetermined distance from the position of the autonomous vehicle at time t from a high-definition map. m = [l1, ,l M ] be the m-th lane centerline segment, where l k= [x, y] is the coordinates of the position points constituting the lane center line segment. m In order to visualize L, first, all position point coordinates in the segment are converted into a coordinate system centered on the position and heading of the autonomous vehicle at time t. Thereafter, the driving environment context information generating unit 112 converts L m A straight line is drawn on the image connecting the position coordinates within the range of 1 to 100. At this time, the driving environment context information generating unit 112 changes the color of the line depending on the direction of the line connecting two consecutive position coordinates. For example, k+1 and k The color of the line connecting and is determined as follows:
[0049] 1) Vector v connecting two coordinates k+1 =l k+1 -l k =[v x ,v y ], then the vector direction d=tan -1 (v y ,v x ) to calculate
[0050] 2) Determine the hue by dividing the vector direction (degrees) by 360, set saturation and value to 1, and then convert the (hue, saturation, value) values to (R, G, B) values.
[0051] The driving environment context information generating unit 112 converts the converted (R, G, B) values into k+1 and k The color of the straight line connecting the points is determined and drawn on the image. In Fig. 5A, a solid line indicates a red line, a dotted line indicates a green line, a dashed line indicates a blue line, and a two-dot dashed line indicates a yellow line (the same applies to Figs. 2, 6, and 9).
[0052] Next, the driving environment context information generator 112 draws the crosswalk segment on the same image or a different image. For example, the driving environment context information generator 112 may draw the crosswalk segment on the lane centerline image, or may draw the crosswalk segment on the crosswalk image after generating another crosswalk image. In the case of a crosswalk, it is drawn with a gray value of a specific brightness. For reference, when the driving environment context information generator 112 draws the crosswalk segment on the crosswalk image, the driving environment context information generator 112 combines the crosswalk image in the channel direction of the lane centerline image to form an image set.
[0053] The driving environment context information generating unit 112 may draw elements of the high-definition map other than the lane centerline and the crosswalk, and may determine different colors according to the direction as described above, or may draw them with a specific brightness gray value. When the driving environment context information generating unit 112 draws elements of the high-definition map on a separate image other than an existing image, the separate image on which the elements of the high-definition map are drawn is combined in the channel direction of the lane centerline image to form an image set. The driving environment context information generating unit 112 may receive and use elements of the high-definition map from an external source, or may extract and use elements of the high-definition map database 114. It goes without saying that the elements of the high-definition map include a lane centerline segment and a crosswalk segment.
[0054] Next, the driving environment context information generating unit 112 draws the past movement trajectory of the moving object on the image. FIG. 5B is an example of a 2D image related to the past movement trajectory of the object. The driving environment context information generating unit 112 draws the past movement trajectory of the moving object on the image. i Past movement trajectory of X i To draw on the image, we go through the following process. First, X i Transform all position coordinates in to a coordinate system centered on the position and heading of the autonomous vehicle at time t. Next, iEach position in the image is drawn in the shape of a specific figure such as a circle. At this time, positions close to the current time t are drawn bright, and positions farther away are drawn dark. Also, the shape or size of the figure is varied according to the type of object. The generated image is attached in the channel direction of the lane centerline image.
[0055] The size of the driving environment context information image (I) generated by the driving environment context information generator 112 can be expressed as H*W*C, where C is the same as the number of channels of the image generated by the driving environment context information generator 112.
[0056] The driving environment feature map generating unit 113 inputs the driving environment context information image (I) into a CNN (Convolutional Neural Network) to generate a driving environment feature map (scene context feature map, F). The CNN used in the driving environment feature map generating unit 113 may include a layer specialized for generating the driving environment feature map. In addition, a conventionally widely used neural network such as ResNet may be used as the CNN as is, or a conventionally known neural network may be partially modified to configure the CNN.
[0057] The future trajectory prediction module 120 includes a coordinate system conversion unit 121 , an object past trajectory information extraction unit 122 , an object center context information extraction unit 123 , and a future trajectory generation unit 124 .
[0058] If the number of types of objects processed by the autonomous driving system is M, there are a total of M future trajectory prediction modules 120 having the same structure. i Type is C k If so, the moving object A iThe future trajectory for is generated by the future trajectory prediction module 120-K. When there are multiple future trajectory prediction modules 120 (120-1, . . . , 120-M), the future trajectory prediction module 120-1 is configured to include a coordinate system conversion unit 121-1, an object past trajectory information extraction unit 122-1, an object center context information extraction unit 123-1, and a future trajectory generation unit 124-1, and the future trajectory prediction module 120-M is configured to include a coordinate system conversion unit 121-M, an object past trajectory information extraction unit 122-M, an object center context information extraction unit 123-M, and a future trajectory generation unit 124-M. Each future trajectory prediction module has the same basic function, except for the type of object to be processed. Below, the function of each component of the future trajectory prediction module 120 will be described in detail.
[0059] The coordinate system conversion unit 121 converts the past trajectory information of the object received from the shared information generation module 110 into an object-centered coordinate system, and transmits the object movement trajectory information in the object-centered coordinate system to the object past trajectory information extraction unit 122 and the object-centered context information extraction unit 123. The coordinate system conversion unit 121 converts all past position information of the object included in the past trajectory of the object into a coordinate system centered on the position and heading of the moving object at the current time t.
[0060] The object past trajectory information extraction unit 122 extracts object A i The past trajectory of is encoded using a long short-term memory (LSTM) network to generate a motion feature vector (m i The object past trajectory information extraction unit 122 generates the hidden state vector most recently output from the LSTM as object A. i The motion feature vector m i The hidden state vector is used as the current object A i It can be said that this vector reflects the past movement trajectory information of the object.
[0061] The object-centric context information extraction unit 123 extracts an agent feature map (F), which is a feature map for a specific object, from the driving environment feature map (F). i To this end, the object-centric context information extraction unit 123 performs the following tasks:
[0062] 1) The object-centered context information extraction unit 123 extracts a lattice template R=[r0,...,r K ], where r k =[r x ,r y ] denotes one location point in the lattice template. Figure 6(a) shows an example of a lattice template, where the black circle indicates the central location point r0 = [0,0] and the shaded circles are the remaining location points that are spaced apart by G meters from each other.
[0063] 2) Object A i All positions in the lattice template are moved to a coordinate system whose center is the position and orientation of the current time t. Figure 6(b) shows an example of this.
[0064] 3) Extract feature vectors from positions in the driving environment feature map (F) corresponding to each position point in the transformed lattice template to generate an agent feature map (F) for the object. i ) is generated. Figure 6(c) shows this process.
[0065] The object-centric context information extraction unit 123 extracts the agent feature map (F i) is input to a convolutional neural network (CNN) to generate the moving object scene feature vector (s i )
[0066] The object-centered context information extraction unit 123 may vary the distance between position points in the lattice template according to the type of object, resulting in different horizontal and vertical lengths of the lattice template. For example, in the case of a vehicle, since the front area is more important than the rear area, the vertical length may be made longer than the horizontal length, and the central position point may be located in the bottom area of the lattice template. FIG. 7 shows an example of this.
[0067] The future trajectory generator 124 calculates the motion feature vector (m i ), object environment feature vector (s i ), and a random noise vector (z), the object (A i The future trajectory generation unit 124 first generates future trajectory information of the motion feature vector (m i ), object environment feature vector (s i ), and a vector (f i ) is fed to MLP (multi-layer perceptron) to obtain the object (A i )'s future trajectory information (
number
number
[0068] The future trajectory generation unit 124 generates a random noise vector (z) using a variational auto-encoder (VAE) method. Specifically, the future trajectory generation unit 124 generates the random noise vector (z) using a neural network (NN) defined by an encoder and a prior. During learning, the future trajectory generation unit 124 generates the random noise vector (z) based on a mean vector and a variance vector generated by the encoder, and during testing, generates the random noise vector (z) based on a mean vector and a variance vector generated by the prior. The encoder and the prior are composed of a multi-layer perceptron (MLP).
[0069] The correct future trajectory (Y) is encoded by the LSTM network and the result is m i Y Then, the encoder generates a motion feature vector (m i ), object environment feature vector (s i ), the future trajectory of the encoded correct answer (m i Y ) are connected to the input, and the mean and variance vectors are output. The prior is also the motion feature vector (m i ), object environment feature vector (s i) to output the mean and variance vectors from the input.
[0070] The learning module 130 trains the artificial neural networks included in the shared information generation module 110 and the future trajectory prediction module 120. As shown in FIG. 8, the shared information generation module 110 generates a driving environment feature map (F) using a CNN based on a driving environment context information image (I), and the future trajectory prediction module 12 ... i ) and the driving environment feature map (F), we use LSTM, CNN, VAE (MLP), and MLP to predict the future trajectory of the object (
number
number
[0071] In equation (1), β is an arbitrary constant, KL(||) denotes the KL divergence, and Q and P are Gaussian distributions defined by the outputs (mean vector and variance vector) of the encoder and prior, respectively.
[0072] The learning module 130 may increase the learning data to improve the learning performance of the DNN. For example, the learning module 130 may increase the driving environment context information image (I) input to the CNN of the driving environment feature map generating unit 113 as follows to improve the learning effect of the DNN. To this end, the learning module 130 may control the driving environment context information generating unit 112.
[0073] (1) Horizontal inversion of the driving environment context information image (I): The image I used during learning is horizontally inverted. At the same time, the sign of the value of the y-component of the object's past movement position point (a direction rotated 90 degrees from the direction of travel of the autonomous vehicle) is changed. As a result, the learning data is doubled.
[0074] (2) When generating the driving environment context information image (I), an arbitrary angle ΔD (degrees) is added to the direction (degrees) of the line connecting two consecutive position coordinates within the lane centerline segment: As described above, the method for determining the color depending on the direction of the line connecting the two position coordinates is as follows.
[0075] 1) Vector v connecting two coordinates k+1 =l k+1 -l k =[v x ,v y ], then the vector direction d=tan -1 (v y ,v x ) to calculate
[0076] 2) Determine the hue by dividing the vector direction (degrees) by 360, set saturation and value to 1, and then convert the (hue, saturation, value) values to (R, G, B) values.
[0077] In the above process 1), an arbitrary angle ΔD is added to d, and then the new d' can be determined by dividing the value by 360. This can be summarized as in equation (2).
number
[0078] For reference, when the driving environment context information generating unit 112 generates one driving environment context information image (I), ΔD can be applied to all lane centerline segments. When generating the next image (I), ΔD can be changed to a random new value by the learning module 130. FIG. 9 is a diagram showing a case where a new driving environment context information image is generated by adding an arbitrary angle to the driving environment context information image. (a) shows the driving environment context information image (I) when ΔD=0, and (b) shows the driving environment context information image (I) when ΔD=90. It can be seen that the hue of the lane centerline etc. has changed due to the difference in hue value.
[0079] The learning module 130 can increase the learning data by the above-mentioned methods (1) and (2), and the DNN can learn to more easily recognize lanes in different directions. For example, by increasing the driving environment context information image (I) used for learning by adding an arbitrary angle ΔD (degree), the DNN can generate future trajectories based on the difference between hue values rather than the specific hue value itself.
[0080] FIG. 10 is a flow chart illustrating a method for training an artificial neural network to predict future trajectories of multiple objects according to one embodiment of the present invention.
[0081] A method for training an artificial neural network for predicting future trajectories of multiple objects according to an embodiment of the present invention includes steps S210, S220, and S230.
[0082] The artificial neural network, as described above, calculates the past movement trajectory of the object (X i ) and the driving environment context information image (I) to generate an object (A i )'s future trajectory information (
number
[0083] Step S210 is a learning data generation step. The multi-object future trajectory prediction device 100 generates past movement trajectory information (X i ) is generated. The multiple object future trajectory prediction device 100 can collect object information during a predetermined time range before a reference time t, and combine object position information included in the object information for each object in chronological order to generate past movement trajectory information for each object. The multiple object future trajectory prediction device 100 can convert the past movement trajectory information to conform to an object-centered coordinate system for DNN input. At this time, there may be multiple objects around the autonomous vehicle. In addition, the multiple object future trajectory prediction device 100 generates all lane information and past movement trajectories [X1,...,X N] is drawn on a 2D image of a size of H*W to generate a driving environment context information image (I). In the learning process according to this embodiment, the reference time t is a specific time point in the past. The multiple object future trajectory prediction device 100 can increase the driving environment context information image (I) used for learning by using methods such as the left-right inversion and ΔD summation described above. Furthermore, the multiple object future trajectory prediction device 100 can receive the trajectories of the objects after the reference time t (correct future trajectory, Y) and use them as learning data. Alternatively, the multiple object future trajectory prediction device 100 can combine position information of the objects at a predetermined time after the reference time t in chronological order to generate the trajectory of the object (correct future trajectory, Y). Data for DNN learning, i.e., learning data, is information on past movement trajectories of objects (X i ), the driving environment context information image (I), and the correct future trajectory (Y). For details about step S210, please refer to the above-mentioned contents about the shared information generation module 110, the future trajectory prediction module 120, and the learning module 130.
[0084] Step S220 is a step of inputting learning data to the DNN to generate future trajectory information and calculate a loss function value. i ), the driving environment context information image (I), and the correct future trajectory (Y) are input to the DNN to calculate the future trajectory of the object (
number
number
[0085] Step S230 is a DNN update step. The multi-object future trajectory prediction device 100 trains the DNN for multi-object prediction by adjusting the parameters (e.g., weighting) of each neural network in the DNN in a direction that minimizes the loss function value. For details about step S230, please refer to the above-mentioned contents about the learning module 130.
[0086] In the learning method according to this embodiment, steps S210 to S230 may be repeated, or only steps S220 and S230 may be repeated. Furthermore, if the loss function value is within a predetermined range as a result of proceeding with step S220, learning can be terminated without proceeding to step S230.
[0087] FIG. 11 is a flowchart illustrating a method for predicting future trajectories of multiple objects according to an embodiment of the present invention.
[0088] The method for predicting future trajectories of multiple objects according to an embodiment of the present invention includes steps S310 to S370.
[0089] Step S310 is a step for generating past trajectories of objects. The multiple object future trajectory prediction device 100 receives in real time the type, position and attitude information (object information) of moving objects around the autonomous vehicle detected in the recognition process, and stores and manages the information for each object. The multiple object future trajectory prediction device 100 generates past trajectories of objects based on the position information of the objects. Moving object A obtained at the current time t i Past T obsThe movement trajectory in seconds is X i =[x t-Hobs ,x t ] where x t = [x, y] is object A at time t i is the position of the object, which is generally expressed in the global coordinate system. obs =T obs *Sampling Rate (Hz). When a total of N objects are detected at the current time t, the multi-object future trajectory prediction device 100 calculates past movement trajectories [X1, ..., X N ] can be obtained.
[0090] Step S320 is a driving environment context information image generation step. The multi-object future trajectory prediction device 100 generates all lane information and past movement trajectories [X1, ..., X2, ...] of objects within a predetermined distance (e.g., R meters) from the position of the autonomous vehicle at the current time t. N ] is drawn on a 2D image having a size of H*W to generate a driving environment context information image I. For details on step S320, refer to the driving environment context information generating unit 112.
[0091] Step S330 is a driving environment feature map generation step. The multi-object future trajectory prediction device 100 inputs the driving environment context information image (I) into a CNN (Convolutional Neural Network) to generate a driving environment feature map (scene context feature map, F). The CNN used in step S330 may include a layer specialized for generating the driving environment feature map. In addition, a conventionally widely used neural network such as ResNet may be used as the CNN as is, or a conventionally known neural network may be partially modified to configure the CNN.
[0092] Step S340 is a step of converting the past movement trajectories of the object into an object-centered coordinate system. The multiple object future trajectory prediction device 100 converts the past movement trajectories of the object (past trajectory information of the object) into an object-centered coordinate system. Specifically, the multiple object future trajectory prediction device 100 converts all past position information of the object included in the past trajectory of the object into a coordinate system centered on the position and heading of the moving object at the current time t.
[0093] Step S350 is a step of generating a motion feature vector. As described above, a "motion feature vector" is a vector in which past movement trajectory information of an object is encoded. i The past trajectory of is encoded using a long short-term memory (LSTM) network to generate a motion feature vector (m i The multi-object future trajectory prediction device 100 generates a hidden state vector most recently output from the LSTM for object A. i The motion feature vector m i Used as.
[0094] Step S360 is a step of generating an object environment feature vector. As described above, the "object environment feature vector" is a vector in which information on the road and traffic conditions around the object and the types and movement trajectories of other objects is encoded. The multi-object future trajectory prediction device 100 generates an agent feature map (F), which is a feature map for a specific object, from the driving environment feature map (F). i To this end, the multi-object future trajectory prediction device 100 performs the following tasks.
[0095] 1) A lattice template R = [r0,...,r K ], where r k =[r x ,r y ] denotes one location point in the lattice template. Figure 6(a) shows an example of a lattice template, where the black circle indicates the central location point r0 = [0,0] and the shaded circles are the remaining location points that are spaced apart by G meters from each other.
[0096] 2) Object A i All positions in the lattice template are moved to a coordinate system whose center is the position and orientation of the current time t. Figure 6(b) shows an example of this.
[0097] 3) Extract feature vectors from positions in the driving environment feature map (F) corresponding to each position point in the transformed lattice template to generate an agent feature map (F) for the object. i ) is generated. Figure 6(c) shows this process.
[0098] The multi-object future trajectory prediction device 100 uses an agent feature map (F i ) is fed to a convolutional neural network (CNN) to obtain a moving object scene feature vector (s i )
[0099] The multi-object future trajectory prediction device 100 may vary the distance between position points in the lattice template according to the type of object, resulting in different horizontal / vertical lengths of the lattice template. For example, in the case of a vehicle, since the front area is more important than the rear area, the vertical length may be made longer than the horizontal length, and the central position point may be located in the bottom area of the lattice template.
[0100] Step S370 is a step of generating a future trajectory of an object. i ), object environment feature vector (s i ), and a random noise vector (z), the object (A i The multi-object future trajectory prediction device 100 first generates future trajectory information of the motion feature vector (m i ), object environment feature vector (s i ), and a vector (f i ) is fed to MLP (multi-layer perceptron) to obtain the object (A i )'s future trajectory information (
number
number
[0101] The multi-object future trajectory prediction device 100 generates a random noise vector (z) using a variational auto-encoder (VAE) technique. Specifically, the multi-object future trajectory prediction device 100 generates a random noise vector (z) using a neural network (NN) defined by an encoder and a prior. The multi-object future trajectory prediction device 100 generates a random noise vector (z) based on a mean vector and a variance vector generated by the encoder during learning, and generates a random noise vector (z) based on a mean vector and a variance vector generated by the prior during testing. The encoder and the prior are composed of a multi-layer perceptron (MLP).
[0102] The correct future trajectory (Y), which is the information for learning, is encoded by the LSTM network and the result is m i Y Then, the encoder generates a motion feature vector (m i ), object environment feature vector (s i ), the future trajectory of the encoded correct answer (m i Y ) are connected to the input, and the mean and variance vectors are output. The prior is also the motion feature vector (m i ), object environment feature vector (s i ) to output the mean and variance vectors from the input.
[0103] The method for training an artificial neural network for predicting future trajectories of multiple objects and the method for predicting future trajectories of multiple objects described above have been described with reference to the flowcharts shown in the drawings. For ease of explanation, the method has been illustrated and described as a series of blocks, but the present invention is not limited to the order of the blocks, and some blocks may occur in a different order or simultaneously with other blocks than those illustrated and described herein, and various other branches, flow paths, and orders of blocks that achieve the same or similar results are possible. Also, not all of the blocks shown may be required to implement the method described herein.
[0104] The above-mentioned artificial neural network learning method for predicting future trajectories of multiple objects and the method for predicting future trajectories of multiple objects can be linked together. That is, the prediction method can be performed after the DNN for predicting future trajectories of multiple objects according to the present invention is trained by the learning method.
[0105] Meanwhile, in the description with reference to Figures 10 to 11, each step may be further divided into additional steps or combined into fewer steps according to an embodiment of the present invention. Also, some steps may be omitted as necessary, and the order of steps may be changed. In addition, the contents of Figures 1 to 9 are applicable to the contents of Figures 10 to 11 even if other contents are omitted. Also, the contents of Figures 10 to 11 are applicable to the contents of Figures 1 to 9.
[0106] For reference, components according to embodiments of the present invention can be implemented in software or hardware form such as a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC) and can perform a given function.
[0107] However, the term "component" is not meant to be limited to software or hardware, and each component may be configured to reside on an addressable storage medium and to execute on one or more processors.
[0108] Thus, by way of example, components include components such as software components, object oriented software components, class components and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.
[0109] The components and the functionality provided within them may be combined into fewer components or further separated into additional components.
[0110] It will be understood at this point that each block of the flowchart illustrations and combinations of the flowchart illustrations can be implemented by computer program instructions. These computer program instructions can be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the functions described in the flowchart blocks. These computer program instructions can also be used with a computer or stored in a computer readable memory that can direct the computer or other programmable data processing apparatus to perform the functions in a particular manner, such that the instructions used with the computer or stored in a computer readable memory can produce an article of manufacture that includes instruction means for performing the functions described in the flowchart blocks. The computer program instructions can also be embodied on a computer or other programmable data processing apparatus such that a series of operational steps are performed on the computer or other programmable data processing apparatus to produce a computer implemented process, and the instructions executed on the computer or other programmable data processing apparatus provide the steps for performing the functions described in the flowchart blocks.
[0111] Also, each block may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function. Furthermore, it should be noted that in some alternative implementations, the functions noted in the blocks may occur out of order. For example, two blocks shown in succession may in fact be performed substantially simultaneously, or the blocks may sometimes be performed in reverse order depending on the corresponding functions.
[0112] At this time, the term "unit" or "module" used in this embodiment means a software or hardware component such as an FPGA or ASIC, and the "unit" or "module" plays some role. However, the "unit" or "module" is not limited to software or hardware. The "unit" or "module" may be configured to be in an addressable storage medium and may be configured to execute one or more processors. Thus, as an example, the "unit" or "module" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided in multiple components, "units" or "modules" may be combined into a smaller number of components, "units" or modules, or further separated into additional components and "units" or "modules". Moreover, the components, "units" and "modules" may be realized to execute one or more CPUs in a device or a security multimedia card.
[0113] Although the present invention has been described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and variations of the present invention can be made without departing from the spirit and scope of the present invention as set forth in the following claims. [Explanation of symbols]
[0114] 100: Multi-object future trajectory prediction device 110: Shared information generation module 111: Object position data receiver 112: Driving environment context information generation unit 113: Driving environment feature map generation unit 114: High-resolution map database 120: Future trajectory prediction module 121: Coordinate system conversion section 122: Object past trajectory information extraction unit 123: Object-centric context information extraction unit 124: Future trajectory generation unit 130: Learning Module
Claims
1. a shared information generation module that collects position information of one or more objects around an autonomous vehicle at a predetermined time, generates a past movement trajectory for the one or more objects based on the position information, and generates a driving environment feature map for the autonomous vehicle based on road information around the autonomous vehicle and the past movement trajectory; a future trajectory prediction module that generates a future trajectory for the one or more objects based on the past movement trajectory and the driving environment feature map; Including, The shared information generation module includes: an object position data receiving unit that collects position information of the one or more objects and generates past movement trajectories for the one or more objects based on the position information; a driving environment context information generating unit that generates a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectory; a driving environment feature map generating unit that inputs the driving environment context information image into a first convolutional neural network to generate the driving environment feature map; A device for predicting future trajectories of multiple objects.
2. The shared information generation module includes: Collecting type information of the one or more objects; The multi-object future trajectory prediction device comprises: a plurality of the future trajectory prediction modules corresponding to each type that the type information may have; The multi-object future trajectory prediction device according to claim 1 .
3. The future trajectory prediction module includes: an object past trajectory information extraction unit that generates a motion feature vector using a long-term memory (LSTM) based on the past movement trajectory; an object-centric context information extractor that generates an object environment feature vector using a second convolutional neural network based on the driving environment feature map; A future trajectory generation unit that generates the future trajectory using a variational auto-encoder (VAE) and an MLP based on the motion feature vector and the object environment feature vector. The multi-object future trajectory prediction device according to claim 1 .
4. The driving environment context information generation unit, extracting the road information including a lane centerline from a high-definition map, and generating the driving environment context information image in a manner of displaying the road information and the past movement trajectory on a 2D image; The multi-object future trajectory prediction device according to claim 1 .
5. The driving environment context information generation unit, extracting the road information including a lane centerline from a high definition map, generating a road image based on the road information, generating a past movement trajectory image based on the past movement trajectory, and combining the road image and the past movement trajectory image in a channel direction to generate the driving environment context information image; The multi-object future trajectory prediction device according to claim 1 .
6. The object-centric context information extraction unit generating a lattice template in which a plurality of position points are arranged in a lattice pattern, moving all of the position points included in the lattice template to a coordinate system centered on the position and heading direction of a specific object, extracting feature vectors from positions in the driving environment feature map corresponding to all of the moved position points to generate an agent feature map, and inputting the agent feature map into a second convolutional neural network to generate the object environment feature vector; The multi-object future trajectory prediction device according to claim 3 .
7. The object-centric context information extraction unit setting at least one of a horizontal interval and a vertical interval between position points included in the grid template based on the type of the specific object; The multi-object future trajectory prediction device according to claim 6.
8. a learning data generating step of generating a past movement trajectory for one or more objects based on position information for a predetermined time for the one or more objects within a predetermined distance range around the autonomous vehicle based on a specific time point, generating a driving environment context information image for the autonomous vehicle by displaying road information around the autonomous vehicle and the past movement trajectory in a 2D image, and generating a correct future trajectory for the one or more objects based on position information for the one or more objects for a predetermined time after the specific time point; inputting the past movement trajectory, the driving environment context information image, and the ground-truth future trajectory into a deep neural network (DNN) to generate a future trajectory of an object, and calculating a value of a loss function based on a difference between the future trajectory of the object and the ground-truth future trajectory; A step of training the DNN so as to reduce the value of the loss function; A method for training an artificial neural network to predict future trajectories of multiple objects, including
9. The learning data generating step includes: The driving environment context information image is increased by at least one of inversion, rotation, and color change, or a combination thereof; 9. A method for training an artificial neural network to predict future trajectories of multiple objects according to claim 8.
10. The loss function is ELBO (Evidence Lower Bound) loss, 9. A method for training an artificial neural network to predict future trajectories of multiple objects according to claim 8.
11. Collecting position information of one or more objects around the autonomous vehicle at a predetermined time, and generating past movement trajectories for the one or more objects based on the position information; generating a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectory; inputting the driving environment context information image into a first convolutional neural network to generate a driving environment feature map; generating a motion feature vector based on the past trajectory using a long short-term memory (LSTM); generating an object environment feature vector using a second convolutional neural network based on the driving environment feature map; generating future trajectories for the one or more objects based on the motion feature vectors and the object environment feature vectors using a variational auto-encoder (VAE) and MLP; A method for predicting future trajectories of multiple objects including:
12. The method further includes a step of transforming the past movement trajectory into a coordinate system centered on each object, The step of generating a motion feature vector includes: generating a motion feature vector using LSTM based on the past movement trajectory transformed into the coordinate system of the object center; The method for predicting future trajectories of multiple objects according to claim 11.
13. The step of generating a driving environment context information image includes: extracting the road information including a lane centerline from a high-definition map, and generating the driving environment context information image in a manner of displaying the road information and the past movement trajectory on a 2D image; The method for predicting future trajectories of multiple objects according to claim 11.
14. The step of generating a driving environment context information image includes: extracting the road information including a lane centerline from a high definition map, generating a road image based on the road information, generating a past movement trajectory image based on the past movement trajectory, and combining the road image and the past movement trajectory image in a channel direction to generate the driving environment context information image; The method for predicting future trajectories of multiple objects according to claim 11.
15. The step of generating the object environment feature vector comprises: generating a lattice template in which a plurality of position points are arranged in a lattice pattern, moving all of the position points included in the lattice template to a coordinate system centered on the position and heading direction of a specific object, extracting feature vectors from positions in the driving environment feature map corresponding to all of the moved position points to generate an agent feature map, and inputting the agent feature map into the second convolutional neural network to generate the object environment feature vector; The method for predicting future trajectories of multiple objects according to claim 11.
16. The step of generating the object environment feature vector comprises: setting at least one of a horizontal interval and a vertical interval between position points included in the grid template based on the type of the specific object; The method for predicting future trajectories of multiple objects according to claim 15.
Citation Information
Patent Citations
Moving body track prediction system
JP2018055141A
Mobile body behavior prediction device
JP2020071495A
Trajectory planing model training method and device, electronic apparatus, computer-readable storage medium, and computer program
JP2022003508A
