Autopilot Trajectory Prediction Method, System and Storage Medium

Abstract

The present invention provides an autonomous driving trajectory prediction method, system and storage medium. The method includes: S1 Scene encoding: Through a trajectory encoder and a map encoder, trajectory features and map features are respectively extracted from the input trajectory information and map information, and the interaction and fusion of the trajectory features and map features are realized through an interaction encoder to form scene features. Among them, the trajectory information includes the historical state of the agent, and the map information includes the lane center line in the high-precision map; S2 Online mapping: The sensor data of the input autonomous driving vehicle is encoded by an online mapping encoder to generate online mapping features; S3 Trajectory decoding: Based on the scene features obtained in step S1 and the online mapping features obtained in step S2, an iterative method is used to generate and optimize the predicted trajectory.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and particularly to an autonomous driving trajectory prediction method, system and storage medium. Background Art

[0002] Trajectory prediction is an important module in modern autonomous driving systems, responsible for observing the historical states of agents in traffic scenes and predicting future trajectories. Due to the complexity and multimodality of traffic scenes leading to uncertainty in the future, predicting accurate trajectories remains challenging. With the popularization of trajectory prediction models based on deep neural networks, the performance of trajectory prediction has been gradually improved. To provide the interaction between agents and traffic scenes, high-precision maps, as additional information about the historical states of agents, are widely used in trajectory prediction models.

[0003] However, the construction process of offline high-precision maps is complex and the labor cost is relatively high. To alleviate this problem, efforts have been made in the construction of end-to-end online high-precision maps in recent years. Currently, the online high-precision map construction method has achieved relatively reliable generation of vectorized map elements. Therefore, combining the trajectory prediction model with online mapping will be an important application direction.

[0004] Existing trajectory prediction models basically rely on offline-generated high-precision maps to provide traffic scene information. However, offline high-precision maps have defects such as complex construction processes and relatively high labor costs. In addition, the update cycle of high-precision maps is long, which restricts the expansion of the driving range of autonomous vehicles. Summary of the Invention

[0005] In order to reduce or even eliminate the dependence of the trajectory prediction model on offline high-precision maps, thereby improving the applicability of the trajectory prediction model. The present invention proposes an autonomous driving trajectory prediction method that combines online mapping features. By combining online mapping with the trajectory prediction model, it is possible to allow the trajectory prediction model to operate outside the range of high-precision maps while ensuring that the performance of the trajectory prediction model is not affected, greatly enhancing the traffic scene adaptation ability of autonomous vehicles.

[0006] To achieve the above object, the technical solution of the present invention provides an autonomous driving trajectory prediction method, which includes the following steps: S1 Scene Encoding: Through a trajectory encoder and a map encoder, trajectory features and map features are respectively extracted from the input trajectory information and map information, and the interaction and fusion of the trajectory features and map features are realized through an interaction encoder to form scene features, where the trajectory information includes the historical state of the agent, and the map information includes the lane centerlines in the high-precision map; S2 Online Mapping: The sensor data of the input autonomous driving vehicle is encoded through an online mapping encoder to generate online mapping features; S3 Trajectory Decoding: Based on the scene features obtained in step S1 and the online mapping features obtained in step S2, the predicted trajectory is generated and optimized in an iterative manner.

[0007] Further, in step S2, an online mapping encoder based on the bird's-eye view (BEV) is used to encode the sensor data to generate online mapping features in the BEV view.

[0008] Further, step S2 further includes: generating vectorized online map lane line elements from the online mapping features through an online mapping decoder as the input of the map encoder for explicit encoding, so as to supplement and correct the high-precision map.

[0009] Further, in step S2, the online mapping features are used as an implicit output for the trajectory decoding in step S3.

[0010] Further, step S3 specifically includes: First, the input scene features are used to generate the predicted trajectories of all agents through the trajectory decoder; Second, the predicted trajectories are interacted with the online mapping features through the mapping feature decoder to fine-tune the predicted trajectories; Finally, through an iterative manner, the predicted trajectories generated by the trajectory decoder are continuously updated to generate the final predicted trajectories.

[0011] Further, in step S3, a mapping feature decoder based on the DETR module is used, and the trajectory prediction position points are used as the reference points of the DETR decoder.

[0012] Further, in step S3, through coordinate transformation, the predicted trajectory position coordinates are transformed into the coordinate system corresponding to the BEV view, and normalized according to the range of the BEV view, and the predicted trajectory position points are transformed into reference points in the BEV view and projected onto the corresponding online mapping feature positions.

[0013] Further, in step S1, the historical state of each agent includes position, speed, acceleration, and heading, and the map lane centerlines are divided into a series of lane line vectors of equal length, and each lane line point feature includes position, orientation, and turning.

[0014] The technical solution of the present invention also provides an automatic driving trajectory prediction system, which includes the following modules: a scene encoding module, configured to extract trajectory features and map features from the input trajectory information and map information respectively through a trajectory encoder and a map encoder, and implement the interaction and fusion of the trajectory features and the map features through an interaction encoder to form scene features, wherein the trajectory information includes the historical state of the agent, and the map information includes the lane centerlines in the high-precision map; an online mapping module, configured to encode the sensor data of the input autonomous driving vehicle through an online mapping encoder to generate online mapping features; a trajectory decoding module, configured to generate and optimize the predicted trajectory iteratively based on the scene features obtained by the scene encoding module and the online mapping features obtained by the online mapping module.

[0015] The technical solution of the present invention also provides a computer-readable storage medium containing a computer program, which, when executed by one or more processors, executes the automatic driving trajectory prediction method as described above. Brief Description of the Drawings

[0016] Figure 1 is a flowchart of the automatic driving trajectory prediction method of the present invention. Detailed Embodiments

[0017] The technical solution of the present invention will be further described below in conjunction with specific embodiments, but the present invention is not limited to these embodiments.

[0018] Online mapping technology uses the on-vehicle sensors of autonomous driving vehicles to achieve real-time detection of lane lines and online construction of high-precision maps, which can effectively make up for the defects of offline high-precision maps. By combining online mapping with the trajectory prediction model, the present invention can allow the trajectory prediction model to run outside the range of the high-precision map while ensuring that the performance of the trajectory prediction model is not affected, greatly improving the traffic scene adaptation ability of autonomous driving vehicles.

[0019] As Figure 1 shown, the trajectory prediction method adopted by the present invention consists of a scene encoding module, an online mapping module and a trajectory decoding module.

[0020] In a specific embodiment, trajectory information and map information are used as the input information of the scene encoding module. The trajectory information is the historical state of the agent. For all agents included in the traffic scene, the historical state of each agent includes position, speed, acceleration, heading, etc. The map information is the center line of the lane in the high-precision map. The map lane center line is divided into a series of lane line vectors with equal lengths. Each lane line point feature includes position, orientation, and turning (going straight, turning left, turning right, and U-turn), etc. The trajectory information and the map information respectively extract trajectory features and map features through a trajectory encoder and a map encoder, and implement information interaction and fusion between the agent and the map lane line through an interaction encoding module, and finally form scene features as the input of the subsequent trajectory decoding module.

[0021] In the online mapping module, the original sensor data of the autonomous driving vehicle (such as cameras, radars, etc.) are used as the input information of the module. The online mapping model adopts the form of an encoder-decoder. An online mapping encoder based on the BEV (Bird's Eye View) is used to encode the sensor data to generate online mapping features in the BEV view. On the one hand, the online mapping features can generate vectorized online map lane line elements through an online mapping decoder, which are used as the input of the map encoder for explicit encoding, so as to supplement and correct the high-precision map. When the online mapping model reaches sufficient accuracy, the online map lane line elements can be used to replace the offline high-precision map while maintaining the overall performance of the trajectory prediction model. On the other hand, the online mapping features are implicitly output. Compared with the explicit vectorized lane line representation, the implicit features contain more information than just lane line features. Directly using the implicit online mapping features for the trajectory decoding module can provide richer features for trajectory prediction, thus improving the trajectory prediction performance.

[0022] The trajectory decoding module takes the scene features and the online mapping features as the input information of the module. The trajectory decoding module generates and optimizes the predicted trajectory in an iterative manner. First, the scene features generate the predicted trajectories of all agents through the trajectory decoder; second, the predicted trajectories interact with the online mapping features through the mapping feature decoder to fine-tune the predicted trajectories. Since the online mapping features are raster image features in the BEV view, and the predicted trajectories are temporal position features, in order to achieve the feature fusion and interaction between the two, this patent adopts a mapping feature decoder based on the DETR (Deformable Transformer) module, and uses the trajectory prediction position points as the reference points of the DETR decoder. Through coordinate transformation, the predicted trajectory position coordinates can be transformed into the coordinate system corresponding to the BEV view, and normalized according to the range of the BEV view, so that the predicted trajectory position points are transformed into reference points in the BEV view and projected onto the corresponding online mapping feature positions. Therefore, through the DETR decoder, the feature information of the online mapping features near the reference points can be extracted, that is, the corresponding online mapping features can be extracted according to the positions of the predicted trajectory points and used as the input of the trajectory decoder, so as to achieve the interaction between the predicted trajectory and the online mapping features. Finally, through an iterative manner, the predicted trajectory generated by the trajectory decoder is continuously updated to generate the final predicted trajectory.

[0023] The mapping feature decoder in the present invention adopts a form based on the DETR module, and uses the trajectory prediction position points to replace the reference points generated by the learnable encoding in the original DETR module, so that the online mapping features at the corresponding positions in the BEV view can be extracted according to the trajectory prediction positions, thus realizing the fusion of different modal trajectory features and BEV view features.

[0024] In an embodiment of the present invention, an autonomous driving trajectory prediction method is provided, which includes the following steps: S1 Scene encoding: Through the trajectory encoder and the map encoder, trajectory features and map features are respectively extracted from the input trajectory information and map information, and the interaction and fusion of the trajectory features and map features are realized through the interaction encoder to form scene features, where the trajectory information includes the historical states of the agents, and the map information includes the lane centerlines in the high-precision map; S2 Online mapping: The sensor data of the input autonomous driving vehicle is encoded through the online mapping encoder to generate online mapping features; S3 Trajectory decoding: Based on the scene features obtained in step S1 and the online mapping features obtained in step S2, the predicted trajectory is generated and optimized in an iterative manner.

[0025] Further, in step S2, an online mapping encoder based on the bird's-eye view (BEV) is used to encode the sensor data to generate online mapping features in the BEV view.

[0026] Further, step S2 further includes: generating vectorized online map lane line elements from the online mapping features through an online mapping decoder, and using them as inputs to the map encoder for explicit encoding, so as to supplement and correct the high-precision map.

[0027] Further, in step S2, the online mapping features are used as implicit outputs for step S3 trajectory decoding.

[0028] Further, step S3 specifically includes: First, generating the predicted trajectories of all agents from the input scene features through a trajectory decoder; Second, the predicted trajectories interact with the online mapping features through a mapping feature decoder to fine-tune the predicted trajectories; Finally, in an iterative manner, continuously update the predicted trajectories generated by the trajectory decoder to generate the final predicted trajectories.

[0029] Further, in step S3, a mapping feature decoder based on the DETR module is adopted, and the trajectory prediction position points are used as the reference points of the DETR decoder.

[0030] Further, in step S3, through coordinate transformation, the predicted trajectory position coordinates are transformed into the coordinate system corresponding to the BEV view, and normalized according to the range of the BEV view, and the predicted trajectory position points are transformed into reference points in the BEV view and projected onto the corresponding online mapping feature positions.

[0031] Further, in step S1, the historical state of each agent includes position, speed, acceleration, and heading. The map lane centerline is divided into a series of lane line vectors of equal length, and each lane line point feature includes position, orientation, and turning.

[0032] In another embodiment of the present invention, an autonomous driving trajectory prediction system is further provided, which includes the following modules: a scene encoding module for respectively extracting trajectory features and map features from the input trajectory information and map information through a trajectory encoder and a map encoder, and realizing the interaction and fusion of the trajectory features and map features through an interaction encoder to form scene features, where the trajectory information includes the historical state of the agent, and the map information includes the lane centerline in the high-precision map; an online mapping module for encoding the sensor data of the input autonomous driving vehicle through an online mapping encoder to generate online mapping features; a trajectory decoding module for generating and optimizing predicted trajectories in an iterative manner based on the scene features obtained by the scene encoding module and the online mapping features obtained by the online mapping module.

[0033] Further, the online mapping module uses an online mapping encoder based on the bird's-eye view (BEV) to encode the sensor data to generate online mapping features in the BEV view.

[0034] Furthermore, the online mapping module is also used to generate vectorized online map lane line elements from the online mapping features through an online mapping decoder, so as to be used as the input of the map encoder for explicit encoding, thereby supplementing and correcting the high-precision map.

[0035] Furthermore, the online mapping module uses the online mapping features as an implicit output for the trajectory decoding module to perform trajectory decoding.

[0036] Furthermore, the trajectory decoding module generates a predicted trajectory by performing the following steps: First, the input scene features are used to generate the predicted trajectories of all agents through the trajectory decoder; Second, the predicted trajectories interact with the online mapping features through the mapping feature decoder to fine-tune the predicted trajectories; Finally, in an iterative manner, the predicted trajectories generated by the trajectory decoder are continuously updated to generate the final predicted trajectory.

[0037] Furthermore, the trajectory decoding module adopts a mapping feature decoder based on the DETR module, and uses the trajectory prediction position points as the reference points of the DETR decoder.

[0038] Furthermore, the trajectory decoding module converts the predicted trajectory position coordinates to the coordinate system corresponding to the BEV view through coordinate transformation, and performs normalization processing according to the range of the BEV view, converts the predicted trajectory position points into reference points in the BEV view, and projects them to the corresponding online mapping feature positions.

[0039] Furthermore, in the scene encoding module, the historical state of each agent includes position, speed, acceleration, and heading, the map lane centerline is divided into a series of lane line vectors with equal lengths, and each lane line point feature includes position, orientation, and turning.

[0040] In other embodiments of the present invention, a computer-readable storage medium containing a computer program is also provided. When the computer program is executed by one or more processors, the above-mentioned autonomous driving trajectory prediction method is executed.

[0041] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the inventive concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. An automatic driving trajectory prediction method, characterized in that, It includes the following steps: S1 Scene Encoding: Through a trajectory encoder and a map encoder, trajectory features and map features are respectively extracted from the input trajectory information and map information, and the interaction and fusion of the trajectory features and map features are realized through an interaction encoder to form scene features. Among them, the trajectory information includes the historical state of the agent, and the map information includes the lane centerlines in the high-precision map; S2 Online Mapping: The sensor data of the input autonomous vehicle is encoded through an online mapping encoder to generate online mapping features; S3 Trajectory Decoding: Based on the scene features obtained in step S1 and the online mapping features obtained in step S2, the predicted trajectory is generated and optimized in an iterative manner; among them, Step S2 further includes: The online mapping features are used to generate vectorized online map lane line elements through an online mapping decoder, which are used as the input of the map encoder for explicit encoding, so as to supplement and correct the high-precision map; and In step S2, the online mapping features are used as an implicit output for the trajectory decoding in step S3; and Step S3 specifically includes: First, the input scene features are used to generate the predicted trajectories of all agents through a trajectory decoder; Second, the predicted trajectories are interacted with the online mapping features through a mapping feature decoder to fine-tune the predicted trajectories; Finally, through an iterative manner, the predicted trajectories generated by the trajectory decoder are continuously updated to generate the final predicted trajectories; and In step S3, a mapping feature decoder based on the DETR module is used, and the trajectory prediction position points are used as the reference points of the DETR decoder.

2. The automatic driving trajectory prediction method according to claim 1, wherein In step S2, an online mapping encoder based on the bird's-eye view (BEV) is used to encode the sensor data to generate online mapping features in the BEV view.

3. The automatic driving trajectory prediction method according to claim 2, wherein In step S3, through coordinate transformation, the predicted trajectory position coordinates are transformed into the coordinate system corresponding to the BEV view, and normalized according to the range of the BEV view, and the predicted trajectory position points are transformed into reference points in the BEV view and projected onto the corresponding online mapping feature positions.

4. The automatic driving trajectory prediction method according to any one of claims 1-3, characterized in that In step S1, the historical state of each agent includes position, speed, acceleration, and heading. The map lane centerlines are divided into a series of lane line vectors of equal length, and each lane line point feature includes position, orientation, and turning.

5. An automatic driving trajectory prediction system, characterized in that, Using the method described in any one of claims 1-4, the system includes the following modules: A scene encoding module for respectively extracting trajectory features and map features from the input trajectory information and map information through a trajectory encoder and a map encoder, and realizing the interaction and fusion of the trajectory features and map features through an interaction encoder to form scene features. Among them, the trajectory information includes the historical state of the agent, and the map information includes the lane centerlines in the high-precision map; An online mapping module for encoding the sensor data of the input autonomous vehicle through an online mapping encoder to generate online mapping features; A trajectory decoding module, which is used to generate and optimize a predicted trajectory in an iterative manner based on the scene features obtained by the scene encoding module and the online mapping features obtained by the online mapping module; wherein, The online mapping module generates vectorized online map lane line elements from the online mapping features through an online mapping decoder, and uses them as inputs to the map encoder for explicit encoding, so as to supplement and correct the high-precision map; And the online mapping module uses the online mapping features as an implicit output for the trajectory decoding module.

6. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is executed by one or more processors, it executes the autonomous driving trajectory prediction method according to any one of claims 1-4.

Images