Obstacle trajectory prediction method, device, electronic device and storage medium

Abstract

The present invention discloses an obstacle trajectory prediction method, device, electronic device and storage medium, wherein the method comprises: obtaining original annotation data of a target vehicle within a preset perception range, wherein the original annotation data comprises trajectory point position data of candidate obstacles at multiple consecutive moments and positioning data of the target vehicle; converting the original annotation data from the original coordinate system to the world coordinate system to obtain target annotation data; determining the obstacle to be predicted according to the trajectory point position data of the candidate obstacles and the positioning data of the target vehicle in the target annotation data; predicting the future trajectory data of the obstacle to be predicted based on the pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted. The technical solution of the embodiment of the present invention improves the prediction effect of the obstacle trajectory prediction model by adopting data collected from a real vehicle, and improves the accuracy of obstacle trajectory prediction.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an obstacle trajectory prediction method, device, electronic device, and storage medium. Background Art

[0002] With the development of the internet and smart terminal technologies, autonomous driving technology is maturing, and self-driving vehicles are gradually becoming commonplace, bringing significant convenience to people's lives. To ensure safe travel, autonomous vehicles must accurately predict the future movements of other traffic participants. Based on the historical movement trajectories of vehicles, non-motorized vehicles, pedestrians, and other objects, as well as high-precision map information, autonomous driving systems predict the behavioral intentions and predicted trajectories of surrounding traffic participants, providing reference information for autonomous vehicle planning and decision-making.

[0003] Prior art methods typically use probability-based or motion-intention-based methods to calculate and predict the trajectory of obstacles around a vehicle. However, in real-world driving environments, the surrounding environment is complex, with numerous obstacles of varying types, high density, and small gaps between them. This can lead to inaccurate trajectory predictions, significantly impacting the ability of autonomous vehicles to accurately plan their own driving trajectories. Summary of the Invention

[0004] The present invention provides an obstacle trajectory prediction method, device, electronic device and storage medium to solve the problem of inaccurate prediction of the driving trajectory of obstacles around a vehicle.

[0005] According to one aspect of the present invention, a method for predicting obstacle trajectories is provided, the method comprising:

[0006] Obtaining original annotated data of the target vehicle within a preset sensing range, wherein the original annotated data includes trajectory point position data of candidate obstacles at multiple consecutive moments and positioning data of the target vehicle;

[0007] Convert the original annotation data from the original coordinate system to the world coordinate system to obtain the target annotation data;

[0008] Determine the obstacle to be predicted based on the trajectory point position data of the candidate obstacle in the target annotation data and the positioning data of the target vehicle;

[0009] The future trajectory data of the obstacle to be predicted is predicted based on the pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted.

[0010] According to another aspect of the present invention, there is provided an obstacle trajectory prediction device, the device comprising:

[0011] A data acquisition module is used to obtain the original annotated data of the target vehicle within the preset perception range, wherein the original annotated data includes the trajectory point position data of the candidate obstacle at multiple consecutive moments and the positioning data of the target vehicle;

[0012] The coordinate conversion module is used to convert the original annotation data from the original coordinate system to the world coordinate system to obtain the target annotation data;

[0013] An obstacle determination module is used to determine the obstacle to be predicted based on the trajectory point position data of the candidate obstacle in the target annotation data and the positioning data of the target vehicle;

[0014] The trajectory prediction module is used to predict the future trajectory data of the obstacle to be predicted based on the pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the obstacle trajectory prediction method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the obstacle trajectory prediction method according to any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention obtains original annotated data of the target vehicle within a preset perception range, then converts the original annotated data from the original coordinate system to the world coordinate system to obtain target annotated data. Furthermore, the obstacle to be predicted is determined based on the trajectory point position data of the candidate obstacle in the target annotated data and the positioning data of the target vehicle. Finally, the future trajectory data of the obstacle to be predicted is predicted based on a pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted. This solves the problem in the prior art of inaccurate obstacle trajectory prediction due to the complex environment around the vehicle, the large variety, high density, and small gaps between obstacles that affect vehicle driving. By using data collected from actual vehicles, the prediction effect of the obstacle trajectory prediction model is improved, and the accuracy of obstacle trajectory prediction is improved.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a flowchart of an obstacle trajectory prediction method provided according to the first embodiment of the present invention;

[0024] Figure 2 This is a flowchart of an obstacle trajectory prediction method provided according to the second embodiment of the present invention;

[0025] Figure 3 This is a flowchart of a method for processing original labeled data in an obstacle trajectory prediction method provided in Example 3 of the present invention;

[0026] Figure 4 This is an algorithm flow chart of an obstacle trajectory prediction method provided in accordance with the third embodiment of the present invention;

[0027] Figure 5 2 is a schematic structural diagram of an obstacle trajectory prediction device provided according to a fourth embodiment of the present invention;

[0028] Figure 6 The figure is a schematic diagram of the structure of an electronic device for implementing the obstacle trajectory prediction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Example 1

[0032] Figure 1 A flowchart of an obstacle trajectory prediction method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of predicting the driving trajectory of obstacles around a vehicle. The method can be executed by an obstacle trajectory prediction device, which can be implemented in the form of hardware and / or software and can be configured in a terminal and / or server. Figure 1 As shown, the method includes:

[0033] S110: Obtain original labeled data of the target vehicle within a preset perception range.

[0034] In this embodiment, the target vehicle may be the host vehicle for which surrounding obstacle trajectory prediction is required. The preset perception range may be a pre-set distance range for data information perception. It should be noted that the preset perception range is variable and can be set according to the perception distance of the data acquisition device. This embodiment does not limit the setting of the preset perception range. For example, the preset perception range may be 50 meters, 100 meters, or 200 meters. The raw annotated data may be data obtained after annotation processing. The raw annotated data may include multi-channel data such as positioning data, perception data, and point cloud data. Optionally, the raw annotated data includes trajectory point location data of candidate obstacles at multiple consecutive moments and the positioning data of the target vehicle. The candidate obstacles may be all traffic participants located around the target vehicle. For example, the candidate obstacles may be motor vehicles, non-motor vehicles, or pedestrians. The trajectory point location data may be the coordinate data of the candidate obstacles in a fixed coordinate system within their motion environment. The positioning data of the target vehicle is the position coordinate data of the target vehicle at a specific point in time.

[0035] Optionally, obtaining original annotated data of the target vehicle within a preset perception range includes: collecting original sampling data within the preset perception range through a radar sensing device set on the target vehicle, and annotating the original sampling data to obtain original annotated data.

[0036] The radar sensing device may be a device that identifies a target object by emitting microwaves and receiving reflected waves. For example, the radar detection device may be an ultrasonic radar, a laser radar, a microwave radar, or a millimeter wave radar.

[0037] In practical applications, a radar sensing device installed on a target vehicle can collect a series of point cloud images within its sensing range. According to the 3D point cloud data annotation standard, the candidate obstacles in the collected point cloud images are annotated with corresponding identity documents (IDs) and trajectory point coordinate data relative to the target vehicle at multiple consecutive moments, thus completing the annotation process and obtaining the original annotated data. The advantages of this setup are that different traffic participants can be distinguished based on identifiers, and data annotation can be used to label data that requires machine recognition, so that the terminal device can continuously learn the characteristics of this data, ultimately achieving the effect of the terminal device autonomously identifying the target data.

[0038] In a specific implementation, a radar sensing device installed on a target vehicle is used to scan the alternative obstacles within the sensing range to obtain the original sampling data of the target vehicle regarding the alternative obstacles. Then, the identifier of the alternative obstacle and the trajectory point position coordinate data relative to the target vehicle at multiple consecutive moments are determined by labeling the original sampling data. The labeled trajectory point position coordinate data of the alternative obstacle and the position data of the target vehicle are used as the original labeled data.

[0039] S120: Convert the original annotation data from the original coordinate system to the world coordinate system to obtain target annotation data.

[0040] In this embodiment, since the original annotation data is collected by a radar sensing device installed on the target vehicle, the position data of the target vehicle in the original annotation data is coordinate data located in the radar coordinate system. Since the trajectory point position data of the alternative obstacle in the original annotation data is relative coordinate data determined with the target vehicle as a reference object, the trajectory point position data of the alternative obstacle is coordinate data located in the main vehicle coordinate system. The main vehicle coordinate system and the radar coordinate system can be the original coordinate systems of the original annotation data.

[0041] Generally, when predicting the trajectory of all obstacles around a target vehicle, it is necessary to accurately predict their trajectory based on the road information around the obstacles and the relevant information about the obstacles in the world coordinate system. Therefore, it is necessary to convert the original annotation data from the original coordinate system to the world coordinate system. The world coordinate system uses a Cartesian right-hand coordinate system to determine the position of each point in the image. The X-axis is the horizontal direction, the Y-axis is the vertical direction, and the Z-axis is the direction perpendicular to the XY plane. The coordinates of the far point O are (0,0,0). The world coordinate system is a fixed coordinate system used to describe absolute and relative positions on the earth.

[0042] For example, the original annotation data can be converted from the original coordinate system to the world coordinate system through the conversion matrix. Taking the trajectory point position data of the alternative obstacle as an example, the coordinate conversion process is from the main vehicle coordinate system to the radar coordinate system, then from the radar coordinate system to the inertial measurement unit (IMU) coordinate system, and finally to the world coordinate system. Further, taking the conversion of the main vehicle coordinate system to the radar coordinate system as an example, the converted coordinates can be obtained based on the following formula:

[0043] new_position=R*[position,1]

[0044] Among them, new_positon represents the coordinate in the world coordinate system, position represents the coordinate in the main vehicle coordinate system, and R represents the transformation matrix.

[0045] In practical applications, the transformation matrix R is determined based on quaternions and displacements. Specifically, the transformation matrix is ​​determined based on the following formula:

[0046]

[0047] Among them, the quaternion is qua={q0,q1,q2,q3}, and the displacement is trans={t x ,t y ,t z}.

[0048] Optionally, the original annotation data is converted from the original coordinate system to the world coordinate system to obtain the target annotation data. The conversion process can be implemented based on the following formula:

[0049] world_position=R_IMU*R_LIDAR*R_AV*[act_position, 1]

[0050] Among them, world_position represents the trajectory point position data of the alternative obstacle in the world coordinate system, R_AV represents the conversion matrix from the main vehicle coordinate system to the radar coordinate system, R_LIDAR represents the conversion matrix from the radar coordinate system to the IMU coordinate system, R_IMU represents the conversion matrix from the IMU coordinate system to the world coordinate system, and act_position represents the trajectory point position data of the alternative obstacle in the original coordinate system.

[0051] It should be noted that the positioning data of the target vehicle in the original annotation data is also converted from the radar coordinate system to the world coordinate system through the above-mentioned coordinate conversion formula to obtain the positioning data of the target vehicle in the world coordinate system, and this data and the trajectory point position data of the alternative obstacle in the world coordinate system are used as the target annotation data.

[0052] S130 : Determine an obstacle to be predicted based on the trajectory point position data of the candidate obstacle in the target annotation data and the positioning data of the target vehicle.

[0053] In this embodiment, the obstacle to be predicted may be an obstacle that meets the prediction conditions among all obstacles around the target vehicle and requires trajectory prediction.

[0054] Generally speaking, among all the obstacles around the target vehicle, not every obstacle’s trajectory has predictive value. Therefore, it is necessary to determine the final obstacle to be predicted through various screening conditions, and other obstacles can serve as participants.

[0055] Optionally, the obstacle to be predicted is determined based on the trajectory point position data of the alternative obstacle in the target annotation data and the positioning data of the target vehicle, including: determining the perceived distance between the target vehicle and the alternative obstacle; determining the data missing rate of the trajectory point position data of the alternative obstacle; determining the trajectory length of the alternative obstacle; when at least one of the perceived distance, the data missing rate and the trajectory length meets a preset screening condition, the alternative obstacle is determined as the obstacle to be predicted.

[0056] The perception distance can be the range of surrounding obstacles detected by a radar sensor installed on the target vehicle. Generally, because the on-board radar sensor has a limited range and not all candidate obstacles can be accurately detected, it is necessary to determine the perception distance between the target vehicle and the candidate obstacle and determine whether it is a predicted obstacle based on the perception distance.

[0057] For example, the perceived distance between the target vehicle and the candidate obstacle can be determined based on the following formula:

[0058] L = dist{AV_position i ,positioni}

[0059] Among them, dist{·} represents the distance, AV_position i Represents the position data of the target vehicle at the i-th moment, position i Represents the trajectory point position data of the candidate obstacle at the i-th moment.

[0060] In this embodiment, the data missing rate can be the ratio of missing data due to unstable perception to the total data in the trajectory point position data of the candidate obstacle. Generally, due to the unstable perception of the on-board radar sensing device, some data may be missing during the process of collecting trajectory point position data. In order to ensure the accuracy of the final trajectory prediction result, it is necessary to ensure that the data missing rate of the trajectory point position data of the predicted obstacle is within a certain threshold.

[0061] Optionally, determining the data missing rate of the trajectory point position data of the candidate obstacle includes: determining the data missing rate of the candidate obstacle based on the following formula:

[0062] r=1-((COUNT(AGENT_track)) / CONUT(AV_track))

[0063] Where r represents the data missing rate of the candidate obstacle, COUNT(·) represents a counter, AGENT_track represents the trajectory point sequence of the candidate obstacle, and AV_track represents the trajectory point sequence of the target vehicle.

[0064] Generally speaking, when predicting the driving trajectory of obstacles around the target vehicle, the driving trajectory of obstacles in motion has predictive value. Therefore, it is necessary to determine the trajectory length of the alternative obstacles. The motion state of the alternative obstacles can be determined by the trajectory length. Obstacles with too short a trajectory length may be stationary and may not be considered in this embodiment.

[0065] Alternatively, the trajectory length of the candidate obstacle can be determined based on the following formula:

[0066] D = dist {position 起点 ,position 终点}

[0067] Among them, position 起点 Indicates the starting point position coordinates in the trajectory point position data of the candidate obstacle, position 终点 Indicates the end point position coordinates in the trajectory point position data of the candidate obstacle.

[0068] In this embodiment, the preset screening conditions may be pre-set conditions for screening the candidate obstacles to be predicted. For example, for the perception distance, the preset screening condition is that the perception distance is less than a certain distance threshold, for example, the distance threshold may be 100 meters; for the data loss rate, the preset screening condition is that the data loss rate is less than a certain loss rate threshold, for example, the loss rate threshold may be 70%; and for the trajectory length, the preset screening condition is that the trajectory length is greater than a certain length threshold, for example, the length threshold may be 10 meters.

[0069] Specifically, after obtaining the target annotation data that has completed the coordinate system conversion, the perceived distance between the candidate obstacle and the target vehicle, the data missing rate of the trajectory point position data, and the trajectory length of the candidate obstacle can be determined based on the trajectory point position data of the candidate obstacle in the target annotation data and the positioning data of the target vehicle. When at least one of the perceived distance, the data missing rate, and the trajectory length meets the preset screening conditions, the candidate obstacle can be determined as the obstacle to be predicted, so that the driving trajectory of the determined obstacle to be predicted can be predicted.

[0070] S140 , predicting future trajectory data of the obstacle to be predicted based on the pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted.

[0071] In this embodiment, the obstacle trajectory prediction model may be a neural network model for predicting the trajectory of obstacles around the target vehicle. Exemplarily, the obstacle trajectory prediction model may be composed of at least one of a convolutional neural network, a recurrent neural network, or a deep neural network, although this embodiment does not limit this.

[0072] Optionally, the obstacle trajectory prediction model may include a feature processing module, a feature fusion module, and a trajectory prediction module. The feature processing module can abstract the map information around the target vehicle into a topological network structure and process the position data of the trajectory points to be predicted into feature vectors. The feature fusion module extracts the trajectory information and map information of the obstacle to be predicted through a feature pyramid network (FPN) and a lane graph convolutional network (LGCN), respectively, and then performs feature fusion in sequence. The trajectory prediction module decodes through a multi-layer perceptron (MLP) to obtain the prediction results and corresponding confidence levels of several future trajectories, and uses the predicted trajectory with the highest confidence level as the output of the obstacle trajectory prediction model.

[0073] Specifically, after determining the obstacle to be predicted from all candidate obstacles around the target vehicle, the future trajectory data of the obstacle to be predicted can be predicted based on the pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted, so that the future driving trajectory of the obstacle to be predicted can be determined based on the prediction results, reducing the impact of the obstacle to be predicted on the driving trajectory of the target vehicle.

[0074] On the basis of the above technical solution, it also includes: processing the trajectory point position data of the predicted obstacle according to a preset data processing method, and saving it in a preset file format.

[0075] The preset data processing method may be a pre-set method for processing various data items. Optionally, the preset data processing method includes smoothing and interpolation. The preset file format may be a pre-set file format for saving trajectory point position data. For example, the preset file format may be a comma-separated values ​​(CSV) file format.

[0076] In practical applications, since the trajectory point position data of the obstacle to be predicted is the perception data collected by the vehicle-mounted radar sensing device, errors may occur, resulting in a certain degree of jitter in the driving trajectory of the obstacle to be predicted. Therefore, the trajectory point position data of the obstacle to be predicted needs to be smoothed.

[0077] Optionally, the trajectory point position data of the predicted obstacle is processed according to a preset data processing method, including: smoothing the trajectory point position data of the predicted obstacle based on a Gaussian filtering method.

[0078] In this embodiment, Gaussian filtering is essentially a signal filter used for signal smoothing. For example, there are two common implementations of Gaussian filtering: discretized window sliding convolution and Fourier transform. This embodiment uses discretized window sliding convolution to smooth the trajectory point position data.

[0079] Optionally, the Gaussian kernel in the Gaussian filtering method is determined based on the following formula:

[0080]

[0081] Among them, G σ represents the Gaussian kernel, x, y represent the coordinates of the trajectory point, and σ represents the hyperparameter.

[0082] In practical applications, the sampling frequency of the original annotation data collected by the radar sensing device is 3 Hz, while the sampling frequency of the input data in the obstacle trajectory prediction model is 10 Hz. In order to make up for the missing data, it is necessary to interpolate the trajectory point position data of the obstacle to be predicted obtained by the radar sensing device. Exemplary data interpolation methods may include but are not limited to inverse distance weighted interpolation, minimum curvature, natural neighbor interpolation, nearest neighbor interpolation or cubic spline interpolation, etc. For example, this embodiment uses the cubic spline interpolation method to expand the trajectory point position data. The specific method is to divide the original trajectory point position data into n intervals, and in the interval [T i ,T i+1 ](i∈(0,…,n-1)) is used to fit the coordinates of X and Y. Taking the fitting of X coordinate as an example, the cubic polynomial is S=a i +b i x+c i x 2 +dx 3 , in the interval [T i ,T i+1 ] satisfies S(T i )=X i , and S(T i+1 )=X i+1 , then the inserted data is (T,,X,), where,T,=T+0.1,X,=S(T,).

[0083] Specifically, after the trajectory point position data of the obstacle to be predicted is smoothed and interpolated, the processed trajectory point position data is saved in a pre-set file format so that it can be subsequently input into the obstacle trajectory prediction model for trajectory prediction.

[0084] The technical solution of the embodiment of the present invention obtains original annotated data of the target vehicle within a preset perception range, then converts the original annotated data from the original coordinate system to the world coordinate system to obtain target annotated data. Furthermore, the obstacle to be predicted is determined based on the trajectory point position data of the candidate obstacle in the target annotated data and the positioning data of the target vehicle. Finally, the future trajectory data of the obstacle to be predicted is predicted based on a pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted. This solves the problem in the prior art of inaccurate obstacle trajectory prediction due to the complex environment around the vehicle, the large variety, high density, and small gaps between obstacles that affect vehicle driving. By using data collected from actual vehicles, the prediction effect of the obstacle trajectory prediction model is improved, and the accuracy of obstacle trajectory prediction is improved.

[0085] Example 2

[0086] Figure 2 This is a flowchart of an obstacle trajectory prediction method provided in the second embodiment of the present invention. This embodiment is a refinement of S140 in the above embodiment. Figure 2 As shown, the method includes:

[0087] S210: Obtain original labeled data of the target vehicle within a preset perception range.

[0088] S220: Convert the original annotation data from the original coordinate system to the world coordinate system to obtain target annotation data.

[0089] S230: Segment the trajectory point position data of the candidate obstacle based on a preset sliding window.

[0090] In this embodiment, the preset sliding window may be a pre-set window that can slide two boundaries in a certain direction. A sliding window is an abstract concept commonly used in array or string problems. A window is typically a set of elements in an array or string defined by a start and end index.

[0091] In practical applications, the model input data of the obstacle trajectory prediction model is data of fixed size. However, the length of each segment of the trajectory point position data of the candidate obstacle is greater than the segment length of the model's input data. Therefore, the trajectory point position data of the candidate obstacle needs to be segmented. Exemplarily, the segmented data in each preset sliding window includes the historical trajectory point position data of the candidate obstacle and the predicted real trajectory point position data. For example, the model input data of the obstacle trajectory prediction model is 5 seconds of data, of which the first 2 seconds are historical trajectory point position data and the last 3 seconds are predicted real trajectory point position data, while the trajectory point position data of the candidate obstacle is 50 seconds of data. Therefore, it is necessary to segment the trajectory point position data of all candidate obstacles according to the window size of 5 seconds.

[0092] S240: Determine an obstacle to be predicted based on the trajectory point position data of the candidate obstacle in the target annotation data and the positioning data of the target vehicle.

[0093] S250: Obtain map data of obstacles to be predicted within a preset search range.

[0094] The preset search range may be a pre-set range threshold for searching map data around an obstacle to be detected. For example, the preset search range may be 50 meters, 100 meters, or 200 meters.

[0095] In this embodiment, the map data may be lane data around the obstacle to be predicted. Optionally, the map data includes at least one of lane coordinates, whether the lane turns left, whether the lane turns right, whether it is a traffic intersection, and whether traffic control is in effect. For example, for each lane around the obstacle to be predicted, the lane may be divided into a number of road nodes according to a preset sampling distance. For example, each lane may be sampled at a distance of 1 meter, i.e., lane = {node0,…,node i-1 ,node i}, i = length(lane) / 1, where lane represents the lane around the obstacle to be predicted, node represents the road node on each lane, and each node includes information such as the location coordinates of the lane road node, whether to turn left, whether to turn right, whether it is a traffic intersection, and whether traffic control is in effect.

[0096] In practical applications, due to the complexity and diversity of the actual prediction scenarios in which the obstacles to be predicted are located, and the map feature information around the obstacles to be predicted in the prediction scenarios is of great value in predicting their driving trajectories, it is necessary to obtain the map data around the obstacles to be predicted after determining the obstacles to be predicted, so as to make the final trajectory prediction results of the obstacles to be predicted more accurate.

[0097] S260: Inputting the trajectory point position data and map data of the obstacle to be predicted into a pre-trained obstacle trajectory prediction model to obtain a prediction result of the future trajectory data of the obstacle to be predicted.

[0098] In the specific implementation, the trajectory point position data and map data of the obstacle to be predicted are input into a pre-trained obstacle trajectory prediction model. First, the map data is abstracted into a topological network structure through the feature processing module, and the trajectory point position data of the obstacle to be predicted is processed into the form of a feature vector. Then, the feature fusion module first extracts the trajectory feature information in the trajectory point position data and the map feature information in the map data, and then performs feature fusion with each other in sequence. Finally, the trajectory prediction module decodes the data to obtain several future trajectory data corresponding to the trajectory point position data of the obstacle to be predicted and their corresponding confidence levels. The future trajectory data corresponding to the predicted trajectory with the highest confidence level is used as the prediction result of the obstacle trajectory prediction model.

[0099] It should be noted that the obstacle trajectory prediction model is trained based on the trajectory point position data of candidate obstacles in the public dataset and the target vehicle's self-collected real vehicle dataset, and the map data corresponding to the candidate obstacles.

[0100] Public datasets can be datasets obtained from public databases. For example, they can be Argo prediction datasets. Public datasets can be collected using sensors, radar, and other equipment in certain scenarios, such as urban or suburban areas, and can include trajectory information of moving objects and map information of the surrounding environment.

[0101] In this embodiment, the advantage of using a public dataset and a self-collected real vehicle dataset of the target vehicle to train the obstacle trajectory prediction model is that it can enable the model to incorporate the characteristics of the real vehicle scene on the basis of existing features, thereby improving the model's prediction results, making the model not limited to the public dataset, and increasing the amount of available information for the obstacle trajectory prediction model.

[0102] The technical solution of the embodiment of the present invention obtains original annotated data of a target vehicle within a preset perception range, then converts the original annotated data from an original coordinate system to a world coordinate system to obtain target annotated data. Furthermore, the trajectory point position data of candidate obstacles is segmented based on a preset sliding window, and the obstacle to be predicted is determined based on the trajectory point position data of the candidate obstacles in the target annotated data and the positioning data of the target vehicle. Finally, map data of the obstacle to be predicted within a preset search range is obtained, and the trajectory point position data and map data of the obstacle to be predicted are input into a pre-trained obstacle trajectory prediction model to obtain a prediction result of the future trajectory data of the obstacle to be predicted. This solves the problem in the prior art of inaccurate obstacle trajectory prediction due to the complex environment around the vehicle, the large variety, high density, and small gaps between obstacles that affect vehicle driving, and the resulting inaccurate obstacle trajectory prediction. By using data collected from actual vehicles, the prediction effect of the obstacle trajectory prediction model is improved, the accuracy of obstacle trajectory prediction is improved, and the prediction capability of the model in actual vehicle scenarios is enhanced.

[0103] Example 3

[0104] Figure 3 This is a flowchart of a method for processing original labeled data in an obstacle trajectory prediction method provided in Example 3 of the present invention. Figure 4 This is an algorithm flow chart of an obstacle trajectory prediction method provided by the third embodiment of the present invention. This embodiment of the present invention is a preferred embodiment of the above-mentioned invention embodiments, such as Figure 3 and Figure 4 As shown, the embodiment of the present invention may include the following steps:

[0105] 1. Original annotation data processing flow:

[0106] 1. Collect raw sampling data through the radar sensing device installed on the target vehicle and perform annotation processing to obtain raw annotated data;

[0107] 2. Since the data used in trajectory prediction is only related to the perception module and positioning module in the radar perception device, the data channels of the original annotated data are filtered to retain only the data information of the perception module and positioning module;

[0108] 3. Convert the original annotation data from the original coordinate system to the world coordinate system to obtain the target annotation data;

[0109] 4. Segment the target annotation data through a preset sliding window;

[0110] 5. Traverse the trajectory point position data of each candidate obstacle according to the preset screening conditions to determine the obstacle to be predicted;

[0111] 6. Use Gaussian filtering for smoothing;

[0112] 7. Interpolate the trajectory point position data and save it;

[0113] 8. Obtain map data around the obstacle to be predicted and use it together with the trajectory point position data of the obstacle to be predicted as the model input data of the obstacle trajectory prediction model.

[0114] 2. Algorithm Process

[0115] 1. Obtain the original annotation data of the target vehicle;

[0116] 2. After the above raw labeled data processing process, the real vehicle dataset of the target vehicle is obtained;

[0117] 3. Preprocess the self-collected real vehicle dataset and public dataset, and train the obstacle trajectory prediction model;

[0118] 4. Evaluate the model's prediction performance. Use the trained obstacle trajectory prediction model to conduct real-world predictions to verify the model's prediction performance. If the performance is good, stop the model iteration. If the performance is poor, identify the scenario with the poor prediction performance, collect data from that scenario, and iterate the model again.

[0119] The technical solution of the embodiment of the present invention obtains original annotated data of the target vehicle within a preset perception range, then converts the original annotated data from the original coordinate system to the world coordinate system to obtain target annotated data. Furthermore, the obstacle to be predicted is determined based on the trajectory point position data of the candidate obstacle in the target annotated data and the positioning data of the target vehicle. Finally, the future trajectory data of the obstacle to be predicted is predicted based on a pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted. This solves the problem in the prior art of inaccurate obstacle trajectory prediction due to the complex environment around the vehicle, the large variety, high density, and small gaps between obstacles that affect vehicle driving. By using data collected from actual vehicles, the prediction effect of the obstacle trajectory prediction model is improved, and the accuracy of obstacle trajectory prediction is improved.

[0120] Example 4

[0121] Figure 5 This is a schematic diagram of the structure of an obstacle trajectory prediction device provided by the fourth embodiment of the present invention. Figure 5 As shown, the device includes: a data acquisition module 310 , a coordinate conversion module 320 , an obstacle determination module 330 and a trajectory prediction module 340 .

[0122] Among them, the data acquisition module 310 is used to obtain the original annotation data of the target vehicle within a preset perception range, wherein the original annotation data includes the trajectory point position data of the alternative obstacle at multiple consecutive moments and the positioning data of the target vehicle; the coordinate conversion module 320 is used to convert the original annotation data from the original coordinate system to the world coordinate system to obtain the target annotation data; the obstacle determination module 330 is used to determine the obstacle to be predicted based on the trajectory point position data of the alternative obstacle in the target annotation data and the positioning data of the target vehicle; the trajectory prediction module 340 is used to predict the future trajectory data of the obstacle to be predicted based on the pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted.

[0123] The technical solution of the embodiment of the present invention obtains original annotated data of the target vehicle within a preset perception range, then converts the original annotated data from the original coordinate system to the world coordinate system to obtain target annotated data. Furthermore, the obstacle to be predicted is determined based on the trajectory point position data of the candidate obstacle in the target annotated data and the positioning data of the target vehicle. Finally, the future trajectory data of the obstacle to be predicted is predicted based on a pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted. This solves the problem in the prior art of inaccurate obstacle trajectory prediction due to the complex environment around the vehicle, the large variety, high density, and small gaps between obstacles that affect vehicle driving. By using data collected from actual vehicles, the prediction effect of the obstacle trajectory prediction model is improved, and the accuracy of obstacle trajectory prediction is improved.

[0124] Optionally, the data acquisition module 310 is further configured to collect original sampled data within a preset sensing range through a radar sensing device provided on the target vehicle, and to annotate the original sampled data to obtain original annotated data.

[0125] Optionally, the obstacle determination module 330 includes a distance determination unit, a data missing rate determination unit, a trajectory length determination unit, and an obstacle determination unit; wherein the distance determination unit is used to determine the perceived distance between the target vehicle and the alternative obstacle; the data missing rate determination unit is used to determine the data missing rate of the trajectory point position data of the alternative obstacle; the trajectory length determination unit is used to determine the trajectory length of the alternative obstacle; and the obstacle determination unit is used to determine the alternative obstacle as the obstacle to be predicted when at least one of the perceived distance, the data missing rate, and the trajectory length meets a preset screening condition.

[0126] Optionally, the data missing rate determining unit is further configured to determine the data missing rate of the candidate obstacle based on the following formula:

[0127] r=1-((COUNT(AGENT_track)) / CONUT(AV_track))

[0128] Where r represents the data missing rate of the candidate obstacle, COUNT(·) represents a counter, AGENT_track represents the trajectory point sequence of the candidate obstacle, and AV_track represents the trajectory point sequence of the target vehicle.

[0129] Optionally, the device further includes: a data processing module for processing the trajectory point position data of the predicted obstacle according to a preset data processing method and saving it in a preset file format, wherein the preset data processing method includes smoothing processing and interpolation processing.

[0130] Optionally, the data processing module is further configured to smooth the trajectory point position data of the predicted obstacle based on a Gaussian filtering method.

[0131] Optionally, the Gaussian kernel in the Gaussian filtering method is determined based on the following formula:

[0132]

[0133] Among them, G σ represents the Gaussian kernel, x, y represent the coordinates of the trajectory point, and σ represents the hyperparameter.

[0134] Optionally, the trajectory prediction module 340 is further used to obtain map data of the obstacle to be predicted within a preset search range, wherein the map data includes at least one of lane coordinates, whether the lane turns left, whether the lane turns right, whether it is a traffic intersection, and whether traffic control is performed; the trajectory point position coordinate data and map data of the obstacle to be predicted are input into a pre-trained obstacle trajectory prediction model to obtain a prediction result of the future trajectory data of the obstacle to be predicted, wherein the obstacle trajectory prediction model is trained based on the trajectory point position data of alternative obstacles in a public data set and a self-collected real vehicle data set of the target vehicle and the map data corresponding to the alternative obstacles.

[0135] Optionally, before determining the obstacle to be predicted based on the trajectory point position data of the alternative obstacle in the target annotation data and the positioning data of the target vehicle, the device also includes: a data segmentation module, which is used to segment the trajectory point position data of the alternative obstacle based on a preset sliding window, wherein the segmented data in each preset sliding window includes the historical trajectory point position data of the alternative obstacle and the predicted actual trajectory point position data.

[0136] The obstacle trajectory prediction device provided in the embodiment of the present invention can execute the obstacle trajectory prediction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] Example 5

[0138] Figure 6A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0139] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0140] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0141] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the obstacle trajectory prediction method.

[0142] In some embodiments, the obstacle trajectory prediction method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the obstacle trajectory prediction method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the obstacle trajectory prediction method in any other appropriate manner (e.g., via firmware).

[0143] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0147] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0148] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0149] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0150] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for predicting obstacle trajectories, characterized in that: include: Acquire original annotation data of the target vehicle within a preset sensing range, wherein the original annotation data includes trajectory point position data of the candidate obstacle at multiple consecutive moments and positioning data of the target vehicle; Convert the original annotation data from the original coordinate system to the world coordinate system to obtain target annotation data; Determining the obstacle to be predicted according to the trajectory point position data of the candidate obstacle in the target annotation data and the positioning data of the target vehicle, including: determining the perceived distance between the target vehicle and the candidate obstacle; determining the data missing rate of the trajectory point position data of the candidate obstacle; determining the trajectory length of the candidate obstacle; when at least one of the perceived distance, the data missing rate and the trajectory length meets a preset screening condition, determining the candidate obstacle as the obstacle to be predicted; Predicting the future trajectory data of the obstacle to be predicted based on the pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted; The data missing rate of the candidate obstacle is determined based on the following formula: r=1-((COUNT(AGENT_track)) / CONUT(AV_track)) r represents the data missing rate of the candidate obstacle, COUNT(·) represents the counter, AGENT_track represents the trajectory point sequence of the candidate obstacle, and AV_track represents the trajectory point sequence of the target vehicle.

2. The method according to claim 1, characterized in that: The obtaining of original labeling data of the target vehicle within a preset sensing range includes: The original sampling data within a preset sensing range is collected by a radar sensing device arranged on the target vehicle, and the original sampling data is labeled to obtain original labeled data.

3. The method according to claim 1, characterized in that Also includes: The trajectory point position data of the obstacle to be predicted is processed according to a preset data processing method and saved in a preset file format, wherein the preset data processing method includes smoothing processing and interpolation processing.

4. The method according to claim 3, characterized in that The processing of the trajectory point position data of the obstacle to be predicted according to a preset data processing method includes: The trajectory point position data of the obstacle to be predicted is smoothed based on the Gaussian filtering method.

5. The method according to claim 4, characterized in that The Gaussian kernel in the Gaussian filtering method is determined based on the following formula: Among them, G σ represents the Gaussian kernel, x, y represent the coordinates of the trajectory point, and σ represents the hyperparameter.

6. The method according to claim 1, characterized in that The predicting of future trajectory data of the obstacle to be predicted based on the pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted includes: Acquire map data of the obstacle to be predicted within a preset search range, wherein the map data includes at least one of lane coordinates, whether the lane turns left, whether the lane turns right, whether it is a traffic intersection, and whether traffic control is performed; The trajectory point position coordinate data of the obstacle to be predicted and the map data are input into a pre-trained obstacle trajectory prediction model to obtain a prediction result of the future trajectory data of the obstacle to be predicted, wherein the obstacle trajectory prediction model is trained based on the trajectory point position data of the candidate obstacles in the public data set and the self-collected real vehicle data set of the target vehicle and the map data corresponding to the candidate obstacles.

7. The method according to claim 1, characterized in that Before determining the obstacle to be predicted according to the trajectory point position data of the candidate obstacle in the target annotation data and the positioning data of the target vehicle, the method further includes: The trajectory point position data of the candidate obstacle are segmented based on a preset sliding window, wherein the segmented data in each preset sliding window includes the historical trajectory point position data and the predicted real trajectory point position data of the candidate obstacle.

8. An obstacle trajectory prediction device, characterized in that: include: A data acquisition module, used to acquire original annotation data of the target vehicle within a preset perception range, wherein the original annotation data includes trajectory point position data of the candidate obstacle at multiple consecutive moments and positioning data of the target vehicle; A coordinate conversion module, used to convert the original annotation data from the original coordinate system to the world coordinate system to obtain target annotation data; An obstacle determination module, used to determine the obstacle to be predicted based on the trajectory point position data of the candidate obstacle in the target annotation data and the positioning data of the target vehicle; A trajectory prediction module, used to predict the future trajectory data of the obstacle to be predicted based on the pre-trained obstacle trajectory prediction model and the trajectory point position data of the obstacle to be predicted; The obstacle determination module includes a distance determination unit, a data missing rate determination unit, a trajectory length determination unit and an obstacle determination unit; The distance determination unit is used to determine the perceived distance between the target vehicle and the candidate obstacle; The data missing rate determination unit is used to determine the data missing rate of the trajectory point position data of the candidate obstacle; The trajectory length determination unit is used to determine the trajectory length of the candidate obstacle; The obstacle determination unit is configured to determine the candidate obstacle as the obstacle to be predicted when at least one of the perception distance, the data missing rate and the trajectory length meets a preset screening condition; The data missing rate determining unit determines the data missing rate of the candidate obstacle based on the following formula: r=1-((COUNT(AGENT_track)) / CONUT(AV_track)) Where r represents the data missing rate of the candidate obstacle, COUNT(·) represents the counter, AGENT_track represents the track point sequence of the candidate obstacle, and AV_track represents the track point sequence of the target vehicle.

Images