Trajectory prediction method and device, vehicle and storage medium

Abstract

The invention relates to a trajectory prediction method and device, a vehicle and a storage medium, and the method comprises the steps: extracting map element features and obstacle features in a multi-view image to generate a multi-view feature image, converting the multi-view feature image into a BEV bird's-eye view space, generating BEV query data, and carrying out the target detection from the BEV query data, the method comprises the steps of determining obstacle identification information of a to-be-predicted obstacle and map element information associated with the obstacle identification information, determining a motion interaction relationship between the to-be-predicted obstacle and map elements according to the map element information and the obstacle identification information, and determining the to-be-predicted obstacle based on the motion interaction relationship. And predicting the movement track of the to-be-predicted obstacle in the map element, and generating predicted track data of the to-be-predicted obstacle. Therefore, trajectory prediction in a map-free scene is realized, the prediction precision is improved, and the trajectory prediction cost is saved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving, and in particular to a trajectory prediction method, device, vehicle and storage medium. Background Art

[0002] With the development of autonomous driving technology, the importance of trajectory prediction in autonomous driving systems has increased significantly in recent years. Trajectory prediction aims to predict the movement trajectory and driving intention of traffic participants in the future, which can provide prior guidance for the planning and control modules of autonomous driving vehicles and avoid collisions in advance.

[0003] For trajectory prediction algorithms, high-precision maps are usually one of the important inputs. High-precision maps can provide effective prior information about the movement patterns of moving obstacles. However, high-precision maps require high human annotation costs and high purchase costs, which to some extent has become a bottleneck in the development of trajectory prediction technology. Moreover, in places where high-precision maps do not exist, the effect of trajectory prediction is difficult to guarantee. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a trajectory prediction method, device, vehicle and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a trajectory prediction method is provided, including:

[0006] Extracting map element features and obstacle features from multi-view images to generate multi-view feature images;

[0007] Converting the multi-view feature image into a BEV bird's-eye view space to generate BEV query data;

[0008] Performing target detection from the BEV query data to determine obstacle identification information of the obstacle to be predicted and map element information associated with the obstacle identification information;

[0009] Determining a motion interaction relationship between the obstacle to be predicted and the map element according to the map element information and the obstacle identification information;

[0010] Based on the motion interaction relationship, the motion trajectory of the obstacle to be predicted in the map element is predicted to generate predicted trajectory data of the obstacle to be predicted.

[0011] Optionally, the performing target detection from the BEV query data to determine obstacle identification information of the obstacle to be predicted, and map element information associated with the obstacle identification information, includes:

[0012] Performing target detection on obstacles in the BEV query data to determine obstacle identification information of the obstacle to be predicted, wherein the obstacle identification information includes obstacle category information, obstacle location information, and interaction information between obstacles;

[0013] Map element information associated with the obstacle identification information is determined from the BEV query data.

[0014] Optionally, determining the motion interaction relationship between the obstacle to be predicted and the map element according to the map element information and the obstacle identification information includes:

[0015] Determining an initial interaction relationship between the obstacle to be predicted and other obstacles according to the obstacle identification information;

[0016] Based on the initial interaction relationship, the interaction between the obstacle to be predicted and the map element is achieved through a preset cross-attention mechanism to generate the motion interaction relationship.

[0017] Optionally, determining, according to the obstacle identification information, an initial interaction relationship between the obstacle to be predicted and other obstacles includes:

[0018] Performing coordinate normalization processing on the obstacle identification information to generate a relative position relationship between the obstacle to be predicted and the other obstacles;

[0019] The initial interaction relationship is determined according to the relative position relationship.

[0020] Optionally, determining the map element information associated with the obstacle identification information from the BEV query data includes:

[0021] Acquire map point cloud data associated with the obstacle to be predicted from the BEV query data;

[0022] Updating preset historical map data based on the map point cloud data to generate initial map data;

[0023] The map element information associated with the obstacle to be predicted is identified from the initial map data.

[0024] Optionally, the performing target detection on obstacles in the BEV query data to determine obstacle identification information of the obstacle to be predicted includes:

[0025] identifying a plurality of horizontal distances between each obstacle and the vehicle from the BEV query data;

[0026] Determine the obstacle to be predicted from the multiple obstacles according to the multiple horizontal distances, the horizontal distance of the obstacle to be predicted being less than a set threshold;

[0027] The obstacle identification information of the obstacle to be predicted is determined from the BEV query data.

[0028] Optionally, determining the obstacle identification information of the obstacle to be predicted from the BEV query data includes:

[0029] Determining association information between the obstacle to be predicted and the BEV query data according to a preset variable attention mechanism;

[0030] Based on the association information, identifying the interaction information between the obstacles from the BEV query data through a multi-head attention mechanism;

[0031] Based on the association information, the obstacle category information and the obstacle position information are determined from the BEV query data through preset classification and regression branches.

[0032] Optionally, determining the map element information associated with the obstacle identification information from the BEV query data includes:

[0033] Determining a plurality of initial map element information associated with the obstacle to be predicted from the BEV query data;

[0034] Acquire a plurality of prediction confidences corresponding one-to-one to the plurality of initial map element information, and a plurality of distances corresponding one-to-one between the plurality of initial map element information and the obstacles to be predicted;

[0035] When the target prediction confidence is greater than the confidence threshold and the target distance is less than the distance threshold, determining that the initial map element information corresponding to the target prediction confidence is the map element information;

[0036] According to a second aspect of an embodiment of the present disclosure, a trajectory prediction device is provided, the device comprising:

[0037] A first generating module is configured to extract map element features and obstacle features in the multi-view image to generate a multi-view feature image;

[0038] A second generating module is configured to convert the multi-view feature image into a BEV bird's-eye view space to generate BEV query data;

[0039] a first determination module configured to perform target detection from the BEV query data, determine obstacle identification information of the obstacle to be predicted, and map element information associated with the obstacle identification information;

[0040] a second determination module, configured to determine a motion interaction relationship between the obstacle to be predicted and the map element according to the map element information and the obstacle identification information;

[0041] The execution module is configured to predict the movement trajectory of the obstacle to be predicted in the map element based on the movement interaction relationship, and generate predicted trajectory data of the obstacle to be predicted.

[0042] According to a third aspect of an embodiment of the present disclosure, a vehicle is provided, comprising:

[0043] processor;

[0044] a memory for storing processor-executable instructions;

[0045] Wherein, the processor is configured to implement the steps of any one of the methods described in the first aspect of the present disclosure when executing the executable instructions.

[0046] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the trajectory prediction method provided in the first aspect of the present disclosure are implemented.

[0047] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0048] Through the above method, map element features and obstacle features in multi-view images are extracted to generate multi-view feature images, and the multi-view feature images are converted into BEV bird's-eye view space to generate BEV query data. Target detection is performed from the BEV query data to determine obstacle identification information of the obstacle to be predicted and map element information associated with the obstacle identification information. According to the map element information and obstacle identification information, the motion interaction relationship between the obstacle to be predicted and the map element is determined. Based on the motion interaction relationship, the motion trajectory of the obstacle to be predicted in the map element is predicted to generate predicted trajectory data of the obstacle to be predicted. Thus, trajectory prediction in a map-free scene is realized, and based on the interaction relationship between obstacles and between obstacles and map scenes, the tasks of obstacle target detection, map prediction and motion prediction are completed, the prediction accuracy is improved, and the trajectory prediction cost is saved.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0051] Figure 1 The figure is a flow chart of a trajectory prediction method according to an exemplary embodiment.

[0052] Figure 2 The figure is a flow chart of a method for determining obstacle recognition information according to an exemplary embodiment.

[0053] Figure 3 The figure is a block diagram of a trajectory prediction device according to an exemplary embodiment.

[0054] Figure 4 is a block diagram of a vehicle 400 according to an exemplary embodiment. DETAILED DESCRIPTION

[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0056] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0057] There are two types of trajectory prediction algorithms in related technologies. One is a trajectory prediction method based on perception output results, which includes the following steps:

[0058] (1) Obtain the detection results of perceived obstacles, the vehicle’s positioning information, and high-precision map information within a certain range of the vehicle;

[0059] (2) Using a learning-based or rule-based trajectory prediction algorithm to obtain the obstacle’s movement trajectory and driving intention in the future;

[0060] (3) The system is output to the downstream regulation and control module to regulate the motion trajectory and driving intention, and output the predicted trajectory of obstacles.

[0061] Another approach is to integrate perception and prediction into the same end-to-end framework, which includes the following steps:

[0062] (1) Obtaining raw sensor data such as cameras and lasers mounted on the vehicle, vehicle positioning information, and high-precision map information within a certain range of the vehicle;

[0063] (2) Using an end-to-end framework algorithm that unifies perception and prediction, the system obtains obstacle detection results and other static attributes, including 3D bounding box (directional matrix bounding box), as well as dynamic attributes such as speed. It also generates the obstacle's motion trajectory and driving intention for a period of time in the future.

[0064] (3) The system is output to the downstream regulation and control module to regulate the motion trajectory and driving intention, and output the predicted trajectory of obstacles.

[0065] The above trajectory prediction method has the following problems:

[0066] (1) It is impossible to obtain good prediction results in map-free scenarios, or even the prediction results cannot be used. This problem will arise when autonomous driving needs to be expanded to more scenarios.

[0067] (2) Trajectory prediction methods that strongly rely on high-precision map information are not easy to respond to environmental changes, because the update of high-precision maps is often delayed. If they only rely on high-precision map information as the judgment of static scene elements, it is easy to produce wrong trajectory prediction results in areas where high-precision maps change;

[0068] (3) High-precision map annotation and acquisition are costly and difficult to obtain, which limits the application of trajectory prediction methods in autonomous driving scenarios.

[0069] In view of this, this proposal proposes a trajectory prediction method, device, vehicle connection and storage medium. Figure 1 is a flow chart of a trajectory prediction method according to an exemplary embodiment. Figure 1 As shown, the method is used in a vehicle-mounted terminal and includes the following steps.

[0070] In step S11, map element features and obstacle features in the multi-view image are extracted to generate a multi-view feature image.

[0071] For example, the vehicle terminal of this embodiment is equipped with a multi-perspective camera sensor for collecting multi-perspective images around the vehicle. The multi-perspective images are used to allow the vehicle terminal to identify the current environment of the vehicle. Therefore, the multi-perspective images can be spliced ​​into a complete overhead view, which can be used to reflect the environmental information within a set range around the vehicle. In order to improve the accuracy of vehicle terminal recognition, the multi-perspective image is usually of high definition, and the corresponding transmission resources in the occupied vehicle are large, and the transmission efficiency is low. For obstacle trajectory prediction, it is only necessary to collect the line features of the obstacles and the surrounding environment in the environment, and perform trajectory prediction analysis based on the line features, without the need to predict the trajectory of the obstacle through the original high-definition image. In addition, by extracting the corresponding features in the multi-perspective image to perform obstacle trajectory analysis, it is possible to avoid interference of other interference elements in the multi-perspective image on the prediction result.

[0072] In this embodiment, feature extraction is performed on the multi-view images collected by the multi-view camera sensor, and map element features and obstacle features in the multi-view images are collected. Wherein, based on the need for obstacle trajectory prediction, the map elements are road line features, map scene features, map road shape features, etc. in the map elements within the preset range of the current vehicle, and the obstacle features include obstacles within the lane line captured within the viewing range of the multi-view camera sensor, obstacles outside the lane line, etc. The obstacles may include moving objects, as well as non-moving objects within the viewing range, and objects that hinder the driving of the current vehicle may be regarded as obstacles. For example, in this embodiment, a backbone network that has been trained for extracting map element features and obstacle features, such as ResNet-50 (deep learning network), etc., may be used to extract map element features and obstacle features of each frame of the multi-view image to generate a multi-view feature image.

[0073] In step S12, the multi-view feature image is converted into the BEV bird's-eye view space to generate BEV query data.

[0074] For example, in this embodiment, in order to identify the environmental features around the vehicle, the direction of the image captured by the multi-view camera sensor is usually based on the horizontal direction of the multi-view camera sensor away from the vehicle, so the corresponding multi-view image and multi-view feature image are 3D image data. When predicting the trajectory of the obstacle, the environment and obstacles around the vehicle are converted to a bird's-eye view, which can better predict the movement trajectory of the obstacle. Therefore, in this embodiment, the multi-view feature image is converted to the BEV (Bird's-Eye View) bird's-eye view space, and then based on the acquisition perspective of each multi-view feature image, the multi-view feature image is spliced ​​to generate BEV query data for reflecting the complete environmental information and complete obstacle information around the vehicle. For example, with reference to BEVFormer (visual perception algorithm) and other related methods, the multi-view feature image is output to the BEV encoder, and the BEV query data is output based on the BEV encoder, so that the multi-view feature image is converted to the BEV space.

[0075] In step S13, target detection is performed from the BEV query data to determine obstacle identification information of the obstacle to be predicted and map element information associated with the obstacle identification information.

[0076] For example, based on the BEV query data generated by the conversion, target detection is performed on the obstacles to be predicted in the data to obtain obstacle identification information of the obstacles to be predicted. Among them, in this embodiment, in order to reduce the burden of trajectory prediction, obstacles in the BEV query data are usually screened, and obstacles to be predicted that meet the prediction conditions are selected. For example, the distance between each obstacle and the vehicle can be determined based on the BEV query data, and obstacles with a distance less than a set threshold are selected as obstacles to be predicted. The obstacle information of the obstacles to be predicted in the BEV query data is identified to determine the obstacle identification information, wherein the obstacle identification information includes obstacle category information, obstacle shape information, obstacle size information, obstacle position information, etc. For example, the BEV query data generated by the BEV encoder is decoded through the trained classification and regression branches, and the obstacle category detection score, obstacle object shape information, and obstacle position information of the obstacles to be predicted are detected respectively. Among them, the obstacle position information is the spatial position information of the obstacle relative to the vehicle. For example, the center of gravity of the vehicle can be used as the coordinate origin in the spatial coordinate system, and the spatial coordinates of the obstacle to be predicted relative to the coordinate origin can be determined according to the BEV query data, and the obstacle position information is generated based on the spatial coordinates.

[0077] After determining the obstacle identification information of the obstacle to be predicted, map element information associated with the obstacle identification information is identified based on the BEV query data. In this embodiment, the map element information is associated with the obstacle identification information. The map element information determined from the BEV query data is generated by identifying the map elements based on the obstacle identification information.

[0078] Optionally, in one implementation, the above step S13 includes:

[0079] Obtaining map point cloud data associated with the obstacle to be predicted from the BEV query data;

[0080] Updating preset historical map data based on map point cloud data to generate initial map data;

[0081] Map element information associated with the obstacle to be predicted is identified from the initial map data.

[0082] For example, in this embodiment, in order to improve the data processing speed and reduce the interference of unnecessary color interference factors on the trajectory prediction algorithm, the map point cloud data associated with the obstacle to be predicted in the BEV query data can be extracted, the map data can be converted in the form of point cloud, and the map elements in the preset historical map data of the current location of the obstacle to be predicted can be updated based on the map point cloud data to generate initial map data. The map element information associated with the obstacle to be predicted is identified from the initial map data.

[0083] The obstacle identification information is the obstacle location information, and the map element information within the preset radius of the obstacle location information is identified from the BEV query data according to the obstacle location information. For example, in the process of identifying the map element information, the map information can be represented in the form of point cloud data, and then the map encoder is used to input the point-form map information into the map decoder, and the classification and regression branches are used respectively to determine the map element information associated with the obstacle identification information.

[0084] Optionally, in one implementation, the above step of “performing target detection from BEV query data and determining map element information associated with obstacle identification information” includes:

[0085] Determining a plurality of initial map element information associated with the obstacle to be predicted from the BEV query data;

[0086] Acquire a plurality of prediction confidences corresponding one-to-one to the plurality of initial map element information, and a plurality of distances corresponding one-to-one between the plurality of initial map element information and obstacles to be predicted;

[0087] When the target prediction confidence is greater than the confidence threshold and the target distance is less than the distance threshold, the initial map element information corresponding to the target prediction confidence is determined as the map element information.

[0088] For example, in this embodiment, before encoding the map information, in order to avoid interference of map elements with low precision on the prediction results, the map elements in the BEV query will be filtered according to the obstacles to be predicted. Therefore, in this embodiment, the generated map element information will be scored for confidence, and the map elements whose prediction confidence exceeds the confidence threshold will be screened out as the map element information associated with the obstacles to be predicted.

[0089] Optionally, in one implementation, the above step S13 includes:

[0090] Performing target detection on obstacles in the BEV query data to determine obstacle identification information of the obstacle to be predicted, wherein the obstacle identification information includes obstacle category information, obstacle location information, and interaction information between obstacles;

[0091] Map element information associated with the obstacle identification information is determined from the BEV query data.

[0092] For example, in this embodiment, based on the purpose of predicting the trajectory of the obstacle, when it is determined that there is an obstacle to be predicted in the BEV query data, the obstacle identification information of the obstacle to be predicted can be identified based on the relevant image recognition neural network, and the obstacle category information, obstacle position information and obstacle interaction information between the obstacle to be predicted and other obstacles can be determined. When determining the obstacle category information of the predicted obstacle, multiple category identification results are generated based on the convolutional neural network, and the matching degree of the multiple category identification results is scored, and the obstacle category with the highest score is determined as the obstacle category information of the obstacle to be predicted. The obstacle interaction information is associated with the obstacle position information of the obstacle at its current location, and the obstacle interaction information includes the interaction information between the obstacle to be predicted and other obstacles in the historical time period, and also includes the interaction information between the obstacle to be predicted and other obstacles in the future.

[0093] For example, based on the obstacle identification information generated in the above steps, map element information in the BEV query data is identified, and map element information associated with the obstacle identification information is screened out. The association relationship between the obstacle identification information and the map element information is an interactive relationship, that is, it includes the map elements that the obstacle to be predicted has traveled through in the historical time period, and the map elements that the obstacle to be predicted is predicted to travel through.

[0094] In step S14, the motion interaction relationship between the obstacle to be predicted and the map element is determined according to the map element information and the obstacle identification information.

[0095] In this embodiment, when predicting the trajectory of the obstacle to be predicted, it is necessary to determine the motion interaction relationship between the obstacle to be predicted and the map element, and predict the trajectory of the obstacle to be predicted based on the motion interaction relationship. The motion interaction relationship includes the motion interaction relationship between the obstacle to be predicted and the map element in the historical time period. For example, the motion interaction relationship includes the obstacle to be predicted moving away from the corresponding map element, the obstacle to be predicted moving toward the corresponding map element, the obstacle to be predicted and the lane line map element are in a parallel moving state, etc.

[0096] For example, in this embodiment, the trained MHSA (Multi-Head Self-Attention) can be used to identify the information between the obstacles to be predicted, and the trained deformable attention mechanism can be used to identify the association information between the obstacles to be predicted and the map element information. The motion interaction relationship between the obstacles to be predicted and the map elements is determined based on the association information.

[0097] Optionally, in one implementation, the above step S14 includes:

[0098] Determining an initial interaction relationship between the obstacle to be predicted and other obstacles according to the obstacle identification information;

[0099] Based on the initial interaction relationship, the interaction between the obstacle to be predicted and the map element is achieved through a preset cross-attention mechanism to generate the motion interaction relationship.

[0100] For example, in this embodiment, based on the obstacle recognition information, Self Attention can be used to learn the initial interaction relationship between the obstacle to be predicted and other obstacles, and then based on the initial interaction relationship, the map elements in the current scene are integrated into the interaction relationship, and the interaction between the obstacle to be predicted and the map elements is realized through the preset cross-attention mechanism to generate a motion interaction relationship. For example, the motion interaction relationship can be generated by the following formula:

[0101] Q' motion =FFN(Self Attention(Q motion ))

[0102] Among them, Q' motionis the motion interaction relationship between the obstacle to be predicted and other obstacles and map elements, FFN() is the forward network (feedforward neural network algorithm), Self Attention() is the self-attention mechanism algorithm, Q motion The initial interaction relationship.

[0103] Optionally, in one implementation, the above step of “determining, according to the obstacle identification information, an initial interaction relationship between the obstacle to be predicted and other obstacles” includes:

[0104] Performing coordinate normalization processing on the obstacle identification information to generate a relative position relationship between the obstacle to be predicted and the other obstacles;

[0105] The initial interaction relationship is determined according to the relative position relationship.

[0106] The trajectory prediction method in the related art only considers one obstacle at a time, resulting in certain limitations in the predicted trajectory of the obstacle to be predicted. In this proposal, all obstacles in the scene will be predicted at the same time. If the global coordinate system is used as a reference for the prediction of all obstacles, the predicted trajectory will be inaccurate. However, if the vehicle coordinate system is used as a reference, it is also unreasonable, because in the vehicle coordinate system, obstacles in two different positions are likely to have similar motion patterns, such as both going straight, but the coordinate systems of their future motion trajectories may be very different. Therefore, in order to reduce the differences between obstacles, in this embodiment, each obstacle is transferred to a coordinate system with itself as the origin, and the coordinates of the obstacle position information of each obstacle are normalized to generate the relative position relationship between the obstacle to be predicted and other obstacles. For example, the position information of the obstacle can be normalized in the following way:

[0107] P' Agent,t =P Agent,t+1 -P Agent,t

[0108] Among them, P Agent,t represents the coordinate system information of the obstacle at time t, P Agent,t+1 Represents the coordinate system information of the obstacle at time t+1, P' Agent,t It is the relative position relationship of the obstacle based on its own coordinate system.

[0109] For example, the initial interaction relationship between the obstacle to be predicted and the map element is determined according to the relative position relationship.

[0110] In step S15, based on the motion interaction relationship, the motion trajectory of the obstacle to be predicted in the map element is predicted to generate predicted trajectory data of the obstacle to be predicted.

[0111] For example, in this embodiment, based on the motion interaction relationship, an interactive connection between map element information and predicted obstacles is established. According to the interactive connection, the motion interaction process between the predicted obstacle and the map element information in the historical time period can be simulated in the virtual space. Then, based on the motion interaction process, the motion trajectory of the predicted obstacle in the future period is simulated, and the extension of the map element information in the time period is predicted to generate derived map element information. And according to the generated motion interaction process of the predicted obstacle in the future period and the derived map element information, the predicted trajectory data of the obstacle to be predicted is determined.

[0112] Through the above method, map element features and obstacle features in multi-view images are extracted to generate multi-view feature images, and the multi-view feature images are converted into BEV bird's-eye view space to generate BEV query data. Target detection is performed from the BEV query data to determine obstacle identification information of the obstacle to be predicted and map element information associated with the obstacle identification information. According to the map element information and obstacle identification information, the motion interaction relationship between the obstacle to be predicted and the map element is determined. Based on the motion interaction relationship, the motion trajectory of the obstacle to be predicted in the map element is predicted to generate predicted trajectory data of the obstacle to be predicted. Thus, trajectory prediction in a map-free scene is realized, and based on the interaction relationship between obstacles and between obstacles and map scenes, the tasks of obstacle target detection, map prediction and motion prediction are completed, the prediction accuracy is improved, and the trajectory prediction cost is saved.

[0113] Figure 2 is a flow chart of a method for determining obstacle identification information according to an exemplary embodiment. Figure 2 As shown, the above step S13 may include the following steps.

[0114] In step S131 , a plurality of horizontal distances between each obstacle and the vehicle are identified from the BEV query data.

[0115] For example, the BEV query data is a bird's-eye view data information centered on the vehicle. The vehicle-mounted terminal can calculate and determine multiple horizontal distances between the obstacle and the vehicle based on multiple image distances between each obstacle and the vehicle in the query data and the corresponding scaling ratio.

[0116] In step S132, an obstacle to be predicted is determined from a plurality of obstacles according to a plurality of horizontal distances, and the horizontal distance of the obstacle to be predicted is less than a set threshold.

[0117] In the process of predicting the trajectory of obstacles, this embodiment only targets obstacles that pose a collision threat to the current vehicle. Therefore, when multiple obstacles are identified in the BEV query data, it is necessary to screen out the obstacles to be predicted that will affect the vehicle's driving from the multiple obstacles. In this embodiment, obstacles whose horizontal distance is less than a set threshold are determined from the multiple obstacles as obstacles to be predicted.

[0118] In step S133 , obstacle identification information of the obstacle to be predicted is determined from the BEV query data.

[0119] After the obstacle to be predicted is determined through the above steps, obstacle identification information associated with the obstacle to be predicted is screened out from the BEV query data, wherein the method for determining the obstacle identification information is the same as that in the above step S13, and reference may be made to the above step S13, which will not be repeated here.

[0120] Optionally, in one implementation, the above step S133 includes:

[0121] Determine the association information between the obstacle to be predicted and the BEV query data according to the preset variable attention mechanism;

[0122] Based on the association information, the interaction information between obstacles is identified from the BEV query data through a multi-head attention mechanism;

[0123] Based on the association information, obstacle category information and obstacle location information are determined from the BEV query data through preset classification and regression branches.

[0124] For example, in this embodiment, obstacle features are learned from BEV queries, where the encoder is composed of: using MHSA (Multi-Head Self-Attention, multi-head attention mechanism) to learn information between obstacles, using deformable attention mechanism to learn the association information between obstacles and BEV queries, and the decoder uses classification and regression branches to respectively predict detection scores and obstacle positions and other information.

[0125] Through the above method, the obstacles and map element information in the BEV query data are screened, thereby avoiding interference of other interference factors on the prediction results and improving the accuracy of the prediction results.

[0126] Figure 3 is a block diagram of a trajectory prediction device according to an exemplary embodiment. Figure 3 The device 100 includes: a first generating module 110, a second generating module 120, a first determining module 130, a second determining module 140 and an executing module 150.

[0127] The first generating module 110 is configured to extract map element features and obstacle features in the multi-view image to generate a multi-view feature image;

[0128] The second generating module 120 is configured to convert the multi-view feature image into a BEV bird's-eye view space to generate BEV query data;

[0129] The first determination module 130 is configured to perform target detection from the BEV query data, determine obstacle identification information of the obstacle to be predicted, and map element information associated with the obstacle identification information;

[0130] The second determination module 140 is configured to determine the motion interaction relationship between the obstacle to be predicted and the map element according to the map element information and the obstacle identification information;

[0131] The execution module 150 is configured to predict the movement trajectory of the obstacle to be predicted in the map element based on the movement interaction relationship, and generate predicted trajectory data of the obstacle to be predicted.

[0132] Optionally, the first determining module 130 includes:

[0133] A first determination submodule is configured to perform target detection on obstacles in the BEV query data to determine obstacle identification information of the obstacle to be predicted, wherein the obstacle identification information includes obstacle category information, obstacle location information, and interaction information between obstacles;

[0134] The second determination submodule is configured to determine map element information associated with the obstacle identification information from the BEV query data.

[0135] Optionally, the second determining module 140 includes:

[0136] A third determination submodule is configured to determine an initial interaction relationship between the obstacle to be predicted and other obstacles according to the obstacle identification information;

[0137] The generation submodule is configured to realize the interaction between the obstacle to be predicted and the map element based on the initial interaction relationship through a preset cross-attention mechanism to generate a motion interaction relationship.

[0138] Optionally, the third determining submodule is configured to:

[0139] Perform coordinate normalization processing on obstacle identification information to generate the relative position relationship between the obstacle to be predicted and other obstacles;

[0140] According to the relative position relationship, the initial interaction relationship is determined.

[0141] Optionally, the first determining module 130 is configured to:

[0142] Obtaining map point cloud data associated with the obstacle to be predicted from the BEV query data;

[0143] Updating preset historical map data based on map point cloud data to generate initial map data;

[0144] Map element information associated with the obstacle to be predicted is identified from the initial map data.

[0145] Optionally, the first determining module 130 includes:

[0146] an identification submodule configured to identify a plurality of horizontal distances between each obstacle and the vehicle from the BEV query data;

[0147] A fourth determination submodule is configured to determine, according to the multiple horizontal distances, a to-be-predicted obstacle from the multiple obstacles, wherein the horizontal distance of the to-be-predicted obstacle is less than a set threshold;

[0148] The fifth determination submodule is configured to determine obstacle identification information of the obstacle to be predicted from the BEV query data.

[0149] Optionally, the fifth determining submodule is configured to:

[0150] Determine the association information between the obstacle to be predicted and the BEV query data according to the preset variable attention mechanism;

[0151] Based on the association information, the interaction information between obstacles is identified from the BEV query data through a multi-head attention mechanism;

[0152] Based on the association information, obstacle category information and obstacle location information are determined from the BEV query data through preset classification and regression branches.

[0153] Optionally, the first determining module 130 is configured to:

[0154] Determining a plurality of initial map element information associated with the obstacle to be predicted from the BEV query data;

[0155] Acquire a plurality of prediction confidences corresponding one-to-one to the plurality of initial map element information, and a plurality of distances corresponding one-to-one between the plurality of initial map element information and obstacles to be predicted;

[0156] When the target prediction confidence is greater than the confidence threshold and the target distance is less than the distance threshold, the initial map element information corresponding to the target prediction confidence is determined as the map element information.

[0157] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0158] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the trajectory prediction method provided by the present disclosure are implemented.

[0159] Figure 4 4 is a block diagram of a vehicle 400 according to an exemplary embodiment. For example, the vehicle 400 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle 400 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0160] Reference Figure 4 , the vehicle 400 may include various subsystems, for example, an infotainment system 410, a perception system 420, a decision control system 430, a drive system 440, and a computing platform 450. The vehicle 400 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 400 may be interconnected by wire or wireless means.

[0161] In some embodiments, the infotainment system 410 may include a communication system, an entertainment system, and a navigation system, among others.

[0162] The perception system 420 may include several sensors for sensing information about the environment around the vehicle 400. For example, the perception system 420 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0163] The decision control system 430 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0164] The drive system 440 may include components that provide power movement for the vehicle 400. In one embodiment, the drive system 440 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0165] Some or all functions of the vehicle 400 are controlled by a computing platform 450. The computing platform 450 may include at least one processor 451 and a memory 452, and the processor 451 may execute instructions 453 stored in the memory 452.

[0166] The processor 451 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0167] Memory 452 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0168] In addition to instructions 453 , memory 452 may also store data, such as road maps, route information, and data such as the location, direction, and speed of the vehicle. The data stored in memory 452 may be used by computing platform 450 .

[0169] In the embodiment of the present disclosure, the processor 451 may execute the instruction 453 to complete all or part of the steps of the above-mentioned trajectory prediction method.

[0170] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device. The computer program has a code portion for executing the above trajectory prediction method when executed by the programmable device.

[0171] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0172] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A trajectory prediction method, characterized in that: include: Extracting map element features and obstacle features from multi-view images to generate multi-view feature images; Converting the multi-view feature image into a BEV bird's-eye view space to generate BEV query data; Performing target detection from the BEV query data to determine obstacle identification information of the obstacle to be predicted and map element information associated with the obstacle identification information; Determining a motion interaction relationship between the obstacle to be predicted and the map element according to the map element information and the obstacle identification information; Based on the motion interaction relationship, the motion trajectory of the obstacle to be predicted in the map element is predicted to generate predicted trajectory data of the obstacle to be predicted.

2. The method according to claim 1, characterized in that The performing target detection from the BEV query data, determining obstacle identification information of the obstacle to be predicted, and map element information associated with the obstacle identification information, includes: Performing target detection on obstacles in the BEV query data to determine obstacle identification information of the obstacle to be predicted, wherein the obstacle identification information includes obstacle category information, obstacle location information, and interaction information between obstacles; Map element information associated with the obstacle identification information is determined from the BEV query data.

3. The method according to claim 1, characterized in that The determining, according to the map element information and the obstacle identification information, a motion interaction relationship between the obstacle to be predicted and the map element includes: Determining an initial interaction relationship between the obstacle to be predicted and other obstacles according to the obstacle identification information; Based on the initial interaction relationship, the interaction between the obstacle to be predicted and the map element is achieved through a preset cross-attention mechanism to generate the motion interaction relationship.

4. The method according to claim 3, characterized in that The determining, according to the obstacle identification information, an initial interaction relationship between the obstacle to be predicted and other obstacles includes: Performing coordinate normalization processing on the obstacle identification information to generate a relative position relationship between the obstacle to be predicted and the other obstacles; The initial interaction relationship is determined according to the relative position relationship.

5. The method according to claim 1, characterized in that The determining, from the BEV query data, map element information associated with the obstacle identification information comprises: Acquire map point cloud data associated with the obstacle to be predicted from the BEV query data; Updating preset historical map data based on the map point cloud data to generate initial map data; The map element information associated with the obstacle to be predicted is identified from the initial map data.

6. The method according to claim 1, characterized in that The performing target detection on the obstacle in the BEV query data to determine obstacle identification information of the obstacle to be predicted includes: identifying a plurality of horizontal distances between each obstacle and the vehicle from the BEV query data; Determine the obstacle to be predicted from the multiple obstacles according to the multiple horizontal distances, the horizontal distance of the obstacle to be predicted being less than a set threshold; The obstacle identification information of the obstacle to be predicted is determined from the BEV query data.

7. The method according to claim 6, characterized in that The determining the obstacle identification information of the obstacle to be predicted from the BEV query data includes: Determining association information between the obstacle to be predicted and the BEV query data according to a preset variable attention mechanism; Based on the association information, identifying the interaction information between the obstacles from the BEV query data through a multi-head attention mechanism; Based on the association information, the obstacle category information and the obstacle position information are determined from the BEV query data through preset classification and regression branches.

8. The method according to claim 1, characterized in that The performing target detection from the BEV query data and determining map element information associated with the obstacle identification information includes: Determining a plurality of initial map element information associated with the obstacle to be predicted from the BEV query data; Acquire a plurality of prediction confidences corresponding one-to-one to the plurality of initial map element information, and a plurality of distances corresponding one-to-one between the plurality of initial map element information and the obstacles to be predicted; When the target prediction confidence is greater than the confidence threshold and the target distance is less than the distance threshold, the initial map element information corresponding to the target prediction confidence is determined to be the map element information.

9. A trajectory prediction device, characterized in that: The device comprises: A first generating module is configured to extract map element features and obstacle features in the multi-view image to generate a multi-view feature image; A second generating module is configured to convert the multi-view feature image into a BEV bird's-eye view space to generate BEV query data; a first determination module configured to perform target detection from the BEV query data, determine obstacle identification information of the obstacle to be predicted, and map element information associated with the obstacle identification information; a second determination module, configured to determine a motion interaction relationship between the obstacle to be predicted and the map element according to the map element information and the obstacle identification information; The execution module is configured to predict the movement trajectory of the obstacle to be predicted in the map element based on the movement interaction relationship, and generate predicted trajectory data of the obstacle to be predicted.

10. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the steps of the method described in any one of claims 1-8 when executing the executable instructions.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

Images