Track planning method and device, electronic equipment and computer readable medium

Abstract

The invention discloses a trajectory planning method and device, electronic equipment and a computer readable medium, and relates to the technical field of computers.The specific implementation mode comprises the steps that multi-modal sensing data at the current moment is obtained according to a received trajectory planning request, a preset sensing task is executed based on the multi-modal sensing data, and the trajectory planning request is executed according to the preset sensing task; obtaining instance features of a preset dimension; updating an instance feature memory queue based on the instance features to obtain an updated instance feature memory queue; determining vehicle instance features at the current moment, and performing interactive fusion on the vehicle instance features and the updated instance feature memory queue through an attention mechanism to obtain aggregated spatio-temporal context features; and determining a planned trajectory based on the aggregated spatio-temporal context features and a preset optimization condition, and outputting the planned trajectory to the vehicle terminal. Therefore, the trajectory planning efficiency and accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a trajectory planning method and device, electronic equipment and computer readable medium. BACKGROUND

[0002] In the field of automatic driving, the existing modular system usually decomposes the automatic driving task into multiple independent tasks, which are processed by independent modules respectively. This method is prone to cause information loss and error accumulation between modules. The bird's eye view (BEV) is a dense two-dimensional representation, in which each pixel corresponds to a specific area and encodes the point information in the area. When using dense bird's eye view (BEV) features for trajectory planning, the calculation overhead is large, there are a large number of redundant features, and the existing technology is relatively simple in the design of prediction and trajectory planning tasks. The automatic driving trajectory planning is low in efficiency and accuracy. SUMMARY

[0003] Therefore, the embodiments of the present application provide a trajectory planning method and device, electronic equipment and computer readable medium, which can solve the problem of low efficiency and accuracy of the existing automatic driving trajectory planning.

[0004] To achieve the above object, according to an aspect of the embodiments of the present application, a trajectory planning method is provided, comprising:

[0005] Obtaining the multi-modal sensing data of the current time according to the received trajectory planning request, and performing a preset perception task based on the multi-modal sensing data to obtain instance features of a preset dimension;

[0006] Updating the instance feature memory queue based on the instance features to obtain an updated instance feature memory queue;

[0007] Determining the vehicle instance features of the current time, and interacting and fusing the vehicle instance features with the updated instance feature memory queue through an attention mechanism to obtain aggregated spatio-temporal context features;

[0008] Determining a planned trajectory based on the aggregated spatio-temporal context features and a preset optimization condition, and outputting the planned trajectory to a self-vehicle terminal.

[0009] Optionally, determining the vehicle instance features of the current time comprises:

[0010] Obtaining the planned trajectory of the previous time of the self-vehicle as the initial trajectory of the current time, and then determining the vehicle instance features of the current time based on the initial trajectory.

[0011] Optionally, updating the instance feature memory queue based on the instance features to obtain the updated instance feature memory queue comprises:

[0012] determine a current time coordinate system corresponding to the instance feature, and convert the instance features at the historical time in the instance feature memory queue into instance features at the current time in the current time coordinate system through pose projection;

[0013] determine the confidence of each instance feature in the current time coordinate system, and obtain an updated instance feature memory queue at the current time according to the confidence.

[0014] Optionally, the updated instance feature memory queue includes dynamic target instance features, static environment instance features, and online map instance features; and

[0015] The vehicle instance feature is interactively fused with the updated instance feature memory queue through an attention mechanism to obtain aggregated spatio-temporal context features, including:

[0016] The vehicle instance feature is used as query data in the attention mechanism;

[0017] Based on each dynamic target instance feature in the updated instance feature memory queue, a dynamic target is determined, the motion state data of the dynamic target at a future time is predicted, the collision risk between the ego vehicle and the dynamic target is predicted based on the query data and the motion state data, and collision risk information is obtained;

[0018] Based on each static environment instance feature in the updated instance feature memory queue, a static environment at the current time is constructed to obtain static environment data, and the driving feasibility of the ego vehicle in the physical space of the static environment is determined based on the query data and the static environment data to obtain physical space driving feasibility information;

[0019] Based on each online map instance feature in the updated instance feature memory queue, an online map at the current time is constructed to obtain online map data, and a drivable route of the ego vehicle is determined based on the query data and the online map data to obtain drivable route information;

[0020] The aggregated spatio-temporal context features are obtained based on the collision risk information, the physical space driving feasibility information, and the drivable route information.

[0021] Optionally, based on the aggregated spatio-temporal context features and a preset optimization condition, a planning trajectory is determined, including:

[0022] The aggregated spatio-temporal context features are input into a trajectory generation model, so that the trajectory generation model generates a plurality of candidate trajectories based on the preset optimization condition;

[0023] The comprehensive score of each candidate trajectory is calculated through a multi-objective decision algorithm, and the planning trajectory is determined from the plurality of candidate trajectories according to the comprehensive score.

[0024] Optionally, the comprehensive score of each candidate trajectory is calculated through a multi-objective decision algorithm, including:

[0025] The geometric-based collision detection algorithm or the probability-based method evaluates the possibility of collision of the ego vehicle when driving in each candidate trajectory to obtain a safety score of each candidate trajectory;

[0026] The acceleration change value and the direction change value of the ego vehicle when driving in each candidate trajectory are evaluated, and it is determined whether the acceleration change value and the direction change value corresponding to each candidate trajectory exceed a preset range, and a smoothness score of each candidate trajectory is determined according to the determination result;

[0027] The energy consumption of the ego vehicle when driving in each candidate trajectory is evaluated to obtain an energy consumption score of each candidate trajectory;

[0028] Based on the safety score, the smoothness score and the energy consumption score, a comprehensive score of each candidate trajectory is obtained.

[0029] In addition, the application also provides a trajectory planning device, comprising:

[0030] The sparse perception unit is configured to obtain multi-modal sensing data of the current time according to the received trajectory planning request, and perform a preset perception task based on the multi-modal sensing data to obtain instance features of a preset dimension;

[0031] The update unit is configured to update the instance feature memory queue based on the instance features to obtain an updated instance feature memory queue;

[0032] The interactive fusion unit is configured to determine the vehicle instance features of the current time, and interactively fuse the vehicle instance features with the updated instance feature memory queue through an attention mechanism to obtain aggregated spatio-temporal context features;

[0033] The trajectory planning unit is configured to determine a planning trajectory based on the aggregated spatio-temporal context features and a preset optimization condition, and output the planning trajectory to the ego vehicle terminal.

[0034] Optionally, the interactive fusion unit is further configured to:

[0035] Obtain the planning trajectory of the ego vehicle at the previous time as the initial trajectory of the current time, and then determine the vehicle instance features of the current time based on the initial trajectory.

[0036] Optionally, the update unit is further configured to:

[0037] Determine the coordinate system of the current time corresponding to the instance features, and convert the instance features of the historical time in the instance feature memory queue into instance features in the coordinate system of the current time through pose projection;

[0038] Determine the confidence of each instance feature in the current time coordinate system, and obtain the updated instance feature memory queue in the current time according to the confidence.

[0039] Optionally, the updated instance feature memory queue includes dynamic target instance features, static environment instance features, and online map instance features.

[0040] The interaction fusion unit is further configured to:

[0041] Take the vehicle instance feature as query data in the attention mechanism;

[0042] Determine the dynamic target based on each dynamic target instance feature in the updated instance feature memory queue, predict the motion state data of the dynamic target at a future time, and predict the collision risk between the ego vehicle and the dynamic target based on the query data and the motion state data to obtain collision risk information;

[0043] Construct the static environment at the current time based on each static environment instance feature in the updated instance feature memory queue to obtain static environment data, and determine the driving feasibility of the ego vehicle in the physical space of the static environment based on the query data and the static environment data to obtain physical space driving feasibility information;

[0044] Construct the online map at the current time based on each online map instance feature in the updated instance feature memory queue to obtain online map data, and determine the drivable route of the ego vehicle based on the query data and the online map data to obtain drivable route information;

[0045] Obtain the aggregated spatio-temporal context feature based on the collision risk information, the physical space driving feasibility information, and the drivable route information.

[0046] Optionally, the trajectory planning unit is further configured to:

[0047] Input the aggregated spatio-temporal context feature into the trajectory generation model to enable the trajectory generation model to generate multiple candidate trajectories based on a preset optimization condition;

[0048] Calculate the comprehensive score of each candidate trajectory through a multi-objective decision algorithm, and determine the planning trajectory from the multiple candidate trajectories according to the comprehensive score.

[0049] Optionally, the trajectory planning unit is further configured to:

[0050] Evaluate the possibility of collision when the ego vehicle drives in each candidate trajectory based on a geometric collision detection algorithm or a probabilistic method to obtain a safety score of each candidate trajectory;

[0051] The acceleration change value and the direction change value of the ego vehicle when driving in each candidate trajectory are evaluated, whether the acceleration change value and the direction change value corresponding to each candidate trajectory exceed a preset range is judged, and the smoothness score of each candidate trajectory is determined according to the judgment result;

[0052] The energy consumption of the ego vehicle when driving in each candidate trajectory is evaluated to obtain the energy consumption score of each candidate trajectory;

[0053] Based on the safety score, the smoothness score and the energy consumption score, a comprehensive score of each candidate trajectory is obtained.

[0054] In addition, the present application also provides a trajectory planning electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the trajectory planning method as described above.

[0055] In addition, the present application also provides a computer readable medium having a computer program stored thereon, the program being executed by a processor to implement the trajectory planning method as described above.

[0056] To achieve the above-mentioned purpose, according to another aspect of the embodiments of the present application, a computer program product is provided.

[0057] The computer program product of the embodiments of the present application comprises a computer program, which is executed by a processor to implement the trajectory planning method provided by the embodiments of the present application.

[0058] The above-mentioned embodiment of the application has the following advantages or beneficial effects: the present application obtains the multi-modal sensing data at the current time according to the received trajectory planning request, performs the preset perception task based on the multi-modal sensing data to obtain the instance features of the preset dimension, updates the instance feature memory queue based on the instance features to obtain the updated instance feature memory queue, determines the vehicle instance features at the current time, and interacts and fuses the vehicle instance features with the updated instance feature memory queue through the attention mechanism to obtain the aggregated spatio-temporal context features. Based on the aggregated spatio-temporal context features and the preset optimization condition, the planning trajectory is determined and output to the ego vehicle terminal. Thus, the trajectory planning efficiency and accuracy can be improved.

[0059] The further effects of the above-mentioned non-conventional optional mode will be described in the following combined with the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings serve to better understand the present application and do not constitute an improper limitation on the present application. Among them:

[0061] Figure 1is a schematic diagram of the main flow of the trajectory planning method according to an embodiment of the present application;

[0062] Figure 2 is a schematic diagram of the main flow of the trajectory planning method according to an embodiment of the present application;

[0063] Figure 3 is a schematic diagram of the main flow of the trajectory planning method according to an embodiment of the present application;

[0064] Figure 4 is a schematic diagram of the main unit of the trajectory planning device according to an embodiment of the present application;

[0065] Figure 5 is an exemplary system architecture diagram to which embodiments of the present application can be applied;

[0066] Figure 6 is a structural schematic diagram of a computer system of a terminal device or a server suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0067] Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding, which should be considered only as exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present application comply with relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present application, some industry existing solutions, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution. In the technical solutions of the present application, the collection, collection, updating, analysis, processing, use, transmission, storage, etc. of user personal information comply with relevant laws and regulations, are used for legal and reasonable purposes, and do not violate public order and good customs, are not shared, leaked or sold outside these legal uses, and are subject to supervision and management by regulatory authorities. Necessary measures are taken for user personal information to prevent illegal access to such user personal information data, to maintain user personal information security, network security and national security, to ensure that personnel with access to personal information data comply with relevant laws and regulations, and to ensure user personal information security. Once these user personal information data are no longer needed, the risk should be minimized by limiting or even prohibiting data collection and / or deleting data.

[0068] When in use, including in certain related applications, user privacy is protected by de-identifying data, such as by removing specific identifiers, controlling the amount or specificity of data stored, controlling how data is stored, and / or other methods of de-identification.

[0069] Figure 1 is a schematic diagram of the main flow of the trajectory planning method according to an embodiment of the present application, as shown, the trajectory planning method mainly includes the following steps S101-S104. Figure 1

[0070] Step S101, according to the received trajectory planning request, obtain the multi-modal sensor data at the current time, perform a preset perception task based on the multi-modal sensor data, and obtain instance features of a preset dimension.

[0071] In this embodiment, the execution subject of the trajectory planning method (for example, it can be a server of a trajectory planning system) can receive a trajectory planning request through wired connection or wireless connection. The trajectory planning request may, for example, be a request for trajectory planning of an autonomous vehicle. The current time is the time when the trajectory planning request is received. According to the trajectory planning request, the vehicle identifier (i.e., the vehicle identifier of the ego vehicle) for which trajectory planning needs to be performed is obtained, and the multi-modal sensor data of the vehicle (i.e., the ego vehicle) corresponding to the vehicle identifier at the current time is obtained. For example, the multi-modal sensor data can be multi-channel camera images, laser radar point clouds, and millimeter radar point clouds. Feature extraction is performed on the multi-modal sensor data, such as feature extraction on multi-channel camera images to obtain visual image features, feature extraction on laser radar point clouds to obtain laser radar point cloud features, and feature extraction on millimeter wave radar point clouds to obtain millimeter wave radar point cloud features. The extracted features of the multi-modal sensor data (such as visual image features, laser radar point cloud features, and millimeter wave radar point cloud features) are input into a sparse perception module to complete a preset perception task (such as a sparse dynamic target detection task, a sparse static environment construction task, and a sparse online map construction task), so as to obtain instance features of a preset dimension (such as dynamic target instance features, static environment instance features, and online map instance features).

[0072] Step S102, update the instance feature memory queue based on the instance features to obtain an updated instance feature memory queue.

[0073] Specifically, updating the instance feature memory queue based on the instance features to obtain an updated instance feature memory queue includes: determining a current time coordinate system corresponding to the instance features, converting the instance features at the historical time in the instance feature memory queue into instance features in the current time coordinate system through pose projection; determining the confidence of each instance feature in the current time coordinate system, and obtaining the updated instance feature memory queue at the current time according to the confidence. ​

[0074] In an example, the determination step S101 obtains the current time coordinate system corresponding to the preset dimension instance features, and converts the instance features at the previous t-1 time points (i.e., the instance features at the historical time points, for example, the dynamic target instance features, the static environment instance features, and the online map instance features at the t-n, …, t-2, t-1 time points) in the instance feature memory queue to the current time coordinate system through the pose projection matrix from the time points corresponding to the instance features to the current time. Then, the top K instance features are selected through the confidence of the instance features to be reserved in the instance feature memory queue, so as to obtain the updated instance feature memory queue at the current time.

[0075] In step S103, the vehicle instance feature at the current time is determined, and the vehicle instance feature is interactively fused with the updated instance feature memory queue through the attention mechanism to obtain the aggregated spatio-temporal context feature.

[0076] The planning trajectory at the previous time point on the ego vehicle is used as the initial trajectory at the current time, and the ego vehicle instance feature is initialized based on the initial trajectory to obtain the vehicle instance feature at the current time. The vehicle instance feature actually implies the direction, speed, acceleration, and other information of the future motion of the ego vehicle.

[0077] Specifically, the updated instance feature memory queue includes the dynamic target instance feature, the static environment instance feature, and the online map instance feature; and the vehicle instance feature is interactively fused with the updated instance feature memory queue through the attention mechanism to obtain the aggregated spatio-temporal context feature, including: taking the vehicle instance feature as the query data (i.e., query) in the attention mechanism; determining a dynamic target (e.g., a traffic participant, which can be a pedestrian or a vehicle driving around) based on each dynamic target instance feature in the updated instance feature memory queue, predicting motion state data (i.e., predicting the moving speed, position, size, and orientation of the dynamic target at the future time) of the dynamic target at the future time, predicting the collision risk of the ego vehicle and the dynamic target based on the query data and the motion state data to obtain collision risk information. Specifically, the collision risk between the ego vehicle and the dynamic target can be predicted using a geometric-based collision detection algorithm or a probabilistic-based method in combination with the query data and the motion state data to obtain the collision risk information. In an example, the collision risk information can include the collision probability of the ego vehicle and the dynamic target at the future time.

[0078] The static environment at the current time is constructed based on each static environment instance feature in the updated instance feature memory queue (for example, the static environment can include static obstacles, roads, etc.), to obtain static environment data (for example, the static environment data can be the position, size, spacing between obstacles, spacing between obstacles and the edge of the road in the static environment, etc. of the static obstacles included in the static environment), and the driving feasibility of the ego vehicle in the physical space in the static environment is determined based on the query data and the static environment data, to obtain the physical space driving feasibility information. For example, the physical space driving feasibility information can include the probability of collision between the ego vehicle and the static obstacles, the edge of the road, etc. included in the static environment.

[0079] The online map at the current time is constructed based on each online map instance feature in the updated instance feature memory queue, to obtain online map data (for example, the online map data can include map data that formats static map elements into a polyline with N points, road traffic marking data, traffic signal position data, etc.), and the drivable route of the ego vehicle is determined based on the query data and the online map data, to obtain the drivable route information.

[0080] The aggregated spatio-temporal context features are obtained based on the collision risk information, the physical space driving feasibility information, and the drivable route information. For example, the collision risk features of the collision risk information, the physical space driving feasibility features of the physical space driving feasibility information, and the drivable route features of the drivable route information can be extracted respectively, and the extracted collision risk features, physical space driving feasibility features, and drivable route features are aggregated to accurately obtain the aggregated spatio-temporal context features.

[0081] In step S104, the planning trajectory is determined based on the aggregated spatio-temporal context features and the preset optimization condition, and the planning trajectory is output to the ego vehicle terminal.

[0082] The preset optimization condition can include, for example, minimizing path deviation, optimizing ride comfort (for example, minimizing acceleration change), and maximizing efficiency.

[0083] The aggregated spatio-temporal context features are input into the trajectory generation model, so that the trajectory generation model plans multiple ego vehicle trajectories based on the preset optimization condition, to obtain multiple candidate trajectories. The safety, smoothness, and efficiency of the obtained multiple candidate trajectories are evaluated, and then the comprehensive score of each candidate trajectory is obtained, the candidate trajectory with the highest comprehensive score is selected as the final planning trajectory, and the final planning trajectory is output to the ego vehicle terminal to accurately guide the driving of the ego vehicle.

[0084] The embodiment obtains multi-modal sensing data at the current time according to the received trajectory planning request, performs a preset perception task based on the multi-modal sensing data to obtain instance features of a preset dimension, updates an instance feature memory queue based on the instance features to obtain an updated instance feature memory queue, determines a vehicle instance feature at the current time, interacts and fuses the vehicle instance feature with the updated instance feature memory queue through an attention mechanism to obtain aggregated spatio-temporal context features, determines a planning trajectory based on the aggregated spatio-temporal context features and a preset optimization condition, and outputs the planning trajectory to a terminal of the ego vehicle. Thus, the trajectory planning efficiency and accuracy can be improved.

[0085] Figure 2 is a main flow diagram of a trajectory planning method according to an embodiment of the present application, as shown in Figure 2 The trajectory planning method mainly includes the following steps S201-S207.

[0086] Step S201: Multi-modal sensing data at the current time is obtained according to a received trajectory planning request, and a preset perception task is performed based on the multi-modal sensing data to obtain instance features of a preset dimension.

[0087] For example, the obtained multi-modal sensing data of the ego vehicle at the current time can be multi-channel camera images, laser radar point clouds and millimeter wave radar point clouds. Feature extraction is performed on the multi-modal sensing data, for example, feature extraction is performed on the multi-channel camera images to obtain visual image features, feature extraction is performed on the laser radar point clouds to obtain laser radar point cloud features, and feature extraction is performed on the millimeter wave radar point clouds to obtain millimeter wave radar point cloud features. The extracted features of the multi-modal sensing data (for example, visual image features, laser radar point cloud features and millimeter wave radar point cloud features) are input into a sparse perception module to complete a preset perception task (for example, a sparse dynamic target detection task, a sparse static environment construction task and a sparse online map construction task) to obtain instance features of a preset dimension (for example, dynamic target instance features, static environment instance features and online map instance features).

[0088] Step S202: The instance feature memory queue is updated based on the instance features to obtain an updated instance feature memory queue.

[0089] For example, the coordinate system corresponding to the instance features of the preset dimension obtained in step S201 is determined. The instance features of the previous t-1 time steps (i.e., instance features of historical time steps, such as dynamic target instance features, static environment instance features, and online map instance features at time steps tn, ..., t-2, t-1) in the instance feature memory queue are transformed to the coordinate system of the current time step using the pose projection matrix of these instance features from the corresponding time step to the current time step. Then, a preset number of instance features are selected based on the confidence level of these instance features and retained in the instance feature memory queue to accurately obtain the instance feature memory queue updated at the current time step.

[0090] Step S203: Obtain the planned trajectory of the vehicle at the previous moment as the initial trajectory at the current moment, and then determine the vehicle instance features at the current moment based on the initial trajectory.

[0091] The planned trajectory of the vehicle at the previous time (e.g., time t-1) is used as the initial trajectory at the current time (e.g., time t). The vehicle's direction features, speed features, acceleration features, and other features corresponding to this initial trajectory are extracted as vehicle instance features at the current time.

[0092] Step S204: The vehicle instance features are interacted and fused with the updated instance feature memory queue through an attention mechanism to obtain aggregated spatiotemporal context features.

[0093] The vehicle instance features are used as the query in the attention mechanism. The updated instance features in the memory queue are concatenated as the key and value in the attention mechanism. Interactive fusion is achieved through the calculation of the attention mechanism to obtain aggregated spatiotemporal context features, thereby achieving the fusion of time and space dimensions and improving the accuracy of trajectory planning.

[0094] Step S205: The aggregated spatiotemporal context features are input into the trajectory generation model so that the trajectory generation model generates multiple candidate trajectories based on preset optimization conditions.

[0095] The trajectory generation model can be, for example, a multilayer perceptron (MLP) model, which is trained using a deep reinforcement learning optimization strategy. The obtained aggregated spatiotemporal context features are input into the multilayer perceptron model. Based on the input data, the model drives trajectory generation according to preset optimization conditions (e.g., minimizing path deviation, minimizing acceleration change, and maximizing efficiency), ultimately generating multiple candidate trajectories for the vehicle.

[0096] Step S206, calculate the comprehensive score of each candidate trajectory by the multi-objective decision algorithm, and determine the planning trajectory from the multiple candidate trajectories according to the comprehensive score.

[0097] Specifically, the comprehensive score of each candidate trajectory is calculated by the multi-objective decision algorithm, including:

[0098] The safety score of each candidate trajectory is obtained by evaluating the possibility of collision of the ego vehicle when driving in each candidate trajectory based on a geometric-based collision detection algorithm or a probabilistic-based method. For example, collision detection is performed on each generated candidate trajectory, and the collision possibility with dynamic targets and static obstacles is calculated. The safety of each candidate trajectory can be evaluated using a geometric-based collision detection algorithm or a probabilistic-based method to obtain a safety score of each candidate trajectory. The higher the safety score, the higher the safety of the ego vehicle when driving according to the corresponding candidate trajectory. It is ensured that the final planning trajectory is safe in all possible situations, i.e. avoiding collision between the ego vehicle and any obstacle.

[0099] The acceleration change value and the direction change value of the ego vehicle when driving in each candidate trajectory are evaluated, and it is determined whether the acceleration change value and the direction change value corresponding to each candidate trajectory exceed a preset range. The smoothness score of each candidate trajectory is determined according to the determination result. The higher the smoothness score, the higher the smoothness of the ego vehicle when driving according to the corresponding candidate trajectory. The smoothness of the candidate trajectory is evaluated by the smoothness score to ensure that the acceleration and direction change of the ego vehicle when driving according to the final planning trajectory are within a reasonable range, so as to provide a comfortable riding experience.

[0100] The energy consumption of the ego vehicle when driving in each candidate trajectory is evaluated to obtain an energy consumption score of each candidate trajectory. The higher the energy consumption score, the lower the energy consumption of the ego vehicle when driving according to the corresponding candidate trajectory. The energy efficiency of the ego vehicle is considered, and the candidate trajectory with lower energy consumption (e.g. energy consumption score higher than a preset threshold) is preferentially selected.

[0101] The comprehensive score of each candidate trajectory is obtained based on the safety score, the smoothness score and the energy consumption score. For example, the safety score, the smoothness score and the energy consumption score corresponding to each candidate trajectory can be added to obtain the comprehensive score of each candidate trajectory, which is used to accurately determine the planning trajectory in the candidate trajectory. The planning trajectory is determined from the multiple candidate trajectories according to the comprehensive score. For example, the candidate trajectory corresponding to the highest comprehensive score can be taken as the final planning trajectory.

[0102] Step S207, output the planning trajectory to the ego vehicle terminal.

[0103] The planning trajectory is output to the ego vehicle terminal to accurately guide the actual driving behavior of the ego vehicle.

[0104] Figure 3is a schematic diagram of a trajectory planning method according to an embodiment of the present application. The trajectory planning method of the present application can unify multiple tasks in autonomous driving into a fully differentiable framework by using an end-to-end approach, allowing the optimization of the entire end-to-end model in a self-vehicle planning oriented manner, and fully learning the habits of excellent human drivers to complete safer and more comfortable autonomous driving trajectory planning. The embodiments of the present application can ensure the integrity of the information used for autonomous driving trajectory planning, reduce error accumulation, and improve trajectory planning efficiency and accuracy. The trajectory planning method of the embodiments of the present application can be implemented through a trajectory planning system. The trajectory planning system can include a sensor feature extraction module, a sparse perception module, a time sequence feature fusion module, and a motion planning module. The input data of the trajectory planning system can be point cloud data and two-dimensional images. The point cloud data can be the output data of the laser radar and millimeter wave radar of the autonomous vehicle (i.e., the ego vehicle). The two-dimensional images can be 2D images output by multiple cameras with different orientations on the autonomous vehicle (i.e., the ego vehicle). Based on the input data, the trajectory planning system outputs the final planning trajectory through the process of Figure 3 , to guide the end-to-end vehicle autonomous driving. As shown in Figure 3 , the trajectory planning method of the embodiments of the present application, i.e., the sparse representation end-to-end autonomous driving method, involves the following four modules: a sensor feature extraction module, a sparse perception module, a time sequence feature fusion module, and a motion planning module. Specifically, the trajectory planning method can be implemented through the following steps 1-4:

[0105] Step 1: using the sensor feature extraction module to extract features from the multi-modal sensor data (such as multi-camera images, laser radar point clouds, and millimeter radar point clouds) of the autonomous vehicle.

[0106] ① Visual image feature extraction: using an image preprocessing module to process the input multi-camera images I of the autonomous vehicle to obtain processed multi-camera images, and then using a visual feature extraction module composed of a convolutional neural network (CNN) to extract features of the processed multi-camera images to obtain visual image features F I . This process can be implemented through a convolutional neural network composed of multiple convolutional layers, activation functions, and pooling layers. The formula can be expressed as: F I =CNN(I).

[0107] ② Laser radar point cloud feature extraction: using a laser radar point cloud preprocessing module to process the input laser radar point cloud P L to obtain processed laser radar point clouds, and then using a laser radar feature extraction module (such as PointNet or a Voxel-based 3D CNN) to extract features of the processed laser radar point clouds to obtain laser radar point cloud features F PLThe formula can be expressed as: F PL = PointNet (P L ).

[0108] ③ Millimeter wave radar point cloud feature extraction: using the millimeter wave radar point cloud preprocessing module to process the input millimeter wave radar point cloud P R to obtain the processed millimeter wave radar point cloud, and then using the millimeter wave radar feature extraction module (such as PointNet or Voxel-based 3D CNN) to extract features from the processed millimeter wave radar point cloud, to obtain millimeter wave radar point cloud features F PR . The formula can be expressed as: F PR = PointNet (P R ).

[0109] Therefore, through the above feature extraction method, the sensor features (such as visual image features, laser radar point cloud features and millimeter wave radar point cloud features) of typical automatic driving vehicles can be extracted, thereby serving as the input for the next step.

[0110] Step 2: input each sensor feature (such as visual image feature, laser radar point cloud feature and millimeter wave radar point cloud feature) obtained in step 1 into the sparse perception module (which performs detection, tracking and online map construction tasks through a symmetric model architecture to learn a completely sparse scene representation) to complete the perception task, identify the traffic participants, obstacles and map elements around the vehicle, and obtain three groups of instance features as the input of the time sequence feature fusion module in step 3. Step 2 includes the following ①-③ three parts:

[0111] ① Dynamic target instance feature extraction: input all sensor features in step 1 into the sparse dynamic target detection branch to use a set of dynamic target instance features (NxC) to represent the surrounding dynamic targets, where N represents the number of potential dynamic targets (instances), and C represents the feature vector dimension of each dynamic target (instance). The sparse dynamic target detection branch takes the sensor features obtained in step 1 and randomly initialized instance features as input, and outputs updated dynamic target instance features (NxC).

[0112] ② Static environment instance feature extraction: input all sensor features in step 1 into sparse static environment construction branch to represent the static environment of 3D space within the vehicle's (i.e. ego vehicle) surrounding environment using a set of static environment instance features (H x W x Z x C), where H, W and Z represent the length, width and height size of the 3D space, and C represents the feature vector dimension at each spatial location. The sparse static environment construction branch takes the sensor features obtained in step 1 and randomly initialized static environment instance features as input, and outputs updated static environment instance features (H x W x Z x C).

[0113] ③ Online map instance feature extraction: input all sensor features in step 1 into sparse online map construction branch to represent the map elements within the vehicle's surrounding environment using a set of online map instance features. The sparse online map construction branch takes the sensor features obtained in step 1 and randomly initialized map instance features as input, and outputs updated online map instance features. Static map elements are formatted as a polyline with N points.

[0114] Step 3: Temporal feature fusion module: update temporal instance features based on the dynamic target instance features, static environment instance features and online map instance features obtained in step 2 at the current time (e.g. time t).

[0115] ① Update the dynamic target instance features, static environment instance features and online map instance features obtained in step 2 at the current time (e.g. time t) to the instance feature memory queue. The specific update method is: convert the instance features at the previous t-1 time (e.g. dynamic target instance features, static environment instance features and online map instance features at time t-n, …, t-2, t-1) in the instance feature memory queue to the current time coordinate system through the pose projection matrix from the corresponding time to the current time. Then select topK instance features to remain in the instance feature memory queue through the confidence of these instance features.

[0116] ② Use the updated dynamic target instance features, static environment instance features and online map instance features in the instance feature memory queue to predict the corresponding 3D target detection results, static environment construction results and online map construction results respectively through a multi-layer perception model. These results actually do not act on the subsequent motion planning, but are output as some intermediate results in autonomous driving to enhance the explainability of the system and facilitate the understanding of users or developers.

[0117] ③ Sparse dynamic target tracking: through the sparse dynamic target tracking branch, the dynamic target instance features in the instance feature memory queue updated in step 3 and the 3D target detection results obtained in step 3 are used to perform corresponding target tracking, the motion state data of the 3D target (i.e. dynamic target) at the future time (i.e. the moving speed, position, size and orientation of the 3D target (i.e. dynamic target) at the future time) is predicted, and the 3D target (i.e. dynamic target) tracking result is obtained. Among them, once the detection confidence of the instance exceeds the pre-set threshold, it will be locked as a target and assigned an ID, which remains unchanged in the time propagation process.

[0118] Step 4: Motion planning module (this module is designed through end-to-end, simultaneously performing motion prediction and planning, generating multiple candidate trajectories, and ensuring the rationality and safety of planning through trajectory planning selection strategy).

[0119] ① Self-vehicle instance initialization: the planning trajectory of the self-vehicle at the previous time is used to initialize the self-vehicle instance feature. Specifically, the planning trajectory of the self-vehicle at the previous time is used as the initial trajectory at the current time, and the self-vehicle instance feature is initialized based on the initial trajectory to obtain the vehicle instance feature at the current time. This vehicle instance feature actually implies the direction, speed, acceleration and other information of the future motion of the self-vehicle, so that these future motion direction, speed, acceleration and other information of the self-vehicle and the surrounding traffic participants and the map are used to interact in the subsequent process to make better game and decision. In addition, historical trajectory information can be used to smooth the transition to a new trajectory planning period, ensuring the continuity and stability of vehicle motion.

[0120] ②Instance Feature Spatio-temporal Interaction: The vehicle instance feature is interacted with the updated dynamic target instance feature, static environment instance feature, and online map instance feature obtained in step 3 respectively through the instance feature spatio-temporal interaction module to aggregate the spatio-temporal context feature. Specifically, the instance feature spatio-temporal interaction module can interact and fuse the vehicle instance feature with the updated dynamic target instance feature, static environment instance feature, and online map instance feature obtained in step 3 through attention mechanism cross-attention, wherein the vehicle instance feature is taken as the query in the attention mechanism, the dynamic target instance feature, static environment instance feature, and online map instance feature are concatenated as the key and value in the attention mechanism, the interaction is realized through the calculation of the attention mechanism cross-attention, and finally the fused feature is obtained. Since the input dynamic target instance feature, static environment instance feature, and online map instance feature are the updated continuous time sequence fused features output by step 3, the interaction here actually realizes a fusion in the time and space dimensions, thereby integrating the spatio-temporal context information of different sources to obtain the aggregated spatio-temporal context feature. Essentially, the interaction process can be visualized as:

[0121] a) Dynamic Target Interaction: The vehicle instance feature actually implies the direction, speed, acceleration, and other information of the future motion of the ego vehicle. Through the instance feature spatio-temporal interaction module, dynamic target trajectory prediction can also be performed, and then dynamic target interaction can be realized based on the vehicle instance feature and the predicted dynamic target trajectory. Specifically, the motion intention behavior of the ego vehicle at a future time can be evaluated with the relative position and relative motion trend of the surrounding dynamic targets (such as pedestrians and vehicles) through the vehicle instance feature and the predicted dynamic target trajectory, and the possible collision risk can be predicted to obtain collision risk information.

[0122] b) Static Environment Interaction: Combined with the motion intention behavior of the ego vehicle at a future time implied by the vehicle instance feature, the surrounding road structure, obstacles, and other static elements are analyzed to ensure that the planned trajectory is physically feasible, to obtain physical space driving feasibility information.

[0123] c) Map Feature Interaction: Combined with the map information, it is ensured that the trajectory follows the traffic rules and road restrictions, such as lane keeping, turning restrictions, etc., to obtain drivable route information.

[0124] Based on the obtained collision risk information, physical space driving feasibility information, and drivable route information, the aggregated spatio-temporal context feature is obtained.

[0125] ③Trajectory Prediction and Generation: The aggregated spatio-temporal context feature is input into the trajectory generation model for multiple ego vehicle trajectory planning to obtain multiple candidate trajectories.

[0126] a) The trajectory generation model can be a multilayer perceptron (MLP) model trained using deep reinforcement learning optimization strategies. The aggregated spatio-temporal context features are input into the multilayer perceptron (MLP) model to drive the model based on the input data for trajectory generation, which ultimately generates multiple candidate trajectories for the ego vehicle.

[0127] b) Multi-objective optimization: Multiple loss functions (optimization objectives) are designed when the model generates trajectories. Each loss function corresponds to minimizing path deviation, optimizing ride comfort (e.g., minimizing acceleration changes), and maximizing efficiency.

[0128] ④ Trajectory planning post-processing: Final trajectory planning is performed through the trajectory planning post-processing module. Specifically, safety can be guaranteed based on rules:

[0129] a) Collision detection and avoidance: Collision detection is performed on each candidate trajectory to calculate its collision probability with dynamic targets and static obstacles. A geometric-based collision detection algorithm or a probabilistic-based method can be used to evaluate the safety of each candidate trajectory, resulting in a safety score for each candidate trajectory.

[0130] b) Trajectory scoring and selection: A comprehensive score is calculated for each candidate trajectory, considering factors such as safety, smoothness, efficiency, etc. The comprehensive score is optimized through a multi-objective decision algorithm. The multi-objective mainly includes the following three aspects: safety priority: ensure that the final planned trajectory selected is safe in all possible situations, i.e., avoid collisions with any obstacles. Smoothness and comfort: evaluate the smoothness of the trajectory to ensure that the acceleration and direction changes of the vehicle motion are within a reasonable range to provide a comfortable ride experience. Efficiency and energy efficiency: consider the energy efficiency of the vehicle and preferentially select candidate trajectories with lower energy consumption.

[0131] ⑤ Final trajectory output: The candidate trajectory with the highest comprehensive score in the trajectory planning post-processing is selected as the final planned trajectory output. This output trajectory will serve as a reference for the vehicle's motion in the next planning period, guiding the actual driving behavior of the vehicle.

[0132] Figure 4 is a schematic diagram of the main units of the trajectory planning device according to an embodiment of the present application. As shown in Figure 4 , the trajectory planning device 400 includes a sparse perception unit 401, an update unit 402, an interactive fusion unit 403, and a trajectory planning unit 404.

[0133] The sparse perception unit 401 is configured to obtain multi-modal sensing data of a current time according to a received trajectory planning request, perform a preset perception task based on the multi-modal sensing data to obtain instance features of a preset dimension.

[0134] The updating unit 402 is configured to update the instance feature memory queue based on the instance features to obtain an updated instance feature memory queue.

[0135] The interaction fusion unit 403 is configured to determine vehicle instance features of the current time, and interactively fuse the vehicle instance features with the updated instance feature memory queue through an attention mechanism to obtain aggregated spatio-temporal context features.

[0136] The trajectory planning unit 404 is configured to determine a planning trajectory based on the aggregated spatio-temporal context features and a preset optimization condition, and output the planning trajectory to a vehicle terminal.

[0137] In some embodiments, the interaction fusion unit 403 is further configured to: obtain a planning trajectory of a previous time of the vehicle as an initial trajectory of the current time, and then determine the vehicle instance features of the current time based on the initial trajectory.

[0138] In some embodiments, the updating unit 402 is further configured to: determine a coordinate system of the current time corresponding to the instance features, convert instance features of historical times in the instance feature memory queue into instance features in the coordinate system of the current time through pose projection; determine a confidence of each instance feature in the coordinate system of the current time, and obtain the updated instance feature memory queue of the current time according to the confidence.

[0139] In some embodiments, the dynamic target instance features, the static environment instance features, and the online map instance features are included in the updated instance feature memory queue; and the interaction fusion unit 403 is further configured to: take the vehicle instance features as query data in an attention mechanism; determine dynamic targets based on the respective dynamic target instance features in the updated instance feature memory queue, predict motion state data of the dynamic targets at a future time, and predict a collision risk between the ego vehicle and the dynamic targets based on the query data and the motion state data to obtain collision risk information; construct a static environment at the current time based on the respective static environment instance features in the updated instance feature memory queue to obtain static environment data, and determine a driving feasibility of the ego vehicle in a physical space of the static environment based on the query data and the static environment data to obtain physical space driving feasibility information; construct an online map at the current time based on the respective online map instance features in the updated instance feature memory queue to obtain online map data, and determine a drivable route of the ego vehicle based on the query data and the online map data to obtain drivable route information; and obtain aggregated spatio-temporal context features based on the collision risk information, the physical space driving feasibility information, and the drivable route information.

[0140] In some embodiments, the trajectory planning unit 404 is further configured to: input the aggregated spatio-temporal context features into a trajectory generation model, so that the trajectory generation model generates a plurality of candidate trajectories based on a preset optimization condition; calculate a comprehensive score of each candidate trajectory by a multi-objective decision algorithm, and determine a planned trajectory from the plurality of candidate trajectories according to the comprehensive scores.

[0141] In some embodiments, the trajectory planning unit 404 is further configured to: evaluate a possibility of a collision of the ego vehicle when driving in each candidate trajectory by a geometric-based collision detection algorithm or a probabilistic-based method to obtain a safety score of each candidate trajectory; evaluate an acceleration change value and a direction change value of the ego vehicle when driving in each candidate trajectory, determine whether the acceleration change value and the direction change value corresponding to each candidate trajectory exceed a preset range, and determine a smoothness score of each candidate trajectory according to a result of the determination; evaluate an energy consumption of the ego vehicle when driving in each candidate trajectory to obtain an energy consumption score of each candidate trajectory; and obtain a comprehensive score of each candidate trajectory based on the safety score, the smoothness score, and the energy consumption score.

[0142] It should be noted that the trajectory planning method and the trajectory planning device of the present application have a corresponding relationship in the specific implementation content, and therefore repeated content will not be described.

[0143] Figure 5 An exemplary system architecture 500 to which the trajectory planning method or the trajectory planning device of the embodiments of the present application can be applied is shown.

[0144] As Figure 5As shown, the system architecture 500 can include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 is a medium for providing communication links between the terminal devices 501, 502, 503 and the server 505. The network 504 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0145] A user can use the terminal devices 501, 502, 503 to interact with the server 505 through the network 504 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0146] The terminal devices 501, 502, 503 can be various electronic devices with trajectory planning processing screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0147] The server 505 can be a server providing various services, such as a background management server providing support for trajectory planning requests submitted by users using the terminal devices 501, 502, 503 (only as an example). The background management server can obtain multi-modal sensor data at the current time according to the received trajectory planning request, perform a preset perception task based on the multi-modal sensor data to obtain instance features in a preset dimension; update the instance feature memory queue based on the instance features to obtain an updated instance feature memory queue; determine a vehicle instance feature at the current time, and interact and fuse the vehicle instance feature with the updated instance feature memory queue through an attention mechanism to obtain an aggregated spatio-temporal context feature; determine a planning trajectory based on the aggregated spatio-temporal context feature and a preset optimization condition, and output the planning trajectory to the ego vehicle terminal. Thus, the trajectory planning efficiency and accuracy can be improved.

[0148] It should be noted that the trajectory planning method provided by the embodiments of the present application is generally executed by the server 505, and correspondingly, the trajectory planning device is generally arranged in the server 505.

[0149] It should be understood that Figure 5 The number of terminal devices, networks, and servers in the system architecture 500 is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks, and servers.

[0150] Reference is made below to Figure 6 which shows a structural schematic diagram of a computer system 600 of a terminal device suitable for implementing the embodiments of the present application. Figure 6The terminal device shown is merely an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0151] As shown in Figure 6 The computer system 600 includes a central processing unit (CPU) 601 which can perform various appropriate actions and processes in accordance with a program stored in a read only memory (ROM) 602 or a program loaded from the storage section 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0152] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable media 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read out therefrom is installed into the storage section 608 as necessary.

[0153] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 609, and / or installed from the removable media 611. When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the system of the present application are executed.

[0154] Note that the computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or a combination thereof. The computer-readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, the computer-readable storage medium can be any tangible medium that contains or stores a program used by an instruction execution system, apparatus, or device to function according to the program. In the present application, the computer-readable signal medium can include a computer-readable storage medium or any computer-readable medium that transmits, propagates, or transfers programs used by an instruction execution system, apparatus, or device to function according to the programs. The program code contained in the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber, RF, etc., or any suitable combination of the foregoing.

[0155] The flow diagrams and block diagrams in the drawings are schematic illustrations of possible architectures, functions, and operations of systems, methods, and computer program products in accordance with various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0156] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware. The described units can also be arranged in a processor, for example, a processor can be described as including a sparse perception unit, an updating unit, an interaction fusion unit and a trajectory planning unit. In some cases, the names of these units do not constitute a limitation on the units themselves.

[0157] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, when the one or more programs are executed by the device, the device is caused to acquire multi-modal sensing data at a current time according to a received trajectory planning request, perform a preset perception task based on the multi-modal sensing data to obtain instance features of a preset dimension, update an instance feature memory queue based on the instance features to obtain an updated instance feature memory queue, determine a vehicle instance feature at the current time, interact and fuse the vehicle instance feature and the updated instance feature memory queue through an attention mechanism to obtain aggregated spatio-temporal context features, determine a planned trajectory based on the aggregated spatio-temporal context features and a preset optimization condition, and output the planned trajectory to a terminal of the ego vehicle.

[0158] The computer program product of the present application includes a computer program, which, when executed by a processor, implements the trajectory planning method in the embodiments of the present application.

[0159] According to the technical solutions of the embodiments of the present application, the trajectory planning efficiency and accuracy can be improved.

[0160] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A trajectory planning method characterized by, The method comprises the following steps: According to the received trajectory planning request, obtain the multi-modal sensor data at the current time, and perform a preset perception task based on the multi-modal sensor data to obtain instance features of a preset dimension; Update the instance feature memory queue based on the instance features to obtain an updated instance feature memory queue; Determine the vehicle instance feature at the current time, and interact and fuse the vehicle instance feature with the updated instance feature memory queue through an attention mechanism to obtain an aggregated spatio-temporal context feature; Based on the aggregated spatio-temporal context feature and a preset optimization condition, determine a planning trajectory, and output the planning trajectory to a vehicle terminal.

2. The method of claim 1, wherein, The determination of the vehicle instance feature at the current time comprises: Obtain the planning trajectory of the vehicle at the previous time as the initial trajectory at the current time, and then determine the vehicle instance feature at the current time based on the initial trajectory.

3. The method of claim 1, wherein, The updating of the instance feature memory queue based on the instance features to obtain the updated instance feature memory queue comprises: Determine the current time coordinate system corresponding to the instance feature, and convert the instance features at the historical time in the instance feature memory queue into instance features in the current time coordinate system through pose projection; Determine the confidence of each instance feature in the current time coordinate system, and obtain the updated instance feature memory queue at the current time according to the confidence.

4. The method of claim 1, wherein, The updated instance feature memory queue comprises dynamic target instance features, static environment instance features, and online map instance features. And The interaction and fusion of the vehicle instance feature with the updated instance feature memory queue through the attention mechanism to obtain the aggregated spatio-temporal context feature comprises: Take the vehicle instance feature as the query data in the attention mechanism; Determine the dynamic target based on each dynamic target instance feature in the updated instance feature memory queue, predict the motion state data of the dynamic target at the future time, predict the collision risk between the vehicle and the dynamic target based on the query data and the motion state data to obtain collision risk information; Construct the static environment at the current time based on each static environment instance feature in the updated instance feature memory queue to obtain static environment data, and determine the driving feasibility of the vehicle in the physical space of the static environment based on the query data and the static environment data to obtain physical space driving feasibility information; Construct the online map at the current time based on each online map instance feature in the updated instance feature memory queue to obtain online map data, and determine the drivable route of the vehicle based on the query data and the online map data to obtain drivable route information; Obtain the aggregated spatio-temporal context feature based on the collision risk information, the physical space driving feasibility information, and the drivable route information.

5. The method of claim 1, wherein, The determination of the planning trajectory based on the aggregated spatio-temporal context feature and the preset optimization condition comprises: Input the aggregated spatio-temporal context feature into a trajectory generation model, so that the trajectory generation model generates multiple candidate trajectories based on the preset optimization condition; The multi-objective decision algorithm is used to calculate a comprehensive score of each candidate trajectory, and a planning trajectory is determined from the plurality of candidate trajectories according to the comprehensive score.

6. The method of claim 5, wherein, The multi-objective decision algorithm is used to calculate a comprehensive score of each candidate trajectory, and a planning trajectory is determined from the plurality of candidate trajectories according to the comprehensive score. A geometric-based collision detection algorithm or a probabilistic-based method is used to evaluate the possibility of collision when the ego vehicle travels along each candidate trajectory, so as to obtain a safety score of each candidate trajectory; The acceleration change value and the direction change value of the ego vehicle when traveling along each candidate trajectory are evaluated, and it is determined whether the acceleration change value and the direction change value corresponding to each candidate trajectory exceed a preset range, and a smoothness score of each candidate trajectory is determined according to the determination result; The energy consumption of the ego vehicle when traveling along each candidate trajectory is evaluated, so as to obtain an energy consumption score of each candidate trajectory; The safety score, the smoothness score and the energy consumption score are used to obtain a comprehensive score of each candidate trajectory.

7. A trajectory planning device characterized by comprising: The method comprises the following steps: a sparse perception unit configured to obtain multi-modal sensing data at a current time according to a received trajectory planning request, and perform a preset perception task based on the multi-modal sensing data to obtain instance features in a preset dimension; an updating unit configured to update an instance feature memory queue based on the instance features to obtain an updated instance feature memory queue; an interactive fusion unit configured to determine vehicle instance features at the current time, and interactively fuse the vehicle instance features with the updated instance feature memory queue through an attention mechanism to obtain aggregated spatio-temporal context features; a trajectory planning unit configured to determine a planning trajectory based on the aggregated spatio-temporal context features and a preset optimization condition, and output the planning trajectory to an ego vehicle terminal.

8. A trajectory planning electronic device, comprising: The method comprises the following steps: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-6.

9. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method according to any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1-6.

Images