End-to-end autonomous driving method and device based on sparse paradigm

Abstract

The application relates to the technical field of intelligent networked automobile automatic driving, in particular to an end-to-end automatic driving method and device based on a sparse paradigm, wherein the method comprises the following steps: extracting image perspective features; obtaining dynamic instance features, dynamic instance anchors, static instance features and static instance anchors based on the sparse paradigm; determining self-vehicle instance features of a self-vehicle and determining corresponding splicing instance features; determining scene information required when the self-vehicle and surrounding vehicles perform future trajectory prediction, so as to obtain surrounding vehicle multi-modal trajectories of the surrounding vehicles and surrounding vehicle trajectory scores corresponding to different surrounding vehicle multi-modal trajectories, self-vehicle multi-modal trajectories of the self-vehicle and self-vehicle trajectory scores corresponding to different self-vehicle multi-modal trajectories; determining a final trajectory of the self-vehicle and controlling the self-vehicle to drive. Thus, the problems that the calculation amount is large, the time delay is high, the perception range is limited and the model performance is limited by relying on bird's-eye view feature calculation are solved; in addition, the problems that the simple sequential connection of multiple tasks does not consider task correlation and limits information transmission are solved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology for intelligent connected vehicles, and in particular to an end-to-end autonomous driving method and device based on a sparse paradigm. Background Technology

[0002] As the application scenarios of autonomous driving in intelligent connected vehicles continue to expand, rule-based autonomous driving planning methods are gradually approaching their limits. Due to the complexity of the real world, traditional rule-based methods struggle to cover all real-world scenarios. Data-driven end-to-end autonomous driving, with its excellent scalability, is gradually becoming the forefront of research in this field.

[0003] In related technologies, end-to-end autonomous driving uses sensor data such as images and LiDAR point clouds as input, and directly outputs decision trajectories or control signals through neural networks; alternatively, a modular end-to-end approach can be used to use perception and prediction tasks as intermediate supervision signals to accelerate model convergence, while the output of intermediate results makes the end-to-end model more interpretable.

[0004] However, in related technologies, relying on bird's-eye view features not only requires additional computation and latency, but also requires a preset perception distance, which limits the perception range. Furthermore, high compression can easily lead to information loss, thus limiting model performance. In addition, the design of related tasks follows perception, prediction, and planning, with multiple tasks simply connected in sequence, without considering the correlation between tasks, which limits the information transfer between tasks and urgently needs improvement. Summary of the Invention

[0005] This application provides an end-to-end autonomous driving method and device based on a sparse paradigm to address the problems in related technologies, such as the need for additional computation and latency due to reliance on bird's-eye view features, the requirement for preset perception distance leading to limited perception range, and the tendency for high compression to cause information loss, thereby limiting model performance; in addition, the design of related tasks follows perception, prediction, and planning, with multiple tasks simply connected in sequence, without considering the correlation between tasks, thus limiting information transmission between tasks.

[0006] The first aspect of this application provides an end-to-end autonomous driving method based on a sparse paradigm, comprising the following steps: acquiring image data in an autonomous driving scenario and extracting at least one corresponding image perspective feature from the image data; based on a sparse paradigm, projecting a first anchor point of a dynamic instance and a second anchor point of a static instance in the autonomous driving scenario onto the at least one image perspective feature for feature sampling to obtain dynamic instance features and dynamic instance anchors of the dynamic instance, and static instance features and static instance anchors of the static instance; determining the vehicle instance features of the autonomous vehicle in the autonomous driving scenario based on the front view image features in the image data, and obtaining the vehicle instance features based on the dynamic instance features and the vehicle instance features. The system uses spliced ​​instance features in a driving scenario. Based on these spliced ​​instance features, dynamic instance anchors, static instance features, and static instance anchors, it determines the scene information needed for the autonomous vehicle and surrounding vehicles to predict future trajectories. Based on this scene information, it determines the surrounding vehicle's multimodal trajectory and the trajectory scores corresponding to different multimodal trajectories, as well as the autonomous vehicle's multimodal trajectory and the trajectory scores corresponding to different multimodal trajectories. Based on the surrounding vehicle's multimodal trajectory, the trajectory scores corresponding to different multimodal trajectories, the autonomous vehicle's multimodal trajectory, and the trajectory scores corresponding to different multimodal trajectories, it determines the final trajectory of the autonomous vehicle and controls its driving according to the final trajectory.

[0007] Optionally, in one embodiment of this application, the step of projecting the first anchor point of the dynamic instance and the second anchor point of the static instance in the autonomous driving scenario onto the at least one image perspective feature for feature sampling based on the sparse paradigm to obtain the dynamic instance features and dynamic instance anchor of the dynamic instance, as well as the static instance features and static instance anchor of the static instance, includes: obtaining the initial dynamic instance features and initial dynamic instance anchor of the dynamic instance and the initial static instance features and initial static instance anchor of the static instance; inputting the initial dynamic instance features and initial dynamic instance anchor of the dynamic instance and the initial static instance features and initial static instance anchor of the static instance into a pre-constructed decoding layer to obtain the dynamic instance features and dynamic instance anchor of the dynamic instance, as well as the static instance features and static instance anchor of the static instance, respectively.

[0008] Optionally, in one embodiment of this application, the step of inputting the initial dynamic instance features and initial dynamic instance anchors of the dynamic instance and the initial static instance features and initial static instance anchors of the static instance into a pre-constructed decoding layer to obtain the dynamic instance features and dynamic instance anchors of the dynamic instance, and the static instance features and static instance anchors of the static instance, includes: predicting the size scale corresponding to the dynamic instance anchor and the static instance anchor based on the initial dynamic instance features and the initial static instance features, respectively, to generate corresponding 3D key points; projecting the 3D key points onto image data through camera parameters and sampling them to obtain sampled features; predicting the weights of the 3D key points based on the initial dynamic instance features and the initial static instance features, respectively, to obtain the weight values ​​of the 3D key points; updating the corresponding initial dynamic instance features and initial static instance features through the weighted sum of the sampled features and the weight values, respectively, to obtain the dynamic instance features and the static instance features; and predicting the anchor offsets of the dynamic instance anchor and the static instance anchor based on the dynamic instance features and the static instance features, respectively, to obtain the dynamic instance anchor and the static instance anchor.

[0009] Optionally, in one embodiment of this application, determining the scene information required for predicting the future trajectory of the vehicle and surrounding vehicles based on the stitched instance features, the dynamic instance anchor, the static instance features, and the static instance anchor includes: performing instance-time interaction on the stitched instance features to obtain corresponding time-series instance features; performing instance-instance interaction on the time-series instance features to obtain corresponding instance features; performing instance-map interaction on the instance features to obtain corresponding map instance features; and using the map instance features to determine the scene information.

[0010] Optionally, in one embodiment of this application, determining the final trajectory of the vehicle based on the surrounding vehicle multimodal trajectory, the surrounding vehicle trajectory scores corresponding to different surrounding vehicle multimodal trajectories, the vehicle multimodal trajectory, and the vehicle trajectory scores corresponding to different vehicle multimodal trajectories includes: selecting a corresponding vehicle multimodal trajectory candidate set from the vehicle multimodal trajectories based on navigation commands; determining the probability of a collision between each candidate trajectory in the vehicle multimodal trajectory candidate set and the surrounding vehicle multimodal trajectory based on the vehicle multimodal trajectory candidate set, the surrounding vehicle multimodal trajectory, and the surrounding vehicle trajectory scores corresponding to different surrounding vehicle multimodal trajectories; re-determining the candidate trajectory score of each candidate trajectory based on the probability; and determining the final trajectory based on the candidate trajectory scores.

[0011] A second aspect of this application provides an end-to-end autonomous driving device based on a sparse paradigm, comprising: an acquisition module, configured to acquire image data in an autonomous driving scenario and extract at least one corresponding image perspective feature from the image data; a generation module, configured to, based on a sparse paradigm, project a first anchor point of a dynamic instance and a second anchor point of a static instance in the autonomous driving scenario onto the at least one image perspective feature for feature sampling, to obtain dynamic instance features and dynamic instance anchors of the dynamic instance, and static instance features and static instance anchors of the static instance; and a first determination module, configured to, based on the front view image features in the image data, determine the vehicle instance features of the autonomous driving scenario, and obtain the vehicle instance features based on the dynamic instance features and the vehicle instance features. The system comprises three modules: a first module for determining the vehicle's trajectory and a second module for determining the scene information required for future trajectory prediction of the vehicle and surrounding vehicles based on the spliced ​​instance features, the dynamic instance anchors, the static instance features, and the static instance anchors; a third module for determining the vehicle's trajectory based on the scene information, including the vehicle's multimodal trajectory and the trajectory scores corresponding to different multimodal trajectories, and the vehicle's multimodal trajectory and the trajectory scores corresponding to different multimodal trajectories; and a fourth module for determining the vehicle's final trajectory based on the multimodal trajectory, the trajectory scores corresponding to different multimodal trajectories, the vehicle's multimodal trajectory, and the trajectory scores corresponding to different multimodal trajectories, in order to control the vehicle's driving according to the final trajectory.

[0012] Optionally, in one embodiment of this application, the acquisition unit is used to acquire the initial dynamic instance features and initial dynamic instance anchor of the dynamic instance and the initial static instance features and initial static instance anchor of the static instance; the first generation unit is used to input the initial dynamic instance features and initial dynamic instance anchor of the dynamic instance and the initial static instance features and initial static instance anchor of the static instance into a pre-constructed decoding layer to obtain the dynamic instance features and dynamic instance anchor of the dynamic instance, and the static instance features and static instance anchor of the static instance, respectively.

[0013] Optionally, in one embodiment of this application, the generation unit includes: a first generation subunit, configured to predict the size of the dynamic instance anchor and the static instance anchor respectively based on the initial dynamic instance features and the initial static instance features, to generate corresponding 3D key points; a second generation subunit, configured to project the 3D key points onto image data respectively using camera parameters and sample them to obtain sampled features; a third generation subunit, configured to predict the weights of the 3D key points respectively using the initial dynamic instance features and the initial static instance features, to obtain the weight values ​​of the 3D key points; an update subunit, configured to update the corresponding initial dynamic instance features and initial static instance features respectively using the weighted sum of the sampled features and the weight values, to obtain the dynamic instance features and the static instance features; and a prediction subunit, configured to predict the anchor offset of the dynamic instance anchor and the static instance anchor respectively using the dynamic instance features and the static instance features, to obtain the dynamic instance anchor and the static instance anchor.

[0014] Optionally, in one embodiment of this application, the second determining module includes: a second generating unit, configured to perform instance and temporal interaction on the stitched instance features to obtain corresponding temporal instance features; a third generating unit, configured to perform instance and instance interaction on the temporal instance features to obtain corresponding instance features; a fourth generating unit, configured to perform instance and map interaction on the instance features to obtain corresponding map instance features; and a first determining unit, configured to determine the scene information using the map instance features.

[0015] Optionally, in one embodiment of this application, the third determining module includes: a fifth generating unit, configured to select a corresponding candidate set of autonomous vehicle multimodal trajectories from the autonomous vehicle multimodal trajectory based on navigation commands; a second determining unit, configured to determine the probability of a collision between each candidate trajectory in the candidate set of autonomous vehicle multimodal trajectories and the surrounding vehicle multimodal trajectory based on the candidate set of autonomous vehicle multimodal trajectories, the surrounding vehicle multimodal trajectories, and the surrounding vehicle trajectory scores corresponding to different surrounding vehicle multimodal trajectories; a third determining unit, configured to redetermine the candidate trajectory score of each candidate trajectory based on the probability; and a fourth determining unit, configured to determine the final trajectory based on the candidate trajectory scores.

[0016] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the end-to-end autonomous driving method based on the sparse paradigm as described in the above embodiments.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described sparse paradigm-based end-to-end autonomous driving method.

[0018] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, implements the above-described sparse paradigm-based end-to-end autonomous driving method.

[0019] This application embodiment can extract image perspective features from image data in autonomous driving scenarios, and then, based on a sparse paradigm, determine the dynamic instance features of dynamic instances and the static instance features of static instances. By concatenating the dynamic instance features and the vehicle's instance features, the scene information required for predicting the future trajectory of the vehicle and surrounding vehicles is determined. This leads to the determination of the surrounding vehicle's multimodal trajectory and the surrounding vehicle trajectory scores corresponding to different multimodal trajectories, as well as the vehicle's multimodal trajectory and the vehicle trajectory scores corresponding to different multimodal trajectories. Thus, the final trajectory of the vehicle is obtained, and the vehicle is controlled to drive according to the final trajectory. The symmetric perception using the sparse paradigm avoids explicit construction of bird's-eye view features, thereby avoiding information loss and improving model efficiency. It also unifies the design of dynamic and static tasks, and outputs a reasonable trajectory through multiple sub-modules such as instance initialization, spatiotemporal interaction, multimodal prediction, and hierarchical planning, ensuring driving safety. This solves the problems in related technologies, such as the need for additional computation and latency due to reliance on bird's-eye view features, the requirement for preset perception distance leading to limited perception range, and the tendency for high compression to cause information loss, thus limiting model performance. In addition, the design of related tasks follows perception, prediction, and planning, with multiple tasks simply connected in sequence, without considering the correlation between tasks, which limits the information transfer between tasks.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0022] Figure 1 This is a flowchart of an end-to-end autonomous driving method based on a sparse paradigm provided according to an embodiment of this application;

[0023] Figure 2 This is a block diagram illustrating the principle of a symmetric sparse sensing module according to an embodiment of this application.

[0024] Figure 3 This is a flowchart for determining the final trajectory of a vehicle according to one embodiment of this application;

[0025] Figure 4 This is a flowchart illustrating the working principle of an end-to-end autonomous driving method based on a sparse paradigm according to an embodiment of this application;

[0026] Figure 5 This is a block diagram of an end-to-end autonomous driving device based on a sparse paradigm according to an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0028] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0029] The following description, with reference to the accompanying drawings, describes an end-to-end autonomous driving method and apparatus based on a sparse paradigm according to embodiments of this application. Addressing the issues mentioned in the background art, which highlight that relying on bird's-eye view features not only requires additional computation and latency but also necessitates a preset perception distance, resulting in limited perception range, and that high compression easily leads to information loss, thus limiting model performance; furthermore, the design of related tasks follows perception, prediction, and planning, with multiple tasks simply connected sequentially without considering the correlation between tasks, thus limiting information transfer between tasks, this application provides an end-to-end autonomous driving method based on a sparse paradigm. In this method, image perspective features can be extracted from image data in an autonomous driving scenario, and then, based on a sparse paradigm, dynamic instance features of dynamic instances and static instance features of static instances are determined. This method concatenates state instance features and vehicle instance features to determine the scene information needed for predicting the future trajectories of both the vehicle and surrounding vehicles. This allows for the determination of the surrounding vehicle's multimodal trajectory and trajectory scores for different trajectories, as well as the vehicle's multimodal trajectory and trajectory scores for different trajectories. The final trajectory of the vehicle is then obtained, and its driving is controlled according to this trajectory. A sparse paradigm of symmetric perception is used to avoid explicitly constructing bird's-eye view features, thus preventing information loss and improving model efficiency. Furthermore, it unifies the design of dynamic and static tasks, outputting a reasonable trajectory through multiple sub-modules such as instance initialization, spatiotemporal interaction, multimodal prediction, and hierarchical planning, ensuring driving safety. This addresses issues in related technologies where relying on bird's-eye view features not only requires additional computation and latency but also necessitates pre-setting perception distances, limiting the perception range. High compression can also lead to information loss, thus limiting model performance. Additionally, related task designs often follow perception, prediction, and planning, with simple sequential connections between tasks without considering inter-task correlations, thus restricting information transfer between tasks.

[0030] Specifically, Figure 1 This is a flowchart of an end-to-end autonomous driving method based on a sparse paradigm provided in an embodiment of this application.

[0031] like Figure 1 As shown, this end-to-end autonomous driving method based on the sparse paradigm includes the following steps:

[0032] In step S101, image data in the autonomous driving scenario is acquired, and at least one corresponding image perspective feature is extracted from the image data.

[0033] It is understood that in the embodiments of this application, the image perspective feature is a key information extracted from the image based on the principle of perspective that can reflect the spatial relationship and three-dimensional structure of the object. Its core is to capture the differences in shape, size and position of the object caused by the change of distance by simulating the observation mode of the human eye or camera, thereby constructing a visual expression with a sense of depth and three-dimensionality.

[0034] Furthermore, in the embodiments of this application, image perspective features can be extracted from image data through the backbone network, or other methods can be used to extract image perspective features. The specific settings can be made by those skilled in the art according to the actual situation, and this application does not impose any specific limitations.

[0035] In some embodiments, the present application can acquire image data in an autonomous driving scenario and then extract the corresponding image perspective features through a backbone network.

[0036] For example, in this embodiment of the application, the camera image data in the autonomous driving scenario is set as I = {I} v |v=1,…,V}, where V is the number of multi-view surround-view cameras deployed on the autonomous vehicle, and the set contains elements I. v ∈R H×W×3 The image corresponding to the v-th camera contains RGB (Red, Green, Blue) three-channel pixel information with a height of H and a width of W.

[0037] Furthermore, in this embodiment, the image data is processed by the backbone network to generate image perspective features F = {F}. v |v=1,…,V}, where element F v Let be the perspective feature of the image corresponding to the v-th camera image.

[0038] In step S102, based on the sparse paradigm, the first anchor point of the dynamic instance and the second anchor point of the static instance in the autonomous driving scenario are projected onto at least one image perspective feature for feature sampling to obtain the dynamic instance feature and dynamic instance anchor of the dynamic instance, as well as the static instance feature and static instance anchor of the static instance.

[0039] It is understood that, in the embodiments of this application, dynamic instances can be understood as other traffic participants besides the vehicle itself in an autonomous driving scenario, and static instances can be understood as map elements in the environment. It should be noted that each instance consists of instance features and instance anchors. The first anchor point of a dynamic instance is represented by a 3D bounding box box = (x, y, z, l, w, h, yaw, vx, vy), and each point contains attributes such as position, size, heading angle, and velocity. The second anchor point of a static instance is represented by a point column map = {(x... i ,y i |i=1,…,M p The points are represented as follows: each point contains two-dimensional location information, and the points are connected in sequence to form map elements.

[0040] In some embodiments, this application uses a sparse paradigm of perception to project the first anchor point of a dynamic instance and the second anchor point of a static instance in an autonomous driving scenario onto image perspective features for feature sampling, so as to obtain the dynamic instance features of the dynamic instance and the static instance features of the static instance.

[0041] Optionally, in one embodiment of this application, based on the sparse paradigm, the first anchor point of a dynamic instance and the second anchor point of a static instance in an autonomous driving scenario are projected onto at least one image perspective feature for feature sampling to obtain the dynamic instance features and dynamic instance anchor of the dynamic instance, as well as the static instance features and static instance anchor of the static instance. This includes: obtaining the initial dynamic instance features and initial dynamic instance anchor of the dynamic instance and the initial static instance features and initial static instance anchor of the static instance; and inputting the initial dynamic instance features and initial dynamic instance anchor of the dynamic instance and the initial static instance features and initial static instance anchor of the static instance into a pre-built decoding layer to obtain the dynamic instance features and dynamic instance anchor of the dynamic instance, as well as the static instance features and static instance anchor of the static instance, respectively.

[0042] In some embodiments, the present application embodiments may utilize a pre-built decoding layer to obtain the dynamic instance features and dynamic instance anchors of a dynamic instance based on the initial dynamic instance features and initial dynamic instance anchors of the dynamic instance.

[0043] In some embodiments, embodiments of this application may utilize a pre-built decoding layer to obtain the static instance features and static instance anchors of a static instance based on the initial static instance features and initial static instance anchors of the static instance.

[0044] Optionally, in one embodiment of this application, the initial dynamic instance features and initial dynamic instance anchors of dynamic instances and the initial static instance features and initial static instance anchors of static instances are input into a pre-constructed decoding layer to obtain the dynamic instance features and dynamic instance anchors of dynamic instances, as well as the static instance features and static instance anchors of static instances. This includes: predicting the size scale corresponding to the dynamic instance anchors and static instance anchors based on the initial dynamic instance features and initial static instance features to generate corresponding 3D key points; projecting the 3D key points onto image data using camera parameters and sampling them to obtain sampled features; predicting the weights of the 3D key points using the initial dynamic instance features and initial static instance features to obtain the weight values ​​of the 3D key points; updating the corresponding initial dynamic instance features and initial static instance features by weighting the sampled features and weight values ​​to obtain dynamic instance features and static instance features; and predicting the anchor offsets of the dynamic instance anchors and static instance anchors using the dynamic instance features and static instance features to obtain the dynamic instance anchors and static instance anchors.

[0045] It is understood that, in order to obtain meaningful instance features and accurate instance locations from initially obtained meaningless instances, this application embodiment requires the interaction of instances and image features through multiple decoding layers.

[0046] Furthermore, embodiments of this application can be combined with Figure 2 As shown, taking a dynamic instance as an example, the process for a static instance is similar and will not be elaborated further. First, the dynamic instance is initialized with its initial features set to 0. The first anchor point is obtained by performing K-means clustering on the dataset. 3D keypoints are generated near the initial dynamic instance anchor obtained from the clustering. The expression for this keypoint can be, but is not limited to, as follows:

[0047] kps 3d = (x,y,z)+scale*(l,w,h),

[0048] Here, scale represents the size of each 3D keypoint, (x,y,z) represents the position of each 3D dynamic instance anchor, and (l,w,h) represents the size of each 3D dynamic instance anchor. Keypoints are generated near the 3D dynamic instance anchors.

[0049] Furthermore, in the embodiments of this application, the scale of each 3D keypoint can be determined by the initial dynamic instance feature F. ins It is obtained by predicting using an MLP (Multilayer Perceptron) and normalizing it with a sigmoid activation function. Its expression can be, but is not limited to, as follows:

[0050] scale = (scale l ,sclae w ,scale h =Sigmoid(MLP(F) ins )),

[0051] Simultaneously, for each 3D keypoint, instance features predict a weight and normalize it using the Softmax function: weight = Softmax(MLP(F ins After obtaining 3D keypoints, they are directly projected onto the camera plane using camera parameters for feature sampling, avoiding explicit construction of features from the bird's-eye view: kps 2d =K×kps 3d Where K is the camera matrix composed of camera intrinsics and extrinsic parameters. For 2D keypoints projected onto the image, features are sampled using bilinear interpolation: f = bilinear_sampling(F, kps) 2d For features obtained from sampling multiple key points, aggregate them according to their corresponding weights and update the instance features: F′ ins =F ins+MLP(∑ i weight i ×f i The updated dynamic instance features acquire richer image features, enabling the prediction of anchor offsets for a more accurate output: offset = MLP(F) ins ), box = box + offset.

[0052] The above process constitutes a decoding layer. This embodiment can consist of six decoding layers. The first decoding layer is a single-frame decoding layer. The dynamic instance features and the first anchor point are all derived from zero initialization and clustering. To introduce temporal information, the second decoding layer selects a portion of the instances output from the first decoding layer and a portion of the instances output from the previous frame to form the input. After multiple feature samplings and iterative updates of the dynamic instance anchor, dynamic instance features rich in semantic and geometric information and accurate dynamic instance anchors are finally obtained. Furthermore, the dynamic instance features predict a classification score. cls =Sigmoid(MLP(F ins )), representing the probability that the current dynamic instance belongs to each category, and the predicted category is determined by the category with the highest score: cls = argmax(score) cls ).

[0053] In addition, for dynamic instances, this application embodiment can also track information to determine the correspondence between historical instances and current output instances. For instances from the single-frame decoding layer, a new tracking number is assigned, and for instances from historical moments, the tracking number remains consistent with the historical instance.

[0054] In step S103, based on the front view image features in the image data, the vehicle instance features of the autonomous driving scenario are determined, and the stitched instance features in the autonomous driving scenario are obtained according to the dynamic instance features and the vehicle instance features.

[0055] In some embodiments, this application can determine the vehicle instance features in an autonomous driving scenario based on the front view image features in the image data, and then obtain the stitched instance features in the autonomous driving scenario based on the dynamic instance features and the vehicle instance features. Specifically, this mainly includes the following:

[0056] Vehicle instance initialization: Both the vehicle trajectory prediction and vehicle planning tasks involve predicting future trajectories, so rich semantic information can be used for scene understanding, and rich geometric information can be used for motion prediction. For vehicle trajectory prediction, the instance features output at the current time step already contain rich information, but for the vehicle instance features F... egoRelated technologies often employ random initialization, while this application proposes a vehicle instance initialization method, endowing vehicle planning with richer initial features. For semantic features, this application, based on the importance and relevance of the front view image in multi-view images to vehicle driving, and considering that the front view image features contain coarse scene information such as weather, time, and lighting, uses global average pooling of the front view image features as the initialization of vehicle features: F ego =AvgPool(F cam_front For geometric information, a 3D bounding box is also provided for the vehicle, where position and size are inherent information of the vehicle, the heading angle is fixed at 0 in the vehicle coordinate system, and the velocity is initialized using the velocity from the previous frame. This vehicle instance initialization provides semantic and geometric information to the vehicle, laying the foundation for subsequent planning.

[0057] Furthermore, in this embodiment, dynamic instance features and vehicle instance features are concatenated to obtain concatenated instance features F in the autonomous driving scenario. a =Concat([F ins1 ,F ins2 ,…,F ego This allows for parallel processing of the chariot and the chariot.

[0058] In step S104, based on the spliced ​​instance features, dynamic instance anchors, static instance features, and static instance anchors, the scene information required for the vehicle and surrounding vehicles to predict future trajectories is determined. Based on the scene information, the surrounding vehicle's multimodal trajectory and the surrounding vehicle trajectory score corresponding to different surrounding vehicle multimodal trajectories are determined, as well as the vehicle's multimodal trajectory and the vehicle trajectory score corresponding to different vehicle multimodal trajectories.

[0059] In some embodiments, this application can determine the scene information required for predicting the future trajectories of the vehicle and the surrounding vehicle based on the splicing instance features and static instance features, and then determine the surrounding vehicle's multimodal trajectory and corresponding surrounding vehicle trajectory score, and the vehicle's multimodal trajectory and corresponding vehicle trajectory score.

[0060] Furthermore, in this embodiment, both vehicle prediction and autonomous vehicle planning inherently involve uncertainty. Most related technologies consider the uncertainty of vehicle prediction, but only output a fixed trajectory for autonomous vehicle planning. This single-modal output cannot capture the uncertainty of the autonomous vehicle planning action space, affecting its performance. Therefore, this embodiment outputs multimodal trajectories and corresponding trajectory scores for both vehicle prediction and autonomous vehicle planning. For vehicles in the same location (i.e., other traffic participants), the output is N. mode Traj multimodal trajectory of the vehicle ins and the corresponding weekly trajectory score insFor the vehicle, considering multiple navigation commands (cmd), for each command, N_mode multimodal trajectories for the vehicle are output, for a total of N cmd ×N_mode multimodal trajectories traj_ego and the corresponding vehicle trajectory score score_ego.

[0061] Optionally, in one embodiment of this application, the scene information required for predicting the future trajectories of the vehicle and surrounding vehicles is determined based on the stitched instance features, dynamic instance anchors, static instance features, and static instance anchors. This includes: performing instance-time interaction on the stitched instance features to obtain corresponding time-series instance features; performing instance-instance interaction on the time-series instance features to obtain corresponding instance features; performing instance-map interaction on the instance features to obtain corresponding map instance features; and using the map instance features to determine scene information.

[0062] It is understandable that in this embodiment, both the surrounding vehicle prediction and the autonomous vehicle planning require a two-way, high-level interactive game, meaning that the autonomous vehicle's actions will affect the surrounding vehicle's reaction in the next moment, and conversely, will affect the autonomous vehicle's actions in the next moment. Related technologies that design prediction and planning tasks sequentially ignore this point, first predicting the surrounding vehicle's trajectory without autonomous vehicle interaction, and then predicting the autonomous vehicle's trajectory based on the surrounding vehicle's trajectory. To consider the high-level interaction among all traffic participants, this embodiment can use three attention interaction mechanisms for parallel spatiotemporal interaction, thereby modeling the high-level interactive game between the surrounding vehicle and the autonomous vehicle.

[0063] In some embodiments, the present application embodiments can perform instance and time-series interaction on the spliced ​​instance features to obtain the corresponding time-series instance features, and perform instance and instance interaction on the time-series instance features to obtain the corresponding instance features, and then perform instance and map interaction on the instance features to obtain the corresponding map instance features, thereby determining scene information.

[0064] For example, in the embodiments of this application, the splicing instance feature F a Three attention interaction mechanisms can be used to extract features needed for future trajectory prediction, which can respectively achieve instance-temporal interaction to obtain temporal instance features F. ah =MHA(F a ,F his ,F his ), instance-to-instance interaction, to obtain instance features F aa =MHA(F ah ,F ah ,F ah ), instance-map interaction, to obtain map instance features F am =MHA(F aa ,F m ,F mMHA is a multi-head attention mechanism, with three inputs: query, key, and value. his For dynamic instance output of historical frames, F m For the static map instance output in the current frame, through spatiotemporal interaction, F a It contains the necessary scene information for predicting future trajectories.

[0065] In step S105, the final trajectory of the vehicle is determined based on the multimodal trajectory of the surrounding vehicle, the trajectory score of the surrounding vehicle corresponding to different multimodal trajectories, the multimodal trajectory of the vehicle itself, and the trajectory score of the vehicle itself corresponding to different multimodal trajectories, so as to control the vehicle to drive according to the final trajectory.

[0066] In some embodiments, this application can simultaneously output the future trajectories of surrounding traffic participants and the vehicle itself, that is, simultaneously complete the tasks of surrounding vehicle trajectory prediction and vehicle trajectory planning, thereby obtaining the final trajectory of the vehicle and controlling its movement. Unlike related technologies that simply design the prediction and planning tasks sequentially, this application deeply considers the correlation between the prediction and planning tasks, summarizing three similarities: the richness of semantic and geometric information in surrounding vehicle trajectory prediction and vehicle trajectory planning, high-level bidirectional interactive game, and inherent trajectory uncertainty. Correspondingly, it designs vehicle instance initialization in S103, parallel spatiotemporal interaction in S104, and surrounding vehicle and vehicle multimodal trajectory prediction. For example, the process of determining the final trajectory of the vehicle in this application is as follows: Figure 3 As shown, the main content is as follows:

[0067] Step S301: Spatiotemporal interaction.

[0068] In this application embodiment, spatiotemporal interaction can be performed based on splicing instance features.

[0069] Furthermore, in the embodiments of this application, spatiotemporal interaction may include, but is not limited to, instance-time sequence interaction, i.e. instance-history interaction; instance-instance interaction, i.e. instance-instance interaction; instance-map interaction, i.e. instance-map interaction. This application does not impose specific limitations.

[0070] Step S302: Multimodal prediction.

[0071] In this application embodiment, multimodal trajectory prediction of the surrounding vehicle and multimodal trajectory prediction of the self vehicle can be performed.

[0072] Step S303: Hierarchical planning.

[0073] In this embodiment, the corresponding candidate set of vehicle multimodal trajectories can be selected according to the current navigation command; then the candidate trajectories are re-scored according to the degree of danger; and finally the final trajectory is selected according to the highest score.

[0074] Optionally, in one embodiment of this application, determining the final trajectory of a vehicle based on the surrounding vehicle multimodal trajectory, the surrounding vehicle trajectory scores corresponding to different surrounding vehicle multimodal trajectories, the vehicle's own multimodal trajectory, and the vehicle's own trajectory scores corresponding to different vehicle's own multimodal trajectories includes: selecting a corresponding set of vehicle multimodal trajectory candidates from the vehicle's own multimodal trajectories based on navigation commands; determining the probability of a collision between each candidate trajectory in the vehicle's own multimodal trajectory candidate set and the surrounding vehicle multimodal trajectory based on the vehicle's own multimodal trajectory candidate set, the surrounding vehicle multimodal trajectory, and the surrounding vehicle trajectory scores corresponding to different surrounding vehicle multimodal trajectories; re-determining the candidate trajectory score of each candidate trajectory based on the probability; and determining the final trajectory based on the candidate trajectory scores.

[0075] In some embodiments, this application proposes a hierarchical planning method. First, a corresponding candidate set of vehicle multimodal trajectories, traj_ego_cmd, is selected based on the current navigation command cmd. Then, the candidate trajectories are re-scored according to their degree of danger. For example, if a candidate trajectory traj_i collides with a surrounding vehicle multimodal trajectory, the score scoer_i of the corresponding candidate trajectory is set to 0. Finally, the final trajectory is selected based on the highest score. Through multimodal modeling and hierarchical planning, the uncertainty of vehicle planning is captured, while driving safety is ensured.

[0076] The working principle of the end-to-end autonomous driving method based on the sparse paradigm proposed in this application will be explained below with reference to a specific embodiment.

[0077] in, Figure 4 This is a flowchart illustrating the working principle of an end-to-end autonomous driving method based on a sparse paradigm according to an embodiment of this application.

[0078] Step S401: Extract perspective image features.

[0079] In this embodiment, image data from an autonomous driving scenario can be used as data, and perspective image features can be extracted through a backbone network.

[0080] Step S402: Symmetric sparse sensing module.

[0081] The schematic diagram of the symmetric sparse sensing module in this embodiment is as follows: Figure 2 As shown.

[0082] Furthermore, embodiments of this application can project dynamic and static instance anchor points onto image plane features based on a symmetric sparse perception module, and output the category, 3D bounding box, tracking number of dynamic traffic participants and the category and location of static maps.

[0083] Step S403: Parallel predictive planning module.

[0084] The schematic diagram of the parallel prediction planning module in this embodiment is as follows: Figure 3 As shown.

[0085] Furthermore, embodiments of this application can initialize prediction and planning instances based on a parallel prediction and planning module, perform spatiotemporal feature interaction through three attention interaction mechanisms, predict multimodal trajectories, and output the final trajectory through hierarchical planning.

[0086] The end-to-end autonomous driving method based on the sparse paradigm proposed in this application can extract image perspective features from image data in autonomous driving scenarios. Then, based on the sparse paradigm, it determines the dynamic instance features of dynamic instances and the static instance features of static instances. By concatenating the dynamic instance features and the vehicle's instance features, it determines the scene information needed for the vehicle and surrounding vehicles to predict future trajectories. This allows it to determine the surrounding vehicle's multimodal trajectory and the surrounding vehicle trajectory scores corresponding to different multimodal trajectories, as well as the vehicle's multimodal trajectory and the vehicle trajectory scores corresponding to different multimodal trajectories, thus obtaining the vehicle's final trajectory. The vehicle is then controlled to drive according to the final trajectory. By using symmetric perception based on the sparse paradigm, it avoids explicitly constructing bird's-eye view features, thereby avoiding information loss and improving model efficiency. It also unifies the design of dynamic and static tasks and outputs a reasonable trajectory through multiple sub-modules such as instance initialization, spatiotemporal interaction, multimodal prediction, and hierarchical planning, ensuring driving safety. This solves the problems in related technologies, such as the need for additional computation and latency due to reliance on bird's-eye view features, the requirement for preset perception distance leading to limited perception range, and the tendency for high compression to cause information loss, thus limiting model performance. In addition, the design of related tasks follows perception, prediction, and planning, with multiple tasks simply connected in sequence, without considering the correlation between tasks, which limits the information transfer between tasks.

[0087] Next, with reference to the accompanying drawings, an end-to-end autonomous driving device based on the sparse paradigm proposed according to an embodiment of this application is described.

[0088] Figure 5 This is a block diagram of an end-to-end autonomous driving device based on a sparse paradigm according to an embodiment of this application.

[0089] like Figure 5 As shown, the end-to-end autonomous driving device 10 based on the sparse paradigm includes: an acquisition module 100, a generation module 200, a first determination module 300, a second determination module 400, and a third determination module 500.

[0090] The acquisition module 100 is used to acquire image data in an autonomous driving scenario and extract at least one corresponding image perspective feature from the image data.

[0091] The generation module 200 is used to project the first anchor point of a dynamic instance and the second anchor point of a static instance in an autonomous driving scenario onto at least one image perspective feature for feature sampling based on a sparse paradigm, so as to obtain the dynamic instance features and dynamic instance anchors of the dynamic instance, as well as the static instance features and static instance anchors of the static instance.

[0092] The first determining module 300 is used to determine the vehicle instance features of the vehicle in the autonomous driving scenario based on the front view image features in the image data, and to obtain the stitched instance features in the autonomous driving scenario based on the dynamic instance features and the vehicle instance features.

[0093] The second determining module 400 is used to determine the scene information required for the vehicle and the surrounding vehicle to predict future trajectories based on spliced ​​instance features, dynamic instance anchors, static instance features, and static instance anchors. Based on the scene information, it determines the surrounding vehicle's multimodal trajectory and the surrounding vehicle trajectory score corresponding to different surrounding vehicle multimodal trajectories, as well as the vehicle's multimodal trajectory and the vehicle trajectory score corresponding to different vehicle multimodal trajectories.

[0094] The third determining module 500 is used to determine the final trajectory of the vehicle based on the multimodal trajectory of the surrounding vehicle, the trajectory score of the surrounding vehicle corresponding to different multimodal trajectories, the multimodal trajectory of the vehicle itself, and the trajectory score of the vehicle itself corresponding to different multimodal trajectories, so as to control the vehicle to drive according to the final trajectory.

[0095] Optionally, in one embodiment of this application, the generation module 200 includes: an acquisition unit and a first generation unit.

[0096] The acquisition unit is used to acquire the initial dynamic instance characteristics and initial dynamic instance anchor of the dynamic instance, and the initial static instance characteristics and initial static instance anchor of the static instance.

[0097] The first generation unit is used to input the initial dynamic instance features and initial dynamic instance anchors of dynamic instances and the initial static instance features and initial static instance anchors of static instances into a pre-built decoding layer to obtain the dynamic instance features and dynamic instance anchors of dynamic instances, as well as the static instance features and static instance anchors of static instances, respectively.

[0098] Optionally, in one embodiment of this application, the generation unit includes: a first generation subunit, a second generation subunit, a third generation subunit, an update subunit, and a prediction subunit.

[0099] The first generation subunit is used to predict the size of the dynamic instance anchor and the static instance anchor based on the initial dynamic instance features and the initial static instance features, respectively, so as to generate the corresponding 3D key points.

[0100] The second generation subunit is used to project 3D key points onto image data using camera parameters and sample them to obtain sampled features.

[0101] The third generation subunit is used to predict the weights of 3D keypoints using initial dynamic instance features and initial static instance features respectively, so as to obtain the weight values ​​of 3D keypoints.

[0102] The update sub-unit is used to update the corresponding initial dynamic instance features and initial static instance features respectively by weighting the sampled features and weight values, so as to obtain dynamic instance features and static instance features.

[0103] The prediction subunit is used to predict the anchor offsets of dynamic instance anchors and static instance anchors respectively using dynamic instance features and static instance features, so as to obtain dynamic instance anchors and static instance anchors.

[0104] Optionally, in one embodiment of this application, the second determining module 400 includes: a second generating unit, a third generating unit, a fourth generating unit, and a first determining unit.

[0105] The second generation unit is used to perform instance and temporal interaction on the spliced ​​instance features to obtain the corresponding temporal instance features.

[0106] The third generation unit is used to perform instance-to-instance interaction on the time-series instance features to obtain the corresponding instance features.

[0107] The fourth generation unit is used to perform instance and map interaction on instance features to obtain the corresponding map instance features.

[0108] The first determining unit is used to determine scene information using map instance features.

[0109] Optionally, in one embodiment of this application, the third determining module 500 includes: a fifth generating unit, a second determining unit, a third determining unit, and a fourth determining unit.

[0110] The fifth generation unit is used to select the corresponding candidate set of vehicle multimodal trajectories from the vehicle multimodal trajectories based on navigation commands.

[0111] The second determining unit is used to determine the probability of a collision between each candidate trajectory in the candidate set of the self-vehicle multimodal trajectory and the surrounding multimodal trajectory, based on the candidate set of the self-vehicle multimodal trajectory and the surrounding multimodal trajectory, as well as the surrounding trajectory scores corresponding to different surrounding multimodal trajectories.

[0112] The third determining unit is used to redetermine the candidate trajectory score for each candidate trajectory based on probability.

[0113] The fourth determining unit is used to determine the final trajectory based on the candidate trajectory scores.

[0114] It should be noted that the foregoing explanation of the end-to-end autonomous driving method embodiment based on the sparse paradigm also applies to the end-to-end autonomous driving device based on the sparse paradigm in this embodiment, and will not be repeated here.

[0115] The end-to-end autonomous driving device based on the sparse paradigm proposed in this application can extract image perspective features from image data in an autonomous driving scenario. Then, based on the sparse paradigm, it determines the dynamic instance features of dynamic instances and the static instance features of static instances. By concatenating the dynamic instance features and the vehicle's instance features, it determines the scene information needed for future trajectory prediction of the vehicle and surrounding vehicles. This allows it to determine the surrounding vehicle's multimodal trajectory and the trajectory scores corresponding to different surrounding vehicle multimodal trajectories, as well as the vehicle's multimodal trajectory and the trajectory scores corresponding to different vehicle multimodal trajectories, thus obtaining the vehicle's final trajectory. The device then controls the vehicle's driving according to this final trajectory. Using symmetric perception based on the sparse paradigm avoids explicitly constructing bird's-eye view features, thereby avoiding information loss and improving model efficiency. It also unifies the design of dynamic and static tasks, outputting a reasonable trajectory through multiple sub-modules such as instance initialization, spatiotemporal interaction, multimodal prediction, and hierarchical planning, ensuring driving safety. This solves the problems in related technologies, such as the need for additional computation and latency due to reliance on bird's-eye view features, the requirement for preset perception distance leading to limited perception range, and the tendency for high compression to cause information loss, thus limiting model performance. In addition, the design of related tasks follows perception, prediction, and planning, with multiple tasks simply connected in sequence, without considering the correlation between tasks, which limits the information transfer between tasks.

[0116] Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. The electronic device may include:

[0117] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0118] When the processor 602 executes the program, it implements the end-to-end autonomous driving method based on the sparse paradigm provided in the above embodiments.

[0119] Furthermore, electronic devices also include:

[0120] Communication interface 603 is used for communication between memory 601 and processor 602.

[0121] The memory 601 is used to store computer programs that can run on the processor 602.

[0122] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0123] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0124] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0125] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0126] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described end-to-end autonomous driving method based on the sparse paradigm.

[0127] This application also provides a computer program product, including a computer program that, when executed, implements the above-described end-to-end autonomous driving method based on the sparse paradigm.

[0128] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0129] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0130] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0131] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting or otherwise processing as necessary, and then stored in computer memory.

[0132] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0133] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0135] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An end-to-end autonomous driving method based on the sparse paradigm, characterized in that, Includes the following steps: Acquire image data from an autonomous driving scenario, and extract at least one corresponding image perspective feature from the image data; Based on the sparse paradigm, the first anchor point of the dynamic instance and the second anchor point of the static instance in the autonomous driving scenario are projected onto the at least one image perspective feature for feature sampling to obtain the dynamic instance feature and dynamic instance anchor of the dynamic instance, as well as the static instance feature and static instance anchor of the static instance; wherein, it includes: obtaining the initial dynamic instance feature and initial dynamic instance anchor of the dynamic instance and the initial static instance feature and initial static instance anchor of the static instance. The initial dynamic instance features and initial dynamic instance anchors of the dynamic instance and the initial static instance features and initial static instance anchors of the static instance are input into a pre-built decoding layer to obtain the dynamic instance features and dynamic instance anchors of the dynamic instance, as well as the static instance features and static instance anchors of the static instance; wherein, based on the initial dynamic instance features and the initial static instance features, the size scale corresponding to the dynamic instance anchor and the static instance anchor is predicted respectively to generate corresponding 3D key points; The 3D key points are projected onto the image data using camera parameters, and then sampled to obtain sampled features. The weights of the 3D keypoints are predicted using the initial dynamic instance features and the initial static instance features, respectively, to obtain the weight values ​​of the 3D keypoints. The initial dynamic instance features and the initial static instance features are updated by weighting the sampling features and the weight values ​​respectively, so as to obtain the dynamic instance features and the static instance features; The anchor offsets of the dynamic instance anchor and the static instance anchor are predicted by the dynamic instance features and the static instance features, respectively, to obtain the dynamic instance anchor and the static instance anchor. Based on the front view image features in the image data, the vehicle instance features of the vehicle in the autonomous driving scenario are determined, and the stitched instance features in the autonomous driving scenario are obtained according to the dynamic instance features and the vehicle instance features. Based on the spliced ​​instance features, the dynamic instance anchors, the static instance features, and the static instance anchors, the scene information required for the autonomous vehicle and the surrounding vehicle to perform future trajectory prediction is determined. Based on the scene information, the surrounding vehicle's multimodal trajectory and the surrounding vehicle trajectory score corresponding to different multimodal trajectories are determined, as well as the autonomous vehicle's multimodal trajectory and the autonomous vehicle trajectory score corresponding to different multimodal trajectories. Based on the vehicle's multimodal trajectory, the trajectory scores corresponding to different vehicle multimodal trajectories, the vehicle's multimodal trajectory, and the trajectory scores corresponding to different vehicle multimodal trajectories, the final trajectory of the vehicle is determined, and the vehicle's movement is controlled according to the final trajectory; wherein, The front view image features are used for global average pooling as the initialization of the vehicle features.

2. The method according to claim 1, characterized in that, The step of determining the scene information needed for the vehicle and surrounding vehicles to predict future trajectories based on the spliced ​​instance features, the dynamic instance anchors, the static instance features, and the static instance anchors includes: The spliced ​​instance features are subjected to instance and temporal interaction to obtain the corresponding temporal instance features; The time-series instance features are processed through instance-to-instance interactions to obtain the corresponding instance features; The instance features are interacted with the map to obtain the corresponding map instance features; The scene information is determined using the map instance features.

3. The method according to claim 1, characterized in that, The step of determining the final trajectory of the vehicle based on the surrounding vehicle multimodal trajectory, the surrounding vehicle trajectory scores corresponding to different surrounding vehicle multimodal trajectories, the vehicle's own multimodal trajectory, and the vehicle's own trajectory scores corresponding to different vehicle's multimodal trajectories includes: Based on navigation commands, a corresponding candidate set of vehicle multimodal trajectories is selected from the vehicle multimodal trajectories; Based on the candidate set of the autonomous vehicle multimodal trajectory and the multimodal trajectory of the surrounding vehicle, as well as the surrounding vehicle trajectory scores corresponding to different multimodal trajectories of the surrounding vehicle, the probability of each candidate trajectory in the candidate set of the autonomous vehicle multimodal trajectory colliding with the multimodal trajectory of the surrounding vehicle is determined; Based on the probability, the candidate trajectory score for each candidate trajectory is re-determined; The final trajectory is determined based on the candidate trajectory scores.

4. An end-to-end autonomous driving device based on a sparse paradigm, characterized in that, include: An acquisition module is used to acquire image data in an autonomous driving scenario and extract at least one corresponding image perspective feature from the image data. A generation module is used to project the first anchor point of a dynamic instance and the second anchor point of a static instance in the autonomous driving scenario onto the at least one image perspective feature for feature sampling based on a sparse paradigm, so as to obtain the dynamic instance feature and dynamic instance anchor of the dynamic instance, and the static instance feature and static instance anchor of the static instance. The first determining module is used to determine the vehicle instance features of the vehicle in the autonomous driving scenario based on the front view image features in the image data, and to obtain the stitched instance features in the autonomous driving scenario based on the dynamic instance features and the vehicle instance features. The second determining module is used to determine the scene information required for the vehicle and the surrounding vehicle to perform future trajectory prediction based on the spliced ​​instance features, the dynamic instance anchor, the static instance features, and the static instance anchor, so as to determine the surrounding vehicle's multimodal trajectory and the surrounding vehicle trajectory score corresponding to different multimodal trajectories, the vehicle's multimodal trajectory and the vehicle trajectory score corresponding to different multimodal trajectories based on the scene information; The third determining module is used to determine the final trajectory of the vehicle based on the surrounding vehicle multimodal trajectory, the surrounding vehicle trajectory score corresponding to different surrounding vehicle multimodal trajectories, the vehicle multimodal trajectory, and the vehicle trajectory score corresponding to different vehicle multimodal trajectories, so as to control the vehicle to drive according to the final trajectory; The generation module includes: The acquisition unit is used to acquire the initial dynamic instance features and initial dynamic instance anchor of the dynamic instance and the initial static instance features and initial static instance anchor of the static instance. The first generation unit is used to input the initial dynamic instance features and initial dynamic instance anchors of the dynamic instance and the initial static instance features and initial static instance anchors of the static instance into a pre-built decoding layer to obtain the dynamic instance features and dynamic instance anchors of the dynamic instance, and the static instance features and static instance anchors of the static instance, respectively. The generation module includes: The first generation subunit is used to predict the size of the dynamic instance anchor and the static instance anchor based on the initial dynamic instance features and the initial static instance features, respectively, so as to generate corresponding 3D key points; The second generation subunit is used to project the 3D key points onto the image data according to the camera parameters, and to sample them to obtain sampled features; The third generation subunit is used to predict the weights of the 3D key points using the initial dynamic instance features and the initial static instance features, respectively, so as to obtain the weight values ​​of the 3D key points. An update subunit is used to update the corresponding initial dynamic instance features and initial static instance features respectively by weighting the sampled features and the weight values, so as to obtain the dynamic instance features and the static instance features; A prediction subunit is configured to predict the anchor offsets of the dynamic instance anchor and the static instance anchor using the dynamic instance features and the static instance features, respectively, to obtain the dynamic instance anchor and the static instance anchor; wherein, The front view image features are used for global average pooling as the initialization of the vehicle features.

5. The apparatus according to claim 4, characterized in that, The second determining module includes: The second generation unit is used to perform instance and temporal interaction on the spliced ​​instance features to obtain the corresponding temporal instance features; The third generation unit is used to perform instance-to-instance interaction on the time-series instance features to obtain the corresponding instance features; The fourth generation unit is used to perform instance and map interaction on the instance features to obtain the corresponding map instance features; The first determining unit is used to determine the scene information using the map instance features.

6. The apparatus according to claim 4, characterized in that, The third determining module includes: The fifth generation unit is used to select a corresponding candidate set of vehicle multimodal trajectories from the vehicle multimodal trajectories based on navigation commands; The second determining unit is used to determine the probability of a collision between each candidate trajectory in the candidate set of the autonomous vehicle multimodal trajectory and the surrounding vehicle multimodal trajectory, based on the candidate set of autonomous vehicle multimodal trajectories, the surrounding vehicle multimodal trajectories, and the surrounding vehicle trajectory scores corresponding to different surrounding vehicle multimodal trajectories; The third determining unit is used to redetermine the candidate trajectory score of each candidate trajectory based on the probability. The fourth determining unit is used to determine the final trajectory based on the candidate trajectory scores.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the end-to-end autonomous driving method based on the sparse paradigm as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the sparse paradigm-based end-to-end autonomous driving method as described in any one of claims 1-3.

9. A computer program product, characterized in that, Includes a computer program, which, when executed, is used to implement the sparse paradigm-based end-to-end autonomous driving method as described in any one of claims 1-3.

Images