End-to-end autonomous driving method, device, and storage medium

Abstract

Provided in the present application are an end-to-end autonomous driving method, a device, and a storage medium. The method comprises: acquiring sensor data of an ego vehicle at the current moment; processing the sensor data at the current moment, so as to obtain a sparse sensor feature vector; performing interaction processing on a reference detection sparse query vector and the sparse sensor feature vector, so as to obtain a detection sparse query vector; performing interaction processing on a reference map sparse query vector and the sparse sensor feature vector, so as to obtain a map sparse query vector; performing obstacle detection on the basis of the detection sparse query vector, so as to obtain an obstacle detection result, and performing map construction on the basis of the map sparse query vector, so as to obtain an online map; and performing autonomous driving on the ego vehicle on the basis of the obstacle detection result and the online map. Therefore, all information is represented as sparse query vectors, and on the basis of the sparse query vectors, sub-tasks of autonomous driving are cascaded, so as to realize high-efficiency and high-accuracy end-to-end autonomous driving.

Description

An end-to-end autonomous driving method, device and storage medium

[0001] Cross Reference to Related Applications

[0002] This application claims priority to the Chinese patent application No. 202410417712.7, filed on April 8, 2024, and entitled "An end-to-end autonomous driving method, device and storage medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of image processing, and in particular, to an end-to-end autonomous driving method, device and storage medium. BACKGROUND

[0004] An autonomous driving system integrates multiple tasks, such as detection, tracking, online mapping, motion prediction and ego planning / control. The mainstream autonomous driving system decouples each task into a separate model and optimizes independently. Handset post-processing steps are added between each model to handle redundant information or difficult cases, which makes the entire process very complex. Moreover, the original sensor data cannot be used for downstream tasks, and errors are gradually accumulated in the system information transmission, which leads to potential safety problems.

[0005] To solve the above problems, an end-to-end autonomous driving system takes the original sensor data as input and returns the planning result in a more concise way. In the existing end-to-end autonomous driving method, the entire driving scene is represented by dense overhead map features, and multi-sensor data and time information are taken as the original input of the end-to-end autonomous driving system and play a role in each sub-task.

[0006] Although such a method can obtain a planning result concisely and optimize end-to-end, it solves the problems of accumulated errors and inconsistent optimization goals, but the performance on each sub-task of autonomous driving is far behind the corresponding single-task method. Moreover, the fusion of time information and sensor data is achieved through dense overhead map features, which significantly increases the computational cost and memory occupancy, making it difficult to implement model deployment. SUMMARY

[0007] In view of the above problems, the embodiments of the present application provide an end-to-end autonomous driving method, device and storage medium to overcome the above problems or at least partially solve the above problems.

[0008] In a first aspect, the embodiments of the present application disclose an end-to-end autonomous driving method, comprising:

[0009] obtaining sensor data of a current time of a ego vehicle;

[0010] processing the current moment sensor data to obtain a sparse sensor feature vector;

[0011] interacting the reference detection sparse query vector with the sparse sensor feature vector to obtain a detection sparse query vector, the reference detection sparse query vector at least comprising a historical detection sparse query vector determined based on historical sensor data of the ego vehicle;

[0012] interacting the reference map sparse query vector with the sparse sensor feature vector to obtain a map sparse query vector, the reference map sparse query vector at least comprising a historical map sparse query vector determined based on historical sensor data of the ego vehicle;

[0013] performing obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and performing map construction according to the map sparse query vector to obtain an online map;

[0014] performing automatic driving of the ego vehicle based on the obstacle detection result and the online map.

[0015] Optionally, the method further comprises:

[0016] determining an obstacle reference trajectory prediction sparse query vector according to obstacle historical position information;

[0017] determining an ego vehicle reference trajectory prediction sparse query vector according to ego vehicle historical position information;

[0018] interacting the obstacle reference trajectory prediction sparse query vector with the detection sparse query vector and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector;

[0019] interacting the ego vehicle reference trajectory prediction sparse query vector with the detection sparse query vector and the map sparse query vector to obtain an ego vehicle trajectory prediction sparse query vector;

[0020] performing trajectory prediction according to the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain an obstacle predicted trajectory and an ego vehicle predicted trajectory;

[0021] performing automatic driving of the ego vehicle based on the obstacle detection result and the online map, comprising:

[0022] performing automatic driving of the ego vehicle based on the obstacle detection result, the online map, the obstacle predicted trajectory and the ego vehicle predicted trajectory.

[0023] Optionally, the method further comprises:

[0024] interacting the self-vehicle trajectory prediction sparse query vector and the self-vehicle navigation command information to obtain a self-vehicle trajectory prediction sparse query vector containing navigation information;

[0025] optimizing the self-vehicle predicted trajectory under the plurality of constraint conditions by using the self-vehicle trajectory prediction sparse query vector containing navigation information to obtain a self-vehicle path planning result;

[0026] automatically driving the self-vehicle based on the obstacle detection result and the online map, including:

[0027] automatically driving the self-vehicle based on the self-vehicle path planning result, the obstacle detection result, the online map, the obstacle predicted trajectory, and the self-vehicle predicted trajectory.

[0028] Optionally, in the case that the plurality of constraint conditions includes a kinetic constraint, optimizing the self-vehicle predicted trajectory under the plurality of constraint conditions by using the self-vehicle trajectory prediction sparse query vector containing navigation information to obtain a self-vehicle path planning result, including:

[0029] regressing motion information of the self-vehicle by using the self-vehicle trajectory prediction sparse query vector containing navigation information, the motion information including a speed and an acceleration of the self-vehicle;

[0030] optimizing the self-vehicle predicted trajectory by using the motion information to obtain a self-vehicle path planning result.

[0031] Optionally, in the case that the plurality of constraint conditions includes a safety distance constraint, optimizing the self-vehicle predicted trajectory under the plurality of constraint conditions by using the self-vehicle trajectory prediction sparse query vector containing navigation information to obtain a self-vehicle path planning result, including:

[0032] predicting a relative position relationship between the obstacle and the self-vehicle according to the self-vehicle trajectory prediction sparse query vector containing navigation information and the obstacle trajectory prediction sparse query vector;

[0033] optimizing the self-vehicle predicted trajectory by using the relative position relationship for distance constraint to obtain a self-vehicle path planning result.

[0034] Optionally, determining an obstacle reference trajectory prediction sparse query vector according to obstacle historical position information, including:

[0035] determining a bounding box in the obstacle detection result whose bounding box confidence score exceeds a score threshold as a tracking target bounding box;

[0036] determining a sparse query vector corresponding to the tracking target bounding box as a tracking sparse query vector;

[0037] Encode the obstacle historical position information corresponding to the tracking sparse query vector into a historical position sparse query vector;

[0038] Interact the historical position sparse query vector and the tracking sparse query vector to obtain an obstacle reference trajectory prediction sparse query vector.

[0039] Optionally, the reference detection sparse query vector is obtained by the following steps:

[0040] Obtain a historical detection sparse query vector and a tracking sparse query vector at a previous time, the historical detection sparse query vector comprising: historical detection sparse query vectors at N previous times, N being an integer greater than 1;

[0041] Obtain a reference detection sparse query vector according to the historical detection sparse query vector and the tracking sparse query vector at the previous time.

[0042] Optionally, the current time sensor data is processed to obtain a sparse sensor feature vector, comprising:

[0043] Feature extraction is performed on the current time sensor data to obtain a sparse initial sensor feature vector;

[0044] According to the current time sensor data, the spatial position information of the sparse initial sensor feature vector is calculated;

[0045] The spatial position information and the sparse initial sensor feature vector are fused to obtain a sparse sensor feature vector.

[0046] Optionally, the end-to-end automatic driving is realized through a multi-task processing network;

[0047] In the case that the end-to-end automatic driving includes obstacle detection and map construction, the label of the training data of the multi-task processing network includes: real position information of the obstacle and real map information;

[0048] In the case that the end-to-end automatic driving includes obstacle detection, map construction and trajectory prediction, the label of the training data of the multi-task processing network includes: real position information of the obstacle, real map information, real trajectory of the obstacle and real trajectory of the ego vehicle;

[0049] In the case that the end-to-end automatic driving includes obstacle detection, map construction, trajectory prediction and path planning, the label of the training data of the multi-task processing network includes: real position information of the obstacle, real map information, real trajectory of the obstacle, real trajectory of the ego vehicle, real motion information of the ego vehicle, and real relative position relationship between the obstacle and the ego vehicle.

[0050] In a second aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the steps of the end-to-end autonomous driving method according to the first aspect of the present application.

[0051] In a third aspect, the present application provides a computer readable storage medium, having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the end-to-end autonomous driving method according to the first aspect of the present application.

[0052] In a fourth aspect, the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the end-to-end autonomous driving method according to the first aspect of the present application.

[0053] The embodiments of the present application have the following advantages:

[0054] In the embodiments of the present application, all information in the spatial dimension and the time dimension in the entire autonomous driving scene is represented by sparse query vectors, without using any dense bird's eye view features. The current time sensor data of the ego vehicle is obtained, and for the current time sensor data of the ego vehicle, the current time sensor data of the ego vehicle is processed into a sparse sensor feature vector; and the reference detection sparse query vector and the sparse sensor feature vector are interactively processed to obtain a detection sparse query vector; the reference map sparse query vector and the sparse sensor feature vector are interactively processed to obtain a map sparse query vector; thus, obstacle detection is performed according to the detection sparse query vector to obtain an obstacle detection result, and map construction is performed according to the map sparse query vector to obtain an online map, so as to drive the ego vehicle autonomously based on the obstacle detection result and the online map.

[0055] Since the reference detection sparse query vector comprises a historical detection sparse query vector determined based on historical sensor data of the ego vehicle, and the reference map sparse query vector comprises a historical map sparse query vector determined based on historical sensor data of the ego vehicle, the detection sparse query vector obtained based on the reference detection sparse query and the sparse sensor feature vector introduces historical time sequence information and current time sensor data, and the map detection sparse query vector obtained based on the reference map sparse query vector and the sparse sensor feature vector introduces historical time sequence information and current time sensor data. Thanks to the introduction of the historical time sequence information and the current time sensor data, each task (such as obstacle detection and map construction) in the end-to-end autonomous driving has better performance. Moreover, the sparse query vector can more effectively utilize long-time historical information, extend to more modalities and tasks, and reduce the computing cost and memory occupation. In this way, all information in different modalities, different tasks, space and time is represented as a sparse query vector, and various sub-tasks of autonomous driving are connected based on the sparse query vector, so as to realize efficient and high-accuracy end-to-end autonomous driving. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0057] FIG. 1 is a step flowchart of an end-to-end autonomous driving method provided by an embodiment of the present application;

[0058] FIG. 2 is a structural schematic diagram of a sparse perception network provided by an embodiment of the present application;

[0059] FIG. 3 is a structural schematic diagram of a motion planning network provided by an embodiment of the present application;

[0060] FIG. 4 is a step flowchart of another end-to-end autonomous driving method provided by an embodiment of the present application;

[0061] FIG. 5 is a structural schematic diagram of a multi-task processing network provided by an embodiment of the present application;

[0062] FIG. 6 is a structural schematic diagram of an end-to-end autonomous driving device provided by an embodiment of the present application;

[0063] FIG. 7 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED EMBODIMENTS

[0064] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0065] In the related art, the entire driving scene is represented by dense bird's eye view features, and the fusion of time sequence information and sensor data is achieved by dense bird's eye view features, which significantly increases the computing cost, memory occupation, and is difficult to implement model deployment, and the performance on each sub-task in autonomous driving is far behind the corresponding single task processing method. In order to overcome the limitations of the above problems, the applicant proposes the following technical concept: a pure sparse method is used, and map construction and obstacle detection are performed without relying on any dense bird's eye view features. All information in different modalities, different tasks, space and time is represented as a sparse query vector, and based on this, various sub-tasks of autonomous driving are connected in series to achieve efficient and high-accuracy end-to-end autonomous driving.

[0066] Based on the above technical concept, the embodiments of the present application provide an end-to-end autonomous driving method. Referring to FIG. 1, FIG. 1 is a step flowchart of an end-to-end autonomous driving method provided by the embodiments of the present application. As shown in FIG. 1, the method can include steps S101 to S104:

[0067] Step S101: Obtain current time sensor data of a vehicle.

[0068] In the embodiments of the present application, the current time sensor data is environmental data reflecting the entire autonomous driving scene of the vehicle collected at the current time, and the current time sensor data includes different types of sensor data (i.e., multi-modal data). In one embodiment, the current time sensor data includes point cloud data collected by a radar and multi-view images collected by a vehicle-mounted camera. The number of views of the multi-view images and the number of views can vary depending on the arrangement of the vehicle-mounted camera, and the multi-view images can include a front view image, a left front view image, a right front view image, a rear view image, a left rear view image, and a right rear view image of the vehicle. In addition, the multi-view images can be video frame images extracted from a video at the current time, or images directly collected by the vehicle-mounted camera at the current time.

[0069] Step S102: Process the current time sensor data to obtain a sparse sensor feature vector.

[0070] In the embodiments of the present application, the sparse sensor feature vector is a more discrete and sparse three-dimensional feature representation compared with the bird's eye view feature. The sparse sensor feature vector represents the target object in the driving scene and does not contain redundant information in the driving scene. For example, the sparse sensor feature vector represents an obstacle in the driving scene or a map element in the driving scene. Since the sparse sensor feature vector represents the target object in the driving scene, the sparse sensor feature vector contains semantic information and position information of the target object. In specific implementation, different feature extraction networks are used to process different types of sensor data (for example, image data and point cloud data) to obtain the sparse sensor feature vector. The support for multi-modal data can be expanded by increasing the number of encoders. Moreover, when the feature encoding of the current time sensor data is performed, the position information of the data in the three-dimensional space is also encoded, and then the feature (semantic information) and the position are unified into the sparse sensor feature vector.

[0071] In an optional embodiment, processing the current time sensor data to obtain the sparse sensor feature vector includes: performing feature extraction on the current time sensor data to obtain a sparse initial sensor feature vector; calculating spatial position information of the sparse initial sensor feature vector according to the current time sensor data; and fusing the spatial position information and the sparse initial sensor feature vector to obtain the sparse sensor feature vector.

[0072] In the embodiments of the present application, the sparse initial sensor feature vector refers to a feature vector containing semantic information of a target object. In order to accurately represent the target object in the driving scene, after obtaining the sparse initial sensor feature vector, the position of each sparse initial sensor feature vector is encoded to obtain a sparse sensor feature vector containing semantic information and position information.

[0073] The current time sensor data includes multiple different types of sensor data. For different types of sensor data, different feature extraction networks are used to extract features of different types of sensor data, and the features of different types of sensor data are flattened and spliced together to obtain a sparse initial sensor feature vector. For each sparse initial sensor feature vector, the corresponding spatial position information is calculated, and the spatial position information and the sparse initial sensor feature vector are fused to obtain a sparse sensor feature vector.

[0074] For example, the different types of sensor data include image data and cloud data. The image data is represented as I ∈ R N×H×W×3 where N is the number of input image perspectives; the point cloud data is represented as where N p is the number of points in the point cloud data, and C is the number of channels of the point cloud data.p is the dimension of each point cloud data. The image data and the point cloud data are respectively subjected to feature extraction using a feature extraction network to obtain image data features and point cloud data features, and the image data features and the point cloud data features are flattened and spliced together to obtain a sparse initial sensor feature vector where N f is the number of the sparse initial sensor feature vectors, and C is the dimension of the sparse initial sensor feature vectors. Then, for each sparse initial sensor feature vector, corresponding spatial position information is calculated, and the spatial position information is fused with the sparse initial sensor feature vector F t to obtain a sparse sensor feature vector.

[0075] Step S103: Interacting the reference detection sparse query vector with the sparse sensor feature vector to obtain a detection sparse query vector, the reference detection sparse query vector at least including a historical detection sparse query vector determined based on historical sensor data of the ego vehicle.

[0076] In the embodiments of the present application, the detection sparse query vector refers to a sparse query vector used for detection and tracking at the current moment, and each detection sparse query vector represents an obstacle object, for example, each detection sparse query vector corresponds to a vehicle in the driving scene. The historical detection sparse query vector refers to the historical detection sparse query vector at the N moments before the current moment, and the historical detection sparse query vector will also be updated as the autonomous driving proceeds.

[0077] Specifically, the interaction between the reference detection sparse query vector and the sparse sensor feature vector refers to: through a self-attention mechanism and a cross-attention mechanism, the reference detection sparse query vector and the sparse sensor feature vector are interacted to obtain a detection sparse query vector. Since the reference detection sparse query vector includes the historical detection sparse query vector, the historical time sequence information and the sensor data at the current moment are introduced into the detection sparse query vector through the interaction, and the detection sparse query vector can be used to well realize obstacle detection. After obtaining the detection sparse query vector, the detection sparse query vector is saved for the next moment of autonomous driving.

[0078] In an optional embodiment, the reference detection sparse query vector is obtained by the following steps: obtaining a historical detection sparse query vector and a tracking sparse query vector at the last moment, the historical detection sparse query vector including: historical detection sparse query vectors at N moments before the current moment, N being an integer greater than 1; and obtaining the reference detection sparse query vector according to the historical detection sparse query vector and the tracking sparse query vector at the last moment.

[0079] In the embodiments of the present application, the tracking sparse query vector of the previous moment is determined according to the obstacle detection result of the previous moment, and is used to track the obstacles with high credibility. In specific implementation, the historical detection sparse query vector and the tracking sparse query vector of the previous moment are stored in an end-to-end multi-task memory bank, and thus the historical detection sparse query vector and the tracking sparse query vector of the previous moment are obtained from the end-to-end multi-task memory bank.

[0080] Specifically, according to the historical detection sparse query vector and the tracking sparse query vector of the previous moment, the initialization detection sparse query vector, the historical detection sparse query vector and the tracking sparse query vector of the previous moment are spliced to obtain a reference detection sparse query vector. The initialization detection sparse query vector is a learnable random initialization vector.

[0081] Step S104: The reference map sparse query vector and the sparse sensor feature vector are interactively processed to obtain a map sparse query vector, and the reference map sparse query vector at least includes a historical map sparse query vector determined based on historical sensor data of the ego vehicle.

[0082] In the embodiments of the present application, the map sparse query vector refers to a sparse query vector used for online mapping at the current moment, and each map sparse query vector represents a map element, for example, each map sparse query vector corresponds to a line segment in the map. The historical map sparse query vector refers to the historical map sparse query vector of the N moments before the current moment, and the historical map sparse query vector will also be updated as the autonomous driving proceeds.

[0083] Specifically, the interactive processing of the reference map sparse query vector and the sparse sensor feature vector refers to: the reference map sparse query vector and the sparse sensor feature vector are interactively processed through a self-attention mechanism and a cross-attention mechanism to obtain a map sparse query vector. Since the reference map sparse query vector includes the historical map sparse query vector, the historical time sequence information and the sensor data at the current moment are introduced into the map sparse query vector through the interactive processing, and the map sparse query vector can be used to well realize the map construction. Moreover, after the map sparse query vector is obtained, the map sparse query vector is saved for the autonomous driving at the next moment.

[0084] In an optional embodiment, the reference map sparse query vector is obtained by the following steps: obtaining a historical map sparse query vector, the historical map sparse query vector comprising: historical map sparse query vectors of N time instants before a current time instant, N being an integer greater than 1; and splicing the historical map sparse query vector and an initial map sparse query vector to obtain a historical map sparse query vector, wherein the initial map sparse query vector is a learnable random initial vector.

[0085] Step S105: performing obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and performing map construction according to the map sparse query vector to obtain an online map.

[0086] In the embodiments of the present application, each detection sparse query vector represents an obstacle object, and the detection network is used to process the detection sparse query vector to obtain an obstacle detection result. The obstacle can be a vehicle in the driving scene or a pedestrian in the driving scene. The obstacle detection result is represented by a detection box and a detection box confidence score. The higher the detection box confidence score, the more reliable the corresponding obstacle detection result. Each map sparse query vector represents a map element, and the map construction network is used to process the map sparse query vector to obtain an online map. The online map includes lane line information and road sign information.

[0087] Step S106: performing automatic driving of the ego vehicle based on the obstacle detection result and the online map.

[0088] In the embodiments of the present application, the obstacle detection result and the online map are obtained by processing sensor data at the current time instant, and multiple sub-tasks of automatic driving are implemented based on the sensor data at the current time instant. In this way, all information of different modalities, different tasks, space and time is represented as sparse query vectors, and all sub-tasks of automatic driving are connected based on the sparse query vectors, thereby realizing efficient and high-accuracy end-to-end automatic driving.

[0089] In the embodiments of the present application, the end-to-end automatic driving is realized by a multi-task processing network. In the case where the end-to-end automatic driving includes obstacle detection and map construction, the labels of the training data of the multi-task processing network include real position information and real map information of the obstacle. By carrying the labels of the real position information and the real map information of the obstacle in the training data, end-to-end optimization of the obstacle detection task and end-to-end optimization of the map construction task are realized. In this way, the obstacle detection task and the map construction task are connected, thereby realizing efficient and high-accuracy end-to-end automatic driving.

[0090] Specifically, the multi-task processing network comprises a feature extraction network and a sparse perception network. The feature extraction network is configured to process the sensor data at the current time to obtain a sparse sensor feature vector. The sparse perception network is configured to perform the obstacle detection task and the map construction task. For example, FIG. 2 is a structural schematic diagram of a sparse perception network according to an embodiment of the present application. The sparse perception network comprises two time encoders, an obstacle object detection module (i.e., an obstacle output head), a map construction module (i.e., a map construction output head), and a data updating module.

[0091] The process of performing the obstacle detection task and the map construction task is as follows: obtaining a historical detection sparse query vector, a historical map sparse query vector, and a tracking sparse query vector at the previous time from the end-to-end multi-task memory bank; obtaining a reference detection sparse query vector according to the historical detection sparse query vector, the tracking sparse query vector at the previous time, and an initial detection sparse query vector; obtaining a reference map sparse query vector according to the historical map sparse query vector and an initial map sparse query vector; performing interactive processing on the reference detection sparse query vector and the sparse sensor feature vector by using one of the time encoders to obtain a detection sparse query vector, and performing obstacle detection on the detection sparse query vector by using the obstacle object detection module to obtain an obstacle detection result; performing interactive processing on the reference map sparse query vector and the sparse sensor feature vector by using the other time encoder to obtain a map sparse query vector, and performing map construction on the map sparse query vector by using the map construction module to obtain an online map. Finally, the data updating module updates the historical map sparse query vector, the historical detection sparse query vector, and the tracking sparse query vector in the end-to-end multi-task memory bank for the end-to-end autonomous driving at the next time.

[0092] In an optional embodiment, in order to obtain the obstacle prediction trajectory and the ego vehicle prediction trajectory, the ego vehicle is autonomously driven based on the obstacle prediction trajectory and the ego vehicle prediction trajectory on the basis of the obstacle detection result and the online map. After step S105, the following steps S107 to S1012 are further included.

[0093] Step S107: determining an obstacle reference trajectory prediction sparse query vector according to obstacle historical position information.

[0094] In the embodiment of the present application, the obstacle historical position information refers to the obstacle historical position information used for tracking, and the obstacle historical position information is represented by a coordinate position in the map. The obstacle reference trajectory prediction sparse query vector refers to an initial vector used for obstacle trajectory prediction.

[0095] Specifically, according to the obstacle historical position information, the obstacle reference trajectory prediction sparse query vector is determined, including: determining a bounding box in the obstacle detection result whose bounding box confidence score exceeds a score threshold as a tracking target bounding box; determining a sparse query vector corresponding to the tracking target bounding box as a tracking sparse query vector; encoding obstacle historical position information corresponding to the tracking sparse query vector as a historical position sparse query vector; and performing interaction processing on the historical position sparse query vector and the tracking sparse query vector to obtain the obstacle reference trajectory prediction sparse query vector.

[0096] In the embodiments of the present application, the score threshold is flexibly set according to actual conditions, and the obstacle detection result is represented by a bounding box and a bounding box confidence score. The higher the bounding box confidence score, the more reliable the corresponding obstacle detection result. Therefore, a bounding box in the obstacle detection result whose bounding box confidence score exceeds a score threshold is determined as a tracking target bounding box, and a sparse query vector corresponding to the tracking target bounding box is determined as a tracking sparse query vector. This realizes prediction of the obstacle trajectory corresponding to only the obstacle detection result with high confidence.

[0097] The historical position sparse query vector is a high-dimensional historical position feature compared with the obstacle historical position information. The obstacle historical position information is encoded as a historical position sparse query vector, so that the obstacle historical position information and the tracking sparse query vector can be interactively processed based on a multi-head attention mechanism. In a specific implementation, a learnable embedding vector is introduced to determine the obstacle reference trajectory prediction sparse query vector. The interaction processing result of the obstacle historical position information and the tracking sparse query vector is spliced with the learnable embedding vector to obtain the obstacle reference trajectory prediction sparse query vector.

[0098] Step S108: determining a vehicle reference trajectory prediction sparse query vector according to vehicle historical position information.

[0099] In the embodiments of the present application, the vehicle historical position information refers to the position of the vehicle in the driving scene, and the vehicle historical position information can be determined by a positioning device of the vehicle. The vehicle reference trajectory prediction sparse query vector refers to an initialization vector for vehicle trajectory prediction. Similarly, a learnable embedding vector is introduced to determine the vehicle reference trajectory prediction sparse query vector. According to the vehicle historical position information, the vehicle reference trajectory prediction sparse query vector is determined, including: encoding the vehicle historical position information to obtain encoded vehicle historical position information; and splicing the encoded vehicle historical position information and the vehicle learnable embedding vector to obtain the vehicle reference trajectory prediction sparse query vector.

[0100] Step S109: Interacting the obstacle reference trajectory prediction sparse query vector with the detection sparse query vector and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector.

[0101] In the embodiments of the present application, the obstacle trajectory prediction sparse query vector contains obstacle position information and map information for predicting the obstacle trajectory; different obstacles correspond to different obstacle trajectory prediction sparse query vectors. The detection sparse query vector refers to the detection sparse query vector obtained in the above step S103, and the map sparse query vector refers to the map sparse query vector obtained in the above step S104. The obstacle reference trajectory prediction sparse query vector refers to an initialization vector for obstacle trajectory prediction, and the obstacle reference trajectory prediction sparse query vector is interactively processed with the detection sparse query vector and the map sparse query vector through a multi-head attention mechanism to interactively update the obstacle reference trajectory prediction sparse query vector, thereby obtaining the obstacle trajectory prediction sparse query vector.

[0102] Step S1010: Interacting the ego vehicle reference trajectory prediction sparse query vector with the detection sparse query vector and the map sparse query vector to obtain an ego vehicle trajectory prediction sparse query vector.

[0103] In the embodiments of the present application, the ego vehicle trajectory prediction sparse query vector contains ego vehicle position information, surrounding obstacle position information, and map information for predicting the ego vehicle trajectory. The detection sparse query vector refers to the detection sparse query vector obtained in the above step S103, and the map sparse query vector refers to the map sparse query vector obtained in the above step S104. The ego vehicle reference trajectory prediction sparse query vector is interactively updated with the detection sparse query vector and the map sparse query vector through a multi-head attention mechanism to obtain the ego vehicle trajectory prediction sparse query vector.

[0104] Step S1011: Trajectory prediction is performed according to the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain an obstacle predicted trajectory and an ego vehicle predicted trajectory.

[0105] In the embodiments of the present application, the obstacle predicted trajectory refers to a predicted driving trajectory of the obstacle in the future, and the ego vehicle predicted trajectory refers to a predicted driving trajectory of the ego vehicle in the future. By predicting the obstacle predicted trajectory and the ego vehicle predicted trajectory, path planning for automatic driving of the ego vehicle can be achieved. Specifically, the trajectory prediction network is used to process the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector respectively to obtain the obstacle predicted trajectory and the ego vehicle predicted trajectory.

[0106] Step S1012: automatically driving the ego vehicle based on the obstacle detection result, the online map, the obstacle prediction trajectory and the ego vehicle prediction trajectory.

[0107] In the embodiments of the present application, the obstacle detection result, the online map, the obstacle prediction trajectory and the ego vehicle prediction trajectory are obtained by processing the sensor data at the current time, so that multiple sub-tasks of automatic driving are realized based on the sensor data at the current time. In this way, all information in different modalities, different tasks, space and time is represented as a sparse query vector, and various sub-tasks of automatic driving are connected based on the sparse query vector, so that efficient and high-accuracy end-to-end automatic driving is realized.

[0108] In the embodiments of the present application, the end-to-end automatic driving is realized by a multi-task processing network; in the case where the end-to-end automatic driving includes obstacle detection, map construction and trajectory prediction, the labels of the training data of the multi-task processing network include: real position information of the obstacle, real map information, real trajectory of the obstacle and real trajectory of the ego vehicle. By carrying the labels of the real position information of the obstacle, the real map information, the real trajectory of the obstacle and the real trajectory of the ego vehicle in the training data, end-to-end optimization of the obstacle detection task, end-to-end optimization of the map construction task and end-to-end optimization of the trajectory prediction task are realized. In this way, the obstacle detection task, the map construction task and the trajectory prediction task are connected, so that efficient and high-accuracy end-to-end automatic driving is realized.

[0109] In an optional embodiment, in order to realize trajectory planning of the ego vehicle, the ego vehicle is automatically driven based on the ego vehicle path planning result in addition to the obstacle detection result, the online map, the obstacle prediction trajectory and the ego vehicle prediction trajectory. After step S1011, steps S1013 to S1015 are further included:

[0110] Step S1013: interactively processing the ego vehicle trajectory prediction sparse query vector and the ego vehicle navigation command information to obtain an ego vehicle trajectory prediction sparse query vector containing navigation information.

[0111] Step S1014: using the ego vehicle trajectory prediction sparse query vector containing navigation information to optimize the ego vehicle prediction trajectory under multiple constraint conditions to obtain an ego vehicle path planning result.

[0112] Step S1015: automatically driving the ego vehicle based on the ego vehicle path planning result, the obstacle detection result, the online map, the obstacle prediction trajectory and the ego vehicle prediction trajectory.

[0113] In the embodiments of the present application, the ego vehicle navigation command information refers to a navigation command for controlling the vehicle, such as a left turn, a right turn, acceleration, deceleration, and the like. The plurality of constraints include dynamic constraints and safety distance constraints, wherein the dynamic constraints include speed constraints and acceleration constraints; the safety distance constraints refer to the relative position relationship between the ego vehicle and the predicted obstacle satisfying a safety distance. In order to realize the trajectory planning of the ego vehicle, the ego vehicle navigation command information is introduced to optimize the ego vehicle predicted trajectory, so as to obtain the ego vehicle path planning result.

[0114] Since the obstacle detection result, the online map, the obstacle predicted trajectory, the ego vehicle predicted trajectory, and the ego vehicle path planning result are all obtained by processing the sensor data at the current time, the plurality of sub-tasks of automatic driving are realized based on the sensor data at the current time. In this way, all information of different modalities, different tasks, space, and time is represented as a sparse query vector, and various sub-tasks of automatic driving are connected based on the sparse query vector, so as to realize efficient and high-accuracy end-to-end automatic driving.

[0115] The optimization processing of the ego vehicle predicted trajectory under the conditions of dynamic constraints and safety distance constraints will be described below.

[0116] (1) In the case where the plurality of constraints include dynamic constraints, the ego vehicle trajectory prediction sparse query vector containing navigation information is used to optimize the ego vehicle predicted trajectory under the plurality of constraints, so as to obtain the ego vehicle path planning result, including: using the ego vehicle trajectory prediction sparse query vector containing navigation information to regress the motion information of the ego vehicle, the motion information including the speed and acceleration of the ego vehicle; using the motion information to optimize the ego vehicle predicted trajectory, so as to obtain the ego vehicle path planning result.

[0117] Wherein, the regression of the motion information of the ego vehicle is equivalent to predicting the motion information (speed and acceleration) of the ego vehicle under the ego vehicle predicted trajectory, so as to facilitate the optimization processing of the motion information on the ego vehicle predicted trajectory. For example, if the speed of the ego vehicle exceeds the safe driving speed, the ego vehicle predicted trajectory is adjusted so that the adjusted trajectory satisfies the safe driving speed constraint. For another example, if the acceleration of the ego vehicle exceeds the safe driving acceleration (i.e., the acceleration is too fast), the ego vehicle predicted trajectory is adjusted so that the adjusted trajectory satisfies the safe driving acceleration constraint.

[0118] (2) in the case that the plurality of constraint conditions include a safety distance constraint, using the ego vehicle trajectory prediction sparse query vector containing navigation information to optimize the ego vehicle predicted trajectory under the plurality of constraint conditions to obtain an ego vehicle path planning result, including: predicting a relative position relationship between the ego vehicle and the obstacle according to the ego vehicle trajectory prediction sparse query vector containing navigation information and the obstacle trajectory prediction sparse query vector; using the relative position relationship to perform distance constraint optimization processing on the ego vehicle predicted trajectory to obtain the ego vehicle path planning result.

[0119] Specifically, a certain safety distance needs to be satisfied between the ego vehicle and the obstacle, and the distance constraint optimization processing on the ego vehicle predicted trajectory according to the relative position relationship means adjusting the ego vehicle predicted trajectory so that the adjusted trajectory satisfies the safety distance constraint.

[0120] In the embodiment of the application, the end-to-end autonomous driving is implemented through a multi-task processing network; in the case that the end-to-end autonomous driving includes obstacle detection, map construction, trajectory prediction and path planning, the labels of the training data of the multi-task processing network include: real position information of the obstacle, real map information, real trajectory of the obstacle, real trajectory of the ego vehicle, real motion information of the ego vehicle, and real relative position relationship between the obstacle and the ego vehicle.

[0121] By carrying the labels of the real position information of the obstacle, the real map information, the real trajectory of the obstacle, the real trajectory of the ego vehicle, the real motion information of the ego vehicle and the real relative position relationship between the obstacle and the ego vehicle in the training data, end-to-end optimization of the obstacle detection task, end-to-end optimization of the map construction task, end-to-end optimization of the trajectory prediction task, and end-to-end optimization of the path planning are implemented. In this way, the obstacle detection task, the map construction task, the trajectory prediction task and the path planning task are connected in series to implement efficient and high-accuracy end-to-end autonomous driving.

[0122] The multi-task processing network includes a feature extraction network, a sparse perception network and a motion planning network. The motion planning network is used to perform the trajectory prediction task and the path planning task. Specifically, FIG. 3 is a structural schematic diagram of a motion planning network provided by an embodiment of the application, and the motion planning network includes a trajectory prediction module and a path planning module.

[0123] Specifically, the process of performing the trajectory prediction task and the path planning task is: determining an obstacle reference trajectory prediction sparse query vector according to obstacle historical position information; and determining a self-vehicle reference trajectory prediction sparse query vector according to self-vehicle historical position information; then, obtaining a historical obstacle reference trajectory prediction sparse query vector, a historical self-vehicle reference trajectory prediction sparse query vector, a detection sparse query vector and a map sparse query vector from the end-to-end multi-task memory bank, and inputting the obstacle reference trajectory prediction sparse query vector and the self-vehicle reference trajectory prediction sparse query vector into an interactive prediction module for interactive processing and trajectory prediction to obtain a self-vehicle trajectory prediction sparse query vector. The interactive processing includes: the obstacle reference trajectory prediction sparse query vector interacts with the corresponding historical obstacle reference trajectory prediction sparse query vector, and the obstacle reference trajectory prediction sparse query vector interacts with the detection sparse query vector and the map sparse query vector; the self-vehicle reference trajectory prediction sparse query vector interacts with the corresponding historical self-vehicle reference trajectory prediction sparse query vector, and the self-vehicle reference trajectory prediction sparse query vector interacts with the detection sparse query vector and the map sparse query vector.

[0124] Then, the path planning module performs interactive processing on the self-vehicle trajectory prediction sparse query vector and self-vehicle navigation command information to obtain a self-vehicle trajectory prediction sparse query vector containing navigation information; and uses the self-vehicle trajectory prediction sparse query vector containing navigation information to perform optimization processing on the self-vehicle predicted trajectory under multiple constraint conditions to obtain a self-vehicle path planning result.

[0125] The present application will be described below in conjunction with a specific embodiment.

[0126] FIG. 4 is a step flowchart of another end-to-end automatic driving method provided by an embodiment of the present application, as shown in FIG. 4, including steps S401 to S4013:

[0127] Step S401: Obtain current time sensor data of a self-vehicle.

[0128] Step S402: Process the current time sensor data to obtain a sparse sensor feature vector.

[0129] Step S403: Perform interactive processing on a reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least includes a historical detection sparse query vector determined based on historical sensor data of the self-vehicle.

[0130] Step S404: Interacting the reference map sparse query vector and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least includes a historical map sparse query vector determined based on historical sensor data of the ego vehicle.

[0131] Step S405: Performing obstacle detection based on the detection sparse query vector to obtain an obstacle detection result, and performing map construction based on the map sparse query vector to obtain an online map.

[0132] Step S406: Determining an obstacle reference trajectory prediction sparse query vector based on historical position information of the obstacle.

[0133] Step S407: Determining an ego vehicle reference trajectory prediction sparse query vector based on historical position information of the ego vehicle.

[0134] Step S408: Interacting the obstacle reference trajectory prediction sparse query vector, the detection sparse query vector, and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector.

[0135] Step S409: Interacting the ego vehicle reference trajectory prediction sparse query vector, the detection sparse query vector, and the map sparse query vector to obtain an ego vehicle trajectory prediction sparse query vector.

[0136] Step S4010: Performing trajectory prediction based on the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain an obstacle predicted trajectory and an ego vehicle predicted trajectory.

[0137] Step S4011: Interacting the ego vehicle trajectory prediction sparse query vector and ego vehicle navigation command information to obtain an ego vehicle trajectory prediction sparse query vector with navigation information.

[0138] Step S4012: Utilizing the ego vehicle trajectory prediction sparse query vector with navigation information to perform optimization processing on the ego vehicle predicted trajectory under multiple constraint conditions to obtain an ego vehicle path planning result.

[0139] Step S4013: Performing automatic driving of the ego vehicle based on the ego vehicle path planning result, the obstacle detection result, the online map, the obstacle predicted trajectory, and the ego vehicle predicted trajectory.

[0140] In the embodiments of the present application, the obstacle detection task, the map construction task, the trajectory prediction task and the path planning task are connected in series, and based on the sensor data of the ego vehicle at the current time, efficient and high-accuracy end-to-end automatic driving can be realized. Specifically, obstacle detection and map construction are realized by performing steps S401 to S405; trajectory prediction is realized by performing steps S406 to S4010, to obtain the obstacle prediction trajectory and the ego vehicle prediction trajectory; in order to realize path planning, steps S4011 to S4012 are performed, to obtain the ego vehicle path planning result that satisfies multiple constraint conditions (such as kinetic constraint, safety distance constraint), so that step S4013 is performed to automatically drive the ego vehicle.

[0141] To sum up, in the embodiments of the present application, all information in the spatial dimension and the time dimension in the entire automatic driving scene is represented by sparse query vectors, without using any dense bird's eye view features, effectively utilizing long-time historical information, extending to more modalities and tasks, and reducing the computing cost and the memory occupation. Thanks to the introduction of the historical time sequence information and the sensor data at the current time, each task in the end-to-end automatic driving has better performance. In this way, the problems of single-task performance lag, high computing cost, large memory occupation and difficult model deployment of the end-to-end automatic driving model are solved.

[0142] In the embodiments of the present application, the end-to-end automatic driving is realized by a multi-task processing network. For example, FIG. 5 is a structural schematic diagram of a multi-task processing network provided by the embodiments of the present application, and the multi-task processing network includes a feature extraction network, a sparse perception network and a motion planning network.

[0143] Specifically, the sensor data (including image data and point cloud data) of the ego vehicle at the current time is input into the feature extraction network, the feature extraction network performs feature extraction and position encoding on the sensor data at the current time, and then inputs the sparse sensor feature vector.

[0144] The sparse perception network obtains the historical detection sparse query vector, the historical map sparse query vector and the tracking sparse query vector at the last time from the end-to-end multi-task memory bank, obtains the reference detection sparse query vector according to the historical detection sparse query vector and the tracking sparse query vector at the last time, and obtains the reference map sparse query vector according to the historical map sparse query vector; then, the reference detection sparse query vector and the sparse sensor feature vector are interactively processed to obtain the detection sparse query vector; the reference map sparse query vector and the sparse sensor feature vector are interactively processed to obtain the map sparse query vector; and obstacle detection is performed according to the detection sparse query vector to obtain the obstacle detection result, and map construction is performed according to the map sparse query vector to obtain the online map.

[0145] The motion planning network takes the detection sparse query vector, the map sparse query vector and the tracking sparse query vector as inputs, determines an obstacle reference trajectory prediction sparse query vector according to historical position information of the obstacle, and determines a self-vehicle reference trajectory prediction sparse query vector according to historical position information of the self-vehicle. Then, the obstacle reference trajectory prediction sparse query vector is interactively processed with the detection sparse query vector and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector, and the self-vehicle reference trajectory prediction sparse query vector is interactively processed with the detection sparse query vector and the map sparse query vector to obtain a self-vehicle trajectory prediction sparse query vector. Then, trajectory prediction is performed according to the obstacle trajectory prediction sparse query vector and the self-vehicle trajectory prediction sparse query vector to obtain an obstacle predicted trajectory and a self-vehicle predicted trajectory. Finally, the self-vehicle predicted trajectory is optimized based on self-vehicle navigation command information to obtain a self-vehicle path planning result that satisfies multiple constraints.

[0146] The embodiment of the present application also provides an end-to-end automatic driving device. Referring to FIG. 6, FIG. 6 is a structural schematic diagram of an end-to-end automatic driving device provided by the embodiment of the present application. The device comprises:

[0147] A data acquisition module 610 is configured to acquire current time sensor data of a self-vehicle.

[0148] A data processing module 620 is configured to process the current time sensor data to obtain a sparse sensor feature vector.

[0149] A first interaction module 630 is configured to interactively process a reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least comprises a historical detection sparse query vector determined based on historical sensor data of the self-vehicle.

[0150] A second interaction module 640 is configured to interactively process a reference map sparse query vector and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least comprises a historical map sparse query vector determined based on the historical sensor data of the self-vehicle.

[0151] A detection construction module 650 is configured to perform obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and perform map construction according to the map sparse query vector to obtain an online map.

[0152] An automatic driving module 660 is configured to perform automatic driving of the self-vehicle based on the obstacle detection result and the online map.

[0153] It can be understood that the end-to-end automatic driving device in the embodiments of the present application can implement the end-to-end automatic driving method in the above embodiments, and the end-to-end automatic driving device has the same advantages as the end-to-end automatic driving method described above relative to the prior art, which will not be described here.

[0154] The embodiments of the present application also provide an electronic device. Referring to FIG. 7, FIG. 7 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. As shown in FIG. 7, the electronic device 700 includes a memory 710 and a processor 720, the memory 710 and the processor 720 are in communication connection through a bus, and the memory 710 stores a computer program, the computer program can run on the processor 720, and then the steps of the end-to-end automatic driving method described in the embodiments of the present application are implemented.

[0155] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the end-to-end automatic driving method described in the embodiments of the present application.

[0156] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the end-to-end automatic driving method described in the embodiments of the present application.

[0157] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other.

[0158] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the method and device according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal equipment generate the device for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0159] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing terminal equipment to work in a specific way, so that the instructions stored in the computer readable storage medium generate a product including instruction devices, which implement the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0160] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more flows of the flowchart and / or one or more blocks of the block diagram.

[0161] Although the preferred embodiments of the application have been described, those skilled in the art will be able to make additional changes and modifications to these embodiments once they have the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the application.

[0162] Finally, it should also be noted that, in this document, relational terms such as first and second and the like can only be used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or terminal device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or terminal device including the element.

[0163] The above describes in detail the end-to-end automatic driving method, device and storage medium provided by the application. The principles and implementation manners of the application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the application and its core idea; meanwhile, for those skilled in the art, according to the idea of the application, the specific implementation manner and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the application.

Claims

1. An end-to-end autonomous driving method, characterized in that: The method comprises: Get the current sensor data of the vehicle; Processing the current moment sensor data to obtain a sparse sensor feature vector; Interactively processing a reference detection sparse query vector and the sparse sensor feature vector to obtain a detection sparse query vector, wherein the reference detection sparse query vector at least includes: a historical detection sparse query vector determined based on historical sensor data of the vehicle; Interactively processing a reference map sparse query quantity and the sparse sensor feature vector to obtain a map sparse query vector, wherein the reference map sparse query vector at least includes: a historical map sparse query vector determined based on historical sensor data of the vehicle; performing obstacle detection according to the detection sparse query vector to obtain an obstacle detection result, and performing map construction according to the map sparse query vector to obtain an online map; The vehicle is automatically driven based on the obstacle detection result and the online map.

2. The method according to claim 1, characterized in that The method further comprises: Determine the obstacle reference trajectory prediction sparse query vector based on the historical position information of the obstacle; Determine the sparse query vector for the ego vehicle reference trajectory prediction based on the ego vehicle's historical position information; Interactively processing the obstacle reference trajectory prediction sparse query vector, the detection sparse query vector, and the map sparse query vector to obtain an obstacle trajectory prediction sparse query vector; Interactively processing the ego vehicle reference trajectory prediction sparse query vector, the detection sparse query vector, and the map sparse query vector to obtain an ego vehicle trajectory prediction sparse query vector; Performing trajectory prediction based on the obstacle trajectory prediction sparse query vector and the ego vehicle trajectory prediction sparse query vector to obtain an obstacle prediction trajectory and an ego vehicle prediction trajectory; Automatically driving the vehicle based on the obstacle detection result and the online map, including: Automatically drive the vehicle based on the obstacle detection result, the online map, the obstacle prediction trajectory, and the vehicle prediction trajectory.

3. The method according to claim 2, characterized in that The method further comprises: Interactively processing the ego-vehicle trajectory prediction sparse query vector and ego-vehicle navigation command information to obtain an ego-vehicle trajectory prediction sparse query vector containing navigation information; Utilizing the ego vehicle trajectory containing navigation information to predict a sparse query vector, optimizing the ego vehicle predicted trajectory under multiple constraints to obtain an ego vehicle path planning result; Automatically driving the vehicle based on the obstacle detection result and the online map, including: The self-vehicle is automatically driven based on the self-vehicle path planning result, the obstacle detection result, the online map, the obstacle prediction trajectory and the self-vehicle prediction trajectory.

4. The method according to claim 3, characterized in that When the multiple constraints include dynamic constraints, the ego vehicle trajectory prediction sparse query vector containing navigation information is used to optimize the ego vehicle predicted trajectory under the multiple constraints to obtain an ego vehicle path planning result, including: Using the ego vehicle trajectory containing navigation information to predict a sparse query vector, regressing ego vehicle motion information, the motion information including ego vehicle speed and acceleration; The motion information is used to optimize the predicted trajectory of the vehicle to obtain a vehicle path planning result.

5. The method according to claim 3 or 4, characterized in that When the multiple constraints include a safety distance constraint, the ego vehicle trajectory including navigation information is used to predict a sparse query vector, and the predicted ego vehicle trajectory is optimized under the multiple constraints to obtain an ego vehicle path planning result, including: Predicting a relative position relationship between the obstacle and the ego vehicle based on the ego vehicle trajectory prediction sparse query vector containing navigation information and the obstacle trajectory prediction sparse query vector; The relative position relationship is used to perform distance constraint optimization processing on the predicted trajectory of the ego vehicle to obtain a ego vehicle path planning result.

6. The method according to claim 2, characterized in that Based on the historical position information of the obstacle, the obstacle reference trajectory prediction sparse query vector is determined, including: Determine the detection frame whose detection frame confidence score exceeds the score threshold in the obstacle detection result as the tracking target detection frame; Determine the sparse query vector corresponding to the tracking target detection box as the tracking sparse query vector; Encoding the obstacle historical position information corresponding to the tracking sparse query vector into a historical position sparse query vector; The historical position sparse query vector and the tracking sparse query vector are interactively processed to obtain an obstacle reference trajectory prediction sparse query vector.

7. The method according to claim 1, characterized in that The reference detection sparse query vector is obtained by the following steps: Obtaining a historical detection sparse query vector and a tracking sparse query vector at a previous moment, wherein the historical detection sparse query vector includes: historical detection sparse query vectors at N moments before the current moment, where N is an integer greater than 1; A reference detection sparse query vector is obtained according to the historical detection sparse query vector and the tracking sparse query vector at the last moment.

8. The method according to claim 1, characterized in that The current moment sensor data is processed to obtain a sparse sensor feature vector, including: Performing feature extraction on the current moment sensor data to obtain a sparse initial sensor feature vector; Calculating spatial position information of the sparse initial sensor feature vector based on the current sensor data; The spatial position information and the sparse initial sensor feature vector are fused to obtain a sparse sensor feature vector.

9. The method according to any one of claims 1 to 8, characterized in that: End-to-end autonomous driving is achieved through a multi-tasking processing network; In the case where the end-to-end autonomous driving includes obstacle detection and map construction, the labels of the training data of the multi-task processing network include: real position information of obstacles and real map information; In the case where the end-to-end autonomous driving includes obstacle detection, map construction, and trajectory prediction, the labels of the training data of the multi-task processing network include: real position information of obstacles, real map information, real trajectory of obstacles, and real trajectory of the vehicle; In the case where the end-to-end autonomous driving includes obstacle detection, map construction, trajectory prediction and path planning, the labels of the training data of the multi-task processing network include: the real position information of the obstacle, the real map information, the real trajectory of the obstacle, the real trajectory of the vehicle, the real motion information of the vehicle, and the real relative position relationship between the obstacle and the vehicle.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the end-to-end autonomous driving method described in any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the end-to-end autonomous driving method described in any one of claims 1 to 9 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the end-to-end autonomous driving method as described in any one of claims 1 to 9 are implemented.

Images