End-to-end autonomous driving method, system and electronic equipment

Abstract

This application provides an end-to-end autonomous driving method, system, and electronic device. The method includes: acquiring camera data and LiDAR data installed on a vehicle; encoding the camera data and LiDAR data to obtain encoder features; inputting the encoder features into a first multilayer perceptron to predict the vehicle's coarse-grained trajectory and control signals; and iteratively optimizing the coarse-grained trajectory and control signals to obtain fine-grained trajectory and control signals. This scheme achieves iterative improvement from coarse-grained to fine-grained trajectory and control signals based on existing predictions, which can reduce the difficulty of model training and the risk of overfitting.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and specifically relates to an end-to-end autonomous driving method, system and electronic device. Background Technology

[0002] End-to-end autonomous driving relies on the collaborative efforts of artificial intelligence, computer vision, and monitoring devices to enable the vehicle's computer to operate autonomously without human intervention. The end-to-end autonomous driving model maps input sensor data into trajectory / control signals, thereby avoiding the cascading errors and complex manually defined rules inherent in traditional modular autonomous driving designs.

[0003] Existing end-to-end autonomous driving methods all follow the encoder-decoder principle, where the encoder typically transmits the input obtained from the decoder directly to a multi-layer perception machine.

[0004] The drawback of existing methods is that the decoder is responsible for finding the features important for driving from the complex global features obtained from the encoder. However, the decoder is often a simple multilayer perceptron. Therefore, the simple structure of the decoder makes it difficult for it to complete such a complex task. Simply increasing the width and depth of the decoder will lead to overfitting. Summary of the Invention

[0005] The purpose of the embodiments in this specification is to provide an end-to-end autonomous driving method, system, and electronic device.

[0006] To solve the above-mentioned technical problems, the embodiments of this application are implemented in the following ways:

[0007] In a first aspect, this application provides an end-to-end autonomous driving method, the method comprising:

[0008] Acquire camera data and lidar data from the camera and lidar installed on the vehicle, respectively;

[0009] Encode the camera data and LiDAR data to obtain encoder features;

[0010] The encoder features are input into the first multilayer sensor to predict the vehicle's coarse-grained trajectory and control signals.

[0011] The coarse-grained trajectory and control signal are iteratively optimized to obtain the fine-grained trajectory and control signal.

[0012] In one embodiment, camera data and LiDAR data are encoded to obtain encoder features, including:

[0013] Map camera data to first bird's-eye view features;

[0014] Map the lidar data to second bird's-eye view features;

[0015] The first and second bird's-eye view features are combined to obtain the overall bird's-eye view features;

[0016] The overall bird's-eye view features are flattened from two dimensions to one dimension to obtain the encoder features.

[0017] In one embodiment, the coarse-grained trajectory and control signal are iteratively optimized to obtain a fine-grained trajectory and control signal, including:

[0018] Based on the trajectory predicted in the previous step, the features corresponding to the coordinates of the trajectory on the first bird's-eye view features and the second bird's-eye view features are extracted to obtain the observation features;

[0019] Based on the trajectory and control signals predicted in the previous step, we infer the features corresponding to the scene conditions at future moments and obtain the predicted features.

[0020] The observed features, predicted features, the trajectory predicted in the previous step, and the control signal are spliced ​​together to obtain the spliced ​​features;

[0021] The spliced ​​features are input into the second multilayer perceptron to obtain the correction values ​​of the trajectory predicted in the previous step and the control signal.

[0022] The predicted trajectory and control signal from the previous step are added together with the correction value and used as the input for the next iteration. The iteration continues until the preset number of iterations is reached, resulting in a fine-grained trajectory and control signal.

[0023] In one embodiment, based on the trajectory predicted in the previous step, features corresponding to the coordinates of the trajectory on the first bird's-eye view features and the second bird's-eye view features are extracted to obtain observation features, including:

[0024] Map the coordinates in the trajectory predicted in the previous step to the image coordinate system and the point cloud coordinate system to obtain the image coordinate system coordinates and the point cloud coordinate system coordinates respectively;

[0025] Extract the first feature of the image coordinate system location from the first bird's-eye view features, and extract the second feature of the point cloud coordinate system location from the second bird's-eye view features;

[0026] By concatenating the first feature and the second feature, the observed feature is obtained.

[0027] In one embodiment, based on the trajectory and control signal predicted in the previous step, features corresponding to the future scene conditions are inferred to obtain predicted features, including:

[0028] The predicted trajectory and control signal from the previous step are stitched together with the features of the overall bird's-eye view to obtain the stitched features;

[0029] The stitched features are input into the ConvLSTM network, which outputs two-dimensional future bird's-eye view features.

[0030] The two-dimensional bird's-eye view features are flattened into one dimension to obtain one-dimensional bird's-eye view features;

[0031] The features of the one-dimensional bird's-eye view are input into the third multilayer perceptron to obtain the predicted features.

[0032] Secondly, this application provides an end-to-end autonomous driving system, the system comprising:

[0033] The encoder is used to acquire camera data and LiDAR data installed on the vehicle; and to encode the camera data and LiDAR data to obtain encoder features;

[0034] The first multilayer perceptron is used to predict the vehicle's coarse-grained trajectory and control signals based on encoder features;

[0035] The decoder is used to iteratively optimize the coarse-grained trajectory and control signals to obtain fine-grained trajectory and control signals.

[0036] In one embodiment, the encoder includes:

[0037] The first backbone network is used to map camera data into first bird's-eye view features;

[0038] The second backbone network is used to map LiDAR data into second bird's-eye view features;

[0039] The merging module is used to merge the first bird's-eye view features and the second bird's-eye view features to obtain the overall bird's-eye view features.

[0040] The dimensionality reduction module is used to flatten the overall bird's-eye view features from two dimensions into one dimension, thus obtaining the encoder features.

[0041] In one embodiment, the decoder includes:

[0042] An observation model is used to extract the features of the coordinates corresponding to the trajectories on the first and second bird's-eye view features, thus obtaining the observation features.

[0043] In one embodiment, the decoder further includes:

[0044] A predictive model is used to infer the features corresponding to the state of a scenario at a future time, thus obtaining the predicted features.

[0045] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the end-to-end autonomous driving method as described in the first aspect.

[0046] As can be seen from the technical solutions provided in the embodiments of this specification above, this solution: based on the existing prediction, the trajectory and control signal are iterated from coarse-grained to fine-grained, which can reduce the difficulty of model training and the risk of overfitting. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating the end-to-end autonomous driving method provided in this application;

[0049] Figure 2 A schematic diagram of the end-to-end autonomous driving system provided in this application;

[0050] Figure 3 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0052] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0053] Various modifications and variations can be made to the specific embodiments described in this application without departing from the scope or spirit of this application, as will be apparent to those skilled in the art. Other embodiments derived from this application will be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0054] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0056] Reference Figure 1 It illustrates a flowchart applicable to the end-to-end autonomous driving method provided in the embodiments of this application.

[0057] like Figure 1 As shown, an end-to-end autonomous driving method may include:

[0058] S110: Acquire camera data and lidar data from the camera and lidar installed on the vehicle, respectively.

[0059] Specifically, the sensors on the vehicle include cameras and lidar, and there may be multiple cameras.

[0060] S120. Encode the camera data and LiDAR data to obtain encoder features, which may include:

[0061] Map camera data to first bird's-eye view features;

[0062] Map the lidar data to second bird's-eye view features;

[0063] The first and second bird's-eye view features are combined to obtain the overall bird's-eye view features;

[0064] The overall bird's-eye view features are flattened from two dimensions to one dimension to obtain the encoder features.

[0065] For camera data from multiple cameras, the classic LSS (Lift, Splat, Shoot) method can be used for processing, and methods such as BEVFormer and BEVDepth can be used to map the camera data into first bird's-eye view features.

[0066] For LiDAR data, the classic SECOND (Sparsely Embedded Convolutional Detection) method can be used to obtain second bird's-eye view features, and methods such as PointPilllars and VoxelNet can also be used.

[0067] Understandably, the above encoding process can use other encoding methods, as long as the encoder features are ultimately obtained.

[0068] S130. Input the encoder features into the first multilayer sensor to predict the vehicle's coarse-grained trajectory and control signals.

[0069] S140. Iteratively optimize the coarse-grained trajectory and control signal to obtain a fine-grained trajectory and control signal, which may include:

[0070] Based on the trajectory predicted in the previous step, the features corresponding to the coordinates of the trajectory on the first bird's-eye view features and the second bird's-eye view features are extracted to obtain the observation features;

[0071] Based on the trajectory and control signals predicted in the previous step, we infer the features corresponding to the scene conditions at future moments and obtain the predicted features.

[0072] The observed features, predicted features, the trajectory predicted in the previous step, and the control signal are spliced ​​together to obtain the spliced ​​features;

[0073] The spliced ​​features are input into the second multilayer perceptron to obtain the correction values ​​of the trajectory predicted in the previous step and the control signal.

[0074] The predicted trajectory and control signal from the previous step are added together with the correction value and used as the input for the next iteration. The iteration continues until the preset number of iterations is reached, resulting in a fine-grained trajectory and control signal.

[0075] In one embodiment, based on the trajectory predicted in the previous step, features corresponding to the coordinates of the trajectory on the first bird's-eye view features and the second bird's-eye view features are extracted to obtain observation features, which may include:

[0076] Map the coordinates in the trajectory predicted in the previous step to the image coordinate system and the point cloud coordinate system to obtain the image coordinate system coordinates and the point cloud coordinate system coordinates respectively;

[0077] Extract the first feature of the image coordinate system location from the first bird's-eye view features, and extract the second feature of the point cloud coordinate system location from the second bird's-eye view features;

[0078] By concatenating the first feature and the second feature, the observed feature is obtained.

[0079] Specifically, based on the coordinates of the trajectory predicted in the previous step, they are mapped to the coordinate systems of each sensor, namely the image coordinate system and the point cloud coordinate system. Then, based on the coordinates in their respective sensor coordinate systems, the features of the coordinate locations are extracted from the first bird's-eye view features and the second bird's-eye view features, respectively. The features extracted from each sensor are then stitched together, which is called the observation features.

[0080] In one embodiment, based on the trajectory and control signals predicted in the previous step, the features corresponding to the future scene conditions are inferred to obtain predicted features, which may include:

[0081] The predicted trajectory and control signal from the previous step are stitched together with the features of the overall bird's-eye view to obtain the stitched features;

[0082] The stitched features are input into the ConvLSTM network, which outputs two-dimensional future bird's-eye view features.

[0083] The two-dimensional bird's-eye view features are flattened into one dimension to obtain one-dimensional bird's-eye view features;

[0084] The features of the one-dimensional bird's-eye view are input into the third multilayer perceptron to obtain the predicted features.

[0085] Specifically, the predicted trajectory from the previous step is concatenated with the control signal and the overall bird's-eye view features, and then input into a ConvLSTM network to output the future bird's-eye view features. The 2D future bird's-eye view features are flattened from two dimensions to one dimension and then input into the third multilayer perceptron to obtain the predicted features.

[0086] The end-to-end autonomous driving method provided in this application reduces the difficulty of model training and the risk of overfitting by iterating the predicted trajectory and control signal from coarse to fine granular based on existing methods.

[0087] Experimental verification

[0088] The end-to-end autonomous driving method provided in this application was tested in the Carla simulator using two publicly available end-to-end driving evaluation standards: town05long and longest6. The method significantly outperformed existing methods on both standards. Furthermore, compared to a baseline model that does not use multi-step iterative optimization, the improvement was approximately 8%.

[0089] Reference Figure 2 It shows a schematic diagram of the structure of an end-to-end autonomous driving system described according to an embodiment of this application.

[0090] like Figure 2 As shown, an end-to-end autonomous driving system may include:

[0091] The encoder is used to acquire camera data and LiDAR data installed on the vehicle; and to encode the camera data and LiDAR data to obtain encoder features;

[0092] The first multilayer perceptron is used to predict the vehicle's coarse-grained trajectory and control signals based on encoder features;

[0093] The decoder is used to iteratively optimize the coarse-grained trajectory and control signals to obtain fine-grained trajectory and control signals.

[0094] The encoder includes:

[0095] The first backbone network is used to map camera data into first bird's-eye view features;

[0096] The second backbone network is used to map LiDAR data into second bird's-eye view features;

[0097] The merging module is used to merge the first bird's-eye view features and the second bird's-eye view features to obtain the overall bird's-eye view features.

[0098] The dimensionality reduction module is used to flatten the overall bird's-eye view features from two dimensions into one dimension, thus obtaining the encoder features.

[0099] The decoder includes:

[0100] An observation model is used to extract features from the first bird's-eye view and the features corresponding to the coordinates of the trajectory on the second bird's-eye view, thus obtaining observation features.

[0101] The decoder also includes:

[0102] A predictive model is used to infer the features corresponding to the state of a scenario at a future time, thus obtaining the predicted features.

[0103] This embodiment provides an end-to-end autonomous driving system that can execute the above-described method. Its implementation principle and technical effects are similar, and will not be repeated here.

[0104] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 3 The diagram shows a structural schematic of an electronic device 300 suitable for implementing embodiments of this application.

[0105] like Figure 3 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage section 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the device 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0106] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 310 as needed so that computer programs read from it can be installed into storage section 308 as needed.

[0107] In particular, according to embodiments of this disclosure, the above references Figure 1 The described process can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the end-to-end autonomous driving method described above. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0109] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0110] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a mobile phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0111] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0112] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. An end-to-end autonomous driving method, characterized in that, The method includes: Acquire camera data and lidar data from the camera and lidar installed on the vehicle, respectively; Encoding the camera data and the lidar data includes: mapping the camera data into a first bird's-eye view feature; mapping the lidar data into a second bird's-eye view feature; merging the first bird's-eye view feature and the second bird's-eye view feature to obtain a total bird's-eye view feature; flattening the total bird's-eye view feature from two dimensions to one dimension to obtain an encoder feature; and inputting the encoder feature into a first multilayer perceptron to predict the vehicle's coarse-grained trajectory and control signals. The coarse-grained trajectory and control signal are iteratively optimized, including: based on the trajectory predicted in the previous step, extracting features corresponding to the coordinates of the trajectory on the first bird's-eye view features and the second bird's-eye view features to obtain observation features; based on the trajectory and control signal predicted in the previous step, inferring features corresponding to the scene status at future moments to obtain prediction features; concatenating the observation features, the prediction features, and the trajectory and control signal predicted in the previous step to obtain concatenated features; inputting the concatenated features into a second multilayer perceptron to obtain correction values ​​for the trajectory and control signal predicted in the previous step; adding the trajectory and control signal predicted in the previous step to the correction values ​​as input for the next iteration, until a preset number of iterations is reached and iteration stops to obtain fine-grained trajectory and control signals.

2. The method according to claim 1, characterized in that, Based on the trajectory predicted in the previous step, features corresponding to the coordinates of the trajectory are extracted from the first bird's-eye view features and the second bird's-eye view features to obtain observation features, including: Map the coordinates in the trajectory predicted in the previous step to the image coordinate system and the point cloud coordinate system to obtain the image coordinate system coordinates and the point cloud coordinate system coordinates, respectively. Extract the first feature of the image coordinate system location from the first bird's-eye view features, and extract the second feature from the second feature. Extract the second feature of the point cloud coordinate system location from the bird's-eye view features; The first feature and the second feature are concatenated to obtain the observed feature.

3. The method according to claim 1, characterized in that, The step of inferring the features corresponding to the future scene conditions based on the trajectory and control signals predicted in the previous step, and obtaining the predicted features, includes: The predicted trajectory and control signal from the previous step are stitched together with the overall bird's-eye view features to obtain the stitched features; The stitched features are input into a ConvLSTM network, which outputs two-dimensional future bird's-eye view features. The two-dimensional future bird's-eye view features are flattened into one dimension to obtain one-dimensional bird's-eye view features. The one-dimensional bird's-eye view features are input into a third multilayer perceptron to obtain the predicted features.

4. An end-to-end autonomous driving system, characterized in that, The system includes: An encoder is used to acquire camera data and lidar data installed on a vehicle; and to encode the camera data and lidar data, including: mapping the camera data to a first bird's-eye view feature; mapping the lidar data to a second bird's-eye view feature; merging the first bird's-eye view feature and the second bird's-eye view feature to obtain a total bird's-eye view feature; and flattening the total bird's-eye view feature from two dimensions to one dimension to obtain the encoder feature; The first multilayer perceptron is used to predict the coarse-grained trajectory and control signals of the vehicle based on the encoder features. The decoder is used to iteratively optimize the coarse-grained trajectory and control signal, including: extracting features corresponding to the coordinates of the trajectory on the first bird's-eye view features and the second bird's-eye view features based on the trajectory predicted in the previous step, to obtain observation features; inferring features corresponding to the future scene conditions based on the trajectory and control signal predicted in the previous step, to obtain prediction features; concatenating the observation features, the prediction features, and the trajectory and control signal predicted in the previous step, to obtain concatenated features; inputting the concatenated features into a second multilayer perceptron to obtain a correction value for the trajectory and control signal predicted in the previous step; adding the trajectory and control signal predicted in the previous step to the correction value as the input for the next iteration, until a preset number of iterations is reached and the iteration stops, to obtain fine-grained trajectory and control signal.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the end-to-end autonomous driving method as described in any one of claims 1-3.

Images