Bird-eye view and deep reinforcement learning-based end-to-end automatic driving control method

Abstract

The invention relates to the technical field of automatic driving control, in particular to an end-to-end automatic driving control method based on a bird's-eye view and deep reinforcement learning, and the method comprises the steps: collecting multi-view image data of an environment where a target vehicle is located under the condition that the target vehicle is detected to be in an automatic driving working condition, the multi-view image data is acquired, the multi-view image data is converted into aerial view representation to generate an aerial view feature map of the environment, and each pixel in the aerial view feature map is classified to obtain a semantic segmentation map of the environment; and inputting the aerial view feature map and the semantic segmentation map into a target deep reinforcement learning network to output an automatic driving control signal of the target vehicle so as to control the target vehicle to run. Therefore, the problems that an automatic driving control system in the prior art is high in calculation complexity, low in capacity of adapting to a complex environment and insufficient in interpretability, the overall performance of the system is reduced, and the driving and riding requirements of a user cannot be met are solved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving control technology, and in particular to an end-to-end autonomous driving control method based on bird's-eye view and deep reinforcement learning. Background Art

[0002] The autonomous driving control system in related technologies relies on a series of advanced technologies, including sensor fusion, such as LiDAR (Light Detection And Ranging), cameras, and radars; high-precision positioning, such as GPS (Global Positioning System), IMU (Inertial Measurement Unit), and SLAM (Simultaneous Localization And Mapping); behavior prediction; path planning and control algorithms, such as MPC (Model Predictive Control) and PID (Proportional-Integral-Derivative Control), to achieve environmental perception, decision-making, and action control.

[0003] However, the autonomous driving control system in related technologies has limitations, and multiple independent modules increase the computational complexity of the system. The system has low ability to adapt to complex environments and insufficient interpretability, which reduces the overall performance of the system and fails to meet the user's driving needs, reducing the user's driving experience, which needs to be urgently addressed. Summary of the invention

[0004] This application is based on the following problems and understandings made by the inventor:

[0005] The autonomous driving control system is one of the core technologies that supports the autonomous navigation and operation of vehicles. It uses a variety of sensors and algorithms to achieve perception, decision-making and action control of the surrounding environment. The autonomous driving control system usually includes several main components: perception module, positioning module, prediction module, planning module and control module. The perception module uses a variety of sensors such as LiDAR (Light Detection And Ranging), camera, millimeter wave radar and ultrasonic sensor, combined with computer vision and deep learning technology, to identify and classify surrounding objects and obtain environmental information in real time, including roads, obstacles, traffic signals, etc. The positioning module uses GPS (Global Positioning System), IMU (Inertial Measurement Unit) and high-precision maps to locate the vehicle. Commonly used technologies also include SLAM (Simultaneous Localization And Mapping) to determine the exact location of the vehicle on the map. The prediction module analyzes the movement patterns of traffic participants based on historical data and deep learning models to make safer decisions and predict the future behavior of surrounding traffic participants (such as vehicles, pedestrians, etc.). The planning module calculates the best driving path from the current position to the target position through global path planning and local path planning algorithms, and formulates driving paths and strategies for the vehicle by taking into account dynamic obstacles and traffic rules. The control module uses algorithms such as MPC (Model Predictive Control) and PID (Proportional-Integral-Derivative Control) to accurately control the acceleration, braking and steering of the vehicle to achieve smooth driving and convert the planned path into specific operating instructions.

[0006] However, this modular approach has four shortcomings: error propagation, high computational complexity, lack of end-to-end learning capability, and insufficient interpretability. Error propagation refers to the independence between modules, which means that the error of one module may affect the performance of subsequent modules and reduce the reliability of the overall system; high computational complexity means that multiple independent modules need to perform complex data transmission and processing, which increases the computational burden; lack of end-to-end learning capability means that existing methods cannot fully utilize the advantages of deep learning technology, resulting in limited understanding of the environment and decision-making capabilities. Insufficient interpretability means that the boundaries between modules are blurred, making it difficult to explain the decision-making process of the system as a whole, which limits users' trust in the autonomous driving system and needs to be improved urgently.

[0007] The present application provides an end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning to address the limitations of autonomous driving control systems in related technologies, as well as the fact that multiple independent modules increase the computational complexity of the system, the system has a low ability to adapt to complex environments, and lacks interpretability, which reduces the overall performance of the system, fails to meet the user's driving needs, and reduces the user's driving experience.

[0008] The first aspect of the present application provides an end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning, comprising the following steps: when a target vehicle is detected to be in an autonomous driving condition, collecting multi-perspective image data of the target vehicle's environment; based on a bird's-eye view feature extraction network, converting the multi-perspective image data into a bird's-eye view representation to generate a bird's-eye view feature map of the environment, and classifying each pixel in the bird's-eye view feature map to obtain a semantic segmentation map of the environment; inputting the bird's-eye view feature map and the semantic segmentation map into a target deep reinforcement learning network to output an autonomous driving control signal for the target vehicle, and controlling the target vehicle to travel according to the autonomous driving control signal.

[0009] Optionally, in one embodiment of the present application, the bird's-eye view feature extraction network is used to convert the multi-view image data into a bird's-eye view representation, including: extracting target features of the multi-view image data to generate a target feature map; calculating the depth distribution of each pixel in the target feature map to obtain a target depth value corresponding to each pixel in the target feature map; determining the probability distribution of each depth value, and generating a depth prediction map of the target feature map based on the probability distribution; determining the 3D coordinates of each pixel based on the depth prediction map, and generating a 3D point cloud using the 3D coordinates of each pixel; and projecting the 3D point cloud into a bird's-eye view grid to convert the multi-view image data into the bird's-eye view representation.

[0010] Optionally, in one embodiment of the present application, classifying each pixel in the bird's-eye view feature map to obtain a semantic segmentation map of the environment includes: extracting abstract features of the bird's-eye view feature map using an encoder in a pre-trained target network; and classifying each pixel in the bird's-eye view feature map based on a decoder in the pre-trained target network and the abstract features to obtain a semantic segmentation map of the environment.

[0011] Optionally, in one embodiment of the present application, the inputting the bird's-eye view feature map and the semantic segmentation map into the target deep reinforcement learning network and outputting the autonomous driving control signal of the target vehicle includes: preliminarily processing the bird's-eye view feature map, the semantic segmentation map and the state information of the target vehicle to obtain target-level features; using the target-level features to determine the target probability distribution of each action performed by the target vehicle, and determining the autonomous driving control signal of the target vehicle based on the target probability distribution.

[0012] Optionally, in one embodiment of the present application, the use of the characteristics of the target hierarchy to determine the target probability distribution of the target vehicle performing each action includes: determining the action space dimension of the target vehicle based on the characteristics of the target hierarchy; and generating the target probability distribution of the target vehicle performing each action using the action space dimension.

[0013] The second aspect of the present application provides an end-to-end autonomous driving control device based on a bird's-eye view and deep reinforcement learning, including: an acquisition module, which is used to collect multi-perspective image data of the environment in which the target vehicle is located when it is detected that the target vehicle is in an autonomous driving condition; an acquisition module, which is used to convert the multi-perspective image data into a bird's-eye view representation based on a bird's-eye view feature extraction network to generate a bird's-eye view feature map of the environment, and classify each pixel in the bird's-eye view feature map to obtain a semantic segmentation map of the environment; a control module, which is used to input the bird's-eye view feature map and the semantic segmentation map into a target deep reinforcement learning network to output an autonomous driving control signal of the target vehicle, and control the target vehicle to travel according to the autonomous driving control signal.

[0014] Optionally, in one embodiment of the present application, the acquisition module includes: a first extraction unit, used to extract target features of the multi-view image data to generate a target feature map; a first acquisition unit, used to calculate the depth distribution of each pixel in the target feature map to obtain the target depth value corresponding to each pixel in the target feature map; a first determination unit, used to determine the probability distribution of each depth value, and generate a depth prediction map of the target feature map according to the probability distribution; a generation unit, used to determine the 3D coordinates of each pixel based on the depth prediction map, and generate a 3D point cloud using the 3D coordinates of each pixel; a second acquisition unit, used to project the 3D point cloud into a bird's-eye view grid, so that the multi-view image data is converted into the bird's-eye view representation.

[0015] Optionally, in one embodiment of the present application, the acquisition module includes: a second extraction unit, used to extract abstract features of the bird's-eye view feature map using an encoder in a pre-trained target network; and a third acquisition unit, used to classify each pixel in the bird's-eye view feature map based on a decoder in the pre-trained target network and the abstract features to obtain a semantic segmentation map of the environment.

[0016] Optionally, in one embodiment of the present application, the control module includes: a fourth acquisition unit, used to preliminarily process the bird's-eye view feature map, the semantic segmentation map and the state information of the target vehicle to obtain target level features; a second determination unit, used to determine the target probability distribution of the target vehicle performing each action using the target level features, and determine the automatic driving control signal of the target vehicle based on the target probability distribution.

[0017] Optionally, in one embodiment of the present application, the second determination unit includes: a determination subunit, used to determine the action space dimension of the target vehicle according to the characteristics of the target hierarchy; and a generation subunit, used to generate a target probability distribution of the target vehicle performing each action using the action space dimension.

[0018] An embodiment of the third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning as described in the above embodiments.

[0019] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning.

[0020] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned end-to-end autonomous driving control method based on bird's-eye view and deep reinforcement learning.

[0021] In the embodiment of the present application, when the target vehicle is detected to be in an automatic driving condition, the multi-view image data of the target vehicle's environment can be converted into a bird's-eye view representation, and each pixel in the generated bird's-eye view feature map of the environment can be classified to obtain a semantic segmentation map of the environment, and then the bird's-eye view feature map and the semantic segmentation map are input into the target deep reinforcement learning network to output the automatic driving control signal of the target vehicle, thereby controlling the driving of the target vehicle. Thus, the problem of high computational complexity of the automatic driving control system in the related art, low ability of the system to adapt to complex environments, insufficient interpretability, reduced overall system performance, inability to meet the user's driving needs, and reduced user driving experience is solved.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart of an end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning according to an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of an end-to-end autonomous driving control based on a bird's-eye view and deep reinforcement learning provided according to an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of an end-to-end autonomous driving control device based on a bird's-eye view and deep reinforcement learning according to an embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] The following describes an end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning in an embodiment of the present application with reference to the accompanying drawings. In view of the limitations of the autonomous driving control system in the related art mentioned in the background technology center, and the multiple independent modules increase the computational complexity of the system, the system has a low ability to adapt to complex environments, insufficient interpretability, and reduces the overall performance of the system, and cannot meet the driving needs of users. The present application provides an end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning. In this method, when it is detected that the target vehicle is in an autonomous driving condition, the multi-perspective image data of the target vehicle's environment can be converted into a bird's-eye view representation, and each pixel in the generated bird's-eye view feature map of the environment is classified to obtain a semantic segmentation map of the environment, and then the bird's-eye view feature map and the semantic segmentation map are input into the target deep reinforcement learning network to output the autonomous driving control signal of the target vehicle, thereby controlling the driving of the target vehicle. As a result, the problem that the computational complexity of the autonomous driving control system in the related art is high, the system has a low ability to adapt to complex environments, insufficient interpretability, reduces the overall performance of the system, cannot meet the driving needs of users, and reduces the driving experience of users is solved.

[0030] This application establishes a hierarchical decoupled end-to-end autonomous driving control system based on BEV feature extraction, which includes four modules: multi-sensor input module, BEV feature extraction network, deep reinforcement learning strategy network and semantic segmentation decoding module, as follows:

[0031] (1) Multi-sensor input module: This module consists of multiple RGB cameras surrounding the vehicle, which can capture environmental images in different directions. This module provides real-time environmental perception data, ensures the comprehensiveness and accuracy of information, and provides a basis for subsequent feature extraction and decision-making.

[0032] (2) BEV feature extraction network: The Lift-Splat architecture is used to convert multi-view image data into a unified BEV representation. The network first lifts the 2D image into a 3D point cloud by predicting the depth distribution of each pixel (Lift step), and then projects these point clouds onto the BEV grid (Splat step). This feature extraction network can effectively integrate the view information from different cameras, reduce information errors, and improve the accuracy of environmental understanding.

[0033] (3) Deep reinforcement learning strategy network: Combined with the Actor-Critic architecture, it inputs high-dimensional features extracted from BEV features and outputs control signals (such as acceleration, braking, and steering angles). Through the DRL method, the system can automatically learn the optimal driving strategy in a dynamic environment, adjust vehicle behavior in real time, and enhance decision-making capabilities.

[0034] (4) Semantic segmentation decoding module: Use a pre-trained network (such as ResNet) to decode BEV features, generate semantic segmentation results, and provide interpretable information about objects in the environment. This module converts high-dimensional features into easy-to-understand environmental information, improves the interpretability of the autonomous driving system, and makes the vehicle's decision-making process more transparent.

[0035] He2eAD (Hierarchical End-to-End Autonomous Driving) achieves an efficient and explainable end-to-end autonomous driving solution by combining BEV (Bird's Eye View) feature extraction with DRL (Deep Reinforcement Learning). Unlike traditional modular approaches, the system uses inputs from multiple cameras to build a BEV representation of the vehicle's surroundings, integrating perception, prediction, planning, and control into a unified framework. By directly mapping sensor inputs to control outputs, the information processing process is simplified and error propagation is reduced. Compared with existing autonomous driving control systems, the hierarchical decoupled end-to-end autonomous driving control system can capture environmental information more comprehensively through BEV representation, reduce collision rates, and improve driving safety. At the same time, the decision-making ability of the control system is enhanced: integrating BEV feature extraction and DRL makes the system's decision-making in complex urban environments more accurate and flexible.

[0036] Specifically, Figure 1 A flowchart of an end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning provided in an embodiment of the present application.

[0037] like Figure 1 As shown, the end-to-end autonomous driving control method based on bird's-eye view and deep reinforcement learning includes the following steps:

[0038] In step S101, when it is detected that the target vehicle is in an automatic driving condition, multi-view image data of the environment in which the target vehicle is located is collected.

[0039] It can be understood that the embodiments of the present application can control the vehicle to enter the autonomous driving condition when it is detected that the target vehicle is in the autonomous driving condition. For example, when the current driving area of ​​the vehicle meets the autonomous driving conditions, the vehicle can be controlled to enter the autonomous driving condition, or the user can control the vehicle to enter the autonomous driving condition through physical buttons, and when the vehicle enters the autonomous driving condition, the embodiments of the present application can collect multi-angle environmental information of the vehicle's environment, that is, multi-view image data, through the multi-sensor input module in the above-mentioned system, and can also calibrate, denoise and enhance the collected multi-view image data, thereby effectively improving the quality of the image data.

[0040] In step S102, based on the bird's-eye view feature extraction network, the multi-view image data is converted into a bird's-eye view representation to generate a bird's-eye view feature map of the environment, and each pixel in the bird's-eye view feature map is classified to obtain a semantic segmentation map of the environment.

[0041] It can be understood that the embodiments of the present application can convert multi-view image data into a bird's-eye view representation through the BEV feature extraction network in the above-mentioned system. For example, the 2D image is first upgraded to a 3D point cloud, and then these 3D point clouds are projected onto the BEV grid to generate a bird's-eye view feature map of the environment, which can effectively integrate the perspective information from different cameras, reduce information errors, and improve the accuracy of environmental understanding; in addition, the semantic segmentation decoding module in the above-mentioned system can also be used to classify each pixel in the bird's-eye view feature map to obtain a semantic segmentation map of the environment, effectively providing interpretable information of objects in the environment.

[0042] Among them, in one embodiment of the present application, based on a bird's-eye view feature extraction network, multi-view image data is converted into a bird's-eye view representation, including: extracting target features of the multi-view image data to generate a target feature map; calculating the depth distribution of each pixel in the target feature map to obtain a target depth value corresponding to each pixel in the target feature map; determining the probability distribution of each depth value, and generating a depth prediction map of the target feature map based on the probability distribution; based on the depth prediction map, determining the 3D coordinates of each pixel, and generating a 3D point cloud using the 3D coordinates of each pixel; projecting the 3D point cloud into a bird's-eye view grid, so that the multi-view image data is converted into a bird's-eye view representation.

[0043] In the actual implementation process, Figure 2 As shown, the embodiment of the present application can build a BEV feature extraction network, wherein the BEV feature extraction network in the above system is designed to convert multi-view image data into a bird's-eye view (BEV) representation to achieve a comprehensive understanding of the surrounding environment, and the BEV feature extraction network consists of two main steps: the Lift step and the Splat step.

[0044] Specifically, the Lift step aims to convert 2D multi-view image data into 3D point clouds, and predict the depth information of each pixel in the target feature map extracted based on the multi-view image data, where the input is an image captured by a surround view camera (such as front view, rear view, and side view), usually in RGB format, and multiple convolutional layers are used to extract image features. The ReLU activation function is used to increase the nonlinear expression capability. Through a series of convolution operations, a target feature map is generated to capture image information at different levels. For each pixel in the target feature map, the depth distribution is calculated through the fully connected layer to generate the depth value corresponding to each pixel. The Softmax layer is used to output the probability distribution of each depth value to form a depth prediction map, where the output generates the 3D coordinates (x, y, z) of each pixel to form a 3D point cloud.

[0045] The Splat step aims to project the generated 3D point cloud into the BEV grid to form a unified BEV representation, wherein the size and resolution of the BEV grid are determined, usually centered on the vehicle and covering a certain area. Each point in the 3D point cloud is projected onto the BEV plane using the intrinsic and extrinsic parameters of the camera, and the coordinates of the 3D points are converted into the BEV coordinate system through the transformation matrix. Sum Pooling is used to map 3D points from different perspectives to the corresponding positions of the BEV grid. For multiple points projected to the same grid cell, the features can be integrated by summing or averaging to generate the final BEV feature map, thereby effectively integrating data from multiple sensors, providing a unified and high-dimensional environmental representation, and reducing information loss and error propagation.

[0046] Optionally, in one embodiment of the present application, each pixel in the bird's-eye view feature map is classified to obtain a semantic segmentation map of the environment, including: using an encoder in a pre-trained target network to extract abstract features of the bird's-eye view feature map; based on a decoder and abstract features in the pre-trained target network, each pixel in the bird's-eye view feature map is classified to obtain a semantic segmentation map of the environment.

[0047] As a possible way to achieve this, Figure 2As shown, the embodiment of the present application can build a semantic segmentation network. The goal of the semantic segmentation module in the above system is to classify each pixel in the input image (or bird's-eye view BEV feature) into a specific category (such as vehicles, pedestrians, roads, etc.). The semantic segmentation module usually adopts a convolutional neural network architecture, which can effectively capture the spatial features and contextual information in the image. The input layer receives the feature map from the BEV feature extraction module or directly inputs the image. The input size is usually a fixed size (such as 256x256 or 512x512). The encoder part consists of a convolutional layer and a pooling layer. The convolutional layer consists of multiple convolutional layers. The ReLU activation function is used to extract features. Each convolutional layer is followed by Batch Normalization to accelerate training and improve stability; the pooling layer uses the maximum pooling layer (MaxPooling) to gradually reduce the spatial dimension of the feature map and extract higher-level features. Generally, a 2×2 pooling window is used with a stride of 2. The feature decoding and visualization part inputs the BEV features into the semantic segmentation network, generates a semantic segmentation map of the environment, implements visualization tools, and displays the segmentation results to enhance interpretability.

[0048] In step S103, the bird's-eye view feature map and the semantic segmentation map are input into the target deep reinforcement learning network to output the automatic driving control signal of the target vehicle, and the target vehicle is controlled to travel according to the automatic driving control signal.

[0049] It can be understood that the embodiments of the present application can input the bird's-eye view feature map and the semantic segmentation map into the target deep reinforcement learning network, such as the DRL (Deep Reinforcement Learning) policy network, to output the target vehicle's autonomous driving control signals, such as acceleration, braking and steering angles, and control the target vehicle's driving according to the autonomous driving control signals, so that the embodiments of the present application can automatically learn the optimal driving strategy in a dynamic environment, adjust the vehicle behavior in real time, and enhance decision-making capabilities.

[0050] Among them, in one embodiment of the present application, the bird's-eye view feature map and the semantic segmentation map are input into the target deep reinforcement learning network, and the automatic driving control signal of the target vehicle is output, including: preliminarily processing the bird's-eye view feature map, the semantic segmentation map and the state information of the target vehicle to obtain the characteristics of the target level; using the characteristics of the target level to determine the target probability distribution of each action performed by the target vehicle, and determining the automatic driving control signal of the target vehicle according to the target probability distribution.

[0051] In the embodiment of the present application, the target level features are higher level features extracted in the embodiment of the present application.

[0052] In some embodiments, the embodiments of the present application can realize the construction of a control strategy network based on deep reinforcement learning. For example, the DRL-based strategy network in the above system aims to learn the optimal control strategy from the environmental state. The network usually adopts the Actor-Critic architecture to convert the state input of the environment into an action strategy and a state value estimate. The input features of the input layer come from the feature map of the BEV feature extraction module (such as BEV representation), the vehicle's state information (such as speed, acceleration, direction, etc.), and environmental information (such as road conditions, traffic signals, etc.). The input features are preliminarily processed through a convolutional layer or a fully connected layer to extract higher-level features, and the higher-level features are used to determine the probability distribution of the target vehicle performing each action, and the vehicle's automatic driving control signal is determined according to the target probability distribution, which effectively improves the stability of the vehicle and improves the user's trust.

[0053] Optionally, in one embodiment of the present application, the target probability distribution of the target vehicle performing each action is determined using the characteristics of the target hierarchy, including: determining the action space dimension of the target vehicle based on the characteristics of the target hierarchy; and generating the target probability distribution of the target vehicle performing each action using the action space dimension.

[0054] In some embodiments, the embodiments of the present application can use the Actor network to generate an action strategy under a given state and output the probability distribution of each action, wherein a convolutional layer is used to extract spatial features from the image, a fully connected layer uses multiple fully connected layers to learn a high-dimensional representation from the input features, and the output layer uses a Softmax activation function to output the probability distribution of the action strategy according to the dimension of the action space. In addition, for continuous action space, a Gaussian distribution can be used to output the mean and standard deviation of the action, which can flexibly handle various types of continuous actions and improve the stability and reliability of the vehicle.

[0055] In addition, in the embodiments of the present application, model training is a crucial step in deep reinforcement learning, involving data preparation, model construction, training configuration, and subsequent evaluation and optimization. The following is a detailed training process of the policy network and semantic segmentation module based on deep reinforcement learning:

[0056] Among them, during the data preparation and collection process, RGB image data needs to be collected in the simulation environment CARLA to ensure that the data set covers a variety of driving scenarios, such as urban roads, highways, and rural roads. And apply data enhancement techniques (such as rotation, scaling, translation, and color transformation) to expand the training set and improve the robustness and generalization ability of the model. In deep reinforcement learning, the corresponding actions and rewards are recorded for each state, and a suitable framework is selected, such as using a deep learning framework (such as TensorFlow or PyTorch) to build a semantic segmentation network and a policy network; define the model hierarchy, and define the input, output, and activation function of each layer according to the above design; set the initial learning rate, usually 0.001 or 0.0001, and may use a learning rate decay strategy; set the batch size of training (such as 16, 32) to balance the training speed and memory usage; set the number of training rounds, usually adjusted according to the performance of the validation set, and use the cross entropy loss function to evaluate the difference between the predicted category distribution and the true label; set the loss function of the actor and critic, including the policy loss and value loss, and use the mean square error (MSE) to evaluate the state value.

[0057] It can be seen that the introduction of the semantic segmentation module in the embodiment of the present application enables the embodiment of the present application to provide visualization results of environmental understanding, improve the user's understanding and trust in autonomous driving decisions, and through end-to-end design, the embodiment of the present application achieves a seamless connection between perception, decision-making and control, greatly simplifies the calculation process, and improves real-time response capabilities.

[0058] According to the end-to-end autonomous driving control method based on bird's-eye view and deep reinforcement learning proposed in the embodiment of the present application, when it is detected that the target vehicle is in an autonomous driving condition, the multi-perspective image data of the target vehicle's environment can be converted into a bird's-eye view representation, and each pixel in the generated bird's-eye view feature map of the environment can be classified to obtain a semantic segmentation map of the environment, and then the bird's-eye view feature map and the semantic segmentation map are input into the target deep reinforcement learning network to output the target vehicle's autonomous driving control signal, thereby controlling the target vehicle's travel. This solves the problem that the autonomous driving control system in the related art has high computational complexity, low ability of the system to adapt to complex environments, insufficient interpretability, reduced overall system performance, inability to meet the user's driving needs, and reduced user driving experience

[0059] Secondly, the end-to-end autonomous driving control device based on bird's-eye view and deep reinforcement learning proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0060] Figure 3 It is a block diagram of an end-to-end autonomous driving control device based on a bird's-eye view and deep reinforcement learning according to an embodiment of the present application.

[0061] like Figure 3 As shown, the end-to-end autonomous driving control device 10 based on bird's-eye view and deep reinforcement learning includes: a collection module 100, an acquisition module 200 and a control module 300.

[0062] Specifically, the acquisition module 100 is used to acquire multi-view image data of the environment in which the target vehicle is located when it is detected that the target vehicle is in an automatic driving condition.

[0063] The acquisition module 200 is used to convert the multi-view image data into a bird's-eye view representation based on a bird's-eye view feature extraction network to generate a bird's-eye view feature map of the environment, and classify each pixel in the bird's-eye view feature map to obtain a semantic segmentation map of the environment.

[0064] The control module 300 is used to input the bird's-eye view feature map and the semantic segmentation map into the target deep reinforcement learning network to output the automatic driving control signal of the target vehicle, and control the driving of the target vehicle according to the automatic driving control signal.

[0065] Optionally, in one embodiment of the present application, the acquisition module 200 includes: a first extraction unit, a first acquisition unit, a first determination unit, a generation unit and a second acquisition unit.

[0066] Wherein, the first extraction unit is used to extract target features of multi-view image data to generate a target feature map.

[0067] The first acquisition unit is used to calculate the depth distribution of each pixel in the target feature map to obtain a target depth value corresponding to each pixel in the target feature map.

[0068] The first determination unit is used to determine the probability distribution of each depth value and generate a depth prediction map of the target feature map according to the probability distribution.

[0069] The generation unit is used to determine the 3D coordinates of each pixel based on the depth prediction map, and generate a 3D point cloud using the 3D coordinates of each pixel.

[0070] The second acquisition unit is used to project the 3D point cloud into a bird's-eye view grid, so that the multi-view image data is converted into a bird's-eye view representation.

[0071] Optionally, in one embodiment of the present application, the acquisition module includes: a second extraction unit and a third acquisition unit.

[0072] The second extraction unit is used to extract abstract features of the bird's-eye view feature map using an encoder in a pre-trained target network.

[0073] The third acquisition unit is used to classify each pixel in the bird's-eye view feature map based on the decoder and abstract features in the pre-trained target network to obtain a semantic segmentation map of the environment.

[0074] Optionally, in one embodiment of the present application, the control module includes: a fourth acquisition unit and a second determination unit.

[0075] Among them, the fourth acquisition unit is used to preliminarily process the bird's-eye view feature map, the semantic segmentation map and the state information of the target vehicle to obtain the features of the target level.

[0076] The second determination unit is used to determine the target probability distribution of each action performed by the target vehicle using the characteristics of the target hierarchy, and determine the automatic driving control signal of the target vehicle according to the target probability distribution.

[0077] Optionally, in one embodiment of the present application, the second determining unit includes: a determining subunit and a generating subunit.

[0078] Among them, the determination subunit is used to determine the action space dimension of the target vehicle according to the characteristics of the target level.

[0079] The generation subunit is used to generate the target probability distribution of each action performed by the target vehicle using the action space dimension.

[0080] It should be noted that the aforementioned explanation of the embodiment of the end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning is also applicable to the end-to-end autonomous driving control device based on a bird's-eye view and deep reinforcement learning of this embodiment, and will not be repeated here.

[0081] According to the end-to-end autonomous driving control device based on bird's-eye view and deep reinforcement learning proposed in the embodiment of the present application, when it is detected that the target vehicle is in an autonomous driving condition, the multi-perspective image data of the target vehicle's environment can be converted into a bird's-eye view representation, and each pixel in the generated bird's-eye view feature map of the environment can be classified to obtain a semantic segmentation map of the environment, and then the bird's-eye view feature map and the semantic segmentation map are input into the target deep reinforcement learning network to output the target vehicle's autonomous driving control signal, thereby controlling the target vehicle's travel. This solves the problem that the autonomous driving control system in the related art has high computational complexity, low ability of the system to adapt to complex environments, insufficient interpretability, reduced overall system performance, inability to meet the user's driving needs, and reduced user driving experience

[0082] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0083] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0084] When the processor 402 executes the program, the end-to-end autonomous driving control method based on the bird's-eye view and deep reinforcement learning provided in the above embodiment is implemented.

[0085] Furthermore, the electronic device further comprises:

[0086] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0087] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0088] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0089] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0090] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0091] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0092] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning.

[0093] This embodiment also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned end-to-end autonomous driving control method based on bird's-eye view and deep reinforcement learning.

[0094] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0095] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0096] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0097] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0098] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0099] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0101] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An end-to-end autonomous driving control method based on bird's-eye view and deep reinforcement learning, characterized in that: The following steps are involved: When it is detected that the target vehicle is in an automatic driving state, collecting multi-view image data of the environment in which the target vehicle is located; Based on a bird's-eye view feature extraction network, converting the multi-view image data into a bird's-eye view representation to generate a bird's-eye view feature map of the environment, and classifying each pixel in the bird's-eye view feature map to obtain a semantic segmentation map of the environment; The bird's-eye view feature map and the semantic segmentation map are input into a target deep reinforcement learning network to output an autonomous driving control signal for the target vehicle, and the target vehicle is controlled to travel according to the autonomous driving control signal.

2. The method according to claim 1, characterized in that The step of converting the multi-view image data into a bird's-eye view representation based on a bird's-eye view feature extraction network comprises: Extracting target features of the multi-view image data to generate a target feature map; Calculate the depth distribution of each pixel in the target feature map to obtain a target depth value corresponding to each pixel in the target feature map; Determine a probability distribution of each depth value, and generate a depth prediction map of the target feature map according to the probability distribution; Based on the depth prediction map, determine the 3D coordinates of each pixel, and generate a 3D point cloud using the 3D coordinates of each pixel; The 3D point cloud is projected into a bird's-eye view grid, so that the multi-view image data is converted into the bird's-eye view representation.

3. The method according to claim 1, characterized in that The step of classifying each pixel in the bird's-eye view feature map to obtain a semantic segmentation map of the environment includes: extracting abstract features of the bird's-eye view feature map using an encoder in a pre-trained target network; Based on the decoder in the pre-trained target network and the abstract features, each pixel in the bird's-eye view feature map is classified to obtain a semantic segmentation map of the environment.

4. The method according to claim 1, characterized in that The step of inputting the bird's-eye view feature map and the semantic segmentation map into a target deep reinforcement learning network and outputting an automatic driving control signal for the target vehicle includes: Preliminarily processing the bird's-eye view feature map, the semantic segmentation map, and the state information of the target vehicle to obtain target-level features; The target probability distribution of each action performed by the target vehicle is determined using the characteristics of the target hierarchy, and the automatic driving control signal of the target vehicle is determined based on the target probability distribution.

5. The method according to claim 4, characterized in that The step of determining the target probability distribution of each action performed by the target vehicle by utilizing the features of the target hierarchy includes: Determining the action space dimension of the target vehicle according to the characteristics of the target level; The action space dimension is used to generate a target probability distribution of the target vehicle performing each action.

6. An end-to-end autonomous driving control device based on bird's-eye view and deep reinforcement learning, characterized in that: include: A collection module, used to collect multi-view image data of the environment in which the target vehicle is located when it is detected that the target vehicle is in an automatic driving condition; an acquisition module, configured to convert the multi-view image data into a bird's-eye view representation based on a bird's-eye view feature extraction network to generate a bird's-eye view feature map of the environment, and classify each pixel in the bird's-eye view feature map to obtain a semantic segmentation map of the environment; A control module is used to input the bird's-eye view feature map and the semantic segmentation map into a target deep reinforcement learning network to output an automatic driving control signal of the target vehicle, and control the driving of the target vehicle according to the automatic driving control signal.

7. The device according to claim 6, characterized in that The acquisition module comprises: A first extraction unit, configured to extract target features of the multi-view image data to generate a target feature map; A first acquisition unit, configured to calculate a depth distribution of each pixel in the target feature map to obtain a target depth value corresponding to each pixel in the target feature map; A first determining unit, configured to determine a probability distribution of each depth value, and generate a depth prediction map of the target feature map according to the probability distribution; A generating unit, configured to determine the 3D coordinates of each pixel based on the depth prediction map, and generate a 3D point cloud using the 3D coordinates of each pixel; The second acquisition unit is used to project the 3D point cloud into a bird's-eye view grid, so that the multi-view image data is converted into the bird's-eye view representation.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed by a processor to implement the end-to-end autonomous driving control method based on a bird's-eye view and deep reinforcement learning as described in any one of claims 1 to 5.

Images