Automatic driving control method of vehicle, electronic equipment and storage medium

Abstract

The invention relates to an automatic driving control method of a vehicle, electronic equipment and a storage medium. The method comprises the steps of obtaining multi-view image data and point cloud data in a current environment where a target vehicle is located; performing feature fusion on the multi-view image data and the point cloud data to obtain multi-modal feature data; detecting a lane line in the current environment based on the multi-modal feature data, and predicting a movement track of an obstacle in a target lane corresponding to the lane line; and generating a vehicle control instruction based on the motion trail, and performing automatic driving control on the target vehicle based on the vehicle control instruction. The multi-view image data and the point cloud data acquired by the target vehicle are fused, and the obstacle is detected based on the fused data, so that the dependency relationship between different sensors can be effectively captured, the movement track of the obstacle in the lane line can be accurately predicted, and the target vehicle can be effectively controlled.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to an autonomous driving control method for a vehicle, an electronic device, and a storage medium. Background Art

[0002] With the continuous development of technology, autonomous driving technology has been developed to a great extent. It plays an important role in logistics, public transportation, smart cities and other fields, and promotes the intelligent development of various fields of society. In recent years, due to the rapid development of artificial intelligence technology, end-to-end autonomous driving has gradually become a research hotspot in the field of intelligent transportation.

[0003] In related technologies, various sensors installed on the vehicle collect image and other data, and use convolutional neural network methods to extract features from the data collected by the sensors. However, related technologies cannot effectively capture the dependencies between sensors, resulting in poor robustness of the perception model in different scenarios and inability to effectively control the vehicle. Summary of the invention

[0004] The present disclosure provides an automatic driving control method for a vehicle, an electronic device, and a storage medium.

[0005] According to a first aspect of the present disclosure, there is provided a method for controlling an automatic driving of a vehicle, the method comprising:

[0006] Acquire multi-view image data and point cloud data of the current environment where the target vehicle is located;

[0007] Performing feature fusion on the multi-view image data and the point cloud data to obtain multimodal feature data;

[0008] Based on the multimodal feature data, detect lane lines in the current environment, and predict the motion trajectory of obstacles in the target lane corresponding to the lane lines;

[0009] A vehicle control instruction is generated based on the motion trajectory, and automatic driving control is performed on the target vehicle based on the vehicle control instruction.

[0010] Optionally, before acquiring the multi-view image data and point cloud data in the current environment where the target vehicle is located, the method further includes:

[0011] Collecting original two-dimensional multi-view image data in the current environment by a multi-view image acquisition device in the target vehicle;

[0012] Generate two-dimensional multi-view image data with multiple levels of different resolutions based on the original two-dimensional multi-view image data;

[0013] The two-dimensional multi-view image data with multiple levels of different resolutions are feature-fused to obtain two-dimensional multi-view image fusion data, and the two-dimensional multi-view image fusion data is converted into three-dimensional image data to obtain multi-view image data in the current environment.

[0014] Optionally, converting the two-dimensional multi-view image fusion data into three-dimensional image data includes:

[0015] Downsampling the two-dimensional multi-view image data with multiple levels of different resolutions to obtain image feature data with multiple levels of different sampling rates;

[0016] Projecting the multi-level image feature data with different sampling rates into a three-dimensional voxel space respectively to obtain multi-level dense voxel feature data;

[0017] Performing pooling operations on the multi-level dense voxel feature data in the vertical direction to obtain multi-level three-dimensional bird's-eye view feature data;

[0018] A connection operation is performed on the multi-level three-dimensional bird's-eye view feature data to obtain comprehensive three-dimensional bird's-eye view feature data, and the comprehensive three-dimensional bird's-eye view feature data is used as three-dimensional image data.

[0019] Optionally, before acquiring the multi-view image data and point cloud data in the current environment where the target vehicle is located, the method further includes:

[0020] Collecting original three-dimensional point cloud data in the current environment by using a laser radar in the target vehicle;

[0021] A three-dimensional point cloud feature map is generated based on the original three-dimensional point cloud data to obtain point cloud data in the current environment.

[0022] Optionally, generating a three-dimensional point cloud feature map based on the original three-dimensional point cloud data includes:

[0023] Dividing the original three-dimensional point cloud data into a three-dimensional voxel grid based on a preset rule; the three-dimensional voxel grid includes a plurality of voxel units of the same size;

[0024] Discretizing the three-dimensional point cloud based on the three-dimensional voxel grid, and performing feature aggregation on the discretized three-dimensional voxel grid based on the voxel unit; wherein the feature aggregation is used to map points in the three-dimensional point cloud data to corresponding voxel units;

[0025] The three-dimensional voxel grid after the feature aggregation is projected onto a two-dimensional plane, and a three-dimensional point cloud feature map is generated based on the number of channels of the three-dimensional point cloud data and the projected three-dimensional voxel grid.

[0026] Optionally, projecting the three-dimensional voxel grid after the feature aggregation onto a two-dimensional plane comprises:

[0027] Respectively obtain the characteristic value of the voxel feature in the vertical direction of each voxel unit in the three-dimensional voxel grid after the feature aggregation;

[0028] Based on the eigenvalues, voxel features of each voxel unit having an eigenvalue greater than a threshold in a vertical direction are retained, and the processed three-dimensional voxel grid is projected onto a two-dimensional plane.

[0029] Optionally, the predicting a motion trajectory of an obstacle in a target lane corresponding to the lane line includes:

[0030] Obtaining the spatial position and motion state of the obstacle in the target lane corresponding to the lane line;

[0031] Based on the detected position information of the lane line, the spatial position and the motion state, a motion trajectory of the obstacle is predicted.

[0032] Optionally, generating a vehicle control instruction based on the motion trajectory includes:

[0033] Acquire a current first motion trajectory and driving state of the target vehicle, and predict a second motion trajectory of the target vehicle based on the first motion trajectory and the driving state;

[0034] In the case where the second motion trajectory overlaps with the motion trajectory of the obstacle, a vehicle control instruction is generated; wherein the vehicle control instruction is used to adjust the motion trajectory of the target vehicle.

[0035] According to a second aspect of the present disclosure, an electronic device is provided. The electronic device comprises: a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0036] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0037] The automatic driving control method, electronic device and storage medium of the vehicle provided by the embodiments of the present disclosure obtain multi-view image data and point cloud data in the current environment collected by the target vehicle, and perform feature fusion on the multi-view image data and point cloud data to obtain multi-modal feature data. Based on the multi-modal feature data, the lane lines in the current environment are detected, and the motion trajectory of the obstacles in the target lane corresponding to the lane lines is predicted. Vehicle control instructions are generated based on the motion trajectory, and the target vehicle is controlled based on the vehicle control instructions. In this way, by fusing the multi-view image data and point cloud data collected by the target vehicle, and detecting obstacles based on the fused data, the dependency between different sensors can be effectively captured, the motion trajectory of obstacles in the lane lines can be accurately predicted, and the target vehicle can be effectively controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0039] Figure 1 A flowchart of an automatic driving control method provided by an exemplary embodiment of the present disclosure;

[0040] Figure 2 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0041] Figure 3 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0043] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0044] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0045] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0046] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0047] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0048] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0049] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0050] As an important part of future intelligent transportation, autonomous driving plays an important role in logistics, public transportation, smart cities and other fields, and promotes the intelligent development of various fields of society. In recent years, with the rapid development of artificial intelligence technology, end-to-end autonomous driving has gradually become a research hotspot in the field of intelligent transportation. End-to-end autonomous driving aims to directly generate driving control commands from sensor inputs through an end-to-end model, so that vehicles can automatically navigate and make decisions in various complex road environments. This method avoids the complex staged processing flow (such as perception, decision-making, planning and control) in traditional autonomous driving systems, and can complete perception and decision-making in a unified network architecture, thereby achieving more efficient autonomous driving performance.

[0051] In the related technology, in end-to-end autonomous driving, for multimodal data from multiple sensors (such as cameras and lidars, etc.), the traditional convolutional neural network-based method is usually used for feature extraction of multimodal sensor data, which is difficult to fully and effectively capture the complex dependencies between sensors, resulting in poor robustness of the perception model in the face of different scenarios. Insufficient information sharing and multi-task collaborative processing between perception, prediction and planning tasks cannot fully utilize the interdependencies between tasks, resulting in poor overall performance and real-time performance of the system. In addition, if reinforcement learning methods are used alone for autonomous driving simulation, it is often difficult to handle the uncertainty problems in multimodal sensor inputs and dynamic environments, and efficient learning and optimization cannot be achieved, especially when faced with long-term training and high-dimensional state space, the convergence speed is slow and the stability is poor.

[0052] Therefore, in order to solve the above technical problems, the disclosed embodiment uses the image data obtained by the multi-view camera and the point cloud data obtained by the lidar as data input, and uses the multi-head self-attention mechanism of Transformer to perform feature fusion of multimodal data to capture the dependencies between sensors, which can greatly improve the robustness of perception. The embodiment passes the generated global features to the Transformer-based multi-task network framework to achieve information sharing between different tasks, significantly optimizing the effect of multi-task collaborative processing. And through the reinforcement learning strategy, the driving control commands in the form of probability distribution are generated, and the strategy is continuously optimized in combination with the designed reward function, so that the model can carefully model the relationship between each action and the environment, and provide richer feedback information than deterministic modeling, so as to effectively deal with the uncertainty problems in the planning process.

[0053] In the embodiment, the disclosed embodiment proposes an end-to-end autonomous driving method based on multimodal Transformer and reinforcement learning, which can effectively improve the perception robustness and decision stability of the autonomous driving system, which is mainly reflected in the following aspects:

[0054] (1) Efficient fusion of multimodal sensor data.

[0055] The embodiment fuses multi-view camera images and lidar point cloud features through the Transformer's multi-head self-attention mechanism (Multi-Head Attention), which can effectively capture the complex dependencies between different sensors and generate accurate environmental representation, thereby significantly improving the robustness and accuracy of perception.

[0056] Among them, the multi-head self-attention mechanism is a variant developed on the basis of the self-attention mechanism, which aims to enhance the expressiveness and generalization ability of the model. By using multiple independent attention heads, the attention weights can be calculated separately, and their results can be concatenated or weighted summed to obtain a richer representation.

[0057] (2) Multi-task collaborative processing.

[0058] The embodiment adopts a Transformer-based multi-task network framework to enable tasks such as target detection, target tracking, and scene modeling to share information, thereby enhancing the effect of multi-task collaborative processing.

[0059] Among them, Transformer is a deep learning model based on the attention mechanism, including an encoder and a decoder. It uses the self-attention mechanism to capture the relationship between different positions in the input sequence to process sequence data. In addition, the core of the Transformer model is the self-attention mechanism, which allows the model to weight the information at different positions when processing the input sequence, rather than relying solely on the position order of the sequence. Each layer consists of two sublayers: a multi-head self-attention layer and a feedforward neural network layer. The self-attention layer interacts each position of the input sequence with other positions by calculating the attention weights, thereby capturing important contextual information. The feedforward neural network layer independently maps and transforms the features of each position.

[0060] (3) Path planning and decision optimization based on reinforcement learning.

[0061] The disclosed embodiment uses the PPO (Proximal Policy Optimization) algorithm to train the policy network to generate driving control commands based on perception and prediction results. Through the training of the reinforcement learning model, the system can model the complex relationship between different actions and environmental states, maximize cumulative rewards, optimize path planning and action decisions, and effectively deal with common uncertainty problems in autonomous driving.

[0062] The present disclosure first provides a method for controlling an automatic driving of a vehicle. Figure 1As shown, the method may include the following steps:

[0063] In step S110 , multi-view image data and point cloud data in the current environment where the target vehicle is located are obtained.

[0064] In an embodiment, the multi-view image data and point cloud data may be image data collected by an image acquisition device on the target vehicle and point cloud data collected by a laser radar, respectively, or may be data obtained by processing image data collected by an image acquisition device and point cloud data collected by a laser radar.

[0065] In step S120, feature fusion is performed on the multi-view image data and the point cloud data to obtain multimodal feature data.

[0066] In the embodiment, in order to effectively capture the complex dependencies between sensors and improve the robustness of the perception model in the face of different scenarios, the embodiment can perform feature fusion on multi-view camera image data and point cloud data through the multi-head self-attention mechanism of Transformer, which can effectively capture the complex dependencies between different sensors and generate accurate environmental representation, thereby significantly improving the robustness and accuracy of perception.

[0067] Specifically, in the embodiment, when performing multimodal data feature fusion, the multimodal sensor features (multi-view image data and point cloud data) can be integrated into a unified representation for subsequent perception and prediction tasks. t = {I t ,P t}After the attention layer is processed, the fusion feature is generated to generate the multimodal fusion feature Z t , see formula (1). Where, I t Represents multi-view image data, P t Represents point cloud data.

[0068] Z t = TransformerEncoder(X t )(1) The calculation formula of the attention layer is:

[0069]

[0070] Where Q, K, and V represent query, key, and value vectors respectively. k Indicates the dimension of the key vector.

[0071] The resulting environmental characterization Z t The combined data features from different sensors will be used in subsequent perception and prediction tasks, thereby improving the effect of multi-task collaborative processing.

[0072] In step S130, based on the multimodal feature data, the lane lines in the current environment are detected, and the movement trajectory of the obstacles in the target lane corresponding to the lane lines is predicted.

[0073] In an embodiment, since the multimodal data obtained after feature fusion includes fused multi-view image data and point cloud data, the lane lines in the current environment can be detected based on the multi-view image data fused in the multimodal feature data, and the obstacles in the current environment can be detected and tracked based on the point cloud data fused in the multimodal feature data, and the motion trajectory of the obstacle in the target lane corresponding to the lane line can be predicted.

[0074] Therefore, the embodiment of the present disclosure adopts a Transformer-based multi-task network framework, which enables tasks such as target detection, target tracking, and scene modeling to share information, thereby enhancing the effect of multi-task collaborative processing.

[0075] Specifically, in the embodiment, when the perception module is used to extract features, the perception module can obtain information about the vehicle's surrounding environment, identify objects on the road, and determine their spatial positions and motion states. The perception module may include TrackFormer and MapFormer.

[0076] (1) TrackFormer introduces Detection query and Track query, which can perform target detection and multi-target tracking simultaneously. Detection query and Track query perceive the feature Z t To capture the abstract features of the target, at each time step, the Detection query is used to detect the target and output the feature F t 1 , and the Track query interacts with the previously recorded queries to aggregate the temporal information until the corresponding target disappears completely and outputs the feature F t 2 .

[0077] (2) MapFormer is used for online map segmentation. Map query is used to sparsely represent road elements and encode the location and structure information. It can be further decomposed into Thing query (including road elements, i.e. lane lines, boundaries and sidewalks) and Stuff query (only responsible for semantic elements, i.e. driving area). After Map query is updated by MapFormer, the location and structure information features F are obtained. t 3 .

[0078] In an embodiment, the prediction module can be used to predict the motion trajectory of obstacles in the target lane corresponding to the lane line. The prediction module infers the future behavior and motion trajectory of dynamic objects (such as other vehicles or pedestrians) based on the perceived data, helping the autonomous driving system to foresee potential collision or risk scenarios. The prediction module includes MotionFormer and OccFormer.

[0079] (1) MotionFormer uses information-rich dynamic agent features F t 1 、F t 2 and static map features as input F t 3 , predicting the future trajectories of all agents in the scene, so that the trajectories of multiple agents can be generated through a single forward pass, greatly saving the computational cost of aligning the entire scene with the coordinates of each agent. MotionFormer captures the “environmental features Z t + object + map” as the key and value. At the same time, an Ego-vehicle query is proposed to explicitly model the ego vehicle and interact with other objects, and finally output the trajectory information M t .

[0080] (2) The task of OccFormer is to discover how the occupancy grid map will change in the future. The occupancy grid map is a discretized BEV (bird’s eye view) representation, where each cell contains a confidence level indicating whether it is occupied, which is mostly used for obstacle avoidance. OccFormer uses feature Z t As query, trajectory information M t As key and value, and predict the future multi-step occupancy, and finally output the global scene occupancy state O t .

[0081] In step S140, a vehicle control instruction is generated based on the motion trajectory, and the target vehicle is automatically driven and controlled based on the vehicle control instruction.

[0082] In an embodiment, a vehicle control instruction may be generated according to the movement trajectory of obstacles in the current environment so as to control the safe operation of the vehicle.

[0083] The embodiment can strengthen learning strategy planning, generate actions a through reinforcement learning based on the results of perception and prediction t , realizing path planning and action decision.

[0084] (1) Definition of state, action, state transfer function and reward function: Define the state s of reinforcement learning t is the global state G t={M t ,O t}; action a t Generated by the policy network π, that is, a t =π(s t ); The goal of reinforcement learning is to maximize the cumulative reward where r t is the immediate reward, and γ∈[0,1] is the discount factor.

[0085] (2) Use the PPO algorithm to update the policy network:

[0086]

[0087] Among them, A π (s t , a t )=Q π (s t , a t )-V π (s t ) is the advantage function, which is used to evaluate the superiority of an action relative to other possible actions, Q p (s t ,a t ) is in state s t Next select action a t The action value function of V p (s t ) is the state value function, which means that in state s t The expected return under π θ (a t |s t ) means taking action a in state s t The probability of old represents the parameters of the old policy, and θ is the parameter of the new policy.

[0088] When executing and giving feedback, the action a generated by the policy network can be t , control the actual behavior of the vehicle, and feedback the execution results. Under the driving control command generated by the policy network, the vehicle performs the corresponding physical operations (such as accelerator, brake or steering, etc.) and feeds back the execution results to the model. Through environmental feedback, the system can dynamically update the global state, provide optimization basis for the next decision cycle, and form a closed-loop policy optimization process.

[0089] In the embodiment, the PPO algorithm can be used to train the policy network to generate driving control commands based on perception and prediction results. Through the training of the reinforcement learning model, the system can model the complex relationship between different actions and environmental states, maximize the cumulative reward, optimize path planning and action decision-making, and effectively deal with common uncertainty problems in autonomous driving.

[0090] The embodiment can generate driving control commands in the form of probability distribution through reinforcement learning strategy, and continuously optimize the strategy in combination with reward function, so that the model can model the relationship between each action and the environment, effectively deal with the uncertainty problem in the planning process, thereby improving the safety and reliability of vehicle driving.

[0091] The automatic driving control method for a vehicle provided by the embodiment of the present disclosure obtains multi-view image data and point cloud data in the current environment collected by the target vehicle, and performs feature fusion on the multi-view image data and point cloud data to obtain multi-modal feature data. Based on the multi-modal feature data, the lane lines in the current environment are detected, and the motion trajectory of the obstacles in the target lane corresponding to the lane lines is predicted. Vehicle control instructions are generated based on the motion trajectory, and the target vehicle is controlled based on the vehicle control instructions. In this way, by fusing the multi-view image data and point cloud data collected by the target vehicle, and detecting obstacles based on the fused data, the dependency between different sensors can be effectively captured, the motion trajectory of obstacles in the lane lines can be accurately predicted, and the target vehicle can be effectively controlled.

[0092] Based on the above embodiment, when the multi-view image data and the point cloud data are respectively processed image data collected by the image acquisition device and point cloud data collected by the laser radar, before step S110, the method may further include the following steps:

[0093] In step S101, original two-dimensional multi-view image data in the current environment is collected by a multi-view image collection device in a target vehicle.

[0094] In an embodiment, a plurality of image acquisition devices with different viewing angles, such as cameras or cameras, may be installed on the target vehicle, and original two-dimensional image data with different viewing angles may be acquired in real time through the image acquisition devices.

[0095] In step S102, two-dimensional multi-view image data with multiple levels of different resolutions are generated based on the original two-dimensional multi-view image data.

[0096] In step S103, feature fusion is performed on the two-dimensional multi-view image data of multiple levels of different resolutions to obtain two-dimensional multi-view image fusion data, and the two-dimensional multi-view image fusion data is converted into three-dimensional image data to obtain multi-view image data in the current environment.

[0097] In the embodiment, the original two-dimensional image data of different viewing angles can be collected by a multi-view camera, and the original two-dimensional image data can be processed.

[0098] ① Multi-view image encoder: Input N camera images into ResNet-50 to generate 4 levels of features F1, F2, F3, F4 with different resolutions; then perform feature fusion and downsampling through a 3-layer multi-scale FPN structure to obtain 3 levels of different downsampled comprehensive image features F 1 / 4 ,F 1 / 8 ,F 1 / 16 , and used as the input of Camera-to-BEV View Transform. Camera-to-BEV View Transform is the process of projecting image data features onto the BEV plane.

[0099] ② Camera-to-BEV View Transform first constructs a projection index table T from the 2D image space to the 3D voxel space, with the same dimension as the 3D voxel space. The present invention iterates each voxel unit in the voxel space and calculates the 2D pixel coordinates (x, y) corresponding to the 3D coordinates (X, Y, Z) through the projection matrix P:

[0100]

[0101] Therefore, in the embodiment, when converting the two-dimensional multi-view image fusion data into three-dimensional image data, the two-dimensional multi-view image data with multiple levels of different resolutions can be downsampled to obtain multiple levels of image feature data with different sampling rates. The multiple levels of image feature data with different sampling rates are projected into the three-dimensional voxel space to obtain multiple levels of dense voxel feature data. In this way, the multiple levels of dense voxel feature data are pooled in the vertical direction to obtain multiple levels of three-dimensional bird's-eye view feature data. The multiple levels of three-dimensional bird's-eye view feature data are connected to obtain comprehensive three-dimensional bird's-eye view feature data, and the comprehensive three-dimensional bird's-eye view feature data is used as the three-dimensional image data.

[0102] In the embodiment, if the 2D coordinate (x, y) obtained by projection can be found on the 2D image feature map (0≤x<W, 0≤y<H), where W represents the width of the image and H represents the height of the image, then fill it into the index table to establish a data-independent index mapping:

[0103] T[voxel]=(img,x,y)(5)

[0104] Next, the multi-view image feature F 14 ,F 18 ,F 1 / 16 According to the index table, they are projected into the 3D voxel space respectively. Each voxel unit is filled with the corresponding 2D image features according to the pre-calculated index table. The multi-view image features at different scales correspond to a different voxel space, generating multi-level dense voxel features.

[0105] Finally, a pooling operation is applied along the Z axis to transform the 3-level 4D voxel tensor V i =R H×W×Z×C , i=1 / 4,1 / 8,1 / 16 are converted to 3D BEV features I respectively i =R H×W×(ZC) , i = 1 / 4, 1 / 8, 1 / 16, and the three-level BEV features are connected, that is, superimposed together to obtain the final image 3D BEV feature I t . Where C is the number of channels.

[0106] Based on the above embodiment, in another embodiment provided by the present disclosure, before the above step S110, the method may further include the following steps:

[0107] In step S104, the original three-dimensional point cloud data in the current environment is collected by the laser radar in the target vehicle.

[0108] In an embodiment, a laser radar may be provided on the target vehicle, and the laser radar may be used to collect original three-dimensional point cloud data in the current environment of the target vehicle.

[0109] In step S105, a three-dimensional point cloud feature map is generated based on the original three-dimensional point cloud data to obtain point cloud data in the current environment.

[0110] In an embodiment, the original three-dimensional point cloud data can be divided into a three-dimensional voxel grid based on a preset rule, and the three-dimensional voxel grid includes a plurality of voxel units of the same size. The three-dimensional point cloud is discretized based on the three-dimensional voxel grid, and the discretized three-dimensional voxel grid is feature aggregated based on the voxel unit. Feature aggregation is used to map points in the three-dimensional point cloud data to corresponding voxel units. The three-dimensional voxel grid after feature aggregation is projected onto a two-dimensional plane, and a three-dimensional point cloud feature map is generated based on the number of channels of the three-dimensional point cloud data and the projected three-dimensional voxel grid.

[0111] When the three-dimensional voxel grid after feature aggregation is projected onto a two-dimensional plane, the characteristic value of the voxel feature of each voxel unit in the three-dimensional voxel grid after feature aggregation in the vertical direction can be obtained respectively. Based on the characteristic value, the voxel feature of each voxel unit with a characteristic value greater than a threshold in the vertical direction is retained, and the processed three-dimensional voxel grid is projected onto a two-dimensional plane.

[0112] Specifically, assuming that the voxel grid size in the 3D space is Δx, Δy, Δz, the embodiment first divides the entire continuous 3D point cloud space into a regular voxel grid, and all grid units are voxel units of equal size. Then, the point cloud data is discretized according to the voxel grid, and feature aggregation is performed according to the voxel unit. Each point (X, Y, Z) in the point cloud is mapped to the corresponding unit in the voxel grid:

[0113]

[0114] Among them, (i, j, k) is the coordinate of the point cloud data in the voxel grid, (X min ,Y min ,Z min ) is the minimum voxel coordinate. Then the pooling operation is used to aggregate the points belonging to the same voxel unit into a single feature vector.

[0115] After completing the voxelization of the point cloud features, the 3D voxel grid is projected onto the 2D plane. During projection, for each (i, j) grid cell, the maximum value of the voxel feature in the Z-axis direction is taken to retain the most important geometric information:

[0116]

[0117] After the voxel features are aggregated, a W×H×C point cloud feature map P is generated. t , W and H are the resolutions of the BEV space, and C is the number of feature channels.

[0118] In this way, through the above embodiments, the collected original multi-view image data and original point cloud data can be processed into data of the same dimension, namely three-dimensional multi-view image data and three-dimensional point cloud data, respectively, so as to facilitate the fusion of the processed multi-view image data and point cloud data.

[0119] Based on the above embodiment, in another embodiment provided by the present disclosure, when predicting the motion trajectory of the obstacle in the target lane corresponding to the lane line, the above step S130 may further specifically include the following steps:

[0120] In step S131, the spatial position and motion state of the obstacle in the target lane corresponding to the lane line are obtained.

[0121] In step S132, the movement trajectory of the obstacle is predicted based on the position information, spatial position and movement state of the detected lane line.

[0122] In an embodiment, the spatial position of each obstacle in the target lane can be obtained. Specifically, the point cloud data corresponding to each obstacle can be projected onto the ground, so that the coordinate position of each obstacle on the ground can be obtained, and the position information of the lane line of the target lane on the ground can be obtained. In this way, the relative position of each obstacle relative to the lane line of the target lane can be obtained, and combined with the motion state of each obstacle, such as motion direction and motion speed, the motion trajectory of each obstacle can be predicted.

[0123] For example, by acquiring the spatial position and motion state of the target obstacle in real time, if the spatial position of the target obstacle is getting closer to a certain lane line and has a certain motion speed, it is very likely that the target obstacle has a tendency to change lanes, so the motion trajectory of the target obstacle can be predicted to change lanes to another lane. If the distance between the target obstacle and the lane line remains within a certain distance range, it means that the target obstacle has no tendency to change lanes, and the motion trajectory of the target obstacle can be predicted to be traveling within the lane line.

[0124] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S140 may further include the following steps:

[0125] In step S141 , the current first motion trajectory and driving state of the target vehicle are acquired, and the second motion trajectory of the target vehicle is predicted based on the first motion trajectory and the driving state.

[0126] In step S142, when the second motion trajectory overlaps with the motion trajectory of the obstacle, a vehicle control instruction is generated, wherein the vehicle control instruction is used to adjust the motion trajectory of the target vehicle.

[0127] In the embodiment, the next second motion trajectory of the target vehicle can also be predicted based on the current first motion trajectory and driving state of the target vehicle. Since the target vehicle will basically drive according to the current motion trend when no new control instructions are implemented on the target vehicle, the next second motion trajectory of the target vehicle can be well predicted.

[0128] The embodiment predicts the movement trajectory of the obstacle and compares it with the second movement trajectory of the target vehicle. If the second movement trajectory overlaps with the movement trajectory of the obstacle, it indicates that the target vehicle may have a risk of collision with the obstacle. Therefore, a vehicle control instruction can be generated in time, and the vehicle can be controlled by the control instruction, such as changing lanes or slowing down. In this way, by changing the next movement trajectory of the target vehicle, the safety of the target vehicle can be improved.

[0129] It can be seen from the above embodiments that the disclosed embodiments introduce a multi-head self-attention mechanism to perform feature fusion of multi-view camera images and lidar point clouds, which can capture the dependencies between sensors and significantly improve the robustness of perception. In addition, the embodiments integrate the main tasks such as perception and prediction and their subtasks into an end-to-end network framework based on Transformer, so that information sharing can be achieved between different tasks, enhancing the effect of multi-task collaborative processing. This improvement not only improves the overall performance and safety of the autonomous driving system, but also enhances its adaptability to complex and changing traffic scenarios.

[0130] The embodiment generates driving control commands in the form of probability distribution through reinforcement learning strategy, and continuously optimizes the strategy in combination with the reward function, so that the model can model the relationship between each action and the environment, effectively deal with the uncertainty problem in the planning process, and thus provide a safer and more reliable autonomous driving solution.

[0131] Therefore, the embodiments of the present disclosure can achieve remarkable results in solving multi-source data fusion, unified processing of multiple tasks, and enhancing the reliability and stability of autonomous driving systems. It can provide strong technical support for the further development of autonomous driving systems and is expected to play an important role in the future field of intelligent transportation.

[0132] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0133] Figure 2 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0134] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.

[0135] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 3 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 3 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0136] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 3As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0138] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0139] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0140] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0141] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0143] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.

[0144] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0147] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0148] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0149] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for controlling an automatic driving of a vehicle, characterized in that: The method comprises: Acquire multi-view image data and point cloud data of the current environment where the target vehicle is located; Performing feature fusion on the multi-view image data and the point cloud data to obtain multimodal feature data; Based on the multimodal feature data, detect lane lines in the current environment, and predict the motion trajectory of obstacles in the target lane corresponding to the lane lines; A vehicle control instruction is generated based on the motion trajectory, and automatic driving control is performed on the target vehicle based on the vehicle control instruction.

2. The method according to claim 1, characterized in that Before acquiring the multi-view image data and point cloud data in the current environment where the target vehicle is located, the method further includes: Collecting original two-dimensional multi-view image data in the current environment by a multi-view image acquisition device in the target vehicle; Generate two-dimensional multi-view image data with multiple levels of different resolutions based on the original two-dimensional multi-view image data; The two-dimensional multi-view image data with multiple levels of different resolutions are feature-fused to obtain two-dimensional multi-view image fusion data, and the two-dimensional multi-view image fusion data is converted into three-dimensional image data to obtain multi-view image data in the current environment.

3. The method according to claim 2, characterized in that The converting the two-dimensional multi-view image fusion data into three-dimensional image data comprises: Downsampling the two-dimensional multi-view image data with multiple levels of different resolutions to obtain image feature data with multiple levels of different sampling rates; Projecting the multi-level image feature data with different sampling rates into a three-dimensional voxel space respectively to obtain multi-level dense voxel feature data; Performing pooling operations on the multi-level dense voxel feature data in the vertical direction to obtain multi-level three-dimensional bird's-eye view feature data; A connection operation is performed on the multi-level three-dimensional bird's-eye view feature data to obtain comprehensive three-dimensional bird's-eye view feature data, and the comprehensive three-dimensional bird's-eye view feature data is used as three-dimensional image data.

4. The method according to claim 1, characterized in that Before acquiring the multi-view image data and point cloud data in the current environment where the target vehicle is located, the method further includes: Collecting original three-dimensional point cloud data in the current environment by using a laser radar in the target vehicle; A three-dimensional point cloud feature map is generated based on the original three-dimensional point cloud data to obtain point cloud data in the current environment.

5. The method according to claim 4, characterized in that The generating of a three-dimensional point cloud feature map based on the original three-dimensional point cloud data comprises: Dividing the original three-dimensional point cloud data into a three-dimensional voxel grid based on a preset rule; the three-dimensional voxel grid includes a plurality of voxel units of the same size; Discretizing the three-dimensional point cloud based on the three-dimensional voxel grid, and performing feature aggregation on the discretized three-dimensional voxel grid based on the voxel unit; wherein the feature aggregation is used to map points in the three-dimensional point cloud data to corresponding voxel units; The three-dimensional voxel grid after the feature aggregation is projected onto a two-dimensional plane, and a three-dimensional point cloud feature map is generated based on the number of channels of the three-dimensional point cloud data and the projected three-dimensional voxel grid.

6. The method according to claim 5, characterized in that The projecting the three-dimensional voxel grid after the feature aggregation onto a two-dimensional plane comprises: Respectively obtain the characteristic value of the voxel feature in the vertical direction of each voxel unit in the three-dimensional voxel grid after the feature aggregation; Based on the eigenvalues, voxel features of each voxel unit having an eigenvalue greater than a threshold in a vertical direction are retained, and the processed three-dimensional voxel grid is projected onto a two-dimensional plane.

7. The method according to claim 1, characterized in that The predicting the motion trajectory of the obstacle in the target lane corresponding to the lane line includes: Obtaining the spatial position and motion state of the obstacle in the target lane corresponding to the lane line; Based on the detected position information of the lane line, the spatial position and the motion state, a motion trajectory of the obstacle is predicted.

8. The method according to any one of claims 1 to 7, characterized in that: The generating of the vehicle control instruction based on the motion trajectory comprises: Acquire a current first motion trajectory and driving state of the target vehicle, and predict a second motion trajectory of the target vehicle based on the first motion trajectory and the driving state; In the case where the second motion trajectory overlaps with the motion trajectory of the obstacle, a vehicle control instruction is generated; wherein the vehicle control instruction is used to adjust the motion trajectory of the target vehicle.

9. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 8.

Images