End-to-end automatic driving control method, device, equipment and medium

Abstract

The invention provides an end-to-end automatic driving control method and device, equipment and a medium, and relates to the technical field of automatic driving, and the method comprises the steps: obtaining a target image of a user in a driving scene, inputting the target image into a trained automatic driving model, obtaining a target track outputted by the automatic driving model, and outputting the target track to the user; an automatic driving control instruction is obtained according to the target track; the target image comprises an obstacle target, a traffic marking and a traffic signal. According to the method, more abundant semantic feature information is obtained by understanding a complex driving environment, so that more appropriate lanes, speeds, overtaking speeds and the like are selected, more accurate target tracks are planned, generalization and robustness of an end-to-end automatic driving model are improved, traffic safety is improved, traffic accidents are reduced, and the method is suitable for popularization and application. And more efficient automatic driving experience is provided for the user.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous driving control method, device, equipment and medium. Background Art

[0002] With the rapid development of autonomous driving technology, vehicles can drive autonomously without human intervention. In autonomous driving, in order to prevent autonomous vehicles from hitting surrounding vehicles and pedestrians, it is usually necessary to predict the future movement trajectories of surrounding vehicles and humans, so as to plan a safe route.

[0003] At present, the data collected from drivers driving their vehicles reveal autonomous behaviors of drivers such as lane deviation, serpentine driving, arbitrary braking, lane changing, etc. There is also navigation information of left turns, right turns, and straight driving that are not carried out in accordance with regulations at traffic intersections. As a result, the routes planned based on these data lack the decision-making process for traffic rules, lack the rich semantic feature information that can be obtained in vision, and are relatively inaccurate.

[0004] Therefore, how to obtain richer semantic feature information of driving scenes and thus better predict the behavior of vehicles and pedestrians in driving scenes is an important topic in the current field of autonomous driving. Summary of the invention

[0005] In view of this, the present invention proposes an end-to-end autonomous driving control method, device, equipment and medium to solve the problem that the current route planned by autonomous driving lacks a decision-making process for traffic rules and lacks rich semantic feature information that can be obtained in vision.

[0006] The technical solution of the present invention is achieved in this way:

[0007] According to a first aspect, an embodiment of the present invention provides an end-to-end autonomous driving control method, the method comprising:

[0008] Obtain a target image of the user in the driving scene, and input the target image into a trained autonomous driving model to obtain a target trajectory output by the autonomous driving model, and obtain an autonomous driving control instruction from the target trajectory; the target image includes obstacle targets, traffic markings, and traffic signals;

[0009] The autonomous driving model obtains the target trajectory through the following steps:

[0010] Extracting target image features of the target image, converting the target image features into bird's-eye view features, and restoring the bird's-eye view features into an image to be processed;

[0011] Obstacle proxy query information, traffic line query information and traffic signal query information are extracted from the image to be processed, and future trajectory prediction is performed on the obstacle proxy query information, the traffic line query information and the traffic signal query information respectively to predict obstacle position information, traffic line position information and traffic signal position information;

[0012] Obtaining navigation information and driving information of the user, and obtaining initial vehicle query information based on the navigation information, the driving information and the bird's-eye view feature; the driving information includes vehicle speed, vehicle status and vehicle features;

[0013] Cross-attention learning is sequentially performed between the initial self-vehicle query information and the obstacle target information, the traffic marking information and the traffic signal information to obtain first, second and third self-vehicle query information respectively; the obstacle target information includes obstacle agent query information and obstacle position information, the traffic marking information includes traffic marking query information and traffic marking position information, and the traffic signal information includes traffic signal query information and traffic signal position information;

[0014] Determine the initial trajectory, perform cross-attention learning based on the first, second, and third ego-vehicle query information and the initial trajectory, and obtain the target trajectory.

[0015] In combination with the first aspect, in a first implementation manner of the first aspect, the obstacle proxy query information, the traffic marking query information, and the traffic signal query information are extracted from the image to be processed, and future trajectory prediction is performed on the obstacle proxy query information, the traffic marking query information, and the traffic signal query information, respectively, to predict obstacle position information, traffic marking position information, and traffic signal position information, specifically including:

[0016] Extracting obstacle proxy query information, traffic line query information and traffic signal query information from the image to be processed;

[0017] Performing agent-agent interactive attention learning on the obstacle agent query information to predict the obstacle position information;

[0018] Performing agent-map interactive attention learning on the traffic marking query information to predict the traffic marking position information;

[0019] The traffic signal query information is subjected to agent-traffic signal interactive attention learning to predict the traffic signal location information.

[0020] In combination with the first aspect, in a second implementation of the first aspect, the obtaining of navigation information and driving information of the user, and obtaining initial vehicle query information based on the navigation information, the driving information, and the bird's-eye view feature specifically includes:

[0021] Encoding the driving information to obtain an encoded navigation map speed;

[0022] Extracting the vehicle speed from the driving information;

[0023] Perform deformable attention learning on the navigation information, the vehicle speed, and the bird's-eye view feature to obtain the initial vehicle query information.

[0024] In combination with the first aspect, in a third implementation manner of the first aspect, the cross-attention learning between the initial self-vehicle query information and the obstacle target information, the traffic marking information, and the traffic signal information is sequentially performed to obtain the first, second, and third self-vehicle query information, respectively, specifically including:

[0025] Performing a first interactive attention learning between the initial ego vehicle query information and the obstacle query information to obtain first ego vehicle query information; in the first interactive attention learning, the initial ego vehicle query information is used as a first query vector, the obstacle proxy query information is used as a first key vector and a first value vector, the ego vehicle position information encoded by the first multilayer perceptron is embedded as a first query position, and the obstacle position information encoded by the first multilayer perceptron is embedded as a first key position;

[0026] Performing a second interactive attention learning between the first self-vehicle query information and the traffic line query information to obtain second self-vehicle query information; in the second interactive attention learning, using the first self-vehicle query information as a second query vector, using the traffic line query information as a second key vector and a second value vector, using the self-vehicle position information encoded by the second multi-layer perceptron as a second query position embedding, and using the traffic line position information encoded by the second multi-layer perceptron as a second key position embedding;

[0027] A third interactive attention learning is performed between the second self-vehicle query information and the traffic signal query information to obtain third self-vehicle query information; in the third interactive attention learning, the first self-vehicle query information is used as a third query vector, the traffic signal query information is used as a third key vector and a third value vector, the self-vehicle position information encoded by the third multi-layer perceptron is embedded as a third query position, and the traffic signal position information encoded by the third multi-layer perceptron is embedded as a third key position.

[0028] In combination with the third implementation of the first aspect, in the fourth implementation of the first aspect, the expression of the first cross-attention learning is:

[0029] Q ego1 =TransformerDecoder(q1,k1,v1,q pos1 , k pos1 )

[0030] Among them, Q ego1 represents the first self-vehicle query information obtained after the first cross-attention learning; q1 represents the first query vector of the first cross-attention learning; k1 represents the first key vector of the first cross-attention learning; v1 represents the first value vector of the first cross-attention learning; q pos1 represents the first query position embedding learned by the first cross-attention; k pos1 represents the first key position embedding learned by the first cross-attention; q1 = Q ego0 , k1=v1=Q a ,q pos1 =PE1(P ego ), k po =PE1(P a ), Q ego0 represents the initial vehicle query information, Q a Indicates obstacle agent query information, PE1(P ego ) represents the vehicle position information P after being encoded by the first multi-layer perceptron ego , PE1(P a ) represents the obstacle position information P after being encoded by the first multi-layer perceptron a ;

[0031] The expression of the second cross attention learning is:

[0032] Q ego2 =TransformerDecoder(q2,k2,v2,q pos2 , k pos2 )

[0033] Among them, Q ego2 represents the second self-vehicle query information obtained after the second cross-attention learning; q2 represents the second query vector of the second cross-attention learning; k2 represents the second key vector of the second cross-attention learning; v2 represents the second value vector of the second cross-attention learning; q po represents the second query position embedding learned by the second cross-attention; k pos2 represents the second key position embedding learned by the second cross-attention; q2 = Q ego1, k2=v2=Q m ,q pos2 =PE2(P ego ), k pos2 =PE2(P m ), Q m Indicates traffic line query information, PE2(P ego ) represents the vehicle position information P after being encoded by the second multi-layer perceptron ego , PE2(P m ) represents the traffic line position information P after being encoded by the second multi-layer perceptron m ;

[0034] The expression of the third cross-attention learning is:

[0035] Q ego3 =TransformerDecoder(q3,k3,v3,q pos , k pos3 )

[0036] Among them, Q ego3 represents the third self-vehicle query information obtained after the third cross-attention learning; q3 represents the third query vector of the third cross-attention learning; k3 represents the third key vector of the third cross-attention learning; v3 represents the third value vector of the third cross-attention learning; q pos3 represents the third query position embedding learned by the third cross-attention; k pos3 The third key position embedding representing the third cross-attention learning; q3 = Q ego , k3=v3=Q T ,q pos3 =PE3(P ego ), k po =PE3(P T ), Q T Indicates traffic signal query information, PE3(P ego ) represents the vehicle position information P after being encoded by the third multi-layer perceptron ego , PE3(P T ) represents the traffic signal map position information P after the second multi-layer perceptron encoding m .

[0037] In combination with the fourth implementation of the first aspect, in the fifth implementation of the first aspect, the expression of the target trajectory is:

[0038] V ego =MLP(TransformerDecoder(q4,k4,v4),C)

[0039] Among them, Vego represents the target trajectory; q4 represents the query vector of trajectory planning; k4 represents the key vector of trajectory planning; v4 represents the value vector of trajectory planning; C represents the driving instruction; q4=Concat[S ego , f ego , Q ego1 , Q ego2 , Q ego3 ], k4=v4=T ego , S ego represents the state of the vehicle, f ego represents the characteristics of the vehicle, T ego represents the initial trajectory.

[0040] In combination with the first aspect, in a sixth implementation of the first aspect, the method further includes the following steps:

[0041] Image data of a vehicle driving scene is obtained, and the image data is preprocessed using a trained encoder-decoder processing framework to obtain a target image; the encoder-decoder processing framework includes an encoder and a decoder, the encoder is used to extract original image features in the image data, and the decoder is used to restore the original image features to obtain the target image.

[0042] According to a second aspect, an embodiment of the present invention further provides an end-to-end autonomous driving control device, the device comprising:

[0043] The autonomous driving module is used to obtain a target image of the user in the driving scene, and input the target image into a trained autonomous driving model to obtain a target trajectory output by the autonomous driving model, and obtain an autonomous driving control instruction from the target trajectory; the target image includes obstacle targets, traffic markings, and traffic signals;

[0044] The autonomous driving model includes:

[0045] A perception neural network module is used to extract target image features of the target image, convert the target image features into bird's-eye view features, and restore the bird's-eye view features to the image to be processed, and extract obstacle proxy query information, traffic marking query information, and traffic signal query information from the image to be processed;

[0046] A motion prediction neural network module, used to respectively predict the future trajectory of the obstacle proxy query information, the traffic line query information and the traffic signal query information, and predict the obstacle position information, the traffic line position information and the traffic signal position information;

[0047] A planning neural network module is used to obtain navigation information and user's driving information, and obtain initial self-vehicle query information based on the navigation information, the driving information and the bird's-eye view feature, and sequentially perform cross-attention learning between the initial self-vehicle query information and obstacle target information, traffic marking information and traffic signal information to obtain first, second and third self-vehicle query information respectively, determine an initial trajectory, and perform cross-attention learning based on the first, second and third self-vehicle query information and the initial trajectory to obtain a target trajectory; the driving information includes self-vehicle speed, self-vehicle vehicle status and self-vehicle characteristics; the obstacle target information includes obstacle proxy query information and obstacle position information, the traffic marking information includes traffic marking query information and traffic marking position information, and the traffic signal information includes traffic signal query information and traffic signal position information.

[0048] According to the third aspect, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the end-to-end autonomous driving control method as described in any one of the above are implemented.

[0049] According to a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the end-to-end autonomous driving control method as described in any one of the above-mentioned methods.

[0050] According to the fifth aspect, an embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the end-to-end autonomous driving control method as described in any one of the above items.

[0051] The end-to-end autonomous driving control method, device, equipment and medium of the present invention have the following beneficial effects compared with the prior art:

[0052] By inputting the target image into the trained autonomous driving model, the autonomous driving model will extract the target image features of the target image, convert the target image features into BEV features, and restore the BEV features to the image to be processed, predict the future trajectories of obstacle targets, traffic markings and traffic signals from the image to be processed, and then perform cross-attention learning on the initial self-vehicle query information and the obstacle target information, traffic marking information and traffic signal information in sequence to obtain the first, second and third self-vehicle query information respectively, and perform cross-attention learning based on the first, second and third self-vehicle query information and the initial trajectory to obtain the target trajectory, based on the visual understanding of obstacle targets, traffic markings and traffic rules in the driving scene, the complex driving environment is understood to obtain richer semantic feature information, and then a more appropriate lane, speed, overtaking speed, etc. are selected, and a more accurate target trajectory is planned, thereby improving the generalization and robustness of the end-to-end autonomous driving model, thereby improving traffic safety and reducing traffic accidents, and providing users with a more efficient autonomous driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A schematic diagram of a flow chart of an end-to-end autonomous driving control method according to an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a target trajectory generated by the end-to-end autonomous driving control method of the present invention;

[0056] Figure 3 It is a flowchart diagram of another embodiment of the end-to-end autonomous driving control method of the present invention;

[0057] Figure 4 A schematic diagram showing the structure of an end-to-end autonomous driving control device provided by the present invention is shown;

[0058] Figure 5 A schematic diagram of the structure of an end-to-end autonomous driving model provided by the present invention is shown;

[0059] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] With the rapid development of autonomous driving technology, vehicles are able to drive autonomously without human intervention. Autonomous driving uses various sensors, computer vision, artificial intelligence, and machine learning technologies to perceive, analyze, and make decisions about the road environment, thereby achieving autonomous navigation and control of the vehicle. The development of autonomous driving technology aims to improve traffic safety, reduce traffic accidents, and provide more efficient traffic flow. It can eliminate traffic accidents caused by human errors, fatigue driving, and distraction. In addition, autonomous driving can also improve road utilization, reduce congestion, and reduce energy consumption and emissions.

[0062] In autonomous driving, in order to prevent autonomous vehicles from hitting surrounding vehicles and pedestrians, it is usually necessary to predict the future movement trajectories of surrounding vehicles and humans, so as to plan a safe route. It can be seen that the quality of relevant data is crucial to the training and evaluation of autonomous driving systems. The existing data are collected for the purpose of training the corresponding perception model. However, through the data collected by the driver driving the vehicle, there are autonomous behaviors of the driver such as lane deviation, serpentine driving, arbitrary braking, lane change, etc., and there are also navigation information of left turn, right turn and straight driving that are not carried out in accordance with regulations at the traffic intersection. As a result, the route planned based on these data lacks the decision-making process of traffic rules and lacks the rich semantic feature information that can be obtained in vision, which is relatively inaccurate. Therefore, in order to ensure that autonomous driving can be applied in practice, it is necessary to learn and simulate human driving behavior, and the collected vehicle driving trajectory should also need to comply with traffic rules. For example, the direction of travel (left turn, right turn or straight driving) needs to be predicted in advance at the traffic intersection.

[0063] In summary, how to obtain richer semantic feature information of driving scenes and thus better predict the behavior of vehicles and pedestrians in driving scenes is an important topic in the current field of autonomous driving.

[0064] The end-to-end autonomous driving control method provided in this specification can be applied to electronic devices with computer vision and corresponding processing capabilities for deep learning capabilities. The electronic devices may include notebooks, desktop computers, smart phones, smart wearable devices (virtual reality glasses, smart watches, etc.), tablet computers, etc. Of course, the end-to-end autonomous driving control method provided in this specification can also be applied to applications running in the above-mentioned electronic devices. For example, the autonomous driving control method can be applied to a browser with computer vision and corresponding processing capabilities for deep learning capabilities, and can also be applied to real-time corresponding processing software with computer vision and deep learning capabilities.

[0065] See also Figure 1 , Figure 1 A flow chart of an end-to-end autonomous driving control method according to an embodiment of the present invention is shown. The method may include the following steps:

[0066] S101. Obtain a target image of the user in the driving scene, and input the target image into the trained autonomous driving model to obtain a target trajectory output by the autonomous driving model, and obtain an autonomous driving control instruction from the target trajectory. In this embodiment, the target image needs to include at least dynamic obstacle targets, i.e., obstacle targets (such as vehicles, pedestrians, and cones in the driving scene), static traffic markings (such as vectorized lane lines, stop lines, road boundary lines, etc.), and static traffic signals.

[0067] It should be noted that the vehicle in the driving scene is a non-user vehicle that is not driven by the current driver.

[0068] The target image may be stored in advance in the electronic device, or may be acquired by the electronic device from the outside, for example, the electronic device acquires the target image from an external acquisition device such as various cameras arranged in the road scene, or the electronic device extracts a video image frame from a relevant image from the outside, etc.

[0069] There is no restriction on the specific acquisition method of the target image, and it only needs to ensure that the electronic device can acquire the target image.

[0070] This autonomous driving model is an end-to-end transform model. After obtaining the target trajectory through the trained autonomous driving model, the target trajectory can be converted into the vehicle's autonomous driving control instructions, including acceleration, braking, steering and other actions, to achieve automatic driving of the vehicle.

[0071] Among them, the processing flow of the autonomous driving model is:

[0072] S102, extracting target image features of the target image, converting the target image features into bird's eye view (BEV) features, and restoring the BEV features into the image to be processed.

[0073] In this embodiment, the above-mentioned target image features can be obtained through the existing Convolutional Neural Networks (CNN) model for image feature extraction, and then the target image features are converted into BEV features in the BEV space through a BEV encoder, and then the BEV features are decoded through a BEV decoder, that is, the above-mentioned image to be processed is obtained using a BEV encoder-BEV decoder framework.

[0074] S103. Since the target image includes obstacle targets, traffic markings and traffic signals, the image to be processed by mining the semantic feature information of the target image will also include these three types of targets. Obstacle agent query information, traffic marking query information (map query) and traffic signal query information are extracted from the image to be processed. The above query information is feature information of the corresponding targets, which will be used as scene information of the driving scene. Future trajectory prediction is then performed on the obstacle agent query information, traffic signal query information and traffic signal query information, respectively, to predict obstacle position information, traffic marking position information and traffic signal position information.

[0075] More specifically, in step S103, a group of agents are used to learn agent-level features from the image to be processed through multi-head self-attention. The group of agents includes an obstacle agent, a map agent, and a traffic signal agent. The properties of these three agents (including position, category score, orientation, etc.) can be decoded from the agent query by a preset decoder.

[0076] In order to enrich the features of the agent for predicting future trajectories, the autonomous driving model uses the attention mechanism to perform agent-agent interaction attention learning, agent-map interaction attention learning, and agent-traffic light interaction attention learning to predict the future trajectory of each agent and express it as a multimodal motion vector, that is, in, represents the multimodal motion vector of the predicted agent, R represents the set of predicted agents, N a Indicates the number of predicted agents, N k represents the number of modalities of the predicted agent, T fIndicates the number of future timestamps. In this way, based on the obstacle agent, map agent and traffic signal agent, the obstacle position information, traffic marking position information and traffic signal position information are obtained respectively.

[0077] The motion vectors of each modality are used to constrain the self-planned trajectory and avoid collisions, and the agent queries are used as scene information.

[0078] S104: Obtain navigation information and user's driving information, and obtain initial vehicle query information based on the navigation information, driving information and BEV characteristics.

[0079] In this embodiment, the navigation information includes map speed limit information and navigation steering information. The navigation information can be obtained through the navigation system, and the driving information can be obtained through various sensor systems arranged on the vehicle. The sensor systems include but are not limited to: surround cameras, rear-view cameras, front-view cameras, lidars, positioning sensors, and domain controllers. The driving information specifically includes the vehicle speed, vehicle status, and vehicle characteristics.

[0080] Then, based on the navigation information, driving information and the previously obtained BEV characteristics, the initial vehicle query information (vector) is modeled.

[0081] S105, cross-attention learning is performed in sequence between the initial vehicle query information and the obstacle target information, the traffic marking information system, and the traffic signal information, that is, a total of three cross-attention learnings are performed, so as to obtain the first, second, and third vehicle query information respectively. Among them, the obstacle target includes the obstacle agent query information and the obstacle position information, the traffic marking information includes the traffic marking query information and the traffic marking position information, and the traffic signal information includes the traffic signal query information and the traffic signal position information.

[0082] S106, determining an initial trajectory, performing cross-attention learning based on the first, second, and third ego-vehicle query information and the initial trajectory, obtaining a target trajectory, and generating an automatic driving control instruction for the vehicle based on the target trajectory.

[0083] In this embodiment, the initial trajectory T can be determined by the traditional Lattice planning algorithm. ego. The Lattice planning algorithm is a path planning algorithm for autonomous driving. It transforms continuous path planning problems into discrete decision-making problems and simplifies computational complexity by constructing a Lattice network. Specifically, the Lattice planning algorithm uses the global path given by the navigation system as the coordinate axis of the Frenet coordinate system and determines the vehicle's coordinate system in the Frenet coordinate system. After local sampling, it obtains a set of alternative trajectories, calculates the corresponding loss function, and transforms the optimal local trajectory back to the Cartesian coordinate system. The optimal local trajectory is selected from multiple alternative trajectories. The Lattice planning algorithm calculates the trajectory for a period of time in the future (for example, 3s), samples a data point every preset time (for example, 0.5s), collects the trajectory coordinates of all data points, and obtains the initial trajectory T ego .

[0084] More specifically, the autonomous driving model includes a three-layer network structure, namely the perception neural network layer, the motion prediction neural network layer and the planning neural network layer. Accordingly, the autonomous driving model is divided into two stages for training. The first stage trains the perception and motion prediction neural network layers to plan the parameters of the network to obtain stable weight initialization. The second stage trains all task modules together to avoid problems such as excessive network depth, gradient vanishing and explosion in Loss gradient calculation, and to avoid modules learning false and useless information. In addition, during training, there is no need to use the traditional planning algorithm module to calculate the trajectory in real time. You only need to input the pre-calculated trajectory into the planning neural network layer.

[0085] In this embodiment, all known historical trajectory data are collected as sample data to provide massive data support for the subsequent training of the autonomous driving model. The historical trajectory data can be stored in the electronic device in advance or obtained by the electronic device from the outside. There is no restriction on the specific form of obtaining the historical trajectory data, as long as the electronic device can obtain the historical trajectory data.

[0086] Preferably, the historical trajectory data is obtained by:

[0087] First, a preset driving route is planned for the driver, and then the driver drives the vehicle according to the preset driving route and receives the map speed limit information and turning information given by the navigation system during driving. Then, the final end-to-end turning information at the intersection is calculated by combining the turning angle information of the navigation system and the turning angle information of the vehicle, and the future estimated trajectory points of the vehicle (for example, the trajectory points of the next 3s) are calculated using planning algorithms such as EMPlanner. Then, the distance error (L2 level) between the actual historical trajectory and the calculated future estimated trajectory points is compared in real time frame by frame. If the error is within the preset threshold, the actual historical trajectory is used as the historical estimated data. If the error exceeds the preset threshold, it is judged as erroneous data. The final historical trajectory data is based on the future estimated trajectory calculated based on planning algorithms such as EMPlanner, and the data is labeled and recorded to facilitate the tracing of data during subsequent autonomous driving model training.

[0088] After the historical trajectory data is collected, the data is integrated into a unified data set in a certain format and labeled, and can be directly used to train the end-to-end autonomous driving model.

[0089] In particular, it will determine whether the data generated by the driver driving along the preset driving route is in accordance with the prescribed speed, and record the data that does not meet the requirements (including speeding too fast or too slow). For example, the navigation map speed limit information is input into the vehicle-side domain control. If there are no other vehicles ahead, the current driver is reminded to accelerate to the speed limit value on the map, that is: ΔV = V navi -V i , where V navi Indicates the speed limit of the navigation system, V i Indicates the current real-time speed of the vehicle. When ΔV exceeds 5km / h, a warning message is issued to the driver, informing him that he should accelerate to reach the navigation map interface.

[0090] See also Figure 2 As shown, Figure 2 This is a right-turn driving scenario, in which the blue lines represent predicted traffic markings (including vectorized lane lines, stop lines, road boundary lines, etc.), the gray lines represent zebra crossings, the red boxes represent predicted obstacle targets, the green boxes represent the ego vehicle, the orange lines starting from the green box represent the initial trajectory obtained based on the traditional autonomous driving algorithm, and the green lines starting from the green box represent the target trajectory predicted by the autonomous driving model in this embodiment. It can be seen that the target trajectory is smoother and more in line with real human driving behavior.

[0091] The end-to-end autonomous driving control method of the present invention inputs a target image into a trained autonomous driving model, and the autonomous driving model extracts target image features of the target image, converts the target image features into BEV features, and restores the BEV features to the image to be processed, predicts the future trajectory of obstacle targets, traffic markings and traffic signals from the image to be processed, and then sequentially performs cross-attention learning on the initial self-vehicle query information and the obstacle target information, traffic marking information and traffic signal information to obtain the first, second and third self-vehicle query information respectively, performs cross-attention learning based on the first, second and third self-vehicle query information and the initial trajectory to obtain the target trajectory, and based on the visual understanding of obstacle targets, traffic markings and traffic rules in the driving scene, understands the complex driving environment, obtains richer semantic feature information, and then selects more appropriate lanes, speeds, overtaking speeds, etc., and plans a more accurate target trajectory, thereby improving the generalization and robustness of the end-to-end autonomous driving model, thereby improving traffic safety and reducing traffic accidents, and providing users with a more efficient autonomous driving experience.

[0092] In this embodiment, step S104 specifically obtains initial vehicle query information based on the navigation map speed after the navigation information is encoded, the vehicle speed in the driving information, and the BEV characteristics, that is:

[0093] Q ego =DeformAttn(Q,MLP(V q ), B) (1)

[0094] In formula (1), Q ego0 represents the initial egoquery information (vector); Q represents the egoquery speed; MLP (V q ) represents the navigation map speed after multilayer perceptron (MLP) encoding; B represents the BEV feature.

[0095] Accordingly, step S105 specifically includes:

[0096] S1051. In order to learn the position and motion information of other dynamic obstacles, that is, dynamic traffic participants, in this embodiment, the first interactive attention learning between the initial vehicle query information and the obstacle query information is first performed to obtain the first vehicle query information. Specifically:

[0097] Q ego1 =TransformerDecoder(q1,k1,v1,q pos1 , k pos ) (2)

[0098] In formula (2), Q ego1 represents the first self-vehicle query information (vector) obtained after the first cross-attention learning; q1 represents the first query vector of the first cross-attention learning; k1 represents the first key vector of the first cross-attention learning; v1 represents the first value vector of the first cross-attention learning; q pos1 represents the first query position embedding learned by the first cross-attention; k pos1 Represents the first key position embedding learned by the first cross-attention.

[0099] In the first cross-attention learning, the initial self-car query information Q ego0 As the query vector in this interactive attention learning, the obstacle agent query information Q a As the key (keyword) vector and value vector in this interactive attention learning, that is, q1 = Q ego0 , k1=v1=Q a . Vehicle position information P ego and obstacle position information P a will be encoded by the first multi-layer perceptron (MLP PE1) of the single layer and embedded as the first query position in the interactive attention learning process q pos1 and the first key position embedding k pos1 , that is, q pos1 =PE1(P ego ), k pos1 =PE1(P a ).

[0100] S1052, after the first interactive attention learning with the obstacle query information, the second interactive attention learning between the first self-inspection query information and the traffic line query information is performed to obtain the second self-vehicle query information, specifically:

[0101] Q ego2 =TransformerDecoder(q2,k2,v2,q pos2 , k pos2 ) (3)

[0102] In formula (3), Q ego represents the second self-vehicle query information (vector) obtained after the second cross-attention learning; q2 represents the second query vector of the second cross-attention learning; k2 represents the second key vector of the second cross-attention learning; v2 represents the second value vector of the second cross-attention learning; q pos2 represents the second query position embedding learned by the second cross-attention; k pos2 Represents the second key position embedding learned by the second cross-attention.

[0103] In the second cross-attention learning, the first self-car query information Q ego1 As the query vector in this interactive attention learning, the traffic line query information Q m As the key (keyword) vector and value vector in this interactive attention learning, that is, q2 = Q ego1 , k2=v2=Q m . Vehicle position information P ego and traffic line position information P m will be encoded by the second multi-layer perceptron (MLP PE2) of the single layer and embedded as the second query position q in the interactive attention learning process. pos2 and the second key position embedding k pos2 , that is, q pos2 =PE2(P ego ), k pos2 =PE2(P m ).

[0104] S1053, after the second interactive attention learning with the traffic line query information, the third interactive attention learning between the second self-vehicle query information and the traffic signal query information is performed to obtain the third self-vehicle query information, specifically:

[0105] Q ego3 =TransformerDecoder(q3,k3,v3,q pos , k pos ) (4)

[0106] In formula (4), Q ego3 represents the third self-vehicle query information (vector) obtained after the third cross-attention learning; q3 represents the third query vector of the third cross-attention learning; k3 represents the third key vector of the third cross-attention learning; v3 represents the third value vector of the third cross-attention learning; q pos3 represents the third query position embedding learned by the third cross-attention; k pos3 Represents the third key position embedding learned by the third cross-attention.

[0107] In the third cross-attention learning, the second self-car query information Q ego2 As the query vector in this interactive attention learning, the traffic signal query information Q T As the key (keyword) vector and value vector in this interactive attention learning, that is, q3 = Q ego2 , k3=v3=Q T . Vehicle position information P egoand traffic signal location information P T will be encoded by the single-layer third multilayer perceptron (MLP PE3) and embedded as the third query position q in the interactive attention learning process. pos3 and the third key position embedding k pos3 , that is, q pos3 =PE3(P ego ), k pos3 =PE3(P T ).

[0108] In this embodiment, the structures of the first, second and third multilayer perceptrons are different, so as to obtain richer vector information.

[0109] The first vehicle query information Q obtained in this way ego1 contains the dynamic obstacle target information in the driving scene, and the second vehicle query information Q ego2 contains the dynamic obstacle targets and static traffic markings in the driving scene. The third vehicle query information Q ego3 The information about the static traffic signal in the driving scene is included in the information about the dynamic obstacle target and the static traffic markings. Therefore, the third self-vehicle query information Q is finally obtained. ego It includes as much dynamic and static information as possible in the driving scene.

[0110] Accordingly, the target trajectory is obtained as follows:

[0111] V ego =MLP(TransformerDecoder(q4,k4,v4),C) (5)

[0112] In formula (4), V ego represents the target trajectory; q4 represents the query vector of trajectory planning; k4 represents the key vector of trajectory planning; v4 represents the value vector of trajectory planning; C represents the driving instruction, which usually includes left turn, right turn and straight driving. ego , vehicle characteristics f ego 、First car query information Q ego1 、Second car query information Q ego2 、Third car query information Q ego3 Together we determine the query vector q4 for trajectory planning, i.e., q4 = Concat[S ego , f ego , Q ego , Q ego2 , Q ego3], the initial trajectory obtained by traditional algorithms such as the Lattice planning algorithm is used as the key (keyword) vector and value vector of trajectory planning, that is, k4 = v4 = T ego .

[0113] Through the current vehicle state S ego and the vehicle characteristics f ego It interacts with the trajectory calculated by the Lattice planning algorithm to learn potential features and provide guarantees for the trajectory features predicted by the autonomous driving model.

[0114] See also Figure 3 , Figure 3 A flow chart of an end-to-end autonomous driving control method according to an embodiment of the present invention is shown. The method may include the following steps:

[0115] S201, obtaining image data of a vehicle driving scene, and preprocessing the image data using a trained encoder-decoder processing framework to obtain a target image. The encoder-decoder processing framework includes an encoder and a decoder, the encoder in the encoder-decoder processing framework is used to extract original image features in the image data, and the decoder is used to restore the original image features to obtain the target image.

[0116] First, we collect image data of vehicle driving scenes, that is, image data that has not been manually annotated, and obtain the original image based on these image data. Then, we construct an encoder-decoder processing framework to preprocess the original image, process the original image using the encoder, extract the original image features in the original image, and then input the original image features into the decoder for restoration processing to obtain the target image. Since the target image can be restored from the feature vector, it can be considered that the feature vector obtained by the encoder can represent the underlying semantic feature information of the original image. By fully mining the semantic feature information in vision, it can greatly help predict the behavior of dynamic objects in driving scenes, including vehicles and humans.

[0117] Since there may be duplicate images captured by the camera, in this embodiment, the similarity between different original images is also calculated, and duplicate images with a similarity greater than a preset threshold (for example, a similarity of 90%) are eliminated. The unlabeled image data is screened to obtain the original image, avoiding repeated calculations and reducing computing overhead.

[0118] In this embodiment, in order to ensure that the above-mentioned encoder-decoder processing framework can obtain the image features of the original image without distortion and the target image based on the image features as much as possible, the encoder-decoder processing framework will be iteratively trained. During the training process, the original image will first be randomly occluded so that the image after occlusion processing is different from the image data that has not been manually labeled, and the training goal is to minimize the difference between the output image and the input image.

[0119] Among them, both the encoder and the decoder can be CNN models.

[0120] It should be noted that the image data may be stored in advance in the electronic device, or may be acquired by the electronic device from the outside. For example, the electronic device acquires the image data from an external acquisition device such as various cameras arranged in a road scene, or the electronic device extracts video image frames from external video data, etc.

[0121] There is no restriction on the specific form of acquiring the image data, and it only needs to ensure that the electronic device can acquire the image data.

[0122] S202, obtain the target image of the user in the driving scene, and input the target image into the trained autonomous driving model to obtain the target trajectory output by the autonomous driving model, and obtain the autonomous driving control instructions from the target trajectory. In this embodiment, the target image needs to contain at least dynamic obstacle targets (such as vehicles, pedestrians and cones in the driving scene), static traffic markings (such as vectorized lane lines, stop lines, road boundary lines, etc.) and static traffic signals. For specific content, refer to step S101, and the processing method of the autonomous driving model refers to steps S102-S106.

[0123] The following describes an apparatus provided by an embodiment of the present invention. The apparatus described below and the method described above can refer to each other.

[0124] See also Figure 4 , Figure 4 A schematic diagram showing the structure of an end-to-end autonomous driving control device according to an embodiment of the present invention is shown. The device may include:

[0125] The automatic driving module 10 is used to obtain the target image of the user in the driving scene, and input the target image into the trained automatic driving model to obtain the target trajectory output by the automatic driving model, and obtain the automatic driving control instruction from the target trajectory. In this embodiment, the target image needs to contain at least dynamic obstacle targets (such as vehicles, pedestrians and cones in the driving scene), static traffic markings (such as vectorized lane lines, stop lines, road boundary lines, etc.) and static traffic signals.

[0126] It should be noted that the vehicles in the driving scene include user vehicles, i.e. the vehicles being driven by the driver, and other non-user vehicles.

[0127] The target image may be stored in advance in the electronic device, or may be acquired by the electronic device from the outside, for example, the electronic device acquires the image from an external acquisition device such as various cameras arranged in the road scene, or the electronic device extracts a video image frame from an external target image, etc.

[0128] There is no restriction on the specific acquisition method of the target image, and it only needs to ensure that the electronic device can acquire the target image.

[0129] This autonomous driving model is an end-to-end model. After obtaining the target trajectory through the trained autonomous driving model, the target trajectory can be converted into the vehicle's autonomous driving control instructions, including acceleration, braking, steering and other actions, to achieve automatic driving of the vehicle.

[0130] The specific network architecture of the autonomous driving model is as follows: Figure 5 As shown, including:

[0131] The perception neural network module is used to extract target image features of the target image, convert the target image features into BEV features, and restore the BEV features to the image to be processed, and extract obstacle agent query information, traffic marking query information and traffic signal query information from the image to be processed.

[0132] In this embodiment, the above-mentioned target image features can be obtained through the existing CNN model for image feature extraction, and then the target image features can be converted into BEV features in the BEV space through a BEV encoder, and then the BEV features can be decoded through a BEV decoder to restore the BEV features to the image to be processed.

[0133] The motion prediction neural network module is used to predict the future trajectory of the obstacle proxy query information, traffic line query information and traffic signal query information respectively, predict the obstacle position information, traffic line position information and traffic signal position information, and sequentially perform cross-attention learning between the initial self-vehicle query information and the obstacle target information, traffic line information and traffic signal information to obtain the first, second and third self-vehicle query information respectively. Among them, the obstacle target information includes obstacle proxy query information and obstacle position information, the traffic line information includes traffic line query information and traffic line position information, and the traffic signal information includes traffic signal query information and traffic signal position information. Determine the initial trajectory, perform cross-attention learning based on the first, second and third self-vehicle query information and the initial trajectory, obtain the target trajectory, and generate the vehicle's automatic driving control instructions based on the target trajectory.

[0134] The neural network module is planned to obtain navigation information and driving information of the user, and obtain initial vehicle query information based on the navigation information, driving information and BEV characteristics.

[0135] In this embodiment, the navigation information includes map speed limit information and navigation steering information. The navigation information can be obtained through the navigation system, and the driving information can be obtained through various sensor systems arranged on the vehicle. The sensor systems include but are not limited to: surround view camera, rear view camera, front view camera, laser radar, positioning sensor, domain controller. The driving information specifically includes the vehicle speed, vehicle status and vehicle characteristics.

[0136] Then, based on the navigation information, driving information and the previously obtained BEV characteristics, the initial vehicle query information (vector) is modeled.

[0137] In this embodiment, the initial trajectory T can be determined by the traditional Lattice planning algorithm. ego . The Lattice planning algorithm is a path planning algorithm for autonomous driving. It transforms continuous path planning problems into discrete decision-making problems and simplifies computational complexity by constructing a Lattice network. Specifically, the Lattice planning algorithm uses the global path given by the navigation system as the coordinate axis of the Frenet coordinate system and determines the vehicle's coordinate system in the Frenet coordinate system. After local sampling, it obtains a set of alternative trajectories, calculates the corresponding loss function, and transforms the optimal local trajectory back to the Cartesian coordinate system. The optimal local trajectory is selected from multiple alternative trajectories. The Lattice planning algorithm calculates the trajectory for a period of time in the future (for example, 3s), samples a data point every preset time (for example, 0.5s), collects the trajectory coordinates of all data points, and obtains the initial trajectory T ego .

[0138] The end-to-end autonomous driving control device of the present invention inputs a target image into a trained autonomous driving model, and the autonomous driving model extracts target image features of the target image, converts the target image features into BEV features, and restores the BEV features to the image to be processed, predicts the future trajectory of obstacle targets, traffic markings and traffic signals from the image to be processed, and then sequentially performs cross-attention learning on the initial self-vehicle query information and the obstacle target information, traffic marking information and traffic signal information to obtain the first, second and third self-vehicle query information respectively, performs cross-attention learning based on the first, second and third self-vehicle query information and the initial trajectory to obtain the target trajectory, and based on the visual understanding of obstacle targets, traffic markings and traffic rules in the driving scene, understands the complex driving environment to obtain richer semantic feature information, and then selects a more appropriate lane, speed, overtaking speed, etc., and plans a more accurate target trajectory, thereby improving the generalization and robustness of the end-to-end autonomous driving model, thereby improving traffic safety and reducing traffic accidents, and providing users with a more efficient autonomous driving experience.

[0139] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610 (processor), a communication interface 620 (Communications Interface), a memory 630 (memory) and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic command in the memory 630 to execute an end-to-end autonomous driving control method, which includes:

[0140] Obtain a target image of the user in the driving scene, and input the target image into a trained autonomous driving model to obtain a target trajectory output by the autonomous driving model, and obtain an autonomous driving control instruction from the target trajectory; the target image includes obstacle targets, traffic markings, and traffic signals;

[0141] The autonomous driving model obtains the target trajectory through the following steps:

[0142] Extracting target image features of the target image, converting the target image features into bird's-eye view features, and restoring the bird's-eye view features into an image to be processed;

[0143] Obstacle proxy query information, traffic line query information and traffic signal query information are extracted from the image to be processed, and future trajectory prediction is performed on the obstacle proxy query information, the traffic line query information and the traffic signal query information respectively to predict obstacle position information, traffic line position information and traffic signal position information;

[0144] Obtaining navigation information and driving information of the user, and obtaining initial vehicle query information based on the navigation information, the driving information and the bird's-eye view feature; the driving information includes vehicle speed, vehicle status and vehicle features;

[0145] Cross-attention learning is performed on the initial ego vehicle query information, the obstacle target information, the traffic marking information, and the traffic signal information in sequence to obtain first, second, and third ego vehicle query information, respectively; the obstacle target information includes obstacle agent query information and obstacle position information, the traffic marking information includes traffic marking query information and traffic marking position information, and the traffic signal information includes traffic signal query information and traffic signal position information;

[0146] Determine the initial trajectory, perform cross-attention learning based on the first, second, and third ego-vehicle query information and the initial trajectory, and obtain the target trajectory.

[0147] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memor), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0148] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute the end-to-end autonomous driving control method provided by the above methods, the method comprising:

[0149] Obtain a target image of the user in the driving scene, and input the target image into a trained autonomous driving model to obtain a target trajectory output by the autonomous driving model, and obtain an autonomous driving control instruction from the target trajectory; the target image includes obstacle targets, traffic markings, and traffic signals;

[0150] The autonomous driving model obtains the target trajectory through the following steps:

[0151] Extracting target image features of the target image, converting the target image features into bird's-eye view features, and restoring the bird's-eye view features into an image to be processed;

[0152] Obstacle proxy query information, traffic line query information and traffic signal query information are extracted from the image to be processed, and future trajectory prediction is performed on the obstacle proxy query information, the traffic line query information and the traffic signal query information respectively to predict obstacle position information, traffic line position information and traffic signal position information;

[0153] Obtaining navigation information and driving information of the user, and obtaining initial vehicle query information based on the navigation information, the driving information and the bird's-eye view feature; the driving information includes vehicle speed, vehicle status and vehicle features;

[0154] Cross-attention learning is performed on the initial ego vehicle query information, the obstacle target information, the traffic marking information, and the traffic signal information in sequence to obtain first, second, and third ego vehicle query information, respectively; the obstacle target information includes obstacle agent query information and obstacle position information, the traffic marking information includes traffic marking query information and traffic marking position information, and the traffic signal information includes traffic query signal information and traffic signal position information;

[0155] Determine the initial trajectory, perform cross-attention learning based on the first, second, and third ego-vehicle query information and the initial trajectory, and obtain the target trajectory.

[0156] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program is implemented to execute the above-mentioned control method for performing end-to-end autonomous driving, the method comprising:

[0157] Obtain a target image of the user in the driving scene, and input the target image into a trained autonomous driving model to obtain a target trajectory output by the autonomous driving model, and obtain an autonomous driving control instruction from the target trajectory; the target image includes obstacle targets, traffic markings, and traffic signals;

[0158] The autonomous driving model obtains the target trajectory through the following steps:

[0159] Extracting target image features of the target image, converting the target image features into bird's-eye view features, and restoring the bird's-eye view features into an image to be processed;

[0160] Obstacle proxy query information, traffic line query information and traffic signal query information are extracted from the image to be processed, and future trajectory prediction is performed on the obstacle proxy query information, the traffic line query information and the traffic signal query information respectively to predict obstacle position information, traffic line position information and traffic signal position information;

[0161] Obtaining navigation information and driving information of the user, and obtaining initial vehicle query information based on the navigation information, the driving information and the bird's-eye view feature; the driving information includes vehicle speed, vehicle status and vehicle features;

[0162] Cross-attention learning is performed on the initial ego vehicle query information, the obstacle target information, the traffic marking information, and the traffic signal information in sequence to obtain first, second, and third ego vehicle query information, respectively; the obstacle target information includes obstacle agent query information and obstacle position information, the traffic marking information includes traffic marking query information and traffic marking position information, and the traffic signal information includes traffic signal query information and traffic signal position information;

[0163] Determine the initial trajectory, perform cross-attention learning based on the first, second, and third ego-vehicle query information and the initial trajectory, and obtain the target trajectory.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An end-to-end autonomous driving control method, characterized in that: The method comprises: Obtain a target image of the user in the driving scene, and input the target image into a trained autonomous driving model to obtain a target trajectory output by the autonomous driving model, and obtain an autonomous driving control instruction from the target trajectory; the target image includes obstacle targets, traffic markings, and traffic signals; The autonomous driving model obtains the target trajectory through the following steps: Extracting target image features of the target image, converting the target image features into bird's-eye view features, and restoring the bird's-eye view features into an image to be processed; Obstacle proxy query information, traffic line query information and traffic signal query information are extracted from the image to be processed, and future trajectory prediction is performed on the obstacle proxy query information, the traffic line query information and the traffic signal query information respectively to predict obstacle position information, traffic line position information and traffic signal position information; Obtaining navigation information and driving information of the user, and obtaining initial vehicle query information based on the navigation information, the driving information and the bird's-eye view feature; the driving information includes vehicle speed, vehicle status and vehicle features; Cross-attention learning is sequentially performed between the initial self-vehicle query information and the obstacle target information, the traffic marking information and the traffic signal information to obtain first, second and third self-vehicle query information respectively; the obstacle target information includes obstacle proxy query information and obstacle position information, the traffic marking information includes traffic marking query information and traffic marking position information, and the traffic signal information includes traffic signal query information and traffic signal position information; Determine the initial trajectory, perform cross-attention learning based on the first, second, and third ego-vehicle query information and the initial trajectory, and obtain the target trajectory.

2. The end-to-end autonomous driving control method according to claim 1, characterized in that: The extracting of obstacle proxy query information, traffic line query information and traffic signal query information from the image to be processed, and performing future trajectory prediction on the obstacle proxy query information, the traffic line query information and the traffic signal query information, respectively, to predict obstacle position information, traffic line position information and traffic signal position information, specifically includes: Extracting obstacle proxy query information, traffic line query information and traffic signal query information from the image to be processed; Performing agent-agent interactive attention learning on the obstacle agent query information to predict the obstacle position information; Performing agent-map interactive attention learning on the traffic marking query information to predict the traffic marking position information; The traffic signal query information is subjected to agent-traffic signal interactive attention learning to predict the traffic signal location information.

3. The end-to-end autonomous driving control method according to claim 1, characterized in that: The obtaining of navigation information and driving information of the user, and obtaining initial vehicle query information based on the navigation information, the driving information and the bird's-eye view feature, specifically includes: Encoding the driving information to obtain an encoded navigation map speed; Extracting the vehicle speed from the driving information; Perform deformable attention learning on the navigation information, the vehicle speed, and the bird's-eye view feature to obtain the initial vehicle query information.

4. The end-to-end autonomous driving control method according to claim 1, characterized in that: The cross-attention learning between the initial self-vehicle query information and the obstacle target information, the traffic line information and the traffic signal information is sequentially performed to obtain the first, second and third self-vehicle query information respectively, specifically including: Performing a first interactive attention learning between the initial ego vehicle query information and the obstacle query information to obtain first ego vehicle query information; in the first interactive attention learning, the initial ego vehicle query information is used as a first query vector, the obstacle proxy query information is used as a first key vector and a first value vector, the ego vehicle position information encoded by the first multilayer perceptron is embedded as a first query position, and the obstacle position information encoded by the first multilayer perceptron is embedded as a first key position; Performing a second interactive attention learning between the first self-vehicle query information and the traffic line query information to obtain second self-vehicle query information; in the second interactive attention learning, using the first self-vehicle query information as a second query vector, using the traffic line query information as a second key vector and a second value vector, using the self-vehicle position information encoded by the second multi-layer perceptron as a second query position embedding, and using the traffic line position information encoded by the second multi-layer perceptron as a second key position embedding; A third interactive attention learning is performed between the second self-vehicle query information and the traffic signal query information to obtain third self-vehicle query information; in the third interactive attention learning, the first self-vehicle query information is used as a third query vector, the traffic signal query information is used as a third key vector and a third value vector, the self-vehicle position information encoded by the third multi-layer perceptron is embedded as a third query position, and the traffic signal position information encoded by the third multi-layer perceptron is embedded as a third key position.

5. The end-to-end autonomous driving control method according to claim 4, characterized in that: The expression of the first cross attention learning is: Q ego1 =TransformerDecoder(q1,k1,v1,q pos1 ,k pos ) Among them, Q ego1 represents the first self-vehicle query information obtained after the first cross-attention learning; q1 represents the first query vector of the first cross-attention learning; k1 represents the first key vector of the first cross-attention learning; v1 represents the first value vector of the first cross-attention learning; q pos1 represents the first query position embedding learned by the first cross-attention; k pos1 represents the first key position embedding learned by the first cross-attention; q1 = Q ego0 , k1=v1=Q a ,q pos1 =PE1(P ego ), k pos1 =PE1(P a ), Q ego0 represents the initial vehicle query information, Q a Indicates obstacle proxy query information, PE1(P ego ) represents the vehicle position information P after being encoded by the first multi-layer perceptron ego , PE1(P a ) represents the obstacle position information P after being encoded by the first multi-layer perceptron a ; The expression of the second cross attention learning is: Q ego2 =TransformerDecoder(q2,k2,v2,q pos2 ,k pos2 ) Among them, Q ego2 represents the second self-vehicle query information obtained after the second cross-attention learning; q2 represents the second query vector of the second cross-attention learning; k2 represents the second key vector of the second cross-attention learning; v2 represents the second value vector of the second cross-attention learning; q pos2 represents the second query position embedding learned by the second cross-attention; k pos2 represents the second key position embedding learned by the second cross-attention; q2 = Q ego1 , k2=v2=Q m ,q pos =PE2(P ego ), k pos2 =PE2(P m ), Q m Indicates traffic line query information, PE2(P ego ) represents the vehicle position information P after being encoded by the second multi-layer perceptron ego , PE2(P m ) represents the traffic line position information P after being encoded by the second multi-layer perceptron m ; The expression of the third cross-attention learning is: Q ego3 =TransformerDecoder(q3,k3,v3,q pos3 ,k pos3 ) Among them, Q ego3 represents the third self-vehicle query information obtained after the third cross-attention learning; q3 represents the third query vector of the third cross-attention learning; k3 represents the third key vector of the third cross-attention learning; v3 represents the third value vector of the third cross-attention learning; q pos3 represents the third query position embedding learned by the third cross-attention; k pos3 The third key position embedding representing the third cross-attention learning; q3 = Q ego2 , k3=v3=Q T ,q pos =PE3(P ego ), k pos =PE3(P T ), Q T Indicates traffic signal query information, PE3(P ego ) represents the vehicle position information P after being encoded by the third multi-layer perceptron ego , PE3(P T ) represents the traffic signal map position information P after the second multi-layer perceptron encoding m .

6. The end-to-end autonomous driving control method according to claim 5, characterized in that: The expression of the target trajectory is: V ego =MLP(TransformerDecoder(q4,k4,v4),C) Among them, V ego represents the target trajectory; q4 represents the query vector of trajectory planning; k4 represents the key vector of trajectory planning; v4 represents the value vector of trajectory planning; C represents the driving instruction; q4=Concat[S ego , f ego , Q ego1 , Q ego2 , Q ego3 ], k4=v4=T ego , S ego represents the state of the vehicle, f ego represents the characteristics of the vehicle, T ego represents the initial trajectory.

7. The end-to-end autonomous driving control method according to claim 1, characterized in that: The method further comprises the following steps: Image data of a vehicle driving scene is obtained, and the image data is preprocessed using a trained encoder-decoder processing framework to obtain a target image; the encoder-decoder processing framework includes an encoder and a decoder, the encoder is used to extract original image features in the image data, and the decoder is used to restore the original image features to obtain the target image.

8. An end-to-end autonomous driving control device, characterized in that: The device comprises: The autonomous driving module is used to obtain a target image of the user in the driving scene, and input the target image into a trained autonomous driving model to obtain a target trajectory output by the autonomous driving model, and obtain an autonomous driving control instruction from the target trajectory; the target image includes obstacle targets, traffic markings, and traffic signals; The autonomous driving model includes: A perception neural network module is used to extract target image features of the target image, convert the target image features into bird's-eye view features, and restore the bird's-eye view features to the image to be processed, and extract obstacle proxy query information, traffic marking query information, and traffic signal query information from the image to be processed; A motion prediction neural network module, used to respectively predict the future trajectory of the obstacle proxy query information, the traffic line query information and the traffic signal query information, and predict the obstacle position information, the traffic line position information and the traffic signal position information; A planning neural network module is used to obtain navigation information and user's driving information, and obtain initial self-vehicle query information based on the navigation information, the driving information and the bird's-eye view feature, and sequentially perform cross-attention learning between the initial self-vehicle query information and obstacle target information, traffic marking information and traffic signal information to obtain first, second and third self-vehicle query information respectively, determine an initial trajectory, and perform cross-attention learning based on the first, second and third self-vehicle query information and the initial trajectory to obtain a target trajectory; the driving information includes self-vehicle speed, self-vehicle vehicle status and self-vehicle characteristics; the obstacle target information includes obstacle proxy query information and obstacle position information, the traffic marking information includes traffic marking query information and traffic marking position information, and the traffic signal information includes traffic signal query information and traffic signal position information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the end-to-end autonomous driving control method as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the end-to-end autonomous driving control method as described in any one of claims 1 to 7 are implemented.

Images