Vectorized map and multi-task feature embedding combined end-to-end automatic driving decision planning model and device

Abstract

The invention discloses an end-to-end automatic driving decision planning model and device combining a vectorized map and multi-task feature embedding. The end-to-end automatic driving decision planning model comprises six task branches of different modules such as perception, decision and control. An end-to-end automatic driving framework is utilized to reduce errors between different model transmissions, and a multi-task constraint mode is utilized to improve the fitting ability of the model. Vectorized map and aerial view target detection are combined to serve as perception representation of a multi-task model, dynamic and static target information needed in aerial view space is provided, environment perception information and decision navigation point output are unified to the same space, and model prediction precision is improved. Information transmission is performed by using a cross attention mechanism between different tasks, training conflicts caused by simple sharing of feature maps in a traditional multi-task model are overcome, efficient coupling between different tasks is realized, and model prediction precision is improved.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving of smart cars and relates to an autonomous driving decision-making planning model and equipment based on vectorized maps and multi-task feature embedding. Background Art

[0002] With the further development of artificial intelligence and smart transportation, autonomous driving has become a hot topic in current research. As the core component of the autonomous driving system, the decision-making and planning module is a direct manifestation of the intelligence of autonomous driving. It is responsible for combining perception information with the direction of travel to formulate future behavior strategies and trajectory planning. Traditional rule-based autonomous driving decision-making and planning algorithms are difficult to ensure algorithm robustness and decision-making safety when facing open road scenes with high dynamics, strong interactivity and uncertainty. Therefore, a series of decision-making and planning algorithms based on deep learning neural networks are widely used in the field of intelligent driving control.

[0003] At present, common autonomous driving systems usually optimize perception and decision planning as separate tasks. This method uses a layered progressive architecture coupling, with clear logic between different tasks and strong interpretability. However, this method also has certain shortcomings. For example, the separate optimization of different modules leads to error transmission between modules, making it difficult to achieve global optimization; at the same time, the perception task and decision module exist separately, resulting in conflicts between the perception task output and the decision module, further leading to the disconnection between perception and decision. To overcome the above problems, a series of scientific researchers have proposed end-to-end autonomous driving decision planning, which directly uses sensor information as model input, outputs future behavior strategies and planning trajectories, and uses deep learning models to optimize perception and decision planning modules at the same time. However, simple end-to-end autonomous driving models have the disadvantages of difficult model convergence and lack of model interpretability. To further overcome the above problems, relevant scholars have proposed an end-to-end autonomous driving decision model based on multi-task learning. By introducing perception and decision-related auxiliary tasks and adding perception-related constraints, the model's ability to perceive the surrounding environment is improved while increasing the interpretability of the model output. For this type of multi-task autonomous driving network, how to select appropriate auxiliary tasks to perceive environmental information and how to build an efficient multi-task coupling module are the key and difficulty in building a safe and efficient decision-making and planning model.

[0004] In view of this, the present invention proposes an end-to-end autonomous driving decision-making planning model that combines vectorized maps with multi-task feature embedding. Using the network paradigm of imitation learning, combined with vectorized maps and targets in the bird's-eye view space as perception auxiliary tasks, a multi-task feature embedding method is proposed to couple the perception and decision-making models to achieve a decision-making planning model that fully considers the surrounding environmental factors. A multi-time series cross-fusion module is further proposed, and historical multi-frame information is used to further enhance the security and stability of the decision-making model. Summary of the invention

[0005] This paper proposes an autonomous driving decision planning model based on vectorized maps and multi-task feature embedding. It mainly includes three design parts: environment perception module, multi-time series cross attention fusion module and decision planning prediction module. Specifically, it includes the following contents:

[0006] Step 1, create an imitation learning dataset: with the help of the CARLA-0.9.10 simulation simulator, set up different traffic scenes and driving tasks, and use a rule-based expert driving algorithm combined with a high-precision map to collect data. The main sensor information set includes six surround cameras and GPS navigation. The collected data types mainly include: image information (I0, I1, I2, I3, I4, I5) collected by six different cameras, each image size is (256, 256, 3); speed information at the current moment; target point at the next moment; navigation point information for the next T time steps obtained using expert experience And the expected speed output corresponding to each moment; perception task label: including high-precision map vector information within the lateral distance [32m, -32m] and longitudinal distance [32m, -32m] at each moment and the position information of other traffic participants in the bird's-eye view space within this interval, the semantic segmentation corresponding label of the front view and the label corresponding to the depth map.

[0007] Step 2: Build the environment perception module, which mainly includes three parts: 1) Build the feature extraction module; 2) Build the depth-semantic-image cross-attention fusion module; 3) Build the vectorized map and bird's-eye view target detection task branch.

[0008] Step three, for the feature extraction module in step two, the present invention adopts ResNet-50 as the main feature extraction network, and fuses multi-scale image features through the pyramid module. When the input image is (I0, I1, I2, I3, I4, I5), the corresponding feature maps extracted by the feature extraction module are (f0, f1, f2, f3, f4, f5).

[0009] Step 4: For the depth-semantic-image cross attention fusion module in step 2, the feature maps (f0, f1, f2, f3, f4, f5) collected in step 3 are input into the depth information extraction network and the semantic information extraction network respectively to obtain the corresponding depth information features. And semantic information features To ensure that the extracted information contains depth information and semantic information, a 1×1 dimensional convolution is used for dimensional transformation, and the collected semantic segmentation labels and depth information labels are combined for supervision. Then, the self-attention mechanism is used to respectively Depth Information Features And semantic information features For further feature encoding, the feature encoding process of three different information is similar. Taking image feature encoding as an example, the calculation formula is as follows:

[0010] Q=w q f i ,K=w k f i ,V=w v f i

[0011]

[0012] Among them, w q , w k , w v They represent the weight parameters that the model needs to learn, Softmax represents the activation function, and K T is the transposed matrix of K. Q represents the input f i The query features of K represent the input f i The corresponding key feature, V represents the input f i The corresponding value feature, Q I represents the output feature of self-attention, and d represents the feature dimension. The calculation process for deep coding and semantic coding is similar. According to the above process, the image coding feature Q can be obtained respectively. I , deep encoding feature Q d And the semantic encoding feature Q e .

[0013] In order to further integrate the above different information to achieve the bird's-eye view feature output of clear image features, depth features and semantic features, the present invention first initializes the query vector Q according to the vectorized map output and the target detection output s =[Q m ,Q o ], where Q m represents the vectorized map query vector. The dimension of the vectorized map query vector is set to 100×256 in the present invention, which means that there are at most 100 vectorized map information items that need to be defined within a 32m range. o represents the target detection query vector under the bird's-eye view. The dimension of the present invention is set to 50×256, which means that there are at most 50 other traffic participants within the surrounding 32m range. First, the initialized semantic encoding feature query vector Q s After the self-attention mechanism, the calculation process is as follows:

[0014] Q0=w q0 Q s ,K0=w k0Q s ,V0=w v0 Q s

[0015]

[0016] Among them, w q0 , w k0 , w v0 Represents the weight parameter that the model needs to learn, Q0 represents Q s The corresponding query feature, K represents Q s Corresponding key features, V represents input Q s The corresponding value feature, Softmax represents the activation function, and Q1 is the output query feature. Then Q1 is combined with the deep feature Q d Perform cross attention fusion:

[0017] Q′1=w q1 Q1,K′1=w k1 Q d ,V′1=w v1 Q d

[0018]

[0019] Among them, w q1 , w k1 , w v1 represents the weight parameter that the model needs to learn, Q′1 represents the query feature corresponding to Q1, and K′1 represents Q d The corresponding key feature, V′1 represents the input Q d The corresponding value feature, Q2, represents the output query feature. The output query feature Q2 obtained above is combined with the semantic feature Q e Perform cross attention fusion:

[0020] Q′2=w q2 Q2,K′2=w k2 Q e ,V′2=w v2 Q e

[0021]

[0022] Among them, w q2 , w k2 , w v2 represents the weight parameter that the model needs to learn, Q′2 represents the query feature corresponding to Q2, and K′2 represents Q e The corresponding bond characteristics, V′2 represents Q eThe corresponding value feature, Q3, represents the output query feature. The output query feature Q3 obtained above is combined with the image feature Q I Perform cross attention fusion:

[0023] Q′3=w q3 Q3,K′3=w k3 Q I ,V′3=w v3 Q I

[0024]

[0025] Among them, w q3 , w k3 , w v3 represents the weight parameter that the model needs to learn, Q′3 represents the query feature corresponding to Q3, and K′3 represents Q s The corresponding key feature, V′3 represents the input Q I The corresponding value feature, Q4, represents the output query feature. Through the above cross attention mechanism, we can get the query feature output Q4 that is fully integrated with the depth information and semantic information, and then pass Q4 through the forward propagation neural network to increase the output nonlinearity of the model, so as to get the final query feature output Q F .

[0026] Step 5: For the vectorized map and bird’s-eye view target detection task branches in step 2, construct output heads for different tasks and use the query feature Q in step 4 F As input, Q F According to the definition in step 4, it is divided into Q o and Q m , where the target detection output head uses a fully connected neural network combined with the query feature Q o Predict other traffic participants around, and the output is Where N1=50 represents the maximum number of other traffic participants around, and output P o It includes the predicted category of traffic participants, the length and width of the bird's-eye view space. For vectorized maps, a fully connected neural network is used in combination with the query feature Q m Predict the surrounding vector map information and output it as Where N2 = 100, representing the maximum predicted number of surrounding vectorized map information, output P m Contains the category information of the vectorized map and the predicted location information of ten points.

[0027] Step 6: Build a multi-temporal cross-attention fusion module. FTo further ensure its effectiveness, the present invention cross-attends the fusion with the feature information of historical moments to achieve the feature output of clear historical information. F Q with historical moments h Perform cross-attention feature fusion, which is calculated as follows:

[0028] Q′ F =w qf Q F ,K′ h =w kh Q h ,V′ h =w vh Q h

[0029]

[0030] where w qf , w kh , w vh Represents the weight parameter that the model needs to learn, Q′ F Representative Q F The corresponding query feature, K′ h Representative Q h The corresponding bond feature, V′ h Represents input Q h The corresponding value feature, Q″ F is the fused output feature. According to the above cross attention mechanism, the fusion of the current moment feature and the historical moment feature can be obtained, and then the above feature Q″ F Input into the forward feedback neural network to obtain the final agent feature output Q a , and update the historical moment feature Q h =Q a .

[0031] Step seven, build a decision-making planning prediction module, with the input being the agent features obtained after time series fusion in step six, which mainly includes two parts: 1) expected speed prediction model, 2) navigation point prediction model.

[0032] Step 8: For the expected speed prediction model in step 7, a Transformer-based model decoder is used. Specifically, first, in order to obtain the features for predicting the expected speed, the expected speed query features are set. And randomly initialize it and compare it with the obtained agent feature Q a Cross-fusion is performed, and the calculation method is as follows:

[0033] Q′ vp =w qa Qvp ,K′ vp =w ka Q a ,V′ vp =w va Q a

[0034]

[0035] where w qa , w ka , w va Represents the weight parameter that the model needs to learn, Q′ vp Representative Q vp The corresponding query feature, K′ vp Representative Q a The corresponding bond feature, V′ vp Represents input Q a The corresponding value feature, Q″ vp represents the desired speed characteristic of the output, and then Q″ vp The output is sent to the forward feedback neural network and the fully connected neural network to obtain the final expected speed prediction.

[0036] Step 9: For the navigation point prediction model in step 7, the Transformer-based model decoder is also used. Specifically, first, in order to obtain the features for predicting the navigation points, the navigation point query features are set And randomly initialized, where T represents the future prediction time step of the navigation point, which is set to 10 in the present invention. It is then input into the self-attention module, and the calculation method is as follows:

[0037] Q′ wp =w qw Q wp ,K′ kw =w ka Q wp ,V′ vw =w va Q wp

[0038]

[0039] Among them, w qw , w kw , w vw Represents the weight parameter that the model needs to learn, Q′ wp , K′ kw , V′ vw Represents input Q wp The corresponding query features, key features, and value features, Q″ wp is the output query feature and is combined with the obtained agent feature Qa Cross-fusion is performed, and the calculation method is as follows:

[0040] Q″′ wp =w′ qw Q″ wp ,K″ kw =w′ kw Q a ,V″ vw =w′ vw Q a

[0041]

[0042] Among them, w′ qw , w′ kw , w′ v Represents the weight parameter that the model needs to learn, Q″′ wp Represents Q″ wp The corresponding query feature, K″ kw Representative Q a The corresponding bond feature, V′ vp Represents input Q a The corresponding value feature, Represents the output navigation point features, and then The output is sent to the forward feedback neural network to increase the nonlinearity of the features and then input into the GRU time series prediction model to obtain the final navigation point prediction.

[0043] Step 10: The GRU time series prediction model in step 9 consists of ten GRU prediction units. Each GRU prediction unit combines a layer of fully connected neural network to output a navigation point output corresponding to the time step. Specifically, for the input navigation point feature Divide it into navigation point features at different time steps The target point w tar Input to the fully connected neural network to get As the initial hidden feature. Then the initial hidden feature and the initial time step navigation point feature Input to the first GRU prediction unit to get the output feature h 1 , and use it as the hidden feature of the next moment, and output the feature h 0 Input into the fully connected neural network to get the navigation point prediction at the initial moment And so on, the navigation point output for the next ten time steps is obtained.

[0044] Step 11, build a control execution module, and construct a control model based on the expected speed prediction obtained in step 8 and the navigation point prediction obtained in step 9 as input. The control model mainly includes two parts: longitudinal control for controlling the throttle and brake and lateral control for controlling the steering wheel angle.

[0045] Step 12: For vertical control, first set a threshold T v , when the predicted expected speed is less than T v When the brake is set to 1, the braking control is realized. Otherwise, the predicted expected speed is input into the PID controller to obtain the throttle opening at this time to realize the acceleration control.

[0046] Step 13: For lateral control, firstly, the predicted output of the navigation point is averaged, and the inverse tangent is calculated to obtain the predicted angle A at this time. p :

[0047]

[0048] Where (x i ,y i ) is the navigation point The coordinates of the predicted angle A are then p Input to the PID controller to get the steering wheel angle u p .

[0049] Step 13: Train the end-to-end autonomous driving multi-task decision-making planning model constructed above. Corresponding loss functions are constructed for different task modules. The depth estimation task and expected speed prediction use the L1 loss function, and the semantic segmentation uses the cross entropy loss function. The loss function for target detection and vectorized map construction includes two parts: the cross entropy loss function for category prediction and the L1 loss function for regression. Thus, the overall loss function can be obtained. for:

[0050]

[0051] in, They represent semantic segmentation, depth estimation, vectorized map construction, target detection, navigation point prediction, and expected speed prediction loss functions respectively.

[0052] Beneficial effects of the present invention:

[0053] (1) The present invention proposes an end-to-end autonomous driving decision-making planning model based on multi-task coupling, and designs six task branches including different modules such as perception, decision-making and control. The end-to-end autonomous driving framework is used to reduce the error between different model transmissions, and the multi-task constraint method is used to increase the model's fitting ability.

[0054] (2) The present invention combines vectorized maps and bird's-eye view target detection as the perceptual representation of the multi-task model. It provides the dynamic and static target information required in the bird's-eye view space, and unifies the environmental perception information and the navigation point output of the decision into the same space, thereby improving the model prediction accuracy.

[0055] (3) The present invention adopts a coupling method of multi-task feature embedding and uses the cross-attention mechanism between different tasks to transfer information, overcome the training conflict caused by simply sharing feature maps in traditional multi-task models, and achieve efficient coupling between different tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Network architecture diagram for combining vectorized maps with multi-task feature embedding;

[0057] Figure 2 It is the structure diagram of the control model; DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods, but the protection scope of the present invention is not limited thereto.

[0059] Figure 1 To combine the vectorized map with the multi-task feature embedding network architecture, it mainly includes three parts: perception module, multi-time series processing module and decision output model, as follows:

[0060] (1) Create an imitation learning dataset: Create an imitation learning dataset: With the help of the CARLA-0.9.10 simulation simulator, set different traffic scenes and driving tasks, and use a rule-based expert driving algorithm combined with a high-precision map to collect data. The main sensor information set includes six surround cameras and GPS navigation. The collected data types mainly include: image information (I0, I1, I2, I3, I4, I5) collected by six different cameras, each image size is (256, 256, 3); speed information at the current moment; target point at the next moment; navigation point information for the next T time steps obtained using expert experience And the expected speed output corresponding to each moment; perception task label: including high-precision map vector information within the lateral distance [32m, -32m] and longitudinal distance [32m, -32m] at each moment and the position information of other traffic participants in the bird's-eye view space within this interval, the semantic segmentation corresponding label of the front view and the label corresponding to the depth map.

[0061] (2) Build an environmental perception module, which mainly includes three parts: 1) Build a feature extraction module; 2) Build a depth-semantic-image cross-attention fusion module; 3) Build a vectorized map and bird's-eye view target detection task branch.

[0062] (3) The present invention adopts ResNet-50 as the main feature extraction network, and fuses multi-scale image features through the pyramid module. When the input image is (I0, I1, I2, I3, I4, I5), the corresponding feature maps extracted by the feature module are (f0, f1, f2, f3, f4, f5).

[0063] (4) Build a depth-semantic-image cross-attention fusion module and input the collected feature maps (f0, f1, f2, f3, f4, f5) into the deep information extraction network and the semantic information extraction network respectively to obtain the corresponding deep information features. And semantic information features To ensure that the extracted information contains depth information and semantic information, a 1×1 dimensional convolution is used for dimensional transformation, and the collected semantic segmentation labels and depth information labels are combined for supervision. Then, the self-attention mechanism is used to respectively Depth Information Features And semantic information features For further feature encoding, the feature encoding process of three different information is similar. Taking image feature encoding as an example, the calculation formula is as follows:

[0064] Q=w q f i ,K=w k f i ,V=w v f i

[0065]

[0066] Among them, w q , w k , w v They represent the weight parameters that the model needs to learn, Softmax represents the activation function, and K T is the transposed matrix of K. Q represents the input f i The query features of K represent the input f i The corresponding key feature, V represents the input f i The corresponding value feature, Q I represents the output feature of self-attention, and d represents the feature dimension. The calculation process for deep coding and semantic coding is similar. According to the above process, the image coding feature Q can be obtained respectively. I , deep encoding feature Qd And the semantic encoding feature Q e .

[0067] In order to further integrate the above different information to achieve the bird's-eye view feature output of clear image features, depth features and semantic features, the present invention first initializes the query vector Q according to the vectorized map output and the target detection output s =[Q m ,Q o ], where Q m represents the vectorized map query vector. The dimension of the vectorized map query vector is set to 100×256 in the present invention, which means that there are at most 100 vectorized map information items that need to be defined within a 32m range. o represents the target detection query vector under the bird's-eye view. The dimension of this invention is set to 50×256, which means that there are at most 50 other traffic participants within the surrounding 32m range. First, the query vector Q is initialized s After the self-attention mechanism, the calculation process is as follows:

[0068] Q0=w q0 Q s ,K0=w k0 Q s ,V0=w v0 Q s

[0069]

[0070] Among them, w q0 , w k0 , w v0 represents the weight parameter that the model needs to learn, Softmax represents the activation function, and d represents the number of feature dimensions. Then the output query feature Q1 and the depth feature Q d Perform cross attention fusion:

[0071] Q′1=w q1 Q1,K′1=w k1 Q d ,V′1=w v1 Q d

[0072]

[0073] Among them, w q0 , w k0 , w v0 Represents the weight parameter that the model needs to learn, Q0 represents Q s The corresponding query feature, K represents Q s Corresponding key features, V represents input Q sThe corresponding value feature, Softmax represents the activation function, d represents the number of dimensions of the feature, and Q1 is the output query feature. Then Q1 is combined with the deep feature Q d Perform cross attention fusion:

[0074] Q′1=w q1 Q1,K′1=w k1 Q d ,V′1=w v1 Q d

[0075]

[0076] Among them, w q1 , w k1 , w v1 represents the weight parameter that the model needs to learn, Q′1 represents the query feature corresponding to Q1, and K′1 represents Q d The corresponding key feature, V′1 represents the input Q d The corresponding value feature, Q2, represents the output query feature. The output query feature Q2 obtained above is combined with the semantic feature Q e Perform cross attention fusion:

[0077] Q′2=w q2 Q2,K′2=w k2 Q e ,V′2=w v2 Q e

[0078]

[0079] Among them, w q2 , w k2 , w v2 represents the weight parameter that the model needs to learn, Q′2 represents the query feature corresponding to Q2, and K′2 represents Q e The corresponding key feature, V′2 represents the input Q e The corresponding value feature, Q3, represents the output query feature. The output query feature Q3 obtained above is combined with the image feature Q I Perform cross attention fusion:

[0080] Q′3=w q3 Q3,K′3=w k3 Q I ,V′3=w v3 Q I

[0081]

[0082] Among them, w q3 , wk3 , w v3 represents the weight parameter that the model needs to learn, Q′3 represents the query feature corresponding to Q3, and K′3 represents Q s The corresponding key feature, V′3 represents the input Q I The corresponding value feature, Q4, represents the output query feature. Through the above cross attention mechanism, we can get the query feature output Q4 that is fully integrated with the depth information and semantic information, and then pass Q4 through the forward propagation neural network to increase the output nonlinearity of the model, so as to get the final query feature output Q F .

[0083] (5) Construct vectorized map and bird's-eye view target detection task branches, using query feature Q F As input, Q F According to the definition in step 4, it is divided into Q o and Q m , where the target detection output head uses a fully connected neural network combined with the query feature Q o Predict other traffic participants around, and the output is Where N1=50 represents the maximum number of other traffic participants around, and the output includes the category, the length and width of the bird's-eye view space. For vectorized maps, a fully connected neural network is used in combination with the query feature Q m Predict the surrounding vector map information and output it as Where N2=100, representing the maximum predicted number of surrounding vectorized map information, and the output includes the category information of the vectorized map and the predicted position information of ten points.

[0084] (6) Build a multi-temporal cross-attention fusion module. After obtaining the query feature Q that can effectively perceive the surrounding environment information F To further ensure its effectiveness, the present invention cross-attends the fusion with the feature information of historical moments to achieve the feature output of clear historical information. F Q with historical moments h Perform cross-attention feature fusion, which is calculated as follows:

[0085] Q′ F =w qf Q F ,K′ h =w kh Q h ,V′ h =w vh Q h

[0086]

[0087] where w qf , w kh , w vh Represents the weight parameter that the model needs to learn, Q′ F Representative Q F The corresponding query feature, K′ h Representative Q h The corresponding bond feature, V′ h Represents input Q h The corresponding value feature, Q″ F is the fused output feature. According to the above cross attention mechanism, the fusion of the current moment feature and the historical moment feature can be obtained, and then the above feature Q″ F Input into the forward feedback neural network to obtain the final agent feature output Q a , and update the historical moment feature Q h =Q a .

[0088] (5) Build a decision-making planning prediction module, with the input being the agent features obtained after time series fusion in step 6, which mainly includes two parts: 1) expected speed prediction model, and 2) navigation point prediction model.

[0089] (7) A Transformer-based model decoder is used to construct an expected speed prediction model. Specifically, first, in order to obtain the features used to predict the expected speed, the expected speed query features are set And perform random initialization, cross-fusion with the obtained agent features, and the calculation method is as follows:

[0090] Q′ vp =w qa Q vp ,K′ vp =w ka Q a ,V′ vp =w va Q a

[0091]

[0092] where w qa , w ka , w va Represents the weight parameter that the model needs to learn, Q′ vp Representative Q vp The corresponding query feature, K′ vp Representative Q a The corresponding bond feature, V′ vp Represents input Q a The corresponding value feature, Q″ vprepresents the desired speed characteristic of the output, and then Q″ vp The output is sent to the forward feedback neural network and the fully connected neural network to obtain the final expected speed prediction.

[0093] (8) A Transformer-based model decoder is used to build a navigation point prediction model. Specifically, in order to obtain the features for predicting navigation points, the navigation point query features are set. And randomly initialized, where T represents the future prediction time step of the navigation point, which is set to 10 in the present invention. It is then input into the self-attention module, and the calculation method is as follows:

[0094] Q′ wp =w qw Q wp ,K′ kw =w ka Q wp ,V′ vw =w va Q wp

[0095]

[0096] Among them, w qw , w kw , w vw Represents the weight parameter that the model needs to learn, Q′ wp , K′ kw , V′ vw Represents input Q wp The corresponding query features, key features, and value features, Q″ wp is the output query feature and is combined with the obtained agent feature Q a Cross-fusion is performed, and the calculation method is as follows:

[0097] Q″′ wp =w′ qw Q″ wp ,K″ kw =w′ kw Q a ,V″ vw =w′ vw Q a

[0098]

[0099] Among them, w′ qw , w′ kw , w′ vw Represents the weight parameter that the model needs to learn, Q″′ wp Represents Q″ wp The corresponding query feature, K″kw Representative Q a The corresponding bond feature, V′ vp Represents input Q a The corresponding value feature, Represents the output navigation point features, and then The output is sent to the forward feedback neural network to increase the nonlinearity of the feature, and then input into the GRU time series prediction model to obtain the final navigation point prediction. The GRU time series prediction model consists of ten GRU prediction units, each of which is combined with a layer of fully connected neural network to output a navigation point output corresponding to the time step. Specifically, for the input navigation point feature Divide it into navigation point features at different time steps Take the target point w tar Input to the fully connected neural network to get As the initial hidden feature. Then the initial hidden feature and the initial time step navigation point feature Input to the first GRU prediction unit to get the output feature h 1 , and use it as the hidden feature of the next moment, and output the feature h 0 Input into the fully connected neural network to get the navigation point prediction at the initial moment And so on, the navigation point output for the next ten time steps is obtained.

[0100] (10) Build a control execution model. Based on the expected speed prediction and navigation point prediction, a control model is constructed. The control model mainly includes two parts: longitudinal control for controlling the throttle and brake and lateral control for controlling the steering wheel angle. The network structure is as follows: Figure 2 For longitudinal control, first set a threshold T v , when the predicted expected speed is less than T v , set the brake at this time to 1, otherwise input the predicted expected speed into the PID controller to obtain the throttle opening at this time. For lateral control, first calculate the average value of the predicted output of the navigation point, and calculate its tangent to obtain the predicted angle A at this time p :

[0101]

[0102] Where (x i ,y i ) is the navigation point The coordinates of the predicted angle A are then obtained. p Input to the PID controller to get the steering wheel angle u p .

[0103] (11) Train the end-to-end autonomous driving multi-task model constructed above. Corresponding loss functions are constructed for different task modules. The depth estimation task and expected speed prediction use the L1 loss function, the semantic segmentation uses the cross entropy loss function, and the loss function for target detection and vectorized map construction includes two parts: the cross entropy loss function for category prediction and the L1 loss function for regression. Thus, the overall loss function can be obtained: for:

[0104]

[0105] in, They represent semantic segmentation, depth estimation, vectorized map construction, target detection, navigation point prediction, and expected speed prediction loss functions respectively.

[0106] (12) The embodiments of the present invention uniformly propose an intelligent driving vehicle device, which utilizes a trained model in combination with an on-board industrial computer, a central controller, a camera and other equipment to implement the autonomous driving decision-making planning described in the invention.

[0107] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. An end-to-end autonomous driving decision-making and planning model combining vectorized maps and multi-task feature embedding, characterized in that: Obtained by: S1, build the environment perception module, which mainly includes: feature extraction module, depth-semantic-image cross attention fusion module, and vectorized map and bird's-eye view target detection task branch module; The feature extraction module is used to extract features of the input image; The depth-semantic-image cross attention fusion module obtains the corresponding depth information features and semantic information features according to the features extracted by the feature extraction module, and encodes the extracted image features, depth information features and semantic information features to obtain image encoding features, depth encoding features and semantic encoding features, and performs information fusion to obtain the final query feature Q F ; The vectorized map and bird's-eye view object detection task branch modules take query features as input, construct output heads for different tasks, and ultimately output categories, length and width in the bird's-eye view space, category information of the vectorized map, and predicted location information of ten points; S2, build a multi-temporal cross-attention fusion module, and obtain the query feature Q that can effectively perceive the surrounding environment information. F Finally, in order to further ensure its effectiveness, it is cross-attended and fused with the feature information of historical moments to obtain the time-series fused agent feature Q a , in order to achieve the characteristic output of clear historical information; S3, build a decision planning prediction module, which mainly includes two parts: 1) expected speed prediction model, 2) navigation point prediction model; The expected speed prediction model cross-integrates the expected speed query features with the intelligent agent features of time series fusion to obtain the expected speed prediction; The navigation point prediction model first performs a self-attention operation on the navigation point query features, and then cross-integrates them with the agent features, and performs GRU time series prediction to obtain the navigation point prediction.

2. The end-to-end autonomous driving decision planning model combining vectorized map and multi-task feature embedding according to claim 1, characterized in that: In S1: the feature extraction module uses ResNet-50 as the main feature extraction network, and fuses multi-scale image features through the pyramid module. When the input image is (I0, I1, I2, I3, I4, I5), the corresponding feature maps extracted by the feature extraction module are (f0, f1, f2, f3, f4, f5); The depth-semantic-image cross attention fusion module inputs the feature maps (f0, f1, f2, f3, f4, f5) extracted by the feature extraction module into the depth information extraction network and the semantic information extraction network respectively to obtain the corresponding depth information features. And semantic information features To ensure that the extracted information contains depth information and semantic information, a 1×1 dimensional convolution is used for dimensional transformation, and the collected semantic segmentation labels and depth information labels are combined for supervision. Then, the self-attention mechanism is used to respectively Depth Information Features And semantic information features For further feature encoding, the feature encoding process of the three different information is the same. Taking image feature encoding as an example, the calculation formula is as follows: Q0=w q0 Q s ,K0=w k0 Q s ,V0=w v0 Q s Among them, w q , w k , w v They represent the weight parameters that the model needs to learn, Softmax represents the activation function, and K T is the transposed matrix of K. Q represents the input f i The query features of K represent the input f i The corresponding key feature, V represents the input f i The corresponding value feature, Q I represents the output feature of self-attention, and d represents the feature dimension. For deep coding and semantic coding, the calculation process is similar. According to the above process, the image coding feature Q can be obtained respectively. I , deep encoding feature Q d And the semantic encoding feature Q e ; In order to fuse the above different information to achieve the bird's-eye view feature output of clear image features, depth features and semantic features, the query vector Q is first initialized according to the vectorized map output and the target detection output. s =[Q m ,Q o ], where Q m represents the vectorized map query vector. The dimension of the vectorized map query vector is set to 100×256 in the present invention, which means that there are at most 100 vectorized map information items that need to be defined within a 32m range. o represents the target detection query vector under the bird's-eye view. The dimension of the present invention is set to 50×256, which means that there are at most 50 other traffic participants within the surrounding 32m range. First, the initialized query vector Q s After the self-attention mechanism, the calculation process is as follows: Q0=w q0 Q s ,K0=w k0 Q s ,V0=w v0 Q s Among them, w q0 , w k0 , w v0 Represents the weight parameter that the model needs to learn, Q0 represents Q s The corresponding query feature, K represents Q s Corresponding key features, V represents input Q s The corresponding value feature, Softmax represents the activation function, and Q1 is the output query feature. Then the output query feature Q1 is combined with the depth feature Q d Perform cross attention fusion: Q′1=w q1 Q1,K′1=w k1 Q d ,V′1=w v1 Q d Among them, w q1 , w k1 , w v1 represents the weight parameter that the model needs to learn, Q′1 represents the query feature corresponding to Q1, and K′1 represents Q d The corresponding key feature, V′1 represents the input Q d The corresponding value feature, Q2, represents the output query feature. The output query feature Q2 obtained above is combined with the semantic feature Q e Perform cross attention fusion: <h2 style=";text-align:left;direction:ltr">Q′2 = w<h2 style=";text-align:left;direction:ltr"> q2 <h2 style=";text-align:left;direction:ltr"> Q2,K′2 = w<h2 style=";text-align:left;direction:ltr"> k2 <h2 style=";text-align:left;direction:ltr"> Q<h2 style=";text-align:left;direction:ltr"> s <h2 style=";text-align:left;direction:ltr"> V′2 = w<h2 style=";text-align:left;direction:ltr"> v2 <h2 style=";text-align:left;direction:ltr"> Q<h2 style=";text-align:left;direction:ltr"> s Among them, w q2 , w k2 , w v2 represents the weight parameter that the model needs to learn, Q′2 represents the query feature corresponding to Q2, and K′2 represents Q e The corresponding bond characteristics, V′2 represents Q e The corresponding value feature, Q3, represents the output query feature. The output query feature Q3 obtained above is combined with the image feature Q I Perform cross attention fusion: Q′3=w q3 Q3,K′3=w k3 Q I ,V′3=w v3 Q I Among them, w q3 , w k3 , w v3 represents the weight parameter that the model needs to learn, Q′3 represents the query feature corresponding to Q3, and K′3 represents Q s The corresponding key feature, V′3 represents the input Q I The corresponding value feature, Q4, represents the output query feature. Through the above cross attention mechanism, we can get the query feature output Q4 that is fully integrated with the depth information and semantic information, and then pass Q4 through the forward propagation neural network to increase the output nonlinearity of the model, so as to get the final query feature output Q F . .

3. The end-to-end autonomous driving decision planning model combining vectorized map and multi-task feature embedding according to claim 2, characterized in that: The vectorized map and bird's-eye view target detection task branch modules in S1 build output heads for different tasks and convert the query feature Q in S3 F As input, Q F Divided into Q o and Q m , where the target detection output head uses a fully connected neural network combined with the query feature Q o Predict other traffic participants around, and the output is Where N1=50 represents the maximum number of other traffic participants around, and output P o Contains categories, length and width in bird's-eye view space; for vector maps, a fully connected neural network is used in combination with query features Q m Predict the surrounding vector map information and output it as Where N2 = 100, representing the maximum predicted number of surrounding vectorized map information, output P m Contains the category information of the vectorized map and the predicted location information of ten points.

4. The end-to-end autonomous driving decision planning model combining vectorized map and multi-task feature embedding according to claim 1, characterized in that: The specific implementation of S2 includes: Using the current Q F Q with historical moments h Perform cross-attention feature fusion, which is calculated as follows: Q′ F =w qf Q F ,K′ h =w kh Q h ,V′ h =w vh Q h where w qf , w kh , w vh Represents the weight parameter that the model needs to learn, Q′ F Representative Q F The corresponding query feature, K′ h Representative Q h The corresponding bond feature, V′ h Represents input Q h The corresponding value feature, Q″ F is the fused output feature. According to the above cross attention mechanism, the fusion of the current moment feature and the historical moment feature can be obtained, and then the above feature Q″ F Input into the forward feedback neural network to obtain the final agent feature output Q a , and update the historical moment feature Q h =Q a .

5. The end-to-end autonomous driving decision planning model combining vectorized map and multi-task feature embedding according to claim 1, characterized in that: The expected speed prediction model of S3 adopts a Transformer-based model decoder. First, in order to obtain the features for predicting the expected speed, the expected speed query features are set And randomly initialize it and compare it with the obtained agent feature Q a Cross-fusion is performed, and the calculation method is as follows: Q′ vp =w qa Q vp ,K′ vp =w ka Q a ,V′ vp =w va Q a where w qa , w ka , w va Represents the weight parameter that the model needs to learn, Q″ vp represents the desired speed characteristic of the output, and then Q″ vp Output to the forward feedback neural network and the fully connected neural network to obtain the final expected speed prediction; The navigation point prediction model adopts a Transformer-based model decoder. First, in order to obtain the features for predicting the navigation points, the navigation point query features are set. And randomly initialized, where T represents the future predicted time step of the navigation point, which is then input into the self-attention module, which is calculated as follows: Q′ wp =w qw Q wp ,K′ kw =w ka Q wp ,V′ vw =w va Q wp Among them, w qw , w kw , w vw Represents the weight parameter that the model needs to learn, Q′ wp , K′ kw , V′ vw Represents input Q wp The corresponding query features, key features, and value features, Q″ wp is the output query feature and is combined with the obtained agent feature Q a Cross-fusion is performed, and the calculation method is as follows: Q″′ wp =w′ qw Q″ wp ,K″ kw =w′ kw Q a ,V″ vw =w′ vw Q a Among them, w′ qw , w′ kw , w′ vw Represents the weight parameter that the model needs to learn, Q″′ wp Represents Q″ wp The corresponding query feature, K″ kw Representative Q a The corresponding bond feature, V′ vp Represents input Q a The corresponding value feature, Represents the output navigation point features, and then The output is sent to the forward feedback neural network to increase the nonlinearity of the features and then input into the GRU time series prediction model to obtain the final navigation point prediction.

6. The end-to-end autonomous driving decision-making planning model combining vectorized map and multi-task feature embedding according to claim 5, characterized in that: The GRU time series prediction model includes ten GRU prediction units, each of which combines a layer of fully connected neural network to output a navigation point output corresponding to the time step; specifically, for its input navigation point feature Divide it into navigation point features at different time steps Take the target point w tar Input to the fully connected neural network to get As the initial hidden feature; Then the initial hidden features and the initial time step navigation point features are combined Input to the first GRU prediction unit to get the output feature h 1 , and use it as the hidden feature of the next moment, and output the feature h 0 Input into the fully connected neural network to get the navigation point prediction at the initial moment And so on, the navigation point output for the next ten time steps is obtained.

7. The end-to-end autonomous driving decision planning model combining vectorized map and multi-task feature embedding according to claim 1, characterized in that: It also includes S4, which builds a control execution module to achieve longitudinal control of the accelerator and brake and lateral control of the steering wheel angle based on the expected speed prediction and navigation point prediction; For longitudinal control, first set a threshold T v , when the predicted expected speed is less than T v When the brake is set to 1, the braking control is realized; otherwise, the predicted expected speed is input into the PID controller to obtain the corresponding throttle opening to realize the acceleration control; For lateral control, firstly, the predicted output of the navigation point is averaged, and the inverse tangent is calculated to obtain the predicted angle A at this time. p : Where (x i ,y i ) is the navigation point The coordinates of the predicted angle A are then obtained. p Input to the PID controller to get the steering wheel angle u p .

8. The end-to-end autonomous driving decision planning model combining vectorized map and multi-task feature embedding according to claim 1, characterized in that: It also includes a dataset for making model training: With the help of the CARLA-0.9.

10. simulation simulator, different traffic scenes and driving tasks were set, and a rule-based expert driving algorithm combined with a high-precision map was used for data collection. The set sensor information included six surround-view cameras and a GPS navigator; the collected data types included: image information (I0, I1, I2, I3, I4, I5) collected by six different cameras, each image size was (256, 256, 3), speed information at the current moment, target point at the next moment, and navigation point information for the next T time steps obtained using expert experience And the expected speed output corresponding to each moment, perception task label: including high-precision map vector information within the lateral distance [32m, -32m] and longitudinal distance [32m, -32m] at each moment and the position information of other traffic participants in the bird's-eye view space within this interval, the semantic segmentation corresponding label of the front view and the label corresponding to the depth map.

9. The end-to-end autonomous driving decision planning model combining vectorized map and multi-task feature embedding according to claim 8, characterized in that: It also includes: using the prepared data set to train the model, constructing corresponding loss functions for different task modules, wherein the depth estimation task and expected speed prediction use the L1 loss function, the semantic segmentation uses the cross entropy loss function, and the loss function for target detection and vectorized map construction includes two parts: the cross entropy loss function for category prediction, and the L1 loss function for regression, so that the overall loss function can be obtained. for: in, They represent semantic segmentation, depth estimation, vectorized map construction, target detection, navigation point prediction, and expected speed prediction loss functions respectively.

10. A control device for an intelligent driving vehicle, characterized in that: The device deploys an end-to-end autonomous driving decision-making planning model that combines vectorized maps and multi-task feature embedding, which is trained according to claim 9.

Images