An end-to-end automatic driving decision planning method and device combining meta-learning multi-task optimization

Abstract

The application discloses an end-to-end automatic driving decision planning method and equipment combined with meta-learning multi-task optimization, adopts an end-to-end learning mode to overcome error transmission, simultaneously optimizes the convergence of a model by using a multi-task constraint method, and further increases the explainability of the model by multi-branch output results of the multi-task. Eight different sub-task branches are designed, including a perception task for an environment and a cognition for a self state, and a model of the sub-task is used to convert original sensor features to a bird's-eye view space, thereby improving the accuracy of decision planning prediction. Further, in order to optimize different multi-task combinations, firstly, a multi-task closeness is used to evaluate different task combinations to obtain initial task weights, and then a multi-task optimization method based on the combination of meta-learning is used to dynamically adjust the weights between different tasks in the training process, so that the optimization of the main task prediction is realized.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving for intelligent vehicles, and relates to an end-to-end autonomous driving decision planning method and device that combines meta-learning multi-task optimization. Background Technology

[0002] Autonomous driving systems, a recent research hotspot, have attracted widespread attention from scholars and related enterprises. A well-developed autonomous driving system is crucial for optimizing travel and improving driving safety. The decision-making and planning module, a vital component of autonomous driving systems, integrates perceived information, plans future paths, and executes commands, acting as the "brain" of the system and ensuring its effectiveness. Current autonomous driving strategies mostly employ rule-based decision-making and planning algorithms; however, established, simple logical rules often exhibit poor robustness and minimal efficiency in complex operating conditions and dynamic scenarios. In recent years, with the rise of deep learning, data-driven imitation learning algorithms have achieved remarkable results in the field of decision-making and planning.

[0003] Imitation learning is a widely adopted deep learning approach in the field of autonomous driving. It uses sensor information as model input and expert driving commands as labels, employing supervised learning to constrain the model to fit expert experience, thereby achieving the effect of mimicking expert driving. Currently, common imitation learning approaches can be divided into hierarchical and end-to-end methods. Hierarchical approaches progressively input sensor information into the perception module, cognitive prediction module, and decision-making and planning module. While this hierarchical approach optimizes the convergence of decision-making and planning, it suffers from error propagation and model redundancy. End-to-end algorithms directly output decision commands from sensor input, but they are difficult to converge and have poor model interpretability. To overcome these problems, researchers have proposed autonomous driving decision-making algorithms based on multi-task constraints. These algorithms overcome error propagation through deep learning end-to-end models and optimize model convergence using multi-task constraints. The multi-branch output results from multiple tasks further increase the model's interpretability. However, simple multi-task learning may have negative effects, leading to local optimization during training and reducing prediction accuracy across different tasks.

[0004] In view of this, this invention proposes an end-to-end driving decision-making and planning method based on multi-task optimization. First, a multi-task autonomous driving decision-making and planning model based on vision combined with perception prediction is proposed. Simultaneously, to optimize the weights of different tasks in multi-task learning, decision-making and planning are selected as the main task. Combining a multi-task affinity evaluation criterion, a meta-learning-based multi-task optimization method is proposed, dynamically adjusting the weights of different tasks during training to achieve optimal prediction of the main task. Summary of the Invention

[0005] This invention proposes an end-to-end autonomous driving decision-making and planning method combining meta-learning and multi-task optimization. It mainly comprises two components: 1. Employing a multi-task learning approach, by adding multi-task constraints such as perception and prediction, sharing sensor features, and optimizing the prediction accuracy of the decision-making and planning model. Simultaneously, the output results of multiple tasks further enhance the interpretability of the end-to-end model. 2. Employing a meta-learning-based multi-task optimization algorithm to dynamically adjust the combination relationships between different tasks, thereby achieving optimization of the main task. The specific steps are as follows:

[0006] Step one involves selecting a set of potential subtasks. The multi-task branches constructed in this invention mainly include nine subtasks: throttle prediction, steering wheel prediction, brake prediction, bird's-eye view prediction, object detection, depth map prediction, semantic segmentation prediction, heading angle prediction, and driving speed prediction. Among these, throttle prediction, steering wheel prediction, and brake prediction are the main tasks of this invention, while the remaining six branches serve as auxiliary tasks.

[0007] Step two: Prepare the dataset based on the selected sub-task. The input information for the decision-making and programming model mainly includes sensor information and driving commands. The sensors used in this invention mainly include three cameras (I, II, III, IV, V, V) on the front, left, and right. f ,I l ,I r Each camera captures an image of size (256, 256, 3). It also needs to collect the vehicle's speed information and driving commands at each moment, specifically: forward, left turn, right turn, lane change, and U-turn. The model's labeling information mainly includes: 1) information on the driver's decision-making actions during manual driving, including throttle input, steering wheel angle, and braking; 2) a bird's-eye view and target detection location information within a 32m radius around the vehicle; 3) depth maps and semantic segmentation labels corresponding to the three camera views. The bird's-eye view, semantic segmentation, and depth map labels are of size (256, 256, 3). The semantic segmentation category information for the bird's-eye view includes six categories (vehicles, roads, lane lines, red lights, green lights, and others).

[0008] Step 3: Construct a multi-task-based autonomous driving decision-making model. To optimize model convergence, multi-task constraints are introduced to share some network features. Furthermore, to achieve cross-domain prediction from sensor image input to control output, this invention transforms the pre-constructed feature map into a bird's-eye view space for prediction of some task branches. The model of this invention mainly includes five task modules: 1) image feature extraction module, 2) state feature extraction module, 3) depth estimation and semantic segmentation prediction module, 4) bird's-eye view and 3D object detection module, and 5) action and state prediction module.

[0009] Step 4: For the image feature extraction module in Step 3: Select ResNet-50 pre-trained on the ImageNet dataset as the image encoder. Input the camera information into the image encoder to obtain the image features f. I and embedded features f e ;

[0010] For the state extraction module: a two-layer fully connected layer is used to decode the vehicle's speed and driving commands, and then the obtained intermediate features are compared with the embedded features f. e The state feature f is obtained by performing a connection. s .

[0011] Step 5: For the depth estimation and semantic segmentation prediction modules in Step 3, this invention uses two decoder branches to decode the feature map to obtain the prediction result. The decoder mainly includes three upsampling layers and two convolutional layers. The semantic segmentation convolutional layer uses the Softmax activation function, and the depth estimation output uses the Sigmoid activation function.

[0012] Step Six: For the bird's-eye view and 3D object detection modules in Step Three, use the viewpoint transformation module to transform the feature map f... I The feature map X is obtained by converting it to a bird's-eye view space. bev The system uses a decoder to obtain the bird's-eye view prediction result and an object detection prediction head to obtain the object detection result. The viewpoint conversion module mainly consists of two parts: a dimensional viewpoint conversion module and a dimensional transformation module. The specific method of the dimensional viewpoint conversion module is as follows:

[0013] First, feature f I The input is processed by spatial average pooling and fully connected layers to obtain the dimension weights M. c Then combine it with feature map f I Multiply and then perform pointwise convolution to obtain the transformed dimensional feature vector X. c The specific calculation method is as follows:

[0014] M c =Mlp(Avergepool) c (f I )) (1)

[0015]

[0016] Among them, Avergepool c This indicates spatial average pooling, Mlp represents a fully connected layer, and FPW represents pointwise convolution.

[0017] Subsequently, a similar spatial transformation is applied to the dimensional feature vector, followed by dimensional average pooling to obtain the dimensional average vector. This vector is then expanded dimensionally and transformed again using a fully connected layer to obtain the spatial weights M. s Then combine it with the feature vector X c Multiplying and then passing through depthwise convolution, the resulting features are finally transformed using a dimension transformation module to obtain the original dimensions, resulting in the final output X. bev The specific calculation method is as follows:

[0018] M s =Mlp(Reshape(Avergepool) s (X c (3)

[0019]

[0020] Among them, Reshape and Reshape b This represents a dimensionality transformation operation, followed by a decoder upsampling the feature map to the original label size to obtain the bird's-eye view prediction result.

[0021] The 3D target detection module for surrounding vehicles first identifies the aforementioned bird's-eye view features X. bev A location feature map P∈[0,1] is obtained through convolutional layer operations. 64×64 The location feature map is used to determine whether a point exists as a target in the bird's-eye view space. Then, the surrounding 360-degree direction is roughly divided into 12 directions (30 degrees each), thus obtaining the approximate directions of surrounding vehicles through classification. Finally, a regression feature map is predicted. The regression prediction graph mainly contains three types of information: vehicle size. Location information deviation and deviation of angle information The positional information deviation is used to correct errors in the positional feature map, while the angle information deviation is used to correct errors caused by estimating the rough direction.

[0022] Step 7: For the action and state prediction module in Step 3: First, use a flattening operation to transform the bird's-eye view features X bev Expand into a one-dimensional feature vector, and then pass through a fully connected operation to obtain the intermediate feature f. c , will the intermediate feature f c With state feature f s The hybrid feature f is obtained by splicing. m =[f c ,f s ], then f m The output speed p after full connection is obtained from the feature output. v and heading angle prediction po At the same time, f m The output action decision prediction A is obtained through a fully connected and gated recurrent neural network (GRU). The action prediction specifically includes accelerator prediction, steering wheel prediction, and brake prediction. The time step of the gated recurrent neural network is set to 4.

[0023] Step 8: Train the multi-task autonomous driving decision network model constructed above. First, construct loss functions for different task branches. Depth prediction uses the L1 loss function, semantic segmentation and bird's-eye view construction use the cross-entropy loss function, and object detection's loss function includes three parts: binary cross-entropy loss function for position encoding, multi-class cross-entropy loss function for coarse direction estimation, and L1 loss function for regression features. L1 loss function is used for steering wheel, throttle, and yaw angle prediction, as well as driving speed prediction; and binary cross-entropy loss function is used for braking prediction. Therefore, the overall loss function can be obtained as follows:

[0024]

[0025] in These represent the loss functions for throttle, steering wheel, and brakes, respectively. These represent the loss functions for semantic segmentation, bird's-eye view prediction, depth prediction, velocity prediction, and heading angle prediction, respectively, with λ... i (i = 1, 2, 3, 4, 5) are the corresponding weights.

[0026] Step nine involves multi-task optimization of the constructed model. After obtaining the aforementioned loss function, to further optimize the model and improve the prediction accuracy of the main task, this invention uses task relevance to evaluate the influence of different sub-tasks on the main task. The specific calculation method is as follows: First, based on the attached... Figure 2 The model parameters are divided into task-specific parameters. and shared network layer parameters θ s Taking the calculation of task affinity between task A and the main task as an example, assume the loss function of the main task is... The loss of task A at time t is L t (A) Use the loss obtained at this time to update the feature-specific parameters and shared network layer parameters of task A. Simultaneously utilize the updated shared network layer parameters and the main task-specific parameters from the previous moment. Obtain the updated main task loss function Finally, the task relevance R between task i and the main task is calculated using the following method. i :

[0027]

[0028] Where lr represents the learning rate at this point, and when the task relevance R... i When the value is positive, it indicates that updating the shared parameters for task i is beneficial to reducing the loss of the main task; conversely, it proves that updating the shared parameters for task i is not beneficial to reducing the loss of the main task.

[0029] Step 10: Determine the initial multi-task relationship weights based on task relevance. However, the relationships between different tasks are not fixed during training. Therefore, to further dynamically adjust the relationships between multi-tasks, a meta-learning-based multi-task optimization algorithm is used to update the weights of the multi-tasks, and the dataset is divided into a training set X. train and validation set X val The training set is used to optimize the model parameters, and the validation set is used to optimize the weights between multiple tasks. Assume the training batch at time t is... The validation set data is First, the model parameters are updated using the training set loss. The specific calculation formula is as follows:

[0030]

[0031] Where θ t Let α represent the model parameters at time t, and α represent the learning rate. This represents the weight value for task i at time t-1. This represents the shared parameters of different tasks at time t-1. This represents the model-specific parameters for task i at time t-1. The multi-task weights of the model are then adjusted using the validation set loss, calculated as follows:

[0032]

[0033] Where β is the learning rate of meta-learning. Finally, the model parameters are further updated by combining the updated multi-task weights and the new training set loss, calculated as follows:

[0034]

[0035] The optimized multi-task weights can be obtained through the above calculation method. However, this method requires two iterations, consuming significant computational resources and GPU memory. To simplify the process, an approximate iterative approach is used to replace part of the gradient descent process. Assuming L0(θ,λ), where L(θ,λ) represents the main task loss and the overall loss of all tasks under the weight combination λ, the gradient update for λ can be approximated by the following calculation process:

[0036]

[0037] in This represents an update of network parameters. ε is a very small value. λ = 1 means that all tasks have the same weight.

[0038] Step 11: For the above model training process, the basic settings are as follows: both the model training and meta-learning optimizer are selected as the Adam optimizer, the initial learning rate is set to 0.0001, the number of training batches is set to 12, the total number of training rounds is 100, and the ratio of training set to validation set is 10:1, that is, every ten training cycles, a weight adjustment learning between multiple tasks is performed.

[0039] Step 12: Utilize the trained model in conjunction with the autonomous driving central controller and onboard cameras to implement the aforementioned autonomous driving decision-making and planning method.

[0040] The beneficial effects of this invention are:

[0041] (1) This invention proposes a multi-task optimized end-to-end autonomous driving decision planning method. While reducing model error propagation, it accelerates model convergence by using multi-task constraints. At the same time, the output of multi-task branches further increases the solvability and interpretability of the end-to-end model.

[0042] (2) In the end-to-end autonomous driving decision planning model based on multi-task optimization, this invention designs eight different sub-task branches, including the task of perceiving the environment and the cognition of its own state. By designing the sub-task model, the original sensor features are transformed into a bird's-eye view space, which improves the accuracy of decision planning and prediction.

[0043] (3) The present invention adopts a multi-task optimization model based on task relevance and meta-learning. First, the relevance between different tasks is evaluated using task relevance. Based on the evaluation results, the initial relationship between different tasks is determined. Then, a multi-task optimization algorithm based on meta-learning is used to realize dynamic adjustment between different sub-tasks, thereby further improving the prediction accuracy of the main task. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the end-to-end autonomous driving decision-making and planning process based on meta-learning multi-task optimization.

[0045] Figure 2 Multi-task autonomous driving decision-making and planning model. Detailed Implementation

[0046] The invention will now be further described with reference to the accompanying drawings.

[0047] Figure 1 This is a schematic diagram of the overall end-to-end autonomous driving decision-making and planning process based on multi-task optimization. The specific implementation process is as follows:

[0048] (1) Selection of sub-task set: The multi-task branch constructed in this invention mainly includes eight sub-tasks: throttle prediction, steering wheel prediction, brake prediction, bird's-eye view prediction, target detection, depth map prediction, semantic segmentation prediction, heading angle prediction, and driving speed prediction. Among them, throttle prediction, steering wheel prediction, and brake prediction are the main tasks of this invention, while the remaining six branches are the auxiliary tasks of this invention.

[0049] (2) Dataset Construction: The model's input information mainly includes sensor information and driving commands. The sensors used in this invention are primarily vision-based, including three cameras (I, II, III, IV, V, V) on the front, left, and right. f ,I l ,I r Each camera captures an image of size (256, 256, 3), and needs to collect driving commands at each moment, including (forward, left turn, right turn, U-turn). The model's labeling information includes the expert's driving command information, namely the expert's throttle input, steering wheel angle, and whether braking is used at the current moment (the three main task labels), a bird's-eye view of the area within 32m around the vehicle, object detection location information, depth maps corresponding to the three perspectives, and semantic segmentation labels. The bird's-eye view label size is (256, 256, 3), and the category information includes six categories (vehicles, roads, lane lines, traffic lights, and other categories).

[0050] (3) Constructing a multi-task-based autonomous driving decision-making model. To optimize model convergence and achieve cross-domain prediction from sensor image input to control output, this invention transforms the pre-constructed feature map into a bird's-eye view space for prediction of some task branches. The different task modules of this invention's model are mainly divided into five modules, including: 1) image feature extraction module, 2) state feature extraction module, 3) depth estimation and semantic segmentation prediction module, 4) bird's-eye view and 3D object detection module, and 5) action and state prediction module, such as... Figure 2 As shown.

[0051] (4) Construct an end-to-end multi-task decision-making and planning model, the network structure diagram of which is attached. Figure 2 As shown, this invention selects ResNet-18 as the image encoder module, and optimizes the utilization of computational resources by using weight sharing for three different image inputs. After the encoder, three different feature maps (f) can be obtained. f ,f l ,f r Then, the three feature maps are concatenated to obtain the image feature f. I and embedded features fe .

[0052] (5) For the state extraction module: a two-layer fully connected layer is used to decode the vehicle speed and driving commands, and then the obtained intermediate features are combined with the embedded features f e The state feature f is obtained by performing a connection. s For the deep semantic and semantic segmentation prediction modules, this invention uses two decoder branches to decode the feature map to obtain the prediction results.

[0053] (6) For bird's-eye view prediction and branching, a view transformation module is used to transform the feature map f I The feature map X is obtained by converting it to a bird's-eye view space. bev The system uses a decoder to obtain the bird's-eye view prediction result and an object detection prediction head to obtain the object detection result. The viewpoint conversion module mainly consists of two parts: a dimensional visual conversion module and a dimensional transformation module. The specific method of the dimensional viewpoint conversion module is as follows:

[0054] First, the feature map f I The dimension weights M are obtained using spatial average pooling and fully connected layers. c Then multiply it with the feature map F and perform pointwise convolution to obtain the transformed feature vector X. c The specific calculation method is as follows:

[0055] M c =Mlp(Avergepool) c (F)) (1)

[0056]

[0057] Among them, Avergepool c Let Mlp represent the spatial average pooling operation, Mlp represent a fully connected layer, and FPW represent pointwise convolution. Then, a similar spatial transformation is applied to the dimensional feature vector, followed by dimensional average pooling to obtain the dimensional average vector. This vector is then unfolded, and subsequently transformed again using a fully connected layer to obtain the spatial weights M. s Then combine it with the feature vector X c The features are multiplied and then processed through a depthwise convolution. Finally, the original dimensions are obtained using a dimension transformation module, resulting in the final output X. bev The specific calculation method is as follows:

[0058] M s =Mlp(Reshape(Avergepool) s (X c (3)

[0059]

[0060] Among them, Reshape and Reshape b The operation represents a dimensionality transformation. Then, a two-layer decoder is used to upsample the bird's-eye view features to the original label size, thereby obtaining the bird's-eye view prediction result.

[0061] (7) For the 3D target detection module for surrounding vehicles, the above bird's-eye view features are first processed through a convolutional layer to obtain a location feature map P∈[0,1]. 64×64 The location feature map is used to determine whether a point exists as a target in the bird's-eye view space. Then, the surrounding 360-degree direction is roughly divided into 12 directions (30 degrees each), thus obtaining the approximate directions of surrounding vehicles through classification. Finally, a regression feature map is predicted. The regression prediction graph mainly contains three types of information: vehicle size. Location information deviation and deviation of angle information The positional information deviation is used to correct errors in the positional feature map, while the angle information deviation is used to correct errors caused by estimating the rough direction.

[0062] (7) For the action and state prediction module: First, a flattening operation is used to transform the bird's-eye view features X bev Expand into a one-dimensional feature vector, and then pass through a fully connected operation to obtain the intermediate feature f. c , will the intermediate feature f c With state feature f s The hybrid feature f is obtained by splicing. m =[f c ,f s ], then f m The output speed p after full connection is obtained from the feature output. v and heading angle prediction p o At the same time, f m The output action decision prediction A is obtained through a fully connected and gated recurrent neural network (GRU). The action prediction specifically includes accelerator prediction, steering wheel prediction, and brake prediction. The time step of the gated recurrent neural network is set to 4.

[0063] (8) Train the multi-task network constructed above. First, construct loss functions for different task branches. Depth prediction uses the L1 loss function, semantic segmentation and bird's-eye view construction use the cross-entropy loss function, steering wheel, throttle heading angle prediction, and speed prediction use the L1 loss function, while braking prediction uses a binary classification cross-entropy loss function. Thus, the overall loss function can be obtained as follows:

[0064]

[0065] in These represent the accelerator, steering wheel, and braking loss functions, respectively, and are the main focus of this prediction. The loss functions are semantic segmentation, bird's-eye view prediction, depth prediction, velocity prediction, and heading angle prediction, respectively, with λ as the loss function. i (i = 1, 2, 3, 4, 5) are the corresponding weights.

[0066] (9) Multi-task optimization is performed on the constructed model. After obtaining the above loss function, in order to further optimize the model and improve the prediction accuracy of the main task, this invention attempts to use task relevance to evaluate the influence of different sub-tasks on the main task. The specific calculation method is as follows: First, according to the appendix... Figure 2 The model parameters are divided into task-specific parameters. and shared network layer parameters θ s Taking the calculation of task affinity between task A and the main task as an example, assume the loss function of the main task is... The loss of task A at time t is L t (A), at this point, the loss obtained at this point is used to update the feature-specific parameters of task A and the parameters of the shared network layers. Simultaneously utilize the updated shared network layer parameters and the main task-specific parameters from the previous moment. Obtain the updated main task loss function Finally, the task relevance R between task i and the main task is calculated using the following method. i :

[0067]

[0068] Where lr represents the learning rate at this point, and when the task relevance R... i When the value is positive, it indicates that the update of the shared parameters by task i is beneficial to the reduction of the loss of the main task; conversely, it proves that the update of the shared parameters by task i is not beneficial to the reduction of the loss of the main task, thus obtaining the influence of task i on the main task.

[0069] (10) To further dynamically adjust the relationships between multiple tasks, a meta-learning-based multi-task optimization model is used to update the weights of the multiple tasks, and the dataset is divided into a training set X. train and validation set X val The training set is used to optimize the model parameters, and the validation set is used to optimize the weights between multiple tasks. Assume the training batch at time t is... The validation set data is First, the model parameters are updated using the training set loss. The specific calculation formula is as follows:

[0070]

[0071] Where θ t Let α represent the model parameters at time t, and α represent the learning rate. This represents the weight value for task i at time t-1. This represents the shared parameters of different tasks at time t-1. This represents the model-specific parameters for task i at time t-1. The multi-task weights of the model are then adjusted using the validation set loss, calculated as follows:

[0072]

[0073] Where β is the learning rate of meta-learning. Finally, the model parameters are further updated by combining the updated multi-task weights and the new training set loss, calculated as follows:

[0074]

[0075] The optimized multi-task weights can be obtained through the above calculation method. However, this method requires two iterations, consuming significant computational resources and GPU memory. To simplify the process, an approximate iterative approach is used to replace part of the gradient descent process. Assuming L0(θ,λ), where L(θ,λ) represents the main task loss and the overall loss of all tasks under the weight combination λ, the gradient update for λ can be approximated by the following calculation process:

[0076]

[0077] in This represents an update of network parameters. ε is a very small value. λ = 1 represents that all tasks have the same weight. Using the above calculation method, the weights of multiple tasks can be dynamically adjusted using the validation set.

[0078] (10) For the above model training process, the basic settings of the training process are as follows: the model training and meta-learning optimizer are both selected as the adam optimizer, the initial learning rate is set to 0.0001, the number of training batches is set to 12, the total number of training rounds is 100, and the ratio of training set to validation set is 10:1, that is, every ten training cycles, a weight adjustment learning between multiple tasks is performed.

[0079] (11) This embodiment of the invention also proposes an autonomous driving vehicle device, which uses a trained model in combination with an autonomous driving central controller and on-board cameras and other devices to realize the autonomous driving decision planning method.

[0080] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. An end-to-end autonomous driving decision-making and planning method combining meta-learning and multi-task optimization, characterized in that, Including the following: S1. Using a multi-task learning approach, a set of potential sub-tasks is selected, and the constructed multi-task branches include: throttle prediction, steering wheel prediction, brake prediction, bird's-eye view prediction, object detection, depth map prediction, semantic segmentation prediction, heading angle prediction, and driving speed prediction. Among them, throttle prediction, steering wheel prediction, and brake prediction are the main tasks, and the remaining six branches are the auxiliary tasks. S2. Prepare the dataset based on the selected subtasks; S3. Build an autonomous driving decision-making model based on multi-tasks. Introduce multi-task constraints to share some network features. At the same time, in order to realize cross-domain prediction between sensor image input and control output, the pre-feature map is transformed into the bird's-eye view space to predict some task branches. The model includes five task modules: 1) Image feature extraction module, 2) State feature extraction module, 3) Depth estimation and semantic segmentation prediction module, 4) Bird's-eye view and 3D object detection module, and 5) Action and state prediction module. S4. Train the constructed multi-task autonomous driving decision network model; S5. Perform multi-task optimization on the model; the specific process of S5 is as follows: The task relevance method is used to evaluate the impact of different subtasks on the main task. The specific calculation method is as follows: First, the model parameters are divided into task-specific parameters. and shared network layer parameters Taking the calculation of task affinity between task A and the main task as an example, assuming the loss function of the main task is... The loss of task A at time t is The loss obtained at this time is used to update the feature-specific parameters of task A and the parameters of the shared network layers. At the same time, utilize the updated shared network layer parameters and the main task-specific parameters from the previous moment. The updated main task loss function is obtained. Finally, the task relevance between task i and the main task is calculated using the following method. : （6） in This represents the learning rate at this point, when the task relevance is... When the value is positive, it indicates that the update of the shared parameters by task i is beneficial to the reduction of the loss of the main task; otherwise, it proves that the update of the shared parameters by task i is not beneficial to the reduction of the loss of the main task. S6. Utilize the trained and optimized model to implement autonomous driving decision-making and planning.

2. The end-to-end autonomous driving decision-making and planning method combining meta-learning multi-task optimization according to claim 1, characterized in that, The specific implementation of S2 includes: The input information for the decision-making and planning model mainly includes sensor information and driving commands. The sensors mainly include three cameras: front, left, and right. Each camera captures an image of size (256, 256, 3), simultaneously collecting the vehicle's speed information and driving commands at each moment, specifically: forward, left turn, right turn, lane change, and U-turn. The model's label information includes: 1) information on the expert's driving decisions, i.e., the expert's throttle input, steering wheel angle, and whether braking is applied at the current moment; 2) a bird's-eye view and target detection location information within a 32m radius around the vehicle; 3) depth maps and semantic segmentation labels corresponding to the three cameras, where the bird's-eye view, semantic segmentation, and depth map labels are of size (256, 256, 3). The semantic segmentation category information for the bird's-eye view includes six categories: vehicles, roads, lane lines, red lights, green lights, and others.

3. The end-to-end autonomous driving decision-making and planning method combining meta-learning multi-task optimization according to claim 1, characterized in that, The image feature extraction module in S3 selects a ResNet-50 pre-trained on the ImageNet dataset as the image encoder. After inputting camera information into the image encoder, image features are obtained. and embedded features ; For the state extraction module: a two-layer fully connected layer is used to decode the vehicle's speed and driving commands, and then the obtained intermediate features are compared with the embedded features. Connect to obtain state characteristics .

4. The end-to-end autonomous driving decision-making and planning method combining meta-learning multi-task optimization according to claim 3, characterized in that, The depth estimation and semantic segmentation prediction module in S3 uses two decoder branches to decode the feature map to obtain the prediction result. The decoder mainly includes three upsampling layers and two convolutional layers. The semantic segmentation convolutional layer uses the Softmax activation function, and the depth estimation output uses the Sigmoid activation function.

5. The end-to-end autonomous driving decision-making and planning method combining meta-learning multi-task optimization according to claim 4, characterized in that, The bird's-eye view and 3D object detection module in S3 utilize visual transformation to convert feature maps... Feature map obtained by converting to bird's-eye view space The decoder is used to obtain the bird's-eye view prediction result, and the target detection prediction head is used to obtain the target detection result; The specific method for the visual transformation is as follows: First, spatial average pooling and a fully connected layer are used to obtain dimensional weights. Then combine it with the feature map Multiply and then perform pointwise convolution to obtain the transformed dimensional feature vector. The specific calculation method is as follows: （1） （2） in This indicates a space-average pooling operation. Represents a fully connected layer. This represents pointwise convolution; subsequently, a similar spatial transformation is applied to the dimensional feature vector, followed by dimensional average pooling to obtain the dimensional average vector, which is then unfolded and transformed again using a fully connected layer to obtain the spatial weights. Then combine it with the feature vector The features are multiplied and then subjected to depthwise convolution. Finally, the dimensions of the resulting features are transformed back to their original dimensions to obtain the final output. The specific calculation method is as follows: （3） （4） in and This represents a dimensionality transformation operation, followed by a decoder processing the feature map. Upsample to the original label size to obtain the bird's-eye view prediction result; The 3D target detection module for surrounding vehicles first obtains a location feature map by performing convolutional layer operations on the bird's-eye view features. The location feature map is used to determine whether a target exists at the point corresponding to the location feature map in the bird's-eye view space. Then, the surrounding 360-degree direction is roughly divided into 12 directions, each with a degree of 30 degrees, so as to obtain the rough direction of the surrounding vehicles through classification. Finally, a regression feature map is predicted. The regression prediction graph mainly contains three types of information: vehicle size. Location information deviation and deviation of angle information The positional information deviation is used to correct errors in the positional feature map, while the angle information deviation is used to correct errors caused by estimating the rough direction.

6. The end-to-end autonomous driving decision-making and planning method combining meta-learning multi-task optimization according to claim 1, characterized in that, The specific process of S4 is as follows: First, loss functions are constructed for different task branches. Depth prediction uses the L1 loss function, semantic segmentation and bird's-eye view construction use the cross-entropy loss function, and object detection's loss function consists of three parts: a binary cross-entropy loss function for position encoding, a multi-class cross-entropy loss function for coarse direction estimation, and an L1 loss function for regression features. For steering wheel, throttle, and yaw angle prediction and speed prediction, the L1 loss function is used, and for braking prediction, a binary cross-entropy loss function is used. Therefore, the overall loss function is: （6） in These represent the throttle, steering wheel, and brake loss functions, respectively. , , , , These represent the loss functions for semantic segmentation, bird's-eye view prediction, depth prediction, velocity prediction, and heading angle prediction, respectively. The values ​​i = 1, 2, 3, 4, 5 represent the corresponding weights.

7. The end-to-end autonomous driving decision-making and planning method combining meta-learning multi-task optimization according to claim 1, characterized in that, The S5 also includes: To dynamically adjust the relationships between multiple tasks, a meta-learning-based multi-task optimization method is used to update the weights of the tasks, and the dataset is divided into a training set and a... and validation set The training set is used to optimize the model parameters, and the validation set is used to optimize the weights between multiple tasks; assuming the training batch at time t is... The validation set data is First, the model parameters are updated using the training set loss. The specific calculation formula is as follows: （7） in This represents the model parameters at time t. Indicates the learning rate. This represents the weight value for task i at time t-1. This represents the shared parameters of different tasks at time t-1. This represents the model-specific parameters for task i at time t-1. Represents the training set loss function Gradient calculation of model parameters, Let represent the training set loss function, which is then combined with the validation set loss to adjust the multi-task weights of the model. The calculation formula is as follows: （8） in The learning rate for meta-learning. Represents the validation set loss function Gradient calculation for task weights, Represents the validation set loss function; Finally, the model parameters are further updated by combining the updated multi-task weights and the new training set loss, as shown in the following formula: （9）。 8. The end-to-end autonomous driving decision-making and planning method combining meta-learning multi-task optimization according to claim 7, characterized in that, We adopt an approximate iterative approach to replace part of the gradient descent process, assuming... , Representative at The main task loss and the overall loss of all tasks under the weighted combination are targeted at The gradient update can be approximated by the following calculation process: in This represents an update of network parameters. , For a very small value, This means that all tasks have the same weight.

9. An autonomous driving vehicle device, characterized in that, The device is capable of performing the autonomous driving decision-making and planning method as described in any one of claims 1-8.

Images