A Multi-Task Perception Method and System for Autonomous Driving Based on Gradient Covariance Eigenvalue Decomposition
By using a multi-task gradient fusion update algorithm based on gradient covariance feature decomposition, the gradient direction of the autonomous driving multi-task detection model is optimized, solving the gradient conflict problem during model training and achieving efficient environmental perception and detection results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2024-08-14
- Publication Date
- 2026-05-26
AI Technical Summary
Existing autonomous driving multi-task detection models suffer from slow detection speeds due to gradient descent direction conflicts and magnitude differences during training. This makes it difficult to improve detection efficiency and performance with limited computing resources.
A multi-task gradient fusion and update algorithm based on gradient covariance eigenvalue decomposition is adopted. By constructing a multi-task detection model, the detection results of traffic targets, drivable areas and lane lines are output in parallel. The gradient direction is adjusted by using covariance matrix eigenvalue decomposition to optimize the backbone network parameters, thereby achieving effective gradient fusion and update.
It improves the efficiency and accuracy of environmental perception in autonomous driving, ensures that each task is fully trained under limited parameters, and enhances detection speed and performance.
Smart Images
Figure CN119007153B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an autonomous driving multi-task perception method and system, and more particularly to an autonomous driving multi-task perception method and system based on gradient covariance eigenvalue decomposition. Background Technology
[0002] The perception capability of autonomous vehicles is crucial for their accurate understanding of the surrounding environment and for providing robust data support for downstream path planning and control. Traffic object detection, drivable area detection, and lane detection are three important perception tasks. Due to limited on-vehicle computing resources, deploying multiple complex models simultaneously can overload the domain controller. Therefore, multi-task detection models are gradually becoming the future trend of autonomous driving perception algorithms. Most existing multi-task detection models for autonomous driving optimize their structure by continuously stacking complex modules to improve accuracy, but this results in low detection speed during actual deployment. Furthermore, traditional training strategies simply add the losses of multiple tasks during backpropagation parameter updates, ignoring potential conflicts or significant differences in gradient descent directions and magnitudes between different tasks. This prevents the model from optimizing towards the global optimum during training. Therefore, researching efficient multi-task model training methods to ensure that each task is adequately trained with limited parameters to improve detection efficiency and performance is a highly valuable research problem. Summary of the Invention
[0003] Purpose of the invention: The purpose of this invention is to provide an autonomous driving multi-task perception method and system based on gradient covariance feature decomposition that can improve detection efficiency and performance.
[0004] Technical solution: The multi-task perception method for autonomous driving based on gradient covariance feature decomposition described in this invention includes:
[0005] (1) Collect driving environment image data, build a multi-task detection model, and input driving environment image data; the multi-task detection model includes a backbone feature extraction network, a feature selection module, and a multi-task detection branch; the multi-task detection branch includes: traffic target detection branch, drivable area detection branch, and lane line detection branch;
[0006] (2) Establish corresponding loss functions for each task detection branch;
[0007] (3) Update the parameters of each task detection branch based on the loss function of each task detection branch;
[0008] (4) Update the parameters of the backbone feature extraction network using a multi-task gradient fusion update algorithm based on covariance matrix eigenvalue decomposition;
[0009] (5) Train the multi-task detection model, use the trained multi-task detection model for detection and recognition, and output the results of traffic target detection, drivable area detection and lane line detection in parallel.
[0010] Furthermore, the backbone feature extraction network is specifically selected as Elannet, which is used to extract image features.
[0011] Furthermore, the feature selection module performs convolution and pooling on the input feature map, normalizes the features to between 0 and 1 using the Sigmoid function, selects the Top-K channels to generate channel masks according to the requirements of downstream tasks, and outputs the filtered feature map.
[0012] Furthermore, the construction of the multi-task detection branch includes:
[0013] Based on the Feature Pyramid Network (FPN), a traffic target detection branch is constructed.
[0014] Choose the ASPP spatial pooling pyramid structure to construct the drivable region detection branch;
[0015] Using shallow and deep features as input, a lane detection branch is constructed based on a strip pooling structure.
[0016] Furthermore, step (2) specifically includes:
[0017] (2.1) Establish the traffic target detection loss function:
[0018] l detect =β1l class +β2l obj +β3l box
[0019] Among them, l class and l obj The loss functions are Focal loss and target object loss, respectively. box For bounding box loss, the CIOU loss function is used; β1, β2, and β3 are the weights corresponding to each loss.
[0020] The formula for calculating the CIoU loss function is as follows:
[0021]
[0022] Among them, b and b gt These are the prediction bounding box and the label bounding box, respectively. gt and h gt ρ represents the width and height of the label box, w and h represent the width and height of the prediction box, ρ represents the distance between the center of the label box and the center of the prediction box, and α is the weighting coefficient.
[0023] (2.2) Establish the drivable area detection loss function:
[0024] l dr =l ce
[0025] Among them, l ce The cross-entropy loss function;
[0026] (2.3) Establish the lane detection loss function:
[0027] l ll =l ce +l IoU +l focal +l tv
[0028] Among them, l IoU l focal and l tv The calculation formulas are as follows:
[0029]
[0030] Among them, TP c FN represents the true number of instances of class c, indicating the number of pixels that the model correctly identified and segmented into lane lines. c FP represents the number of false negatives for class c, indicating the number of pixels that the model failed to segment that actually exist as lane lines. c TN represents the number of false positives for class c, indicating that the model incorrectly labels pixels in non-lane regions as lane lines. c p represents the number of true negative instances of class c, indicating the number of pixels that the model correctly identified in non-lane area. n (c) represents the probability that the nth pixel belongs to class c, g n (c) indicates whether the nth pixel belongs to class c in the true label, γ is a hyperparameter for adjusting the weights, C represents the number of classes, and N represents the total number of pixels in the input image. This is a hyperparameter.
[0031] Furthermore, step (3) specifically includes:
[0032] For each task detection branch, update the parameters:
[0033]
[0034] Among them, w t1 w t2 and w t3 These are the parameters to be updated for the traffic target detection branch, the drivable area detection branch, and the lane line detection branch, respectively. t3 Specifically, it refers to the structural weights on the lane detection branch, w t1′、w t2 ′ and w t3 '' represents the updated parameters for the traffic target detection branch, the drivable area detection branch, and the lane line detection branch, respectively, and η is the learning rate. This indicates the partial derivative.
[0035] Further, step (4) specifically includes:
[0036] (4.1) For each task detection branch m∈{1,2,…,M}, calculate the loss L m gradient g with respect to model parameters θ m :
[0037]
[0038] (4.2) Construct the covariance matrix A of the gradients of multiple tasks. Assuming there are M tasks, and the gradient dimension of each task is d, then the covariance matrix is a d×d matrix:
[0039]
[0040] Where T represents transpose. This represents the mean of the gradients for all tasks:
[0041]
[0042] (4.3) Perform eigenvalue decomposition on the covariance matrix A to obtain the eigenvalues λ. i and eigenvector v i :
[0043]
[0044] Where Λ represents the eigenvalue λ. i A diagonal matrix is represented as follows:
[0045]
[0046] V is the eigenvector matrix, V = [v1, v2, ..., v] d ];
[0047] (4.4) Adjust the gradient using eigenvalues and eigenvectors to generate a new fusion gradient g. fused :
[0048]
[0049] Among them, v i The dimension is d×1, g j The dimension is d×1, g fused The dimension is d×1;
[0050] (4.5) Based on the fusion gradient g fused Update the backbone network parameters.
[0051] Furthermore, step (5) specifically includes:
[0052] (5.1) Train the model using an optimizer;
[0053] (5.2) Based on the TensorRT inference framework, the trained multi-task detection model is quantized to generate an engine file;
[0054] (5.3) Deploy the multi-task detection model to an embedded device to perform detection and recognition, and output traffic target detection, drivable area detection and lane line detection results.
[0055] Based on the same inventive concept, this invention also provides an autonomous driving multi-task perception system based on gradient covariance eigenvalue decomposition, comprising:
[0056] The model building module is used to collect driving environment image data, build a multi-task detection model, and input driving environment image data; the multi-task detection model includes a backbone feature extraction network, a feature selection module, and multi-task detection branches; the multi-task detection branches include: traffic target detection branch, drivable area detection branch, and lane line detection branch;
[0057] The loss function construction module is used to build corresponding loss functions for each task detection branch;
[0058] The branch update module is used to update the parameters of each task detection branch based on the loss function of each task detection branch;
[0059] The backbone update module updates the parameters of the backbone feature extraction network based on a multi-task gradient fusion update algorithm of covariance matrix eigenvalue decomposition.
[0060] The perception and detection module trains a multi-task detection model, uses the trained model to perform detection and recognition, and outputs traffic target detection, drivable area detection, and lane line detection results in parallel.
[0061] Based on the same inventive concept, the present invention also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the autonomous driving multi-task perception method based on gradient covariance eigenvalue decomposition as described in any of the preceding claims.
[0062] Beneficial effects: Compared with the prior art, the present invention filters the features extracted by the shared network by constructing a feature selection module, and selects the more important features of each task to send to each detection branch; the present invention proposes a multi-gradient fusion update algorithm for multi-task gradient covariance matrix eigenvalue decomposition, which effectively updates the shared network parameters of the multi-task model during the backpropagation process of the model, so that traffic target detection, drivable area segmentation and lane line detection tasks are fully trained, thereby improving the efficiency and accuracy of autonomous driving environment perception. Attached Figure Description
[0063] Figure 1 This is a flowchart of the method of the present invention;
[0064] Figure 2 This is a structural diagram of the multi-task detection model according to an embodiment of the present invention;
[0065] Figure 3 This is a network structure diagram of the feature selection module in an embodiment of the present invention;
[0066] Figure 4 This is a schematic diagram of the multi-task detection branch parameter update in an embodiment of the present invention;
[0067] Figure 5 This is a schematic diagram of multi-task gradient fusion update according to an embodiment of the present invention. Detailed Implementation
[0068] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0069] As attached Figure 1 As shown in this embodiment, the multi-task perception method for autonomous driving based on gradient covariance feature decomposition includes:
[0070] (1) Collect driving environment image data, build a multi-task detection model, and input driving environment image data; the multi-task detection model includes a backbone feature extraction network, a feature selection module, and a multi-task detection branch; the multi-task detection branch includes: traffic target detection branch, drivable area detection branch, and lane line detection branch;
[0071] (2) Establish corresponding loss functions for each task detection branch;
[0072] (3) Update the parameters of each task detection branch based on the loss function of each task detection branch;
[0073] (4) Update the parameters of the backbone feature extraction network using a multi-task gradient fusion update algorithm based on covariance matrix eigenvalue decomposition;
[0074] (5) Train the multi-task detection model, use the trained multi-task detection model for detection and recognition, and output the results of traffic target detection, drivable area detection and lane line detection in parallel.
[0075] Step (1) specifically includes the following steps:
[0076] (1.1) Acquire driving environment image data collected by cameras of autonomous vehicles and convert the data format;
[0077] (1.2) Construct a multi-task detection model, such as Figure 2 As shown, it includes a backbone feature extraction network, a feature selection module, and a multi-task detection branch;
[0078] Elannet was chosen as the backbone network for extracting image features;
[0079] Construct a feature selection module, such as Figure 3 As shown, more targeted features are input into each detection branch. This module performs convolution and pooling on the input feature map, and then normalizes the features to between 0 and 1 using the Sigmoid function. Based on the requirements of the downstream task, the top-K channels are selected to generate channel masks, and the filtered feature map is output.
[0080] Based on the Feature Pyramid Network (FPN), a traffic target detection branch is constructed.
[0081] The ASPP spatial pooling pyramid structure is selected to construct the drivable area detection branch. The drivable area refers to the area where a vehicle can drive safely and legally in a specific scenario. Specifically, it includes lane areas, obstacle avoidance areas, and areas restricted by traffic rules on the road. For example, in urban roads, the drivable area includes lanes divided by lane lines; in highways, it includes driving lanes and emergency stopping lanes; and in parking lots, it includes parking space areas and lane passages.
[0082] The lane detection branch is constructed based on a strip pooling structure, using shallow and deep features as inputs. The shallow features, which are feature maps from the output of the first block of Elannet, refer to features extracted from layers close to the input in the network. These features mainly capture low-level information of the image, such as edges, textures, and basic shapes. The deep features are the entire output of Elannet, that is, the output after passing through four blocks. These are features extracted from deeper layers in the network. These features are more abstract and can capture more complex patterns and high-level semantic information.
[0083] Step (2) specifically includes the following steps:
[0084] (2.1) Regarding traffic target detection loss,
[0085] l detect =β1l class +β2l obj +β3l box
[0086] Among them, l box For bounding box loss, the CIOU loss function is used; obj l class The loss consists of target object loss and classification loss, respectively. The target object loss represents the difference between the predicted confidence of the target's existence and the true label, and is used to evaluate whether each detection box contains the target object. The Focal loss function is used, with β1, β2, and β3 being the weights corresponding to the three losses mentioned above. The CIoU loss calculation formula is as follows:
[0087]
[0088] Where b, b gt These are the prediction bounding box and the label bounding box, respectively. gt h gt w and h are the width and height of the label box and the prediction box, respectively; ρ represents the distance between the centers of the two rectangles; and α is the weighting coefficient.
[0089] (2.2) For detecting losses in drivable areas,
[0090] l dr =l ce
[0091] Among them, l ce The cross-entropy loss function;
[0092] (2.3) Regarding lane line detection loss,
[0093] l ll =l ce +l IoU +l focal +l tv
[0094] Among them, l IoU l focal and l tv The calculation formulas are as follows:
[0095]
[0096] Among them, TP c FN represents the true number of instances of class c, indicating the number of pixels that the model correctly identified and segmented into lane lines. c FP represents the number of false negatives for class c, indicating the number of pixels that the model failed to segment that actually exist as lane lines. cTN represents the number of false positives for class c, indicating that the model incorrectly labels pixels in non-lane regions as lane lines. c p represents the number of true negative instances of category c, indicating the number of pixels that the model correctly identified in non-lane line regions. n (c) represents the probability that the nth pixel belongs to class c; g n (c) indicates whether the nth pixel belongs to class c in the true label, γ is a hyperparameter for adjusting the weights, C represents the number of classes, and N represents the total number of pixels in the input image. It's a hyperparameter, set to 0.7.
[0097] Step (3) specifically includes the following steps, the basic structure of the multi-task model parameter update is as follows: Figure 4 As shown:
[0098] (3.1) For each detection branch, update the parameters:
[0099]
[0100] Among them, w t1 w t2 w t3 These are the parameters to be updated for the three branches, w t1 ′、w t2 ′ and w t3 '' represents the updated parameters for the traffic target detection branch, the drivable area detection branch, and the lane line detection branch, respectively, and η is the learning rate. This indicates the partial derivative.
[0101] Step (4) specifically includes the following steps, with the multi-task gradient update direction as follows: Figure 5 As shown:
[0102] (4.1) For each task detection branch m∈{1,2, … ,M}, calculate the loss L m gradient g with respect to model parameters θ m :
[0103]
[0104] (4.2) Construct the covariance matrix A of the gradients of multiple tasks. Assuming there are M tasks, and the gradient dimension of each task is d, then the covariance matrix is a d×d matrix:
[0105]
[0106] Where T represents transpose. This represents the mean of the gradients for all tasks:
[0107]
[0108] (4.3) Perform eigenvalue decomposition on the covariance matrix A to obtain the eigenvalues λ. i and eigenvector v i :
[0109]
[0110] Where Λ represents the eigenvalue λ. i A diagonal matrix is represented as follows:
[0111]
[0112] V is the eigenvector matrix, V = [v1, v2, ..., v3, ..., v4, ... … ,v d ];
[0113] (4.4) Adjust the gradient using eigenvalues and eigenvectors to generate a new fusion gradient g. fused :
[0114]
[0115] Among them, v i The dimension is d×1, g j The dimension is d×1, g fused The dimension is d×1; in the calculation, the eigenvalue λ i As a scalar, first calculate (A d×d matrix), then with the gradient vector g j Multiply the values, and finally sum them by weights to obtain the fused gradient g. fused (A d×1 vector). Represents the eigenvector v i The outer product. The outer product here. It is a d×d matrix that represents the eigenvector v. i Projection and orientation in the feature space. Wherein, through... For gradient g j A transformation is performed to obtain the gradient components along the eigenvector directions. Finally, a weighted sum is taken over all eigenvector directions to obtain the fused gradient g. fused .
[0116] (4.5) Based on the fusion gradient g fused Update the backbone network parameters.
[0117] Step (5) specifically includes the following steps:
[0118] (5.1) Train the model for 200 epochs using the Adam optimizer;
[0119] (5.2) Based on the TensorRT inference framework, the trained multi-task detection model is quantized to generate an engine file;
[0120] (5.3) Deploy the multi-task detection model to an embedded device and output traffic target detection, drivable area detection and lane line detection results.
[0121] Based on the same inventive concept, this invention also provides an autonomous driving multi-task perception system based on gradient covariance eigenvalue decomposition, comprising:
[0122] The model building module is used to collect driving environment image data, build a multi-task detection model, and input driving environment image data; the multi-task detection model includes a backbone feature extraction network, a feature selection module, and multi-task detection branches; the multi-task detection branches include: traffic target detection branch, drivable area detection branch, and lane line detection branch;
[0123] The loss function construction module is used to build corresponding loss functions for each task detection branch;
[0124] The branch update module is used to update the parameters of each task detection branch based on the loss function of each task detection branch;
[0125] The backbone update module updates the parameters of the backbone feature extraction network based on a multi-task gradient fusion update algorithm of covariance matrix eigenvalue decomposition.
[0126] The perception and detection module trains a multi-task detection model, uses the trained model to perform detection and recognition, and outputs traffic target detection, drivable area detection, and lane line detection results in parallel.
[0127] Based on the same inventive concept, the present invention also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the autonomous driving multi-task perception method based on gradient covariance eigenvalue decomposition as described in any of the preceding claims.
[0128] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A multi-task perception method for autonomous driving based on gradient covariance eigenvalue decomposition, characterized in that, include: (1) Collect driving environment image data, build a multi-task detection model, and input driving environment image data; The multi-task detection model includes a backbone feature extraction network, a feature selection module, and a multi-task detection branch; The multi-task detection branches include: traffic target detection branch, drivable area detection branch, and lane line detection branch; (2) Establish corresponding loss functions for each task detection branch; specifically: (2.1) Establish the traffic target detection loss function: ； in, and The loss functions Focal are used for classification loss and target object loss, respectively. For bounding box loss, the CIoU loss function is used; , and These are the weights corresponding to each loss; The formula for calculating the CIoU loss function is as follows: ； in, and These are the prediction bounding box and the label bounding box, respectively. and For the width and height of the label box, and To predict the width and height of the bounding box, This represents the distance between the center of the label box and the center of the prediction box. These are the weighting coefficients; (2.2) Establish the drivable area detection loss function: ； in, The cross-entropy loss function; (2.3) Establish the lane detection loss function: ； in, , and The calculation formulas are as follows: ； ； ； in, For category The true number of instances represents the number of pixels that the model correctly identified and segmented into lane lines. For category The number of false negatives indicates the number of pixels in the lane lines that the model failed to segment. For category The number of false positives indicates that the model incorrectly labels pixels in non-lane line regions as lane lines. For category The true negative count represents the number of pixels that the model correctly identified in non-lane line regions. Indicates the first Each pixel belongs to The probability of the category, Indicates the first Does each pixel belong to the actual label? kind, To adjust the hyperparameters of the weights, Indicates the number of categories. This represents the total number of pixels in the input image. For hyperparameters; (3) Update the parameters of each task detection branch based on the loss function of each task detection branch; (4) Update the parameters of the backbone feature extraction network using a multi-task gradient fusion update algorithm based on covariance matrix eigenvalue decomposition; (5) Train the multi-task detection model, use the trained multi-task detection model for detection and recognition, and output the results of traffic target detection, drivable area detection and lane line detection in parallel.
2. The autonomous driving multi-task perception method based on gradient covariance feature decomposition according to claim 1, characterized in that, The backbone feature extraction network is specifically selected as Elannet, which is used to extract image features.
3. The autonomous driving multi-task perception method based on gradient covariance feature decomposition according to claim 1, characterized in that, The feature selection module performs convolution and pooling on the input feature map, normalizes the features to between 0 and 1 using the Sigmoid function, selects the Top-K channels to generate channel masks according to the requirements of downstream tasks, and outputs the filtered feature map.
4. The autonomous driving multi-task perception method based on gradient covariance feature decomposition according to claim 1, characterized in that, Construct a multi-task detection branch, including: Based on the Feature Pyramid Network (FPN), a traffic target detection branch is constructed. Choose the ASPP spatial pooling pyramid structure to construct the drivable region detection branch; Using shallow and deep features as input, a lane detection branch is constructed based on a strip pooling structure.
5. The autonomous driving multi-task perception method based on gradient covariance feature decomposition according to claim 1, characterized in that, Step (3) specifically involves: For each task detection branch, update the parameters: ； ； ； in, , and These are the parameters to be updated for the traffic target detection branch, the drivable area detection branch, and the lane line detection branch, respectively. , and These are the updated parameters for the traffic target detection branch, the drivable area detection branch, and the lane line detection branch, respectively. For learning rate, This indicates the partial derivative.
6. The autonomous driving multi-task perception method based on gradient covariance feature decomposition according to claim 5, characterized in that, Step (4) specifically involves: (4.1) For each task detection branch Calculate the loss Relative to model parameters gradient : ； (4.2) Construct the covariance matrix of gradients for multiple tasks Assuming there is There are tasks, and the gradient dimension of each task is . Then the covariance matrix is The matrix: ； in, Indicates transpose. This represents the mean of the gradients for all tasks: ； (4.3) For the covariance matrix Perform eigenvalue decomposition to obtain eigenvalues. and eigenvectors : ； in, For including eigenvalues A diagonal matrix is represented as follows: ； It is the eigenvector matrix. ; (4.4) Adjust the gradient using eigenvalues and eigenvectors to generate a new fused gradient. : ； in, The dimension is , The dimension is , The dimension is ; (4.5) Based on fusion gradient Update the backbone network parameters.
7. The autonomous driving multi-task perception method based on gradient covariance feature decomposition according to claim 6, characterized in that, Step (5) specifically involves: (5.1) Train the model using an optimizer; (5.2) Based on the TensorRT inference framework, the trained multi-task detection model is quantized to generate an engine file; (5.3) Deploy the multi-task detection model to an embedded device to perform detection and recognition, and output traffic target detection, drivable area detection and lane line detection results.
8. An autonomous driving multi-task perception system based on gradient covariance eigenvalue decomposition, which implements the autonomous driving multi-task perception method based on gradient covariance eigenvalue decomposition according to any one of claims 1-7, characterized in that, include: The model building module is used to collect driving environment image data, build a multi-task detection model, and input driving environment image data; The multi-task detection model includes a backbone feature extraction network, a feature selection module, and a multi-task detection branch; The multi-task detection branches include: traffic target detection branch, drivable area detection branch, and lane line detection branch; The loss function construction module is used to build corresponding loss functions for each task detection branch; The branch update module is used to update the parameters of each task detection branch based on the loss function of each task detection branch; The backbone update module updates the parameters of the backbone feature extraction network based on a multi-task gradient fusion update algorithm of covariance matrix eigenvalue decomposition. The perception and detection module trains a multi-task detection model, uses the trained model to perform detection and recognition, and outputs traffic target detection, drivable area detection, and lane line detection results in parallel.
9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the autonomous driving multi-task perception method based on gradient covariance eigenvalue decomposition according to any one of claims 1-7.