An automatic driving car road scene understanding method based on multi-task learning

Abstract

This paper presents a multi-task learning-based method for road scene understanding in autonomous vehicles, relating to vehicle intelligence and autonomous driving. The method comprises: 1) Designing a shared encoder for the YOLO-ODL multi-task model for road scene understanding in autonomous vehicles; 2) Designing an object detection decoder for the YOLO-ODL multi-task model; 3) Designing drivable area decoders and lane line decoders for the YOLO-ODL multi-task model using semantic segmentation; 4) Designing a multi-task loss function, defining the total loss as the weighted sum of the losses from the three tasks; 5) Using TensorRT to accelerate the deployment of the multi-task model and reduce its computational complexity; and 6) Establishing a road scene understanding dataset for training and evaluating the multi-task learning model. This method achieves joint detection of traffic objects, drivable areas, and lane lines, saving computational resources, reducing overfitting risk, and improving network flexibility.

Description

Technical Field

[0001] This invention belongs to the field of automotive intelligence and autonomous driving, and in particular relates to a method for understanding road scenes of autonomous vehicles based on multi-task learning. Background Technology

[0002] Autonomous vehicles can be functionally divided into three modules: environmental perception, decision-making and planning, and vehicle control. Environmental perception is a fundamental and crucial component for achieving autonomous driving, serving as the foundation for other key technologies. Its performance determines whether an autonomous vehicle can adapt to complex and ever-changing traffic environments. Visual perception will play a key role in the development of autonomous vehicles, and visual sensors offer advantages such as mature technology, low cost, and comprehensive detection capabilities.

[0003] Effectively detecting perceived objects in various environments can significantly improve the safety of autonomous vehicles. However, factors such as road scenes, lighting, and weather severely limit detection performance. With the development of big data, computing power, and algorithms, the accuracy of deep learning has been rapidly improving, achieving significant breakthroughs in the field of autonomous driving and making the detection challenge a potential solution. Reference 1 (Tian Z, Shen C, Chen H, et al. FCOS: Fully convolutional one-stage object detection[C]. Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2019:9627-9636.) uses a pixel-wise prediction method to predict target bounding boxes and adds a Center-ness branch to suppress low-quality target bounding boxes. Reference 2 (Teichmann M, Weber M, Zoellner M, et al. MultiNet: Real-time joint semantic reasoning for autonomous driving[C]. Proceedings of the IEEE Intelligent Vehicles Symposium (IV). 2018:1013-1020.) designs a joint classification, detection, and semantic segmentation network.

[0004] Although deep learning has made significant progress, runtime becomes crucial when deploying applications in practice. Summary of the Invention

[0005] The purpose of this invention is to address the problems of redundant computation, slow speed, and low accuracy in existing technologies by providing a multi-task learning-based method for road scene understanding in autonomous vehicles. This method achieves joint and efficient detection of targets, drivable areas, and lane lines through a multi-task model based on hard parameter sharing. The YOLO-ODL multi-task model, based on hard parameter sharing, enables joint and efficient detection of targets, drivable areas, and lane lines, and provides accelerated deployment and application of the model, thereby providing stable and reliable conditions for decision-making, planning, and execution control.

[0006] The technical solution of this invention is as follows: A multi-task model, YOLO-ODL, based on hard parameter sharing is constructed. By sharing neural network parameters for three tasks—traffic targets, drivable areas, and lane lines—joint and efficient detection of targets, drivable areas, and lane lines is achieved. TensorRT neural network acceleration is employed to improve model deployment performance, significantly increasing model inference speed while maintaining model accuracy. This invention includes the following steps:

[0007] Step 1: Design a shared encoder for YOLO-ODL, a multi-task model for road scene understanding in autonomous vehicles.

[0008] Step 2: Design the object detection decoder for YOLO-ODL, a multi-task model for road scene understanding in autonomous vehicles.

[0009] Step 3: Design the drivable area decoder and lane line decoder of the multi-task model YOLO-ODL using semantic segmentation.

[0010] Step 4: Link the losses of multiple tasks together, design a multi-task loss function, and define the total loss as the weighted sum of the losses of the three tasks.

[0011] Step 5: Use TensorRT to accelerate the deployment of multi-task models and reduce the computational complexity of the models.

[0012] Step 6: Establish a road scene understanding dataset, and use the dataset to train and evaluate the multi-task learning model.

[0013] In step 1, the specific steps for designing the shared encoder of the YOLO-ODL multi-task model for road scene understanding in autonomous vehicles include:

[0014] Step 1: The Backbone and FPN structures in the YOLOv5s object detection model are adopted. The Backbone extracts common image features in the scene, and the FPN fuses image features at different scales.

[0015] Step 2: Generate feature maps at three scales: W / 4×H / 4×128, W / 8×H / 8×64, and W / 16×H / 16×128.

[0016] In step 2, the specific steps for designing the object detection decoder of the YOLO-ODL multi-task model for road scene understanding in autonomous vehicles include:

[0017] Step 1: The PANet and Detection Head structures in the YOLOv5s object detection model are adopted. PANet further integrates features of different scales, and the Detection Head uses convolutional layers with a kernel size of 1×1 and a stride of 1 to adjust the number of channels.

[0018] Step 2: Add shallow high-resolution features of 160×96 to replace the initial deep low-resolution features of 20×12, and finally generate feature maps of three scales: 160×96×18, 80×48×18, and 40×24×18.

[0019] Step 3: Each grid is responsible for 3 prior boxes, with a total of 61,440 prediction outputs. Each prediction output includes 4 parameters related to the position of the prediction box, 1 confidence parameter, and 1 vehicle category parameter. Therefore, the output feature map has 3×6=18 channels.

[0020] In step 3, the specific steps for designing the drivable region decoder and lane line decoder of the multi-task model YOLO-ODL using the semantic segmentation method include:

[0021] Step 1: The drivable area decoder and lane line decoder consist of two CSPup layers, a CBS layer, and a Sigmoid layer.

[0022] The second step is to transform the low-resolution W / 4×H / 4×128 feature map back to the high-resolution W×H×2 feature map, with the two channels corresponding to the number of categories. Image features are then further extracted to generate a denser feature map. Finally, a sigmoid layer is used to generate semantic probability outputs for drivable areas and lane line segmentation.

[0023] Step 3: Post-process the outputs of the three encoders respectively, and plot the post-processed results on the original image.

[0024] In step 4, the specific steps for linking the losses of multiple tasks, designing a multi-task loss function, and defining the total loss as the weighted sum of the losses of the three tasks include:

[0025] Step 1: Define the total loss as the weighted sum of the losses of the three tasks, so that the loss scales of the three tasks are closer.

[0026] The second step is to introduce homoscedastic uncertainty to balance multiple tasks. Learnable noise parameters are added to the loss of each task, so that the multi-task network can automatically adjust the weight parameters during training.

[0027] In step 5, the specific steps for accelerating the deployment of the multi-task model using TensorRT include:

[0028] Step 1: Use FP16 quantization to accelerate the deployment of multi-task models, compressing higher-precision data into lower-precision data, thereby improving the model's throughput while reducing the model's storage size and memory consumption.

[0029] The second step is to optimize the structure of the network model to significantly improve the inference speed while ensuring the model's accuracy.

[0030] In step 6, the specific steps for establishing a road scene understanding dataset and using the dataset to train and evaluate the multi-task learning model include:

[0031] Step 1: Collect datasets of road targets, drivable areas, and lane lines, and set the ratio of training set, validation set, and test set.

[0032] Step 2: Target detection performance is evaluated using recall and mean average precision (mAP). MIoU is used to measure drivable area detection performance, while pixel accuracy and IoU are used to measure lane line detection performance.

[0033] Step 3: Set the initial learning rate to 0.001, weight decay to 0.0005, and momentum to 0.937. Use the Adam optimizer for training optimization.

[0034] The effects and benefits of this invention are: This invention proposes a road scene understanding method for autonomous vehicles based on multi-task learning, which realizes the joint detection of traffic targets, drivable areas and lane lines, effectively saving computing resources, reducing the risk of overfitting and improving network flexibility. Attached Figure Description

[0035] Figure 1 This is a structural diagram of the multi-task learning model of the present invention.

[0036] Figure 2 Data augmentation structure diagram. Detailed Implementation

[0037] The following embodiments will further illustrate the present invention with reference to the accompanying drawings.

[0038] Considering the advantages of hard parameter sharing in multi-task models, such as simple structure, high efficiency, and reduced overfitting risk, a multi-task model based on hard parameter sharing, YOLO-ODL, is built to achieve joint and efficient detection of traffic targets, drivable areas, and lane lines. The YOLO-ODL model structure is as follows: Figure 1 As shown, the model employs an Encoder-Decoder scheme, consisting of a shared encoder and three task-specific decoders: a detection encoder, a drivable area encoder, and a lane line encoder. Compared to models using single tasks, the YOLO-ODL multi-task model shares resources, avoids redundant computation of image features, and effectively improves model efficiency.

[0039] Step 1: Design the shared encoder for the multi-task model YOLO-ODL.

[0040] Step 1.1: The Backbone and FPN structures in the YOLOv5s object detection model are adopted. The Backbone extracts common image features in the scene, and the FPN fuses image features at different scales.

[0041] Step 1.2: Generate feature maps at three scales: W / 4×H / 4×128, W / 8×H / 8×64, and W / 16×H / 16×128.

[0042] Step 2: Design the YOLO-ODL multi-task model object detection decoder.

[0043] Step 2.1: The PANet and Detection Head structures in the YOLOv5s object detection model are adopted. PANet further integrates features of different scales. The Detection Head uses convolutional layers with a kernel size of 1×1 and a stride of 1 to adjust the number of channels.

[0044] Step 2.2: Add shallow high-resolution features of 160×96 to replace the initial deep low-resolution features of 20×12, and finally generate feature maps of three scales: 160×96×18, 80×48×18, and 40×24×18.

[0045] Step 2.3: Each grid is responsible for 3 prior boxes, with a total of 61,440 prediction outputs. Each prediction output includes 4 parameters related to the position of the prediction box, 1 confidence parameter, and 1 vehicle category parameter. Therefore, the output feature map has 3×6=18 channels.

[0046] Step 3: Design the drivable area decoder and lane line decoder for the multi-task model YOLO-ODL using semantic segmentation.

[0047] Step 3.1: The Drivable Area Encoder and Lane Line Encoder consist of two CSPup layers, a CBS layer, and a Sigmoid layer.

[0048] Step 3.2: Transform the low-resolution W / 4×H / 4×128 feature map back into a high-resolution W×H×2 feature map. The two channels correspond to the number of categories. Further extract image features to generate a denser feature map. Finally, generate semantic probability outputs for drivable area and lane line segmentation through a Sigmoid layer.

[0049] Step 3.3 Perform post-processing on the outputs of the three encoders respectively, and plot the post-processing results on the original image.

[0050] Step 4: Link the losses of multiple tasks such as target detection, drivable area detection and lane detection, design a multi-task loss function, and define the total loss as the weighted sum of the losses of the three tasks.

[0051] Step 4.1: Define L total The total loss is a weighted sum of the losses from the three tasks, to make the loss scales of the three tasks more similar.

[0052] L total =α box L box +α obj L obj +α cls L cls +α drivable L drivable +α lane L lane (1)

[0053] In the formula, α box α obj and α cls These are the three loss weights for the object detection task; α drivable It is the loss weight for the drivable area detection task; α lane These are the loss weights for lane detection tasks; bounding box prediction is a regression problem, L box The CIOU loss function is used, while the BCE loss function is used for other tasks which are classification problems.

[0054] Step 4.2: Introduce homoscedastic uncertainty to balance multiple tasks, and add learnable noise parameters to the loss of each task so that the multi-task network can automatically adjust the weight parameters during training.

[0055] Step 4.2.1: Define f(x; θ) as the output of the model, x as the input, θ as the parameters of the neural network, and y as the labels of the dataset. For regression problems, the likelihood function satisfying a Gaussian distribution is defined as follows:

[0056] p(y|f(x;θ))=N(f(x;θ),σ 2 (2)

[0057] In the formula, σ represents a learnable noise parameter; N represents a Gaussian distribution.

[0058] Step 4.2.2: In the maximum likelihood inference process, we aim to maximize the log-likelihood of the model. Therefore, the log-likelihood function in the regression problem is as follows:

[0059]

[0060] Step 4.2.3: Assuming the model calculates the output probability using the Softmax function, the likelihood function in the classification problem is as follows:

[0061]

[0062] Step 4.2.4: The log-likelihood function for the classification problem is expressed as follows:

[0063]

[0064] In the formula, c represents the correctly classified category; c′ represents all categories.

[0065] Step 4.2.5: Assume the multi-task model consists of two tasks: regression and classification, with output labels y1 and y2, and the joint loss L(θ) sh The formulas (θ1, θ2, σ1, σ2) can be obtained from the previous formulas. The simplified formula process is as follows:

[0066]

[0067] In the formula, θ sh θ1 and θ2 represent shared neural network parameters; θ1 and θ2 represent task-specific neural network parameters; L1(θ sh ,θ1)=‖y1-f(x;θ sh ,θ1)‖ 2 The output f(x; θ) of the regression task sh The Euclidean distance between y1 and y2; L2(θ1) sh,θ2)=-log Softmax(y2;f(x;θ sh ,θ2)) is the cross-entropy loss in the classification task; the two log terms on the right side of the formula can be interpreted as regularization terms, which can prevent σ from being too large and causing serious training imbalance.

[0068] Step 4.2.6: To avoid generating negative loss values, the multi-task loss weight balancing formula is used as shown in equation (7), and the original loss weight α is added.

[0069]

[0070] In the formula, Y = {box,obj,cls,drivable,lane} includes three sub-tasks of object detection, drivable area detection, and lane detection.

[0071] Step 5: Use TensorRT to accelerate the deployment of multi-task models and reduce the computational complexity of the models.

[0072] Step 5.1: Use FP16 quantization to accelerate the deployment of multi-task models, compressing higher-precision data into lower-precision data, thereby improving the model's throughput while reducing the model's storage volume and memory consumption.

[0073] Step 5.2: Optimize the structure of the network model to significantly improve the inference speed while ensuring model accuracy.

[0074] Step 5.2.1: Eliminate useless operations and unused output layers to avoid unnecessary computations.

[0075] Step 5.2.2: Convert the network into a simpler representation to improve runtime efficiency, for example, by merging convolution, bias, and activation functions into a single layer.

[0076] Step 5.2.3: Perform aggregation operations or horizontal layer fusion using the same source tensor and similar parameters to produce larger layers and further improve model performance.

[0077] Step 5.2.4: Direct the outputs of each layer of the network to the correct destination to merge the concat layers, thereby reducing the number of network operations.

[0078] Step 6: Establish a road scene understanding dataset, and use the dataset to train and evaluate the multi-task learning model.

[0079] Step 6.1: Collect a dataset of road targets, drivable areas, and lane lines, totaling 100,000 images with a resolution of 1280×720. The training set contains 70,000 images, the validation set contains 10,000 images, and the test set contains 20,000 images.

[0080] Step 6.2: Target detection performance is evaluated using recall and mean average precision (mAP). MIoU is used to measure drivable area detection performance, while pixel accuracy and IoU are used to measure lane line detection performance.

[0081] Step 6.3: The initial learning rate is set to 0.001, weight decay to 0.0005, and momentum to 0.937. The Adam optimizer is used for training. The input size of the model is adjusted to 640×384 to speed up the model, and the input image is normalized.

[0082] Step 6.4: To avoid overfitting and improve the model's generalization ability, augment the data in the dataset, such as... Figure 2 As shown, the data augmentations used include rotation, scaling, translation, color space enhancement, and left / right flipping.

[0083] Step 6.5: Jointly train the three decoders and compare them with various publicly available models to complete the testing and experimental evaluation.

[0084] The above description, in conjunction with preferred technical solutions, provides a further detailed explanation of the present invention and should not be construed as limiting the specific implementation of the invention to these descriptions. For those skilled in the art, simple deductions and substitutions can be made without departing from the concept of the present invention, and all such deductions and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for understanding road scenes of autonomous vehicles based on multi-task learning, characterized in that... Includes the following steps: Step 1: Design the shared encoder for the YOLO-ODL multi-task model for road scene understanding in autonomous vehicles. Specific steps include: Step 1: Using the Backbone and FPN structures in the YOLOv5s object detection model, the Backbone extracts common image features in the scene, and the FPN fuses image features at different scales; Step 2: Generate feature maps at three scales: W / 4×H / 4×128, W / 8×H / 8×64, and W / 16×H / 16×128. Step 2: Design the object detection decoder for the YOLO-ODL multi-task model for road scene understanding in autonomous vehicles. Specific steps include: Step 1: The PANet and Detection Head structures in the YOLOv5s object detection model are adopted. PANet further integrates features at different scales. The Detection Head uses convolutional layers with a kernel size of 1×1 and a stride of 1 to adjust the number of channels. Step 2: Add a shallow high-resolution feature of 160×96 to replace the initial deep low-resolution feature of 20×12, and finally generate feature maps of three scales: 160×96×18, 80×48×18, and 40×24×18. Step 3: Each grid is responsible for 3 prior boxes, with a total of 61,440 prediction outputs. Each prediction output includes 4 parameters related to the position of the prediction box, 1 confidence parameter, and 1 vehicle category parameter. Therefore, the output feature map has 3×6=18 channels. Step 3: Design the drivable region decoder and lane line decoder for the multi-task model YOLO-ODL using semantic segmentation methods. Specific steps include: Step 1: The drivable area decoder and lane line decoder consist of two CSPup layers, a CBS layer, and a Sigmoid layer; The second step is to transform the low-resolution W / 4×H / 4×128 feature map back to the high-resolution W×H×2 feature map, with the two channels corresponding to the number of categories. Image features are further extracted to generate a denser feature map. Finally, the semantic probability output of drivable area and lane line segmentation is generated through the Sigmoid layer. Step 3: Post-process the outputs of the three encoders respectively, and plot the post-processed results on the original image; Step 4: Link the losses of multiple tasks together, design a multi-task loss function, and define the total loss as the weighted sum of the losses of the three tasks; Step 5: Use TensorRT to accelerate the deployment of multi-task models and reduce the computational complexity of the models; Step 6: Establish a road scene understanding dataset, and use the dataset to train and evaluate the multi-task learning model.

2. The method for understanding road scenes of autonomous vehicles based on multi-task learning as described in claim 1, characterized in that... In step 4, the specific steps for linking the losses of multiple tasks, designing a multi-task loss function, and defining the total loss as the weighted sum of the losses of the three tasks include: Step 1: Define the total loss as the weighted sum of the losses from the three tasks to make the loss scales of the three tasks more similar; The second step is to introduce homoscedastic uncertainty to balance multiple tasks. Learnable noise parameters are added to the loss of each task, so that the multi-task network can automatically adjust the weight parameters during training.

3. The method for understanding road scenes of autonomous vehicles based on multi-task learning as described in claim 1, characterized in that... In step 5, the specific steps for accelerating the deployment of the multi-task model using TensorRT include: Step 1: Use FP16 quantization to accelerate the deployment of multi-task models, compressing higher-precision data into lower-precision data, thereby improving the model's throughput while reducing the model's storage size and memory consumption. The second step is to optimize the structure of the network model to significantly improve the inference speed while ensuring the model's accuracy.

4. The method for understanding road scenes of autonomous vehicles based on multi-task learning as described in claim 1, characterized in that... In step 6, the specific steps for establishing a road scene understanding dataset and using the dataset to train and evaluate the multi-task learning model include: Step 1: Collect datasets of road targets, drivable areas, and lane lines, and set the ratio of training set, validation set, and test set; Step 2: Use recall and mean precision to evaluate target detection performance; use MIoU to measure drivable area detection performance; and use pixel accuracy and IoU to measure lane line detection performance. Step 3: Set the initial learning rate to 0.001, weight decay to 0.0005, and momentum to 0.

937. Use the Adam optimizer for training optimization.

Images