Intelligent driving multi-task visual environment perception method

Abstract

Environmental perception is a key technology for ensuring safe and stable driving of an intelligent vehicle. In order to solve the problem that an existing single-task model cannot meet diversified environment perception, a multi-task environment perception algorithm MTP-Net is provided, and a vehicle detection task and a lane line segmentation task can be completed at the same time. The MTP-Net is composed of an efficient shared backbone network CSP-Darknet53, a bidirectional feature pyramid structure (BiFPN) and multi-level feature map fused vehicle detection branch, and a lane line detection branch with global context features and fine-grained positioning information comprehensively reserved, the overall performance of multi-task learning is remarkably improved in combination with the design of a weighted loss function, and the detection efficiency is improved. The TensorRT is used for accelerating reasoning, the MTP-Net can be deployed and operated on embedded equipment, and the requirements for real-time performance and accuracy of intelligent driving environment perception are fully met.

Description

Technical Field

[0001] The present invention relates to the field of target detection and semantic segmentation, and in particular to a multi-task environment perception method for intelligent driving. Background Art

[0002] Real-time and accurate environmental perception technology can help intelligent vehicles make correct plans and decisions while driving, and achieve safe and stable driving of vehicles. Vehicle detection and lane segmentation are basic tasks in intelligent driving environment perception. The vehicle detection task can detect surrounding vehicles and other traffic driving elements in real time. The detection results can help intelligent vehicles understand the surrounding driving environment and make safer and more efficient driving decisions. The lane segmentation task can segment various lane signs on the road, help intelligent vehicles determine the lanes and road boundaries, and judge whether the car has deviated from the predetermined lane, thereby improving driving safety. With the rapid development of computer vision technology, many methods for solving single tasks have been proposed in the field of intelligent driving visual environment perception, and excellent results have been achieved. Considering that single-task models cannot meet the requirements of diversified environmental perception, in addition, running multiple single-task models at the same time will not only pose a huge challenge to the vehicle computing platform, but also cause inconsistencies in the prediction results in the time series. In order to achieve simultaneous perception of multiple tasks, reduce computing resource usage and improve computing efficiency, it is particularly important to develop a high-precision multi-task environmental perception algorithm that can run in real time on the vehicle platform, which has important practical research significance. Summary of the invention

[0003] The purpose of the present invention is to propose a multi-task environment perception algorithm MTP-Net to address the problem that the existing single-task model of intelligent driving cannot meet the needs of diversified perception and insufficient utilization of computing resources. This method saves on-board computing resources by setting up a shared backbone network, and sets vehicle detection branches and lane line segmentation branches to achieve real-time and accurate perception of multiple tasks.

[0004] The technical solution to achieve the above purpose is as follows:

[0005] Step 1: Data collection and organization. Clean and process the BDD100K autonomous driving dataset. Divide the dataset into training set, validation set, and test set in a ratio of 7:2:1. Label each image in the training set and validation set with the vehicle category and generate a lane mask image.

[0006] Step 2: Design of the intelligent driving multi-task visual environment perception algorithm MTP-Net, using CSP-Darknet53 in YOLOv8 as the shared backbone network, setting up vehicle detection branches and lane segmentation branches, and combining the design of weighted loss functions to complete vehicle detection tasks and lane segmentation tasks at the same time.

[0007] Step 3: MTP-Net model training. Through the design of the network structure and weighted loss function in step 2, combined with the training hyperparameter tuning, the MTP-Net algorithm in step 2 is trained to obtain the optimal model weights.

[0008] Step 4: Verify the effect of the MTP-Net algorithm. Use the optimal model weights obtained in step 3 and the test set in step 1 to verify the MTP-Net algorithm and evaluate the algorithm reasoning effect.

[0009] Step 5: Embedded deployment of the MTP-Net algorithm on the edge computing device. Simplify and convert the optimal model weights obtained in step 3, and deploy them to the embedded development board using TensorRT and C++ programming to achieve real-time and fast reasoning for vehicle detection and lane segmentation.

[0010] The BDD100K dataset in step 1 is a large-scale and diverse autonomous driving dataset, released by the AI ​​Lab (BAIR) of the University of California, Berkeley in 2018. The BDD100K dataset contains 100,000 high-definition video sequences with a total duration of more than 1,100 hours. Each video is about 40 seconds long, with a resolution of 720p and a frame rate of 30fps. The key frame is sampled at the 10th second of each video to obtain 100,000 images (with a size of 1280x720), which are annotated in detail, including the bounding boxes, types, occlusion conditions, and whether they are truncated, etc. of various objects. The annotated data also contains key information such as lane lines, traffic lights, and road signs.

[0011] The CSP-Darknet53 in step 2 is the backbone network (Backbone) used in YOLOv8. It is built on the Darknet53 network and introduces the CSP (Cross Stage Partial) structure. CSP divides the network into multiple stages, each of which contains multiple convolutional layers. In each stage, a part of the feature map directly skips the convolutional layer processing and is passed to the next stage, while the other part of the feature map is processed by the convolutional layer. This structure can reduce the amount of calculation while maintaining the accuracy of the model. CSP-Darknet53 uses deep separable convolution to decompose the convolution operation into deep convolution and point-by-point convolution. This decomposition method can significantly reduce the number of parameters of the model while maintaining the performance of the model. CSP-Darknet53 also uses residual connection to add the input feature map and output feature map of each stage. This connection can alleviate the gradient disappearance problem and improve the training stability of the model.

[0012] The TensorRT in step 5 is a high-performance deep learning inference optimization and runtime library launched by NVIDIA, which is designed to accelerate the inference process of deep learning models on NVIDIA GPUs. TensorRT significantly improves the inference speed and reduces latency by optimizing the computational graph, fusion layer, and precision reduction of deep learning models. It is very suitable for application scenarios that require high throughput and low latency, such as autonomous driving, real-time recommendation systems, and speech recognition.

[0013] Compared with the existing multi-task learning algorithms, the present invention has the following innovations:

[0014] (1) Introduced the high-precision, low-parameter shared backbone network CSP-Darknet53;

[0015] (2) In the vehicle detection branch, a multi-level feature fusion method integrating BIFPN is proposed, and a small target detection layer is added to the vehicle detection branch based on actual application scenarios;

[0016] (3) In the lane segmentation branch, the DySample lightweight upsampling operator is applied to maximize the retention of image semantic detail information, and the encoder-decoder structure and DCC module stacking are designed to achieve accurate prediction of lane line pixels;

[0017] (4) In view of the difference in complexity between vehicle detection and lane segmentation tasks, a weighted loss function is designed to balance multi-task learning and improve the performance of the MTP-Net algorithm.

[0018] Beneficial effects of the present invention:

[0019] (1) The efficient and lightweight shared backbone network and multi-task branch design make it possible to deploy multi-task learning algorithms on edge devices with insufficient computing power.

[0020] (2) The design of weighted loss function provides a reference for balancing the complexity of subtasks in multi-task learning, improving model performance, dynamically adjusting weights, and simplifying hyperparameter adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of the steps of the present invention

[0022] Figure 2 MTP-Net network structure diagram

[0023] Figure 3 MTP-Net shared backbone network structure diagram

[0024] Figure 4 Branch network structure diagram for vehicle detection

[0025] Figure 5BiFPN structure diagram

[0026] Figure 6 Lane segmentation branch network structure diagram

[0027] Figure 7 Flowchart for DySample upsampling

[0028] Figure 8 MTP-Net detection result diagram DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, a flowchart of a method for multi-task visual environment perception of intelligent driving is shown, and the specific implementation steps are as follows:

[0031] Step 1: Data collection and organization. The BDD100K autonomous driving dataset is cleaned and processed. The training set, validation set, and test set are divided in a ratio of 7:2:1. Each image in the training set and validation set is labeled with the vehicle category and a lane line mask image is generated.

[0032] Step 2, the design of the intelligent driving multi-task visual environment perception algorithm MTP-Net, using CSP-Darknet53 in YOLOv8 as the shared backbone network, setting up vehicle detection branches and lane segmentation branches, combined with the design of weighted loss functions, can complete vehicle detection tasks and lane segmentation tasks at the same time.

[0033] Step 3, MTP-Net model training, through the design of the network structure and weighted loss function in step 2, combined with the training hyperparameter tuning, the MTP-Net algorithm in step 2 is trained to obtain the optimal model weights.

[0034] Step 4: Verify the effect of the MTP-Net algorithm. Use the optimal model weights obtained in step 3 and the test set in step 1 to verify the MTP-Net algorithm and evaluate the algorithm reasoning effect.

[0035] Step 5: Embedded deployment of the MTP-Net algorithm on the edge computing device. The optimal model weights obtained in step 3 are simplified and converted into a new format. They are then deployed to the embedded development board using TensorRT and C++ programming to achieve real-time and fast reasoning for vehicle detection and lane segmentation.

[0036] The implementation process of step 1 is as follows:

[0037] Step 1: Use Python scripts to process the JSON label files in the BDD100K image training set and validation set, and merge car, bus, truck, and train into the Vehicle class for vehicle detection branch training.

[0038] Step 2: Use the Python script to process the lane control points in the JSON label file to generate a lane mask binary image for lane segmentation branch training.

[0039] Step 3: The processed dataset contains 70,000 training set images and the corresponding lane line mask binary images, 20,000 validation set images and the corresponding lane line mask binary images, and 10,000 test set images. The resolution of all images is unified to 1280*720.

[0040] The implementation process of step 2 is as follows:

[0041] like Figure 2 As shown in the figure, the network structure design process and the design of the algorithm loss function are explained in detail in combination with the MTP-Net network structure diagram.

[0042] Step 1: Use CSP-Darknet53 in YOLOv8 as the shared backbone network, such as Figure 3 As shown in the figure, it includes convolutional layer (Conv), C2f feature extraction module, and fast spatial pyramid pooling structure (SPPF). After sharing the backbone network, the input size of the feature map changes from 640*640 to 20*20, and the feature map contains richer semantic information.

[0043] Step 2: For the vehicle detection branch, such as Figure 4 As shown in the figure, the 20*20 feature map is used as the input of the branch. After upsampling, BiFPN multi-level feature fusion, C2f feature extraction module and other structures, three feature maps of 80*80, 40*40, and 20*20 are obtained for vehicle detection. In order to improve the accuracy of small target detection, a 160*160 feature map is added for tiny target capture, which effectively improves the accuracy of the vehicle detection branch.

[0044] In the second step, BiFPN is a new feature fusion method with the following structure: Figure 5 As shown in the figure. For the Pe and Ps layers, the nodes with only one input edge that contribute little to the fusion feature network are deleted, which can simplify the network structure. For the Pn and Pm layers, jump connections are added from the input to the output nodes, which can fuse more semantic features with less additional cost. Compared with the traditional PAN-FPN fusion method, BiFPN has significant advantages in cross-layer connection and weighted feature fusion.

[0045] Step 3: For the lane segmentation branch, such as Figure 6 As shown in the figure, the 20*20 feature map is used as the input of the branch and passes through the DCC module (DySample+Concat+C2f). Among them, DySample dynamic upsampling can retain the image semantic information to the greatest extent, Concat can splice feature maps of the same size to enrich the image semantic information, and C2f feature extraction can extract image features. Through the DCC module, the lane line segmentation branch can accurately segment the pixels belonging to the lane line in the image, improving the algorithm accuracy. After the lane line segmentation branch network structure, the 20*20 input feature map becomes 640*640, which is consistent with the original input image size, and decoding is achieved.

[0046] In the third step, the DySample upsampling operator dynamically selects sampling points on the feature map instead of using dynamic convolution to reorganize the feature map, which significantly reduces the computational complexity. Figure 7 As shown, given an upsampling scale factor s, a feature map χ of size C×H×W, and a point sampling set ξ of size 2g×sH×sW. The grid_sample function uses the position in the point sampling set ξ to resample the feature map χ to χ′ of size C×sH×sW, and the calculation formula is as follows:

[0047] χ'=grid_sample(χ,ξ)

[0048] Step 4: MTP-Net completes the two tasks of vehicle detection and lane segmentation at the same time, so the loss calculation also consists of two parts: L det and L llseg Among them, L det is the loss of the vehicle detection branch, which is composed of the classification loss L class , regression loss L bbox , and confidence loss L object After weighted processing, α1, α2, and α3 are weight coefficients (0.05, 0.5, and 1, respectively). The classification loss and confidence loss are both binary cross entropy losses. The regression loss is calculated using Alpha-IoU fused with CIoU. The calculation formula is as follows:

[0049] L det =α1L class +α2L bbox +α3L object

[0050] In the lane line segmentation task, the difference between the number of pixels belonging to the lane line and the number of background pixels is too large, and there is a problem of class imbalance. Therefore, the loss function L of the lane line segmentation branch is designed llseg The expression is as follows, where LFocal Focal Loss stands for Focal Loss, which is a loss function used to deal with the problem of class imbalance. It reduces the loss weight of easy-to-classify samples and increases the attention to difficult-to-classify samples, thereby improving the performance of the algorithm. tversky It stands for Tversky Loss, which improves image segmentation performance by balancing the weights between false positives and false negatives.

[0051] L llseg =L Focal +L tvsky

[0052] Combining the loss functions of the two branches, the total loss function L of the MTP-Net algorithm sum The calculation method is as follows, where ω1 and ω2 are weighted coefficients, which are 0.5 and 0.5 respectively.

[0053] L sum =ω1L det +ω2L llseg

[0054] The implementation process of step 3 is as follows:

[0055] The training was conducted on an Ubuntu 20.04 computer, with an AMD EPYC 7T83 CPU, an RTX4090 GPU, and PyTorch as the deep learning framework. The training hyperparameters were set as follows: 200 iterations, 16 batch sizes, an Adam optimizer, an initial learning rate of 0.001, and a cosine annealing algorithm to dynamically adjust the learning rate.

[0056] The implementation process of step 4 is as follows:

[0057] According to the various evaluation indicators of the model weights saved in each round of training on the validation set (vehicle detection branch: average detection precision (mAP50), recall rate (Recall), lane line segmentation branch: intersection over union (IoU), accuracy (ACC)), the best model weights are selected and saved, and the best weights are loaded on the test set to intuitively verify the effect of the MTP-Net algorithm.

[0058] The implementation process of step 5 is as follows:

[0059] Step 1: Convert the MTP-Net best model weights saved in step 4 to ONNX format.

[0060] Step 2: Convert the weights in ONNX format to Engine format using the TensorRT tool.

[0061] Step 3: Based on the weight file in Engine format, use C++ and TensorRT to accelerate program writing and deploy it to the embedded development board NVIDIA Jetson Xavier NX to achieve real-time inference of MTP-Net on low-computing edge devices.

[0062] The inference results of the MTP-Net algorithm are as follows Figure 8 As shown, the vehicle and ground lane lines in the picture can be predicted simultaneously, providing sufficient information for intelligent driving environment perception.

Claims

1. A multi-task visual environment perception method for intelligent driving, characterized in that: The steps include: Step 1: Data collection and organization. Clean and process the BDD100K autonomous driving dataset. Divide the dataset into training set, validation set, and test set in a ratio of 7:2:

1. Label each image in the training set and validation set with the vehicle category and generate a lane mask image. Step 2: Design of the multi-task visual environment perception algorithm MTP-Net, using CSP-Darknet53 as the shared backbone network, setting up vehicle detection branches and lane segmentation branches, and combining the design of the weighted loss function to complete vehicle detection tasks and lane segmentation tasks at the same time. Step 3: MTP-Net model training. Through the design of the network structure and weighted loss function in step 2, combined with the training hyperparameter tuning, the MTP-Net algorithm in step 2 is trained to obtain the optimal model weights. Step 4: Verify the effect of the MTP-Net algorithm. Use the optimal model weights obtained in step 3 and the test set in step 1 to verify the MTP-Net algorithm and evaluate the algorithm reasoning effect. Step 5: Embedded deployment of the MTP-Net algorithm on the edge computing device. Simplify and convert the optimal model weights obtained in step 3, and deploy them to the embedded development board using TensorRT and C++ programming to achieve real-time and fast reasoning for vehicle detection and lane segmentation.

2. The intelligent driving multi-task visual environment perception method according to claim 1 is characterized in that: In step 1, the BDD100K dataset is processed as follows: for each image, car, bus, truck, and train are merged into the Vehicle class for vehicle detection branch training, and the lane control points are processed to generate lane mask binary images for lane segmentation branch training. The processed dataset contains 70,000 training set images and corresponding lane mask binary images, 20,000 validation set images and corresponding lane mask binary images, and 10,000 test set images. The resolution of all images is unified to 1280*720.

3. The intelligent driving multi-task visual environment perception method according to claim 1 is characterized in that: In step 2, the main design innovations include the following: (1) The efficient and lightweight CSP-Darknet53 network in YOLOv8 is used as the shared backbone network of MTP-Net, which enhances the feature extraction capability of the algorithm and reduces the computational complexity; (2) In the vehicle detection branch, BiFPN multi-level feature fusion is used and a small target detection layer is added; (3) In the lane segmentation branch, the DySample lightweight upsampling operator is used to replace the commonly used nearest neighbor interpolation sampling, and the encoder-decoder lane segmentation network structure is designed. The DCC module stacking is used to complete the accurate lane pixel segmentation; (4) Weighted loss function design: In view of the complexity differences between vehicle detection and lane segmentation tasks, a weighted loss function is designed to balance multi-task learning in order to achieve the best training effect.

4. The intelligent driving multi-task visual environment perception method according to claim 1 is characterized in that: In step 3, the training is performed on an Ubuntu 20.04 computer, with an AMD EPYC 7T83 CPU, an RTX4090 GPU, and PyTorch as the deep learning framework. The training hyperparameters are set as follows: 200 iterations, 16 batch sizes, Adam optimizer, 0.001 initial learning rate, and cosine annealing algorithm to dynamically adjust the learning rate.

5. The intelligent driving multi-task visual environment perception method according to claim 1 is characterized in that: In step 4, the best model weights are selected and saved based on various evaluation indicators of the model weights saved in each round of training on the validation set (vehicle detection branch: average detection precision (mAP50), recall rate (Recall), lane line segmentation branch: intersection over union (Iou), accuracy (ACC)), and the MTP-Net algorithm effect is intuitively verified on the test set.

6. The intelligent driving multi-task visual environment perception method according to claim 1 is characterized in that: In step 5, the best model weights saved in step 4 are converted to ONNX format, and then converted to Engine format through the TensorRT tool. They are then deployed to the embedded development board NVIDIA Jetson Xavier NX using C++ programming and TensorRT acceleration to achieve real-time inference of MTP-Net on low-computing edge devices.

Images