A High-Precision Lightweight Multi-Task Network Method for Traffic Scenarios

By introducing EALAN and Ghostconv2 modules into the multi-task network, feature representation and lightweighting are enhanced, solving the problems of low accuracy and poor real-time performance in small target detection, and realizing the real-time perception capability of a high-precision, lightweight multi-task network.

CN117854043BActive Publication Date: 2026-11-10DALIAN UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410031881.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-09
Publication Date
2026-11-10
Estimated Expiration
2044-01-09

Smart Images

  • Figure CN117854043B_ABST
    Figure CN117854043B_ABST
Patent Text Reader

Abstract

This invention discloses a high-precision, lightweight multi-task network method for traffic scenarios, comprising the following steps: designing the overall architecture of the multi-task network; building a backbone feature extraction network; building a neck feature extraction network; building task branch networks; and training the overall network. In the backbone feature extraction network, this invention uses a constructed EALAN module to improve feature extraction capabilities by controlling gradient paths, while also focusing on information between different channels and spaces to improve feature representation. A small target detection head is designed in target detection to make the network pay more attention to minute vehicle information, further improving target detection accuracy. In the neck feature extraction network, this invention uses the lightweight module Ghostconv2, which not only captures global information but also makes the network lightweight, improving its operating speed. Therefore, this invention can perceive and respond to different traffic scenarios in real time on embedded devices.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent driving, and more particularly to a high-precision, lightweight, multi-task network method for traffic scenarios. Background Technology

[0002] In panoramic driving perception, visual information provides rich scene data and helps guide vehicles to achieve intelligent driving. In the transportation field, the main visual perception tasks include object detection, drivable area segmentation, and lane detection. Object detection helps vehicles identify the size and positional relationships of surrounding objects, while drivable area segmentation and lane detection provide crucial information for planning safe driving routes. Despite the rapid development of deep learning, single object detection, segmentation, and lane detection algorithms are insufficient to meet the complex requirements of intelligent driving. Therefore, multi-task networks with high-precision real-time perception capabilities play a vital role in panoramic driving. Multi-task networks employ a shared encoder approach to generate multi-scale feature maps containing different semantic information and pass them to decoders for different tasks to achieve parallel processing of multiple tasks.

[0003] In current multi-task network methods, the downsampling factor is relatively large, which prevents the network from effectively learning the feature information of small targets. Due to this limitation, small target detection faces false positives and false negatives. Furthermore, the network model is relatively complex with a large number of parameters, thus reducing the real-time performance of inference. For example, a multi-task network road target detection method for autonomous driving (patent application number: 202010002358.3) uses Ghost lightweight convolutions to improve the inference speed of the network, but it does not consider small target detection, thus resulting in false positives and false negatives. For example, a multi-task target detection and recognition method and network structure based on YOLOv5 (patent application number: 202111455472.2) uses the Mosaic method for data augmentation, enabling the simultaneous processing of multiple images in a single forward propagation to improve computational efficiency; however, this method does not consider small targets, resulting in low detection accuracy in target detection tasks. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention proposes a high-precision, lightweight, multi-task network method for traffic scenarios that can improve the accuracy of target detection, drivable area segmentation, and lane detection, while also solving the problem of poor real-time performance when deployed on embedded platforms.

[0005] The basic idea of ​​this invention is as follows: First, an Efficient AttentionLayer Aggregation Network (EALAN) module is designed. This module performs convolutions on the feature maps through an efficient attention path, enabling the network to focus on the channel and spatial feature information of the feature maps, thereby enhancing feature representation capabilities. Then, the Ghostconv2 module is referenced in the network's Neck part. By employing a lightweight convolutional structure, the network efficiency is improved while capturing global information, accelerating network inference speed. Finally, a small object detection head is designed, enabling the network to decode category, location, and confidence features in feature maps downsampled by 4 times. This better captures the semantic information of small objects in high-resolution feature maps, significantly improving recall and mAP in object detection tasks.

[0006] The technical solution of this invention is as follows: A high-precision, lightweight, multi-task network method for traffic scenarios, comprising the following steps:

[0007] A. Design the overall architecture of a multi-task network

[0008] The overall architecture of the multi-task network consists of an encoder and a decoder. The encoder includes a backbone feature extraction network and a neck feature extraction network. The decoder consists of an object detection branch network, a drivable region segmentation branch network, and a lane line detection branch network. The specific steps are as follows:

[0009] A1. Input the image with a resolution of 640×640 into the backbone feature extraction network.

[0010] A2. Input the obtained feature maps with resolutions of 20×20, 40×40, 80×80, and 160×160 into the neck feature extraction network to enrich the semantic information of the feature maps.

[0011] A3. Input the feature maps with resolutions of 20×20, 40×40, 80×80, and 160×160 encoded by the neck feature extraction network into the target detection branch network, the drivable area segmentation branch network, and the lane line detection branch network for decoding to obtain the final detection results.

[0012] B. Constructing the backbone feature extraction network

[0013] Feature extraction is performed in the backbone feature extraction network using convolutional layers and the constructed efficient attention layer aggregation network module, namely the EALAN module. The specific steps are as follows:

[0014] B1. The image is processed through a convolutional layer to extract features and downsample them to obtain a feature map.

[0015] B2. Constructing the EALAN module. First, the feature map is transformed by using 1×1 convolutions to change the number of channels. Then, the feature map is sequentially processed by a channel attention module (CAM) and a spatial attention module (SAM) to extract key features. Finally, the obtained feature information is input into an efficient gradient path network formed by stacked convolutions for feature extraction. The EALAN module formula is as follows:

[0016]

[0017] F2=Concat[CBS*4(F1),CBS*2(F1),CBS(X),F1]

[0018] In the formula, X is the input feature map, F1 is the intermediate feature map after extracting key features, and F2 is the output feature map. The asterisk (*) represents the number of consecutive iterations of the CBS module, which consists of convolutional normalization and the SiLU activation function. CAM processes the input feature map through two parallel max-pooling and average-pooling layers to obtain two C×1×1 feature maps. These two 1×1 feature maps are then passed through a fully connected layer and a ReLU activation function to obtain two activation results. These two activation results are element-wise summed and then passed through a sigmoid activation function to obtain the CAM output. Finally, the CAM output is multiplied by the input feature map to obtain a feature map with channel attention. SAM processes the feature map through max-pooling and average-pooling to obtain two 1×H×W feature maps. These two 1×H×W feature maps are then concatenated and passed through a 7×7 convolution and a sigmoid activation function to obtain the SAM feature map. Finally, the SAM feature map is multiplied by the input feature map to obtain a feature map with spatial attention.

[0019] B3. After the feature map is output from the EALAN module, it is downsampled using a max pooling layer to obtain a low-resolution feature map.

[0020] C. Construct a neck feature extraction network

[0021] In the neck feature extraction network, a lightweight attention module, namely the Ghostconv2 module, is used to extract features. The specific steps are as follows:

[0022] C1. Introducing the GhostConv2 module. First, the feature map is downsampled using an average pooling layer to reduce the number of parameters while preserving the overall feature information of the image. Then, a fully connected layer progressively obtains long-distance correlation features in the horizontal and vertical directions. Finally, the feature map obtained by upsampling using the Sigmoid activation function and bilinear interpolation is element-wise multiplied with the GhostConv phantom convolution module to obtain the final result. The GhostConv2 module formula is as follows:

[0023] F1=Sigmoid(Vertical_FC(Horizontal_FC(Convolution(Avg_pool(X)))))

[0024]

[0025] Vertical_FC represents a fully connected feature map in the vertical direction, Horizontal_FC represents a fully connected feature map in the horizontal direction, Convolution represents a convolution operation on the feature map, Avg_pool represents an average pooling operation on the input feature map, and GhostConv represents a phantom convolution on the feature map.

[0026] C2. Input the output feature map of the backbone feature extraction network into the feature pyramid structure in the neck feature extraction network for feature splicing.

[0027] D. Build a task branch network

[0028] The task branch network is built by constructing an object detection branch network, a drivable area segmentation branch network, and a lane line detection branch network to achieve feature decoding for the object detection task, the drivable area segmentation task, and the lane line detection task.

[0029] D1. Constructing the target detection branch network

[0030] A small target detection head is added to the target detection branch network, which consists of four detection heads and simultaneously detects small, small-medium-small, medium-sized, and large target vehicles. The loss function for target detection is L. det The target classification loss L cls Target confidence loss L obj and the target bounding box L box The weighted summation is calculated using the following formula:

[0031] L det =a1L cls +a2L obj +a3L box

[0032] In the formula, a1 is L cls The weight of the loss, a2 is L obj The weight of the loss, a3 is L box The weight of the loss.

[0033] D2. Construct a drivable area segmentation branch network.

[0034] The feature map, after concatenating shallow information from the neck feature extraction network, is input into the drivable region segmentation branch network. An upsampling method using four-fold bilinear interpolation is applied to restore the feature map to its original size, thus obtaining the predicted category. The loss function for drivable region segmentation is as follows:

[0035] L da-seg =L CE

[0036] Where L CE This is the cross-entropy loss function.

[0037] D3. Construct a lane line detection branch network

[0038] The feature map output from the neck feature extraction network is input into the lane detection branch network. After two upsampling operations, the resolution is restored to the original image size, resulting in a final feature map with two channels. These two channels represent the probabilities of predicting the lane line background and the lane line foreground, respectively. The higher probability value is selected as the final prediction result to achieve lane line detection. The loss function for lane line detection is as follows:

[0039] L ll-seg =L CE +L IoU

[0040]

[0041] Among them, L CE Let L be the cross-entropy loss function. IoU The IOU loss function is used for lane detection. ll-seg It consists of the cross-entropy loss function and the IoU loss function.

[0042] E. Training the overall network

[0043] E1. The training dataset used is the BDD100K dataset. This dataset contains information on object detection, drivable region segmentation, and lane line detection.

[0044] E2. The total loss of the network is a weighted sum of the object detection loss, the drivable region segmentation loss, and the lane detection loss, as shown in the following formula:

[0045] L all =γ1L det +γ2L da-seg +γ3L ll-seg

[0046] In the formula, γ1 is L det The weight of the total loss, γ2 is L da-seg The weight of the total loss, γ3 is L ll-segThe weight of the total loss.

[0047] Compared with existing technologies, the present invention has the following advantages:

[0048] 1. This invention uses a constructed EALAN module in the backbone feature extraction network to improve feature extraction capability by controlling the gradient path, while also focusing on information between different channels and spaces to improve feature representation. A small target detection head is designed in target detection to make the network pay more attention to small vehicle information, further improving target detection accuracy.

[0049] 2. This invention incorporates the lightweight module Ghostconv2 in the neck feature extraction network, which not only captures global information but also reduces the network's weight and improves its operating speed. Therefore, this invention can perceive and respond to different traffic scenarios in real time on embedded devices. Attached Figure Description

[0050] Figure 1 This is the overall architecture diagram of the present invention.

[0051] Figure 2 This is the structural diagram of the backbone feature extraction network of the present invention.

[0052] Figure 3 This is a structural diagram of the EALAN module of the present invention.

[0053] Figure 4 This is a diagram of the neck feature extraction network structure of the present invention.

[0054] Figure 5 This is a structural diagram of the Ghostconv2 module of the present invention.

[0055] Figure 6 This is a diagram of the target detection branch network structure of the present invention.

[0056] Figure 7 This is a diagram of the drivable area segmentation branch network structure of the present invention.

[0057] Figure 8 This is a diagram of the lane line detection branch network structure of the present invention. Detailed Implementation

[0058] The invention will be further described below with reference to the accompanying drawings. Figure 1 As shown, the present invention includes the following steps:

[0059] A. Design the overall architecture of a multi-task network

[0060] like Figure 1This is an overall architecture diagram of a high-precision, lightweight, multi-task network method for traffic scenarios. The overall architecture of the multi-task network consists of an encoder and a decoder. The encoder comprises two parts: a backbone feature extraction network and a neck feature extraction network. The decoder consists of an object detection branch network, a drivable region segmentation branch network, and a lane detection branch network. The specific steps are as follows:

[0061] A1. Input the image with a resolution of 640×640 into the backbone feature extraction network.

[0062] A2. Input the obtained feature maps with resolutions of 20×20, 40×40, 80×80, and 160×160 into the neck feature extraction network to enrich the semantic information of the feature maps.

[0063] A3. Input the feature maps with resolutions of 20×20, 40×40, 80×80, and 160×160 encoded by the neck feature extraction network into the target detection branch network, the drivable area segmentation branch network, and the lane line detection branch network for decoding to obtain the final detection results.

[0064] B. Constructing the backbone feature extraction network

[0065] Figure 2 This is the backbone feature extraction network structure diagram of a high-precision, lightweight, multi-task network method for traffic scenarios. The EALAN module is used for feature extraction in the backbone feature extraction network, and the specific steps are as follows:

[0066] B1. The 640×640 resolution image is subjected to feature extraction and downsampling using the convolutional normalized SiLU activation function to obtain a feature map with a resolution of 160×160.

[0067] B2. Construct the EALAN module. Figure 3This is a diagram of the EALAN module structure, a high-precision, lightweight, multi-task network method for traffic scenarios. First, 1×1 convolutions are applied to change the number of channels in the feature map. Then, the feature map is sequentially passed through the Channel Attention Module (CAM) and the Spatial Attention Module (SAM) for inference. In the Channel Attention Module, the feature map is first subjected to global max pooling and global average pooling, and the pooled features are element-wise summed and activated using the Sigmoid function to obtain the channel attention map. In the Spatial Attention Module, the feature map is subjected to max pooling and average pooling along the channel dimension, and the pooled features are stacked along the channel dimension. Finally, convolution and sigmoid activation are performed to obtain the spatial attention map. Multiplying the obtained attention map with the input feature map, the weighted weights of the attention map highlight important features in different channels and spatial locations, thereby enhancing the network's ability to perceive key regions of the image. Finally, the obtained feature information is input into an efficient gradient path network formed by stacking convolutional modules. This network performs a concat operation on four consecutive 3×3 convolutional modules for feature extraction. The EALAN module formula is as follows:

[0068]

[0069] F2=Concat[CBS*4(F1),CBS*2(F1),CBS(X),F1]

[0070] In this formula, X is the input feature map, F1 is the intermediate feature map after extracting key features, and F2 is the output feature map. The multiplication is represented by a symbol (*), and the number of consecutive iterations of the CBS module is indicated by a symbol (*). The CBS module consists of convolutional normalization and the SiLU activation function. CAM processes the input feature map through two parallel max-pooling and average-pooling layers to obtain two C×1×1 feature maps. These two 1×1 feature maps are then processed through a fully connected layer and a ReLU activation function to obtain two activation results. These two activation results are element-wise summed and then passed through a sigmoid activation function to obtain the CAM output. Finally, the CAM output is multiplied by the input feature map to obtain a feature map with channel attention. SAM processes the feature map through max-pooling and average-pooling to obtain two 1×H×W feature maps. These two 1×H×W feature maps are then concatenated and passed through a 7×7 convolution and a sigmoid activation function to obtain the SAM feature map. Finally, the SAM feature map is multiplied by the input feature map to obtain a feature map with spatial attention.

[0071] B3. After the feature map is output from the EALAN module, it is downsampled using a max-pooling layer to obtain a low-resolution feature map. According to... Figure 2As shown, after three consecutive EALAN modules and max pooling layer downsampling, inputting the EALAN module again yields a feature map with a resolution of 20×20.

[0072] C. Construct a neck feature extraction network

[0073] Figure 4 This is a network structure diagram for neck feature extraction, a high-precision, lightweight, multi-task network method for traffic scenarios. The lightweight attention module Ghostconv2 is used in the neck feature extraction network to extract features. The specific steps are as follows:

[0074] C1. Import the Ghostconv2 module. Figure 5 This is the architecture diagram of the GhostConv2 module, a high-precision, lightweight, multi-task network method for traffic scenarios. First, the feature map is downsampled using an average pooling layer to reduce the number of parameters while preserving the overall feature information of the image. Then, long-distance correlation features in the horizontal and vertical directions are gradually obtained through horizontal and vertical fully connected layers. Finally, the feature map obtained by upsampling using the Sigmoid activation function and bilinear interpolation is element-wise multiplied with the GhostConv phantom convolutional module to obtain the final result. The GhostConv2 module formula is as follows:

[0075] F1=Sigmoid(Vertical_FC(Horizontal_FC(Convolution(Avg_pool(X)))))

[0076]

[0077] Where X is the input feature map, F1 is the intermediate feature map, and F2 is the output feature map. This is element-wise multiplication.

[0078] Here, Horizontal_FC uses a 1×5 convolution kernel, representing a fully connected feature map in the horizontal direction. Vertical_FC uses a 5×1 convolution kernel, representing a fully connected feature map in the vertical direction. Convolution represents performing a convolution operation on the feature map, Avg_pool represents performing an average pooling operation on the input feature map, and GhostConv represents performing a phantom convolution on the feature map.

[0079] C2. Input the output feature map of the backbone feature extraction network into the neck feature extraction network. The feature map is first passed through a spatial pyramid, then through the Ghostconv2 module, upsampling for feature extraction, and then concatenated along the channel by a feature layer with the same resolution as the backbone feature extraction network to finally obtain the output feature map.

[0080] D. Build a task branch network

[0081] The task branch network is built by constructing an object detection branch network, a drivable area segmentation branch network, and a lane line detection branch network to achieve feature decoding for the object detection task, the drivable area segmentation task, and the lane line detection task.

[0082] D1. Constructing the target detection branch network

[0083] Figure 6 This is a high-precision, lightweight, multi-task network method for traffic scenarios, featuring a target detection branch network structure. In the target detection branch, a path aggregation network structure is used for feature fusion and concatenation. PAN extracts features from the bottom up using the EALAN module, reduces resolution using a max-pooling layer, and concatenates the feature map with the neck feature extraction network to enrich the semantic information of the feature map, achieving multi-scale object detection and effectively improving target detection accuracy. This invention adds a small target detection head to the feature map downsampled by 4 times in the PAN structure. Because this layer's feature map has higher resolution and contains more semantic information, the network can more easily capture details and local features in the image, thus improving the detection performance of small targets. Therefore, this network predicts category information, location information, and confidence information for objects at different scales on feature maps downsampled by 32 times, 16 times, 8 times, and 4 times. Each detection head initially generates three-scale anchorboxes using the k-means method. These anchorboxes can be moved and resized based on the acquired feature information to achieve target location regression. Introducing a small object detection head makes it easier for the network to focus on the features of small objects, enabling the simultaneous detection of small, medium-small, medium, and large target vehicles, significantly improving the recall and mAP of object detection. The loss function for object detection is L. det It is the target classification loss L cls Target confidence loss L obj and the target bounding box L box The weighted summation is calculated using the following formula:

[0084] L det =a1L cls +a2L obj +a3L box

[0085] In this formula, α1 is L cls The weight of the loss, L cls It is the loss used to measure the classification accuracy of the target category, α2 is L obj The weight of the loss, L objThe loss indicates whether the target is included in the prediction box; α3 is the loss of L. box The weight of the loss, L box It is a loss function that measures the accuracy of the bounding box location. In this invention, L cls ,L obj ,L box All are 1.

[0086] D2. Construct a drivable area segmentation branch network.

[0087] Figure 7 This is a high-precision, lightweight, multi-task network method for traffic scenarios, specifically a drivable region segmentation branch network structure diagram. A 40×40 resolution feature map from the neck feature extraction network is input into the drivable region segmentation branch network. A four-fold bilinear interpolation upsampling method is used to restore the feature map to its original size. Each pixel in the feature map has two channels, representing the probability of predicting the background and the probability of predicting the drivable road area, respectively. These two probabilities are compared, and the value with the higher probability is selected as the final prediction category, thus achieving drivable region segmentation. The loss function for drivable region segmentation is as follows:

[0088] L da-seg =L CE

[0089] Where L CE This is the cross-entropy loss function.

[0090] D3. Construct a lane line detection branch network

[0091] Figure 8 This is a high-precision, lightweight, multi-task network method for traffic scenarios, presenting a lane detection branch network structure. The 160×160 resolution feature map output from the neck feature extraction network is input into the lane detection branch network. After two EALAN module feature extractions and upsampling operations, the resolution is restored to the original image size. The branch network has two channels, representing the probabilities of predicting the lane line background and the lane line foreground, respectively. The higher probability value is selected as the final prediction result to achieve lane line detection. The loss function for lane detection is as follows:

[0092] L ll-seg =L CE +L IoU

[0093]

[0094] Where L CE Let L be the cross-entropy loss function. IoU The IOU loss function is used for lane detection. ll-segIt consists of the cross-entropy loss function and the IoU loss function.

[0095] E. Training Network

[0096] E1. The dataset used for training in this invention is the BerkeleyDeepDrive Dataset (BDD100K). This dataset contains information such as object detection, drivable area segmentation, and lane detection, and includes 70,000 training images, 10,000 validation images, and 20,000 test images.

[0097] E2. The total loss of the network is a weighted sum of the object detection loss, the drivable region segmentation loss, and the lane detection loss, as shown in the following formula:

[0098] L all =γ1L det +γ2L da-seg +γ3L ll-seg

[0099] In the formula, γ1 is L det The weight of the total loss, γ2 is L da-seg The weight of the total loss, γ3 is L ll-seg The weights of the losses are assigned to the total loss. Loss weights balance the training difficulty across different tasks, ensuring the network can fully learn all tasks. In this invention, the weight coefficients for object detection loss, drivable region segmentation loss, and lane detection loss are all 0.2.

[0100] E3. This model was trained using an NVIDIA GeForce RTX 3090 GPU with a batch size of 8, for a total of 200 rounds of end-to-end training. AdamW was used as the optimizer during training, with an initial learning rate of 0.001.

[0101] The basic principles, main features, and advantages of this invention have been described above. Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed.

Claims

1. A method for constructing a high-precision, lightweight, multi-task network for traffic scenarios, characterized in that: Includes the following steps: A. Design the overall architecture of a multi-task network The overall architecture of the multi-task network consists of an encoder and a decoder; the encoder includes a backbone feature extraction network and a neck feature extraction network; the decoder consists of an object detection branch network, a drivable region segmentation branch network, and a lane line detection branch network; the specific steps are as follows: A1. Input the image with a resolution of 640×640 into the backbone feature extraction network; A2. Input the obtained feature maps with resolutions of 20×20, 40×40, 80×80, and 160×160 into the neck feature extraction network to enrich the semantic information of the feature maps; A3. Input the feature maps with resolutions of 20×20, 40×40, 80×80, and 160×160 encoded by the neck feature extraction network into the target detection branch network, the drivable area segmentation branch network, and the lane line detection branch network for decoding to obtain the final detection result. B. Constructing the backbone feature extraction network Feature extraction is performed in the backbone feature extraction network using convolutional layers and the constructed efficient attention layer aggregation network module, namely the EALAN module. The specific steps are as follows: B1. The image is processed through a convolutional layer to extract features and downsample them to obtain a feature map; B2. Construct the EALAN module; First, change the number of channels in the feature map using 1×1 convolutions; then, extract key features from the feature map sequentially using the Channel Attention Module (CAM) and the Spatial Attention Module (SAM); finally, input the obtained feature information into an efficient gradient path network formed by stacked convolutions for feature extraction; the EALAN module formula is as follows: F1=SAM[CAM(CBS(X)) CBS(X)] [CAM(CBS(X)) CBS(X)] F2=Concat[CBS 4(F1),CBS 2(F1),CBS(X),F1] In the formula, X is the input feature map, F1 is the intermediate feature map after extracting key features, and F2 is the output feature map. Represents numerical multiplication. This represents the number of consecutive CBS modules performed. The CBS module consists of convolutional normalization and the SiLU activation function. CAM passes the input feature map through two parallel max pooling layers and average pooling layers to obtain two feature maps with a resolution of C×1×1. Then, the two feature maps with a resolution of C×1×1 are passed through a fully connected layer and the ReLU activation function to obtain two activation results. The results of these two activations are added element-wise, and then passed through a sigmoid activation function to obtain the output of CAM. Finally, the output of CAM is multiplied with the input feature map to obtain the feature map with channel attention. SAM obtains two 1×H×W feature maps by max pooling and average pooling. The two 1×H×W feature maps are then concatenated, and then passed through a 7×7 convolution and a sigmoid activation function to obtain the feature map of SAM. Finally, the feature map of SAM is multiplied with the input feature map to obtain the feature map with spatial attention. B3. After the feature map is output from the EALAN module, it is downsampled using a max pooling layer to obtain a low-resolution feature map. C. Construct a neck feature extraction network The Ghostconv2 module, a lightweight attention module, is used in the neck feature extraction network to extract features. The specific steps are as follows: C1. Introducing the Ghostconv2 module: First, the feature map is downsampled using an average pooling layer to reduce the number of parameters while preserving the overall feature information of the image. Then, long-distance correlation features in the horizontal and vertical directions are gradually obtained through a fully connected layer. Finally, the feature map obtained by upsampling using the Sigmoid activation function and bilinear interpolation is multiplied element-wise by the Ghostconv2 module to obtain the final result. The formula for the Ghostconv2 module is as follows: F1=Sigmoid(Vertical_FC(Horizontal_FC(Convolution(Avg_pool(X))))) F2=GhostConv(X) F1 in Vertical_FC This represents a fully connected feature map in the vertical direction. Horizontal_FC This represents a fully connected feature map in the horizontal direction. Convolution This represents performing a convolution operation on the feature map. Avg_pool This represents performing average pooling on the input feature map. GhostConv This represents performing a phantom convolution on the feature map; C2. Input the output feature map of the backbone feature extraction network into the feature pyramid structure in the neck feature extraction network for feature splicing; D. Build a task branch network The task branch network is built by building an object detection branch network, a drivable area segmentation branch network, and a lane line detection branch network to achieve feature decoding for the object detection task, the drivable area segmentation task, and the lane line detection task. D1. Constructing the target detection branch network A small target detection head is added to the target detection branch network, which consists of four detection heads and can simultaneously detect small, medium-small, medium and large target vehicles. Loss function L for object detection det The target classification loss L cls Target confidence loss L obj and the target bounding box L box The weighted summation is calculated using the following formula: L det =a1L cls +a2L obj +a3L box In the formula, a1 is L cls The weight of the loss, a2 is L obj The weight of the loss, a3 is L box The weight of the loss; D2. Construct a drivable area segmentation branch network. The feature map, after concatenating shallow information from the neck feature extraction network, is input into the drivable region segmentation branch network. A 4-order bilinear interpolation upsampling method is used to restore the feature map to its original size, thus obtaining the predicted category result. The loss function for drivable region segmentation is as follows: L da-seg =L CE Where L CE The cross-entropy loss function; D3. Construct a lane line detection branch network The feature map output from the neck feature extraction network is input into the lane detection branch network. The feature map is then upsampled twice to restore its resolution to the original image size, resulting in a final feature map with two channels. These two channels represent the probabilities of predicting the lane line background and the lane line foreground, respectively. The higher probability value is selected as the final prediction result to achieve lane line detection. The loss function for lane line detection is as follows: L ll-seg =L CE +L IoU Among them, L CE Let L be the cross-entropy loss function. IoU The IOU loss function; the lane detection loss function L ll-seg It consists of the cross-entropy loss function and the IoU loss function; E. Training the overall network E1. The dataset used for training is the BDD100K dataset; the dataset contains information on object detection, drivable area segmentation, and lane line detection. E2. The total loss of the network is a weighted sum of the object detection loss, the drivable region segmentation loss, and the lane detection loss, as shown in the following formula: L all =g1L det +g2L da-seg +g3L ll-seg In the formula, g1 is L det The weight of the total loss, g2 is L da-seg Weight of total loss, g3 is L ll-seg The weight of the total loss.

Citation Information

Patent Citations

  • A predictive decision fusion method for intelligent vehicles considering two-way vehicle interaction

    CN110936954B

  • Multitask target detection and identification method and network structure based on yov5

    CN114267049A