Lane line detection and classification method and system based on bidirectional separation attention

Abstract

The application provides a lane line detection and classification method and system based on bidirectional separated attention, and the method comprises the following steps: constructing a lightweight lane line detection and classification network LNet, wherein the lightweight lane line detection and classification network LNet comprises a symmetric unit and a dense symmetric block, and the dense symmetric block is used to densely connect no less than two symmetric units; and a bidirectional separated attention mechanism is constructed, so as to construct a dependence relationship between long-distance pixels and obtain global position information and local detail information. The application solves the technical problems of large parameter quantity and poor segmentation performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a lane line detection and classification method and system based on bidirectional separation attention. BACKGROUND

[0002] Image segmentation is a multi-classification problem of pixel deep features, which attempts to combine the pixels in the image that meet certain specific semantics by assigning each pixel a class label of the corresponding attribute, and presents different class regions that meet the visual judgment. Therefore, the present application applies the segmentation idea to the lane line detection and classification task. The visual-based segmentation method can provide the lane line position and category information in the road scene for the environment perception of the autonomous vehicle, which plays a very important role in the safety of the autonomous vehicle during driving.

[0003] In early image segmentation tasks, features such as color, gradient and geometric shape are mostly selected for image segmentation. Threshold-based, edge-based, clustering-based, graph-based and region-based segmentation are common segmentation algorithms. In recent years, due to strong feature extraction capability and semantic information understanding capability, CNN has been applied to multiple image segmentation fields. Although semantic segmentation networks such as FCN, UNet, SegNet and Deeplab series networks have achieved good segmentation performance, the network parameter quantity is large. Subsequently, ENet, ERFNet and BiseNet series networks consider the problem of too many network parameters, and realize the balance between segmentation performance and parameter quantity by constructing light modules. However, since there is a correlation between the targets in the image, if only a single pixel is focused on and the dependency between pixels is not considered, the segmentation performance of the network will be reduced. The existing patent literature with the publication number CN110688971A, "A method, device and equipment for detecting dashed lane lines" includes: performing feature extraction on a to-be-detected road image to obtain a feature map of the road image; determining a lane line region in the road image and an endpoint pixel point in the road image according to the feature map; the endpoint pixel point is a pixel point that may be an endpoint of a dashed lane line in the road image; based on the lane line region and the endpoint pixel point, determine the dashed lane line in the road image. From the refinement content in the prior art solution, it can be seen that the prior art uses FCN as a feature extraction network, and after multiple convolution downsampling, a high-dimensional feature map of the road sample image is obtained, and then the high-dimensional feature map conv1 is deconvolution up-sampled to obtain an image feature us_conv1. Then the image feature us_conv1 is input to a region prediction network and an endpoint prediction network. It can be seen that the prior art does not further disclose a feasible solution for image distortion and feature precision loss in the down-sampling and up-sampling processes, and cannot guarantee the accuracy of lane line feature extraction. At the same time, the prior art does not fully consider the correlation between targets in the image, which reduces the segmentation performance of the network. The existing patent literature with the publication number CN112287912A, "Lane line detection method and device based on deep learning" includes: acquiring a first picture to be detected; inputting the first picture into a target neural network model to obtain a target feature map, wherein the target neural network model includes a neural network model generated based on a multi-scale attention mechanism and a depth separable convolution model, and the target feature map is used to represent the probability that each pixel in the first picture is a lane line pixel; performing image post-processing on the target feature map to obtain a target detection result. The specification embodiment of the prior art solution records: the first feature map is subjected to convolution operation respectively, and different convolution kernels are used to fuse the information of lane line elements under different receptive field scales.Three feature maps are output, specifically, two second type feature maps and one first type feature map, and the correlation between elements is calculated to determine the importance of each pixel position in the feature map to the global inference, and then a method in statistics is used to count the value of each position element from the dimension of the channel. It can be seen that although the correlation between elements in the feature map is considered, using the statistical processing logic disclosed by the prior art, only the connected domain not belonging to the lane line can be excluded and the conventional lane can be recognized, and the local features and global features in the feature image cannot be fully fused, which still restricts the image segmentation performance of the network model in specific application scenarios such as lane line occlusion, and the network parameter quantity in the network model used by the prior art is large, which increases the operation difficulty.

[0004] In summary, the prior art has the technical problems of large parameter quantity and poor segmentation performance. SUMMARY

[0005] The technical problem to be solved by the present application is how to solve the technical problems of large parameter quantity and poor segmentation performance in the prior art.

[0006] The present application solves the above technical problems by adopting the following technical solutions: a lane line detection and classification method based on bidirectional separated attention includes:

[0007] S1, constructing a lightweight lane line detection and classification network LNet, wherein the lightweight lane line detection and classification network LNet includes: a symmetric unit and a dense symmetric block, and the dense symmetric block is used to densely connect not less than 2 symmetric units, wherein step S1 includes:

[0008] S11, processing the input image by the encoder downsampling of the lightweight lane line detection and classification network LNet to gradually reduce the resolution of the input image, so as to obtain a small resolution feature map;

[0009] S12, processing the last layer convolution image in the encoder by the decoder upsampling to restore the small resolution feature map to the original size of the input image, wherein the output features of the decoder and the encoder are fused to obtain a fusion feature;

[0010] S13, training the lightweight lane line detection and classification network LNet by using a weighted cross-entropy loss function;

[0011] S2, constructing a bidirectional separated attention mechanism to build a long-distance pixel dependency relationship and obtain global position information and local detail information, and step S2 includes:

[0012] S21, adding a bidirectional separated attention module TSA to the lightweight lane line detection and classification network LNet to construct a bidirectional attention lightweight lane line detection and classification network TSANet;

[0013] S22, the target texture and target position information are integrated from the horizontal pixels and the vertical pixels respectively by using the bidirectional separation attention module TSA, the fusion feature is taken as an input feature map, the bidirectional separation attention module TSA is used to process the input feature map, and a suitable mapping relationship is obtained as a dependency relationship between long-distance pixels.

[0014] The application is based on the idea of semantic segmentation, and the encoder-decoder model is redesigned to generate a lightweight network (LNet) with fewer parameters. Secondly, in order to make the model robust to challenging environments, a bidirectional separation attention (TSA) is introduced to construct the dependency relationship between long-distance pixels. Finally, the TSA is integrated into the LNet to obtain more accurate lane detection and classification results. The TSANet used in the application improves the lane detection accuracy on TuSimple and other datasets, optimizes the image segmentation effect, and is suitable for real-time lane detection and classification in intelligent driving scenarios.

[0015] The application proposes a symmetric unit with a symmetric structure. Compared with the subunit of the LLNet used in the prior art, the symmetric unit uses one-dimensional convolution with a smaller convolution kernel, which requires less computational complexity and reduces the parameter amount.

[0016] In a more specific technical solution, in step S1, the bidirectional separation attention module TSA is placed in each dense connection block in the encoder, and the bidirectional attention lightweight lane detection and classification network TSANet is used to extract target features and semantic information.

[0017] The application obtains more scale information by using the dense connection method, thereby solving the gradient vanishing problem in the network and preventing overfitting phenomenon.

[0018] In a more specific technical solution, in step S11, the lightweight lane detection and classification network LNet fuses the output resolution of the multi-scale features extracted in the encoder stage in the upsampling process to make up for the information loss.

[0019] The application constructs a lightweight lane detection and classification network (LNet) with a symmetric unit and a dense symmetric block. The application constructs a bidirectional separation attention mechanism (TSA) to construct the dependency relationship between long-distance pixels. The application adds the bidirectional separation attention module (TSA) to the LNet to construct a lightweight lane detection and classification network based on bidirectional attention (TSANet), and enhances the ability of the network to obtain the position of the target area and the dependency relationship between long-distance pixels.

[0020] In a more specific technical solution, step S12 includes:

[0021] S121, fusing processing multi-scale features output by the first encoder DS1 and the seventh encoder DS7;

[0022] S122, fusing processing multi-scale features output by the second encoder DS2 and the sixth encoder DS6;

[0023] S123, fusing processing multi-scale features output by the third encoder DS3 and the fifth encoder DS5.

[0024] The LNet adopted in the application gradually increases the resolution of the output in the upsampling process by fusing the multi-scale features extracted in the encoder stage, and fuses the features of the encoder stage and the features of the decoder stage containing rich global information. In this way, the partial loss of information is made up, and the low-level features are fused into the decoder, which can make up for the lost information as the network depth increases, so that the final result has both global information and retains local detail information.

[0025] The application only considers the DS block composed of two and three symmetry units to reduce the rapid increase of the number of channels caused by dense connection, to improve the segmentation accuracy in the specific network depth scenario, and to ensure the real-time performance of the network.

[0026] In a more specific technical solution, in step S13, the lightweight lane line detection classification network LNet is trained by using the following weighted cross-entropy loss function:

[0027]

[0028] In the formula, n is the number of pixels, yi is the id of the i-th pixel, wherein id represents a feature lane line or background, and pi represents the class prediction probability of the i-th pixel.

[0029] In a more specific technical solution, the weight parameter is constrained to an applicable area by using a regularization parameter L2 to reduce the probability of overfitting.

[0030] The application uses a weighted cross-entropy loss function to train the network, wherein the regularization L2 can constrain the weight parameter to a smaller area, and can reduce the probability of overfitting.

[0031] In a more specific technical solution, the bidirectional separation attention module TSA in step S22 is an independent module.

[0032] The TSA used in the application is an independent module, which can be placed at any position on the network, improving the system applicability.

[0033] In a more specific technical solution, step S22 includes:

[0034] S221, respectively, maximum pooling and average pooling operations are performed on the input feature map from the horizontal and vertical directions by using the following logic to obtain the horizontal feature map and the vertical feature map;

[0035]

[0036]

[0037] In the formula, MP and AP represent maximum and average pooling, respectively;

[0038] S222, the horizontal feature map and the vertical feature map are transmitted to the sharing module, and the horizontal shared feature and the vertical shared feature are obtained by processing; the horizontal shared feature and the vertical shared feature are processed by using the Sigmoid activation function and the following logic, so as to obtain the horizontal activation feature H n and the vertical activation feature W n :

[0039] H n '=f(Θ(H n ))

[0040] W n '=f(Θ(W n ))

[0041] In the formula, f is the Sigmoid activation function, and Θ represents the sharing module;

[0042] S223, the horizontal activation feature and the vertical activation feature are multiplied by using the following logic, so as to obtain the attention feature and construct the long-distance dependence relationship between pixels:

[0043] A n =F n ×H n '×W n '.

[0044] The TSA adopted by the application can effectively integrate the texture and position information of the target from the horizontal and vertical pixels respectively without introducing a large amount of overhead, thereby improving the segmentation efficiency. The TSA of the application can respectively encode the image pixels in the horizontal and vertical directions, so as to effectively integrate the spatial coordinate information into the generated attention feature. Therefore, the TSA can effectively integrate the coordinate information into the generated attention feature, and use the spatial positions between pixels to construct the long-distance dependence relationship.

[0045] In a more specific technical solution, the sharing module in step S22 includes: 2 convolution layers.

[0046] In a more specific technical solution, the lane line detection and classification system based on bidirectional separated attention includes:

[0047] The lightweight lane line detection classification module is configured to construct a lightweight lane line detection classification network LNet, wherein the lightweight lane line detection classification network LNet comprises: symmetric units and dense symmetric blocks, and the dense symmetric blocks are used to densely connect no less than two symmetric units, wherein the lightweight lane line detection classification module comprises:

[0048] The resolution reduction module is configured to perform encoder down-sampling processing on the input image by using the lightweight lane line detection classification network LNet, so as to gradually reduce the resolution of the input image, and obtain a small-resolution feature map;

[0049] The feature fusion module is configured to perform decoder up-sampling processing on the last layer of the convolution image in the encoder, so as to restore the small-resolution feature map to the original size of the input image, wherein the output features of the decoder and the encoder are fused to obtain fused features, and the feature fusion module is connected with the resolution reduction module;

[0050] The network training module is configured to train the lightweight lane line detection classification network LNet by using a weighted cross-entropy loss function;

[0051] The position detail information module is configured to construct a bidirectional separated attention mechanism, so as to construct a long-distance pixel dependency relationship and obtain global position information and local detail information, and the position detail information module is connected with the lightweight lane line detection classification module, and the position detail information module comprises:

[0052] The bidirectional attention network construction module is configured to add the bidirectional separated attention module TSA to the lightweight lane line detection classification network LNet, so as to construct a bidirectional attention lightweight lane line detection classification network TSANet;

[0053] The dependency relationship acquisition module is configured to use the bidirectional separated attention module TSA to respectively integrate target texture and target position information from horizontal pixels and vertical pixels, and use the bidirectional separated attention module TSA to process the input feature map according to the fused features, so as to obtain an applicable mapping relationship as a long-distance pixel dependency relationship, and the dependency relationship acquisition module is connected with the bidirectional attention network construction module.

[0054] Compared with the prior art, the present application has the following advantages: the present application is based on the idea of semantic segmentation, and the encoder-decoder model is redesigned to generate a lightweight network (LNet) with fewer parameters. Secondly, in order to make the model robust to challenging environments, a bidirectional separable attention (TSA) is introduced to build the dependency relationship between long-distance pixels. Finally, the TSA is integrated into the LNet to obtain more accurate lane line detection and classification results. The TSANet used in the present application improves the lane line detection accuracy on TuSimple and other datasets, optimizes the image segmentation effect, and is suitable for real-time lane line detection and classification in intelligent driving scenarios.

[0055] The present application proposes a symmetric unit with a symmetric structure, compared with the subunit of the LLNet used in the prior art, the symmetric unit uses one-dimensional convolution with a smaller convolution kernel, only needs less calculation amount, and also reduces the parameter amount.

[0056] The present application acquires more scale information by adopting a dense connection mode, thereby solving the gradient vanishing problem in the network and preventing the occurrence of overfitting phenomenon.

[0057] The present application constructs a lightweight lane line detection and classification network (LNet) with a symmetric unit and a dense symmetric block. The present application constructs a bidirectional separable attention mechanism (TSA) to build the dependency relationship between long-distance pixels. The present application adds the bidirectional separable attention module (TSA) to the LNet to construct a lightweight lane line detection and classification network based on bidirectional attention (TSANet), and enhances the ability of the network to obtain the position of the target region and the dependency relationship between long-distance pixels.

[0058] The LNet used in the present application gradually increases the resolution of the output in the upsampling process by fusing the multi-scale features extracted in the encoder stage, and fuses the features of the encoder stage containing rich global information and the features of the decoder stage. In this way, the partial loss of information is made up, and the low-level features can be fused into the decoder to make up for the lost information as the network depth increases, so that the final result has both global information and retains local detail information.

[0059] The present application only considers the DS block composed of two and three symmetric units to reduce the rapid increase of the channel number caused by dense connection, to improve the segmentation accuracy in the specific network depth scenario, and to ensure the real-time performance of the network.

[0060] The present application uses a weighted cross-entropy loss function to train the network, wherein the regularization L2 can constrain the weight parameters to a smaller area, and can reduce the probability of overfitting.

[0061] The TSA used in the application is an independent module, which can be placed at any position on the network, improving the system applicability.

[0062] The TSA used in the application can effectively integrate the texture and position information of the target from the horizontal and vertical pixels respectively without introducing a large amount of overhead, improving the segmentation efficiency. The TSA of the application can respectively encode the image pixels in the horizontal and vertical directions to effectively integrate the spatial coordinate information into the generated attention features. Therefore, the TSA can effectively integrate the coordinate information into the generated attention features and utilize the spatial positions between the pixels to construct long-distance dependency relationships. The application solves the technical problems of large parameter quantity and poor segmentation performance in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 The figure is a schematic diagram of the overall network structure of the TSA Net in the lane line detection and classification method based on bidirectional separated attention of embodiment 1 of the application;

[0064] Figure 2 The figure is a schematic diagram of the composition structure of the symmetric unit of embodiment 1 of the application

[0065] Figure 3a The figure is a schematic diagram of the first composition structure of the dense symmetric block of embodiment 1 of the application;

[0066] Figure 3b The figure is a schematic diagram of the second composition structure of the dense symmetric block of embodiment 1 of the application;

[0067] Figure 4 The figure is a schematic diagram of the composition structure of the dimension reduction layer of embodiment 1 of the application;

[0068] Figure 5 The figure is a schematic diagram of the architecture of the LNet of embodiment 1 of the application;

[0069] Figure 6 The figure is a schematic diagram of the structure of the bidirectional separated attention of embodiment 1 of the application;

[0070] Figure 7 The figure is a visual feature map under different combination modes of the dense connection block of embodiment 2 of the application;

[0071] Figure 8 The figure is a schematic diagram of the detection result of the lane line detection network in the TuSimple data set of embodiment 3 of the application;

[0072] Figure 9 The figure is a schematic diagram of the detection result of the lane line detection network in the self-built data set of embodiment 3 of the application. DETAILED DESCRIPTION

[0073] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] Example 1

[0075] like Figure 1 As shown, this invention constructs a lightweight lane detection and classification network (LNet) with symmetric units and dense symmetric blocks. Secondly, to construct dependencies between long-range pixels, a bidirectional split attention mechanism (TSA) is built. To enhance the network's ability to acquire the location of the target region and the dependencies between long-range pixels, a bidirectional split attention module (TSA) is added to the LNet, constructing a lightweight lane detection and classification network based on bidirectional attention (TSANet). TSA is a standalone module that can be placed anywhere on the network. To enhance the network's ability to extract target features and semantic information, in the network constructed in this invention, TSA is placed after each densely connected block in the encoder.

[0076] 1. Symmetrical elements and densely symmetric blocks:

[0077] like Figure 2 As shown, this invention proposes a symmetric unit with a symmetric structure. Compared to the sub-units of LLNet, the symmetric unit uses one-dimensional convolution with a smaller kernel, requiring less computation and reducing the number of parameters. Figure 2 The composition of the symmetric unit is given, which consists of 3×1 and 1×3 convolutional layers.

[0078] like Figure 3a and Figure 3b As shown, dense connections can acquire information at more scales, thereby solving the gradient vanishing problem in the network and preventing overfitting. Therefore, this invention constructs a dense symmetric block (DS). This DS block connects multiple symmetric units through dense connections to enhance information transmission at each layer. It is not true that deeper networks necessarily lead to higher segmentation accuracy, and real-time performance must be guaranteed. Therefore, this invention only considers DS blocks composed of two and three symmetric units to reduce the rapid increase in the number of channels caused by dense connections, where ∪ connects the feature maps.

[0079] like Figure 4As shown, since the DS block adopts dense connection, it will greatly increase the dimension of the feature map, thereby increasing the amount of calculation. Therefore, the present application adopts a dimension reduction layer to reduce the dimension of the feature map after the DS block. In order to reduce the information loss caused by Pooling, the dimension reduction layer does not adopt Pooling, but only adopts a 1x1 convolution layer, and the dimension reduction factor is set to 0.5. Assuming that the input ratio is h x w x c, the dimension of the channel after dimension reduction is h x w x (c x 0.5).

[0080] 2. Architecture of LNet

[0081] As shown in Figure 5 , this paper constructs LNet based on the symmetric unit and dense symmetric block designed above.

[0082] Firstly, the encoder of LNet reduces the resolution of the input image by using down-sampling operation, and reduces the amount of calculation of the network by obtaining a small resolution feature map; the decoder restores the small resolution feature map to the original input image size by using up-sampling operation on the last convolution image in the encoder. However, the resolution of the feature map of the last convolution layer in the encoder is particularly small, and the corresponding detailed information contained is very little. If the up-sampling is directly used to restore the resolution of the feature map to the original input image, a lot of information will be lost, resulting in low segmentation accuracy.

[0083] In order to make up for the partial loss of information, LNet fuses the multi-scale features extracted in the encoder stage during the up-sampling process to gradually increase the resolution of the output, and uses a black dashed line in Figure 4 to represent this operation. The features output by the first DS1 and the features output by the DS7 are fused, the features output by the first DS2 and the features output by the DS6 are fused, and the features output by the first DS3 and the features output by the DS5 are fused. Thus, the present application fuses the features of the encoder stage containing rich global information and the features of the decoder stage. In this way, the low-level features can be fused into the decoder to make up for the information lost with the increase of the network layers, so that the final result has both global information and retains local detailed information.

[0084] This paper uses a weighted cross-entropy loss function to train the network, and the calculation formula of the loss function is shown in formula (1). Regularization L2 can constrain the weight parameters to a smaller area to reduce the probability of overfitting. In formula (1), n is the number of pixels, yi is the id (feature lane line or background) of the ith pixel, and pi represents the class prediction probability of the ith pixel.

[0085]

[0086] 3. Bidirectional separation attention

[0087] To obtain the global position information and local detail information of the target region, and establish the dependency relationship between long-distance pixels, the present application proposes a bidirectional separated attention (TSA). Wherein, the pixel-level multiplication is represented by , and the Sigmoid function is represented by ∫. TSA can effectively integrate the texture and position information of the target from the horizontal and vertical pixels respectively, without introducing a large amount of overhead.

[0088] As Figure 6 shown, assuming that the feature map of the nth layer is F n ∈h n ×w n ×c n . The input of TSA is F n , and the output is A n ∈h n ×w n ×c n . TSA is to find an optimal mapping relationship Γ:

[0089]

[0090] First, for the input feature map F n , Max-pooling and Avg-pooling operations are respectively performed from the horizontal and vertical directions to obtain the horizontal feature map and the vertical feature map , as shown in formula (3) and formula (4). Wherein, MP and AP represent Max-pooling and Avg-pooling respectively.

[0091]

[0092]

[0093] Then, H n and W n are sent to a shared module composed of two convolutional layers, wherein K h =[1,w], K w =[h,1]. As shown in formula (5) and formula (6), the feature outputs of the shared module in two directions are further transmitted to the Sigmoid activation function to generate H n ' and W n ' respectively. In addition, in order to reduce the channel number of the network, TSA also adds a 1×1 convolutional layer with a dimension reduction coefficient r.

[0094] H n '=f(Θ(H n )) (5)

[0095] Wn '=f(Θ(W n (6)

[0096] Where f is the Sigmoid activation function, and Θ represents the shared module.

[0097] Finally, the feature map (H) n 'and W n ') and input feature map (F n Multiply to obtain attention features (A) n As shown in formula (7).

[0098] A n =F n ×H n '×W n (7)

[0099] TSA can encode image pixels in both the horizontal and vertical directions to effectively integrate spatial coordinate information into the generated attention features. Therefore, TSA can effectively integrate coordinate information into the generated attention features and utilize the spatial location between pixels to construct long-range dependencies.

[0100] Example 2

[0101] like Figure 7 As shown, ablation experiments were conducted on densely symmetric blocks. In this embodiment, LNet was selected as the baseline network to obtain the optimal number of symmetric units constituting the set connection block. To eliminate the influence of TSA, TSA and LNet were not combined. This invention designs five modes: 2222, 2223, 2233, 2333, and 3333, as shown... Figure 7 As shown. Pattern 2222 indicates that the number of symmetric elements in DS1, DS2, DS3, and DS4 are each 2. From Figure 7 As can be seen, when using mode 2223, the lane line features in the feature map are clearer and more detailed than those obtained by other modes. Furthermore, mode 2223 achieves superior detection accuracy (94.43%) as shown in Table 1.

[0102] Table 1. Comparison of results for different combinations of densely connected blocks in the TuSimple dataset.

[0103]

[0104] Example 3

[0105] A TSA ablation experiment was conducted. In this embodiment, a TSA was applied after each DS block in the encoder. The feasibility and rationality of this design were demonstrated by analyzing the performance of adding TSA at different locations. As shown in Table 2, combining each DS block in the encoder with a TSA resulted in the best performance, with an accuracy of up to 96.53%. This proves that the layout of placing the TSA after each DS block is superior to other layouts.

[0106] Table 2. Comparison of results for TSA at different positions in LNet in the TuSimple dataset.

[0107]

[0108]

[0109] A comparison and analysis of lane detection network performance was conducted. Table 3 shows that TSANet outperforms other lane detection methods. Although TSANet's detection accuracy is comparable to SCNN and SE+FG+BS+Push, TSANet takes less time. Furthermore, SCNN takes 20 times longer than TSANet. While LaneATT, using ResNet18 as its backbone, has a single-frame processing time of 6.06ms, which is 0.963ms lower than TSANet, LaneATT (ResNet18) has 6 times more network parameters than TSANet. Both ESA and TSA can extract features in two directions from the image. Adding ESA and TSA to ERFNet respectively improves the accuracy of ERFNet-TSA and ERFNet-ESA by 0.85% and 0.92% compared to the baseline ERFNet. This demonstrates that extracting features in both directions from the image is beneficial for detection results. Therefore, TSANet can simultaneously extract local texture and global location information while having fewer parameters, achieving competitive performance.

[0110] Table 3. Comparison and analysis of lane detection network performance on the TuSimple dataset.

[0111]

[0112] like Figure 8 As shown, Figure 8 The results of lane line detection in different scenarios on the TuSimple dataset are presented. Figure 8 As can be seen, TSANet exhibits good robustness in scenarios involving ground obstacles and occlusions. This is because TSANet can obtain local and positional information in two directions, which to some extent avoids false positives and false negatives, thus contributing to better detection results.

[0113] As Figure 9 shown, the performance of TSANet is verified on TuSimple dataset, and the robustness of TSANet is tested on self-built dataset. Eight scenes are selected from the self-built dataset, (a) and (b) represent the scenes without traffic sign interference on the road surface, and the lane line is relatively clear; (c) exists water on the road surface; (d), (e), (f) and (g) exist different traffic sign interference on the road surface; (h) contains winding lane lines. The results show that TSANet can obtain better detection performance by constructing the dependency relationship between long-distance pixels.

[0114] Comparison and analysis of lane line classification network performance

[0115] As can be seen from the results in Table 4, the mIoU of TSANet is 69.7%, the F1-score is 95.3%, and the parameter amount is only 0.21M higher than that of LNet. The outputs of Cascade-CNN, LNet and TSANet are all semantic category result maps, and TSANet has competitive advantages in mIoU, F1-score, real-time performance and parameter amount.

[0116] Table 4. Comparison and analysis of lane line classification network performance in TuSimple dataset

[0117]

[0118] In summary, based on the idea of semantic segmentation, the encoder-decoder model is redesigned to generate a lightweight network (LNet) with fewer parameters. Secondly, in order to make the model robust to challenging environments, bidirectional separated attention (TSA) is introduced to construct the dependency relationship between long-distance pixels. Finally, TSA is integrated into LNet to obtain more accurate lane line detection and classification results. The TSANet adopted by the present application improves the lane line detection accuracy on TuSimple and other datasets, optimizes the image segmentation effect, and is suitable for real-time lane line detection and classification in intelligent driving scenarios.

[0119] The present application proposes a symmetric unit with a symmetric structure, compared with the subunit of the LLNet adopted in the prior art, the symmetric unit adopts one-dimensional convolution with smaller convolution kernel, only needs less calculation amount, and also reduces the parameter amount.

[0120] The present application acquires more scale information by adopting the dense connection mode, so as to solve the gradient disappearance problem in the network and prevent the occurrence of overfitting phenomenon.

[0121] The application constructs a lightweight lane line detection and classification network (LNet) with symmetric units and dense symmetric blocks. A bidirectional separation attention mechanism (TSA) is constructed to build the dependency relationship between long-distance pixels. The bidirectional separation attention module (TSA) is added to the LNet to construct a lightweight lane line detection and classification network based on bidirectional attention (TSA Net), which enhances the ability of the network to obtain the location of the target area and the dependency relationship between long-distance pixels.

[0122] The LNet adopted in the application gradually increases the resolution of the output in the upsampling process by fusing the multi-scale features extracted in the encoder stage, and fuses the features of the encoder stage containing rich global information and the features of the decoder stage. In this way, the partial loss of information is made up, and the low-level features can be fused into the decoder to make up for the lost information as the network depth increases, so that the final result has both global information and retains local detail information.

[0123] The application only considers the DS blocks composed of two and three symmetric units to reduce the rapid increase of the number of channels caused by dense connection, to improve the segmentation accuracy in the specific network depth scenario, and to ensure the real-time performance of the network.

[0124] The application uses a weighted cross-entropy loss function to train the network, wherein the regularization L2 can constrain the weight parameters to a smaller area, and can reduce the probability of overfitting.

[0125] The TSA used in the application is an independent module that can be placed at any position on the network, improving the system applicability.

[0126] The TSA used in the application can effectively integrate the texture and position information of the target from the horizontal and vertical pixels respectively without introducing a large amount of overhead, improving the segmentation efficiency. The TSA of the application can respectively encode the image pixels in the horizontal and vertical directions to effectively integrate the spatial coordinate information into the generated attention features. Therefore, the TSA can effectively integrate the coordinate information into the generated attention features, and use the spatial positions between the pixels to build long-distance dependency relationships. The application solves the technical problems of large number of parameters and poor segmentation performance in the prior art.

[0127] The above embodiments are only used to illustrate the technical solutions of the application, but not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A lane line detection and classification method based on bidirectional separation attention, characterized in that, The method includes the following steps: S1. Construct a lightweight lane detection and classification network LNet, wherein the lightweight lane detection and classification network LNet includes: symmetric units and dense symmetric blocks, and dense symmetric blocks are used to densely connect no less than 2 symmetric units. Step S1 includes: the symmetric units are composed of 3×1 and 1×3 convolutional layers. S11. The encoder of the lightweight lane detection and classification network LNet is used to downsample the input image to gradually reduce the resolution of the input image in order to obtain a low-resolution feature map. S12. The last convolutional image in the encoder is processed on the decoder to restore the small-resolution feature map to the original size of the input image. The output features of the decoder and encoder are fused to obtain the fused features. S13. Train a lightweight lane detection and classification network LNet using the weighted cross-entropy loss function; S2. Construct a bidirectional separation attention mechanism to build dependencies between long-distance pixels and obtain global position information and local detail information. Step S2 includes: S21. Add the bidirectional separation attention module TSA to the lightweight lane detection and classification network LNet to construct the bidirectional attention lightweight lane detection and classification network TSANet. S22. Using the bidirectional split attention module TSA, target texture and target position information are integrated from horizontal and vertical pixels respectively. The fused features are used as input feature maps. The bidirectional split attention module TSA processes the input feature maps to obtain applicable mapping relationships as dependencies between long-distance pixels. The shared module in step S22 includes: 2 convolutional layers. Step S22 includes: S221. Using the following logic, perform max pooling and average pooling operations on the input feature map from the horizontal and vertical directions respectively to obtain the horizontal feature map and the vertical feature map. In the formula, , These represent max pooling and average pooling, respectively. , ; S222. The horizontal and vertical feature maps are sent to the sharing module for processing to obtain horizontal and vertical shared features. Using the Sigmoid activation function, the horizontal and vertical shared features are processed according to the following logic to obtain the horizontal activation features. and longitudinal activation features : In the formula, It is the Sigmoid activation function. Indicates a shared module; S223. Multiply the horizontal activation features and vertical activation features using the following logic to obtain attention features, thereby constructing long-distance dependencies between pixels: 。 2. The lane line detection and classification method based on bidirectional separation attention as described in claim 1, characterized in that, In step S1, the bidirectional separation attention module TSA is placed into each dense connection block in the encoder, so that the bidirectional attention lightweight lane detection and classification network TSANet can extract target features and semantic information.

3. The lane line detection and classification method based on bidirectional separation attention as described in claim 1, characterized in that, In step S11, the lightweight lane detection classification network LNet fuses the output resolution of multi-scale features extracted during the encoder stage during the upsampling process to compensate for information loss.

4. The lane line detection and classification method based on bidirectional separation attention as described in claim 1, characterized in that, Step S12 includes: S121, Fuse and process the multi-scale features output from the first encoder DS1 and the seventh encoder DS7; S122, Fuse and process the multi-scale features output from the second encoder DS2 and the sixth encoder DS6; S123, fuses the multi-scale features output from the third encoder DS3 and the fifth encoder DS5.

5. The lane line detection and classification method based on bidirectional separation attention according to claim 1, characterized in that, In step S13, the lightweight lane detection classification network LNet is trained using the following weighted cross-entropy loss function: In the formula, n is the number of pixels, and y i Let p be the id of the i-th pixel, where id represents the feature lane line or background. i Describe the predicted class probability of the i-th pixel; This represents the regularization parameter.

6. The lane line detection and classification method based on bidirectional separation attention as described in claim 5, characterized in that, Using regularization parameters The weight parameters are constrained to the applicable region to reduce the probability of overfitting.

7. The lane line detection and classification method based on bidirectional separation attention according to claim 1, characterized in that, The bidirectional separate attention module TSA in step S22 is an independent module.

8. A lane detection and classification system based on bidirectional separate attention, used to execute the lane detection and classification method based on bidirectional separate attention as described in any one of claims 1 to 7, characterized in that, The system includes: A lightweight lane line detection and classification module is provided to construct a lightweight lane line detection and classification network (LNet). The LNet comprises symmetric units and dense symmetric blocks, wherein at least two symmetric units are densely connected using the dense symmetric blocks. The lightweight lane line detection and classification module further includes: The resolution reduction module is used to downsample the input image with the encoder of the lightweight lane detection classification network LNet to progressively reduce the resolution of the input image in order to obtain a low-resolution feature map. A feature fusion module is used to process the last convolutional image in the encoder on the decoder to restore the low-resolution feature map to the original size of the input image, wherein the output features of the decoder and the encoder are fused to obtain fused features, and the feature fusion module is connected to the resolution reduction module; The network training module is used to train the lightweight lane detection and classification network LNet using the weighted cross-entropy loss function; A location detail information module is used to construct a bidirectional separation attention mechanism to build dependencies between long-distance pixels and obtain global location information and local detail information. This location detail information module is connected to the lightweight lane line detection and classification module. The location detail information module includes: A bidirectional attention network construction module is used to add the bidirectional separate attention module TSA to the lightweight lane detection and classification network LNet, thereby constructing the bidirectional attention lightweight lane detection and classification network TSANet; The dependency acquisition module is used to integrate target texture and target position information from horizontal and vertical pixels using the bidirectional separation attention module TSA, and use the fused features as input feature maps. Based on the input feature maps, the bidirectional separation attention module TSA processes them to obtain applicable mapping relationships as the dependencies between long-distance pixels. The dependency acquisition module is connected to the bidirectional attention network construction module.

Images