Lightweight lane line detection method based on improved MobileNet network

Abstract

The invention discloses a lightweight lane line detection method based on an improved MobileNet network, and the method comprises the following steps: (1) receiving a video stream signal through a camera, intercepting the video stream signal into an image, and carrying out the preprocessing of the image; (2) constructing an improved MobileNet-based lightweight lane line detection model, and carrying out training by using the preprocessed data set; the MobileNet lightweight lane line detection model comprises a backbone network EMA-MNet, a network structure of a neck is established by adopting BiFPN, and a detection head of CLRNet is adopted; (3) outputting a detection result; according to the invention, efficient detection of the lane line is realized with extremely small parameter quantity and extremely low model complexity.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a lightweight lane line detection method based on an improved MobileNet network. Background Art

[0002] At present, similar technologies for lane line detection are anchor-based methods, such as LineCNN and LaneATT, which preset a large number of linear anchors, regress the offset between the sampling point and the predefined anchor, and use NMS to select the lane line with the highest confidence. On this basis, CLRNet reduces the number of preset linear anchors and proposes learnable anchors to further improve the accuracy of detection. However, in real environments with multiple lanes, high traffic density or intersections, lane lines may be staggered or blocked, and lane line detection may also be affected by weather factors such as rainy days, strong or dark light, making detection difficult. The current lane line detection algorithm has poor detection effect in the presence of occlusion and bad weather. Most of the backbone networks of current technologies use ResNet series and DLA series networks to extract features. Although they have high accuracy, they usually have deeper network layers and more parameters, resulting in large computational complexity and are not suitable for mobile devices or edge devices. In particular, the real-time performance will be greatly reduced on resource-constrained devices. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a lightweight lane line detection method based on an improved MobileNet network, which can effectively perform lane line detection under high traffic density, poor lighting conditions, and other adverse conditions where lane lines may be blurred.

[0004] Technical solution: The lightweight lane line detection method based on the improved MobileNet network described in the present invention comprises the following steps:

[0005] (1) Receive video stream signals through a camera, capture the video stream signals into images and perform preprocessing;

[0006] (2) Build an improved MobileNet lightweight lane detection model based on , and use the preprocessed dataset for training; the MobileNet lightweight lane detection model includes: a backbone network EMA-MNet, a network structure using BiFPN to build the neck, and a detection head using CLRNet;

[0007] (3) Output the test results.

[0008] Furthermore, step (1) is as follows: use python+opencv to write a program to extract video frames, use OpenCV (cv2) library to process videos, open video files through cv2.VideoCapture(video_path), read the video frame by frame through video.read(), and save the read frames as image files. The preprocessing steps include: randomly cropping images, randomly selecting a cropping area, cropping the area from each input image, and enhancing the randomness of the data. Randomly rotating images to simulate rotation transformations and increase the model's insensitivity to rotation. Randomly adding Gaussian blur to some images in the image group to simulate image blur scenarios. Randomly flipping images horizontally, applying flipping with a probability of 50%, and enhancing the model's insensitivity to horizontal flipping.

[0009] Furthermore, in step (2), the backbone network EMA-MNet is specifically as follows: bilinear interpolation is added before layer0 in the MobileNetV4-smal model to crop and scale the image so that the input image size is changed from 1640*590 to 800*800, and the last fully connected layer in the network model is removed, and the EMA attention mechanism is added to the output positions of layer1 and layer2 as well as layer4 and layer5 to construct a residual structure.

[0010] Furthermore, in step (2), the network structure of the neck is constructed using BiFPN as follows: the input of BiFPN is changed from 5 feature inputs to 3 feature inputs, the number of convolution kernels of the 3 horizontal layers is set to 64, and the number of channels of the input feature map is unified to facilitate subsequent calculations of the model.

[0011] Furthermore, in step (2), the detection head of CLRNet is used as follows: ROIGather in the original detection head is modified, and the InceptionNext structure composed of LKD large kernel decomposition convolution is introduced, and the large kernel convolution is decomposed into multiple parallel branches, including 1 / 3 of the channels with 3×3 kernel, 1 / 3 of the channels with 1×11 kernel, 1 / 3 of the channels with 11×1 kernel and an identity mapping.

[0012] Furthermore, in step (2), LineIoU is used as a new regression loss function to measure the accuracy of the predicted sampling points in lane line detection; Focal Loss is used as a classification loss function to focus on the attribution of different pixels and determine whether they belong to the lane line; L1 loss is used when regressing the coordinates of the anchor line starting point, the angle of the anchor line, and the length of the anchor line; the LineIoU loss function draws on the calculation idea of ​​IoU in target detection.

[0013] Furthermore, LineIoU is used as a new regression loss function to measure the accuracy of the predicted sampling points in lane line detection. The details are as follows: define the true value point G and the predicted point P of the lane line, and then use -e, +e to offset the horizontal coordinates x of the true value point G and the predicted point P to the left and right, and form two line segments with point G and point P as the midpoints respectively. The ratio of the intersection and union of the two line segments is used to measure the overlap degree of the two points.

[0014] The lightweight lane detection system based on the improved MobileNet network described in the present invention includes:

[0015] Preprocessing module: used to receive video stream signals through the camera, intercept the video stream signals into images and perform preprocessing;

[0016] Lane detection model module: used to build and train the improved MobileNet lightweight lane detection model; the MobileNet lightweight lane detection model includes: the backbone network EMA-MNet, the network structure using BiFPN to build the neck, and the detection head using CLRNet;

[0017] Detection module: outputs detection results.

[0018] An electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, it implements any one of the lightweight lane line detection methods based on the improved MobileNet network.

[0019] A storage medium described in the present invention stores a computer program, and when the computer program is executed by a processor, it implements any one of the lightweight lane line detection methods based on the improved MobileNet network.

[0020] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: using the improved EMA-MNet network structure, BiFPN multi-scale fusion layer, LKROIGather, etc., to enhance the feature extraction capability, expand the receptive field of the model and improve the ability to capture long-distance dependencies, reduce the interference of complex environments on lane line detection, reduce the complexity of the model, and improve real-time performance. The method of the present invention achieves efficient detection of lane lines with extremely small parameters and extremely low model complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the present invention;

[0022] Figure 2 It is the overall structure diagram of the network model of the present invention;

[0023] Figure 3 It is a structural diagram of the EMA-MNet backbone network of the present invention;

[0024] Figure 4 is a structural diagram of the neck network of the present invention;

[0025] Figure 5 It is a structural diagram of LKROIGather of the present invention;

[0026] Figure 6 This is the LKD large kernel decomposition convolution structure diagram of the present invention;

[0027] Figure 7 Schematic diagram for LineIoU loss function calculation;

[0028] Figure 8 This is a diagram showing the detection effect of the present invention under traffic congestion and dark light conditions. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0030] The embodiment of the present invention provides a lightweight lane line detection method based on an improved MobileNet network, comprising the following steps:

[0031] S1 obtains the public datasets CULane and TuSimple and performs preprocessing; the preprocessing is as follows: The preprocessing steps include: randomly cropping the image, randomly selecting a cropping area, and cropping the area from each input image to enhance the randomness of the data. Randomly rotating the image to simulate the rotation transformation and increase the model's insensitivity to rotation. Randomly adding Gaussian blur to some images in the image group to simulate the image blur scene. Randomly flipping the image horizontally, applying flipping with a probability of 50%, and enhancing the model's insensitivity to horizontal flipping.

[0032] S2 builds a lightweight lane detection network based on the improved MobileNet network, such as Figure 2 As shown in FIG, it includes: an EMA-MNet backbone network, a neck module, and a head module; among them, the EMA-MNet backbone network is improved on the basis of the MobileNetV4-small network structure; as shown in FIG. Figure 3As shown in the figure, the EMA attention mechanism is used to construct a branch residual structure at the positions of layer1, layer2, layer4, and layer5 to better integrate features at different levels, which helps to capture long-term dependencies and enhance the memory capacity of the model. In addition, the inverted residual structure of the MobileNet series network will quickly increase and decrease the dimension of the feature map, which may lead to the loss of feature information. By constructing a residual structure, the risk of information loss in deeper networks can be reduced. In the neck part, the BIFPN module is used to fuse the feature maps of the three levels output by the backbone network. Its special bidirectional feature fusion path ensures that feature information of different scales can interact and complement each other more fully, which helps to capture richer multi-scale information. In the head part, the detection head of the CLRNet network is used for lane line detection. Its internal ROIGather can effectively collect global context with rich semantic information and capture the features of the foreground lane even under occlusion. However, the lane line is long and continuous, and the model needs a certain long-distance modeling capability. In addition, a large receptive field is very effective in improving the detection accuracy of the model. By introducing the InceptionNext structure composed of LKD large-core decomposable convolution, a new LKROIGather module is constructed. LKD decomposes the large-core convolution into multiple parallel branches, reducing the high computational cost brought by the large-core convolution, so as to expand the receptive field of the model and improve the ability to capture long-distance dependencies. In terms of loss function, lineIoU is used as the regression loss function, abandoning the traditional use of L1 loss function to measure the distance between the predicted point and the true value point, and using the union intersection to regress the lane prior as a whole unit, which improves the accuracy of regression. The specific process is as follows:

[0033] The S21EMA-MNet backbone network includes 6 convolutional layer structures from layer0 to layer5, and two branch residual structures containing the EMA attention mechanism. Figure 3As shown in the figure, the input image size is 1640*590, and the number of channels is 3. After the bilinear interpolation operation, the number of image channels remains unchanged, and the size becomes 800*800. The processed image is sent to layer0~layer5 for feature extraction. The branch output of layer2 passes through a 1×1 convolution layer, and the number of channels is adjusted from 64 to 32. At this time, the feature map size is 32*100*100. The new feature map passes through the EMA attention mechanism to obtain a feature map of size 32*100*100. Then, the feature map after the attention mechanism is added element by element to the branch output of layer1 to obtain the feature map L2. The EMA attention mechanism provides a cross-spatial information aggregation method in different spatial dimensions to enrich feature aggregation, establishes local cross-channel interactions in the network without channel dimensionality reduction, and avoids the side effects of modeling cross-channel relationships through channel dimensionality reduction on extracting deep visual representations. Similarly, feature map L0 is the output of layer 4 branch after 1×1 convolution layer and EMA attention mechanism and then added to the output of layer 5, while layer 3 does not have a residual structure, feature map L1 is the output of layer 3 branch. The three output feature maps of the backbone network, namely L0, L1, and L2, will be provided to the neck part for multi-scale feature fusion. The sizes of the three are expressed by the following formulas:

[0034] L0∈R 512*25*25 ,L1∈R 96*50*50 L2∈R 32*100*100 (1)

[0035] S22 Figure 4 As shown in the figure, the neck part uses the BIFPN module to fuse the feature maps L0, L1, and L2. The structure selected by the red dotted line in the figure is the flow direction of the data. BIFPN*3 means that the structure selected by the red dotted line needs to be repeated 3 times, which can be regarded as 3 BIFPN structures connected in series. It can be seen from the figure that the three input feature maps are adjusted to 64 channels through 1×1 convolution, and the obtained right The upsampling operation is performed to double the size of the feature map and then Perform weighted fusion to obtain The calculation formula is as follows:

[0036]

[0037] Among them, ω0 and ω1 are the learned weights, Upsample means upsampling the feature map, ε is a number that tends to 0 but is not 0 to prevent the denominator from being 0, and Conv means convolution operation. Perform an upsampling operation to double the size of the feature map and then Perform weighted fusion to obtain The calculation formula is as follows:

[0038]

[0039] Where ω′1 represents The weight of the feature map, and then The downsampling operation is performed to reduce the feature map size to half of the original size, and then Perform weighted fusion to obtain The calculation formula is as follows:

[0040]

[0041] Where Downsample means downsampling the feature map, ω′2 represents the feature map Then the feature map The downsampling operation is performed to half the original size, and then Perform weighted fusion to obtain The calculation formula is as follows:

[0042]

[0043] Where ω″1 represents The weight value of . So far, the multi-scale feature fusion of L0, L1, and L2 feature maps has been completed, and a new feature map has been obtained. Its size and dimension are expressed by the following formula:

[0044]

[0045] Since the entire neck part is composed of three BIFPN modules in series, the The above process will be repeated as the input of the second BIFPN module until the third BIFPN module outputs the final new feature map L0, L1, L2, whose size and dimension are expressed by the following formula:

[0046] L0∈R 64*25 * 25 L1∈R 64*50*50 L2∈R 64*100*100 (7)

[0047] At this point, the feature map output by the backbone network has undergone three rounds of multi-scale weighted fusion to obtain a new feature map, which will be input into the head detection head.

[0048] The feature map processed by the neck layer in S23 enters the detection head and enters LKROIGather together with the lane line prior information for layer-by-layer refinement. The specific process is as follows: Figure 2 In the head part, the high-level feature L0 and P0 are firstly put into LKROIGather for refinement operation to obtain the first-layer prediction result P′0. P′0 and P0 are concatenated to obtain the lane line prior information P1 of the next layer. Then the feature map L1 and P1 of the next layer are subjected to the same operation to obtain the lane line prior information P2 and the prediction result P′1 of the next layer. Then the feature map L2 and P2 are repeated to obtain the lane line prediction result P′2 of the last layer. t , t∈(0,1,2) represents the prediction result, Pt,t∈(0,1,2) represents the prior information of the lane line, including the coordinates of the starting point x, y and the angle θ of the preset anchor line, and the layer-by-layer refinement calculation formula is:

[0049]

[0050] The operation represented by Rt in the formula is as follows Figure 5 As shown in the figure, LKROIGather takes feature maps and anchor lines as input. Each anchor has 72 sampling points (obtained by sampling at equal intervals in the horizontal direction). First, the roi of the anchor is extracted, and 36 points are sampled from the 72 sampling points on the anchor. The obtained roi passes through a convolution layer and then a fully connected layer to obtain new roi features. At the same time, the feature map of the current level is resized to 64*10*25 through a resize operation, and then the receptive field of the feature map is expanded through LKD large kernel decomposition convolution. The network structure diagram of LKD large kernel decomposition convolution is shown in the figure Figure 6 As shown, the calculation formula is as follows:

[0051] X out =shortcut+MLP(Norm(InceptionDWConv2d(X)) (9)

[0052] Among them, shortcut represents the output of the residual structure, MLP represents the multi-layer perceptron, and Norm represents the normalization layer. InceptionDWConv2d means splitting the feature map according to the number of channels, performing convolutions separately, and then splicing to obtain the final result. The calculation formula is as follows:

[0053] X out =Concat(X id ,DWConv1(X1),DWConv2(X2),DWConv3(X3)) (10)

[0054] Concat means concatenating the feature maps according to the number of channels. Figure 6 The split operation in the input feature map X is divided into X id , X1, X2, X3. Then the feature map output after LKD is flattened to change the feature map from 64*10*25 to 64*250, and then the roi and feature map are used to perform attenion operation to calculate the attention matrix, that is Figure 5 The position marked by w is calculated as follows:

[0055]

[0056] in the formula Indicates transposing the processed roi. represents the square root of the number of channels, X f Represents the feature map after processing. Then multiply the obtained attention matrix and feature map to get the global feature map. Figure 5 The position of F in the equation is calculated as follows:

[0057]

[0058] Where F represents the global feature map, Indicates the feature map transposition, and finally the global feature map and the new roi are added to get the final roi, i.e. P′ t , t∈(0,1,2), such as Figure 5 The output parameters shown in the figure represent the following meanings: Indicates the coordinates of the starting point of the anchor line. Indicates the angle of the anchor line, l i Indicates the length of the anchor line, Indicates the offset of the horizontal sampling point of the anchor line. Represents the probability of lane lines and background, i∈(0, 1, 2).

[0059] S3 obtains three prediction results P′ after step S2. t , t∈(0, 1, 2), the loss of these results is calculated uniformly. Line iou is used for regression when regressing the offset of the horizontal sampling point of the predicted lane line. The calculation process is as follows Figure 7As shown in the figure, the blue line represents the real lane line, and the yellow line represents the predicted result. Define the true value point g and the predicted point p of the lane line, and then use -e and +e to offset the horizontal coordinate x of the true value point g and the predicted point p to the left and right, respectively, and form two line segments with point g and point p as the midpoints. The ratio of the intersection and union of the two line segments is used to measure the overlap of the two points. The calculation formula is as follows:

[0060] in the formula represents the intersection, represents a union, Represents the horizontal coordinate of the real lane line sampling point, Indicates the horizontal coordinate of the predicted lane line sampling point, min() and max() respectively represent the minimum and maximum values ​​of the two. Perform the above operation on each sample and accumulate the results to get the final result. The calculation formula is as follows:

[0062] In the formula, N represents the number of sampling points, which is 72. -1≤Line IoU≤1. When the two lines are far apart, the value is -1, and when the two lines overlap, the value is 1. The loss function is simple and differentiable, and it is easy to implement parallel calculations. At the same time, the lane is predicted as a whole, which helps to improve the overall performance of the model. L1 loss is used when regressing the coordinates of the starting point of the anchor line, the angle of the anchor line, and the length of the anchor line. The classification loss function uses the Focal Loss loss function, and the total loss function is obtained as follows:

[0064] L total =ω cls L cls +ω xytl L xytl +w LIoU L L I oU (15)L cls is the focal loss between prediction and label, L xytl is the L1 loss of the regression of the starting coordinates, angles and lane length, L LIoU is the Line IoU loss between the predicted lane and the ground truth, and ω represents the weight value of the corresponding loss function.

[0065] S4 inputs the processed data set into the created backbone network and uses GPU for training. For the CULane data set, the training rounds are set to 15 rounds, and for the TuSimple data set, the training rounds are set to 70 rounds. The learning rate is set to 0.0003, the batchsize is 8, and the Admw optimizer is used. The learning rate scheduler is the cosine annealing scheduler. The weight file of each round is saved during the training process. After the training is completed, the best weight is saved as best.pth. The comparison results with the indicators of similar algorithms are shown in Table 1 and Table 2:

[0066] Table 1: Test results on the CULane dataset

[0067]

[0068] Table 2: Test results on the TuSimple dataset

[0069]

[0070] In the table, the F1 score is an indicator used to evaluate the balance between precision and recall in a recommendation system or classification system. The larger the F1, the better the model performance. In Table 1, F1@50 indicates the F1 value when the IoU critical value is 0.5. From the two tables, it can be seen that the F1 value of the present invention is not much different from other algorithms, and is even better than LaneATT and ADNet. The FPS value, parameter amount (Params), and model complexity (MACs) values ​​are measured under the conditions of an input image size of 1640*590 and a graphics card of RTX4060. Table 1 shows that the present method achieves real-time and effective lane line detection with very few parameters and very low model complexity. The detection effect diagram is shown in the figure. Figure 8 As shown, the present invention is suitable for deployment on resource-constrained devices and has good real-time performance, and has certain scientific and practical significance.

[0071] The embodiment of the present invention further provides a lightweight lane line detection system based on an improved MobileNet network, comprising:

[0072] Preprocessing module: used to receive video stream signals through the camera, intercept the video stream signals into images and perform preprocessing;

[0073] Lane detection model module: used to build and train the improved MobileNet lightweight lane detection model; the MobileNet lightweight lane detection model includes: the backbone network EMA-MNet, the network structure using BiFPN to build the neck, and the detection head using CLRNet;

[0074] Detection module: outputs detection results.

[0075] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is loaded into the processor, the computer program implements any one of the lightweight lane line detection methods based on the improved MobileNet network.

[0076] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any one of the lightweight lane line detection methods based on an improved MobileNet network.

Claims

1. A lightweight lane detection method based on an improved MobileNet network, characterized in that: The following steps are involved: (1) Receive video stream signals through a camera, capture the video stream signals into images and perform preprocessing; (2) Build an improved MobileNet lightweight lane detection model based on CNN and use the preprocessed dataset for training; The MobileNet lightweight lane detection model includes: the backbone network EMA-MNet, the network structure using BiFPN to build the neck, and the detection head using CLRNet; (3) Output the test results.

2. According to claim 1, a lightweight lane line detection method based on an improved MobileNet network is characterized in that: Step (1) is as follows: use python+opencv to write a program to extract video frames, use OpenCV (cv2) library to process the video, open the video file through cv2.VideoCapture(video_path), read the video frame by frame through video.read(), and save the read frames as image files; The preprocessing steps include: randomly cropping the image, randomly selecting a cropping region, and cropping the region from each input image to enhance the randomness of the data; randomly rotating the image to simulate the rotation transformation and increase the model's insensitivity to rotation; randomly adding Gaussian blur to some images in the image group to simulate the image blur scene; randomly flipping the image horizontally, applying flipping with a probability of 50%, and enhancing the model's insensitivity to horizontal flipping.

3. According to claim 1, a lightweight lane line detection method based on an improved MobileNet network is characterized in that: In step (2), the backbone network EMA-MNet is specifically as follows: in the MobileNetV4-smal model, bilinear interpolation is added before layer0 to crop and scale the image to change the input image size from 1640*590 to 800*800. At the same time, the last fully connected layer in the network model is removed, and the EMA attention mechanism is added to the output positions of layer1 and layer2 as well as layer4 and layer5 to construct a residual structure.

4. According to claim 1, a lightweight lane line detection method based on an improved MobileNet network is characterized in that: In step (2), the network structure of the neck is constructed using BiFPN as follows: the input of BiFPN is changed from 5 feature inputs to 3 feature inputs, the number of convolution kernels of the three horizontal layers is set to 64, and the number of channels of the input feature map is unified to facilitate subsequent calculations of the model.

5. According to claim 1, a lightweight lane line detection method based on an improved MobileNet network is characterized in that: In step (2), the detection head of CLRNet is used as follows: ROIGather in the original detection head is modified, and the InceptionNext structure composed of LKD large kernel decomposition convolution is introduced. The large kernel convolution is decomposed into multiple parallel branches, including 1 / 3 of the channels with 3×3 kernel, 1 / 3 of the channels with 1×11 kernel, 1 / 3 of the channels with 11×1 kernel and an identity mapping.

6. The lightweight lane line detection method based on the improved MobileNet network according to claim 1, characterized in that: In step (2), LineIoU is used as a new regression loss function to measure the accuracy of the predicted sampling points in lane line detection; Focal Loss is used as a classification loss function to focus on the attribution of different pixels and determine whether they belong to the lane line; L1 loss is used when regressing the coordinates of the anchor line starting point, the angle of the anchor line, and the length of the anchor line.

7. The lightweight lane detection method based on the improved MobileNet network according to claim 6, characterized in that: LineIoU is used as a new regression loss function to measure the accuracy of the predicted sampling points in lane line detection. The specific details are as follows: define the true value point G and the predicted point P of the lane line, and then use -e, +e to offset the horizontal coordinates x of the true value point G and the predicted point P to the left and right, and form two line segments with point G and point P as the midpoints respectively. The ratio of the intersection and union of the two line segments is used to measure the overlap degree of the two points.

8. A lightweight lane detection system based on an improved MobileNet network, characterized in that: include: Preprocessing module: used to receive video stream signals through the camera, intercept the video stream signals into images and perform preprocessing; Lane detection model module: used to build and train an improved MobileNet-based lightweight lane detection model; The MobileNet lightweight lane detection model includes: the backbone network EMA-MNet, the network structure using BiFPN to build the neck, and the detection head using CLRNet; Detection module: outputs detection results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into the processor, the lightweight lane line detection method based on the improved MobileNet network according to any one of claims 1 to 7 is implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a lightweight lane line detection method based on an improved MobileNet network is implemented according to any one of claims 1 to 7.

Images