A lane line detection method based on non-site traffic enforcement scenarios

Abstract

This invention discloses a lane line detection method based on off-site traffic enforcement scenarios. The method utilizes a multi-feedback feature pyramid module constructed from ResNet50 and FPN. Then, pixel-level anchorless bounding box regression, combined with hierarchical prediction and centrality calculation, filters out low-quality and meaningless regression boxes, improving the lane line prediction rate and obtaining the coordinates of the outer rectangle of the lane line. Finally, spatial positional filtering, clustering, and line fitting are applied to obtain the final lane line detection result in the image to be detected. This invention can effectively detect lane lines in various complex environments, including shadows and uneven lighting, multi-vehicle interference, rainwater, stains, and reflections on the road surface.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a lane line detection method based on non-on-site traffic enforcement scenarios. Background Technology

[0002] In non-on-site traffic enforcement scenarios, the lane line violation corresponding to the lane line is illegal lane changing. Determining this violation is a crucial part of AI-assisted judgment. AI needs to know the relationship between the vehicle's position and the lane lines to determine whether the vehicle has moved from one side of the lane to the other, thus requiring knowledge of the lane line position. Illegal lane changing severely impacts the normal driving of other vehicles, and this type of violation is quite common. Many enforcement cameras target and capture this violation, and the increase in enforcement cameras means an increased workload for labeling. Therefore, implementing lane line detection in non-on-site traffic enforcement scenarios is extremely necessary.

[0003] Current research on lane detection mostly focuses on autonomous driving scenarios, with very little research on lane detection in off-site traffic scenarios. However, autonomous driving scenarios differ significantly from off-site traffic scenarios, making direct application impossible. Algorithms specifically designed for off-site traffic scenarios need to be developed.

[0004] In summary, lane line detection in non-on-site traffic enforcement scenarios presents the following challenges:

[0005] (1) Passing vehicles will obscure the lane lines, especially large vehicles, which will completely obscure the lane lines.

[0006] (2) The lane lines are long and thin, with indistinct features, and are easily confused with other targets.

[0007] (3) Actual traffic conditions are very complex, such as shadows and uneven lighting, rainwater, stains and reflections on the road surface, which can interfere with the detection of lane lines.

[0008] How to solve the problem of lane line detection in off-site traffic scenarios is a technical issue that urgently needs to be addressed. Summary of the Invention

[0009] The purpose of this invention is to solve the problems existing in the prior art and to provide a lane line detection method based on off-site traffic enforcement scenarios. The specific technical solution adopted by this invention is as follows:

[0010] A lane line detection method based on off-site traffic enforcement scenarios includes:

[0011] S1. Obtain an labeled training dataset, in which each image sample contains an image taken from above by a law enforcement camera, showing lane lines. The images are pre-labeled with the two endpoints of the center line of each lane line and the lane line offset. The lane line offset is used to determine whether the lane line is on the main diagonal or the secondary diagonal within its bounding rectangle. The image samples in the training dataset belong to different intersection scenarios, and all image samples are divided into a daytime image subset taken during the day and a nighttime image subset taken at night according to the shooting time.

[0012] S2. For each image sample in the training dataset, taking into account the two retention principles of prioritizing images with fewer vehicles and prioritizing daytime images over nighttime images, filter all image samples in the same intersection scene by combining the image's grayscale value and the number of vehicles in the image, and remove image samples exceeding the threshold for each intersection scene.

[0013] S3. With the goal of minimizing the loss function, train the lane detection network using the training dataset filtered by S2.

[0014] The lane detection network includes an input module, a multi-feedback feature pyramid module, a classification and regression module, and an output module.

[0015] The input module is used to input the original image of the lane line to be detected into the network;

[0016] The multi-feedback feature pyramid module is composed of multiple cascaded ResNet50-based feature pyramid networks. The first ResNet50-based feature pyramid model uses the original image as its sole input, extracts multi-level features from the bottom up using ResNet50 as the backbone network, and then outputs four feature maps at different scales through the feature pyramid network. Except for the first ResNet50-based feature pyramid model, each of the remaining ResNet50-based feature pyramid models uses the original image as the first input and receives the four feature maps at different scales output from the previous ResNet50-based feature pyramid model as the second input. During the process of the ResNet50 backbone network extracting feature maps from the first input from the bottom up, after each feature map of a certain scale is extracted, it is connected with the feature map of the corresponding scale in the second input and then used to extract the feature map of the next scale through a 3*3 dilated convolution. The four feature maps at different scales output by the feature pyramid network in the current ResNet50-based feature pyramid model are then fused with the second input at the corresponding level, and finally output as the four feature maps at different scales as the final output of the current ResNet50-based feature pyramid model.

[0017] The four feature maps at different scales output by the multi-feedback feature pyramid module serve as inputs to the classification and regression module. The classification and regression module includes a classification network and a regression network. The classification network performs binary classification on each pixel in each feature map, outputting the classification label and confidence score of whether the mapped region of each pixel in the original image belongs to the lane line. The regression network obtains six prediction parameters for each pixel in each feature map through regression, including the distance from the center of the mapped region of each pixel in the original image to the four boundaries of the bounding rectangle of the lane line, the lane line skew, and the centrality.

[0018] In the output module, the first step is to use the binary classification result of the classification network to perform a first screening on the pixels in each feature map, removing pixels that do not belong to the lane lines. Then, a second screening is performed on the remaining pixels that belong to the lane lines, removing pixels whose distance values ​​exceed the threshold range. Finally, non-maximum suppression is performed on the remaining pixels based on the corresponding confidence level, and the prediction result of the bounding rectangle of the lane lines in the original image is obtained. For each bounding rectangle in the prediction result, the diagonal of the bounding rectangle is determined by combining the lane line bias, which is used as the final output lane line prediction result.

[0019] S4. Input the image to be detected containing lane lines into the trained lane line detection network, and output the lane line prediction result in the image to be detected.

[0020] S5. Filter the lane line prediction results in the image to be detected and remove line segments that do not satisfy the spatial position relationship of the lane lines in the image.

[0021] S6. Divide each lane line remaining after S5 filtering into a series of points, and cluster all the points of all lane lines together. Points belonging to the same lane line are clustered into the same cluster.

[0022] S7. Perform line fitting on the points in each cluster, and use the fitted line segment as the final lane line detection result in the image to be detected.

[0023] Preferably, the specific method of S2 is as follows:

[0024] S21. Convert each image sample in the training dataset from an RGB image to a grayscale image. Then calculate the mean grayscale value of all pixels in each image sample. Then calculate the average of the mean grayscale values ​​of all image samples in each subset for the daytime image subset and the nighttime image subset respectively, and use it as the average brightness of the corresponding subset. Use the average brightness of the two subsets as the brightness distinction threshold for distinguishing between day and night.

[0025] S22. Use the trained target detection model to detect vehicles in each image sample in the training dataset to obtain the number of vehicles in each image sample. Then calculate the average number of vehicles in all image samples in the training dataset. Finally, calculate the vehicle weight of each image sample as the ratio of the number of vehicles in the image sample to the average number of vehicles multiplied by the average brightness of the daytime image subset.

[0026] S23. Based on the brightness discrimination threshold and vehicle weight, calculate the quality weight of each image sample in the training dataset = 255 + λ * α * gray - β * carWeight, where gray represents the mean gray value of all pixels in the currently calculated image sample, carWeight represents the vehicle weight corresponding to the currently calculated image sample, α and β are two weights, and λ is the weight determined by the brightness discrimination threshold bound and gray. If gray ≥ bound, then λ = λ1; if gray < bound, then λ = λ2, λ1 + λ2 = 1 and λ1 > λ2.

[0027] S24. For all image samples in each intersection scene in the training dataset, sort them according to their respective quality weights. If the number of image samples in an intersection scene exceeds the threshold, retain the image samples that meet the threshold in descending order of quality weight. If the number of image samples in an intersection scene does not exceed the threshold, retain all image samples.

[0028] Preferably, the weights α and β are 1 and 2 respectively, and the weights λ1 and λ2 are 0.6 and 0.4 respectively.

[0029] Preferably, the centrality of each pixel is calculated from the distances from the center of the mapped region of that pixel in the original image to the four boundaries of the bounding rectangle of the lane line, using the following formula:

[0030]

[0031] In the formula: centra represents the centrality of a pixel, and (l,t,r,b) is the distance from the center of the mapped region of the pixel in the original image to the four boundaries of the bounding rectangle of the lane line; the centrality of a pixel ranges from 0 to 1, the larger the centrality of the pixel, the higher the confidence level, and the smaller the centrality of the pixel, the lower the confidence level.

[0032] Preferably, the loss function of the lane detection network is:

[0033]

[0034] In the formula: N represents the number of classification labels; λ, μ, and ν represent the weights of the three types of losses, respectively; L cls This represents the focus loss, used to calculate the category prediction error; c represents the predicted bounding box category at position (x, y). x,y This represents the category of the true bounding box at position (x, y), with a category value of 1 if it belongs to a lane line and a category value of 0 if it does not belong to a lane line; L reg This represents the IOU loss, used to calculate the error between the predicted bounding box and the ground truth bounding box. It is a conditional function; if If the value is greater than 0, the function value is 1; otherwise, it is 0. x,y This represents the four distances from the position (x, y) to the four boundaries of the ground truth bounding box. L represents the four distances from the position (x, y) to the four boundaries of the prediction box; ctr The binary cross-entropy loss function, representing location centrality, is used to calculate the distance between the center point of the predicted bounding box and the center point of the ground truth bounding box. x,y Let (x, y) be the centrality from the position (x, y) to the center point of the true bounding box. L is the centrality from the location (x,y) to the center point of the predicted bounding box; lean The binary cross-entropy loss function, representing the tilt bias, is used to calculate the bias of lane lines within the outer rectangle. The value representing the prediction bias at position (x,y) is l. x,y This represents the true bias value corresponding to the position (x, y). The bias value is 1 when the lane line is on the main diagonal of the outer rectangle and 0 when it is on the secondary diagonal.

[0035] Preferably, in the output module, the remaining pixels belonging to the lane lines are filtered a second time to remove pixels whose distance values ​​exceed the threshold range. The specific method for this is as follows:

[0036] The five thresholds d0, d1, d2, d3, and d4 are set to 0, 64, 128, 256, and ∞, respectively. For the i-th layer feature map F output by the multi-feedback feature pyramid module... i If i = 1, 2, 3, 4, then if F i The four distance values ​​corresponding to a pixel (l) (i) ,t (i) ,r (i) ,b (i) ) satisfies max(l (i) ,t (i) ,r (i) ,b (i) )<d i-1 or max(l (i) ,t (i) ,r (i),b (i) )>d i If so, then that pixel will be removed.

[0037] Preferably, the multi-feedback feature pyramid module consists of three cascaded feature pyramid networks based on ResNet50.

[0038] Preferably, in step S5, when filtering the lane line prediction results in the image to be detected, the spatial position relationship of the lane lines in the image should be such that the angle between the lane lines and the horizontal line is not within the range of 45 to 90 degrees.

[0039] Preferably, in step S6, the specific method for clustering all lane line points is as follows:

[0040] S61. Place all pixels into the first set, which is initialized to be empty;

[0041] S62. Randomly select a pixel from the current first set and add it to the second set, which is initialized to be empty;

[0042] S63. Traverse all pixels in the first set and determine whether there are any pixels in the second set whose distance to the currently traversed pixel is less than the maximum clustering distance. If there are, add the currently traversed pixel to the second set. The maximum clustering distance is the maximum allowable distance between adjacent pixels on a lane line in the image.

[0043] S64. Repeat S63 until no new pixels are added to the second set. Then, take all the pixels in the second set as a cluster, and the pixels in the cluster belong to the same lane line.

[0044] S65. Repeat S62 to S64 until all pixels in the first set are assigned to clusters, thus obtaining the pixels corresponding to each lane line.

[0045] Preferably, in step S7, the straight line fitting is implemented using the RANSAC algorithm.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1. This invention demonstrates excellent performance in lane line detection within complex scenes. During dataset construction, this invention calculates weights for images within the same scene, selecting images with fewer vehicles (fewer vehicles obstruct lane lines) and better lighting conditions (daytime lighting is superior to nighttime lighting) for training.

[0048] 2. This invention achieves very high accuracy in detecting and recognizing lane lines. Since lane lines are easily obscured by oncoming vehicles, this invention utilizes multiple images of the same scene to simultaneously detect lane lines, and then clusters and fits the detection results from each image, greatly improving the accuracy of detection in this scenario. Attached Figure Description

[0049] Figure 1 This is an example image of an illegal lane change in the embodiment.

[0050] Figure 2 The image shown is the original, unlabeled image used in the example.

[0051] Figure 3 This is a diagram showing the centerline annotation in the embodiment.

[0052] Figure 4 This is a set of image weight sorting images in the embodiment.

[0053] Figure 5 This is a diagram of the ODL-Net network model architecture in the embodiment.

[0054] Figure 6 This is a diagram of the ResNet50 network model structure in the embodiment.

[0055] Figure 7 This is the ResNet50-based FPN network in the embodiment.

[0056] Figure 8 This refers to the MFP module in the embodiment.

[0057] Figure 9 This is a diagram showing the MFP process in the embodiment.

[0058] Figure 10 This is a diagram illustrating the feedback connection process between FPNs in the embodiment.

[0059] Figure 11 The pixel prediction bounding box is shown in the embodiment.

[0060] Figure 12 This is a graph showing the model prediction results in the example.

[0061] Figure 13 This is a RANSAC fitted line graph from the embodiment.

[0062] Figure 14 This is the detection result image of the first ODL-Net algorithm in the embodiment.

[0063] Figure 15 This is a diagram showing the detection results of the second ODL-Net algorithm in the embodiment.

[0064] Figure 16 This is a diagram showing the detection results of the third ODL-Net algorithm in the embodiment.

[0065] Figure 17 This is the detection result image of the fourth ODL-Net algorithm in the embodiment. Detailed Implementation

[0066] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0067] In the description of this invention, it should be understood that when an element is considered to be "connected" to another element, it can be a direct connection to the other element or an indirect connection, i.e., there is an intermediate element. Conversely, when an element is said to be "directly" connected to another element, there is no intermediate element.

[0068] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature.

[0069] In a preferred embodiment of the present invention, a lane line detection method based on non-on-site traffic enforcement scenarios is provided, the specific steps of which are shown in S1 to S7. The specific implementation of each step is described in detail below.

[0070] S1. Obtain an labeled training dataset, in which each image sample contains an image of lane lines taken from above by a law enforcement camera. The images are pre-labeled with the two endpoints of the center line of each lane line and the lane line offset. The lane line offset is used to determine whether the lane line is on the main diagonal or the secondary diagonal within its bounding rectangle. The image samples in the training dataset belong to different intersection scenarios, and all image samples are divided into a daytime image subset taken during the day and a nighttime image subset taken at night according to the shooting time.

[0071] S2. For each image sample in the training dataset, taking into account the two retention principles of prioritizing images with fewer vehicles and prioritizing daytime images over nighttime images, filter all image samples in the same intersection scene by combining the image's grayscale value and the number of vehicles in the image, and remove image samples exceeding the threshold for each intersection scene.

[0072] In this invention, the specific method of S2 described above is as follows:

[0073] S21. Convert each image sample in the training dataset from an RGB image to a grayscale image, then calculate the mean grayscale value of all pixels in each image sample, and then calculate the average of the mean grayscale values ​​of all image samples in each subset for the daytime image subset and the nighttime image subset respectively, as the average brightness of the corresponding subset; use the average of the average brightness of the two subsets as the brightness distinction threshold for distinguishing between day and night.

[0074] S22. Use the trained object detection model to detect vehicles in each image sample in the training dataset to obtain the number of vehicles in each image sample. Then calculate the average number of vehicles in all image samples in the training dataset. Finally, calculate the vehicle weight of each image sample as the ratio of the number of vehicles in the image sample to the average number of vehicles multiplied by the average brightness of the daytime image subset.

[0075] S23. Based on the brightness discrimination threshold and vehicle weight, calculate the quality weight of each image sample in the training dataset = 255 + λ * α * gray - β * carWeight, where gray represents the mean gray value of all pixels in the currently calculated image sample, carWeight represents the vehicle weight corresponding to the currently calculated image sample, α and β are two weights, and λ is the weight determined by the brightness discrimination threshold bound and gray. If gray ≥ bound, then λ = λ1; if gray < bound, then λ = λ2, λ1 + λ2 = 1 and λ1 > λ2.

[0076] In this invention, the weights α and β are preferably 1 and 2, respectively, and the weights λ1 and λ2 are preferably 0.6 and 0.4, respectively.

[0077] S24. For all image samples in each intersection scene in the training dataset, sort them according to their respective quality weights. If the number of image samples in an intersection scene exceeds the threshold, retain the image samples that meet the threshold in descending order of quality weight. If the number of image samples in an intersection scene does not exceed the threshold, retain all image samples.

[0078] S3. With the goal of minimizing the loss function, train the lane detection network using the training dataset filtered by S2.

[0079] The lane detection network described above includes an input module, a multi-feedback feature pyramid module, a classification and regression module, and an output module. The specific functions and implementation of each of the four modules are described in detail below.

[0080] In this invention, the above-mentioned input module is used to input the original image of the lane line to be detected into the network. The original image input during the model training stage is the aforementioned image sample, while the original image input during the inference or practical application stage is the image to be detected.

[0081] In this invention, the aforementioned multi-feedback feature pyramid module is composed of multiple cascaded ResNet50-based feature pyramid networks. The first ResNet50-based feature pyramid model uses the original image as its sole input, extracts multi-level features from bottom to top using ResNet50 as the backbone network, and then outputs four feature maps at different scales through the feature pyramid network. Except for the first ResNet50-based feature pyramid model, in each of the remaining ResNet50-based feature pyramid models, the original image is used as the first input, and the four feature maps at different scales output from the previous ResNet50-based feature pyramid model are used as the second input. During the process of the ResNet50 backbone network extracting feature maps from the first input from bottom to top, after each feature map of a certain scale is extracted, it is connected with the feature map of the corresponding scale in the second input, and then used to extract the feature map of the next scale through a 3*3 dilated convolution. The four feature maps at different scales output by the feature pyramid network in the current ResNet50-based feature pyramid model are then fused with the second input at the corresponding level, and finally output as the four feature maps at different scales as the final output of the current ResNet50-based feature pyramid model.

[0082] Preferably, the aforementioned multi-feedback feature pyramid module is composed of three cascaded feature pyramid networks based on ResNet50.

[0083] It should be noted that when concatenating the four feature maps at different scales output from the previous ResNet50-based feature pyramid model with the feature maps extracted from the bottom up by the subsequent ResNet50 backbone network, it is necessary to maintain scale correspondence so that the two can be concatted. Similarly, when fusing the four feature maps at different scales output from the feature pyramid network in the current ResNet50-based feature pyramid model with the four feature maps in the second input, scale correspondence also needs to be considered so that the two can be fused.

[0084] In this invention, the four feature maps of different scales output by the aforementioned multi-feedback feature pyramid module are used as inputs to the classification and regression module. The classification and regression module includes a classification network and a regression network. The classification network is used to perform binary classification on each pixel in each feature map, outputting the classification label and confidence score of whether the mapped region of each pixel in the original image belongs to the lane line. The regression network is used to obtain six prediction parameters for each pixel in each feature map through regression, including the distance from the center of the mapped region of each pixel in the original image to the four boundaries of the bounding rectangle of the lane line, the lane line skew, and the centrality.

[0085] In this invention, both the classification network and the regression network described above can be implemented using four cascaded 1×1 convolutions. The classification network outputs a 1-channel feature map through four cascaded 1×1 convolutions, representing whether a pixel is located within the lane line area. The regression network outputs a 6-channel feature map through four cascaded 1×1 convolutions, where each pixel can obtain 6 parameters: the distance values ​​(l, t, r, b) from the center of the mapped region of each pixel in the original image to the four boundaries of the bounding rectangle of the lane line, the lane line skew (marking whether the lane line is located on the main diagonal or secondary diagonal of the bounding rectangle), and the centrality.

[0086] In this invention, the centrality of each pixel is calculated from the distance from the center of the mapped region of that pixel in the original image to the four boundaries of the bounding rectangle of the lane line. The calculation formula is as follows:

[0087]

[0088] In the formula: centra represents the centrality of a pixel, and (l,t,r,b) is the distance from the center of the mapped region of the pixel in the original image to the four boundaries of the bounding rectangle of the lane line; the centrality of a pixel ranges from 0 to 1, the larger the centrality of the pixel, the higher the confidence level, and the smaller the centrality of the pixel, the lower the confidence level.

[0089] In this invention, the output module first uses the binary classification result of the classification network to perform a first screening of the pixels in each feature map, removing pixels that do not belong to the lane lines. Then, it performs a second screening of the remaining pixels that belong to the lane lines, removing pixels whose distance values ​​exceed the threshold range. Finally, it performs non-maximum suppression on the remaining pixels based on the corresponding confidence level, and finally obtains the prediction result of the bounding rectangle of the lane lines in the original image. For each bounding rectangle in the prediction result, the diagonal of the bounding rectangle is determined by combining the lane line skew, which is used as the final output lane line prediction result.

[0090] In this invention, the specific method for performing a second screening on the remaining pixels belonging to the lane lines in the above-mentioned output module to remove pixels whose distance values ​​exceed the threshold range is as follows:

[0091] The five thresholds d0, d1, d2, d3, and d4 are set to 0, 64, 128, 256, and ∞, respectively. For the i-th layer feature map F output by the multi-feedback feature pyramid module... i If i = 1, 2, 3, 4, then if F i The four distance values ​​corresponding to a pixel (l) (i) ,t (i) ,r (i) ,b (i) ) satisfies max(l (i) ,t (i) ,r (i) ,b (i) )<d i-1 or max(l (i) ,t (i) ,r (i) ,b (i) )>d i If so, then that pixel will be removed.

[0092] During training, the loss function of the lane detection network described above is:

[0093]

[0094] In the formula: N represents the number of classification labels; λ, μ, and ν represent the weights of the three types of losses, respectively; L cls This represents the focus loss, used to calculate the category prediction error; c represents the predicted bounding box category at position (x, y). x,y This represents the category of the true bounding box at position (x, y), with a category value of 1 if it belongs to a lane line and a category value of 0 if it does not belong to a lane line; L reg This represents the IOU loss, used to calculate the error between the predicted bounding box and the ground truth bounding box. It is a conditional function; if If the value is greater than 0, the function value is 1; otherwise, it is 0. x,y This represents the four distances from the position (x, y) to the four boundaries of the ground truth bounding box. L represents the four distances from the position (x, y) to the four boundaries of the prediction box; ctr The binary cross-entropy loss function, representing location centrality, is used to calculate the distance between the center point of the predicted bounding box and the center point of the ground truth bounding box. x,y Let (x, y) be the centrality from the position (x, y) to the center point of the true bounding box. L is the centrality from the location (x,y) to the center point of the predicted bounding box; leanThe binary cross-entropy loss function, representing the tilt bias, is used to calculate the bias of lane lines within the outer rectangle. The value representing the prediction bias at position (x,y) is l. x,y This represents the true bias value corresponding to the position (x, y). The bias value is 1 when the lane line is on the main diagonal of the outer rectangle and 0 when it is on the secondary diagonal.

[0095] Preferably, the weight values ​​λ, μ, and ν are all 1.

[0096] It should be noted that the predicted bounding box and the actual bounding box mentioned above refer to the predicted and the actually marked lane lines, respectively.

[0097] S4. Input the image to be detected containing lane lines into the trained lane line detection network, and output the lane line prediction results in the image to be detected.

[0098] S5. Filter the lane line prediction results in the image to be detected, and remove line segments that do not satisfy the spatial position relationship of the lane lines in the image.

[0099] In this invention, when filtering the lane line prediction results in the image to be detected, the spatial position relationship of the lane lines in the image should satisfy that the angle between the lane lines and the horizontal line is not within the range of 45 to 90 degrees.

[0100] S6. After filtering by S5, each remaining lane line is divided into a series of points. All lane line points are clustered together, and points belonging to the same lane line are clustered into the same cluster.

[0101] In this invention, the specific method for clustering all lane line points is as follows:

[0102] S61. Place all pixels into the first set, which is initialized to be empty;

[0103] S62. Randomly select a pixel from the current first set and add it to the second set, which is initialized to be empty;

[0104] S63. Traverse all pixels in the first set and determine whether there are any pixels in the second set whose distance to the currently traversed pixel is less than the maximum clustering distance. If there are, add the currently traversed pixel to the second set. The maximum clustering distance is the maximum allowable distance between adjacent pixels on a lane line in the image.

[0105] S64. Repeat S63 until no new pixels are added to the second set. Then, take all the pixels in the second set as a cluster, and the pixels in the cluster belong to the same lane line.

[0106] S65. Repeat S62 to S64 until all pixels in the first set are assigned to clusters, thus obtaining the pixels corresponding to each lane line.

[0107] S7. Perform line fitting on the points in each cluster, and use the fitted line segment as the final lane line detection result in the image to be detected.

[0108] In this invention, the straight line fitting is implemented using the RANSAC algorithm.

[0109] The lane detection methods shown in S1 to S7 above for non-on-site traffic enforcement scenarios will be applied to specific examples to demonstrate their implementation process and the technical effects they can achieve.

[0110] Example

[0111] In this embodiment, the lane line detection method based on non-on-site traffic enforcement scenarios shown in S1 to S7 above is implemented through the following process:

[0112] Step 1. Create a dataset

[0113] This embodiment focuses on real-world data from non-on-site traffic enforcement scenarios. All data originates from traffic monitoring equipment (enforcement cameras) capturing images of vehicles violating traffic rules. The data comprises 5247 images from 968 devices. Figure 1 This paper presents a typical image of an illegal lane change, captured from above by a law enforcement camera, showing lane markings. This invention proposes a lane marking method that differs from existing methods, specifically for non-on-site traffic enforcement scenarios.

[0114] In this embodiment, the tool used for image sample annotation is LabelImg. Unlike existing object detection annotation methods and semantic segmentation annotation methods, which typically use LabelImg to enclose the target with a rectangle that precisely matches the target's edge, existing semantic segmentation annotation methods use multiple points to annotate along the target's edge, forming a closed region that encloses the target, achieving pixel-level annotation—a rather cumbersome process.

[0115] In this embodiment, the lane lines to be detected are straight lines. However, straight lines cannot be labeled using bounding boxes for object detection, and semantic segmentation methods are too cumbersome. Therefore, this embodiment uses line segments to label the straight lines. The line segment annotation tool Labelme is used to label the center line of the lane lines. The reason for labeling the center line instead of the entire lane line segment is that lane lines in the real world are not like line segments in the concept, which have no thickness. A lane line is a long and thin quadrilateral, not just a single line. Therefore, in actual annotation, it is only necessary to label the midpoints of the two short line segments at both ends of the lane line. Connecting the two midpoints represents the center line of the lane line. Figure 2 The original traffic image is unlabeled. Figure 3 This is the annotated image after the center line has been marked. As you can see, the annotation is very simple, requiring only two points.

[0116] After the annotation is completed, Labelme will automatically save the annotation results to a JSON (JavaScript ObjectNotation) file.

[0117] This embodiment uses object detection to detect lane lines, treating them as the diagonals of a rectangle. Therefore, existing line segment labels need to be converted to rectangle labels. First, the information of two points is read from the label JSON file. Then, the two points are sorted, with the point with the smaller x-coordinate designated as (x1, y1) and the point with the larger x-coordinate designated as (x2, y2). Next, the slope of the line segment is calculated using the following formula:

[0118]

[0119] If the slope is positive, the rectangle's coordinates [top, left, bottom, right] will be [y1, x2, y2, x1], and the angle bias will be set to 0. Conversely, if the slope is negative, it will be [y2, x2, y1, x1], and the angle bias will be set to 1. To prevent the slope from being uncalculated if the lines are perpendicular, the slope is not calculated when the x-coordinates of two points are the same; the angle bias is directly set to 1. Because the rectangle has two diagonals, the angle bias is 0 when the line segment is the main diagonal of the rectangle, and 1 when the line segment is the secondary diagonal. This angle bias represents the lane line bias within the outer rectangle. When the outer rectangle is subsequently detected, this bias marker can be used to determine whether the lane line is located on the main or secondary diagonal of the rectangle.

[0120] The converted annotation information is then used to regenerate the annotation file. The final annotation file contains the image name, target category, location information list (top, left, bottom, right), and lane line orientation.

[0121] Step 2. Data Preprocessing

[0122] This embodiment addresses a non-on-site traffic scenario, where each image sample is captured from an overhead enforcement camera at an intersection. Therefore, under the same enforcement camera, the background of the image data is often identical, with only the positions of people and vehicles differing. Each violating vehicle corresponds to three images, meaning that under the same device, this embodiment will obtain a large amount of traffic image data with the same background, differing only in the positions of people and vehicles. Given this context, and considering that various complex environmental factors such as vehicle occlusion, darkness, shadows, uneven lighting, rain, dirt, and reflections on the road surface can significantly impact lane line detection, the dataset should ideally be created using images from relatively ideal environments with minimal vehicle occlusion and good weather conditions. Therefore, weighting and filtering of the dataset are necessary to improve the lane line detection rate.

[0123] To better assess image quality, this embodiment employs a quality weighting formula that combines the image's grayscale value with the number of vehicles in the image. This formula calculates the quality value of all traffic images from the same device; a higher quality value indicates higher image quality. Images are then ranked based on quality, and given a sufficient number of images, higher-quality images are prioritized for lane line detection. This quality weighting method improves the lane line detection rate by filtering image quality.

[0124] Step 2.1 Calculate the thresholds for daytime and nighttime data.

[0125] First, all image data in the dataset is divided into two batches: daytime image data (called the daytime image subset) and nighttime image data (called the nighttime image subset). The average brightness of these two batches is calculated separately. Then, a brightness discrimination threshold that can distinguish between day and night is obtained by calculating the arithmetic mean of the daytime and nighttime image data. The calculation formula is as follows:

[0126] Gray=R*0.299+G*0.587+B*0.114 (2)

[0127]

[0128]

[0129]

[0130] In Equation (2), R, G, and B represent the three channels of the image, and the gray value of a single pixel is calculated using the values ​​of these three channels. Gray represents the gray value of a single pixel. Equation (3) is used to calculate the average gray value of all pixels in an image, where h and w are the height and width of the image, respectively. Equation (4) calculates the average brightness of a batch of image data (i.e., the average of the average gray values ​​of all images), where D represents a subset of data (which can be a batch of daytime data or a batch of nighttime data), and n represents the number of image samples in the subset. In Equation (5), bright day Bright represents the average brightness of a subset of images taken during the day. night The brightness of the nighttime image subset is represented by Equation (4), and both can be calculated using Equation (4). The bound represents the brightness distinction threshold that distinguishes between day and night.

[0131] Step 2.2 Calculate the vehicle weight threshold based on the number of vehicles.

[0132] A YOLOv3 pre-trained model was used to perform transfer learning on the scenario of this invention, resulting in a target detection model applicable to this scenario. This model was then used to detect vehicles in image data, obtaining the number of vehicles and their coordinates, which were then saved to a JSON file. In the JSON file format, the first two values ​​in each list represent the top-left coordinate of the vehicle, and the last two values ​​represent the bottom-right coordinate.

[0133] The vehicle weight is calculated based on the number of vehicles in the image, using the following formula:

[0134]

[0135]

[0136] Equation (6) calculates the average number of vehicles (avg) across all image data, where n represents the number of images in the dataset, and S represents the dataset. Equation (7) calculates the vehicle weights for the image. `carWeight` represents the vehicle weight for the current image, `carNum` represents the number of vehicles in the current image, and `bright` represents the vehicle weight. day This represents the average brightness of a subset of the daytime image; here, the average grayscale value of the daytime image, "bright," is used. day This is to improve the impact of vehicle weighting on image quality.

[0137] Step 2.3. Finally, calculate the quality weight of each image sample in the training dataset based on the brightness discrimination threshold and vehicle weight. The calculation formula is shown in the following formula (8):

[0138] geight=255+λ·α·gray-β·carWeight (8)

[0139]

[0140] Where α and β are the weights of brightness and number of vehicles, respectively. In this embodiment, these two values ​​are set to 1 and 2, respectively. This is because the number of vehicles has a much greater impact on lane lines than day and night, thus increasing the weight of vehicles. gray represents the average gray value of the image mentioned above, carWeight represents the vehicle weight of the image calculated by equation (7), and λ varies depending on the value of gray. When the value of gray is greater than or equal to the boundary value between day and night, the image is considered to be daytime, and the value of λ is 0.6; when the value of gray is less than the boundary value, the image is considered to be nighttime, and the value of λ is 0.4. This approach is mainly to increase the weight of daytime images, but since the impact of day and night is not significant, only the weight of daytime is increased to 0.6, and the weight of nighttime is set to 0.4.

[0141] This embodiment uses the aforementioned quality weights to sort image data captured by the same law enforcement camera at the same intersection. The reason for sorting, rather than directly removing low-quality images by setting a quality weight threshold, is that sometimes only images from nighttime or images with many vehicles are available. In such cases, using a quality weight threshold for removal would eliminate all images, making lane line detection impossible. Therefore, this invention uses a sorting method, selecting images with higher quality weights to obtain higher-quality images and improve lane line detection performance.

[0142] In this embodiment, after calculating the quality weights of all images in the scene, the image samples are sorted according to their quality weights. Then, the first six image samples are retained in ascending order of weight, and the remaining image samples are deleted from the dataset. The threshold of six retained image samples was obtained through multiple experiments to ensure the highest lane line recognition rate and the lowest error rate. If there are fewer than six image samples, there is no need to sort them using the quality formula, and all images are retained.

[0143] Figure 4 A set of images is shown to illustrate the image quality ranking effect of this step: (a) image with few cars during the day, (b) image with few cars at night, (c) image with many cars during the day, and (d) image with many cars at night. Since it is difficult to find an image that simultaneously meets all four conditions using the same device, these four images are from four different devices. The images are weighted according to a weighting formula, and it can be seen that the quality weighting formula of this invention favors images with fewer vehicles.

[0144] Step 3. Construction of ODL-Net, a lane detection network based on multiple feedback feature pyramids

[0145] The lane detection network based on a multi-feedback feature pyramid is named ODL-Net, and its network model architecture diagram is shown below. Figure 5 As shown.

[0146] In this embodiment, the backbone network used is ResNet50. To accommodate subsequent processing, the last fully connected (FC) layer of ResNet50 needs to be removed. The ResNet50 model structure is as follows: Figure 6 As shown, ResNet50 incorporates a residual network, which solves the problems of deep network parameters not being able to learn and gradient vanishing without adding extra parameters or increasing computational complexity. This allows the network to develop into deeper layers, thereby improving feature extraction performance. Furthermore, since this embodiment detects lane lines, which are long, thin, continuous shapes with strong spatial relationships but few external cues, ordinary convolutions often lose spatial hierarchy information, leading to the loss of image detail features. To better extract features, this embodiment converts all standard 3x3 convolutions in ResNet50 into 3x3 dilated convolution modules. These dilated convolution modules are composed of the results of dilated convolutions with a dilation factor of 1 and dilated convolutions with a dilation factor of 3, increasing the receptive field of each pixel in the feature map. This significantly improves the performance of the feature extraction network, ultimately increasing the lane line recognition rate.

[0147] Step 3.1 Construction of the Multiple Feedback Feature Pyramid Module

[0148] To extract lane line linear features more effectively, this embodiment proposes a Multiple Feedback Feature Pyramid (MFP) module, which performs multiple feature extractions on the image and multi-scale feature fusion to extract lane line context information more thoroughly and deeply, greatly improving the detection effect.

[0149] The Multiple Feedback Feature Pyramid (MFP) module proposed in this embodiment is built on top of the ResNet50-based Feature Pyramid Network (FPN). The ResNet50-based FPN structure is as follows: Figure 7 As shown. By incorporating the additional feedback connections of the FPN into the bottom-up backbone layer, a feature extraction network, namely the MFP module, is obtained, as follows. Figure 8 As shown. It can extract features from an image multiple times. The MFP module achieves stronger feature extraction capabilities through multiple image feature extractions. Similar to RFP, the feedback connection directly imports features received from the top layer of the network into the bottom layer of ResNet50 from the bottom up to accelerate training and improve performance. The MFP module proposed in this embodiment implements multiple feature extractions, with the bottom-up backbone and FPN running multiple times, and the final output features depending on the features in the previous steps. The MFP process can be expanded as follows: Figure 9As shown in the diagram. The feedback connection diagram between MFPs is as follows. Figure 10 As shown. This embodiment of MFP only uses a triple feature pyramid. Using more than three features results in only a small improvement in accuracy; instead, it increases the number of parameters and reduces training and detection speed. The specific data processing flow in the triple feature pyramid is as follows:

[0150] The first ResNet50-based feature pyramid model uses the original image as its sole input. ResNet50, as the backbone network, extracts multi-level features from the bottom up, and then the feature pyramid network outputs four feature maps at different scales. Except for this first ResNet50-based feature pyramid model, each subsequent model uses the original image as the first input and receives the four feature maps at different scales output from the previous ResNet50-based model as the second input. During the bottom-up feature map extraction process by the ResNet50 backbone network, after each extracted feature map scale, it is concatenated with the corresponding scale feature map from the second input and then subjected to a 3x3 dilated convolution to extract the next scale feature map. Finally, the four feature maps at different scales output by the feature pyramid network in the current ResNet50-based model are fused with the second input at the corresponding level, resulting in the final output of four feature maps at different scales.

[0151] In this embodiment, all standard 3x3 convolutions of the pyramids are replaced with dilated convolution modules, which can better acquire spatial information and improve the probability of lane line detection; at the same time, batch normalization (BN) can be added between each pyramid to improve the learning convergence speed and accelerate the training speed.

[0152] Step 3.2 Regression Prediction

[0153] This embodiment uses an anchorless method to perform coordinate regression on the target. Currently, most target detection models, such as SSD, RefineDet, YOLO series, RCNN series, RetinaNet, etc., use anchors or proposals to predict targets. That is, they match targets by pre-calculating predefined anchors or proposals, and a prediction is performed for each anchor or proposal.

[0154] For lane line detection in this embodiment, anchors present the following problems: the size and dimensions of the anchors are fixed, and when the target to be detected changes significantly, the anchors can negatively impact the detection performance and generalization ability; a large number of anchor boxes will generate a large number of negative samples, leading to an imbalance between positive and negative samples; at the same time, a large number of IOU calculations are required, which will affect the model's performance, resulting in slow training speed and a large amount of invalid memory consumption; therefore, this embodiment does not use preset anchors for lane line detection, and for this reason, this embodiment is simpler than existing anchor-based detection models.

[0155] By predicting each pixel on the feature map, the target bounding box of the object corresponding to each pixel is directly regressed, thus mapping each pixel on the feature map back to the input image.

[0156] The input image size is 800*800. Define the label of the i-th ground truth bounding box of the input image as box. i Then box i =(x0) (i) ,y0 (i) ,x1 (i) ,y1 (i) ,lean (i) ,c (i) Here (x0) (i) ,y0 (i) (x1) represents the coordinates of the top-left corner of the actual bounding rectangle. (i) ,y1 (i) ) represents the coordinates of the bottom right corner of the actual bounding rectangle, lean (i) This indicates the lane line offset of the true bounding box, with a value of 0 or 1, meaning whether the lane line is a primary diagonal or a secondary diagonal. (i) This indicates the category of the actual bounding box. In this embodiment, only one category is set, so c is 1.

[0157] Let F i Let be the feature map of the i-th layer output by the backbone network, and d represent the current downsampling factor of this feature map. Now, map each pixel on the feature map back to its original image location. Assuming the pixel coordinates are (x0, y0), the corresponding coordinates after mapping back to the original image are:

[0158]

[0159] The pixel position mapped back to the original image has d. 2 In this embodiment, the position is mapped back to the center position, so d / 2 needs to be added to both x and y.

[0160] During model training, instead of directly predicting the target bounding box, the model predicts the distances (l, t, r, b) from the pixel coordinates (x, y) to the four sides of the bounding box. The final target bounding box is then calculated using these four variables. The relationship between (l, t, r, b) and (x, y) is as follows: Figure 11 As shown, since each pixel corresponds to a ground truth bounding box, when a pixel falls into multiple ground truth bounding boxes, it is necessary to decide which bounding box to predict. In this embodiment, the ground truth bounding box with the smallest predicted area is used as the regression target. However, this still affects the model's performance. Therefore, this embodiment proposes hierarchical prediction to solve this problem.

[0161] The MFP module generates multiple feature maps, which can be used to perform hierarchical prediction of targets of different sizes. In this embodiment, the prediction method is to regress the four distances from the pixel to the four sides of the target bounding box. The larger the target, the larger these four distances. Therefore, by limiting the magnitude of these four distances, the size of the predicted target box can be controlled, thus solving the problem mentioned above where a point falls within multiple rectangles. The regression distance (l, t, r, b) at each location on all feature maps is calculated, and the regression distance is limited for each feature map. The ResNet50 backbone network generates four feature maps, F1 to F4, and sets the distances d0 to d4 to 0, 64, 128, 256, and ∞, respectively. If the prediction result at a location satisfies max(l... (i) ,t (i) ,r (i) ,b (i) )<d i-1 or max(l (i) ,t (i) ,r (i) ,b (i) )>d i If the bounding box is not regressed, it is no longer necessary because the size of the bounding box that should be predicted for the pixels on the feature map has been exceeded. In this embodiment, since lane lines are being detected, there is a certain distance between them, and the overlap is small. By limiting the maximum regression distance, this situation can be minimized. Even if there are still cases where the bounding box falls into multiple ground truth bounding boxes, only the ground truth bounding box with the smallest area needs to be selected as the regression target. Hierarchical prediction can largely solve this problem.

[0162] Finally, considering that a target bounding box covers a large area, each pixel will generate a target bounding box. Locations far from the center of the target bounding box will have larger prediction errors, resulting in many low-quality, meaningless bounding boxes and degrading prediction performance. To eliminate these bounding boxes, this embodiment adds a variable to predict the centrality of the location, primarily describing the normalized distance from that location to the regressed target center. The centrality calculation formula is:

[0163]

[0164] Centrality reduces the confidence of bounding boxes that are far from the target center. These low-confidence bounding boxes are then filtered out using Non-Maximum Suppression (NMS), thus improving detection performance. Centrality ranges from 0 to 1, so it can be trained using a binary cross-entropy loss, which is added to the loss function.

[0165] In this embodiment, both the classification network and the regression network can be implemented using four cascaded 1×1 convolutions. The classification network outputs a 1-channel feature map through four cascaded 1×1 convolutions, representing whether a pixel is within the lane line area. The regression network outputs a 6-channel feature map through four cascaded 1×1 convolutions, where each pixel receives six parameters: the distance (l, t, r, b) from the center of the mapped region of each pixel in the original image to the four boundaries of the lane line's bounding rectangle, the lane line skew (marking whether the lane line is located on the main diagonal or secondary diagonal of the bounding rectangle), and the centrality.

[0166] In summary, each pixel will have four output coordinates: lane line skew, centrality, a binary classifier label, and a confidence score related to the centrality. In other words, this embodiment directly treats the target bounding box as a training sample. The predicted boundary distance and the original image position are combined to obtain the predicted bounding box. Then, the skewness is combined to determine the lane line orientation, and finally, the coordinates of the two endpoints of the lane line are obtained.

[0167] Step 3.3 Loss Function

[0168] The loss function for training the ODL-Net network model is defined as follows:

[0169]

[0170] In the formula: N represents the number of classification labels; λ, μ, and ν represent the weights of the three types of losses, respectively; L cls This represents the focus loss, used to calculate the category prediction error; c represents the predicted bounding box category at position (x, y). x,yThis represents the category of the true bounding box at position (x, y), with a category value of 1 if it belongs to a lane line and a category value of 0 if it does not belong to a lane line; L reg This represents the IOU loss, used to calculate the error between the predicted bounding box and the ground truth bounding box. It is a conditional function; if If the value is greater than 0, the function value is 1; otherwise, it is 0. x,y This represents the four distances from the position (x, y) to the four boundaries of the ground truth bounding box. L represents the four distances from the position (x, y) to the four boundaries of the prediction box; ctr The binary cross-entropy loss function, representing location centrality, is used to calculate the distance between the center point of the predicted bounding box and the center point of the ground truth bounding box. x,y Let (x, y) be the centrality from the position (x, y) to the center point of the true bounding box. L is the centrality from the location (x,y) to the center point of the predicted bounding box; lean The binary cross-entropy loss function, representing the tilt bias, is used to calculate the bias of lane lines within the outer rectangle. The value representing the prediction bias at position (x,y) is l. x,y The value represents the true bias corresponding to the position (x, y). The bias value is 1 when the lane line is located on the main diagonal of the outer rectangle, and 0 when it is located on the secondary diagonal. In this embodiment, the weight values ​​λ, μ, and ν are all 1.

[0171] Step 3.4 Model Training and Prediction

[0172] The previous dataset was divided into training, validation, and test sets in a 7:2:1 ratio. A lane detection model based on a multi-feedback feature pyramid was trained on the training set. The training completion was determined by the changes in loss on the training and validation sets. Training ended when the training loss decreased by less than 0.01 within 50 iterations, and the validation loss began to increase. The model was then saved for future use. In this example, the model performance was optimal when the training loss reached 0.43 and the validation loss reached 0.56.

[0173] The trained model is then used to predict images in the test set to evaluate its performance. The model prediction results are as follows: Figure 12 As shown, the model training is very effective, and the prediction results can detect lane lines quite well.

[0174] Step 4: Lane line post-processing

[0175] Multiple images from the same device are preprocessed, and then the aforementioned model is used to detect lane lines in the preprocessed images, yielding lane line detection results. These results are then merged into a single set, and post-processing is applied to these line segments to obtain the final lane lines for the scene. This embodiment will provide a detailed description of the post-processing techniques.

[0176] Step 4.1 Line Segment Filtering

[0177] After calling the model to detect multiple images, multiple line segments will be generated. Some of these line segments are generated by detection errors. Even if they are indeed lane lines, multiple line segments will be generated. Therefore, this embodiment needs to filter the lane lines and remove those that are unnecessary or have been detected incorrectly.

[0178] Statistical analysis of lane line angles in the dataset reveals that lane lines are mostly perpendicular to the x-axis, while stop lines are typically perpendicular to the y-axis. Even with slight angle shifts, the changes are minimal, suggesting that stop line angles are within 10 degrees. Lane line angles are relatively larger than stop line angles, mostly perpendicular to the x-axis, with only minor shifts, which are not excessive. Therefore, lane line angles can be considered between 45 and 90 degrees. This is because the traffic images in this embodiment are captured by law enforcement cameras. Excessive angle shifts could cause vehicle height to obscure lane lines, making it difficult for traffic police to determine if a vehicle is crossing the line, thus affecting violation detection. This is a characteristic of this scenario. Therefore, this embodiment utilizes the angle range of stop lines and lane lines to filter out line segments with angles between 10 and 45 degrees to the horizontal line in the ODL-Net network model's detection results; these segments are considered false detections.

[0179] Step 4.2 Lane line clustering

[0180] After obtaining a large number of lane line detection results, it is necessary to extract the actual lane lines from them, which requires clustering these line segments. However, since line segment clustering is difficult to implement and the lane line detection results still contain some erroneous detection results, various line segments may be intertwined, causing all lines to be clustered into one class. To solve this problem, this embodiment proposes a clustering algorithm PDcluster based on point spacing calculation, which transforms line segment clustering into point clustering.

[0181] To convert line segments into multiple points, the steps are as follows: Divide each line segment into 10-pixel segments. For example, a 100-pixel line segment will be converted into 11 points, with each point spaced 10 pixels apart. Using this method, all line segments are converted into points, and then all points are clustered together.

[0182] The following describes the specific steps of the clustering algorithm based on the distance between nodes:

[0183] 1) Place all pixels into the first set A, which is initialized to be empty;

[0184] 2) Randomly select a pixel from the current first set A and add it to the second set B, which is initialized to be empty;

[0185] 3) Traverse all pixels in the first set A, and determine whether there are any pixels in the second set B whose distance to the currently traversed pixel is less than the maximum clustering distance. If there are, add the currently traversed pixel to the second set B. The maximum clustering distance is the maximum allowable distance between adjacent pixels on a lane line in the image.

[0186] 4) Repeat step 3) until no new pixels are added to the second set B. Then, take all the pixels in the second set B as a cluster, and the pixels in the cluster belong to the same lane line.

[0187] 5) Repeat steps 2) through 4) until all pixels in the first set A have been assigned to a cluster, thus obtaining the pixel corresponding to each lane line. This results in n clusters, each typically containing tens or hundreds of pixels.

[0188] Step 4.3 Lane line fitting

[0189] After clustering, each lane line is grouped into a single class. Fitting the points within each class yields the correct lane line. Since a small portion of each class may contain falsely detected segments, using least squares fitting might result in poor fit. Therefore, this embodiment employs the Random Sample Consensus (RANSAC) algorithm to fit the lane lines. The RANSAC algorithm iteratively estimates the parameters of a mathematical model from a dataset containing outliers. Increasing the number of iterations significantly increases the probability of obtaining the desired result, unaffected by outliers. In this embodiment, the falsely detected segments are converted to points, becoming outliers. The RANSAC algorithm can detect the true lane lines from this outlier-containing data. Using least squares to fit a straight line would cause the segments to be affected by outliers, deviating from their original positions. Furthermore, the fitted lane lines are trimmed, removing excess portions. The RANSAC algorithm's straight line fitting result is shown below. Figure 13 As shown.

[0190] At this point, the complete lane lines have been detected. All detected lane lines can be represented by the coordinates of their two endpoints. Therefore, the detection results can be saved to a JSON file. During manual adjustments, the lane lines can be visualized on the image background, and the manual operator can drag the endpoints of the lane lines requiring correction to make adjustments.

[0191] In summary, this invention demonstrates excellent performance in lane line detection under complex scenarios. By weighting images within the same scene during dataset construction, this invention selects images with fewer vehicles (fewer vehicles obstruct lane lines) and better lighting conditions (daytime lighting is better than nighttime lighting) for training, thus largely mitigating the impact of data from adverse conditions on the model. Figure 14 , Figure 15 , Figure 16 , Figure 17 Examples of test results are given for different scenarios: dark environment with worn lane lines, rain at night with water accumulation and dirt interference, lane lines obstructing reflection at night, and multiple vehicles reflecting water at night.

[0192] Since lane lines are easily obscured by oncoming vehicles, this invention utilizes multiple images of the same scene to simultaneously detect lane lines, and then clusters and fits the detection results of each image, greatly improving the detection accuracy in this scene. As shown in Tables 1 and 2, the detection accuracy of this invention can reach 96.98% and 95.49% respectively on the widely used TuSimple and CULane evaluation datasets for lane line assessment.

[0193] Table 1. Performance Comparison of Different Models under TuSimple Evaluation Metrics

[0194]

[0195] Table 2. Performance Comparison of Different Models under CULane Evaluation Indicators

[0196]

[0197] Therefore, it can be seen that the present invention has a very high accuracy in detecting and recognizing lane lines.

[0198] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A lane line detection method based on non-on-site traffic enforcement scenarios, characterized in that, include: S1. Obtain an labeled training dataset, in which each image sample contains an image taken from above by a law enforcement camera, showing lane lines. The images are pre-labeled with the two endpoints of the center line of each lane line and the lane line offset. The lane line offset is used to determine whether the lane line is on the main diagonal or the secondary diagonal within its bounding rectangle. The image samples in the training dataset belong to different intersection scenarios, and all image samples are divided into a daytime image subset taken during the day and a nighttime image subset taken at night according to the shooting time. S2. For each image sample in the training dataset, taking into account the two retention principles of prioritizing images with fewer vehicles and prioritizing daytime images over nighttime images, filter all image samples in the same intersection scene by combining the image's grayscale value and the number of vehicles in the image, and remove image samples exceeding the threshold for each intersection scene. S3. With the goal of minimizing the loss function, train the lane detection network using the training dataset filtered by S2. The lane detection network includes an input module, a multi-feedback feature pyramid module, a classification and regression module, and an output module. The input module is used to input the original image of the lane line to be detected into the network; The multi-feedback feature pyramid module is composed of multiple cascaded ResNet50-based feature pyramid networks. The first ResNet50-based feature pyramid model uses the original image as its sole input, extracts multi-level features from the bottom up using ResNet50 as the backbone network, and then outputs four feature maps at different scales through the feature pyramid network. Except for the first ResNet50-based feature pyramid model, each of the remaining ResNet50-based feature pyramid models uses the original image as the first input and receives the four feature maps at different scales output from the previous ResNet50-based feature pyramid model as the second input. During the process of the ResNet50 backbone network extracting feature maps from the first input from the bottom up, after each feature map of a certain scale is extracted, it is connected with the feature map of the corresponding scale in the second input and then used to extract the feature map of the next scale through a 3*3 dilated convolution. The four feature maps at different scales output by the feature pyramid network in the current ResNet50-based feature pyramid model are then fused with the second input at the corresponding level, and finally output as the four feature maps at different scales as the final output of the current ResNet50-based feature pyramid model. The four feature maps at different scales output by the multi-feedback feature pyramid module serve as inputs to the classification and regression module. The classification and regression module includes a classification network and a regression network. The classification network performs binary classification on each pixel in each feature map, outputting the classification label and confidence score of whether the mapped region of each pixel in the original image belongs to the lane line. The regression network obtains six prediction parameters for each pixel in each feature map through regression, including the distance from the center of the mapped region of each pixel in the original image to the four boundaries of the bounding rectangle of the lane line, the lane line skew, and the centrality. In the output module, the first step is to use the binary classification result of the classification network to perform a first screening on the pixels in each feature map, removing pixels that do not belong to the lane lines. Then, a second screening is performed on the remaining pixels that belong to the lane lines, removing pixels whose distance values ​​exceed the threshold range. Finally, non-maximum suppression is performed on the remaining pixels based on the corresponding confidence level, and the prediction result of the bounding rectangle of the lane lines in the original image is obtained. For each bounding rectangle in the prediction result, the diagonal of the bounding rectangle is determined by combining the lane line bias, which is used as the final output lane line prediction result. S4. Input the image to be detected containing lane lines into the trained lane line detection network, and output the lane line prediction result in the image to be detected. S5. Filter the lane line prediction results in the image to be detected and remove line segments that do not satisfy the spatial position relationship of the lane lines in the image. S6. Divide each lane line remaining after S5 filtering into a series of points, and cluster all the points of all lane lines together. Points belonging to the same lane line are clustered into the same cluster. S7. Perform line fitting on the points in each cluster, and use the fitted line segment as the final lane line detection result in the image to be detected.

2. The lane line detection method based on non-on-site traffic enforcement scenarios as described in claim 1, characterized in that, The specific method of S2 is as follows: S21. Convert each image sample in the training dataset from an RGB image to a grayscale image. Then calculate the mean grayscale value of all pixels in each image sample. Then calculate the average of the mean grayscale values ​​of all image samples in each subset for the daytime image subset and the nighttime image subset respectively, and use it as the average brightness of the corresponding subset. Use the average brightness of the two subsets as the brightness distinction threshold for distinguishing between day and night. S22. Use the trained target detection model to detect vehicles in each image sample in the training dataset to obtain the number of vehicles in each image sample. Then calculate the average number of vehicles in all image samples in the training dataset. Finally, calculate the vehicle weight of each image sample as the ratio of the number of vehicles in the image sample to the average number of vehicles multiplied by the average brightness of the daytime image subset. S23. Based on the brightness discrimination threshold and vehicle weight, calculate the quality weight of each image sample in the training dataset = 255 + λ * α * gray - β * carWeight, where gray represents the mean gray value of all pixels in the currently calculated image sample, carWeight represents the vehicle weight corresponding to the currently calculated image sample, α and β are two weights, and λ is the weight determined by the brightness discrimination threshold bound and gray. If gray ≥ bound, then λ = λ1; if gray < bound, then λ = λ2, λ1 + λ2 = 1 and λ1 > λ2. S24. For all image samples in each intersection scene in the training dataset, sort them according to their respective quality weights. If the number of image samples in an intersection scene exceeds the threshold, retain the image samples that meet the threshold in descending order of quality weight. If the number of image samples in an intersection scene does not exceed the threshold, retain all image samples.

3. The lane line detection method based on non-on-site traffic enforcement scenarios as described in claim 2, characterized in that, The weights α and β are 1 and 2, respectively, and the weights λ1 and λ2 are 0.6 and 0.4, respectively.

4. The lane line detection method based on non-on-site traffic enforcement scenarios as described in claim 1, characterized in that, The centrality of each pixel is calculated from the distances from the center of the mapped region of that pixel in the original image to the four boundaries of the bounding rectangle of the lane line, using the following formula: In the formula: centra represents the centrality of a pixel, and (l,t,r,b) is the distance from the center of the mapped region of the pixel in the original image to the four boundaries of the bounding rectangle of the lane line; the centrality of a pixel ranges from 0 to 1, the larger the centrality of the pixel, the higher the confidence level, and the smaller the centrality of the pixel, the lower the confidence level.

5. The lane line detection method based on non-on-site traffic enforcement scenarios as described in claim 1, characterized in that, The loss function of the lane detection network is: In the formula: N represents the number of classification labels; λ, μ, and ν represent the weights of the three types of losses, respectively; L cls This represents the focus loss, used to calculate the category prediction error; c represents the predicted bounding box category at position (x, y). x,y This represents the category of the true bounding box at position (x, y), with a category value of 1 if it belongs to a lane line and a category value of 0 if it does not belong to a lane line; L reg This represents the IOU loss, used to calculate the error between the predicted bounding box and the ground truth bounding box. It is a conditional function; if If the value is greater than 0, the function value is 1; otherwise, it is 0. x,y This represents the four distances from the position (x, y) to the four boundaries of the ground truth bounding box. L represents the four distances from the position (x, y) to the four boundaries of the prediction box; ctr The binary cross-entropy loss function, representing location centrality, is used to calculate the distance between the center point of the predicted bounding box and the center point of the ground truth bounding box. x,y Let (x, y) be the centrality from the position (x, y) to the center point of the true bounding box. L is the centrality from the location (x,y) to the center point of the predicted bounding box; lean The binary cross-entropy loss function, representing the tilt bias, is used to calculate the bias of lane lines within the outer rectangle. The value representing the prediction bias at position (x,y) is l. x,y This represents the true bias value corresponding to the position (x, y). The bias value is 1 when the lane line is on the main diagonal of the outer rectangle and 0 when it is on the secondary diagonal.

6. The lane line detection method based on non-on-site traffic enforcement scenarios as described in claim 1, characterized in that, In the output module, the remaining pixels belonging to the lane lines are filtered a second time to remove pixels whose distance values ​​exceed the threshold range. The specific method for this is as follows: The five thresholds d0, d1, d2, d3, and d4 are set to 0, 64, 128, 256, and ∞, respectively. For the i-th layer feature map F output by the multi-feedback feature pyramid module... i If i = 1, 2, 3, 4, then if F i The four distance values ​​corresponding to a pixel (l) (i) ,t (i) ,r (i) ,b (i) ) satisfies max(l (i) ,t (i) ,r (i) ,b (i) )<d i-1 or max(l (i) ,t (i) ,r (i) ,b (i) )>d i If so, then that pixel will be removed.

7. The lane line detection method based on non-on-site traffic enforcement scenarios as described in claim 1, characterized in that, The multi-feedback feature pyramid module consists of three cascaded feature pyramid networks based on ResNet50.

8. The lane line detection method based on non-on-site traffic enforcement scenarios as described in claim 1, characterized in that, In step S5, when filtering the lane line prediction results in the image to be detected, the spatial position relationship of the lane lines in the image should be such that the angle between the lane lines and the horizontal line is not within the range of 45 to 90 degrees.

9. The lane line detection method based on non-on-site traffic enforcement scenarios as described in claim 1, characterized in that, In step S6, the specific method for clustering all lane line points is as follows: S61. Place all pixels into the first set, which is initialized to be empty; S62. Randomly select a pixel from the current first set and add it to the second set, which is initialized to be empty; S63. Traverse all pixels in the first set and determine whether there are any pixels in the second set whose distance to the currently traversed pixel is less than the maximum clustering distance. If there are, add the currently traversed pixel to the second set. The maximum clustering distance is the maximum allowable distance between adjacent pixels on a lane line in the image. S64. Repeat S63 until no new pixels are added to the second set. Then, take all the pixels in the second set as a cluster, and the pixels in the cluster belong to the same lane line. S65. Repeat S62 to S64 until all pixels in the first set are assigned to clusters, thus obtaining the pixels corresponding to each lane line.

10. The lane line detection method based on non-on-site traffic enforcement scenarios as described in claim 1, characterized in that, In step S7, the straight line fitting is implemented using the RANSAC algorithm.

Images