Fast lane line detection method and system based on sketch refinement strategy

Abstract

The invention belongs to the technical field of computer vision, and discloses a sketch refinement strategy-based fast lane line detection method and system, and the method employs a sketch and refinement two-stage strategy to detect a fast lane line. By adopting the lightweight convolutional neural network in the sketch stage, the system can quickly extract image features, further efficiently estimate the local direction of the lane line and generate a preliminary lane line proposal, so that the system can complete detection without complex calculation, the detection speed is up to 278 frames per second, and the detection efficiency is greatly improved. According to the method, the recognition capability of the shape and continuity of the lane line is enhanced through the multi-stage feature fusion and lane section association module in the refining stage, so that the accuracy of lane line detection is greatly improved, the F1 score reaches 78.9%, the detection efficiency is improved, the detection precision is ensured, and the detection efficiency is improved. And the high requirement on real-time detection of the lane line in an automatic driving system is met.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a fast lane line detection method and system based on a sketch refinement strategy. Background Art

[0002] Lane detection is one of the core tasks of intelligent driving and advanced driver assistance systems (ADAS). Its performance is directly related to vehicle positioning, path planning and safety. In practical applications, lane detection needs to be completed accurately and efficiently in complex and changing scenarios, which places high demands on the design of algorithms. Existing lane detection methods can be mainly divided into the following two categories: proposal-based methods and key point-based methods;

[0003] Proposed lane detection methods usually complete lane detection by generating candidate regions and evaluating their possibilities. Such methods have a fast processing speed and can meet real-time requirements. However, since their detection process is usually completed through predefined rules or fixed patterns, they lack sufficient flexibility to cope with complex and changeable road conditions. For example, in the case of curved lanes, widening or multiple forks, proposed methods often find it difficult to generate accurate candidate regions. In addition, such methods usually rely on a large number of heuristic rules. When faced with challenges such as occlusion, light changes or blurred lanes, their performance will be significantly reduced.

[0004] The key point based method locates several key points of the lane line and then generates the complete lane line through fitting or interpolation technology. This type of method is more flexible and can adapt to the changes of lane lines in complex scenarios. However, the key point based method needs to process a large number of subsequent steps, including key point extraction, point association and curve fitting, which makes the algorithm more complex and the computational efficiency lower. Therefore, a new lane line detection method with both efficiency and flexibility is needed for improvement and optimization. Summary of the invention

[0005] The purpose of the present invention is to provide a fast lane line detection method and system based on sketch refinement strategy to solve the problems raised in the above background technology.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a fast lane line detection method and system based on a sketch refinement strategy, the method adopts a two-stage strategy of sketch and refinement to detect the fast lane line, and the specific steps are:

[0007] Step 1: Sketch phase: First, use a lightweight convolutional neural network to extract features from the input image.

[0008] Step 2: Based on the extracted features, the system outputs a local direction map of the lane line, thereby quickly determining the direction trend of the lane line;

[0009] Step 3: Construct an initial lane line proposal sketch based on the local direction information;

[0010] Step 4: Enter the refinement stage. In the refinement stage, the lane line shape will be optimized first through the lane segment association module;

[0011] Step 5: Then the system extracts image features at different levels and performs fusion to achieve the refinement of the lane line proposal sketch;

[0012] Step 6: Finally, based on the refined result, output the accurate lane line position and shape.

[0013] Preferably, in the sketch stage, a lightweight convolutional neural network is used to extract features from the input image, and the extracted features mainly include local texture, edge, and shape information. These information represent the preliminary features of the lane line and can be expressed as:

[0014] D(i,j) = (dx(i,j), dy(i,j))

[0015] where D(i,j) is the direction vector at position (i,j), and dx and dy are the horizontal and vertical components of the direction vector respectively. The direction consistency constraint can be expressed as minimizing the difference between adjacent pixel direction vectors, and the expression is: min ∑(i,j),(i′,j′)∈neighbors ||D(i,j) ― D(i′,j′)|| 2 .

[0016] Preferably, each pixel point of the local direction map represents the direction trend of that point through a direction vector. Using the direction consistency constraint, it is ensured that the directions in adjacent regions of the direction map are smooth and continuous, reducing direction jumps or unstable regions. And the generation of this local direction map is achieved through supervised learning, and the direction of the real lane line is used as the annotation during the training process.

[0017] Preferably, in Step 3, preliminary lane line proposals are generated through local direction estimation. These proposals will be used as the output of the sketch stage and provide input for the refinement stage. The specific steps for constructing the initial lane line proposal sketch are as follows:

[0018] A1. Integrate the direction information in the direction map into lane line segments according to the rules;

[0019] A2. Use direction consistency for inter-segment association, smooth the parts with sudden direction changes, and eliminate possible pseudo-lane line candidate regions;

[0020] A3. The proposals are represented in the form of line segments and point sets. Each line segment represents a local lane line segment. Based on the clustering algorithm, adjacent point segments are aggregated into the initial proposal of the complete lane line;

[0021] A4. Output the generated initial proposal in the form of a point set, a segment set, and a curve, providing input for the refinement stage.

[0022] Preferably, the lane segment association module dynamically adjusts by analyzing the geometric and directional relationships between adjacent lane line segments, further optimizing the preliminary proposal of the lane lines in the sketch stage. Its goal is to ensure the continuity of the lane line shape and avoid unnatural discontinuities or incorrect curve shapes between lane segments.

[0023] Preferably, in Step 5, low-level features and high-level features are extracted. Among them, the low-level features include edge information, texture, and linear structure, while the high-level features involve the global semantics of the lane, the overall trend of the lane shape, and the relative positions between lanes. Then, the low-level and high-level features are fused to ensure the unity of detail and global information. Its expression is:

[0024] F fused = g(F low , F high )

[0025] Among them, let the low-level feature be F low , and the high-level feature be F high . g is a fusion function that fuses information by taking a weighted average of the low-level and high-level features. Let α and β be weight parameters, and α + β = 1. Then the weighted average feature F fused can be expressed as the expression: F fused = g(F low , F high ).

[0026] Preferably, the system includes an input module, a sketch generation module, a detail optimization module, and an output module.

[0027] Preferably, the input module includes an image acquisition unit and an image preprocessing unit. Among them, the image acquisition unit obtains the road image in front during the vehicle's driving in real time through an in-vehicle camera, and the image preprocessing unit is used to perform denoising, normalization, and grayscale processing on the image to reduce image noise;

[0028] Among them, the expression of image preprocessing is:

[0029] In the formula, y i is the true label of the i-th sample, and x i is the predicted probability distribution of the model. The sample imbalance problem is solved by adjusting the weights α of positive and negative samples and the focusing degree γ;

[0030]

[0031] where N is the horizontal pixel equal division number of the picture, and r i，j is the region relation score of the i-th row and j-th column in the image.

[0032] Preferably, the sketch generation module is based on a lightweight convolutional neural network to extract features in the image and generate a local direction map; the detail optimization module adaptively optimizes and adjusts the lane line segments based on the lane segment association module.

[0033] Preferably, the output module outputs the lane lines in the forms of polygons, B-spline curves, and Bezier curves to meet the requirements of different path planning and control systems.

[0034] The beneficial effects of the present invention are as follows:

[0035] 1. By adopting a lightweight convolutional neural network in the sketch stage, the system of the present invention can quickly extract image features, and then efficiently estimate the local direction of the lane lines and generate preliminary lane line proposals, enabling the system to complete detection without complex calculations, with a detection speed of up to 278 frames per second. Through multi-level feature fusion and lane segment association module in the refinement stage, the recognition ability of the lane line shape and continuity is enhanced, thus greatly improving the accuracy of lane line detection, making the F1 score reach 78.9%. While improving the detection efficiency, the detection accuracy is ensured, meeting the high requirements for real-time detection of lane lines in the autonomous driving system.

[0036] 2. In the sketch stage of the present invention, through three key steps of image feature extraction, local direction estimation, and initial lane line proposal construction, a preliminary sketch of the lane line is quickly generated. Compared with traditional methods, this method is based on an efficient and lightweight design in the sketch stage, taking into account both real-time performance and the accuracy of direction estimation, providing reliable input for high-precision optimization in the refinement stage, and significantly reducing the demand for computing resources at the same time. This enables the method to operate efficiently on various devices.

[0037] 3. Through the lightweight design in the sketch stage of the present invention, the consumption of computing resources is effectively reduced, enabling the method to operate efficiently on devices with limited hardware resources. Through the adaptive adjustment in the refinement stage, the method can be applied to various complex environments, reducing the false alarm rate and missed detection rate, and improving the overall detection stability. Description of the Drawings

[0038] Figure 1 is the flowchart of the fast lane line detection method of the present invention;

[0039] Figure 2 is the framework diagram of the fast lane line detection system of the present invention. Detailed Embodiments

[0040] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0041] As Figures 1 to 2 shown, the embodiments of the present invention provide a fast lane line detection method and system based on a sketch refinement strategy. This method uses a two-stage strategy of sketch and refinement to detect fast lane lines, and the specific steps are as follows:

[0042] Step 1, in the sketch stage, first use a lightweight convolutional neural network to extract features from the input image;

[0043] Step 2, based on the extracted features, the system outputs a local direction map of the lane line, thereby quickly determining the direction trend of the lane line;

[0044] Step 3, according to the local direction information, construct an initial lane line proposal sketch;

[0045] Step 4, enter the refinement stage. In the refinement stage, first optimize the lane line shape through the lane segment association module (LSAM);

[0046] Step 5, then the system extracts image features at different levels and performs fusion to realize the refinement of the lane line proposal sketch;

[0047] Step 6, finally, according to the refined result, output the accurate lane line position and shape.

[0048] By using a lightweight convolutional neural network in the sketch stage, the system can quickly extract image features, and then efficiently estimate the local direction of the lane line and generate a preliminary lane line proposal, enabling the system to complete the detection without complex calculations, and the detection speed is as high as 278 frames per second. Through multi-level feature fusion and lane segment association module in the refinement stage, the recognition ability of the lane line shape and continuity is enhanced, thus greatly improving the accuracy of lane line detection, making the F1 score reach 78.9%, improving the detection efficiency while ensuring the detection accuracy, and meeting the high requirements for real-time lane line detection in the autonomous driving system.

[0049] Among them, in the sketch stage, a lightweight convolutional neural network is used to extract features from the input image, and the extracted features mainly include local texture, edge and shape information. These information represent the preliminary features of the lane line and can be expressed as:

[0050] D(i,j) = (dx(i,j), dy(i,j))

[0051] Among them, D(i,j) is the direction vector at position (i,j), and dx and dy are the horizontal and vertical components of the direction vector respectively. The direction consistency constraint can be expressed as minimizing the difference between the direction vectors of adjacent pixels, and the expression is: min ∑(i,j),(i′,j′)∈neighbors ||D(i,j) - D(i′,j′)|| 2 .

[0052] The design of the lightweight network ensures the computational efficiency of this stage and avoids the latency caused by complex network structures.

[0053] Through the lightweight design in the sketch stage, the consumption of computing resources is effectively reduced, enabling this method to operate efficiently on devices with limited hardware resources. Through the adaptive adjustment in the refinement stage, this method can be applied to various complex environments, reducing the false alarm rate and missed detection rate, and improving the overall detection stability.

[0054] Among them, each pixel point of the local direction map represents the direction trend of this point through a direction vector (such as an angle or direction probability). (Performing a convolution operation on the input feature map to predict the probability that each pixel point belongs to the lane line and the extension direction of the lane line), using the direction consistency constraint to ensure that the directions in adjacent regions of the direction map are smooth and continuous, reducing possible direction jumps or unstable regions, and the generation of this local direction map is achieved through supervised learning, with the direction of the real lane line as the annotation during the training process.

[0055] The generation of the local direction map provides key guidance for the initial lane line proposal, enabling the shape and extension trend of the lane line to be predicted quickly and roughly accurately.

[0056] Among them, in step three, preliminary lane line proposals are generated through local direction estimation. These proposals will be used as the output of the sketch stage and provide input for the refinement stage. The specific steps for constructing the initial lane line proposal sketch are as follows:

[0057] A1. Integrate the direction information in the direction map into a set of lane line segments or key points according to the rules;

[0058] A2. Use direction consistency for inter-segment association, smooth the parts with sudden direction changes, and eliminate possible pseudo-lane line candidate regions;

[0059] A3. The proposals are represented in the form of line segments and point sets. Each line segment represents a local lane line segment, and adjacent point segments are aggregated into a complete initial lane line proposal based on the clustering algorithm (DBSCAN);

[0060] A4. Output the generated initial proposals in the form of point sets, segment sets, and curves, providing input for the refinement stage.

[0061] In the sketch stage, three key steps of image feature extraction, local direction estimation, and initial lane line proposal are constructed to quickly generate a preliminary sketch of the lane line. Compared with traditional methods, this method is based on an efficient and lightweight design in the sketch stage, taking into account real-time performance and the accuracy of direction estimation, providing reliable input for high-precision optimization in the refinement stage, and significantly reducing the demand for computing resources. This enables the method to run efficiently on various devices.

[0062] Among them, the lane segment association module dynamically adjusts by analyzing the geometric and directional relationships between adjacent lane line segments to further optimize the preliminary lane line proposal in the sketch stage. Its goal is to ensure the continuity of the lane line shape and avoid unnatural discontinuities or incorrect curve shapes between lane line segments.

[0063] The LSAM module can adaptively adjust the shape of the lane line according to different scenarios. For example, in the face of complex scenarios such as curve segments, multi-lane bifurcations, or occlusions, the LSAM can automatically adjust the direction, curvature, and shape of the lane line, making the detection results of the lane line always reasonable and coherent. For curved roads, bifurcated sections, occlusions, or temporary road changes, the LSAM ensures high robustness of the lane line in various complex environments by introducing a dynamic adjustment mechanism.

[0064] Among them, in step five, low-level features and high-level features are extracted. The low-level features include edge information, texture, and linear structures, while the high-level features involve the global semantics of the lane, the overall trend of the lane shape, and the relative positions between lanes. Then, the low-level and high-level features are fused to ensure the unity of detail and global information. Its expression is:

[0065] F fused = g(F low , F high )

[0066] Among them, let the low-level feature be F low , and the high-level feature be F high . g is a fusion function that fuses information by taking the weighted average of the low-level feature and the high-level feature. Let α and β be weight parameters, and α + β = 1. Then the weighted average feature F fused can be expressed as the expression: F fused = g(F low , F high ).

[0067] By extracting high-level semantic features, the ability to understand and adapt to complex lane patterns on the road is improved, and then through feature stitching technology, the global information and local features are optimized and fused to improve the accuracy and recognizability of the lane line.

[0068] Among them, the system includes an input module, a sketch generation module, a detail optimization module, and an output module.

[0069] Among them, the input module includes an image acquisition unit and an image preprocessing unit. The image acquisition unit obtains the road image in front during the vehicle driving in real time through a vehicle-mounted camera, and the image preprocessing unit is used to perform denoising, normalization, and grayscale processing on the image to reduce image noise.

[0070] The expression of image preprocessing is as follows:

[0071] In the formula, y i is the true label of the i-th sample, and x i is the predicted probability distribution of the model. The sample imbalance problem is solved by adjusting the weights α of positive and negative samples and the focusing degree γ.

[0072]

[0073] In the formula, N is the horizontal equal division number of picture pixels, and r i，j is the regional relationship score of the i-th row and j-th column in the image.

[0074] The input module is responsible for receiving the image data collected in real time by the vehicle-mounted camera and performing necessary preprocessing operations to provide appropriate input for subsequent processing steps.

[0075] Among them, the sketch generation module extracts features in the image and generates a local direction map based on a lightweight convolutional neural network (CNN); the detail optimization module adaptively optimizes and adjusts the lane line segments based on the lane segment association module (LSAM).

[0076] Among them, the output module outputs the lane line in the forms of polygons, B-spline curves, and Bezier curves to meet the requirements of different path planning and control systems.

[0077] Among them, the sketch stage provides a basis for subsequent refinement and optimization through feature extraction, direction estimation, and preliminary proposal generation. Its specific implementation code is as follows:

[0078]

[0079]

[0080] Among them, the refinement stage mainly optimizes the shape and continuity of the lane line through the lane segment association module (LSAM) to ensure that the lane line is smoother and conforms to the actual road structure. Its specific implementation code is as follows:

[0081] class LSAM(nn.Module):

[0082] def

[0083] __init__ (

[0085] *self* ):

[0087] self.attention = MultiHeadAttention()

[0088] self.ffn = FeedForward()

[0089] def forward(

[0090] *self*

[0091] ,

[0092] *segments* ):

[0094] # Calculate inter-segment attention

[0095] attended = self.attention(segments)

[0096] # Feature enhancement

[0097] refined = self.ffn(attended)

[0098] return

[0099] refined;

[0100] Among them, the loss function is used to measure the gap between the prediction result and the true label during the training process. By minimizing the loss function, the model can learn better lane detection effects during training. The specific implementation code is as follows:

[0101]

[0102]

[0103] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0104] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A fast lane line detection method based on sketch refinement strategy, characterized by: This method uses a two-stage strategy of sketching and refinement to detect fast lane lines. The specific steps are as follows: Step 1: Sketch phase: First, use a lightweight convolutional neural network to extract features from the input image. Step 2: Based on the extracted features, the system outputs a local direction map of the lane line, thereby quickly determining the direction trend of the lane line; Step 3: Construct an initial lane line proposal sketch based on local direction information; Step 4: Enter the refinement stage, where the lane line shape is optimized through the lane segment association module; Step 5: The system then extracts image features at different levels and fuses them to refine the lane proposal sketch. Step six: Finally, based on the refined results, output the precise lane line position and shape.

2. The fast lane line detection method based on sketch refinement strategy according to claim 1, characterized in that: In the sketching stage, a lightweight convolutional neural network is used to extract features from the input image. The extracted features mainly include local texture, edge, and shape information. These information represent the preliminary features of the lane line and can be expressed as: D(i,j)=(dx(i,j),dy(i,j)) Where D(i,j) is the direction vector of position (i,j), dx and dy are the horizontal and vertical components of the direction vector respectively, and the direction consistency constraint can be expressed as minimizing the difference between the direction vectors of adjacent pixels, expressed as: min ∑(i,j),(i′,j′)∈neighbors ||D(i,j)―D(i′,j′)|| 2 .

3. The fast lane line detection method based on sketch refinement strategy according to claim 1, characterized in that: Each pixel point of the local directional map represents the directional trend of the point through a directional vector, and uses directional consistency constraints to ensure that the directions of adjacent areas in the directional map are smooth and continuous, reducing directional jumps or unstable areas. The generation of the local directional map is achieved through supervised learning, and the direction of the real lane line is used as an annotation during the training process.

4. The fast lane line detection method based on sketch refinement strategy according to claim 1, characterized in that: In step 3, preliminary lane line proposals are generated through local direction estimation. These proposals will serve as the output of the sketch stage and provide input for the refinement stage. The specific steps for constructing the initial lane line proposal sketch are: A1, integrate the direction information in the direction map into lane segments according to the rules; A2, using directional consistency to associate segments, smoothing the parts with sudden directional changes, and eliminating possible pseudo lane line candidate areas; A3, the proposal is represented in the form of line segments and point sets. Each line segment represents a local lane line segment. Based on the clustering algorithm, adjacent point segments are aggregated into a complete lane line initial proposal; A4,outputs the generated initial proposals in the form of point sets, segment sets, and curves, providing input for the refinement stage.

5. The fast lane line detection method based on sketch refinement strategy according to claim 1, characterized in that: The lane segment association module analyzes the geometric and directional relationships between adjacent lane segments and makes dynamic adjustments to further optimize the initial lane line proposal in the sketch stage. Its goal is to ensure the continuity of the lane line shape and avoid unnatural discontinuities or curve shape errors between lane line segments.

6. The fast lane line detection method based on sketch refinement strategy according to claim 1, characterized in that: The step 5 extracts low-level features and high-level features, where low-level features include edge information, texture, and linear structure, while high-level features involve the global semantics of lanes, the overall trend of lane shapes, and the relative positions between lanes. Low-level and high-level features are then fused to ensure the unity of details and global information. The expression is: F fused =g(F low ,F high ) Among them, let the underlying feature be F low , the high-level feature is F high , g is the fusion function, which fuses information by weighted averaging the underlying features and high-level features. Let α and β be weight parameters, and α+β=1, then the weighted average feature F fused It can be expressed as: F fused =g(F low ,F high ).

7. A fast lane line detection system based on sketch refinement strategy, characterized by: The system includes an input module, a sketch generation module, a detail optimization module and an output module.

8. The fast lane line detection system based on sketch refinement strategy according to claim 7, characterized in that: The input module includes an image acquisition unit and an image preprocessing unit, wherein the image acquisition unit acquires the road image in front of the vehicle in real time through the vehicle-mounted camera, and the image preprocessing unit is used to perform denoising, normalization and grayscale processing on the image to reduce image noise; The expression of image preprocessing is: Where y i is the true label of the i-th sample, x i For the predicted probability distribution of the model, the sample imbalance problem is solved by adjusting the weight α and the focusing degree γ of positive and negative samples; Where N is the horizontal square of the image pixels, r i，j is the relationship score of the region in the i-th row and j-th column in the image.

9. The fast lane line detection system based on sketch refinement strategy according to claim 7, characterized in that: The sketch generation module extracts features from the image and generates a local directional map based on a lightweight convolutional neural network; the detail optimization module adaptively optimizes and adjusts the lane segments based on the lane segment association module.

10. The fast lane line detection system based on sketch refinement strategy according to claim 7, characterized in that: The output module outputs lane lines in the form of polygons, B-spline curves and Bezier curves to meet different path planning and control system requirements.

Images