Monocular 3D lane detection method based on geometric prior

Abstract

The invention discloses a monocular 3D lane detection method based on geometric prior, belongs to the field of computer vision, and aims to improve the precision and robustness of lane line detection in a complex scene. The method comprises the following steps: acquiring a lane line data set and a label thereof; constructing a 3D lane line detection model based on geometric prior; performing training optimization on the model; and inputting a test image into the trained model, and outputting a 3D lane line detection result. The detection model constructed by the invention adopts a spatial transformation network guided by geometric information to adaptively extract the geometric features of the lane lines, and particularly has relatively high recognition capability for slender and curved lane lines in a complex scene, so that the accuracy and robustness of lane line detection are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a monocular 3D lane detection method based on geometric prior. Background Art

[0002] Lane detection technology is a key component of intelligent driving systems and is essential for accurate positioning and reliable navigation of vehicles. Currently, multi-sensor fusion technologies such as monocular, binocular, and lidar are more common in intelligent vehicles, among which monocular vision is the most widely used. Therefore, accurate and robust monocular lane detection has always been the goal pursued by the industry to ensure the safe driving of vehicles in complex traffic environments.

[0003] With the continuous advancement of deep learning technology, 2D lane detection technology has made significant breakthroughs in recent years. However, existing methods still face some challenges in practical applications. These methods mainly focus on 2D detection and often ignore the slope of the real road, resulting in a lack of accurate road morphology information, which is crucial for the operation of intelligent vehicles.

[0004] With the rapid development of autonomous driving technology, accurate lane line detection has become one of the important links to ensure the safe driving of vehicles. In real environments, the geometric structure of lane lines is usually affected by various complex factors, such as road curvature, slope, occlusion, and visual perspective distortion. One of the existing 3D lane line detection methods is to directly predict the 3D structure of lane lines from the front view (FV), and the other is to recover 3D lane line information from the bird's eye view (BEV). The former mostly adopts the Transformer-based architecture, which has high requirements for computing resources and poor real-time performance. The latter projects to the bird's eye view through inverse perspective mapping (IPM), and then recovers the 3D lane line information on this basis. IPM ignores the change in road height, resulting in divergence or convergence of the predicted lane lines when going uphill or downhill, thus affecting the detection accuracy. In addition, when dealing with these non-rigid structural features, convolutional neural networks often lack the ability to dynamically adapt to specific geometric structures. Therefore, it is of great significance for 3D lane line detection to propose a spatial transformation method that can dynamically focus on the structural features of lane lines and adapt to geometric changes. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a monocular 3D lane detection method based on geometric prior, which significantly improves the detection accuracy and robustness by combining the geometric prior of lane lines.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The technical solution of the present invention is a monocular 3D lane detection method based on geometric prior, comprising the following steps:

[0008] Step 1: Obtain lane line dataset and labels;

[0009] Step 2: Build a 3D lane detection model based on geometric priors, including the backbone network, spatial transformation network, upsampling network, and detection head:

[0010] Step 3: Train and optimize the 3D lane detection model;

[0011] Step 4: Use the trained and optimized 3D lane detection model to perform 3D lane detection.

[0012] Preferably, in step 2, the constructed 3D lane line detection model includes a backbone network, spatial conversion, upsampling and a detection head.

[0013] Preferably, in step three, the OpenLane training set images and labels are input into the 3D lane detection model constructed in step two, the training parameters (learning rate, optimizer, batch size and number of iterations) are set, and the model performs loss calculation while back-propagating the optimization model.

[0014] Preferably, in step 4, the OpenLane test set image is input into the model saved in step 3, and the 3D lane line detection result is output.

[0015] Preferably, in the step 2, FV is input into the backbone network to obtain shared features, and the backbone network adopts ResNet-34; the spatial transformation network is used to convert FV to BEV; and the detection network is used to detect 3D lane line information in BEV.

[0016] Preferably, in step 3, the input image size is adjusted to 576×1024×3, and the training process is optimized using the AdamW optimizer, with the beta parameters set to 0.9 and 0.999, and the weight decay is 0.01. The batch size is set to 16, the basic learning rate is 0.0005, and the model is trained for a total of 15 epochs, which takes 30 hours.

[0017] Preferably, in step 4, the size of the test image is adjusted to 576×1024×3, and input into the model trained in step 2 to infer the 3D lane line detection result.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] The present invention designs a monocular 3D lane detection method based on geometric prior, designs an efficient spatial conversion network, combines the geometric prior information of lane lines in the process of building BEV, dynamically focuses on the shape features of lane lines, and realizes accurate information conversion; the present invention effectively reduces the loss of feature information through the fusion of two-level feature layers, and further improves the accuracy and robustness of lane line detection; the present invention divides the grid in the height direction and predicts the offset of the lane line in the grid, so that the detection method can adapt to the three-dimensional geometric changes in complex road scenes and meet the actual application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of an embodiment of the present invention;

[0021] Figure 2 A diagram illustrating a network structure of an embodiment of the present invention;

[0022] Figure 3 A dynamic focusing module according to an embodiment of the present invention;

[0023] Figure 4 A space conversion network according to an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of offset prediction according to an embodiment of the present invention;

[0025] Figure 6 A visualization effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments. The present invention provides a monocular 3D lane detection method based on geometric priors, and the corresponding flow chart is as follows: Figure 1 As shown, the overall network structure is as follows Figure 2 As shown, the dynamic focusing module is Figure 3 As shown, the spatial transformation network is Figure 4 As shown, the offset prediction is Figure 5 As shown in the figure, the model detection visualization effect is as follows Figure 6 shown.

[0027] Figure 1 The figure shows a flow chart of a monocular 3D lane detection method based on geometric priors according to an embodiment of the present invention. The specific steps are as follows:

[0028] Step 1: Obtain lane line dataset and its labels;

[0029] Step 2: Build a 3D lane detection model including a backbone network, a spatial transformation network combined with geometric priors, an upsampling network, and a detection head;

[0030] Step 3: Use the optimization algorithm to train the model parameters and adjust the model weights through the back propagation of the loss function;

[0031] Step 4: Input the test image into the trained model to generate 3D lane detection results.

[0032] Figure 2 The figure shows a structural diagram of a monocular 3D lane detection model based on geometric priors according to an embodiment of the present invention. The specific steps are as follows:

[0033] Step 1: Build a backbone network, use ResNet-34 as the backbone network for lane feature extraction, set the input image size to 576×1024×3, and output the image through the backbone network and The sizes are 18×32×512 and 9×16×1024 respectively. The process includes conventional convolution operations to obtain the global features of the image;

[0034] The specific process of the space conversion network is as follows:

[0035] Figure 4 The geometric information guides the efficient spatial transformation network designed by the present invention, wherein the dynamic focusing module, such as Figure 3 As shown in the figure, the spatial transformer network combines the geometric adaptability of the dynamic focus module and the spatial alignment capability of the view conversion module to retain the complete lane geometry information when constructing the bird's-eye view feature, thereby significantly improving the accuracy, robustness and adaptability of 3D lane detection.

[0036] The specific implementation steps of the dynamic focusing module include:

[0037] Offset prediction: The offset matrix is ​​predicted by the convolutional network, which contains the y-direction offset of each convolution kernel position;

[0038] Coordinate mapping generation: Based on the morphological information and offset of the convolution kernel, a 2D coordinate mapping is dynamically generated to determine the sampling points of the convolution kernel on the image;

[0039] Feature interpolation: Using the generated 2D coordinate mapping, interpolation operations are performed on the input feature map to obtain dynamically adjusted feature representations;

[0040] like Figure 3 As shown in Figure 1, the dynamic focus module avoids the roaming phenomenon that may occur in the perception area of ​​the deformable convolution, thereby more accurately capturing the geometric features of the lane line; the dynamic focus module uses a convolution kernel of size M, expands horizontally, and sets the center as M i , M i-1 , M i+1 For the adjacent left and right positions, the offset is generated by the convolution layer in the y direction:

[0041]

[0042] in is the offset, X is the input, Conv is a 3×3 convolution, GN represents normalization, tanh represents the activation function, and the prediction offset is cumulative to ensure that the convolution kernel conforms to the linear morphology structure:

[0043]

[0044] in Indicates the vertical offset distance of the current grid prediction. Indicates the offset distance of the previous sampling grid; front view and After the dynamic focusing module outputs D 32 and D 64 ;

[0045] In the view conversion application, based on the effective perception of the dynamic focus module, the present invention adopts a two-layer MLP to learn the dependency of all spatial positions between the front view feature map and the BEV feature map, and effectively bridges the information between the two view angles; first, the front view feature map is flattened, and then the relationship between any two pixel positions between the flattened front view feature map and the BEV feature map is learned:

[0046] f t [i] = R i (f t [i],...,f t [j],...,f t [HW])

[0047] where i,j∈[0,HW) is the BEV feature And front view feature map Flatten the index on the dimension, R i The relationship between the i-th pixel in the BEV view feature map and each pixel in the front view feature map is simulated; this design can not only effectively learn the spatial relationship between different views, but also avoid excessive computational burden;

[0048] By fusing features of different scales, the network can capture rich detail information and high-level semantic information. The present invention adopts a two-level feature layer fusion strategy, and the D output by the dynamic focus module is respectively converted into 32 and D 64 The corresponding BEV feature map is fused to obtain more lane feature information:

[0049]

[0050] in and They correspond to the pixel values ​​of the BEV feature map after view conversion, concatenating them according to the channel direction, such as Figure 2 As shown in , after fusion, the BEV feature map H is obtained b ×W b ×C b The dimension is 25×6×512 to achieve effective feature aggregation;

[0051] For the detection head, Figure 5 As shown, n grids are divided along the Z direction, and each grid represents 0.5 meters. First, predict which grid G ​​the lane line is in i In the above example, for the grid G i The lane line in the grid is calculated from its distance Δh to the bottom of the grid. i .

[0052] Step 2: Calculate the loss and back-propagate the optimization model. The implementation is as follows:

[0053] The Z detection offset branch loss in step 1 is the confidence loss Classification Loss and distance loss The lane line confidence branch is a simple binary classification branch used for classification detection of lane line pixels and road background pixels. Each pixel represents a grid unit G i If a G i If there is a lane line passing through, the confidence score is set to 1, otherwise the confidence score is set to 0, and the confidence loss is calculated using binary cross entropy loss. Sum intersection loss To express:

[0054]

[0055] in Expressed as:

[0056]

[0057] where y i represents the true label, p i represents the model prediction probability, and the grid size is set to 0.5m×0.5m;

[0058] Expressed as:

[0059]

[0060] By maximizing the IOU loss through the above formula, |pi ∩y i | represents the intersection between the prediction and the true value, |p i ∪y i | represents the union between the prediction and the true value;

[0061] Category loss, using cross entropy loss:

[0062]

[0063] z i Represents the i-th grid G i The predicted category, Represents i grids G i The real category of

[0064] In order to more accurately represent the position of the lane line, this paper uses the offset branch to predict the precise offset of the lane line in the Z direction. The loss is expressed by the mean square error of the offset:

[0065]

[0066] It should be noted that this paper only calculates the offset for grid cells with positive confidence true values. Indicates whether the i-th grid has a lane line passing through it, σ represents the sigmoid function, Δh i represents the prediction offset, Indicates the deviation from the actual lane line;

[0067] By calculating the loss function, back propagation, and optimizing the model.

[0068] Step 3: Use the OpenLane validation set to input the optimized model in step 2 to detect 3D lane lines. The visualization results are as follows: Figure 6 To verify the effectiveness of the model of the present invention, it is compared with the previous model in different scenarios. The specific acquisition process of the comparative experiment is as follows:

[0069]

[0070] It can be seen that compared with previous models, the performance of the monocular 3D lane detection model we constructed based on geometric prior is the best.

[0071] The specific embodiments described herein are only used to illustrate the spirit and technical solutions of the present invention and do not constitute any limitation on the scope of the present invention. A person skilled in the art of the present invention may make various modifications or supplements to the specific embodiments described or replace them in a similar manner, but they will not deviate from the spirit of the present invention or exceed the scope defined by the attached claims.

Claims

1. A monocular 3D lane detection method based on geometric prior guidance, characterized in that The following steps are involved: Step 1: Obtain lane line dataset and its labels; Step 2: Build a 3D lane detection model including a backbone network, a spatial transformation network combined with geometric priors, an upsampling network, and a detection head; Step 3: Use the optimization algorithm to train the model parameters and adjust the model weights through the back propagation of the loss function; Step 4: Input the test image into the trained model to generate 3D lane detection results.

2. The method for monocular 3D lane detection based on geometric prior guidance according to claim 1, characterized in that: The step 2 at least includes the following steps: The backbone network adopts the ResNet-34 structure to obtain feature maps of different scales. Front view feature map Through the spatial transformation network, combined with the geometric prior information of the lane line, the features are adaptively adjusted and mapped to generate a bird's-eye view (BEV) feature map; The BEV feature map gradually restores high-resolution lane line information through the upsampling network; The up-sampled results are processed into mesh classification through a mesh-based detection head.

3. The monocular 3D lane detection method based on geometric prior guidance according to claim 2, characterized in that: The spatial transformation network includes a dynamic focusing module and a view transformation module; The dynamic focusing module extracts the geometric features of the lane lines through a dynamic deformable convolution kernel, focusing on the slender and tortuous parts of the lane lines; The view conversion module learns the spatial mapping relationship between the front view and the bird's-eye view (BEV) feature map through two layers of MLP, combines geometric priors to guide feature alignment, and reduces computational complexity.

4. The method for monocular 3D lane detection based on geometric prior guidance according to claim 3, characterized in that: The dynamic focusing module adaptively adjusts the convolution offset to focus on the thin and winding parts of the lane line. The specific inference process is as follows: The dynamic focusing module uses a convolution kernel of size M, which is expanded horizontally. i Centered on M i1 , M i+1 For the adjacent left and right positions, the offset is generated by the convolution layer: in is the offset, X is the input, Conv is a 3×3 convolution, GN represents normalization, tanh represents the activation function, and the predicted offset is cumulative to ensure that the convolution shape is consistent with the geometric features of the lane line: in Indicates the vertical offset distance of the current grid prediction. Indicates the offset distance of the previous sampling grid, front view and After the dynamic focusing module outputs D 32 and D 64 .

5. The method for monocular 3D lane detection based on geometric prior guidance according to claim 3, characterized in that: The view conversion network adopts a two-layer MLP structure to effectively learn the pixel-level spatial relationship between the front view feature map and the BEV feature map; the view conversion network guides the learning of feature alignment relationship by combining geometric priors and reduces the computational complexity, including the following steps: Flatten the front-view feature map, and then learn the relationship between any two pixel positions between the flattened front-view feature map and the BEV feature map: f t [i]=R i (f t [i],...,f t [j],...,f t [HW]) where i,j∈[0,HW) is the BEV feature t∈R C×H×W And front view feature map Flatten the index on the dimension, R i The relationship between the i-th pixel in the BEV view feature map and each pixel in the front view feature map is simulated.

6. The method for monocular 3D lane detection based on geometric prior guidance according to claim 3, characterized in that: The converted BEV feature maps are fused in a channel-wise concatenation manner: In the formula and They correspond to the pixel values ​​of the BEV feature map after view conversion respectively. Concate represents feature fusion. The BEV feature maps from different sources are integrated channel by channel to achieve effective feature aggregation and enhance the expression ability of lane line features.

7. The method for monocular 3D lane detection based on geometric prior guidance according to claim 1, characterized in that: The resolution of the BEV feature map is improved through the upsampling network to generate a high-resolution feature map for lane line detection.

8. The method for monocular 3D lane detection based on geometric prior guidance according to claim 1, characterized in that: The detection head performs three-dimensional prediction of the lane line by grid division in the height direction; the detection head predicts the confidence, category and offset of each grid unit through the convolution layer to capture the changes in the spatial position of the lane line.

9. The method for monocular three-dimensional lane detection based on geometric prior guidance according to claim 8, characterized in that: The loss function of the grid unit confidence Using binary cross entropy loss Sum intersection loss Joint Optimization: in Expressed as: where y i represents the true label, p i represents the model prediction probability; Expressed as: where |p i ∩y i | represents the intersection between the prediction and the true value, |p i ∪y i | represents the union between the prediction and the true value; The grid category loss uses cross entropy loss: where z i represents the predicted category of the i-th grid, Represents i grids G i The real category of In order to more accurately represent the position of the lane line, the precise offset of the lane line on each grid in the Z direction is predicted, and its loss is expressed as the mean square error of the offset: in Indicates whether the i-th grid has a lane line passing through it, σ represents the sigmoid function, Δh i represents the prediction offset, Indicates the deviation from the actual lane line.

Images