3D lane line detection method and system fusing image features and traffic target semantics

Abstract

The application discloses a 3D lane line detection method and system fusing image features and traffic target semantics, and the method is based on Gen-LaneNet. Road semantic information in an image is obtained by using a segmentation network for the acquired image. The acquired image and the road semantic information are respectively input into a down-sampling network for processing, and road semantic information features and image features are obtained. A fusion sub-network is used for fusing the image features and the road semantic information, and fused features are obtained. The fused features are projected into a virtual top view, and a lane line in the top view space is predicted by a lane line detection head network. The lane line in the top view space is subjected to geometric transformation to obtain a lane line in a 3D space. A feature fusion module is used for integrating semantic features and visual features, a top view lane line is predicted by using a fused feature map, and real three-dimensional lane points are directly obtained through geometric projection. The lane line prediction of an image far end is more accurate, and the prediction accuracy is improved by nearly one time compared with that without using.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving, specifically relating to a 3D lane line detection method and system that integrates image features and traffic target semantics. Background Technology

[0002] Lane detection has become a significant issue in the field of autonomous driving in recent years. As an important part of intelligent driver assistance, accurate lane line recognition plays a crucial role in advanced driving systems such as Lane Departure Warning (LDW), Blind Spot Monitoring (BSM), and Adaptive Cruise Control (ACC).

[0003] Most lane detection methods treat lane detection as a 2D lane segmentation task. To convert the 2D detection results to 3D space, inverse perspective mapping (IPM) is usually used as a post-processing step. However, real-world traffic scenarios often involve uphill and downhill sections, making this approach impractical in real-world traffic environments.

[0004] In the field of 3D lane perception, inspired by the success of CNNs in monocular depth estimation, 3D-LaneNet designed an end-to-end framework that unifies image encoding, top-view transformation, and 3D curve extraction, directly predicting 3D lane lines from the front view. However, the end-to-end framework is greatly affected by visual variations. Therefore, Gen-LaneNet decouples the image segmentation and 3D feature extraction subnetworks, forming a two-stage subnetwork. In the first stage, the input image is encoded, and then the fused features are decoded into a lane segmentation map. The second stage uses 3D-GeoNet to project the segmentation map onto a virtual top view and predict the lane using a lane detection head, outputting points that directly yield the real 3D lane lines through geometric transformation. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a 3D lane detection method that integrates image features and traffic target semantics. Based on Gen-LaneNet, a segmentation network is used to obtain road semantic information in the image. The input image features and road semantic information are encoded separately and then fused using a fusion sub-network. By fusing the embedded semantic features and visual features, valuable relationships between objects around the lane lines and the lane lines can be effectively mined, resulting in better prediction results.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is: a 3D lane line detection method that integrates image features and traffic target semantics, comprising the following steps: based on Gen-LaneNet, a segmentation network is used to obtain road semantic information in the acquired image;

[0007] The acquired image and road semantic information are respectively input into the downsampling network for processing to obtain road semantic information features and image features. The image features and road semantic information are then fused using a fusion sub-network to obtain the fused features.

[0008] The fused features are projected onto a virtual top view, and the lane line detection head network predicts the lane lines in the top view space.

[0009] The lane lines in the top-view space are geometrically transformed to obtain the lane lines in the 3D space.

[0010] Furthermore, when using segmentation networks to obtain road semantic information in images, two visual semantic segmentation networks, FCN and DeepLabV3+, are employed. The semantic segmentation network classifies each pixel in the image based on the ResNet50 backbone network, generating image masks segmented by category. The road object masks obtained from the segmentation are then used as road semantic information for analyzing and understanding the scene.

[0011] Furthermore, the downsampling network consists of several convolutional networks, and the input image is... The tensor is obtained through a downsampling network. The feature tensor.

[0012] Furthermore, when fusing semantic and visual features, the semantic and visual features are added element-wise on a tensor basis; or the semantic features are used as attention weights and multiplied element-wise with the visual features, using semantic information to adjust the influence of the visual features.

[0013] Furthermore, the fused features are projected onto the virtual top view, and the lane line detection head network predicts the lane lines in the top view space. This process includes: processing the fused features through an upsampling network to obtain a feature tensor; projecting the feature tensor onto the virtual top view through inverse perspective mapping; and then outputting the lane lines in the top view space by the lane line detection head network.

[0014] Furthermore, the output of the first-stage fusion subnet in the lane detection head network A 32-dimensional tensor is projected onto a top-view space tensor via IPM. After several pooling and convolutional layers, the result is... The output tensor of 128 is obtained Each lane line parameter is represented.

[0015] Furthermore, when performing a geometric transformation on the lane lines in the top-view space to obtain the lane lines in 3D space, the 3D lane line points in the vehicle coordinate system... Points in the top-view coordinate system that have been transformed by projection to 2D image pixels (u,v) The transformation from homography matrix to the same 2D image pixel (u,v) can be expressed by the following formula:

[0016]

[0017] in, Let be a rotation matrix. For transformation vectors, For camera internal parameters,

[0018]

[0019] in, For the camera's tilt angle, For camera height; use s and c to represent it. , have to:

[0020]

[0021] The following system of equations is obtained by breaking down the calculations:

[0022]

[0023] right and Further processing yielded:

[0024]

[0025]

[0026] Will Substituting into the system of equations, we get:

[0027]

[0028] This allows points in the top-view space to be directly converted to points in the 3D space through this geometric transformation.

[0029] Based on the above-mentioned concept, this invention provides a 3D lane line detection system that integrates image features and traffic target semantics, including a feature acquisition module, a feature fusion module, a top-view lane line acquisition module, and a geometric transformation module.

[0030] The feature acquisition module is based on Gen-LaneNet. It uses a segmentation network to obtain road semantic information from the acquired image. The acquired image and road semantic information are then input into a downsampling network for processing to obtain road semantic information features and image features.

[0031] The feature fusion module uses a fusion sub-network to fuse image features and road semantic information to obtain fused features;

[0032] The top-view lane line acquisition module projects the fused features into the virtual top view, and the lane line detection head network predicts the lane lines in the top view space.

[0033] The geometric transformation module is used to perform geometric transformations on the lane lines in the top-view space to obtain lane lines in the 3D space.

[0034] The present invention also provides a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the 3D lane line detection method of the present invention that integrates image features and traffic target semantics when executing the program.

[0035] A computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it can realize the 3D lane line detection method that fuses image features and traffic target semantics as described in this invention.

[0036] Compared with the prior art, the present invention has at least the following beneficial effects:

[0037] This invention integrates semantic information from road traffic objects into a deep learning-based 3D visual lane detection model. It integrates the semantic mask of moving road objects into a new branch and uses a feature fusion module to integrate semantic and visual features. The fused feature map is used to predict top-view lane lines, and the true 3D lane points are directly obtained through geometric projection. Testing shows that the method described in this invention improves the overall lane line prediction accuracy on the Apollo dataset; lane line prediction at the far end of the image is more accurate, nearly doubling the prediction accuracy compared to the unused method. The proposed fusion mechanism is effective and versatile for 3D lane detection, and the proposed target segmentation algorithm can be successfully applied in practical applications.

[0038] In the image segmentation sub-network of this invention, RGB images and vehicle masks should be simultaneously input into the encoder. Embedding the vehicle mask can extract valuable information from the detected surrounding objects. Furthermore, this invention provides different segmentation networks to obtain road semantic information for auxiliary detection, and uses two different fusion methods to fuse semantic features and visual features. Attached Figure Description

[0039] Figure 1 This is the two-stage framework of the present invention.

[0040] Figure 2 This paper focuses on semantic mask extraction and its impact on lane detection. Since cars obstruct the leftmost lane line in a 2D image, direct prediction based solely on the image leads to significant long-range errors. However, this invention, which fuses road semantic information, achieves accurate lane line prediction, demonstrating the auxiliary role of object semantics.

[0041] Figure 3 This relates to the representation (anchor) and geometric projection principles of lane lines.

[0042] Figure 4 This is a specific component of a one-stage downsampling network.

[0043] Figure 5 This is a specific component of a one-stage upsampling network.

[0044] Figure 6 This is a specific component of the two-stage lane line detection head network.

[0045] Figure 7 The comparison shows the prediction results of existing methods and the present invention, where the red line represents the predicted lane lines and the blue line represents the actual lane lines on the ground. Detailed Implementation

[0046] like Figure 1 As shown, the present invention comprises two stages and four steps.

[0047] Phase 1: Merging subnets.

[0048] The first step, in order to obtain road semantic information in the image, is to use two mainstream visual semantic segmentation methods: FCN and DeepLabV3+. Figure 2 (a) A semantic segmentation network. The semantic segmentation network uses the ResNet50 backbone network to classify each pixel in the image, thereby generating image masks segmented by category. This invention mainly utilizes the segmented road object masks as road semantic information for further analysis and understanding of the scene.

[0049] The second step, in Figure 2 In the fusion network of (a), the road semantic information obtained in the first step and the acquired image are simultaneously fed into a downsampling network composed of several convolutional networks to obtain road semantic information features and image features, respectively. The specific structure of the downsampling network is as follows: Figure 4 As shown, the input image is The tensor is obtained through a downsampling network. The feature tensor. To fuse semantic and visual features, this invention employs two methods: one is to add semantic and visual features element-wise across the tensor; the other is to use semantic features as attention weights, multiplying them element-wise with visual features to adjust the influence of visual features using semantic information; specifically:

[0050] remember For image feature tensors The Okay, number Column elements, For road semantic information feature tensor The Okay, number Column elements, , The width and height of the image.

[0051] Addition and fusion:

[0052]

[0053] Multiplication and fusion:

[0054]

[0055] in, This indicates element-wise multiplication.

[0056] Next, the fused features are processed by... Figure 2 (b) consists of several deconvolutional networks, the specific structure of which is as follows: Figure 5 As shown, the input image is The tensor is obtained through a downsampling network. A feature tensor of 32.

[0057] The second stage is the 3D geometric subnet.

[0058] The third step, Figure 2 (c) First, the decoded features are projected onto the virtual top view using inverse perspective mapping (IPM). Then, a lane detection head network consisting of several pooling convolutional layers predicts the lane lines in the top view space. The specific composition of the lane detection head network is as follows: Figure 6 As shown, the output of the first-stage fusion subnet A 32-dimensional tensor is projected onto a top-view space tensor via IPM. After several pooling and convolutional layers, the result is... The output tensor of 128 is thus obtained. Each lane line parameter is represented.

[0059] at last, Figure 1 (d) and Figure 3The geometric transformation in (a) directly projects the lane line points from the top-view space into 3D space, resulting in lane line points in 3D space. The specific geometric transformation is as follows:

[0060] like Figure 3 (b) shows the 3D lane line points in the vehicle coordinate system. Points in the top-view coordinate system that have been transformed by projection to 2D image pixels (u,v) The transformation to the same 2D image pixel (u,v) through the homography matrix is ​​expressed by the following formula.

[0061]

[0062] in, Let be a rotation matrix. For transformation vectors, This refers to the camera's internal parameters.

[0063]

[0064] in, For the camera's tilt angle, This is the camera height.

[0065] Replace with s and c , have to:

[0066]

[0067] The following system of equations is obtained by breaking down the calculations:

[0068]

[0069] Simplifying the third expression in the system of equations yields...

[0070]

[0071] Substituting into the second equation, we get:

[0072]

[0073]

[0074] Will Substituting into the system of equations, we get:

[0075]

[0076] This allows points in the top-view space to be directly converted to points in 3D space through this geometric transformation.

[0077] The fused feature maps are used to predict top-view lane lines, and the true 3D lane points are directly obtained through geometric projection. In long-range detection, such as... Figure 7 As shown, the present invention provides more accurate predictions than the original model. Furthermore, various object segmentation algorithms can be successfully applied in practical applications.

[0078] With the same technical concept as the method described above, this invention provides a 3D lane line detection system that integrates image features and traffic target semantics, including a feature acquisition module, a feature fusion module, a top-view lane line acquisition module, and a geometric transformation module;

[0079] The feature acquisition module is based on Gen-LaneNet. It uses a segmentation network to obtain road semantic information from the acquired image. The acquired image and road semantic information are then input into a downsampling network for processing to obtain road semantic information features and image features.

[0080] The feature fusion module uses a fusion sub-network to fuse image features and road semantic information to obtain fused features;

[0081] The top-view lane line acquisition module projects the fused features into the virtual top view, and the lane line detection head network predicts the lane lines in the top view space.

[0082] The geometric transformation module is used to perform geometric transformations on the lane lines in the top-view space to obtain lane lines in the 3D space.

[0083] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the 3D lane line detection method of the present invention that integrates image features and traffic target semantics when executing the computer executable program.

[0084] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the 3D lane line detection method of the present invention that integrates image features and traffic target semantics.

[0085] The computer equipment may be a laptop, desktop computer, vehicle-mounted computer, or workstation.

[0086] The processor described in this invention may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0087] The memory described in this invention can be an internal storage unit of a laptop, desktop computer, vehicle computer, or workstation, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

[0088] Computer-readable storage media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0089] Finally, it should be noted that the above description is only for illustrating specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that any modifications or variations made based on the technical solutions and inventive concepts of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A 3D lane line detection method that integrates image features and traffic target semantics, characterized in that, Includes the following steps: Based on Gen-LaneNet, a segmentation network is used to obtain road semantic information in the acquired images; The acquired image and road semantic information are respectively input into the downsampling network for processing to obtain road semantic information features and image features. The image features and road semantic information are then fused using a fusion sub-network to obtain the fused features. The fused features are projected onto a virtual top view, and the lane line detection head network predicts the lane lines in the top view space. Specifically, the fused features are processed by an upsampling network to obtain a feature tensor, the feature tensor is projected onto the virtual top view through inverse perspective mapping, and then the lane line detection head network predicts the lane lines in the top view space. Perform a geometric transformation on the lane lines in the top-view space to obtain the lane lines in the 3D space. 2.The 3D lane line detection method of fusing image features and traffic target semantics according to claim 1, characterized in that, When using segmentation networks to obtain road semantic information in images, two visual semantic segmentation networks, FCN and DeepLabV3+, are employed. The semantic segmentation network classifies each pixel in the image based on the ResNet50 backbone network, generating image masks segmented by category. The road object masks obtained from the segmentation are then used as road semantic information for analyzing and understanding the scene.

3. The 3D lane line detection method fusing image features and traffic target semantics according to claim 1, characterized in that, The down-sampling network is composed of several convolutional networks, the input image is a tensor of , and a feature tensor of is obtained after the down-sampling network. 4.The 3D lane line detection method of fusing image features and traffic target semantics according to claim 3, characterized in that, When fusing semantic and visual features, the semantic and visual features are added element-wise on a tensor basis; or the semantic features are used as attention weights and multiplied element-wise with the visual features, using semantic information to adjust the influence of the visual features.

5. The 3D lane line detection method of fusing image features and traffic target semantics according to claim 1, characterized in that, Output of the first-stage fusion subnet in the lane detection head network A 32-dimensional tensor is projected onto a top-view space tensor via IPM. After several pooling and convolutional layers, the result is... The output tensor of 128 is obtained Each lane line parameter is represented.

6. The 3D lane line detection method fusing image features and traffic target semantics according to claim 1, characterized in that, When performing a geometric transformation on the lane lines in top-view space to obtain lane lines in 3D space, the 3D lane line points in the vehicle coordinate system... Points in the top-view coordinate system that have been transformed by projection to 2D image pixels (u,v) The transformation from homography matrix to the same 2D image pixel (u,v) can be expressed by the following formula: wherein, is a rotation matrix, is a translation vector, is a camera intrinsic, wherein is the camera tilt angle, is the camera height; replaced by s, c , resulting in: The following system of equations is obtained by breaking down the calculations: To and Further processing gives: Will Substituting into the system of equations, we get: This allows points in the top-view space to be directly converted to points in the 3D space through this geometric transformation.

7. A 3D lane marking detection system that fuses image features with traffic object semantics, characterized in that, It includes a feature acquisition module, a feature fusion module, a top-view lane line acquisition module, and a geometric transformation module; The feature acquisition module is based on Gen-LaneNet. It uses a segmentation network to obtain road semantic information from the acquired image. The acquired image and road semantic information are then input into a downsampling network for processing to obtain road semantic information features and image features. The feature fusion module uses a fusion sub-network to fuse image features and road semantic information to obtain fused features; Specifically, this includes: processing the fused features through an upsampling network to obtain a feature tensor; projecting the feature tensor onto a virtual top view through an inverse perspective mapping; and then having the lane line detection head network predict the lane lines in the top view space. The top-view lane line acquisition module projects the fused features into the virtual top view, and the lane line detection head network predicts the lane lines in the top view space. The geometric transformation module is used to perform geometric transformations on the lane lines in the top-view space to obtain lane lines in the 3D space.

8. A computer device, comprising: The device includes a processor and a memory, the memory being used to store a computer-executable program, the processor reading the computer-executable program from the memory and executing it, and the processor executing the program being able to implement the 3D lane line detection method of any one of claims 1-6, which integrates image features and traffic target semantics.

9. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, enables the 3D lane detection method according to any one of claims 1-6, which integrates image features and traffic target semantics.

Images