A lane line detection method and device
By combining the Transformer model and the BEV-Seg method, efficient extraction and segmentation of lane line features are achieved, solving the problem of insufficient accuracy of existing lane line detection methods in complex scenarios and improving the stability and accuracy of detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA AUTOMOTIVE INNOVATION CORP
- Filing Date
- 2024-03-05
- Publication Date
- 2026-07-24
AI Technical Summary
Existing lane detection methods are insufficient in accuracy, especially in complex scenarios such as highways and urban environments where they perform poorly. Traditional methods also suffer from insufficient transfer and generalization capabilities.
A lane detection method based on the Transformer model is adopted. Lane features are extracted and segmented through a feature extraction network, an attention mechanism model, and the BEV-Seg method. Feature fusion is performed using BEV query vectors and self and cross attention mechanisms to achieve pixel-level semantic segmentation and obtain BEV feature maps.
It improves the accuracy and long-distance detection capability of lane lines, effectively solves the lane line detection problem in complex scenarios, and enhances the stability and accuracy of detection results.
Smart Images
Figure CN118247753B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, and more particularly to a lane line detection method and apparatus. Background Technology
[0002] Lane detection is one of the most fundamental and crucial tasks in the field of autonomous driving. With the development of deep learning technology, lane detection techniques have evolved from traditional image processing-based methods to deep learning-based approaches. Autonomous driving encompasses multiple modules, including simulation systems, perception algorithms, localization algorithms, behavior decision-making, control systems, and vehicle-to-everything (V2X) networks. Obtaining highly reliable, stable, and accurate lane perception results is essential for subsequent traffic control tasks, especially for Lane Keeping Assist (LKA). Meanwhile, in the field of natural language understanding, compared to convolutional neural networks, Generative Pre-Trained Transformer (GPT) models and Bidirectional Encoder Representations from Transformers (BERT) have achieved tremendous success, leading to attempts in computer vision to introduce Transformers into perception tasks.
[0003] Currently, lane detection methods used in autonomous driving scenarios include traditional 2D-based methods. These methods are generally conventional, and their algorithms suffer from significant problems in transfer and generalization capabilities. For example, while they can detect lane lines on highways, they fail in urban areas and winding roads. Another approach involves obtaining high-precision lane lines from high-definition maps, but this method has obvious shortcomings. For instance, in most areas, map provider data does not include high-definition map information, thus failing to meet the lane detection requirements of autonomous driving. Furthermore, LiDAR-based detection schemes use radar to obtain ground elevation and directly predict 3D lane lines; however, due to current cost and operational efficiency considerations, this approach has not been widely adopted. Therefore, lane detection remains a significant challenge.
[0004] To address these issues, researchers proposed predicting lane lines on the image and then mapping the 2D detection results to a 3D plane based on the strong assumption that "the ground plane is flat." However, this method suffers from the problem of the assumption not holding, leading to unsatisfactory lane line detection on uphill or downhill slopes. Inspired by monocular depth estimation, researchers also proposed directly predicting 3D lane lines in the camera coordinate system using the monocular forward-looking image, but this detection method performs poorly for lane lines at greater distances. Furthermore, researchers proposed a lane line detection method based on anchors, but this method suffers from issues such as missed detection of short lines and poor detection of curved lines. Summary of the Invention
[0005] This invention provides a lane line detection method and apparatus, which solves the problem of low accuracy of existing lane line detection results.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] In a first aspect, the present invention provides a lane line detection method, the method comprising:
[0008] Acquire the input image captured by the vehicle's forward-facing camera;
[0009] Based on a pre-built feature extraction network model, a backbone feature extraction network based on the Transformer model is used to extract lane line features from the input image to obtain a multi-layer feature map.
[0010] Based on the vehicle's location and the input image, obtain the vehicle's current and previous BEV query vectors; the BEV query vectors include lane line feature information from the input image.
[0011] Based on the pre-built attention mechanism model, the BEV query vectors of the current time and the previous time, as well as the multi-layer feature map, are fused using the attention mechanism to obtain a temporal fusion feature map.
[0012] Based on the pre-constructed BEV lane line detection task header, the temporal fusion feature map is subjected to pixel-level semantic segmentation processing using the BEV-Seg method to obtain the BEV feature map; the BEV feature map includes lane line information.
[0013] In one possible implementation, the attention mechanism model includes a temporal attention mechanism unit, a cross-attention mechanism unit, and a feedforward neural network unit; the fusion processing of the BEV query vectors at the current time step and the previous time step, as well as the multi-layer feature map, based on the attention mechanism to obtain a temporal fusion feature map specifically includes:
[0014] The BEV query vectors at the current time and the previous time are feature aligned at the current time; the feature alignment is to represent the lane line feature information at the current time.
[0015] The first fusion feature is obtained by performing self-attention calculation on the aligned BEV query vectors of the current time and the previous time using the temporal attention mechanism unit.
[0016] The cross-attention mechanism unit performs cross-attention calculation on the multi-layer feature map and the first fused feature to obtain the second fused feature;
[0017] The second fused feature is convolved by the feedforward neural network unit to obtain a temporal fused feature map.
[0018] In one possible implementation, the feature alignment of the BEV query vectors at the current time and the previous time at the current time specifically includes:
[0019] Obtain the positions of the vehicle at the current time and the previous time, and establish a vehicle coordinate system based on the vehicle's position at the current time;
[0020] The positions of the vehicle at the current time and the previous time are represented in the vehicle coordinate system, and the transition matrix for the vehicle to move from the position at the previous time to the position at the current time is determined.
[0021] The lane line feature information from the previous moment is transferred to the current moment using the transition matrix.
[0022] In one possible implementation, the feature extraction network model includes a Backbone feature extraction unit and an FPN feature pyramid unit; the Transformer-based backbone feature extraction network extracts lane line features from the input image to obtain a multi-layer feature map, specifically including:
[0023] The BackBone feature extraction unit extracts lane line information from the input image to obtain a first feature map;
[0024] The lane line information in the first feature map is extracted layer by layer by using the convolution calculation method of the FPN feature pyramid unit to form a multi-layer feature map.
[0025] In one possible implementation, before obtaining the BEV query vector of the vehicle at the current time and the previous time, the method further includes:
[0026] Initialize a set of grid-like learnable parameters Q(H, W, C) as a BEV query vector; where H×W represents the length and width of the BEV grid plane, and C represents the size of the BEV query vector in the channel dimension when used as a feature.
[0027] In one possible implementation, after performing pixel-level semantic segmentation processing on the temporal fusion feature map based on the BEV-Seg method to obtain the BEV feature map, the method further includes:
[0028] The lane line information is clustered to obtain 3D lane lines.
[0029] In a second aspect, the present invention provides a lane line detection device, comprising: an image acquisition module for acquiring an input image captured by a forward-looking camera of a vehicle;
[0030] The image feature extraction module is used to extract lane line features from the input image based on a pre-built feature extraction network model and a backbone feature extraction network based on the Transformer model, to obtain a multi-layer feature map.
[0031] The time-series data acquisition module is used to acquire the BEV query vector of the vehicle at the current time and the previous time based on the vehicle's position and the input image; the BEV query vector includes lane line feature information in the input image;
[0032] The temporal feature fusion module is used to fuse the BEV query vectors of the current time and the previous time, as well as the multi-layer feature map, based on a pre-built attention mechanism model and an attention mechanism, to obtain a temporal fusion feature map.
[0033] The lane line processing module is used to perform pixel-level semantic segmentation processing on the temporal fusion feature map based on the pre-built BEV lane line detection task head and the BEV-Seg method to obtain a BEV feature map; the BEV feature map includes lane line information.
[0034] In one possible implementation, the attention mechanism model includes a temporal attention mechanism unit, a cross-attention mechanism unit, and a feedforward neural network unit; when fusing the BEV query vectors of the current time step and the previous time step, as well as the multi-layer feature map, based on the attention mechanism to obtain a temporal fused feature map, the temporal feature fusion module is configured to perform:
[0035] The BEV query vectors at the current time and the previous time are feature aligned at the current time; the feature alignment is to represent the lane line feature information at the current time.
[0036] The first fusion feature is obtained by performing self-attention calculation on the aligned BEV query vectors of the current time and the previous time using the temporal attention mechanism unit.
[0037] The cross-attention mechanism unit performs cross-attention calculation on the multi-layer feature map and the temporal fusion feature to obtain the second fusion feature;
[0038] The second fused feature is convolved by the feedforward neural network unit to obtain a temporal fused feature map.
[0039] In one possible implementation, when the BEV query vectors of the current time and the previous time are feature aligned at the current time, the temporal feature fusion module is specifically configured to perform:
[0040] Obtain the positions of the vehicle at the current time and the previous time, and establish a vehicle coordinate system based on the vehicle's position at the current time;
[0041] The positions of the vehicle at the current time and the previous time are represented in the vehicle coordinate system, and the transition matrix for the vehicle to move from the position at the previous time to the position at the current time is determined.
[0042] The lane line feature information from the previous moment is transferred to the current moment using the transition matrix.
[0043] In one possible implementation, the feature extraction network model includes a Backbone feature extraction unit and an FPN feature pyramid unit; when the Transformer-based backbone feature extraction network extracts lane line features from the input image to obtain a multi-layer feature map, the image feature extraction module is specifically configured to perform:
[0044] The BackBone feature extraction unit extracts lane line information from the input image to obtain a first feature map;
[0045] The lane line information in the first feature map is extracted layer by layer by using the convolution calculation method of the FPN feature pyramid unit to form a multi-layer feature map.
[0046] In one possible implementation, the lane detection device further includes an initialization module; before acquiring the BEV query vectors of the vehicle at the current and previous times, the initialization module is configured to perform:
[0047] Initialize a set of grid-like learnable parameters Q(H, W, C) as a BEV query vector; where H×W represents the length and width of the BEV grid plane, and C represents the size of the BEV query vector in the channel dimension when used as a feature.
[0048] In one possible implementation, the lane line detection device further includes a lane line processing module. After performing pixel-level semantic segmentation processing on the temporal fusion feature map based on the BEV-Seg method to obtain the BEV feature map, the lane line processing module is configured to perform:
[0049] The lane line information is clustered to obtain 3D lane lines.
[0050] Thirdly, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the lane line detection method described above.
[0051] Fourthly, the present invention provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded by a processor and executed according to any one of the preceding claims of the lane line detection method.
[0052] In practical applications, the lane detection method and apparatus provided in this invention first acquire an environmental image captured by a vehicle's forward-facing camera as the input image. Second, the input image is input into a pre-constructed feature extraction network model. This model extracts lane line features from the input image based on the backbone feature extraction network of the Transformer model, resulting in a multi-layer feature map. Third, based on the vehicle's position and the input image, the vehicle's current and previous BEV query vectors are acquired. The BEV query vector is a BEV grid plane of a preset size, with the center of the grid plane representing the vehicle's position and the grid plane representing an overhead view of the vehicle and its surrounding environment. Finally, the acquired current and previous BEV query vectors, along with the multi-layer feature map, are input into a pre-constructed attention mechanism model. The model uses the attention mechanism of the Transformer model to fuse input information, obtaining a temporal fusion feature map. Finally, the temporal fusion feature map is input into a pre-built BEV lane detection task head. The BEV lane detection task head performs pixel-level semantic segmentation on the temporal fusion feature map using the BEV-Seg method, obtaining a BEV feature map that includes lane line information. This invention uses environmental images captured by the vehicle's forward-facing camera as input images. Since the BEV query vector includes lane line feature information, this invention utilizes the attention mechanism of the Transformer model to fuse the current and previous BEV query vectors, obtaining continuous lane line features. This enables long-distance lane line detection and effectively improves the accuracy of lane line detection results. Attached Figure Description
[0053] Figure 1 A flowchart illustrating the steps of a lane line detection method provided in an embodiment of the present invention;
[0054] Figure 2 A flowchart illustrating the algorithm for a lane line detection method provided in an embodiment of the present invention;
[0055] Figure 3 BEV feature map of a lane line detection method provided in an embodiment of the present invention;
[0056] Figure 4 This is a structural block diagram of a lane line detection device provided in an embodiment of the present invention. Detailed Implementation
[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0058] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, the use of "based on" or "according to" implies openness and inclusiveness, because processes, steps, calculations, or other actions "based on" or "according to" one or more of the stated conditions or values may in practice be based on additional conditions or beyond the stated values.
[0059] To address the issue of low accuracy in existing lane line detection methods, this invention provides a lane line detection method and apparatus.
[0060] like Figure 1 , Figure 2 As shown, in a first aspect, embodiments of the present invention provide a lane line detection method, which is applied to a lane line detection system. The lane line detection system comprises a feature extraction network model 11, an attention mechanism model 12, and a BEV lane line detection task head 13. Specifically, the method includes:
[0061] Step 101: Obtain the input image captured by the vehicle's forward-facing camera.
[0062] The main function of the vehicle's forward-facing camera is to capture road conditions as the vehicle travels, helping the driver better understand the road conditions and driving environment. The forward-facing camera is usually mounted at the front of the car, generally in the center, and takes pictures through a lens on the windshield. To facilitate shooting at night and in low-light conditions, the forward-facing camera is also equipped with infrared imaging equipment and LED lights.
[0063] While the vehicle is in motion, it uses a forward-facing camera to capture real-time images of the environment in front of it and uses these images as input images.
[0064] Step 102: Based on the pre-constructed feature extraction network model 11, the backbone feature extraction network based on the Transformer model extracts lane line features from the input image to obtain a multi-layer feature map.
[0065] The input image is fed into the feature extraction network model 11. In this embodiment, the feature extraction network model 11 is implemented using the backbone feature extraction network of the Transformer model. The lane line features of the input image are extracted through the feature extraction network model 11 to obtain a multi-layer feature map.
[0066] Step 103: Based on the vehicle's location and the input image, obtain the BEV query vector for the vehicle at the current time and the previous time.
[0067] The BEV query vector includes lane line feature information from the input image.
[0068] Specifically, the BEV query vector is a BEV grid plane of a preset size. The BEV grid plane is composed of multiple squares, and the center of the BEV grid plane represents the position of the vehicle.
[0069] The BEV grid plane represents a top-down view of the vehicle and its surrounding preset area, which is related to the preset size of the BEV grid plane.
[0070] In this embodiment, the forward-view camera can only capture environmental images in front of the vehicle. However, once the vehicle starts running, as time changes, the environmental image currently acquired by the forward-view camera can be used as the image in front of the vehicle, and the previously acquired environmental image can be used as the image behind the current vehicle. With the current vehicle position as the center, an overhead view of the current vehicle and its surrounding preset range can be obtained.
[0071] The environmental image acquired by the forward-looking camera contains lane lines. By extracting features from the environmental image, lane line feature information can be obtained. The lane line feature information in the environmental image is represented in the top-down view, and a BEV query vector containing lane line feature information can be obtained.
[0072] Step 104: Based on the pre-built attention mechanism model 12, the BEV query vectors of the current time step and the previous time step, as well as the multi-layer feature map, are fused to obtain the temporal fusion feature map.
[0073] Specifically, the BEV query vectors obtained at the current time and the previous time, as well as the multi-layer feature map, are input into the pre-built attention mechanism model. The attention mechanism model performs attention mechanism calculations on the input BEV query vectors at the current time and the previous time, as well as the multi-layer feature map, based on the attention mechanism, and the calculation result is a temporal fusion feature map.
[0074] Among them, attention mechanism computation includes self-attention computation and cross-attention computation.
[0075] Step 105: Based on the pre-constructed BEV lane line detection task head 13, perform pixel-level semantic segmentation processing on the temporal fusion feature map using the BEV-Seg method to obtain the BEV feature map; the BEV feature map includes lane line information.
[0076] Among them, the BEV-Seg method is a commonly used existing BEV semantic segmentation method. This method uses an additional parsing network to optimize the projection and incomplete BEV segmentation according to the results.
[0077] like Figure 3 As shown, in this embodiment, the BEV feature map output by the BEV lane detection task head 13 is 200×48 in size. According to the special characteristics of the lane detection task, the height and width of each cell in the BEV feature map correspond to 0.5 meters in the real environment where the vehicle is located. That is to say, the resolution of the BEV feature map is 200×48. Specifically, the height is 200 and the width is 48, which correspond to detection distances of 100 meters and 24 meters in the real environment where the vehicle is located, respectively. In other words, the predicted lane line range is 24 meters to the left and right with the vehicle as the origin, and the forward length is 100 meters.
[0078] Lane line information specifically includes instance embedding results, corresponding to Figure 3 The parts a, b, and c in the diagram indicate that lane lines exist in that section.
[0079] Lane line information also includes the offset of the instance location, corresponding to Figure 3 The arrow in section a represents the offset of the instance position, which is the position of the lane line in the horizontal direction, as well as the confidence level of that position.
[0080] Lane information also includes predicted values for lane height.
[0081] In practical applications, the lane detection method provided in this invention first acquires an environmental image captured by the vehicle's forward-looking camera as the input image. Second, the input image is fed into a pre-constructed feature extraction network model. This model extracts lane line features from the input image based on the backbone feature extraction network of the Transformer model, resulting in a multi-layer feature map. Third, based on the vehicle's position and the input image, the vehicle's current and previous BEV query vectors are obtained. The BEV query vector is a BEV grid plane of a preset size, with the center of the BEV grid plane representing the vehicle's position. The V-grid plane represents an overhead view of the vehicle and its surrounding environment. Next, the acquired BEV query vectors from the current and previous times, along with multi-layer feature maps, are input into a pre-built attention mechanism model. This model, based on the Transformer model's attention mechanism, fuses the input information to obtain a temporal fusion feature map. Finally, the temporal fusion feature map is input into a pre-built BEV lane detection task head. This head performs pixel-level semantic segmentation on the temporal fusion feature map using the BEV-Seg method to obtain a BEV feature map, which includes lane line information.
[0082] This invention uses the environmental image captured by the vehicle's forward-looking camera as the input image. Since the BEV query vector includes lane line feature information, this invention utilizes the attention mechanism of the Transformer model to perform feature fusion on the BEV query vector obtained at the current time and the previous time, which can obtain continuous lane line features, thereby enabling long-distance detection of lane lines and effectively improving the accuracy of lane line detection results.
[0083] like Figure 2 As shown, the attention mechanism model further includes a temporal attention mechanism unit 121, a cross-attention mechanism unit 122, and a feedforward neural network unit 123.
[0084] Based on an attention mechanism, the BEV query vectors from the current time step and the previous time step, as well as the multi-layer feature map, are fused to obtain a temporal fusion feature map, specifically including:
[0085] The BEV query vectors at the current time step and the previous time step are feature aligned at the current time step.
[0086] Feature alignment involves representing the lane line feature information from the previous moment in the current moment. In other words, a vehicle coordinate system is established with the current vehicle position as the center, and the positions of the vehicle and lane line feature information from the previous moment are represented in the current vehicle coordinate system.
[0087] The first fusion feature is obtained by performing self-attention calculation on the aligned BEV query vectors of the current time step and the previous time step through the temporal attention mechanism unit 121.
[0088] Specifically, after aligning the lane line feature information of the current time and the previous time, the BEV query vectors of the current time and the previous time are input into the temporal attention mechanism unit 121.
[0089] The temporal attention mechanism unit 121 performs self-attention calculation on the BEV query vectors of the current time step and the previous time step, and uses the result of the self-attention calculation as the first fusion feature.
[0090] The second fusion feature is obtained by performing cross-attention calculation on the multi-layer feature map and the first fusion feature through the cross-attention mechanism unit 122.
[0091] Specifically, the multi-layer feature map and the first fused feature are input into the cross-attention mechanism unit 122. The cross-attention mechanism unit 122 performs cross-attention calculation on the multi-layer feature map and the first fused feature, and the result of the cross-attention calculation is used as the second fused feature.
[0092] The formula for calculating cross-attention is as follows:
[0093] O = Softmax(Q × K) T )×V;
[0094] Where O represents the second fusion feature, K and V are the input values obtained after linear transformation of the multi-layer feature map, K is the KEY of the cross-attention mechanism, V is the VALUE of the cross-attention mechanism, and Q represents the first fusion feature.
[0095] The second fusion feature is convolved by the feedforward neural network unit 123 to obtain the temporal fusion feature map.
[0096] Among them, the feedforward neural network is the simplest type of neural network. The neurons are arranged in layers, and each neuron is only connected to the neurons in the previous layer, receiving the output of the previous layer and outputting it to the neurons in the next layer.
[0097] In this embodiment, the second fused feature is input into the feedforward neural network unit 123, and the feedforward neural network unit 123 performs convolution calculation on the second fused feature to obtain a temporal fused feature map.
[0098] exist Figure 2In this context, TEMPORAL ATTENTION represents the temporal attention mechanism unit 121, CROSS ATTENTION represents the cross attention mechanism unit 122, and Feed Forward represents the feedforward neural network unit 123.
[0099] Furthermore, feature alignment is performed on the BEV query vectors from the current time step and the previous time step at the current time step, specifically including:
[0100] Obtain the vehicle's position at the current moment and the previous moment, and establish the vehicle's coordinate system based on the vehicle's position at the current moment.
[0101] Specifically, during vehicle operation, the vehicle's onboard position sensors, combined with a high-precision map, can accurately obtain the vehicle's position information at the current and previous moments. Then, the vehicle's coordinate system is established with the current position of the vehicle as the origin.
[0102] Represent the vehicle's current and previous positions in the vehicle coordinate system, and determine the transition matrix for the vehicle to move from its previous position to its current position.
[0103] Specifically, since the positions of the vehicle at the current moment and the previous moment are known, and the vehicle's coordinate system at the current moment is known, the position of the vehicle at the previous moment can be represented at the current moment.
[0104] Based on the current vehicle position and the previous vehicle position in the vehicle coordinate system, the vehicle's transition matrix can be calculated.
[0105] The lane line feature information from the previous time step is transferred to the current time step using a transition matrix.
[0106] Specifically, the product of the lane line feature information from the previous moment and the transition matrix represents the lane line feature information from the previous moment at the current moment.
[0107] The vehicle coordinate system is a three-dimensional coordinate system, including the x-axis, y-axis, and z-axis. The x-axis component represents the vehicle's forward displacement, the y-axis component represents the vehicle's left or right displacement, and the z-axis component represents the vehicle's uphill or downhill displacement along the road surface. A z-axis component of 0 indicates the vehicle is traveling on a flat road, while a non-zero z-axis component indicates the vehicle is going uphill or downhill.
[0108] In this embodiment, by performing self-attention calculation on the BEV query vectors at the current time and the previous time, and then fusing the calculation results of the self-attention mechanism with the multi-layer feature map through a cross-attention mechanism, the expressive power of lane line features can be effectively improved, thereby effectively improving the detection capability of lane lines.
[0109] Furthermore, the feature extraction network model includes a Backbone feature extraction unit 111 and an FPN feature pyramid unit 112; the Transformer-based backbone feature extraction network extracts lane line features from the input image to obtain multi-layer feature maps, specifically including:
[0110] The BackBone feature extraction unit 111 extracts lane line information from the input image to obtain the first feature map.
[0111] The lane line information in the first feature map is extracted layer by layer by using the convolution calculation method of FPN feature pyramid unit 112 to form a multi-layer feature map.
[0112] exist Figure 2 In this diagram, BACKBONE represents the BackBone backbone feature extraction unit 111, and FPN represents the FPN feature pyramid unit 112. The input image acquired by the forward-looking camera is processed by the BackBone backbone feature extraction unit 111 to obtain a first feature map containing lane line information. The first feature map is then input into the FPN feature pyramid unit 112 for target detection at different scales to obtain multi-layer feature maps.
[0113] In this embodiment, an intermediate layer of Feature Extraction Network (FPN) is established through a convolutional neural network, and the final output multi-layer feature map consists of 4 layers.
[0114] Furthermore, before obtaining the BEV query vectors for the vehicle at the current and previous moments, the method also includes:
[0115] Initialize a set of raster-like learnable parameters Q(H, W, C) as the BEV query vector.
[0116] H×W represents the length and width of the BEV grid plane, and C represents the size of the BEV query vector in the channel dimension when it is used as a feature.
[0117] The BEV query vector, or BEV Queries, has H and W dimensions that are consistent with the grid size of the BEV grid plane in the x and y directions. Therefore, it inherits the characteristic that the BEV grid plane is explicitly associated with the horizontal and vertical physical space surrounding the vehicle. However, C is the dimension of the BEV Queries as a feature in the channel dimension and does not explicitly correspond to the physical space size of the BEV grid plane along the z-axis. Q(p)(1, c) located at p = (x, y) is responsible for querying the corresponding grid cell region of the BEV grid plane. Each grid cell region of the BEV grid plane represents s meters in the actual vehicle environment.
[0118] Furthermore, after performing pixel-level semantic segmentation on the temporal fusion feature map based on the BEV-Seg method to obtain the BEV feature map, the method also includes:
[0119] The lane line information is clustered to obtain 3D lane lines.
[0120] In this embodiment, clustering is performed based on the geometric features of lane lines, such as shape, length, and curvature, to classify and group lane lines. Common methods include curvature-based clustering and straight-segment length-based clustering.
[0121] In this embodiment, in order to maintain information flow while avoiding gradient vanishing or exploding problems, thereby improving the training efficiency and performance of the model, Add&Norm layers are provided between the temporal attention mechanism unit 121 and the cross attention mechanism unit 122, between the cross attention mechanism unit 122 and the feedforward neural network unit 123, and between the feedforward neural network unit 123 and the BEV lane line detection task head.
[0122] The Add&Norm layer adds the output and input of the previous layer and performs normalization, then uses the result as the input of the next layer.
[0123] In practical applications, the lane detection method provided by this invention utilizes high-precision maps as training data for the model. This not only eliminates the high manpower and time costs associated with existing LiDAR-based lane line acquisition and labeling, but also outputs BEV feature maps that directly obtain BEV lane lines based on the vehicle body, thus obtaining instance lane lines in an end-to-end manner. This effectively improves the efficiency and accuracy of lane detection.
[0124] like Figure 2 As shown, in a second aspect, embodiments of the present invention also provide a lane line detection device, comprising:
[0125] Image acquisition module 201 is used to acquire input images captured by the vehicle's forward-facing camera;
[0126] The image feature extraction module 202 is used to extract lane line features from the input image based on the backbone feature extraction network of the Transformer model according to the pre-built feature extraction network model, and obtain a multi-layer feature map.
[0127] The time-series data acquisition module 203 is used to acquire the BEV query vector of the vehicle at the current time and the previous time based on the vehicle's position and the input image; the BEV query vector includes lane line feature information in the input image;
[0128] The temporal feature fusion module 204 is used to fuse the BEV query vectors of the current time step and the previous time step, as well as the multi-layer feature map, based on the pre-built attention mechanism model and the attention mechanism to obtain the temporal fusion feature map.
[0129] The lane line processing module 205 is used to perform pixel-level semantic segmentation processing on the temporal fusion feature map based on the BEV-Seg method according to the pre-built BEV lane line detection task head to obtain the BEV feature map; the BEV feature map includes lane line information.
[0130] Furthermore, the attention mechanism model includes a temporal attention mechanism unit, a cross-attention mechanism unit, and a feedforward neural network unit; when fusing the BEV query vectors from the current time step and the previous time step, as well as the multi-layer feature map, based on the attention mechanism to obtain a temporal fused feature map, the temporal feature fusion module 204 is configured to execute:
[0131] The BEV query vectors from the current time step and the previous time step are feature aligned at the current time step; feature alignment means representing the lane line feature information from the previous time step at the current time step.
[0132] The first fusion feature is obtained by performing self-attention calculation on the aligned BEV query vectors of the current time step and the previous time step through the temporal attention mechanism unit.
[0133] The second fused feature is obtained by performing cross-attention calculation on the multi-layer feature map and the temporal fusion feature through the cross-attention mechanism unit;
[0134] The second fusion feature is convolved by a feedforward neural network unit to obtain a temporal fusion feature map.
[0135] Furthermore, when performing feature alignment on the BEV query vectors from the current time step and the previous time step at the current time step, the temporal feature fusion module 204 is specifically configured to perform:
[0136] Obtain the vehicle's position at the current moment and the previous moment, and establish the vehicle's coordinate system based on the vehicle's position at the current moment;
[0137] Represent the vehicle's position at the current time and the previous time in the vehicle coordinate system, and determine the transition matrix for the vehicle to move from the previous time position to the current time position;
[0138] The lane line feature information from the previous time step is transferred to the current time step using a transition matrix.
[0139] Furthermore, the feature extraction network model includes a Backbone feature extraction unit and an FPN feature pyramid unit; when the Transformer-based backbone feature extraction network extracts lane line features from the input image to obtain multi-layer feature maps, the image feature extraction module 202 is specifically configured to perform:
[0140] The first feature map is obtained by extracting lane line information from the input image through the BackBone backbone feature extraction unit;
[0141] The lane line information in the first feature map is extracted layer by layer by using convolution calculation method through the FPN feature pyramid unit to form a multi-layer feature map.
[0142] Furthermore, the lane detection device also includes an initialization module; before acquiring the vehicle's current and previous BEV query vectors, the initialization module is configured to execute:
[0143] Initialize a set of learnable grid-like parameters Q(H, W, C) as the BEV query vector; H×W represents the length and width of the BEV grid plane, and C represents the size of the BEV query vector in the channel dimension when used as a feature.
[0144] Furthermore, the lane detection device also includes a lane processing module 205. After performing pixel-level semantic segmentation processing on the temporal fusion feature map based on the BEV-Seg method to obtain the BEV feature map, the lane processing module 205 is configured to execute:
[0145] The lane line information is clustered to obtain 3D lane lines.
[0146] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0147] Thirdly, embodiments of the present invention also provide an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the lane detection method in the embodiments of the present invention.
[0148] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the lane detection method in the embodiments of the present invention.
[0149] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).
[0150] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A lane line detection method, characterized in that, include: Acquire the input image captured by the vehicle's forward-facing camera; Based on a pre-built feature extraction network model, a backbone feature extraction network based on the Transformer model is used to extract lane line features from the input image to obtain a multi-layer feature map. Based on the vehicle's position and the input image, obtain the vehicle's current and previous BEV query vectors; the BEV query vector is a BEV grid plane of a preset size, the BEV grid plane is composed of multiple squares, the center of the BEV grid plane represents the vehicle's position, and the BEV query vector includes lane line feature information from the input image. Based on the pre-built attention mechanism model, the BEV query vectors of the current time and the previous time, as well as the multi-layer feature map, are fused using the attention mechanism to obtain a temporal fusion feature map. Based on the pre-constructed BEV lane line detection task head, the temporal fusion feature map is subjected to pixel-level semantic segmentation processing based on the BEV-Seg method to obtain the BEV feature map; the BEV feature map includes lane line information. The attention mechanism model includes a temporal attention mechanism unit, a cross-attention mechanism unit, and a feedforward neural network unit; the process of fusing the BEV query vectors of the current time step and the previous time step, as well as the multi-layer feature map, based on the attention mechanism to obtain a temporal fused feature map specifically includes: The BEV query vectors at the current time and the previous time are feature aligned at the current time; the feature alignment is to represent the lane line feature information at the current time. The first fusion feature is obtained by performing self-attention calculation on the aligned BEV query vectors of the current time and the previous time using the temporal attention mechanism unit. The cross-attention mechanism unit performs cross-attention calculation on the multi-layer feature map and the first fused feature to obtain the second fused feature; The second fused feature is convolved by the feedforward neural network unit to obtain a temporal fused feature map.
2. The method according to claim 1, characterized in that, The step of aligning the BEV query vectors at the current time and the previous time with features at the current time specifically includes: Obtain the positions of the vehicle at the current time and the previous time, and establish a vehicle coordinate system based on the vehicle's position at the current time; The positions of the vehicle at the current time and the previous time are represented in the vehicle coordinate system, and the transition matrix for the vehicle to move from the position at the previous time to the position at the current time is determined. The lane line feature information from the previous moment is transferred to the current moment using the transition matrix.
3. The method according to claim 1, characterized in that, The feature extraction network model includes a BackBone backbone feature extraction unit and an FPN feature pyramid unit; The backbone feature extraction network based on the Transformer model extracts lane line features from the input image to obtain a multi-layer feature map, specifically including: The BackBone feature extraction unit extracts lane line information from the input image to obtain a first feature map; The lane line information in the first feature map is extracted layer by layer by using the convolution calculation method of the FPN feature pyramid unit to form a multi-layer feature map.
4. The method according to claim 1, characterized in that, Before obtaining the BEV query vectors of the vehicle at the current time and the previous time, the method further includes: Initialize a set of grid-like learnable parameters Q(H, W, C) as a BEV query vector; the BEV query vector is a BEV grid plane of a preset size, where H×W represents the length and width of the BEV grid plane, and C represents the size of the BEV query vector in the channel dimension when used as a feature.
5. The method according to claim 1, characterized in that, After performing pixel-level semantic segmentation processing on the temporal fusion feature map based on the BEV-Seg method to obtain the BEV feature map, the method further includes: The lane line information is clustered to obtain 3D lane lines.
6. A lane line detection device, characterized in that, include: The image acquisition module is used to acquire the input image captured by the vehicle's forward-facing camera; The image feature extraction module is used to extract lane line features from the input image based on a pre-built feature extraction network model and a backbone feature extraction network based on the Transformer model, to obtain a multi-layer feature map. The time-series data acquisition module is used to acquire the BEV query vector of the vehicle at the current time and the previous time based on the vehicle's position and the input image; the BEV query vector is a BEV grid plane of a preset size, the BEV grid plane is composed of multiple squares, the center of the BEV grid plane represents the vehicle's position, and the BEV query vector includes lane line feature information from the input image. The temporal feature fusion module is used to fuse the BEV query vectors of the current time and the previous time, as well as the multi-layer feature map, based on a pre-built attention mechanism model and an attention mechanism, to obtain a temporal fusion feature map. The lane line processing module is used to perform pixel-level semantic segmentation processing on the temporal fusion feature map based on the BEV-Seg method according to the pre-constructed BEV lane line detection task head, so as to obtain a BEV feature map; the BEV feature map includes lane line information. The attention mechanism model includes a temporal attention mechanism unit, a cross-attention mechanism unit, and a feedforward neural network unit; when fusing the BEV query vectors of the current time step and the previous time step, as well as the multi-layer feature map, based on the attention mechanism to obtain a temporal fused feature map, the temporal feature fusion module is configured to execute: The BEV query vectors at the current time and the previous time are feature-aligned at the current time; The feature alignment is to represent the lane line feature information from the previous time step at the current time step; The first fusion feature is obtained by performing self-attention calculation on the aligned BEV query vectors of the current time and the previous time using the temporal attention mechanism unit. The cross-attention mechanism unit performs cross-attention calculation on the multi-layer feature map and the temporal fusion feature to obtain the second fusion feature; The second fused feature is convolved by the feedforward neural network unit to obtain a temporal fused feature map.
7. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the lane detection method as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the lane detection method as described in any one of claims 1-5.