Lane line detection method and related device
By combining cross-granularity adaptive convolutional networks and prior knowledge of lane line continuity, the problem of instability and false detection in lane line detection methods under complex scenarios is solved, achieving lane line detection with higher accuracy and robustness.
Patent Information
- Application Number
- CN202511366411.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-23
- Publication Date
- 2025-12-12
AI Technical Summary
Existing lane detection methods are unstable in complex scenarios, prone to missed detections and false detections, and lack robustness, making it difficult to meet the accurate perception requirements of autonomous driving.
A multi-layer, multi-granularity feature map is extracted using a cross-granularity adaptive convolutional network. Feature fusion is performed by combining a multi-head attention mechanism. The detection results are smoothed and outlier corrections are performed using prior knowledge of lane line continuity to form the target lane line position matrix.
It improves the accuracy and robustness of lane line detection, effectively alleviates the problems of detection instability and missed detection, and enhances the stability of lane line detection.
Smart Images
Figure CN121121685A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of vehicles, and more particularly to a lane line detection method and related device. BACKGROUND
[0002] Lane line detection is a key link in the automatic driving perception system, which can provide core input for advanced intelligent driving assistance system. The current lane line detection method mostly adopts a network framework based on row anchor points, the core idea of which is to divide the lane line image into specific rows and to divide each row into grids, to realize lane line modeling by judging the specific position of the lane line in each row and combining multi-level feature fusion.
[0003] However, the feature fusion scheme of the existing lane line detection method has certain limitations - there is a lack of correlation between features of different levels, which leads to unstable detection or missed detection problems in complex scenes (such as sudden changes in light, worn road markings, occlusion, etc.); at the same time, the suppression ability of outliers (misdetected points with large deviation from the true lane line position) is weak. Ultimately, the detection accuracy of the existing lane line detection method is low, and the robustness is insufficient, which is difficult to meet the demand for accurate perception of automatic driving. SUMMARY
[0004] Therefore, the present application discloses a lane line detection method and related device to improve the lane line detection accuracy and enhance the robustness of lane line detection.
[0005] A lane line detection method, comprising:
[0006] obtaining lane line image data;
[0007] extracting a multi-layer multi-granularity feature map from the lane line image data;
[0008] extracting features of different positions of different granularities from the multi-layer multi-granularity feature map based on a cross-granularity adaptive convolution network, and performing feature fusion processing on all the features of different positions to obtain a multi-fine-granularity fusion feature map;
[0009] converting the multi-fine-granularity fusion feature map into a lane line position matrix;
[0010] based on lane line continuity prior knowledge, smoothing and correcting outliers of the lane line position shown in the lane line position matrix to obtain a target lane line position matrix composed of corrected lane line positions, and completing lane line detection.
[0011] Optionally, the extracting a multi-layer multi-granularity feature map from the lane line image data comprises:
[0012] The lane line image data is subjected to data enhancement processing to obtain intermediate lane line image data;
[0013] The intermediate lane line image data is subjected to normalization processing to obtain target lane line image data;
[0014] The multi-layer multi-granularity feature maps are extracted from the target lane line image data by using a feature extraction network.
[0015] Optionally, the data enhancement processing of the lane line image data to obtain intermediate lane line image data comprises:
[0016] A plurality of target rows are randomly selected from the lane line image data, and row anchor points in the target rows are set to zero to generate zero rows, thereby completing the data enhancement processing.
[0017] Optionally, the cross-granularity adaptive convolution network extracts features of different positions at different granularities from the multi-layer multi-granularity feature maps, and performs feature fusion processing on all the features of different positions to obtain a multi-fine-granularity fusion feature map, which comprises:
[0018] For each layer of multi-granularity feature maps of the multi-layer multi-granularity feature maps, a two-dimensional aggregation reference point is set;
[0019] A two-dimensional aggregation reference point position offset of a feature point predicted by a convolution layer in each layer of multi-granularity feature maps is predicted;
[0020] The two-dimensional aggregation reference points are subjected to normalization processing, and the two-dimensional aggregation reference points corresponding to each layer of multi-granularity feature maps after normalization and the two-dimensional aggregation reference point position offset are added to obtain an aggregation point of the cross-granularity adaptive convolution network in each layer of multi-granularity feature maps;
[0021] The feature value at the position of the aggregation point is determined as the feature of different positions at different granularities of the corresponding each layer of multi-granularity feature maps;
[0022] The features of different positions are subjected to feature fusion processing by an adaptive convolution network combined with a multi-head attention mechanism to obtain the multi-fine-granularity fusion feature map.
[0023] Optionally, the multi-fine-granularity fusion feature map is converted into a lane line position matrix, which comprises:
[0024] The multi-fine-granularity fusion feature map is flattened to obtain a one-dimensional feature vector;
[0025] The one-dimensional feature vector is processed by a fully connected layer to obtain a lane line position score;
[0026] The lane line position score is reorganized to obtain the lane line position matrix.
[0027] Optionally, based on the lane line continuity prior knowledge, the lane line positions shown in the lane line position matrix are smoothed and outliers are corrected to obtain a target lane line position matrix composed of the corrected lane line positions, and lane line detection is completed, including:
[0028] In the lane line position matrix, through the continuity constraint between the row anchor points, from top to bottom, the next row lane line distribution probability is obtained by using the previous row lane line position.
[0029] After obtaining all the corrected next row lane line distribution probabilities, each of the corrected next row lane line distribution probabilities is compared with a probability threshold in size.
[0030] If there is a target next row lane line distribution probability less than the probability threshold, it is determined that there is an outlier in the lane line corresponding to the target next row lane line distribution probability, and the next row lane line distribution probability is corrected again by using the previous row lane line position from bottom to top through the continuity constraint between the row anchor points, to obtain the target lane line position matrix composed of the corrected lane line positions.
[0031] If there is no target next row lane line distribution probability less than the probability threshold, the target lane line position matrix composed of the corrected lane line positions is directly obtained.
[0032] Optionally, in the lane line position matrix, through the continuity constraint between the row anchor points, from top to bottom, the next row lane line distribution probability is obtained by using the previous row lane line position, including:
[0033] In the lane line position matrix, through the continuity constraint between the row anchor points, from top to bottom, the next row lane line distribution probability is obtained by using the previous row lane line position, including:
[0034] A lane line detection device, comprising:
[0035] An image acquisition unit configured to acquire lane line image data;
[0036] A feature extraction unit configured to extract a multi-layer multi-granularity feature map from the lane line image data;
[0037] The feature fusion unit is used to extract features at different locations with different granularities from the multi-layer multi-granularity feature map based on the cross-granularity adaptive convolutional network, and to perform feature fusion processing on all the features at different locations to obtain a multi-fine-granularity fused feature map.
[0038] The conversion unit is used to convert the multi-fine-grained fused feature map into a lane line position matrix;
[0039] The correction unit is used to smooth the lane line positions shown in the lane line position matrix and correct outliers based on prior knowledge of lane line continuity, so as to obtain the target lane line position matrix composed of the corrected lane line positions and complete the lane line detection.
[0040] A computer storage medium storing at least one instruction that, when executed by a processor, implements any lane line detection method.
[0041] A vehicle includes a vehicle controller, the vehicle controller including a memory and a processor;
[0042] The memory is used to store at least one instruction;
[0043] The processor is used to execute the at least one instruction to implement any lane line detection method.
[0044] As can be seen from the above technical solution, this invention discloses a lane line detection method and related apparatus. The method acquires lane line image data, extracts multi-layer, multi-granularity feature maps from the lane line image data, extracts features at different positions and granularities from the multi-layer, multi-granularity feature maps based on a cross-granularity adaptive convolutional network, and performs feature fusion processing on all features at different positions to obtain a multi-fine-granularity fused feature map. This multi-fine-granularity fused feature map is then transformed into a lane line position matrix. Based on prior knowledge of lane line continuity, the lane line positions shown in the lane line position matrix are smoothed and outliers are corrected to obtain the target lane line position matrix composed of the corrected lane line positions, thus completing lane line detection. This invention strengthens the correlation between multi-level, multi-granularity features through a cross-granularity adaptive convolutional network, thereby better adapting to the slender structural characteristics of lane lines and effectively alleviating problems such as detection instability and missed detections, thereby improving the accuracy of lane line detection. Simultaneously, by introducing prior knowledge of lane line continuity as a constraint condition, the detection results are regularized using this prior knowledge, effectively suppressing outliers and enhancing the robustness of lane line detection. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.
[0046] Figure 1 This is a flowchart of a lane line detection method disclosed in an embodiment of the present invention;
[0047] Figure 2 This is a schematic diagram of lane line detection position for lane anchor classification disclosed in an embodiment of the present invention;
[0048] Figure 3 This is a schematic diagram of a cross-granularity adaptive convolutional network disclosed in an embodiment of the present invention;
[0049] Figure 4 This is a schematic diagram of a lane line outlier point disclosed in an embodiment of the present invention;
[0050] Figure 5 This is a schematic diagram of a lane line detection method based on cross-granularity adaptive convolution disclosed in an embodiment of the present invention;
[0051] Figure 6 This is a flowchart of a method for obtaining multi-fine-grained fused feature maps through feature fusion, as disclosed in an embodiment of the present invention.
[0052] Figure 7 This is a flowchart of a method for determining a target lane line position matrix disclosed in an embodiment of the present invention;
[0053] Figure 8 This is a schematic diagram illustrating the probability correction of the next lane line distribution according to an embodiment of the present invention;
[0054] Figure 9 This is a schematic diagram of the structure of a lane line detection device disclosed in an embodiment of the present invention;
[0055] Figure 10 This is a schematic diagram of a vehicle controller disclosed in an embodiment of the present invention. Detailed Implementation
[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0057] This invention discloses a lane line detection method and related apparatus. By using a cross-granularity adaptive convolutional network, the correlation between multi-level and multi-granularity features is strengthened, thereby better adapting to the slender structural characteristics of lane lines and effectively alleviating problems such as detection instability and missed detections, thus improving the accuracy of lane line detection. At the same time, prior knowledge of lane line continuity is introduced as a constraint condition. The detection results are regularized by the prior knowledge of lane line continuity, which effectively suppresses outliers and enhances the robustness of lane line detection.
[0058] See Figure 1 The present invention discloses a lane line detection method flowchart, the method comprising:
[0059] Step S101: Obtain lane line image data.
[0060] In practical applications, lane line image data can be acquired through cameras (e.g., wide-angle cameras) installed at the front of the vehicle.
[0061] Step S102: Extract multi-layer, multi-granularity feature maps from lane line image data.
[0062] Specifically, (1) data augmentation processing is performed on the lane line image data to obtain the middle lane line image data.
[0063] Considering the slender shape of lane lines, the image enhancement model needs to have the ability to complete lane lines under occlusion conditions in order to effectively restore the occluded parts of the lane lines. This embodiment randomly selects several target rows from the lane line image data, sets the row anchor points of the target rows to zero to generate zero-set rows, and completes the data enhancement process. See details. Figure 2 The diagram shown illustrates the lane line detection positions for row anchor classification. Row anchors represent m rows, column anchors represent n columns, and zero-rows are shown below. Figure 2 As shown.
[0064] Lane line image data typically acquired by cameras mounted on the front of vehicles is 3×704×1280 pixels. Since lane lines are generally concentrated in the upper half of the image, the acquired lane line image data can be cropped before data augmentation, retaining only the portion containing lane lines. The cropping range can be determined by the experimenter using prior knowledge; for example, the cropped lane line image data could be 3×256×1280 pixels. Then, data augmentation processing is performed on the cropped lane line image data to obtain the middle lane line image data.
[0065] (2) Normalize the image data of the middle lane line to obtain the image data of the target lane line.
[0066] In practical applications, in order to better restore the obscured lane lines based on the information around the vehicle, it is necessary to normalize the enhanced middle lane line image data, for example, normalize the middle lane line image data to [-1, 1], so as to perform subsequent operations.
[0067] (3) Use a feature extraction network to extract multi-layer, multi-granularity feature maps from the target lane line image data.
[0068] A preferred feature extraction network could be a pre-trained ResNet34.
[0069] Taking a three-layer multi-granularity feature map as an example, the three-layer multi-granularity feature map includes: conv2_x, conv3_x, conv4_x, and the size of each layer is (128×88×160), (256×44×80), and (512×22×40) respectively.
[0070] Step S103: Based on the cross-granularity adaptive convolutional network, extract features at different locations of different granularities from the multi-layer multi-granularity feature map, and perform feature fusion processing on all the features at different locations to obtain a multi-fine-granularity fused feature map.
[0071] In practical applications, if the feature extraction network in step S102 is a pre-trained ResNet34, the multi-layer, multi-granularity feature map extracted from the lane line image data can be a ResNet (Deep Residual Network) multi-level feature map.
[0072] The Transformer Encoder is one of the core components of the Transformer model. It is mainly used for feature extraction and context understanding of the input sequence. Through multi-layer attention mechanism and feedforward network, it captures the dependencies between elements in the sequence and generates context information.
[0073] This invention is based on the idea of Transformer Encoder and adopts... Figure 3 The cross-granularity adaptive convolutional network shown extracts features at different locations with different granularities from each layer of a multi-layer, multi-granularity feature map. The parameters of the features at different locations are not shared. By fusing the features at different locations, a multi-fine-grained fused feature map is generated. The multi-granularity fusion of lane lines is achieved by using the cross-granularity adaptive convolutional network.
[0074] Step S104: Convert the multi-fine-grained fused feature map into a lane line position matrix.
[0075] Flatten the multi-fine-grained fused feature map to obtain a one-dimensional feature vector;
[0076] The one-dimensional feature vector is processed through a fully connected layer to obtain the lane line position score;
[0077] The lane line position scores are reorganized to obtain the lane line position matrix.
[0078] The recombination process converts the dimensions of the fully connected layer's output into a three-dimensional matrix.
[0079] For example, the final converted lane position matrix is 101×20×4, where 101 represents the position of the lane line in the 101st row of the lane line image, 20 represents the position of the lane line in the 20th column of the lane line image, and 4 represents the 4 predicted lane lines.
[0080] In this embodiment, the lane detection head is the core module in the deep learning model used to ultimately predict the lane line position. It is usually located at the end of the network and is responsible for converting the previously extracted fine-grained features into specific lane line detection results. The lane detection head includes: flattening and fully connected layers.
[0081] Step S105: Based on prior knowledge of lane line continuity, smooth the lane line positions shown in the lane line position matrix and correct outliers to obtain the target lane line position matrix composed of the corrected lane line positions, thus completing lane line detection.
[0082] Lane line continuity prior knowledge is a cognitive framework based on road traffic rules, physical characteristics, and driving logic, forming a coherence and regularity of lane lines in spatial and temporal dimensions. It reflects the "continuity" characteristic that lane lines, as the core function of road markings (guiding vehicles and defining lane boundaries), must possess, and is key prior information in fields such as intelligent driving (e.g., lane line detection, path planning) and traffic engineering.
[0083] In this embodiment, the prior knowledge of lane line continuity serves as a theoretical foundation, which can be implemented through the continuity constraints of line anchor points. That is, the prior knowledge of lane line continuity provides the theoretical basis for the continuity constraints of line anchor points, while the continuity constraints of line anchor points are the technical implementation means.
[0084] Based on this, in practical applications, the lane line positions shown in the lane line position matrix can be smoothed and outliers corrected by the continuity constraint of the row anchor points, so as to obtain the target lane line position matrix composed of the corrected lane line positions, thereby completing lane line detection.
[0085] Set the lane line position matrix as follows: The prediction result for each lane line image data is There are 1 lane markings, and each lane marking has 1 anchor point. Each lane line's anchor point divides the row into lines. Each column position predicts the lane to be in The probability of each lane position and the probability of having no lane line. Therefore, each lane line image data needs to predict... Each category score, such as Figure 2 As shown, each lane line has a location distribution map. Traditional algorithms select... The maximum value in each row determines the position of the lane line within that row. However, since the predictions are not correlated between rows, and there are multiple instance lines within a row, there may be multiple maxima in the prediction results for each row. Taking the maximum value can easily lead to outliers caused by incorrect predictions of a particular lane line in a given row. Figure 4 As shown, solving this problem requires using prior knowledge of lane line continuity to constrain the output of lane line positions, ensuring that the same lane line... The columns in the row are in consecutive positions.
[0086] In summary, this invention discloses a lane line detection method. It acquires lane line image data, extracts multi-layer, multi-granularity feature maps from the lane line image data, extracts features at different locations and granularities from the multi-layer, multi-granularity feature maps using a cross-granularity adaptive convolutional network, and fuses all features at different locations to obtain a multi-fine-granularity fused feature map. This multi-fine-granularity fused feature map is then transformed into a lane line position matrix. Based on prior knowledge of lane line continuity, the lane line positions shown in the lane line position matrix are smoothed and outliers are corrected, resulting in a target lane line position matrix composed of corrected lane line positions, thus completing lane line detection. This invention strengthens the correlation between multi-level, multi-granularity features through a cross-granularity adaptive convolutional network, thereby better adapting to the slender structural characteristics of lane lines and effectively alleviating problems such as detection instability and missed detections, thus improving the accuracy of lane line detection. Simultaneously, by introducing prior knowledge of lane line continuity as a constraint, the detection results are regularized using this prior knowledge, effectively suppressing outliers and enhancing the robustness of lane line detection.
[0087] In one embodiment, see Figure 5 The present invention discloses a lane line detection method based on cross-granularity adaptive convolution, and the entire detection process includes:
[0088] (1) Image acquisition: Specifically, lane line image data is acquired through a camera installed at the front of the vehicle.
[0089] (2) Feature extraction, specifically: extracting multi-layer, multi-granularity feature maps from lane line image data.
[0090] (3) Feature fusion, specifically: based on the cross-granularity adaptive convolutional network, features at different positions of different granularities are extracted from multi-layer multi-granularity feature maps, and features at all different positions are fused to obtain multi-fine-granularity fused feature maps.
[0091] (4) Flatten the multi-fine-grained fusion feature map to obtain a one-dimensional feature vector.
[0092] (5) The one-dimensional feature vector is processed through a fully connected layer to obtain the lane line position score.
[0093] (6) Reorganize the lane line position fractions to obtain the lane line position matrix.
[0094] (7) Lane line continuity selection: Based on prior knowledge of lane line continuity, the lane line positions shown in the lane line position matrix are smoothed and outliers are corrected to obtain the target lane line position matrix composed of the corrected lane line positions, thus completing lane line detection.
[0095] In practical applications, the L2 loss of the modified target lane line position matrix and the classification loss can be combined to train the cross-granularity adaptive convolutional network using a large-scale self-collected labeled dataset until the model converges.
[0096] In summary, the cross-granularity adaptive convolution and network learning fusion strategy proposed in this invention (i.e., multi-granularity feature fusion) effectively fuses global and local features while preserving detailed information in low-resolution lane line images. Compared to traditional solutions using Transformer Encoder and other feature fusion methods, this strategy significantly reduces computational complexity during feature fusion; combined with... Figure 5 The overall architecture of the lane detection model shown offers advantages in lightweight and high efficiency during deployment, making it particularly suitable for practical applications such as embedded systems or low-cost hardware. Furthermore, the method for processing prior knowledge of lane line continuity in this invention, by introducing lane line continuity constraints, fully exploits the spatial coherence and geometric characteristics of lane lines, accurately locating their positions and effectively solving the outlier problem that easily occurs in traditional line anchor point methods for lane detection.
[0097] In one embodiment, see Figure 6 The present invention discloses a flowchart of a method for obtaining multi-fine-grained fused feature maps through feature fusion, specifically step S103 may include:
[0098] Step S201: For each layer of multi-granularity feature map, set a two-dimensional aggregation reference point.
[0099] Specifically, a two-dimensional aggregation reference point is set for each layer of multi-granularity feature maps. The size is: , This indicates the number of input feature points that need to be aggregated for each output feature point. Indicates the height of the multi-fine-grained fused feature map. This represents the width of the multi-fine-grained fused feature map.
[0100] Two-dimensional aggregation reference point This refers to the reference points for the output feature points of the adaptive convolutional network at the locations where the input features need to be aggregated.
[0101] Step S202: Use the convolutional layer to predict the offset of the two-dimensional aggregated reference point position of the output feature point in each multi-granularity feature map.
[0102] For each multi-granularity feature map layer, the convolutional layer is used to predict the size of the feature points in that layer's feature map. The offset of the two-dimensional aggregated reference point position. For each output feature point in H×W, the predicted offset of the two-dimensional aggregated reference point position can be expressed as: .
[0103] Step S203: Normalize the two-dimensional aggregation reference points, and add the normalized two-dimensional aggregation reference points and the position offsets of the two-dimensional aggregation reference points corresponding to each multi-granularity feature map to obtain the aggregation points of the cross-granularity adaptive convolutional network in each multi-granularity feature map.
[0104] Specifically, the aggregation point of the multi-granularity feature map in each layer of the cross-granularity adaptive convolutional network can be represented as: ,in, Represents each output feature point One of the two-dimensional aggregation reference points, This indicates a normalization operation. This indicates the offset of the two-dimensional aggregated reference point position.
[0105] The normalization process for two-dimensional aggregated reference points can be achieved through the following operations: Taking two-dimensional aggregated reference points in feature maps of different granularities (such as original channel numbers of 128, 256, and 512) as targets, upsampling (such as interpolation and deconvolution) or downsampling (such as pooling and stride convolution) operations are used to adjust the feature dimension and spatial size of each two-dimensional aggregated reference point. Specifically, feature maps with low channel numbers (such as 128) are upsampled to increase the channel number to 256, and feature maps with high channel numbers (such as 512) are downsampled to reduce the channel number to 256, resulting in a final size of 256×44×80 for subsequent fusion operations.
[0106] Step S204: Determine the feature value at the aggregation point as the feature at different positions of different granularities in the corresponding multi-granularity feature map of each layer.
[0107] Step S205: Combine an adaptive convolutional network with a multi-head attention mechanism to perform feature fusion processing on features at different locations to obtain a multi-fine-grained fused feature map.
[0108] The feature fusion process using an adaptive convolutional network with multi-head attention mechanism specifically includes:
[0109] The convolution operation performed on the feature values at the aggregation point location is represented as: ,in, This represents the eigenvalue at the aggregation point location. Represents eigenvalues. Represents the convolution value. The eigenvalues are obtained using bilinear interpolation. This indicates bilinear interpolation.
[0110] The aggregated values of the output feature points on each multi-granularity feature map are represented as multiple aggregated positions. Corresponding eigenvalues and convolution values Multiply them and sum the products.
[0111] The aggregated values of the multi-layer, multi-granularity feature maps are summed to obtain L represents the number of layers in the multi-layer, multi-granularity feature map. This represents the offset of the predicted two-dimensional aggregated reference point position.
[0112] Then, an adaptive convolutional network with a multi-head attention mechanism is used for aggregation to obtain the output formula of the cross-granularity adaptive convolutional network (MSDCN), as follows:
[0113] ;
[0114] In the formula, This represents each position of the output feature in the multi-fine-grained fused feature map. The input features represent different granularities, and M represents the number of deformable convolution heads. This represents the normalization coefficient used to balance each deformable convolution, obtained by batch normalization at different positions of the input feature values.
[0115] The final size of the multi-fine-grained fused feature map is: , Indicates the size of the feature dimension. Indicates the height of the multi-fine-grained fused feature map. This represents the width of the multi-fine-grained fused feature map.
[0116] It should be noted that the feature value at the aggregation point is the granular feature of each layer's multi-granularity feature map.
[0117] In one embodiment, see Figure 7 The flowchart of a method for determining a target lane line position matrix disclosed in this embodiment of the invention, specifically step S105, may include:
[0118] Step S301: In the lane position matrix, by using the continuity constraint between row anchor points, from top to bottom, the corrected distribution probability of the next lane line is obtained using the position of the previous lane line.
[0119] Specifically, in the lane position matrix, through the continuity constraint between row anchor points, starting from the position where the lane line appears in the first row, from top to bottom, the position of the previous row of lane lines is taken as the center of the normal distribution, and the center of the normal distribution is multiplied by the predicted probability of the next row of lane lines to obtain the corrected distribution probability of the next row of lane lines.
[0120] For the input lane line position matrix Assuming that in the row where lane lines first appear from top to bottom, the position represented by the q-th column of column m is the position with the highest probability of lane line distribution, then the predicted probability of lane line distribution in the next row is: , representing the probability of a laneless distribution And since its value is not the maximum, the probability distribution of the next lane line after correction of the previous lane line position is:
[0121] ;
[0122] In the formula, Let L and S represent the corrected probability distribution of the next lane, where L and S are normal distribution coefficients, and i represents the i-th lane.
[0123] Corrected probability distribution of the next lane line The maximum value of this value indicates the location of the next lane line. This formula means that the closer the next lane line is to the previous lane line, the greater the probability of it appearing on the next lane.
[0124] See Figure 8The diagram illustrates the correction of the next row's lane line distribution probability. The horizontal axis represents the column anchor, and the vertical axis represents the probability of each lane line in the row anchor. Starting from the position where a lane line appears in the first row, the process proceeds downwards, using the position of the previous row's lane line as the center of the normal distribution. Points on the normal distribution represent probabilities multiplied by a Gaussian coefficient. The center of the normal distribution is then multiplied by the predicted probability of the next row's lane line to obtain the corrected distribution probability of the next row's lane line. This process effectively identifies the correct lane line positions and avoids false lane line detections caused by outliers. The process then iterates downwards row by row, skipping rows where no lane line is detected. For rows containing lane lines, the continuous distribution only corrects the corresponding probability distribution.
[0125] Step S302: After obtaining all the corrected next lane line distribution probabilities, compare the corrected next lane line distribution probability with the probability threshold.
[0126] Step S303: If there is a target lane line distribution probability less than the probability threshold, determine that there are outliers in the lane lines corresponding to the target lane line distribution probability. Through the continuity constraint between row anchor points, from bottom to top, use the position of the previous lane line to correct the next lane line distribution probability again, and obtain the target lane line position matrix composed of the corrected lane line positions.
[0127] Step S304: If there is no target lane line distribution probability less than the probability threshold, the target lane line position matrix composed of the corrected lane line positions is directly obtained.
[0128] In this embodiment, after correcting the probability distribution of all lane lines for each lane line from top to bottom, the probability distribution of each corrected next lane line is compared with a probability threshold to find outliers that deviate from the entire line. If outliers are found, the probability distribution of the next lane line is corrected again from bottom to top using the position of the previous lane line, thus avoiding prediction errors from the very first line.
[0129] The target lane line position matrix, composed of the corrected lane line positions one by one, can be visualized and marked in the lane line image data.
[0130] Corresponding to the above method embodiments, the present invention also discloses a lane line detection device.
[0131] See Figure 9 The present invention discloses a schematic diagram of a lane line detection device, which may include:
[0132] Image acquisition unit 401 is used to acquire lane line image data.
[0133] In practical applications, lane line image data can be acquired through cameras (e.g., wide-angle cameras) installed at the front of the vehicle.
[0134] The feature extraction unit 402 is used to extract multi-layer, multi-granularity feature maps from lane line image data.
[0135] The feature fusion unit 403 is used to extract features at different locations with different granularities from multi-layer multi-granularity feature maps based on a cross-granularity adaptive convolutional network, and to perform feature fusion processing on all features at different locations to obtain a multi-fine-granularity fused feature map.
[0136] The transformation unit 404 is used to transform the multi-fine-grained fused feature map into a lane line position matrix.
[0137] The correction unit 405 is used to smooth the lane line positions shown in the lane line position matrix and correct outliers based on prior knowledge of lane line continuity, so as to obtain the target lane line position matrix composed of the corrected lane line positions and complete the lane line detection.
[0138] In summary, this invention discloses a lane line detection device that acquires lane line image data, extracts multi-layer, multi-granularity feature maps from the lane line image data, extracts features at different locations and granularities from the multi-layer, multi-granularity feature maps based on a cross-granularity adaptive convolutional network, and fuses all features at different locations to obtain a multi-fine-granularity fused feature map. This multi-fine-granularity fused feature map is then transformed into a lane line position matrix. Based on prior knowledge of lane line continuity, the lane line positions shown in the lane line position matrix are smoothed and outliers are corrected to obtain a target lane line position matrix composed of corrected lane line positions, thus completing lane line detection. This invention strengthens the correlation between multi-level, multi-granularity features through a cross-granularity adaptive convolutional network, thereby better adapting to the slender structural characteristics of lane lines and effectively alleviating problems such as detection instability and missed detections, thereby improving the accuracy of lane line detection. Simultaneously, by introducing prior knowledge of lane line continuity as a constraint condition, the detection results are regularized using this prior knowledge, effectively suppressing outliers and enhancing the robustness of lane line detection.
[0139] In one embodiment, the feature extraction unit 402 can be specifically used for:
[0140] Data augmentation processing is performed on the lane line image data to obtain the middle lane line image data;
[0141] The image data of the middle lane line is normalized to obtain the image data of the target lane line;
[0142] By using a feature extraction network, multi-layer, multi-granularity feature maps are extracted from target lane line image data.
[0143] In one embodiment, the feature extraction unit 402 can also be used for:
[0144] Several target rows are randomly selected from the lane line image data, and the row anchor points in the target rows are set to zero to generate zeroed rows, thus completing the data augmentation process.
[0145] In one embodiment, the feature fusion unit 403 can be specifically used for:
[0146] For each layer of multi-granularity feature map, a two-dimensional aggregation reference point is set.
[0147] The offset of the output feature point in the two-dimensional aggregated reference point position in the multi-granularity feature map of each layer is predicted using convolutional layers;
[0148] The two-dimensional aggregation reference points are normalized, and the normalized two-dimensional aggregation reference points and their position offsets corresponding to the multi-granularity feature maps of each layer are added together to obtain the aggregation points of the cross-granularity adaptive convolutional network in each multi-granularity feature map.
[0149] The feature values at the aggregation point are determined as features at different locations of different granularities in the corresponding multi-granularity feature map of each layer;
[0150] An adaptive convolutional network combining multi-head attention mechanism is used to perform feature fusion processing on features at different locations to obtain multi-fine-grained fused feature maps.
[0151] In one embodiment, the conversion unit 404 can be specifically used for:
[0152] Flatten the multi-fine-grained fused feature map to obtain a one-dimensional feature vector;
[0153] The one-dimensional feature vector is processed through a fully connected layer to obtain the lane line position score;
[0154] The lane line position scores are reorganized to obtain the lane line position matrix.
[0155] In one embodiment, the correction unit 405 can specifically be used for:
[0156] In the lane position matrix, by using the continuity constraint between row anchor points, the corrected probability distribution of the next lane line is obtained from top to bottom using the position of the previous lane line.
[0157] After obtaining all the corrected next lane line distribution probabilities, the probability of each corrected next lane line distribution is compared with the probability threshold.
[0158] If there is a target lane line distribution probability less than the probability threshold, it is determined that there are outliers in the lane line corresponding to the target lane line distribution probability. By using the continuity constraint between the row anchors, the next lane line distribution probability is corrected again from bottom to top using the position of the previous lane line, and the target lane line position matrix composed of the corrected lane line positions is obtained.
[0159] If there is no target lane line distribution probability less than the probability threshold, the target lane line position matrix composed of the corrected lane line positions can be obtained directly.
[0160] In one embodiment, the correction unit 405 can also be used for:
[0161] In the lane position matrix, by constraining the continuity between row anchor points, starting from the position where the lane line appears in the first row, from top to bottom, the position of the previous row of lane lines is taken as the center of the normal distribution, and the center of the normal distribution is multiplied by the predicted probability of the next row of lane lines to obtain the corrected distribution probability of the next row of lane lines.
[0162] It should be noted that for the specific working principles of each component in the device embodiment, please refer to the corresponding section of the method embodiment, which will not be repeated here.
[0163] Corresponding to the above embodiments, the present invention also discloses a computer storage medium that stores at least one instruction, which, when executed by a processor, implements the steps shown in the embodiments of the lane line detection method.
[0164] Computer storage media can be tangible media that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. Computer storage media can be machine-readable signal media or machine-readable storage media. Computer storage media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0165] Corresponding to the above embodiments, such as Figure 10 As shown, the present invention also provides a structural schematic diagram of a vehicle controller, wherein the electronic device may include: a processor 1 and a memory 2;
[0166] The processor 1 and memory 2 communicate with each other via communication bus 3.
[0167] Processor 1, for executing at least one instruction;
[0168] Memory 2 is used to store at least one instruction;
[0169] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0170] Memory 2 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0171] The processor executes at least one instruction to implement the lane line detection method.
[0172] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0173] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0174] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A lane line detection method, characterized in that, include: Acquire lane line image data; Multi-layer, multi-granularity feature maps are extracted from the lane line image data; Based on the cross-granularity adaptive convolutional network, features at different locations with different granularities are extracted from the multi-layer multi-granularity feature map, and all the features at different locations are fused to obtain a multi-fine-granularity fused feature map. The multi-fine-grained fused feature map is converted into a lane line position matrix; Based on prior knowledge of lane line continuity, the lane line positions shown in the lane line position matrix are smoothed and outliers are corrected to obtain the target lane line position matrix composed of the corrected lane line positions, thus completing lane line detection.
2. The lane line detection method according to claim 1, characterized in that, The step of extracting multi-layer, multi-granularity feature maps from the lane line image data includes: The lane line image data is subjected to data augmentation processing to obtain the middle lane line image data; The image data of the middle lane line is normalized to obtain the image data of the target lane line; The multi-layer, multi-granularity feature map is extracted from the target lane line image data using a feature extraction network.
3. The lane line detection method according to claim 2, characterized in that, The process of performing data augmentation on the lane line image data to obtain the middle lane line image data includes: Several target rows are randomly selected from the lane line image data, and the row anchor points in the target rows are set to zero to generate zero-set rows, thus completing the data augmentation process.
4. The lane line detection method according to any one of claims 1 to 3, characterized in that, The cross-granularity adaptive convolutional network extracts features at different locations with different granularities from the multi-layer, multi-granularity feature map, and performs feature fusion processing on all the features at different locations to obtain a multi-fine-granularity fused feature map, including: For each layer of the multi-granularity feature map, a two-dimensional aggregation reference point is set. The offset of the two-dimensional aggregated reference point position of the output feature point in the multi-granularity feature map of each layer is predicted using the convolutional layer; The two-dimensional aggregation reference points are normalized, and the normalized two-dimensional aggregation reference points corresponding to each multi-granularity feature map are added together with the position offset of the two-dimensional aggregation reference points to obtain the aggregation points of the cross-granularity adaptive convolutional network in each multi-granularity feature map. The feature value at the aggregation point is determined as the feature at different positions of different granularities in each layer of the multi-granularity feature map; An adaptive convolutional network with multi-head attention mechanism is used to perform feature fusion processing on features at different locations to obtain the multi-fine-grained fused feature map.
5. The lane line detection method according to any one of claims 1 to 3, characterized in that, The step of converting the multi-fine-grained fused feature map into a lane line position matrix includes: Flatten the multi-fine-grained fused feature map to obtain a one-dimensional feature vector; The one-dimensional feature vector is processed through a fully connected layer to obtain the lane line position score; The lane line position fractions are reorganized to obtain the lane line position matrix.
6. The lane line detection method according to claim 1, characterized in that, Based on prior knowledge of lane line continuity, the lane line positions shown in the lane line position matrix are smoothed and outliers are corrected to obtain a target lane line position matrix composed of corrected lane line positions, thus completing lane line detection, including: In the lane position matrix, by using the continuity constraint between row anchor points, the corrected distribution probability of the next lane line is obtained from top to bottom using the position of the previous lane line. After obtaining all the corrected next lane line distribution probabilities, each corrected next lane line distribution probability is compared with a probability threshold. If there is a target next lane line distribution probability less than the probability threshold, it is determined that there is an outlier in the lane line corresponding to the target next lane line distribution probability. By using the continuity constraint between row anchor points, the next lane line distribution probability is corrected again from bottom to top using the position of the previous lane line to obtain the target lane line position matrix composed of the corrected lane line positions. If there is no target lane line distribution probability less than the probability threshold, the target lane line position matrix composed of the corrected lane line positions is directly obtained.
7. The lane line detection method according to claim 6, characterized in that, In the lane position matrix, by using the continuity constraint between row anchor points, the corrected probability distribution of the next lane line is obtained from top to bottom using the position of the previous lane line, including: In the lane line position matrix, by constraining the continuity between row anchor points, starting from the position where the lane line appears in the first row, from top to bottom, the position of the previous row of lane lines is taken as the center of normal distribution, and the center of normal distribution is multiplied by the predicted probability of the next row of lane lines to obtain the corrected distribution probability of the next row of lane lines.
8. A lane line detection device, characterized in that, include: Image acquisition unit, used to acquire lane line image data; A feature extraction unit is used to extract multi-layer, multi-granularity feature maps from the lane line image data; The feature fusion unit is used to extract features at different locations with different granularities from the multi-layer multi-granularity feature map based on the cross-granularity adaptive convolutional network, and to perform feature fusion processing on all the features at different locations to obtain a multi-fine-granularity fused feature map. The conversion unit is used to convert the multi-fine-grained fused feature map into a lane line position matrix; The correction unit is used to smooth the lane line positions shown in the lane line position matrix and correct outliers based on prior knowledge of lane line continuity, so as to obtain the target lane line position matrix composed of the corrected lane line positions and complete the lane line detection.
9. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction, which, when executed by a processor, implements the lane line detection method as described in any one of claims 1 to 7.
10. A vehicle, comprising a vehicle controller, characterized in that, The vehicle controller includes a memory and a processor; The memory is used to store at least one instruction; The processor is used to execute the at least one instruction to implement the lane detection method as described in any one of claims 1 to 7.