A lane line detection method and system based on super anchor

Abstract

The present invention relates to a lane line detection method and system based on super anchor, the method comprising: S1: extracting features from a forward traffic image through a convolutional network to obtain a lane line feature map F b ; S2: Based on F b Input the super anchor generation network to obtain the super anchor probability score map S for each lane line; and calculate the position expectation of S row by row and column by column to generate the super anchor position P of the middle lane and the two side lanes respectively; S3: The lateral features obtained by connecting the convolutional network and the feature pyramid network laterally are combined with F b Fusion is performed to obtain the feature map F loc , F loc After adding the position encoding vector, it is multiplied point by point with S to obtain the lane descriptor D; S4: Perform self-attention-based feature aggregation on D within and between lanes to obtain D loc and D gbl The two prediction heads are used to obtain the lane offset relative to the super anchor and the lane confidence, and the final lane line position is obtained. The method provided by the present invention can obtain the accurate position of the lane line in extreme situations.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a lane line detection method and system based on super-anchors. Background Art

[0002] As a prerequisite for driving tasks such as lane keeping, lane changing, and drivable area detection, lane detection is one of the key perception tasks for autonomous driving. Compared to conventional traffic targets, lane markings have relatively simple texture, structure, and spatial distribution pattern. That is, lane markings are usually thin white or yellow lines on the ground, which makes them relatively easy to process. Under normal road conditions, traditional lane detection methods and recent deep learning-based methods work well. However, when visual cues of lanes are lacking, almost all of these methods suffer from significant performance degradation in challenging scenarios, i.e., the no-lane visual cues problem.

[0003] Lane detection remains a challenging task due to the inherently slender structure of lane markings and the complex external environmental factors that can easily lead to a loss of visual cues. Existing lane detection methods often experience significant performance degradation in the face of extreme lighting conditions and vehicle occlusion. Therefore, improving lane detection accuracy in challenging scenarios such as occlusion and extreme lighting has become a pressing issue. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a lane line detection method and system based on super-anchor.

[0005] The technical solution of the present invention is: a lane line detection method based on super anchor, comprising:

[0006] Step S1: The forward traffic image is passed through the convolutional network for feature extraction to obtain the lane feature map F b ;

[0007] Step S2: Based on F b Input the super-anchor generation network to obtain the super-anchor probability score map S for each lane line; and calculate the position expectation of the super-anchor probability score map row by row and column by column to generate the super-anchor positions of the middle lane and the two side lanes respectively;

[0008] Step S3: Connect the lateral features obtained by lateral connection of the convolutional network and the feature pyramid network to F b Fusion is performed to obtain a feature map F of the same size as S loc , F loc After adding it to the position encoding vector, it is multiplied point by point with S to obtain the lane descriptor D with an adaptive receptive field;

[0009] Step S4: Perform self-attention-based feature aggregation on the lane descriptor D within and between lanes to obtain D loc and D gbl The lane offset relative to the super anchor and the lane confidence are obtained through two prediction heads respectively, and the super anchor position and the offset are added to obtain the final lane line position.

[0010] Compared with the prior art, the present invention has the following advantages:

[0011] The present invention discloses a lane line detection method based on super-anchors. A new type of lane anchor, namely the super-anchor, is constructed for lane line detection. The super-anchor can provide the rough position of the lane line. Based on the super-anchor, a multi-level lane perception feature aggregation method is used to aggregate the features within and between lane lines to capture the local features of the lane itself and the global features between lane lines, thereby obtaining the accurate position of the lane line. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Flowchart of a lane line detection method based on super-anchor in an embodiment of the present invention;

[0013] Figure 2 Schematic diagram of the super-anchor generation network structure in an embodiment of the present invention;

[0014] Figure 3 Schematic diagram of row and column super anchors in an embodiment of the present invention;

[0015] Figure 4A Schematic diagram of lane feature aggregation in an embodiment of the present invention;

[0016] Figure 4B Schematic diagram of lane-to-lane feature aggregation in an embodiment of the present invention;

[0017] Figure 5 The figure is a structural block diagram of a lane detection system based on super-anchor in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention provides a lane line detection method based on super-anchors, which can obtain accurate lane line positions in the face of extreme lighting and vehicle occlusion.

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementation and in conjunction with the accompanying drawings.

[0020] Example 1

[0021] like Figure 1 As shown, a lane line detection method based on super anchor provided by an embodiment of the present invention includes the following steps:

[0022] Step S1: The forward traffic image is passed through the convolutional network for feature extraction to obtain the lane feature map F b ;

[0023] Step S2: Based on F b The super-anchor generation network is input to obtain the super-anchor probability score map S for each lane line. The super-anchor positions for the middle lane and the two side lanes are generated by calculating the position expectation of the super-anchor probability score map row by row and column by column.

[0024] Step S3: Connect the convolutional network to the feature pyramid network to obtain the lateral features and F b Fusion is performed to obtain a feature map F of the same size as S loc , F loc After adding it to the position encoding vector, it is multiplied point by point with S to obtain the lane descriptor D with an adaptive receptive field;

[0025] Step S4: Perform self-attention-based feature aggregation on the lane descriptor D within and between lanes to obtain D loc and D gbl The lane offset relative to the super anchor and the lane confidence are obtained through the two prediction heads respectively. The super anchor position and the offset are added to obtain the final lane line position.

[0026] In one embodiment, the above step S1: extracts features from the forward traffic image through a convolutional network to obtain a lane feature map F b ;

[0027] like Figure 2 As shown, the forward traffic image is used as the input image I∈R H×W×3 , where H and W are the height and width of the input image respectively, 3 is the number of channels, and the input image I is subjected to multi-layer convolution operations of the convolutional network to extract the features of the input image and obtain the lane line feature map F b .

[0028] In one embodiment, the above step S2: F b The super-anchor generation network is input to obtain the super-anchor probability score map S for each lane line. The super-anchor positions for the middle lane and the two lanes are generated by calculating the position expectation of the super-anchor probability score map row by row and column by column. Specifically,

[0029] Step S21: Based on the input image I, predefine a set of evenly distributed rows, called row anchors Among them, N row is the number of rows; a lane line can be described by a set of lane points falling on the row anchor point;

[0030] Step S22: Dynamically generate a set of reference points based on image I It is called super anchor. According to the following formulas (1) to (3), the lane line representation based on the super anchor is obtained

[0031]

[0032]

[0033]

[0034] Where Φ(·) represents super-anchor generation, I is the input image, and θ is the parameter of the super-anchor generation network; x i Indicates the row anchor point y i The actual location of the lane entry point;

[0035] st and end represent the indices of the row anchor points of the start and end lane points, respectively, and conf represents the confidence of the lane prediction;

[0036] Step S23: Similarly, a uniformly distributed column of super anchor points can be obtained; therefore, the super anchor lane is expressed as formulas (4) to (5):

[0037]

[0038]

[0039] Among them, the line super anchor Indicates the lane, column super anchor Indicates side lane, N row and N col Represents the number of row and column anchors respectively;

[0040] Taking into account the position error of lane-by-lane representation, the embodiment of the present invention generalizes the above-mentioned super-anchor points based on uniformly distributed rows to uniformly distributed column super-anchor points, such as Figure 3 As shown, the row super anchors are used for the ego lane and the column super anchors are used for the side lane;

[0041] Step S24: F b The upper branch of the input super-anchor generation network passes through a 1×1 convolutional layer to reduce the feature channels to improve computational efficiency, is flattened and input into the fully connected layer, and finally rearranged and output F gbl , where F gbl The number of channels is C gbl , where C gbl =N ego +N side , N ego and N side are the number of lanes and lanes on both sides, respectively. Figure 2Indicated by gray grid;

[0042] F gbl It includes four groups of features, two of which are row super anchor features and column super anchor features, whose sizes are N row ×N col ×N ego and N row ×N col ×N side The outputs of are used to calculate the row super anchor and column super anchor respectively; for any lane, its corresponding super anchor feature is recorded as M, with a size of N row ×N col By calculating the softmax row by row or column by column, the super anchor probability score map S can be obtained, and S can be expanded by selecting K maximum response values, that is, S = softmax (M). The super anchor P is obtained by calculating the expected row or column position of S to obtain each anchor point position p of the super anchor P. i , as shown in formula (4):

[0043]

[0044] where s i，j ∈S;

[0045] The other two sets of features are the lane start position feature and the lane end position feature, and their sizes are (N row +1)×2×N ego and (N col +1)×2×N ego Used to mark the start and end of the lane; (N row +1) and (N col +1) is in N row and N col On this basis, an additional output is introduced to indicate the situation where there is no lane;

[0046] For example, taking the row hyperanchor calculation as an example, the calculation steps are described as follows:

[0047] Input: a row hyperanchor map The number of super anchor points per lane, K∈R 1 ;

[0048] 1: Denote M as M0 to indicate subscript consistency, and perform row-by-row softmax operation on M0 to obtain the basic score map S0, that is, S0 = row_softmax(M0);

[0049] 2: The base row super anchor point P0 is defined as the row position expectation, and each anchor point position p of P0 is calculated i :

[0050]

[0051] where s i，j ∈S0, i=0, 1, ..., N row -1;

[0052] 3: Let k = 1;

[0053] 4: When k≤K, execute the loop;

[0054] 5: Keep the top k elements with the strongest response on each row of M0 and set the other elements to negative infinity. The server will be converted to 0 through the softmax operation; then a local row hyper-anchor mapping is obtained

[0055] 6: Calculate the supplementary score map S by repeating steps 1 and 2 k and super anchor P k ;

[0056] 7: End the loop

[0057] Output: Super anchor probability score map Super Anchor Set

[0058] Similarly, repeating steps 1 to 7 above can calculate the column super anchors. Each super anchor consists of a reference point on a redefined row and column. In addition to the super anchor P, a super anchor probability score map S is also obtained for constructing the lane descriptor in the subsequent steps. P and S contain K different elements, representing K possible lane positions;

[0059] Step S25: constructing row-by-row classification loss function and regression loss function to constrain super-anchor generation;

[0060] In order to make the generation of super-anchors more accurate, the embodiment of the present invention uses the traditional cross entropy loss function and regression loss function to constrain the training of the super-anchor generation network.

[0061] In one embodiment, the lateral features obtained by connecting the convolutional network to the feature pyramid network are combined with F b Fusion is performed to obtain a feature map F of the same size as S loc , F loc After adding it to the position encoding vector, it is multiplied point by point with S to obtain the lane descriptor D with an adaptive receptive field, which specifically includes:

[0062] Step S31: Use 1×1 convolution to reduce the number of channels of Fb to C fea At the same time, the lateral features obtained by connecting the convolutional network to the feature pyramid network are also reduced to C by using 1×1 convolution. fea, fuse the lateral features with Fb to obtain the feature

[0063] like Figure 2 As shown, in the lower branch of the super-anchor generation network, first a 1×1 convolution is applied to F b And reduce its channel number to C fea In addition, another 1×1 convolution is used to obtain the lateral connection between the convolutional network and the feature pyramid network. F loc The same size as the super anchor probability score map S in step S2, so that loc A one-to-one mapping is formed between them;

[0064] Step S32: Define lane descriptor K is the total number of super anchors for each lane. The lane descriptor is calculated using formula (5):

[0065]

[0066] Among them, S k ∈S, PE is the preset position encoding, which enables the lane descriptor to contain explicit position information; Perform element-wise multiplication and summation along the row or column direction; D k Including lane descriptor and train track description N an c is N row or N col , N lane N ego or N side .

[0067] Although convolutional networks can implicitly encode position information, the present invention adds a learnable position encoding PE to the F loc to provide explicit location information.

[0068] The present invention constructs the lane descriptor using formula (5) instead of adopting single point sampling that would result in a fixed receptive field of the lane descriptor, thereby being able to obtain more clues about the area near the lane line, and collecting local information of the adjacent area in this way, which helps to find the accurate position of the lane.

[0069] The super-anchor provides an overall structural reference for the lane. In order to better solve the lane line detection problem, the present invention further applies the self-attention mechanism to enhance the lane feature representation of a single lane and between different lanes.

[0070] In one embodiment, the above step S4: performing self-attention-based feature aggregation on the lane descriptor D respectively within the lane and between lanes to obtain D loc and D gbl The lane offset relative to the super anchor and the lane confidence are obtained through the two prediction heads respectively. The super anchor position and the offset are added to obtain the final lane line position, which specifically includes:

[0071] Step S41: Based on each lane descriptor D k , according to the predicted corresponding lane starting position index st k and end position index end k Create a mask and use self-attention-based feature aggregation only in the valid lane area to obtain the lane descriptor It is used to predict the offset, and its calculation formula (6) is as follows:

[0072]

[0073] Among them, mask loc According to the lane descriptor D k The starting position index st k The generated mask;

[0074] like Figure 4A As shown, the actual lane length is usually longer than the anchor length N row and N col Therefore, the lane grid is drawn in dark gray and light gray, and the background area is represented in white. Intra-lane feature aggregation is performed in the dark gray and light gray rectangular regions respectively, which can be effectively achieved through appropriate masked self-attention.

[0075] Step S42: Based on multiple different lane descriptors D k and D k+1 , using global feature aggregation based on self-attention to obtain lane-to-lane descriptors D gbl Other lane information is embedded to determine lane confidence, and its calculation formula (7) is as follows:

[0076]

[0077] Among them, mask gbl Based on D k and D k+1 The starting position index st k 、st k+1 and end position index end k 、end k+1 Generate a mask;

[0078] like Figure 4B As shown, an embodiment of the present invention proposes an inter-lane aggregation. Attention is calculated between different lane instances. Background features are ignored in the intra-lane and inter-lane aggregation of this embodiment of the present invention to improve computational efficiency and avoid introducing background noise into the lane descriptor. If no lane is detected, the identity matrix represented by the diagonal white line is used as the corresponding attention map. In the intra-lane aggregation. Similarly, if the number of valid lane segments in the inter-lane aggregation is less than 2, the identity matrix is ​​applied.

[0079] The in-lane descriptor D consists of the local details of a specific lane l oc is used to predict the accurate offset, and the lane-to-lane descriptor D gbl The information of other lane instances is embedded and can be used to determine lane confidence, such as Figure 2 As shown. D loc The predicted offsets help to obtain more accurate lane positions in challenging scenarios, such as extreme lighting and occlusion. Considering the global context, from D gbl The obtained lane confidence can reduce false positives. In addition, compared to applying fixed-direction spatial convolution to the entire feature map, lane-aware feature aggregation allows more flexible information flow direction and focuses on sparse lane regions.

[0080] Step S43: The two prediction heads composed of fully connected layers respectively predict the offset of the lane point relative to the super anchor and the confidence of the lane prediction, thereby determining the lane line position.

[0081] After aggregating local and global features based on the lane descriptor, two prediction heads, each consisting of fully connected layers, are used to predict the lane point offset relative to the super-anchor and the lane prediction confidence. The offset derived from local features effectively locates the exact lane line position, while the lane confidence derived from global features accurately describes the prediction reliability, helping to reduce the number of false lane detections.

[0082] The present invention discloses a lane line detection method based on super-anchor. A new type of lane anchor, namely super-anchor, is constructed for lane line detection. The super-anchor can provide the rough position of the lane line. Based on the super-anchor, a multi-level lane perception feature aggregation method is used to aggregate features within and between lane lines to capture the local features of the lane itself and the global features between lane lines, thereby obtaining the accurate position of the lane line.

[0083] Example 2

[0084] like Figure 5 As shown, an embodiment of the present invention provides a lane detection system based on a super-anchor, comprising the following modules:

[0085] The lane feature map acquisition module 51 is used to extract features from the forward traffic image through a convolutional network to obtain a lane feature map F b ;

[0086] Obtaining the super anchor probability score map module 52 is used to convert F b The super-anchor generation network is input to obtain the super-anchor probability score map S for each lane line. The super-anchor positions for the middle lane and the two side lanes are generated by calculating the position expectation of the super-anchor probability score map row by row and column by column.

[0087] Obtain lane descriptor module 53, which is used to connect the lateral features obtained by the convolutional network and the feature pyramid network laterally with F b Fusion is performed to obtain a feature map F of the same size as S loc , F loc After adding it to the position encoding vector, it is multiplied point by point with S to obtain the lane descriptor D with an adaptive receptive field;

[0088] The lane position calculation module 54 is used to perform self-attention-based feature aggregation on the lane descriptor D within and between lanes to obtain D loc and D gbl The lane offset relative to the super anchor and the lane confidence are obtained through the two prediction heads respectively. The super anchor position and the offset are added to obtain the final lane line position.

[0089] The above embodiments are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention are intended to be within the scope of the present invention.

Claims

1. A lane line detection method based on super anchor, characterized in that: include: Step S1: The forward traffic image is passed through the convolutional network for feature extraction to obtain the lane feature map F b ; Step S2: F b Input the super-anchor generation network to obtain the super-anchor probability score map S for each lane line; and calculate the position expectation of the super-anchor probability score map row by row and column by column to generate the super-anchor positions of the middle lane and the two side lanes respectively; Step S3: Connect the lateral features obtained by lateral connection of the convolutional network and the feature pyramid network to F b Fusion is performed to obtain a feature map F of the same size as S loc , F loc After adding it to the position encoding vector, it is multiplied point by point with S to obtain the lane descriptor D with an adaptive receptive field; Step S4: Perform self-attention-based feature aggregation on the lane descriptor D within and between lanes to obtain D loc and D gbl The lane offset relative to the super anchor and the lane confidence are obtained through two prediction heads respectively, and the super anchor position and the offset are added to obtain the final lane line position.

2. The lane line detection method based on super anchor according to claim 1, characterized in that: Step S2: F b Input the super-anchor generation network to obtain the super-anchor probability score map S for each lane line; The super anchor positions of the middle lane and the two side lanes are generated respectively by calculating the position expectation of the super anchor probability score map row by row and column by column. Specifically, the following steps are performed: Step S21: According to the input image I∈R H×W×3 , pre-define a set of evenly distributed rows, called row anchors Among them, N row is the number of rows; a lane line can be described by a set of lane points falling on the row anchor point; Step S22: Dynamically generate a set of reference points based on image I It is called super anchor. According to the following formulas (1) to (3), the lane line representation based on the super anchor is obtained Where Φ(·) represents super-anchor generation, I is the input image, and θ is the parameter of the super-anchor generation network; x i Indicates the row anchor point y i The actual location of the lane entry point; st and end represent the indices of the row anchor points of the start and end lane points respectively, and conf represents the confidence of the lane prediction; Step S23: Similarly, a uniformly distributed column of super anchor points can be obtained; therefore, the super anchor lane is expressed as formulas (4) to (5): Among them, the line super anchor Indicates the lane, super anchor Indicates side lane, N row and N col Represents the number of row and column anchors respectively; Step S24: F b Input the upper branch of the super anchor generation network, after passing through the 1×1 convolution layer and the fully connected layer, the output F gbl , where F gbl The number of channels is C gbl , where C gbl =N ego +N side , N ego and N side are the number of lanes on this lane and the lanes on both sides; F gbl It includes four groups of features, two of which are row super anchor features and column super anchor features, whose sizes are N row ×N col ×N ego and N row ×N col ×N side The outputs of are used to calculate the row super anchor and column super anchor respectively; for any lane, its corresponding super anchor feature is recorded as M, and the size is N row ×N col By calculating the softmax row by row or column by column, the super anchor probability score map S can be obtained, and S can be expanded by selecting K maximum response values, that is, S = softmax (M). The super anchor P is obtained by calculating the expected row or column position of S to obtain each anchor point position p of the super anchor P. i , as shown in formula (4): where s i,j ∈S; The other two sets of features are the lane start position feature and the lane end position feature, and their sizes are (N row +1)×2×N ego and (N col +1)×2×N ego Used to mark the start and end of the lane; (N row +1) and (N col +1) is in N row and N col On this basis, an additional output is introduced to indicate the situation where there is no lane; Step S25: Construct a row-by-row classification loss function and a regression loss function to constrain the generation of super-anchors.

3. The lane line detection method based on super anchor according to claim 2, characterized in that: Step S3: Connecting the lateral features obtained by lateral connection of the convolutional network and the feature pyramid network to the F b Fusion is performed to obtain a feature map F of the same size as S loc , F loc After adding it to the position encoding vector, it is multiplied point by point with S to obtain the lane descriptor D with an adaptive receptive field, which specifically includes: Step S31: Use 1×1 convolution to apply to F b Reduce the number of channels to C fea At the same time, the lateral features obtained by connecting the convolutional network to the feature pyramid network are laterally connected, and the number of channels is reduced to C by using 1×1 convolution. fea , the lateral features and F b Fusion to obtain features Step S32: Define lane descriptor K is the total number of super anchors for each lane. The lane descriptor is calculated using formula (5): Among them, S k ∈S, PE is a preset position code that enables the lane descriptor to contain explicit position information; Perform element-wise multiplication and summation along the row or column direction; D k Including lane descriptor and train track description N anc N row or N col , N lane N ego or N side .

4. The lane line detection method based on super anchor according to claim 3, characterized in that: Step S4: performing self-attention-based feature aggregation on the lane descriptor D within and between lanes to obtain D loc and D gbl The lane offset relative to the super anchor and the lane confidence are obtained through the two prediction heads respectively. The super anchor position and the offset are added to obtain the final lane position, which specifically includes: Step S41: Based on each lane descriptor D k , according to the predicted corresponding lane starting position index st k and end position index end k Create a mask and use self-attention-based feature aggregation only in the valid lane area to obtain the lane descriptor It is used to predict the offset, and its calculation formula (6) is as follows: Among them, mask loc According to the lane descriptor D k The starting position index st k The generated mask; Step S42: Based on a plurality of different lane descriptors D k and D k+1 , using global feature aggregation based on self-attention to obtain lane-to-lane descriptors D gbl Other lane information is embedded to determine lane confidence, and its calculation formula (7) is as follows: Among them, mask gbl Based on D k and D k+1 The starting position index st k 、st k+1 and end position index end k 、end k+1 Generate a mask; Step S43: The two prediction heads composed of fully connected layers respectively predict the offset of the lane point relative to the super anchor and the confidence of the lane prediction, thereby determining the lane line position.

5. A lane line detection system based on super anchor, characterized in that: Includes the following modules: The lane feature map acquisition module is used to extract features from the forward traffic image through a convolutional network to obtain a lane feature map F. b ; Obtain the super anchor probability score map module, which is used to convert F b Input the super-anchor generation network to obtain the super-anchor probability score map S for each lane line; and calculate the position expectation of the super-anchor probability score map row by row and column by column to generate the super-anchor positions of the middle lane and the two side lanes respectively; Obtain a lane descriptor module for connecting the convolutional network to the feature pyramid network to obtain the lateral features and F b Fusion is performed to obtain a feature map F of the same size as S loc , F loc After adding it to the position encoding vector, it is multiplied point by point with S to obtain the lane descriptor D with an adaptive receptive field; The lane position calculation module is used to perform self-attention-based feature aggregation on the lane descriptor D within and between lanes to obtain D loc and D gbl The lane offset relative to the super anchor and the lane confidence are obtained through two prediction heads respectively, and the super anchor position and the offset are added to obtain the final lane line position.

Images