Video lane line detection method and system based on instance feature guidance

Abstract

The invention discloses a video lane line detection method and system based on instance feature guidance, and the method comprises the steps: constructing a video lane line detection network which comprises a backbone network, a neck network, and a line anchor feature interaction fusion module; the backbone network and the neck network form two multi-level feature extraction branch networks sharing weights; obtaining a current frame image and a reference frame image, and respectively sending the two images into the two multi-level feature extraction branch networks to obtain multi-level image features of the two images; initializing line anchors distributed on the images, sending the line anchors and multi-level image features of the two images into a line anchor feature interaction fusion module, and gradually refining the positions of the line anchors from a high level to a low level to obtain a lane detection result; and training and optimizing the constructed video lane line detection network to obtain an optimal video lane line detection network for lane detection. Under the condition of a difficult scene, time sequence information contained in a video can be fully utilized, lane line instance features in a reference frame are combined to serve as supplement and prompt, a complete lane line instance can be effectively reasoned, detection of a current frame is assisted, and the accuracy and robustness of a model are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and autonomous driving technology, and in particular to a video lane line detection method and system based on instance feature guidance. Background Art

[0002] Lane line detection is an important component of autonomous driving systems and advanced driver assistance systems. The purpose of lane line detection is to help vehicles identify and track lane lines. Accurate lane line detection results can help vehicles achieve automatic cruising, lane keeping, lane departure warning and other functions to ensure stable and safe driving of the vehicle.

[0003] Traditional lane marking methods rely on edge detection, Hough transform, Kalman filtering and other methods. However, these methods are highly dependent on manually designed features and are difficult to cope with various complex scenarios, such as weather changes, low light and reflections, intersections, lane line merging or bifurcation and other challenges.

[0004] In recent years, with the rapid development of deep learning technology, lane detection methods based on deep learning have become the research focus of academia and industry. The first is the image-based deep learning method. According to the modeling method of lane lines, these methods can be divided into segmentation-based, parameter curve-based, key point-based, line anchor-based and other methods, such as "Real-time lane detection networks for autonomous driving", "A keypoint-based global association network for lane detection", "Cross layer refinement network for lane detection", etc. Compared with traditional methods, these methods are more capable of dealing with complex scenes, with better accuracy and robustness. However, this type of image-based method ignores the temporal information contained in the video captured by the on-board camera. With the development of video target detection and tracking technology, some video-based lane detection methods have been proposed, such as "A new dataset and a baseline model for video instance lane detection", "Recursive Video Lane Detection", "Enhancing Video Lane Detection with Temporal These methods achieve better detection results by fusing the information of historical frames with the current frame. However, the fusion granularity of these methods is at the picture level. The lane line detection task is different from the general target detection task in that the lane line has a slender shape and a large span. If we only focus on the feature fusion at the picture level, the background noise in the temporal information may reduce the robustness of the fused features.

[0005] The invention patent with the patent publication number CN118552924A discloses a lane line detection method based on starting point guidance. For the lane line detection network based on line anchors, a starting point detection head is introduced to more accurately locate the lane line, and a method of enhancing the hierarchical fusion feature using the starting point coordinates is designed. However, this method ignores the temporal information in the road video. When faced with scenes such as lane lines being blocked, the effect is poor because the current frame lacks sufficient semantic information to determine the existence of the lane line and locate the lane line.

[0006] The invention patent with the patent publication number CN118506307A discloses a real-time lane line detection method based on optical flow motion prediction. First, the multi-level feature map is aggregated, and the optical flow field between adjacent frames is calculated using the optical flow motion prediction algorithm to obtain the feature-refined current frame feature map, and the lane line is identified and detected by mixing anchor points. However, the optical flow field calculation in this method is large, and some noise in the time series information may also be introduced, resulting in a decrease in the representation ability of the fused features. Summary of the invention

[0007] The technical problem to be solved by the present invention is that the existing lane line detection technology does not make sufficient use of time series information.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] A video lane detection method based on instance feature guidance, comprising:

[0010] Construct a video lane detection network, including a backbone network, a neck network, and a line-anchor feature interactive fusion module; the backbone network and the neck network form two multi-level feature extraction branch networks with shared weights;

[0011] Obtain the current frame image and the reference frame image, and send the two images to two multi-level feature extraction branch networks respectively to obtain multi-level image features of the two images;

[0012] Initialize the line anchors distributed on the image and send them together with the multi-level image features of the two images into the line-anchor feature interactive fusion module to gradually refine the position of the line anchors from high level to low level to obtain the lane detection result;

[0013] The constructed video lane line detection network is trained and optimized to obtain the best video lane line detection network; the best video lane line detection network is used for lane detection.

[0014] Beneficial effects: Considering the continuity of lane lines between adjacent frames in road videos, while fusing global spatial information through line anchors and image features, the temporal information in the reference frame can be extracted more efficiently. Compared with the current video lane line detection method, this method only requires one reference frame. For each feature level, the line anchor feature interactive fusion module is used to selectively fuse the lane line instance features in the reference frame using the attention mechanism. During the reasoning process, the calculation results of the previous frame can be reused, which reduces the amount of calculation, improves the detection speed, and can also reduce the introduction of interfering background information in the historical frame.

[0015] In one embodiment of the present invention, the backbone network uses Resnet18, and the neck network uses the feature pyramid network FPN; multi-level image features of the two images are obtained, including:

[0016] For the previous frame image I t and the reference frame image I r Perform image enhancement;

[0017] The current frame image I after image enhancement t Send to the first multi-level feature extraction branch network; the current frame image I t First, the backbone network is used to extract the current frame image features of multiple different resolutions; then the current frame image features of multiple different resolutions are sent to the neck network for enhancement to obtain the multi-level image features of the current frame n represents the level of image features, and each level corresponds to different feature sizes and number of channels;

[0018] Similarly, the reference frame image I after image enhancement r Send it to the second multi-level feature extraction branch network to obtain the multi-level image features of the reference frame

[0019] In one embodiment of the present invention, a set of line anchors distributed on the image is initialized. These include: using numbers normalized to [0, 1] to represent line anchors on the image, and each line anchor p i Use (x i ,y i ,θ i ) indicates that it is a line starting from (x i ,y i ), and the angle between it and the positive direction of the X-axis is θ i ray; where i represents the number of the line anchor.

[0020] In one embodiment of the present invention, the lane detection result is obtained in the following manner:

[0021] The line-anchor feature interactive fusion module includes a plurality of line-anchor feature interactive fusion modules having the same structure and arranged in sequence;

[0022] In the first line-anchor feature interactive fusion module:

[0023] S310, performing RoI operations on the obtained initialized line anchors and the current frame multi-level image features and the reference frame multi-level image features of the current layer, respectively, to obtain the current frame line anchor features and the reference frame line anchor features;

[0024] S320, cross-attention interaction is performed on the line anchor feature of the current frame, the multi-level image feature of the current frame, and the multi-level image feature of the reference frame to obtain the line anchor feature that integrates the global information of the current frame and the reference frame and line anchor features

[0025] S330, cross-attention interaction is performed on the reference frame line anchor feature with the reference frame multi-level image feature and the current frame multi-level image feature to obtain the line anchor feature that integrates the global information of the reference frame and the current frame and line anchor features

[0026] S340, Line Anchor Feature After linear transformation, the line anchor feature is used as the query Line Anchor Features and line anchor features After connecting the channel dimension and performing linear transformation, the key and value are obtained respectively; the query, key and value are cross-attention interacted, and the features containing temporal and spatial global information are selectively integrated to obtain the line anchor features of enhanced representation;

[0027] S350, linearly projecting the enhanced represented line anchor features, and outputting parameters representing the line anchor confidence and the line anchor position;

[0028] S360, apply the line anchor confidence and line anchor position parameters output by the previous level line anchor feature interactive fusion module to optimize the line anchor position, and input the optimized line anchor into the next level line anchor feature interactive fusion module for further refinement, and repeat steps S310 to 350 until the last level line anchor feature interactive fusion module outputs the final detection result.

[0029] In one embodiment of the present invention, a line anchor feature that combines the global information of the current frame and the reference frame is obtained. Line Anchor Features Line Anchor Features and line anchor features The following steps are involved:

[0030] S321, in the multi-level image features of the current frame, for each line anchor, divide it into N s Anchor points are sampled to obtain the feature representation of the corresponding point position on the feature level, which is used as the feature representation of the line anchor in the current frame. Similarly, in the reference frame, the RoI operation of the multi-level image features is performed to obtain the feature representation of the line anchor on the reference frame.

[0031] S322, multi-level image features of the current frame and reference frame multi-level image features After scaling and flattening, the size is C*H a *W a Feature map Where C is the number of channels, H a and W a are height and width respectively;

[0032] S323, feature representation of line anchor in current frame After linear layer and layer normalization operations, the enhanced ROI feature representation is obtained

[0033] S324, the enhanced ROI feature representation After linear transformation Get query Q tt ; The current frame feature map obtained in step S322 After linear transformation and Get K tt and V tt , then Q tt With K tt and V tt Perform cross-attention interaction to establish the association between the line anchor feature and the global context of the current frame, and then output the line anchor feature through residual connection

[0034] S325, the enhanced ROI feature is represented After linear transformation Get query Q tr ; The reference frame feature map obtained in step S322 After linear transformation and Get K tr and V tr Through cross attention, the global context information of the reference frame is integrated into the feature representation of the line anchor, and the line anchor feature is output through residual connection.

[0035] Similarly, the feature representation of the line anchor on the reference frame As a query, repeat steps S323 to S324 to obtain the line anchor feature and line anchor features

[0036] In one embodiment of the present invention, the line anchor feature is obtained Line Anchor Features Line Anchor Features and line anchor features The calculation process is expressed by the following formula:

[0037]

[0038] Where Q' represents the query of the cross-attention interaction, K' represents the key of the cross-attention interaction, V' represents the value of the cross-attention interaction, O' represents the features of the query Q', key K' and value V' after the attention matrix transformation, softmax represents the activation function, T represents the matrix transpose, dropout represents the random failure of some neurons during the training process, and W O Expressed as the output linear transformation matrix, L i It is represented as the line anchor feature that integrates the global information of the current frame and the reference frame, d' k The vector dimensions are represented as K' and Q'.

[0039] In one embodiment of the present invention, the line anchor feature of the enhanced representation is obtained, and the formula is as follows:

[0040]

[0041] In the formula, It is represented as the feature after concatenation of channel dimensions, concat is represented as the concatenation function, and Q is represented as the line anchor feature After the linear transformation of the query, K and V are represented as line anchor features respectively. Line Anchor Features and line anchor features The keys and values ​​are obtained after concatenation of the channel dimensions and linear transformation; O represents the features of the query Q, key K and value V after transformation by the attention matrix, T represents the matrix transpose, d k It is expressed as, It represents the line anchor feature of enhanced representation, dropout represents the random failure of some neurons during training, and W O It is represented as the output linear transformation matrix, softmax is represented as the activation function, d k Represented as vector dimensions of K and Q.

[0042] In one embodiment of the present invention, S350, linearly projecting the line anchor features of the enhanced representation, and outputting parameters representing the line anchor confidence and the line anchor position, comprises the following steps: the linear projection layer comprises a classification branch and a regression branch, both of which are composed of a linear layer, a ReLU activation function, a linear layer, a ReLU activation function, and a linear layer connected in sequence, wherein the result P output by the classification branch cls , indicating N p The confidence of the line anchor, the result P output by the regression branch reg , including N p The horizontal and vertical coordinates of the starting point of the line anchor and the correction value Δx of the angle with the positive direction of the X axis 0 , Δy 0 and Δθ, and n r An x-direction offset ΔX, followed by Δx 0, Δy 0 and Δθ are used to adjust the position of the line anchor and input into the line-anchor feature interactive fusion module of the next level.

[0043] In one embodiment of the present invention, in the training phase, the acquired lane detection results are matched with the true value through dynamic matching, and the loss is calculated, back-propagated, and trained until the network converges, and the network parameters are saved; in the testing phase, the network parameters obtained in the training phase are loaded, and the prediction results with high confidence are screened out through the non-maximum suppression algorithm, and the final detection results are output.

[0044] The present invention also provides a video lane line detection system based on instance feature guidance, which applies any of the above-mentioned video lane line detection methods based on instance feature guidance, including:

[0045] Model building module, used to build a video lane detection network, including a backbone network, a neck network, and a line anchor feature interactive fusion module; the backbone network and the neck network form two multi-level feature extraction branch networks with shared weights;

[0046] A multi-layer extraction module is used to obtain the current frame image and the reference frame image, and send the two images to two multi-level feature extraction branch networks respectively to obtain multi-level image features of the two images;

[0047] The fusion detection module is used to initialize the line anchors distributed on the image and send them together with the multi-level image features of the two images to the line-anchor feature interactive fusion module to gradually refine the position of the line anchors from high level to low level to obtain the lane detection result;

[0048] The training optimization application module is used to train and optimize the constructed video lane line detection network to obtain the best video lane line detection network; and use the best video lane line detection network to perform lane detection.

[0049] Compared with the prior art, the beneficial effect of the present invention is that for each level of the feature pyramid extracted by the backbone network and the neck network, the present invention fully interactively fuses the scene information and lane line instance features in the reference frame and the current frame through the cross-attention mechanism, and then further screens and fuses the instance features after the initial fusion to enhance the feature representation of the lane line instance, and finally the feature is used to decode the actual lane line. In the line anchor feature interactive fusion module, for each level of the feature pyramid, from high-level features with low resolution but rich semantic information to low-level features with high resolution, the position of the line anchor is gradually refined. In this process, the scene information and lane line instance features in the reference frame and the current frame are fully interactively fused through the cross-attention mechanism. In some complex scenes, the temporal information contained in the reference frame can serve as an important supplement to effectively enhance the feature representation of the line anchor, and then the obtained line anchor features are selectively fused. The actual lane line is decoded in the line anchor feature interactive fusion module of the last layer. When the model faces various complex scenarios, such as weather changes, low light and reflections, lane line merging or bifurcation, etc., it can make full use of the temporal information contained in the video as a supplement to the current features. Compared with the existing video lane line detection method, while reducing the amount of calculation, it can also reduce the interference of redundant background information in the time series on the features of slender lane lines, thereby improving the accuracy and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The present invention is a flow chart of a video lane detection method based on instance feature guidance according to an embodiment of the present invention.

[0051] Figure 2 This is a network framework diagram of video lane detection according to an embodiment of the present invention.

[0052] Figure 3 Schematic diagram of initialization line anchor according to an embodiment of the present invention.

[0053] Figure 4 Schematic diagram of the line anchor feature interactive fusion module according to an embodiment of the present invention.

[0054] Figure 5 The present invention is a block diagram of a video lane detection system based on instance feature guidance according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to facilitate those skilled in the art to understand the technical solution of the present invention, the technical solution of the present invention is further described in conjunction with the accompanying drawings of the specification.

[0056] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0057] See also Figure 1 As shown, the present invention provides a video lane line detection method based on instance feature guidance, comprising:

[0058] S10, construct a video lane line detection network, including a backbone network, a neck network, and a line anchor feature interactive fusion module; the backbone network and the neck network form two multi-level feature extraction branch networks with shared weights.

[0059] See also Figure 1 and Figure 2 As shown, in one embodiment of the present invention, the backbone network adopts Resnet18, the neck network adopts the feature pyramid network FPN, and a plurality of line anchor feature interaction fusion modules STRM are provided.

[0060] S20, obtaining a current frame image and a reference frame image, and sending the two images to two multi-level feature extraction branch networks respectively to obtain multi-level image features of the two images.

[0061] In one embodiment of the present invention, the current frame image I is obtained. t , select a frame from the historical frames of the video as the reference frame image I r After data enhancement, the two images are sent to the backbone network and neck network with shared weights to extract multi-level image features. and Among them, 0, ..., n-1 represents the level of features, the 0th level features are the features with the lowest resolution, and the n-1th level are the features with the highest resolution.

[0062] In this embodiment, first, for current frame images I of different sizes, t , reference frame image I r Perform image enhancement. First, cut the upper 30% of the image area and finally scale it to W 0 ×H 0 , where W 0 Indicates the width of the transformed image, H 0 Indicates the height of the transformed image, which meets the size required by the network input. If it is the training stage, color change, affine transformation, and blur operation are also required for data enhancement.

[0063] In this embodiment, the backbone network is initialized with pre-trained weights, and the backbone network and the neck network are divided into two multi-level feature extraction branch networks with shared weights, which are used to extract features of the current frame and the reference frame respectively. Figure 2 As shown, taking the first multi-level feature extraction branch network as an example, the current frame image I t After the backbone network, four levels of features are extracted, with the height and width of the original image H 0 and W 0 1 / 4, 1 / 8, 1 / 16, 1 / 32 of the features are input into the neck network for fusion, and the in Indicates the fusion feature with the lowest resolution and the highest number of channels. Indicates the fusion feature with the highest resolution and the lowest number of channels; Figure 2 In the second multi-level feature extraction branch network shown in the figure, after similar operations, the fusion features of each level of the reference frame are obtained. In this embodiment, n=3.

[0064] S30, initialize the line anchors distributed on the image, and send them together with the multi-level image features of the two images into the line-anchor feature interactive fusion module, and gradually refine the positions of the line anchors from high level to low level to obtain lane detection results.

[0065] See also Figures 1 to 3 As shown, in one embodiment of the present invention, the line anchor set distributed on the image is initialized. These include: using numbers normalized to [0, 1] to represent the coordinates of the line anchors on the image, and each line anchor p i Use (x i ,y i ,θ i ) indicates that it is a line starting from (x i ,y i ), and the angle between it and the positive direction of the X-axis is θ i Where i is the number of the line anchor.

[0066] In this embodiment, the number of line anchors is set to N pIn this embodiment, there are 192, and the initialization method is to use a number normalized to [0, 1] to represent the coordinates on the image. In the range of [0.5, 1) on the left boundary of the image, 12 endpoints are evenly divided, and each endpoint generates 2 line anchors with horizontal angles of 28.8° and 57.6° respectively. In the range of [0.5, 1) on the right boundary of the image, 12 endpoints are evenly divided, and each endpoint generates 2 line anchors with horizontal angles of 122.4° and 151.2° respectively. In the range of [0, 1] on the lower boundary of the image, 36 endpoints are evenly divided, and each endpoint generates 4 line anchors with horizontal angles of 36°, 72°, 108°, and 144° respectively. Specifically, the line anchors start from the left boundary (blue in the figure), the right boundary (green in the figure), and the lower boundary (red in the figure) respectively.

[0067] See also Figures 1 to 4 As shown, in one embodiment of the present invention, in S30, obtaining the lane detection result includes the following steps:

[0068] S310, input the initialized line anchor set into the first STRM module, perform RoI operation with the current frame features and reference frame features of the current level, and obtain the line anchor features of the reference frame and the current frame respectively.

[0069] S320, the line anchor features of the current frame are cross-attentionally interacted with the image features of the current frame and the image features of the reference frame, respectively. S330, the line anchor features of the reference frame are cross-attentionally interacted with the image features of the reference frame and the image features of the current frame, respectively, to obtain four line anchor features that fuse the global information of the current frame and the reference frame.

[0070] In this embodiment, in this embodiment, the line anchor feature that fuses the global information of the current frame and the reference frame is obtained. Line Anchor Features Line Anchor Features and line anchor features Specifically include:

[0071] S321, for the line anchor set P i , where i represents the level of the feature pyramid, and the line anchor set initialized in step 3 is P 0 , respectively extract the region of interest (RoI) for the current frame and the reference frame. Specifically, taking the current frame feature as an example, for each line anchor, divide it into N in the vertical direction s Anchor points are sampled to obtain the feature representation of the corresponding point position on the feature level, which is used as the feature representation of the line anchor in the current frame. In the RoI operation of the reference frame feature, the feature representation of the line anchor on the reference frame can be obtained in the same way.

[0072] S322, for the feature map of the current frame and the reference frame and After scaling and flattening, the size is C*H a *W a Feature map and Where C is the number of channels, H a and W a are the height and width respectively. For all feature pyramid levels, feature maps of the same size will be obtained after scaling and flattening.

[0073] S323, feature representation of line anchor in current frame and After the linear layer and layer normalization operations, if the STRM module is not the first layer, it will also pass through the convolution layer and the linear layer to fuse the ROI features obtained in the previous layer to enrich the ROI features of the current layer and obtain an enhanced ROI feature representation. and

[0074] S324, for enhanced ROI feature representation After linear transformation Get query Q tt ; For the current frame feature map obtained in step S322 After linear transformation and Get K tt and V tt , then Q tt With K tt and V tt Perform cross-attention interaction to establish the association between the line anchor feature and the global context of the current frame, and then output it through residual connection Similarly, the enhanced ROI feature representation After linear transformation Get query Q tr , the reference frame feature map obtained in step S322 After linear transformation and Get K tr and V tr , through cross attention, the global context information of the reference frame is integrated into the feature representation of the line anchor, and the output is Similarly, the feature representation of the line anchor on the reference frame As a query, repeat steps S323 to S324 to obtain the line anchor feature and line anchor features

[0075] In this embodiment, in some difficult scenarios, such as when the lane line is blocked or damaged in the current frame, the timing information of the reference frame can be used as a supplement.

[0076] In this embodiment, the line anchor feature is obtained Line Anchor Features Line Anchor Features and line anchor features The calculation process is expressed by the following formula:

[0077]

[0078] Where Q' represents the query of the cross-attention interaction, K' represents the key of the cross-attention interaction, V' represents the value of the cross-attention interaction, O' represents the features of the query Q', key K' and value V' after the attention matrix transformation, softmax represents the activation function, T represents the matrix transpose, dropout represents the random failure of some neurons during the training process, and W O Expressed as the output linear transformation matrix, L i It is represented as the line anchor feature that integrates the global information of the current frame and the reference frame, d' k The vector dimensions are represented as K' and Q'.

[0079] S340, for line anchor features After the linear layer As a query, the channel dimension is connected After linear operations, the keys and values ​​are obtained respectively. After cross-attention interactions, the features containing temporal and spatial global information are selectively fused to obtain the line anchor features of enhanced representation.

[0080] In one embodiment of the present invention, the line anchor feature of the enhanced representation is obtained, and the formula is as follows:

[0081]

[0082] In the formula, It is represented as the feature after concatenation of channel dimensions, concat is represented as the concatenation function, and Q is represented as the line anchor feature After the linear transformation of the query, K and V are represented as line anchor features respectively. Line Anchor Features and line anchor features The keys and values ​​are obtained after concatenation of the channel dimensions and linear transformation; O represents the features of the query Q, key K and value V after transformation by the attention matrix, T represents the matrix transpose, d k It is expressed as, It represents the line anchor feature of enhanced representation, dropout represents the random failure of some neurons during training, and WO It is represented as the output linear transformation matrix, softmax is represented as the activation function, d k Represented as vector dimensions of K and Q.

[0083] S350, for the line anchor feature that integrates the enhanced representation of temporal and spatial global information acquisition, after linear projection, outputs parameters representing the line anchor confidence and position.

[0084] In this embodiment, the linear projection layer includes a classification branch and a regression branch, both of which are composed of a linear layer, a ReLU activation function, a linear layer, a ReLU activation function, and a linear layer connected in sequence, wherein the result P output by the classification branch is cls , indicating N p The confidence of the line anchor, the result P output by the regression branch reg , including N p The horizontal and vertical coordinates of the starting point of the line anchor and the correction value Δx of the angle between it and the positive direction of the X axis 0 , Δy 0 and Δθ, and N r An x-direction offset ΔX, followed by Δx 0 , Δy 0 and Δθ are used to adjust the position of the line anchor and input into the line-anchor feature interactive fusion module of the next level.

[0085] S360 uses the parameters output in S350 to optimize the position of the line anchors, and the optimized line anchors are input into the STRM module of the next level for further refinement until the last layer outputs the final detection result.

[0086] S40, training and optimizing the constructed video lane line detection network to obtain an optimal video lane line detection network; and applying the optimal video lane line detection network to perform lane detection.

[0087] In one embodiment of the present invention, in the training phase, the obtained lane detection results are matched with the true value through dynamic matching, and the loss is calculated, back propagated, and trained until the network converges, and the network parameters are saved. In the testing phase, the network parameters obtained in the training phase are loaded, and the prediction results with high confidence are screened out through the non-maximum suppression algorithm, and the final detection results are output.

[0088] In this embodiment, during the training phase, for the results obtained in step S30, each true value is dynamically matched with one or more prediction results, and the loss is calculated. The loss function includes classification loss, regression loss, and lane intersection loss.

[0089] In this embodiment, in the test phase, according to the result P of the classification branch cls, filter out the detection results with confidence greater than the threshold, the threshold is selected as 0.41, further, the non-maximum suppression algorithm is used to filter out the final lane line detection results. For the VIL-100 dataset, the upper limit of the number of output lane lines is set to 6.

[0090] Please refer to Table 1. In one embodiment of the present invention, the experimental results of the present invention on the VIL-100 dataset show that the mIoU and F1 0.8 The indicators have been greatly improved compared with the existing image-based and video-based algorithms.

[0091] Table 1 Comparison of experimental results on the VIL-100 dataset of the present invention

[0092]

[0093] See also Figures 1 to 5 As shown, the present invention also provides a video lane line detection system based on instance feature guidance, which applies the above-mentioned video lane line detection method based on instance feature guidance, including:

[0094] The model building module 10 is used to build a video lane line detection network, including a backbone network, a neck network, and a line anchor feature interactive fusion module. The backbone network and the neck network form two multi-level feature extraction branch networks with shared weights.

[0095] The multi-layer extraction module 20 is used to obtain the current frame image and the reference frame image, and send the two images to two multi-level feature extraction branch networks respectively to obtain multi-level image features of the two images.

[0096] The fusion detection module 30 is used to initialize the line anchors distributed on the image, and send them together with the multi-level image features of the two images to the line anchor feature interactive fusion module, and gradually refine the positions of the line anchors from high level to low level to obtain lane detection results.

[0097] The training optimization application module 40 is used to train and optimize the constructed video lane line detection network, obtain the best video lane line detection network, and apply the best video lane line detection network to perform lane detection.

[0098] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0099] The above-described embodiments merely represent implementation methods of the invention. The protection scope of the present invention is not limited to the above-described embodiments. For those skilled in the art, several modifications and improvements may be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A video lane detection method based on instance feature guidance, characterized in that: include: Construct a video lane detection network, including the backbone network, neck network, and line-anchor feature interactive fusion module; The backbone network and the neck network form two multi-level feature extraction branch networks with shared weights; Obtain the current frame image and the reference frame image, and send the two images to two multi-level feature extraction branch networks respectively to obtain multi-level image features of the two images; Initialize the line anchors distributed on the image and send them together with the multi-level image features of the two images into the line-anchor feature interactive fusion module to gradually refine the position of the line anchors from high level to low level to obtain the lane detection result; The constructed video lane line detection network is trained and optimized to obtain the best video lane line detection network; the best video lane line detection network is used for lane detection.

2. The video lane detection method based on instance feature guidance according to claim 1 is characterized in that: The backbone network uses Resnet18, and the neck network uses the feature pyramid network FPN; multi-level image features of the two images are obtained, including: For the previous frame image I t and the reference frame image I r Perform image enhancement; The current frame image I after image enhancement t Send to the first multi-level feature extraction branch network; the current frame image I t First, the backbone network is used to extract the current frame image features of multiple different resolutions; then the current frame image features of multiple different resolutions are sent to the neck network for enhancement to obtain the multi-level image features of the current frame n represents the level of image features, and each level corresponds to different feature sizes and number of channels; Similarly, the reference frame image I after image enhancement r Send it to the second multi-level feature extraction branch network to obtain the multi-level image features of the reference frame 3. The video lane detection method based on instance feature guidance according to claim 1 is characterized in that: Initialize the set of line anchors distributed on the image These include: using numbers normalized to [0, 1] to represent the coordinates of the line anchors on the image, and each line anchor p i Use (x i ,y i ,θ i ) indicates that it is a line starting from (x i ,y i ), and the angle between it and the positive direction of the X-axis is θ i ray; where i represents the number of the line anchor.

4. The video lane detection method based on instance feature guidance according to claim 1 is characterized in that: Get lane detection results in the following ways: The line-anchor feature interactive fusion module includes a plurality of line-anchor feature interactive fusion modules having the same structure and arranged in sequence; In the first line-anchor feature interactive fusion module: S310, performing RoI operations on the obtained initialized line anchor set and the current frame multi-level image features and the reference frame multi-level image features of the current layer, respectively, to obtain the current frame line anchor features and the reference frame line anchor features; S320, cross-attention interaction is performed on the line anchor feature of the current frame, the multi-level image feature of the current frame, and the multi-level image feature of the reference frame to obtain the line anchor feature that integrates the global information of the current frame and the reference frame and line anchor features S330, cross-attention interaction is performed on the reference frame line anchor feature with the reference frame multi-level image feature and the current frame multi-level image feature to obtain the line anchor feature that integrates the global information of the reference frame and the current frame and line anchor features S340, Line Anchor Feature After linear transformation, the line anchor feature is used as the query Line Anchor Features and line anchor features After connecting the channel dimension and performing linear transformation, the key and value are obtained respectively; the query, key and value are cross-attention interacted, and the features containing temporal and spatial global information are selectively integrated to obtain the line anchor features of enhanced representation; S350, linearly projecting the enhanced line anchor features, and outputting parameters representing the line anchor confidence and the line anchor position; S360, apply the line anchor confidence and line anchor position parameters output by the previous level line anchor feature interactive fusion module to optimize the line anchor position, and input the optimized line anchor into the next level line anchor feature interactive fusion module for further refinement, and repeat steps S310 to 350 until the last level line anchor feature interactive fusion module outputs the final detection result.

5. The video lane detection method based on instance feature guidance according to claim 4 is characterized in that: Get the line anchor feature that combines the global information of the current frame and the reference frame Line Anchor Features Line Anchor Features and line anchor features The following steps are involved: S321, in the multi-level image features of the current frame, for each line anchor, divide it into N s Anchor points are sampled to obtain the feature representation of the corresponding point position on the feature level, which is used as the feature representation of the line anchor in the current frame. Similarly, in the reference frame, the RoI operation of the multi-level image features is performed to obtain the feature representation of the line anchor on the reference frame. S322, multi-level image features of the current frame and reference frame multi-level image features After scaling and flattening, the size is C*H a *W a Feature map Where C is the number of channels, H a and W a are height and width respectively; S323, feature representation of line anchor in current frame After linear layer and layer normalization operations, the enhanced ROI feature representation is obtained S324, the enhanced ROI feature representation After linear transformation Get query Q tt ; The current frame feature map obtained in step S322 After linear transformation and Get K tt and V tt , then Q tt With K tt and V tt Perform cross-attention interaction to establish the association between the line anchor feature and the global context of the current frame, and then output the line anchor feature through residual connection S325, the enhanced ROI feature is represented After linear transformation Get query Q tr ; The reference frame feature map obtained in step S322 After linear transformation and Get K tr and V tr Through cross attention, the global context information of the reference frame is integrated into the feature representation of the line anchor, and the line anchor feature is output through residual connection. Similarly, the feature representation of the line anchor on the reference frame As a query, repeat steps S323 to S324 to obtain the line anchor feature and line anchor features 6. The video lane detection method based on instance feature guidance according to claim 5 is characterized in that: Get line anchor feature Line Anchor Features Line Anchor Features and line anchor features The calculation process is expressed by the following formula: Where Q' represents the query of the cross-attention interaction, K' represents the key of the cross-attention interaction, V' represents the value of the cross-attention interaction, O' represents the features of the query Q', key K' and value V' after the attention matrix transformation, softmax represents the activation function, T represents the matrix transpose, dropout represents the random failure of some neurons during the training process, and W O Expressed as the output linear transformation matrix, L i It is represented as the line anchor feature that integrates the global information of the current frame and the reference frame, d' k The vector dimensions are represented as K' and Q'.

7. The video lane detection method based on instance feature guidance according to claim 5 is characterized in that: Obtain the line anchor feature of enhanced representation, the formula is as follows: In the formula, It is represented as the feature after concatenation of channel dimensions, concat is represented as the concatenation function, and Q is represented as the line anchor feature After the linear transformation of the query, K and V are represented as line anchor features respectively. Line Anchor Features and line anchor features The keys and values ​​are obtained after concatenation of the channel dimensions and linear transformation; O represents the features of the query Q, key K and value V after transformation by the attention matrix, T represents the matrix transpose, d k It is expressed as, It represents the line anchor feature of enhanced representation, and dropout represents the random failure of some neurons during training. O It is represented as the output linear transformation matrix, softmax is represented as the activation function, d k Represented as vector dimensions of K and Q.

8. The video lane detection method based on instance feature guidance according to claim 4 is characterized in that: S350, linearly projecting the enhanced line anchor features, and outputting parameters representing the line anchor confidence and the line anchor position, including the following steps: the linear projection layer includes a classification branch and a regression branch, both of which are composed of a linear layer, a ReLU activation function, a linear layer, a ReLU activation function, and a linear layer connected in sequence, wherein the result P output by the classification branch cls , indicating N p The confidence of the line anchor, the result P output by the regression branch reg , including N p The horizontal and vertical coordinates of the starting point of each line anchor and the correction values ​​of the angle between the anchor and the positive X-axis, Δx0, Δy0 and Δθ, as well as N r The offset ΔX in the x direction is then used to adjust the position of the line anchor and input into the line anchor feature interactive fusion module of the next level.

9. The video lane detection method based on instance feature guidance according to claim 1, characterized in that: In the training phase, the acquired lane detection results are matched with the true value through dynamic matching, and the loss function is calculated, back-propagated, trained until the network converges, and the network parameters are saved; In the testing phase, the network parameters obtained in the training phase are loaded, and the prediction results with high confidence are screened out through the non-maximum suppression algorithm to output the final detection results.

10. A video lane detection system based on instance feature guidance, characterized in that: The video lane line detection method based on instance feature guidance according to any one of claims 1 to 9 is applied, comprising: Model building module, used to build a video lane detection network, including a backbone network, a neck network, and a line anchor feature interactive fusion module; the backbone network and the neck network form two multi-level feature extraction branch networks with shared weights; A multi-layer extraction module is used to obtain the current frame image and the reference frame image, and send the two images to two multi-level feature extraction branch networks respectively to obtain multi-level image features of the two images; The fusion detection module is used to initialize the line anchors distributed on the image and send them together with the multi-level image features of the two images to the line-anchor feature interactive fusion module to gradually refine the position of the line anchors from high level to low level to obtain the lane detection result; The training optimization application module is used to train and optimize the constructed video lane line detection network to obtain the best video lane line detection network; and use the best video lane line detection network to perform lane detection.

Images