Lane Line Detection Method Based on Geometric Feature Extraction and Location Information Encoding

Abstract

The present invention discloses a lane line detection method based on geometric feature extraction and position information encoding, which solves the problems of low accuracy and large computational complexity in the prior art due to the lack of consideration of lane line geometric features and position information. The steps implemented by the present invention include: building a preliminary feature extraction sub-network; building a geometric feature extraction module; building a position information encoding module; constructing an asymmetric bilateral self-attention network; generating a training set; training the asymmetric bilateral self-attention network; and detecting lane lines. The designed geometric feature extraction module and position information encoding module of the present invention can fully extract and utilize the geometric features and position information of lane lines, thereby improving the performance of lane line detection as a whole. At the same time, bar-shaped convolution kernels are used instead of square convolution kernels to reduce the computational complexity, achieving a better balance effect on the accuracy and timeliness of lane line detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and further relates to a lane line detection method based on geometric feature extraction and position information coding in the field of image segmentation. The present invention can be used to detect lane lines during the driving of a motor vehicle through an autonomous driving device. Background Art

[0002] In lane line detection, the exact shape of each lane is obtained from a two-dimensional front view image captured by a camera mounted on a vehicle. In addition to obtaining the direction and shape of the lane lines, the system must also be able to distinguish each lane. For this reason, lane line detection is often regarded as a semantic segmentation task. Most of the previous work has adopted a pure segmentation method, that is, capturing its semantic features through an encoder and then performing post-processing by decoding to obtain the final result. Lane line detection is a relatively special detection problem, which has challenges different from other detection problems, such as high noise (multiple shadow occlusions), a low proportion of pixels belonging to positive samples in the whole image (lane lines are slender), diverse forms (lane lines with different colors and shapes), and diverse scenarios (affected by weather factors). Therefore, if it is only regarded as a traditional semantic segmentation problem and a pure segmentation method is used, it cannot effectively handle some complex scenarios such as shadows, strong light, and night, and has a small scope of application. Moreover, for lane lines, in addition to the most basic semantic features, there is also an important feature that lane lines belong to artificial objects, which is different from general natural objects. They have strong prior geometric features (lane lines are generally slender parallelogram structures, and better detection performance can be obtained by using their geometric features). At the same time, there is position information between lane lines. For example, lane lines in the real world are parallel. Therefore, if position information is not considered, some detection performance will also be lost.

[0003] East China Jiaotong University proposed a method for an autonomous driving device to detect motor vehicle lane lines in its patent document "A Lane Line Detection Method Based on Deep Learning" (Application No.: 2022110592895, Publication No.: CN 115376089 A, Publication Date: November 22, 2022). The implementation steps of this method are as follows: (1) Perform multi-scale image enhancement operations on the existing image samples in the dataset, and then generate training samples according to the marked lane line position information and the image; (2) Construct a suitable convolutional neural network based on the generated images; (3) Use the generated lane line images for training; (4) Evaluate the performance of this convolutional neural network; (5) Use the trained model for lane line detection. The advantage of this method is that it enhances the robustness of the overall model by combining deep learning methods and can effectively detect straight and curved lanes in the case of lane line breakage, occlusion, and shadow. However, the still existing deficiency of this method is that it regards lane line detection as a simple pixel segmentation task, and its network structure is completely composed of ordinary two-dimensional convolutional layers and activation layers, neither capturing the geometric features of lane lines nor considering the position information of lane lines. This results in poor detection performance or even failure to detect lane lines in some scenarios where the semantic information of lane lines is not obvious or even missing.

[0004] Guangdong University of Technology proposed a method for an autonomous driving device to detect motor vehicle lane lines based on a self-attention structure in its patent document "Lane Line Detection Method, System, Computer, and Storage Medium Based on Deep Learning" (Application No.: 2021112642095, Publication No.: CN114022863A, Publication Date: February 8, 2022). The implementation steps are as follows: (1) Use a traffic monitoring camera to shoot videos with lane lines and intercept multiple images with lane lines from the videos to generate a dataset; (2) Preprocess the generated dataset and divide the preprocessed dataset into a training set and a test set; (3) Construct a dual-branch lane line detection model, with a feature extractor and an image segmenter set in each branch; (4) Train the dual-branch lane line detection model with the training set; (5) Detect the test set with the trained dual-branch lane line detection model. This invention introduces an attention mechanism that can more effectively extract image features, especially global information, and the global information can reflect some situations of the lane lines themselves. However, the still existing deficiency of this method is that it still does not consider the position information relationship between lane lines. As a result, when an autonomous driving vehicle encounters a situation where multiple lane lines are arranged closely during driving, it will identify them as one lane line, which in turn leads to the problem of poor detection effect of the lane line detection technology. Summary of the Invention

[0005] The object of the present invention is to provide a lane line detection method based on geometric feature extraction and position information encoding for the deficiencies of the above-mentioned existing technologies, so as to solve the problems of poor detection effect, large computational complexity and low detection accuracy of lane line detection in complex road scenarios in the lane line detection technology.

[0006] The technical idea for realizing the object of the present invention is that the present invention first uses ResNet as an encoder for preliminary feature extraction, and then constructs a geometric feature extraction module and a position information encoding module respectively to process the feature map in parallel up and down. The geometric feature extraction module performs attention operations row by row and from left to right within the row in the form of strip-shaped convolutional kernels. The strip-shaped convolutional kernels fit the geometric shape of the lane lines and can also reduce the computational complexity, and the form of attention within the row can effectively fuse local and global information. Since the strip-shaped convolutional kernels are used instead of the traditional square convolutional kernels, the number of parameters and the computational amount are reduced, alleviating the problem of large computational amount in the existing technology. At the same time, compared with the square convolutional kernels, the strip-shaped convolutional kernels often span the entire feature map and have a larger receptive field in a certain direction, which can provide more context information, help capture long-range dependencies, and solve the deficiency of low detection accuracy of lane line detection in complex road scenarios. When constructing the position information encoding module of the present invention, a relative position vector will be generated first, and then the position vector will be incorporated into the transformer to form an attention mechanism with position information encoding. The position vector encoding can effectively capture the potential position relationship between the lane lines themselves and between the lane lines. Through the parallel processing of the above two methods, finally feature fusion is performed, and then double-branch decoding is performed to obtain the instance branch result and the existence branch result respectively, and then decoding is performed to obtain the lane line detection result. The potential position information of the lane line distribution in the image is mined and captured, and the position vector is directly encoded. Since the situation of multiple closely spaced lane lines generally only exists near the vanishing point of the road surface, the lane lines near the vehicle body are relatively scattered, which is determined by the camera shooting angle. The position encoding of the present invention is initially formed by rearranging the global encoding matrix into a vector. That is to say, for the same picture, the position encoding information of the relatively scattered lane lines will also act on the lane line dense area, and the lane lines in the lane line scattered area are easy to classify, which helps the lane line classification in the lane line dense area and overcomes the defect that in the prior art, when facing the situation where multiple lane lines are arranged closely during the driving of an autonomous vehicle, they will be recognized as one lane line with poor detection effect.

[0007] The implementation steps of the present invention are as follows:

[0008] Step 1, construct a preliminary feature extraction sub-network;

[0009] Step 2, construct a geometric feature extraction module:

[0010] Construct a geometric feature extraction module composed of a high-level feature input layer, an intra-line self-attention sub-module, an inter-line self-attention sub-module, and a geometric feature output layer connected in series in sequence;

[0011] The intra-line self-attention sub-module consists of two branches. The first branch is composed of an input layer, a first-dimensional rearrangement layer, a linear layer, a segmentation layer, and a self-attention layer connected in series in sequence; the second branch is composed of a matrix addition layer, a second-dimensional rearrangement layer, and an output layer connected in series in sequence; the matrix addition layer in the second branch is connected across the input layer and the self-attention layer of the first branch;

[0012] The inter-line self-attention sub-module consists of two branches. The first branch is composed of an inter-line self-attention input layer, a convolutional layer, a first segmentation layer, a stacking layer, a first-dimensional rearrangement layer, a second segmentation layer, and a self-attention layer connected in series in sequence. The second branch is composed of a matrix addition layer, a second-dimensional rearrangement layer, and an inter-line self-attention output layer connected in series in sequence. The matrix addition layer is connected across the inter-line self-attention and the self-attention layer of the first branch; set the convolutional kernel size of the convolutional layer to 1×1, the stride to 1, and the padding to 0, and the self-attention layer adopts standard self-attention operations;

[0013] Step 3, construct a position information encoding module:

[0014] Construct a position information encoding module composed of a low-level feature input layer, a vector encoding layer, a self-attention layer, and a position information encoding output layer connected in series in sequence; the vector encoding layer is composed of a relative position vector encoding layer and a qkv feature generation layer connected in parallel; the self-attention layer in the position information encoding module will perform multi-head attention operations;

[0015] The relative position vector encoding layer is composed of a relative position matrix generation layer, a flattening layer, a subscript selection layer, and a segmentation layer connected in series in sequence;

[0016] The qkv feature generation layer is composed of a first-dimensional rearrangement layer, a convolutional layer, a normalization layer, a second-dimensional rearrangement layer, and a segmentation layer connected in series in sequence; set the convolutional kernel size of the convolutional layer to 1×1, the stride to 1, and the padding to 0;

[0017] The self-attention layer consists of two branches. The first matrix addition layer in the first branch is respectively connected to the first matrix multiplication layer, the second matrix multiplication layer, and the third matrix multiplication layer; the softmax layer in the second branch is respectively connected to the second matrix addition layer, the fourth matrix multiplication layer, and the fifth matrix multiplication layer. The second matrix addition layer is respectively connected to the fourth matrix multiplication layer, the fifth matrix multiplication layer, the connected softmax layer, and the first matrix addition layer of the first branch;

[0018] Step 4, construct an asymmetric bilateral attention network:

[0019] Step 4.1, construct a feature fusion layer;

[0020] Step 4.2, build a decoding prediction sub-network composed of an instance branch and an existence branch in parallel;

[0021] Step 4.3, connect a geometric feature extraction module and a position information encoding module in parallel to form a module group;

[0022] Step 4.4, cascade the preliminary feature extraction sub-network, the module group, the feature fusion layer, and the decoding prediction sub-network into an asymmetric bilateral attention network;

[0023] Step 5, generate a training set:

[0024] Select at least 3000 images containing lane lines to form a sample set. The resolution of each image is 1280×720, and each image includes 0 to 4 lane lines. At the same time, each image corresponds to a binary segmentation label map; perform preprocessing of cropping, rotation, and blurring on each image in the sample set in turn; form a training set with the preprocessed images;

[0025] Step 6, train the asymmetric bilateral attention network:

[0026] Step 6.1, set training parameters: set the momentum rate to 0.9, the weight decay rate to 0.0004, and the initial learning rate to 0.02;

[0027] Step 6.2, initialize the weights. Use the weights of ResNet-34 pre-trained on ImageNet as the initial weights of the asymmetric bilateral attention network. Adopt the SGD optimization algorithm, input the training set into the asymmetric bilateral attention network batch by batch, and use the gradient descent method to iteratively update the network parameters until the total loss function of the network converges. Obtain the trained network and save the weights;

[0028] Step 7, detect lane lines:

[0029] Adopt the same method as in Step 5, input the preprocessed images containing lane lines to be detected into the trained asymmetric bilateral attention network in turn, and output the lane line detection results.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] First, the present invention constructs a geometric feature extraction module, improves the self-attention mechanism by using a strip-shaped convolution kernel, overcomes the deficiencies of large computational complexity in the prior art and low detection accuracy of lane lines in complex road scenes, and enables the present invention to better balance the lane line detection accuracy and timeliness.

[0032] Second, the present invention constructs a position information encoding module, which mines and captures the potential position information of the lane line distribution in the image, directly encodes the position vector, and embeds it into the attention module. This overcomes the defect of the prior art that when facing the situation where multiple lane lines are arranged closely during the driving process of an autonomous vehicle, it will recognize them as one lane line with poor detection effect. It enables the autonomous vehicle of the present invention to make full use of the position information between lane lines during the driving process, effectively improving the accuracy and robustness of lane line detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flowchart of the present invention;

[0034] Figure 2 is a structural diagram of the asymmetric bilateral attention network of the present invention;

[0035] Figure 3 is a structural diagram of the preliminary feature extraction sub-network of the present invention;

[0036] Figure 4 is a structural diagram of the geometric feature module of the present invention;

[0037] Figure 5 is a structural diagram of the intra-row self-attention sub-module in the geometric feature module of the present invention;

[0038] Figure 6 is a structural diagram of the inter-row self-attention sub-module in the geometric feature module of the present invention;

[0039] Figure 7 is a structural diagram of the position information encoding module of the present invention;

[0040] Figure 8 is a structural diagram of the relative position vector encoding layer in the position information encoding module of the present invention;

[0041] Figure 9 is a structural diagram of the qkv feature generation layer in the position information encoding module of the present invention;

[0042] Figure 10 is a structural diagram of the self-attention layer in the position information encoding module of the present invention;

[0043] Figure 11 is a structural diagram of the instance branch in the decoding and prediction sub-network of the present invention;

[0044] Figure 12 is a structural diagram of the existence branch in the decoding and prediction sub-network of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] Refer to Figure 1 , and the implementation steps of the embodiments of the present invention will be further described.

[0047] The asymmetric bilateral attention network in the embodiments of the present invention includes a preliminary feature extraction sub-network, a geometric feature extraction module, a position information encoding module, and a decoding prediction sub-network, as Figure 2 shown.

[0048] Step 1: Construct the preliminary feature extraction sub-network in the asymmetric bilateral attention network.

[0049] Build a preliminary feature extraction sub-network composed of an image input layer, a preliminary feature extraction layer, and an output layer group connected in series in sequence. The output layer group is composed of a high-level feature output layer and a low-level feature output layer connected in parallel, as Figure 3 shown.

[0050] The input layer is the input of the lane line picture. The preliminary feature extraction layer uses ResNet-34 to perform preliminary feature extraction on the input lane line picture. After preliminary feature extraction, the direct output of ResNet-34 is used as the output of the high-level feature output layer, and the first layer feature map of ResNet-34 is used as the output of the low-level feature.

[0051] Step 2: The geometric feature extraction module in the asymmetric bilateral attention network.

[0052] Build a geometric feature extraction module composed of a high-level feature input layer, an intra-row self-attention sub-module, an inter-row self-attention sub-module, and a geometric feature output layer connected in series in sequence, as Figure 4 shown.

[0053] The intra-row self-attention sub-module is composed of two branches. The first branch is composed of an input layer, a first-dimension rearrangement layer, a linear layer, a segmentation layer, and a self-attention layer connected in series in sequence; the second branch is composed of a matrix addition layer, a second-dimension rearrangement layer, and an output layer connected in series in sequence; the matrix addition layer in the second branch is connected across the input layer and the self-attention layer of the first branch, as Figure 5 shown.

[0054] When the feature with the size of (B, C, H, W) Figure XAfter being input into the intra-row self-attention sub-module, first pass through the first-dimension rearrangement layer to rearrange the X dimension into (B*H, W, C), and then expand in the channel dimension through the linear layer to obtain a feature map with a size of (B*H, W, 3C). After passing through the segmentation layer, the feature map with a size of (B*H, W, 3C) is cut into three parts in the channel dimension, respectively obtaining three feature maps with a scale of (B*H, W, C). These three feature maps are respectively used as query, key, and value and input into the self-attention layer. After the standard self-attention operation of this layer, the operation result is added to the feature Figure X through the matrix addition layer, and then its size is changed to (B, C, H, W) through the second-dimension rearrangement layer. This result is used as the output, that is, the result of the intra-row self-attention sub-module. Among them, B represents the number of feature maps in one operation, C represents the number of channels of the feature map, H represents the height of the feature map, and W represents the width of the feature map.

[0055] The described inter-row self-attention sub-module consists of two branches. The first branch is composed of an inter-row self-attention input layer, a convolutional layer, a first segmentation layer, a stacking layer, a first-dimension rearrangement layer, a second segmentation layer, and a self-attention layer connected in series in sequence. The second branch is composed of a matrix addition layer, a second-dimension rearrangement layer, and an inter-row self-attention output layer connected in series in sequence. The matrix addition layer is connected across the inter-row self-attention and the self-attention layer of the first branch, as Figure 6 shown. Set the convolution kernel size of the convolutional layer to 1×1, the stride to 1, and the padding to 0. The self-attention layer adopts the standard self-attention operation.

[0056] When the feature map with a size of (B, C, H, W) enters the inter-row self-attention sub-module, after passing through a convolutional layer with a convolution kernel size of 1×1, a feature map with a size of (B, 3C, H, W) is obtained. After passing through the segmentation layer, it is cut into three parts in the channel dimension, respectively obtaining three feature maps with a scale of (B, C, H, W); after passing through the stacking layer, tensors are stacked in the first dimension to obtain a feature map with a scale of (3B, C, H, W), and then the dimension is rearranged to (3B, H, W*C), and cut into three parts according to the first dimension, and the size of each part is (B, H, W*C). These three tensors are respectively used as query, key, and value and input into the standard self-attention module. The result is added to the feature map of the input layer through matrix addition, and then the scale is changed to (B, C, H, W) through the dimension rearrangement layer, and finally the output result is obtained, that is, the result of the inter-row self-attention.

[0057] Step 3, construct the position information encoding module in the asymmetric bilateral attention network.

[0058] Build a position information encoding module composed of a low-level feature input layer, a vector encoding layer, a self-attention layer, and a position information encoding output layer connected in series in sequence. The vector encoding layer is composed of a relative position vector encoding layer and a qkv feature generation layer connected in parallel, as Figure 7 shown.

[0059] The self-attention layer in the position information encoding module will perform multi-head attention operations. In the embodiments of the present invention, the grouping number G of the multi-head attention is set to 8.

[0060] The relative position vector encoding layer is composed of a relative position matrix generation layer, a flattening layer, a subscript selection layer, and a segmentation layer connected in series in sequence, as Figure 8 shown.

[0061] When a feature map with a size of (B, C, H, W) enters the relative position vector encoding layer, a constant sequence from 0 to 79 is first generated in the relative position matrix generation layer, expanded in the 0th dimension and the 1st dimension respectively, and the latter is subtracted from the former (here the broadcasting mechanism of tensors is utilized) and then 79 is added to obtain a relative position matrix. It is stretched into a vector through the flattening layer to obtain a relative position sequence. In the subscript selection layer, a random number matrix with a size of (2C / G, 80 * 2 - 1) is first generated by using a random function that follows the standard normal function. Taking the relative position sequence as the standard for subscript selection, the random number matrix is rearranged in the 2nd dimension to obtain a tensor with a size of (2C / G, 80, 80). Finally, it is divided into 3 parts in the 1st dimension to obtain 3 position information encoding vectors with sizes of (C / 2G, 80, 80), (C / 2G, 80, 80), and (C / G, 80, 80) respectively, denoted as l q 、l k and l v .

[0062] The qkv feature generation layer is composed of a 1st dimension rearrangement layer, a convolutional layer, a normalization layer, a 2nd dimension rearrangement layer, and a segmentation layer connected in series in sequence, as Figure 9 shown. The convolutional kernel size of the convolutional layer is set to 1×1, the stride is set to 1, and the padding is set to 0; the normalization operation of the normalization layer is performed in the second dimension, and the input and output channels are the same.

[0063] When the feature map of size (B, C, H, W) enters the qkv feature generation layer, its dimensions are first rearranged to (B*W, C, H), and then it passes through a convolution layer with a kernel size of 1×1, a stride of 1, and padding of 0. A BatchNorm operation, i.e., a normalization layer, is performed on the second dimension to obtain a feature map of size (B*W, 2C, H). The feature map is rearranged to obtain a feature map of size (B*W, G, 2C / G, H), and then split in the third dimension to obtain three tensors with sizes of (B*W, G, C / 2G, H), (B*W, G, C / 2G, H), and (B*W, G, C / G, H), respectively, which are denoted as X q , X k and X v .

[0064] The self-attention layer consists of two branches. The first matrix addition layer in the first branch is connected to the first matrix multiplication layer, the second matrix multiplication layer, and the third matrix multiplication layer. The softmax layer in the second branch is connected to the second matrix addition layer, the fourth matrix multiplication layer, and the fifth matrix multiplication layer. The second matrix addition layer is connected to the fourth matrix multiplication layer, the fifth matrix multiplication layer and the connected softmax layer as well as the first matrix addition layer of the first branch. Figure 10 shown.

[0065] The results of the relative position encoding layer and the qkv feature generation layer are passed through the 1st, 2nd, and 3rd matrix multiplication layers to obtain X q × q , X k × k and X q ×X k , after stacking in the first dimension, they are rearranged into (B*W,3,G,H,H), and accumulated in the second dimension, and then softmax operation is performed on the fourth dimension to obtain (B*W,G,H,H). The results are respectively passed through the fourth matrix multiplication layer and l v Multiply and the 5th matrix multiplication layer with X v Multiply them together and then add them together through the second matrix addition layer to get the final result.

[0066] Step 4: Build a feature fusion layer consisting of a tensor stacking layer and a convolutional layer in series, set the convolution kernel size of the convolutional layer to 1×1, the stride to 1, and no padding.

[0067] Step 5: Build a decoding prediction subnetwork consisting of instance branch and existence branch in parallel.

[0068] In step 5.1, the instance branch is composed of a dropout layer, a convolutional layer, and an upsampling layer in series, as shown in Figure 11As shown in the figure. Among them, the dropout probability in the dropout layer is 0.1; the convolution kernel size of the convolution layer is set to 1×1, the stride is set to 1, and the padding is set to 0; the algorithm used in the upsampling layer is the bilinear interpolation algorithm.

[0069] Step 5.2, there is a branch composed of a dropout layer, a convolution layer, a softmax layer, an average pooling layer, a dimension rearrangement layer, a first linear layer, a first activation layer, a second linear layer, and a second activation layer connected in series in sequence, as Figure 12 shown. Among them, the dropout probability in the dropout layer is 0.1; the convolution kernel size of the convolution layer is set to 1×1, the stride is set to 1, and the padding is set to 0; softmax is performed on the second dimension of the input feature map; the pooling kernel size of the average pooling layer is set to 2×2, the stride is set to 2, and the padding is set to 0; the activation function used in the first activation layer is the ReLU function, and the activation function used in the second activation layer is the sigmoid function.

[0070] Step 6, construct an asymmetric bilateral attention network.

[0071] Step 6.1, connect the geometric feature extraction module and the position information encoding module in parallel to form a module group.

[0072] Step 6.2, cascade the preliminary feature extraction sub-network, the module group, the feature fusion layer, and the decoding prediction sub-network into an asymmetric bilateral attention network.

[0073] Step 7, generate a training set.

[0074] Step 7.1, in the embodiment of the present invention, at least 3000 images containing lane lines are selected from the Tusimple dataset to form a training set. The resolution of each image is 1280×720, and each image includes 0 to 4 lane lines. At the same time, each image corresponds to a binary segmentation label map;

[0075] Step 7.2, perform preprocessing on each image in the training set in sequence, including cropping, rotation, and blurring. Among them, use the resize function in OpenCV to crop the image size to 320×800; the rotation operation is a random angle rotation, and the angle is evenly distributed between [-10°, +10°]; the blurring operation uses the GaussianBlur function in OpenCV, the blurring kernel size is 5×5, and the Gaussian kernel standard deviations in the horizontal and vertical directions are 1.0×10 -6 and 0.6.

[0076] Step 8, train the asymmetric bilateral attention network.

[0077] Step 8.1, Set training parameters: Set the momentum rate to 0.9, the weight decay rate to 0.0004, and the initial learning rate to 0.02.

[0078] Step 8.2, Initialize weights. Use the weights of ImageNet-pre-trained ResNet-34 as the initial weights of the asymmetric bilateral attention network. Input the training set into the asymmetric bilateral attention network, and use the gradient descent method to iteratively update the network parameters. Adopt the SGD optimization algorithm. At least 4 images need to be fed into one batch, train for 200 epochs, and use the exponential training warm-up algorithm for the first 1000 batches. Stop training until the value of the total network loss function drops to a stable value, obtain the trained network, and save the weights.

[0079] The total network loss function is as follows:

[0080] L = w BCE L BCE + w CE L CE

[0081] where L represents the total loss function of the asymmetric bilateral attention network, w BCE represents the weight of the binary cross-entropy loss function with branches in the decoding prediction sub-network, L BCE represents the binary cross-entropy loss function with branches in the decoding prediction sub-network, w CE represents the weight of the standard cross-entropy loss function of the instance branch in the decoding prediction sub-network, L CE represents the standard cross-entropy loss function of the instance branch in the decoding prediction sub-network.

[0082]

[0083] where S represents the total number of pixel points in an image input to the decoding prediction sub-network with branches, i represents the serial number of a pixel point in an image input to the decoding prediction sub-network with branches, y i represents the ground truth label of the i-th pixel point, log represents the logarithm operation with base 2, and y' i represents the predicted value of the i-th pixel point obtained by prediction through the decoding prediction sub-network.

[0084]

[0085] where ∑ represents the summation operation, M represents the total number of lane line classifications, j represents the serial number of the lane line, p ij represents the label value indicating whether the i-th pixel point belongs to the j-th lane line, and p' ijIt represents the possibility that the i-th pixel point obtained by decoding and predicting the sub-network belongs to the j-th lane line.

[0086] Step 9: Detect lane lines.

[0087] Using the same method as in Step 7, the pre-processed images containing lane lines to be detected are sequentially input into the trained asymmetric bilateral attention network, and the lane line detection results are output.

[0088] The following further illustrates the effect of the present invention through simulation experiments:

[0089] 1. Simulation experiment conditions:

[0090] The hardware platform for the simulation experiment of the present invention: The graphics card is NVIDIA RTX2080Ti x4, the video memory is 11GB x4, the processor is Intel(R) Core(TM) CPU i9-10900X@3.70GHz, and the memory is 32GB.

[0091] The software platform is a code running environment built in a Python 3.6 virtual environment of Anaconda under the Ubuntu18.04 operating system, which includes important environment libraries such as PyTorch-1.1.0+cu100 and Matplotlib-3.3.3.

[0092] The input images used in the simulation experiment of the present invention are the publicly available dataset Tusimple created by TuSimple. The imaging time of this lane line data is 2017, the image size is 1280×720, collected under medium weather conditions, at different times, and different traffic conditions. The image format is PNG. In this simulation experiment, at least 2000 pictures other than the training set in the Tusimple dataset are selected as the test set and pre-processed according to the method in Step 7.2.

[0093] 2. Simulation content and its result analysis:

[0094] The simulation experiment of the present invention uses the present invention and three existing technologies (LaneAF, LSTR, LaneATT) to detect the input Tusimple lane line images of the TuSimple dataset respectively, and obtain the detection accuracy results.

[0095] In the simulation experiment, the three existing technologies adopted refer to:

[0096] The prior art LaneNet refers to a lane detection method proposed by Abualsaud, Hala et al. in "Laneaf: Robust multi-lane detection with affinity fields. IEEE Robotics and Automation Letters 6.4 (2021): 7477-7484.", abbreviated as LaneAF.

[0097] The prior art LSTR refers to a lane detection method proposed by Liu, Ruijin et al. in "End-to-end lane shape prediction with transformers. Proceedings of the IEEE / CVF winter conference on applications of computer vision. 2021.", abbreviated as LSTR.

[0098] The prior art LaneATT refers to a lane detection method proposed by Tabelini, Lucas et al. in "Keep your eyes on the lane: Real-time attention-guided lane detection. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021.", abbreviated as LaneATT.

[0099] The detection accuracy results of the four methods are evaluated using three evaluation metrics (Accuracy, False Positive Rate FP, and False Negative Rate FN). Using the following formulas, calculate Accuracy, FP, and FN, and plot all the calculation results in Table 1:

[0100]

[0101] Among them, Accuracy represents the accuracy of lane detection, N represents the total number of all images in the test set, i represents the image number, C i represents the correctly predicted points in the i-th image, and S i represents the total number of pixel points in the label map corresponding to the i-th image. The larger the accuracy Accuracy, the better the detection performance.

[0102]

[0103] Among them, FP represents the false positive rate, where F pred represents the number of points that are predicted as lane line points but are actually non-lane line points, and N pred represents the total number of points predicted as lane line points. The smaller the FP, the better the performance.

[0104]

[0105] Among them, FN represents the false negative rate, where M pred represents the number of points that are predicted as non-lane line points but are actually lane line points, and N gt represents the total number of all points in the ground truth label. The smaller the FN, the better the performance.

[0106] The generated test images are input into the trained lane line feature extraction network to detect lane lines. The accuracy rate of lane line detection in the present invention is 96.76%.

[0107] Table 1. Quantitative analysis and comparison table of the classification results of the present invention and various existing technologies in the simulation experiment

[0108] Experimental method The method of the present invention LaneAF LSTR LaneATT Accuracy rate (%) 96.76 96.40 96.18 95.63 FN (%) 3.32 4.18 3.38 3.53 FP (%) 2.80 2.80 2.91 2.92

[0109] It can be seen from Table 1 that the accuracy rate, FN, and FP of the present invention are all better than the existing three methods, indicating the superiority of the performance of the present invention and proving that the geometric feature extraction module and the position information encoding module proposed in the present invention are effective.

[0110] The above simulation experiments show that: different from the previous methods that only utilize the semantic features of lane lines, the geometric feature extraction module and the position information encoding module proposed in the present invention can effectively utilize the geometric characteristics and position information of lane lines, break through the thinking limitation of semantic segmentation for lane line detection, specialize the lane line detection method, thereby improving the detection performance. At the same time, in the module, partial strip convolution kernels are used to replace the square convolution kernels, which reduces the computational amount to a certain extent and is convenient for practical applications. Therefore, the present invention is a deep learning-based lane line detection method that takes both performance and speed into account.

Claims

1. A lane detection method based on geometric feature extraction and position information encoding, characterized in that: A geometric feature extraction module and a position information encoding module are respectively constructed; the steps of this detection method are as follows: Step 1, construct a preliminary feature extraction sub-network; Step 2, construct a geometric feature extraction module: Build a geometric feature extraction module composed of a high-level feature input layer, an intra-line self-attention sub-module, an inter-line self-attention sub-module, and a geometric feature output layer connected in series in sequence; The intra-line self-attention sub-module consists of two branches. The first branch is composed of an input layer, a first-dimension rearrangement layer, a linear layer, a segmentation layer, and a self-attention layer connected in series in sequence; the second branch is composed of a matrix addition layer, a second-dimension rearrangement layer, and an output layer connected in series in sequence; the matrix addition layer in the second branch is connected across the input layer and the self-attention layer of the first branch; The inter-line self-attention sub-module consists of two branches. The first branch is composed of an inter-line self-attention input layer, a convolutional layer, a first segmentation layer, a stacking layer, a first-dimension rearrangement layer, a second segmentation layer, and a self-attention layer connected in series in sequence. The second branch is composed of a matrix addition layer, a second-dimension rearrangement layer, and an inter-line self-attention output layer connected in series in sequence. The matrix addition layer is connected across the inter-line self-attention and the self-attention layer of the first branch; set the convolutional kernel size of the convolutional layer to 1×1, the stride to 1, and the padding to 0, and the self-attention layer adopts standard self-attention operations; Step 3, construct a position information encoding module: Build a position information encoding module composed of a low-level feature input layer, a vector encoding layer, a self-attention layer, and a position information encoding output layer connected in series in sequence; the vector encoding layer is composed of a relative position vector encoding layer and a qkv feature generation layer in parallel; the self-attention layer in the position information encoding module will perform multi-head attention operations; The relative position vector encoding layer is composed of a relative position matrix generation layer, a flattening layer, a subscript selection layer, and a segmentation layer connected in series in sequence; The qkv feature generation layer is composed of a first-dimension rearrangement layer, a convolutional layer, a normalization layer, a second-dimension rearrangement layer, and a segmentation layer connected in series in sequence; set the convolutional kernel size of the convolutional layer to 1×1, the stride to 1, and the padding to 0; The self-attention layer consists of two branches. The first matrix addition layer in the first branch is respectively connected to the first matrix multiplication layer, the second matrix multiplication layer, and the third matrix multiplication layer; the softmax layer in the second branch is respectively connected to the second matrix addition layer, the fourth matrix multiplication layer, and the fifth matrix multiplication layer. The second matrix addition layer is respectively connected to the fourth matrix multiplication layer, the fifth matrix multiplication layer, the connected softmax layer, and the first matrix addition layer of the first branch; Step 4, construct an asymmetric bilateral attention network: Step 4.1, construct a feature fusion layer; Step 4.2, build a decoding prediction sub-network composed of an instance branch and an existence branch in parallel; Step 4.3, connect the geometric feature extraction module and the position information encoding module in parallel to form a module group; Step 4.4, cascade the preliminary feature extraction sub-network, the module group, the feature fusion layer, and the decoding prediction sub-network into an asymmetric bilateral attention network; Step 5, generate a training set: Select at least 3000 images containing lane lines to form a sample set. The resolution of each image is 1280×720, and each image includes 0 to 4 lane lines. At the same time, each image corresponds to a binary segmentation label map. Perform preprocessing of cropping, rotation, and blurring on each image in the sample set in sequence. Compose the preprocessed images into a training set; Step 6, train the asymmetric bilateral attention network: Step 6.1, set training parameters: Set the momentum rate to 0.9, the weight decay rate to 0.0004, and the initial learning rate to 0.02; Step 6.2, weight initialization. Use the weights of the ResNet-34 pre-trained on ImageNet as the initial weights of the asymmetric bilateral attention network. Adopt the SGD optimization algorithm. Input the training set into the asymmetric bilateral attention network batch by batch. Use the gradient descent method to iteratively update the network parameters until the total network loss function converges. Obtain the trained network and save the weights; Step 7, detect lane lines: Adopt the same method as in Step 5. Input the preprocessed images containing lane lines to be detected into the trained asymmetric bilateral attention network in sequence, and output the lane line detection results.

2. The lane line detection method based on geometric feature extraction and position information encoding according to claim 1, characterized in that The preliminary feature extraction sub-network described in Step 1 is composed of an image input layer, a preliminary feature extraction layer, and an output layer group connected in series in sequence. The output layer group is composed of a high-level feature output layer and a low-level feature output layer connected in parallel.

3. The lane line detection method based on geometric feature extraction and position information encoding according to claim 1, characterized in that: The feature fusion layer described in Step 4.1 is composed of a tensor stacking layer and a convolutional layer connected in series. Set the convolutional kernel size of the convolutional layer to 1×1 and the stride to 1.

4. The lane line detection method based on geometric feature extraction and position information coding according to claim 1, wherein The instance branch described in Step 4.2 is composed of a dropout layer, a convolutional layer, and an upsampling layer connected in series in sequence. Set the dropout probability in the dropout layer to 0.1; set the convolutional kernel size of the convolutional layer to 1×1, the stride to 1, and the padding to 0; the upsampling layer adopts the bilinear interpolation algorithm.

5. The lane line detection method based on geometric feature extraction and position information encoding according to claim 1, wherein The existence branch described in Step 4.2 is composed of a dropout layer, a convolutional layer, a softmax layer, an average pooling layer, a dimension rearrangement layer, a first linear layer, a first activation layer, a second linear layer, and a second activation layer connected in series in sequence. Set the dropout probability in the dropout layer to 0.1; set the convolutional kernel size of the convolutional layer to 1×1, the stride to 1, and the padding to 0; set the pooling kernel size of the average pooling layer to 2×2, the stride to 2, and the padding to 0; the first activation layer adopts the ReLU function, and the second activation layer adopts the sigmoid function.

6. The lane line detection method based on geometric feature extraction and position information encoding according to claim 1, characterized in that The preprocessing described in Step 5 refers to using the resize function in OpenCV to crop the image size to 320×800; the rotation operation is a random angle rotation, and the angles are evenly distributed between [-10°, +10°]; the blur operation uses the GaussianBlur function in OpenCV, the blur kernel size is 5×5, and the Gaussian kernel standard deviations in the horizontal and vertical directions are 1.0×10 -6 and 0.6 respectively.

7. The lane line detection method based on geometric feature extraction and position information encoding according to claim 1, characterized in that The total network loss function described in Step 6.2 is as follows: L = w BCE L BCE + w CE L CE Among them, L represents the total loss function of the asymmetric bilateral attention network, w BCE represents the weight of the binary cross-entropy loss function with branches in the decoding prediction sub-network, L BCE represents the binary cross-entropy loss function with branches in the decoding prediction sub-network, w CE represents the weight of the standard cross-entropy loss function of the instance branch in the decoding prediction sub-network, L CE represents the standard cross-entropy loss function of the instance branch in the decoding prediction sub-network.

8. The lane line detection method based on geometric feature extraction and position information coding according to claim 7, wherein The binary cross-entropy loss function in the total network loss function described in Step 6.2 is as follows: Among them, S represents the total number of pixel points in an image with branches input into the decoding prediction sub-network, i represents the serial number of the pixel points in an image with branches input into the decoding prediction sub-network, and y i represents the true label of the i-th pixel point, log represents the logarithmic operation with base 2, and y' i represents the predicted value of the i-th pixel point obtained by prediction through the decoding prediction sub-network.

9. The lane line detection method based on geometric feature extraction and position information coding according to claim 7, wherein The standard cross-entropy loss function in the total network loss function described in Step 6.2 is as follows: Among them, ∑ represents the summation operation, M represents the total number of lane line classifications, j represents the serial number of the lane line, and p ij represents the label value indicating whether the i-th pixel point belongs to the j-th lane line, and p' ij represents the possibility that the i-th pixel point predicted by the decoded prediction sub-network belongs to the j-th lane line.

Images