A method for quickly detecting curved lane lines based on oblique convolution

Abstract

The application provides a curved lane line fast detection method based on oblique convolution, and relates to the technical field of intelligent driving, and the method comprises the following steps: acquiring an RGB image of a target region at a current time; processing the RGB image by using a pre-trained oblique convolution lane line detection model to obtain coordinate information of lane lines on a row anchor; and the oblique convolution lane line detection model extracts whole lane line information of a rotated RGB image through a deformable convolution network with spatial attention. The application improves the detection speed and precision of curved lane lines.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and in particular to a curved lane line rapid detection method based on slant convolution. BACKGROUND

[0002] In lane line detection, since lane lines are generally distributed at an angle and span multiple regions in an image, rather than being completely vertical, the convolution kernel translation operation of ordinary convolution cannot well adapt to the elongated lane lines in the image, resulting in difficulty in capturing lane line information. SUMMARY

[0003] Therefore, the present application provides a curved lane line rapid detection method based on slant convolution to solve the above technical problems.

[0004] In a first aspect, the present application provides a curved lane line rapid detection method based on slant convolution, which comprises:

[0005] obtaining an RGB image of a target region at a current time;

[0006] processing the RGB image using a pre-trained slant convolution lane line detection model to obtain coordinate information of lane lines on a line anchor; the slant convolution lane line detection model extracts whole lane line information of the rotated RGB image through a deformable convolution network with spatial attention.

[0007] Further, the slant convolution lane line detection model comprises a rotation processing module, a feature extraction module and a detection module.

[0008] processing the RGB image using a pre-trained slant convolution lane line detection model to obtain coordinate information of lane lines on a line anchor; comprising:

[0009] filling and rotating the original feature map of the RGB image using the rotation processing module to obtain a rotated feature map;

[0010] performing down-sampling and deformable convolution operations on the rotated feature map using the feature extraction module to obtain a down-sampled feature map containing lane line position features;

[0011] removing rotation and filling from the down-sampled feature map using the detection module, extracting context information from the restored down-sampled feature map, and obtaining coordinate information of lane lines on a line anchor.

[0012] Further, the rotation processing module comprises a first convolution layer, an expansion unit and a rotation unit.

[0013] The rotation processing module is used for filling and rotating the original feature map of the RGB image to obtain a rotated feature map; the method comprises the following steps:

[0014] The first convolutional layer is used for processing the RGB image to obtain an original feature map with 64 channels and a size of HxW;

[0015] The expansion unit is used for calculating the diameter r of the circumscribed circle of the feature map with a size of H / 8xW / 8, adding a region with a length of 2r in the height direction of the original feature map, and adding a region with a length of 2r in the width direction of the original feature map, so that the size of the original feature map is expanded to RxR, wherein R=8r; The rotation unit is used for rotating the filled original feature map according to four predetermined inclination angles to obtain a rotated feature map with 64 channels; the four inclination angles are obtained by the following steps:

[0016] A plurality of RGB image samples with labeled lane lines are obtained;

[0017] The inclination angles of the lane lines in each RGB image sample are calculated;

[0018] The k-means clustering algorithm is used for clustering the inclination angles of the lane lines of all the RGB image samples to obtain four inclination angles corresponding to four clustering centers: θ1, θ2, θ3 and θ4.

[0019] Further, the rotation unit is used for rotating the filled original feature map according to the four predetermined inclination angles to obtain a rotated feature map with 64 channels, which comprises the following steps:

[0020] A first inclination angle value θ1 is obtained, and the filled original feature map is rotated counterclockwise by θ1 to obtain channel values with channel numbers of 0-7; the filled original feature map is rotated clockwise by θ1 to obtain channel values with channel numbers of 8-15;

[0021] A second inclination angle value θ2 is obtained, and the filled original feature map is rotated counterclockwise by θ2 to obtain channel values with channel numbers of 16-23; the filled original feature map is rotated clockwise by θ2 to obtain channel values with channel numbers of 24-31;

[0022] A third inclination angle value θ3 is obtained, and the filled original feature map is rotated counterclockwise by θ3 to obtain channel values with channel numbers of 32-39; the filled original feature map is rotated clockwise by θ3 to obtain channel values with channel numbers of 40-47;

[0023] A third inclination angle value θ3 is obtained, and the filled original feature map is rotated counterclockwise by θ3 to obtain channel values with channel numbers of 32-39; the filled original feature map is rotated clockwise by θ3 to obtain channel values with channel numbers of 40-47;

[0024] ​Obtaining a fourth inclination angle value θ4, rotating the filled original feature map counterclockwise by θ4 to obtain channel values of channels with serial numbers 48-55; rotating the filled original feature map clockwise by θ4 to obtain channel values of channels with serial numbers 56-63;

[0025] Channel splicing is performed on the channel values of channels with serial numbers 0-7, the channel values of channels with serial numbers 8-15, the channel values of channels with serial numbers 16-23, the channel values of channels with serial numbers 24-31, the channel values of channels with serial numbers 32-39, the channel values of channels with serial numbers 40-47, the channel values of channels with serial numbers 48-55, and the channel values of channels with serial numbers 56-63 to obtain a 64-channel rotated feature map.

[0026] Further, the feature extraction module comprises a convolution layer with a residual module, a second downsampling unit, a deformable convolution unit with spatial attention, a third downsampling unit, and a fourth downsampling unit.

[0027] The rotated feature map is subjected to downsampling and deformable convolution operation by using the feature extraction module to obtain a downsampling feature map containing lane line position features, which comprises:

[0028] The 64-channel rotated feature map is processed by using the convolution layer with a residual module to obtain a first feature map f1.

[0029] The first feature map f1 is subjected to downsampling by using the second downsampling unit to obtain a second feature map f2, which has a channel number of 128 and a size of

[0030] The second feature map f2 is subjected to deformable convolution operation by using the deformable convolution unit with spatial attention to obtain a feature map containing lane line position features.

[0031] The feature map containing lane line position features is subjected to downsampling by using the third downsampling unit to obtain a third feature map f3, which has a channel number of 256 and a size of

[0032] The third feature map f3 is subjected to downsampling by using the fourth downsampling unit to obtain a fourth feature map f4, which has a channel number of 512 and a size of

[0033] Further, the second feature map f2 is subjected to deformable convolution operation by using the deformable convolution unit with spatial attention to obtain a feature map containing lane line position features, which comprises:

[0034] Each pixel of the second feature map f2 is subjected to maximum pooling processing in the channel direction to obtain a single-channel feature map with a size of ​

[0035] The upper and lower edges of the single-channel feature map are each increased by a region of , obtaining a feature map f2' of size ;

[0036] A deformable convolution network (DCN) with a convolution kernel size of is used to extract long strip lane line features from the feature map f2', obtaining a feature map f2'' of size ;

[0037] The feature map f2'' is pixel multiplied with the second feature map f2, obtaining a feature map f2''' containing lane line position features.

[0038] Further, the detection module comprises a first rotation removal unit, a first removal expansion unit, a first CCA unit, a second CCA unit, and two fully connected layers.

[0039] The detection module is used to remove rotation and removal expansion from the down-sampled feature map, and extract context information from the restored down-sampled feature map, obtaining coordinate information of lane lines on the row anchor; comprising:

[0040] The first rotation removal unit is used to perform rotation removal processing on the fourth feature map f4, obtaining a fifth feature map f5;

[0041] The first removal expansion unit is used to perform removal expansion processing on the fifth feature map f5, obtaining a sixth feature map f6 of size ;

[0042] The first CCA unit is used to process the sixth feature map f6, obtaining a seventh feature map f7 containing context information of vertical and horizontal long connections;

[0043] The second CCA unit is used to process the seventh feature map f7, obtaining an eighth feature map f8 containing dense context information of vertical and horizontal long connections;

[0044] The two fully connected layers are used to process the eighth feature map f8, obtaining coordinate information of lane lines on the row anchor.

[0045] Further, the first CCA unit comprises three parallel 1*1 convolution layers, an Affinity operation unit, a softmax layer, an Aggregation operation unit, and an adder.

[0046] The first CCA unit is used to process the sixth feature map f6, obtaining a seventh feature map f7 containing context information of vertical and horizontal long connections; comprising:

[0047] The sixth feature map f6 is processed by three parallel 1*1 convolution layers respectively to obtain a feature map Q, a feature map K and a feature map V, wherein the feature map Q, the feature map K and the feature map V have the same size as the feature map f6; the feature map Q and the feature map K have a channel number C', and C' is less than 512, the channel number of the feature map f4; and the feature map V has a channel number of 512;

[0048] The feature map Q and the feature map K are processed by an Affinity operation unit to obtain a feature map D;

[0049] The feature map D is normalized in the channel dimension by a softmax layer to obtain a feature map A;

[0050] The feature map A and the feature map V are contextually and positionally weighted by an Aggregation operation unit to obtain a seventh feature map f7.

[0051] Further, the method further comprises the steps of training the lane line detection model:

[0052] A semantic segmentation module is constructed, which is connected with the deformable convolution unit of the spatial attention of the lane line detection model, the third down-sampling unit and the fourth down-sampling unit respectively; the semantic segmentation module is used to process the feature map output by the DCN-SA unit, the feature map output by the third down-sampling unit and the feature map output by the fourth down-sampling unit to obtain a pixel-level lane line segmentation prediction image;

[0053] A training data set is obtained, which includes a plurality of RGB image samples containing real coordinate information of lane lines on the line anchors, and corresponding pixel-level lane line segmentation annotation images;

[0054] The lane line detection model connected with the semantic segmentation module is used to process the RGB image samples to obtain the predicted coordinate information of the lane lines on the line anchors of the RGB image samples and the pixel-level lane line segmentation prediction image respectively;

[0055] A first loss function L cls is calculated:

[0056]

[0057] Wherein, A is the number of lane lines, B is the number of line anchors, P a,b represents the probability vector of D+1 grid units of the a-th lane on the b-th line anchor; T a,b represents the label of D+1 grid units of the a-th lane on the b-th line anchor; L CE (·,·) represents a cross-entropy loss function;

[0058] A second loss function L is calculated for the constraint lane line being continuous on adjacent rows sim :

[0059]

[0060] wherein ‖‖1 represents an L1 norm; P a,b+1 represents a probability vector of D+1 grid units of the a-th lane on the b+1-th row anchor;

[0061] A third loss function L is calculated for the semantic segmentation module seg :

[0062]

[0063] wherein T h,w is a prediction result of lane line detection of pixel (h, w), T′ h,w is a labeled result of lane line detection, taking values of 0, 1, 2, 3, 4; 0 represents that pixel (h, w) is not on a lane line, 1 represents that pixel (h, w) is on a first lane line, 2 represents that pixel (h, w) is on a second lane line, 3 represents that pixel (h, w) is on a third lane line, and 4 represents that pixel (h, w) is on a fourth lane line;

[0064] A total loss function Loss is calculated:

[0065] Loss=L cls +L sim +L seg

[0066] The model parameters of the lane line detection model connected with the semantic segmentation module are updated by using the total loss function Loss.

[0067] When the model parameters of the lane line detection model connected with the semantic segmentation module are trained, a trained lane line detection model is obtained.

[0068] Further, the semantic segmentation module comprises a second rotation removal unit, a second expansion removal unit, a splicing unit and a second convolution layer.

[0069] The second rotation removal unit is configured to perform rotation removal processing on the feature maps output by the deformable convolution unit of the spatial attention, the feature maps output by the third down-sampling unit and the feature maps output by the fourth down-sampling unit respectively, to obtain three rotation-removed feature maps.

[0070] The second expansion removal unit is configured to perform expansion removal processing on the three rotation-removed feature maps respectively, to obtain feature maps with a size of and a size of ​ and the size of feature maps

[0071] The splicing unit is configured to perform linear interpolation on the feature maps and the feature maps respectively to obtain two feature maps with the size of , splice the two feature maps with the feature maps in the channel to obtain spliced feature maps.

[0072] The second convolutional layer is configured to process the spliced feature maps to obtain a pixel-level lane line segmentation prediction image.

[0073] In a second aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of the embodiments of the present application when executing the computer program.

[0074] In a third aspect, a computer readable storage medium is provided, which stores computer instructions, and the computer instructions are executed by a processor to implement the method of the embodiments of the present application.

[0075] The present application improves the detection speed and accuracy of curved lane lines. BRIEF DESCRIPTION OF DRAWINGS

[0076] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the description of the specific embodiments or the prior art. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0077] Figure 1 The flowchart of the curved lane line fast detection method based on the convolution of the embodiments of the present application is provided.

[0078] Figure 2 The structure diagram of the convolution lane line detection model provided by the embodiments of the present application is provided.

[0079] Figure 3 The deformable convolution network DCN structure provided by the embodiments of the present application is provided.

[0080] Figure 4 The deformable convolution network DCN-SA structure provided by the embodiments of the present application is provided.

[0081] Figure 5A structural diagram of a lane line detection model connected with a semantic segmentation module is provided for an embodiment of the present application.

[0082] Figure 6 A functional structural diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0083] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0084] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0085] First, the design idea of the embodiments of the present application is briefly introduced.

[0086] Lane line detection can identify the accurate position and shape of lane lines in the road, which is of great significance for automatic driving cars to judge their specific position and driving state in the road, and then realize functions such as lane keeping and lane changing. Since lane lines are long and thin, and there are problems such as complex lane line changes, occlusion, and light effects in real scenes, there are still certain challenges in accurately identifying lane lines. In addition, lane line detection algorithms are applied to vehicle-mounted systems, which need to process real-time data, and therefore put forward very high requirements on the real-time performance of the algorithm. How to improve the performance of the algorithm on the basis of reducing the hardware requirements is a challenge faced by the lane line detection task.

[0087] Lane line detection mainly includes traditional image processing method and deep learning based method. Image processing method can be roughly divided into: lane line detection based on Hough transform; lane line detection based on bird's eye view transformation; lane line detection based on fitting; lane line detection based on LSD straight line. Traditional method divides lane line area by edge detection filtering and other methods, and then combines Hough transform, bird's eye view transformation and other algorithms for lane line detection. This kind of algorithm needs to manually adjust the filter operator, and manually adjust the parameters according to the characteristics of the environment. The workload is large and the robustness is poor. Traditional method cannot cope with environmental changes and has poor real-time performance. Deep learning based method mainly includes lane line detection based on segmentation, lane line detection based on detection and lane line detection based on curve parameterization fitting. The method based on segmentation predicts whether each pixel is part of the lane line, which does not make full use of the prior knowledge of lane line, and has low efficiency and cannot meet the real-time requirement. The method based on detection usually uses top-down method to predict lane line, which uses the prior knowledge that lane line extends from near to far in driving perspective to construct lane line instance. For example, design line type Anchor, and regress the offset of sampling point and predefined Anchor. Or, the image is equally divided into horizontal strips, and the position of each lane line in the horizontal strip is detected. Top-down design can better utilize the prior knowledge of lane line and improve the real-time performance of detection. At the same time, it can obtain continuous lane line detection instance in the case of serious occlusion and other situations. But the preset Anchor shape will affect the flexibility of detection. The method based on parameter curve uses pre-set parameter curve to detect lane line shape, which can naturally learn the overall lane representation and has high reasoning speed, but the accuracy is not high.

[0088] Aiming at the problem of effective extraction of lane lines in the automatic driving process, in order to adapt to the distribution of lane lines in the image, based on the change of lane lines in a long distance driving process, a curved lane line fast detection method based on slant convolution is proposed, the slant convolution refers to rotating the image and then processing it through the deformable convolution network of spatial attention; The lane line inclination angle can be obtained according to the lane line circumscribed rectangle, and the clustering center in the data set is obtained as the image rotation angle according to k-means clustering. Since the image information will be lost after rotating for feature extraction and then rotating back, the size of the circumscribed circle of the image is calculated first and filled in the up, down, left and right directions of the image, and then the image is rotated into a slightly straight line type lane line according to the inclination angle and input into the feature extraction model. In the model training process, the network is divided into a main branch and a segmentation branch; The main branch is based on the lane line position selection and classification, and the segmentation branch aggregates multi-scale features for pixel-level segmentation task while adjusting the features of the main branch; The detection method of the present application is more easy to capture the slender features of the inclined lane line. In the Tusimple data set, the method of the present application can achieve an accuracy of 95.56%.

[0089] The present application rotates the image according to the inclination degree of the lane line, so that the lane line in the image is a straight line shape, so as to better utilize the deformable convolution to extract the slender features of the lane line, so that the lane line detection is more efficient and accurate. In the field of automatic driving, the lane line detection task is particularly important, and the method will have a wider use, and the method also gives a certain reference value for other fields.

[0090] After introducing the application scenarios and design ideas of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0091] As shown in Figure 1 The present application provides a curved lane line fast detection method based on slant convolution, which comprises:

[0092] Step 101: acquiring an RGB image of a target area at the current time;

[0093] The lane line of the lane where the driving vehicle is located in the embodiment is collected in real time by the front monocular camera installed on the driving vehicle. The device collecting the RGB image collects the information in front of the driving direction of the driving vehicle, that is, the RGB image is a perspective view corresponding to the information in front of the driving direction of the collecting vehicle.

[0094] Step 102: Process the RGB image using the pre-trained oblique convolutional lane line detection model to obtain the coordinate information of the lane lines on the anchor; the oblique convolutional lane line detection model extracts the entire lane line information of the rotated RGB image through a deformable convolutional network with spatial attention.

[0095] The oblique convolutional lane detection model includes: a rotation processing module, a feature extraction module, and a detection module;

[0096] The RGB image is processed using a pre-trained oblique convolutional lane detection model to obtain the coordinate information of the lane lines on the anchor; including:

[0097] The rotation processing module is used to fill and rotate the original feature map of the RGB image to obtain the rotated feature map;

[0098] The feature extraction module is used to perform downsampling and deformable convolution operations on the rotated feature map to obtain a downsampled feature map containing lane line position features;

[0099] The detection module is used to remove rotation and padding from the downsampled feature map, and context information is extracted from the recovered downsampled feature map to obtain the coordinate information of the lane lines on the anchor.

[0100] Among them, such as Figure 2 As shown, the rotation processing module includes: a first convolutional layer, a padding unit, and a rotation unit;

[0101] The rotation processing module is used to fill and rotate the original feature map of the RGB image to obtain a rotated feature map; including:

[0102] The first convolutional layer is used to process the RGB image to obtain a 64-channel original feature map with a size of H×W;

[0103] The diameter r of the circumcircle of a feature map of size H / 8 × W / 8 is calculated using the expansion unit, and the length is increased both upwards and downwards along the height direction of the original feature map. The region is then expanded to the left and right along the width of the original feature map. For the region, the size of the original feature map is enlarged to R×R, where R=8r;

[0104] Because in the training process of the model, the three feature maps of the down-sampling stage need to be removed and expanded, if the diameter of the circumscribed circle is directly calculated for the HxW feature map, a decimal situation will occur, and it is not easy to determine the size of the removal and expansion in the subsequent process, so the diameter r of the circumscribed circle of the last down-sampled feature map with a size of H / 8xW / 8 is calculated first, so that the number of pixels to be removed is accurately known when the removal and expansion is performed.

[0105] The rotation unit is used to rotate the filled original feature map according to the four predetermined inclination angles, to obtain a 64-channel rotated feature map.

[0106] Obtain a plurality of RGB image samples labeled with lane lines.

[0107] Calculate the inclination angle of the lane line in each RGB image sample.

[0108] Use the k-means clustering algorithm to cluster the inclination angles of the lane lines of all RGB image samples, to obtain four inclination angles corresponding to four clustering centers: θ1, θ2, θ3 and θ4.

[0109] The rotation unit is used to rotate the filled original feature map according to the four predetermined inclination angles, to obtain a 64-channel rotated feature map, including:

[0110] Obtain the first inclination angle value θ1, rotate the filled original feature map counterclockwise by θ1 to obtain the channel values of the channel numbers 0-7; rotate the filled original feature map clockwise by θ1 to obtain the channel values of the channel numbers 8-15.

[0111] Obtain the second inclination angle value θ2, rotate the filled original feature map counterclockwise by θ2 to obtain the channel values of the channel numbers 16-23; rotate the filled original feature map clockwise by θ2 to obtain the channel values of the channel numbers 24-31.

[0112] Obtain the third inclination angle value θ3, rotate the filled original feature map counterclockwise by θ3 to obtain the channel values of the channel numbers 32-39; rotate the filled original feature map clockwise by θ3 to obtain the channel values of the channel numbers 40-47.

[0113] Obtain the fourth inclination angle value θ4, rotate the filled original feature map counterclockwise by θ4 to obtain the channel values of the channel numbers 48-55; rotate the filled original feature map clockwise by θ4 to obtain the channel values of the channel numbers 56-63.

[0114] The channel values ​​of channels 0-7, 8-15, 16-23, 24-31, 32-39, 40-47, 48-55, and 56-63 are concatenated to obtain a 64-channel rotated feature map.

[0115] In this embodiment, the feature extraction module includes: a convolutional layer with a residual module, a second downsampling unit, a deformable convolutional unit with spatial attention, a third downsampling unit, and a fourth downsampling unit;

[0116] The feature extraction module performs downsampling and deformable convolution operations on the rotated feature map to obtain a downsampled feature map containing lane line position features; including:

[0117] The 64-channel rotated feature map is processed using the convolutional layer with residual modules to obtain the first feature map f1;

[0118] The second downsampling unit is used to downsample the first feature map f1 to obtain the second feature map f2, which has 128 channels and a size of [missing information].

[0119] The deformable convolutional unit with spatial attention is used to perform a deformable convolution operation on the second feature map f2 to obtain a feature map containing lane line position features.

[0120] The third downsampling unit is used to downsample the feature map containing lane line position features to obtain a third feature map f3, which has 256 channels and a size of [missing information].

[0121] The fourth downsampling unit is used to downsample the third feature map f3 to obtain the fourth feature map f4, which has 512 channels and a size of [missing information].

[0122] CNN convolutional kernels sample fixed locations in the feature map, making them unsuitable for targets of different scales and shapes. For example... Figure 3 As shown, the deformable convolutional network DCN is as follows: First, the input feature map H×W×C is processed by ordinary convolution to obtain H×W×2N (N = 2 × the square of the convolution kernel size, for example, N = 2 × 3). 2) of each pixel point (including x direction, y direction), since the offset is often a floating point type value, the position of the sampling point is added to the offset, and the four coordinates near the sampling point are obtained by rounding up and down, and then the value of the current sampling pixel point is obtained by the method of bilinear interpolation. Then perform the normal convolution operation. Since the offset in DCN is constantly learning, DCN can change the shape of convolution.

[0123] Since the DCN can learn the offset of each position in the feature map, as shown in Figure 4 , the embodiment designs a deformable convolution unit with spatial attention DCN-SA (Deformable Convolution Networks-Spatial Attentio) based on DCN for extracting the elongated features of lane lines.

[0124] The deformable convolution unit with spatial attention is used to perform deformable convolution operation on the second feature map f2 to obtain a feature map containing lane line position features; comprising:

[0125] Performing maximum pooling processing on each pixel of the second feature map f2 in the channel direction to obtain a single-channel feature map with a size of ;

[0126] Adding a region of to the upper and lower edges of the single-channel feature map to obtain a feature map f2' with a size of ;

[0127] Using a deformable convolution network (DCN) with a convolution kernel size of to extract long strip lane line features from the feature map f2' to obtain a feature map f2'' with a size of ;

[0128] Pixel multiplying the feature map f2'' and the second feature map f2 to obtain a feature map f2''' containing lane line position features.

[0129] In the embodiment, the detection module includes a first anti-rotation unit, a first anti-pad unit, a first CCA unit, a second CCA unit, and two fully connected layers. The ordinary convolution operation can only obtain the features of the local receptive field, such as the ASPP module of Deeplab, the pyramid pooling module of PSPNet, and the dilated convolution operation, which can only obtain features within a certain range and cannot obtain dense context information. The CCA unit of CCNet can reduce the computational complexity while obtaining dense context information.

[0130] The detection module is used for removing rotation and removing padding from the down-sampling feature map, extracting context information from the recovered down-sampling feature map, and obtaining coordinate information of lane lines on the row anchor; comprising:

[0131] The first de-rotation unit is used for performing de-rotation processing on the fourth feature map f4 to obtain a fifth feature map f5;

[0132] The first de-padding unit is used for performing de-padding processing on the fifth feature map f5 to obtain a sixth feature map f6 with a size of

[0133] The first CCA unit is used for processing the sixth feature map f6 to obtain a seventh feature map f7 containing context information of vertical and horizontal long connections;

[0134] The second CCA unit is used for processing the seventh feature map f7 to obtain an eighth feature map f8 containing dense context information of vertical and horizontal long connections;

[0135] The two fully connected layers are used for processing the eighth feature map f8 to obtain coordinate information of lane lines on the row anchor.

[0136] The first CCA unit comprises three parallel 1*1 convolution layers, an Affinity operation unit, a softmax layer, an Aggregation operation unit and an adder;

[0137] The first CCA unit is used for processing the sixth feature map f6 to obtain a seventh feature map f7 containing context information of vertical and horizontal long connections; comprising:

[0138] The three parallel 1*1 convolution layers are used for processing the sixth feature map f6 respectively to obtain a feature map Q, a feature map K and a feature map V, wherein the sizes of the feature map Q, the feature map K and the feature map V are the same as the size of the feature map f6; the channel number of the feature map Q and the feature map K is C', C' is less than 512, the channel number of the feature map V is 512;

[0139] The Affinity operation unit is used for processing the feature map Q and the feature map K to obtain a feature map D;

[0140] The softmax layer is used for performing normalization processing on the feature map D in the channel dimension to obtain a feature map A;

[0141] The Aggregation operation unit is used for performing context position relationship weighting processing on the feature map A and the feature map V to obtain the seventh feature map f7.

[0142] ​The Affinity operation unit is used to process the feature map Q and the feature map K to obtain a feature map D; including:

[0143] The vector Q is obtained from the position u in the spatial dimension of the feature map Q u The feature vector set Ω is obtained by extracting the feature vectors in the same row or column as the position u from the feature map K u , and the number is

[0144] The correlation degree feature map D is calculated, and the number of channels is The feature vector d of the position u of the i-th channel of the feature map D is i,u :

[0145]

[0146] Wherein, Ω i,u is the i-th element of Ω u ,

[0147] The Aggregation operation unit is used to process the feature map A and the feature map V to obtain the seventh feature map f7; including:

[0148] The vector V is obtained from the position u in the spatial dimension of the feature map V u The feature vector set φ is obtained by extracting the feature vectors in the same row or column as the position u from the feature map V u ;

[0149] The feature vector H of the position u of the feature map H is calculated by the Aggregation operation: u

[0150]

[0151] Wherein, A i,u is the scalar value of the channel i and the position u, and φ i,u is the scalar when the channel i and the position u are in the same row and column;

[0152] The addition operation is performed on the feature map f4 and the feature map H by the addition unit to obtain the seventh feature map f7:

[0153] f7 u = H u +f4 u

[0154] Wherein, f7 u is the feature vector of the position u of the seventh feature map f7, which contains the context information of the vertical and horizontal long connection; f4 u ​is a feature vector of the fourth feature map f4 at the position u.

[0155] In addition, the method further comprises a step of training the lane line detection model. In order to obtain global context information, a semantic segmentation module is added in the training process of the model, so that the aggregation of global and local information is realized, and the detection accuracy of the model is improved. Specifically, the training step comprises:

[0156] A semantic segmentation module is constructed, which is connected with the DCN-SA unit, the third downsampling unit and the fourth downsampling unit of the lane line detection model respectively; the semantic segmentation module is used for processing the feature maps output by the DCN-SA unit, the third downsampling unit and the fourth downsampling unit to obtain a pixel-level lane line segmentation prediction image; as shown in Figure 5 The semantic segmentation module comprises a second rotation removal unit, a second expansion removal unit, a splicing unit and a convolution layer.

[0157] The second rotation removal unit is used for respectively performing rotation removal processing on the feature maps output by the DCN-SA unit, the third downsampling unit and the fourth downsampling unit to obtain three rotation-removed feature maps.

[0158] The second expansion removal unit is used for respectively performing expansion removal processing on the three rotation-removed feature maps to obtain feature maps with sizes of

[0159] The splicing unit is used for respectively performing linear interpolation on the feature maps and to obtain two feature maps with a size of , splicing the two feature maps with the feature map in the channel to obtain a spliced feature map.

[0160] The convolution layer is used for processing the spliced feature map to obtain a pixel-level lane line segmentation prediction image.

[0161] A training data set is obtained, which comprises a plurality of RGB image samples containing real coordinate information of lane lines on line anchors, and corresponding pixel-level lane line segmentation annotation images;

[0162] ​​​​​The training data set of the embodiment is Tusimple, which has high-quality data, but it is slightly different from the content actually shot by the camera, depending on the type and position of the camera carried by the vehicle.

[0163] In terms of data augmentation, the embodiment uses methods such as scale change, random rotation, and random occlusion to augment the data set. In terms of scale change, the image is scaled from (1280, 720) to (800, 288), and the label size is scaled to (100, 36) to reduce memory consumption. Then the original image and the label are randomly rotated within a certain rotation angle, and randomly occluded in height and width. The data set is divided into training set, test set in the ratio of 1:3:1 to reduce the overfitting phenomenon.

[0164] In the lane line detection task, the network is based on supervised learning, so lane line labeling needs to be performed on the collected data set. The embodiment performs data labeling on visible light images, marking the obvious lane lines in the images, and also labeling the lane lines in the case of vehicle occlusion and unclearness.

[0165] The lane line detection model connected with the semantic segmentation module is used to process the RGB image sample, and the prediction coordinate information of the lane line on the row anchor of the RGB image sample and the pixel-level lane line segmentation prediction image are obtained respectively;

[0166] The first loss function L cls :

[0167]

[0168] where A is the number of lane lines, B is the number of row anchors, P a,b represents the probability vector of the D+1 grid units of the a-th lane line on the b-th row anchor; T a,b represents the label of the D+1 grid units of the a-th lane line on the b-th row anchor; L CE (·,·) represents the cross-entropy loss function;

[0169] The second loss function L sim :

[0170]

[0171] where ‖‖1 represents the L1 norm; P a,b+1 represents the probability vector of the D+1 grid units of the a-th lane line on the b-th row anchor; T

[0172] The third loss function L seg of the semantic segmentation module is calculated.

[0173]

[0174] wherein, T h,w is the prediction result of lane line detection of pixel (h, w), T' h,w is the labeled result of lane line detection, taking values of 0, 1, 2, 3, 4; 0 represents that pixel (h, w) is not on the lane line, 1 represents that pixel (h, w) is on the first lane line, 2 represents that pixel (h, w) is on the second lane line, 3 represents that pixel (h, w) is on the third lane line, and 4 represents that pixel (h, w) is on the fourth lane line;

[0175] The total loss function Loss is calculated as follows:

[0176] Loss = L cls + L sim + L seg

[0177] The model parameters of the lane line detection model connected with the semantic segmentation module are updated by using the total loss function Loss.

[0178] When the model parameters of the lane line detection model connected with the semantic segmentation module are trained, a trained lane line detection model is obtained.

[0179] Based on the above embodiment, the embodiment of the present application further provides an electronic device, referring to Figure 6 The electronic device 300 provided by the embodiment of the present application at least includes a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301, and the processor 301 implements the lane line fast detection method based on the slant convolution provided by the embodiment of the present application when executing the computer program.

[0180] The electronic device 300 provided by the embodiment of the present application can further include a bus 303 connecting different components (including the processor 301 and the memory 302). Among them, the bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0181] The memory 302 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022, and can further include a read-only memory (ROM) 3023.

[0182] The memory 302 can also include a program utility 3025 having a set (at least one) of program modules 3024 including, but not limited to, an operating system, one or more applications, other program modules, and program data, and each of such examples or some combination thereof, can include implementation of a network environment.

[0183] The electronic device 300 can also communicate with one or more external devices 304 such as a keyboard or a pointing device, for example, by I / O interface 305. Additionally, the electronic device 300 can communicate with one or more devices that enable user interaction with the electronic device 300 (such as a telephone, for example) and / or one or more devices that enable the electronic device 300 to communicate with one or more other electronic devices 300 (such as a router, for example). Such communication can occur via I / O interface 305. Still yet, the electronic device 300 can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the Internet, for example) via network adapter 306. As Figure 6 illustrated, network adapter 306 can communicate with the other components of electronic device 300 via bus 303. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with electronic device 300. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, redundant arrays of independent disks (RAID) subsystems, tape drives, and data archival storage subsystems, etc. Figure 6 It is to be appreciated that the bus 303 shown in FIG. 3 can include any one or combination of several types of busses, including, for example, a memory bus or multiple buses, a peripheral bus, a local bus, a wide bus, a narrow bus, etc.

[0184] It is to be appreciated that the electronic device 300 shown in FIG. 3 is only one example of an electronic device and should not be taken to limit the scope of functionality or use of embodiments of the present application. Figure 6 It is to be appreciated that the electronic device 300 shown in FIG. 3 is only one example of an electronic device and should not be taken to limit the scope of functionality or use of embodiments of the present application.

[0185] The embodiments of the present application also provide a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the method for fast detection of curved lane lines based on oblique convolution provided by the embodiments of the present application. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the method for fast detection of curved lane lines based on oblique convolution provided by the embodiments of the present application by executing the built-in or installed executable program.

[0186] The method for quickly detecting curved lane lines based on oblique convolution provided in the embodiments of the present application can also be implemented as a program product. The program product includes program codes. When the program product can run on the electronic device 300, the program codes are used to make the electronic device 300 execute the method for quickly detecting curved lane lines based on oblique convolution provided in the embodiments of the present application.

[0187] The program product provided in the embodiments of the present application can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. Specifically, more specific examples (non-exhaustive list) of the readable storage medium include an electrical connection with one or more wires, a portable disc, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0188] The program product provided in the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on a computing device. However, the program product provided in the embodiments of the present application is not limited to this. In the embodiments of the present application, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or component.

[0189] It should be noted that although several units or sub-units of the apparatus are mentioned in the foregoing detailed description, such a division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units for embodiment.

[0190] In addition, although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into a single step, and / or divided into multiple steps.

[0191] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the examples, those skilled in the art should understand that modifications or equivalent replacements to the technical solutions of the present application can be made without departing from the spirit and scope of the present application, and all should be covered in the scope of the claims of the present application.

Claims

1. A fast detection method for curved lane lines based on oblique convolution, characterized in that, The method includes: Get the RGB image of the target region at the current time; The RGB image is processed using a pre-trained oblique convolutional lane line detection model to obtain the coordinate information of the lane line on the anchor; the oblique convolutional lane line detection model extracts the entire lane line information of the rotated RGB image through a deformable convolutional network with spatial attention. The oblique convolutional lane detection model includes: a rotation processing module, a feature extraction module, and a detection module; The RGB image is processed using a pre-trained oblique convolutional lane detection model to obtain the coordinate information of the lane lines on the anchor; including: The rotation processing module is used to fill and rotate the original feature map of the RGB image to obtain the rotated feature map; The feature extraction module is used to perform downsampling and deformable convolution operations on the rotated feature map to obtain a downsampled feature map containing lane line position features; The detection module is used to remove rotation and padding from the downsampled feature map, and context information is extracted from the recovered downsampled feature map to obtain the coordinate information of the lane lines on the anchor.

2. The method according to claim 1, characterized in that, The rotation processing module includes: a first convolutional layer, an expansion unit, and a rotation unit; The rotation processing module is used to fill and rotate the original feature map of the RGB image to obtain a rotated feature map; including: The first convolutional layer is used to process the RGB image to obtain a 64-channel original feature map with a size of [size missing]. ; The size is calculated using the expansion unit. The diameter of the circumcircle of the feature map Increase the length by 100 mm both above and below the height of the original feature map. The region is then expanded to the left and right along the width of the original feature map. The region will have its original feature map size expanded to [a larger value]. ,in ; The original feature map after filling is rotated using the rotation unit according to four predetermined tilt angles to obtain a rotated feature map with 64 channels; the four tilt angles are obtained through the following steps: Obtain multiple RGB image samples with lane lines marked; Calculate the tilt angle of the lane lines in each RGB image sample; The k-means clustering algorithm was used to cluster the lane line tilt angles of all RGB image samples, resulting in four tilt angles corresponding to four cluster centers: , , and .

3. The method according to claim 2, characterized in that, The original feature map after filling is rotated using the rotation unit according to four predetermined tilt angles to obtain a 64-channel rotated feature map, including: Get the first tilt angle value Rotate the filled original feature map counterclockwise Obtain the channel values ​​of channels 0-7; rotate the filled original feature map clockwise. Get the channel values ​​of channels 8-15; Get the second tilt angle value Rotate the filled original feature map counterclockwise Obtain the channel values ​​for channels 16-23; rotate the filled original feature map clockwise. Get the channel values ​​for channels 24-31; Obtain the third tilt angle value Rotate the filled original feature map counterclockwise Obtain the channel values ​​for channels 32-39; rotate the filled original feature map clockwise. Get the channel values ​​for channels 40-47; Obtain the fourth tilt angle value Rotate the filled original feature map counterclockwise Obtain the channel values ​​for channels 48-55; rotate the filled original feature map clockwise. Get the channel values ​​for channels 56-63; The channel values ​​of channels 0-7, 8-15, 16-23, 24-31, 32-39, 40-47, 48-55, and 56-63 are concatenated to obtain a 64-channel rotated feature map.

4. The method according to claim 3, characterized in that, The feature extraction module includes: a convolutional layer with a residual module, a second downsampling unit, a deformable convolutional unit with spatial attention, a third downsampling unit, and a fourth downsampling unit; The feature extraction module performs downsampling and deformable convolution operations on the rotated feature map to obtain a downsampled feature map containing lane line position features; including: The 64-channel rotated feature map is processed using the convolutional layer with residual modules to obtain the first feature map f1; The second downsampling unit is used to downsample the first feature map f1 to obtain the second feature map f2, which has 128 channels and a size of [missing information]. ; The deformable convolutional unit with spatial attention is used to perform a deformable convolution operation on the second feature map f2 to obtain a feature map containing lane line position features. The third downsampling unit is used to downsample the feature map containing lane line position features to obtain a third feature map f3, which has 256 channels and a size of [missing information]. ; The fourth downsampling unit is used to downsample the third feature map f3 to obtain the fourth feature map f4, which has 512 channels and a size of [missing information]. .

5. The method according to claim 4, characterized in that, The deformable convolutional unit with spatial attention is used to perform a deformable convolution operation on the second feature map f2 to obtain a feature map containing lane line position features; including: Max pooling is performed on each pixel of the second feature map f2 in the channel direction to obtain a result of size f2. Single-channel feature map; Add to the top and bottom edges of the single-channel feature map The region is obtained with a size of Feature map ; Using a convolution kernel size of ( A deformable convolutional network (DCN) with a length of +1)*1 extracts features from the feature map. Extract the long strip lane line features to obtain a size of Feature map ; feature map Multiplying the second feature map f2 by pixels yields a feature map containing lane line position features. .

6. The method according to claim 4, characterized in that, The detection module includes: a first removal rotation unit, a first removal expansion unit, a first CCA unit, a second CCA unit, and two fully connected layers; The detection module removes rotation and padding from the downsampled feature map, extracts context information from the recovered downsampled feature map, and obtains the coordinate information of the lane lines on the anchor; including: The first removal and rotation unit is used to remove and rotate the fourth feature map f4 to obtain the fifth feature map f5; The first removal and expansion unit is used to perform removal and expansion processing on the fifth feature map f5, resulting in a size of The sixth feature map f6; The first CCA unit is used to process the sixth feature map f6 to obtain the seventh feature map f7, which contains context information of long connections in the vertical and horizontal directions. The seventh feature map f7 is processed using the second CCA unit to obtain an eighth feature map f8 containing dense contextual information of long vertical and horizontal connections; The coordinate information of the lane lines on the anchor is obtained by processing the eighth feature map f8 using the two fully connected layers.

7. The method according to claim 6, characterized in that, The first CCA unit includes: three parallel 1*1 convolutional layers, an affinity operation unit, a softmax layer, an aggregation operation unit, and an adder; The first CCA unit is used to process the sixth feature map f6 to obtain a seventh feature map f7 containing contextual information of long connections in the vertical and horizontal directions; including: The sixth feature map f6 is processed using three parallel 1*1 convolutional layers to obtain feature maps Q, K, and V. The sizes of feature maps Q, K, and V are the same as those of feature map f6. The number of channels in feature maps Q and K is... , The number of channels in feature map f4 is less than 512, while the number of channels in feature map V is 512. The feature map Q and feature map K are processed using the Affinity operation unit to obtain feature map D; The feature map D is normalized along the channel dimension using a softmax layer to obtain the feature map. ; Using the Aggregation operation unit on the feature map With feature map After performing context-based weighted processing, the seventh feature map f7 is obtained.

8. The method according to claim 7, characterized in that, The method further includes the step of training the lane detection model: A semantic segmentation module is constructed, which is connected to the deformable convolutional unit, the third downsampling unit, and the fourth downsampling unit of the spatial attention of the lane detection model, respectively. The semantic segmentation module is used to process the feature maps output by the DCN-SA unit, the feature maps output by the third downsampling unit, and the feature maps output by the fourth downsampling unit to obtain pixel-level lane line segmentation prediction images. Obtain a training dataset, which includes multiple RGB image samples containing the real coordinate information of lane lines on the anchors, and corresponding pixel-level lane line segmentation and annotation images; The lane detection model connected to the semantic segmentation module is used to process RGB image samples to obtain the predicted coordinate information of lane lines on the row anchors of the RGB image samples, as well as the pixel-level lane line segmentation prediction image. Calculate the first loss function : Where A is the number of lane lines and B is the number of anchor points. This represents the probability vector of the D+1 grid cells of the a-th lane at the b-th anchor. This represents the label of lane a on grid cell D+1 at anchor b; Represents the cross-entropy loss function; The second loss function is used to calculate the continuity of the constrained lane lines on adjacent rows. : Where, |||1 represents the L1 norm; Let D+1 represent the probability vector of lane a at the (b+1)th anchor point; Calculate the third loss function of the semantic segmentation module : in, The prediction result for lane line detection at pixel (h,w) is given. The labeling result for lane line detection takes values ​​of 0, 1, 2, 3, and 4; 0 represents that the pixel (h, w) is not on the lane line, 1 represents that the pixel (h, w) is on the first lane line, 2 represents that the pixel (h, w) is on the second lane line, 3 represents that the pixel (h, w) is on the third lane line, and 4 represents that the pixel (h, w) is on the fourth lane line. Calculate the total loss function : Using the total loss function Update the model parameters of the lane detection model that connects to the semantic segmentation module; Once the model parameters of the lane detection model connected to the semantic segmentation module are trained, the trained lane detection model is obtained.

9. The method according to claim 8, characterized in that, The semantic segmentation module includes: a second removal and rotation unit, a second removal and expansion unit, a splicing unit, and a second convolutional layer; The second rotation removal unit is used to perform rotation removal processing on the feature map output by the deformable convolution unit of spatial attention, the feature map output by the third downsampling unit, and the feature map output by the fourth downsampling unit, respectively, to obtain three rotation-removed feature maps. The second removal and augmentation unit is used to perform removal and augmentation processing on the three feature maps after removal and rotation, respectively, to obtain a size of Feature map Size is Feature map and size are Feature map ; The splicing unit is used to process the feature map respectively. and feature map Perform linear interpolation to obtain two values ​​of size . The feature maps, and the two feature maps are combined with the feature map. The feature map is obtained by stitching the images together on the channels. The second convolutional layer is used to process the stitched feature map to obtain a pixel-level lane line segmentation prediction image.

Images