A lane line rapid detection method, device and readable medium

Abstract

The application discloses a kind of lane line quick detection method, device and readable medium, obtain road condition image;Encoder-decoder based neural network is constructed and trained, and lane line detection model is obtained, encoder-decoder based neural network includes main branch and auxiliary branch, main branch includes the encoder network, CBAM attention mechanism module and decoder network connected in turn in the sequentially connected, including improved residual structure in the encoder network, improved residual structure includes first convolution layer and first branch and second branch connected with first convolution layer respectively, first branch includes second convolution layer, second branch includes third convolution layer and fourth convolution layer connected in turn, the input of first branch, second branch and first convolution layer is connected by adder, auxiliary branch is instance segmentation network, and only used in training process;Road condition image is input into lane line detection model, and lane line image is detected, improve the accuracy and detection speed of lane line detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, and in particular to a lane line rapid detection method and device and readable medium. BACKGROUND

[0002] Lane line recognition is an important task of automatic driving environment perception. The lane will not be visible under the influence of factors such as poor lighting, obstacle blocking and ground pollution, resulting in a highly challenging lane detection task. Therefore, accurately and quickly detecting lane lines is a prerequisite for safe and reliable driving of unmanned vehicles. Lane line detection can be divided into traditional visual lane line detection and lane line detection based on deep learning.

[0003] Traditional lane line detection mostly relies on manual annotation to extract lane features, and then realizes lane detection through curve fitting model, lane marker grouping and other operations, but the weak generalization ability of this method makes it only applicable to detecting specific scenarios such as highway lanes, and difficult to be applied to complex scenarios such as cities. With the rapid development of deep learning technology, deep methods with good representation and learning ability gradually replace traditional detection methods. In addition, in order to obtain lane information in real time, detection speed has become the most important model evaluation standard in addition to detection accuracy. In recent years, convolutional neural networks have developed rapidly and achieved good results in target detection and scene semantic segmentation. Therefore, researchers have begun to use deep learning methods based on convolutional neural networks to realize lane line detection and have achieved certain detection effects, but there is still room for improvement in the real-time performance and detection accuracy of the model.

[0004] Current lane detection methods based on deep learning can be divided into two categories: detection-based methods and segmentation-based methods. The detection-based method divides the scene picture into a number of small grid regions of the same size, then uses a target detector such as YOLO to detect each row of grid regions to determine whether they belong to lane lines, and finally fits the grid belonging to lane lines as lane lines. The segmentation-based method classifies each lane line pixel of the scene picture to determine whether the pixel belongs to the lane line or the background. This method requires pixel-level classification of the image, which means extremely high computational cost. With the increase of computational cost, the detection speed of the algorithm will decrease. At the same time, due to the limitation of the number of semantic segmentation categories, the performance in detecting variable number of lane lines is poor, and some post-processing is usually needed to solve this problem, which brings greater time consumption. In the scene of automatic driving, the lane line detection algorithm needs to quickly and accurately locate the lane line, which is still a big challenge. SUMMARY

[0005] To solve the above technical problems, embodiments of the present application propose a lane line rapid detection method, device and readable medium.

[0006] In a first aspect, the present application provides a lane line rapid detection method, comprising the following steps:

[0007] obtaining a road condition image;

[0008] constructing and training an encoder-decoder based neural network to obtain a lane line detection model, the encoder-decoder based neural network comprising a main branch and an auxiliary branch, the main branch comprising an encoder network, a CBAM attention mechanism module and a decoder network connected in sequence, the encoder network comprising an improved residual structure, the improved residual structure comprising a first convolutional layer and a first branch and a second branch connected to the first convolutional layer respectively, the first branch comprising a second convolutional layer, the second branch comprising a third convolutional layer and a fourth convolutional layer connected in sequence, the first branch, the second branch and the input of the first convolutional layer being connected through an adder, the auxiliary branch being an instance segmentation network and being used only in the training process;

[0009] inputting the road condition image into the lane line detection model to obtain a lane line image.

[0010] Preferably, the encoder network comprises a first unit, a second unit, a third unit, a fourth unit and a fifth unit connected in sequence, the first unit comprising a fifth convolutional layer and a max-pooling layer connected in sequence, the second unit and the fifth unit comprising an improved residual structure, the third unit comprising two improved residual structures connected in sequence, and the fourth unit comprising three improved residual structures connected in sequence.

[0011] Preferably, the CBAM attention mechanism module comprises a channel attention module and a spatial attention module, in the channel attention module, the feature map output by the encoder network is input into an average pooling layer and a max-pooling layer respectively to obtain an average pooling feature and a max-pooling feature, and the average pooling feature and the max-pooling feature are input into a multilayer perceptron respectively to obtain two output feature vectors, the two output feature vectors are combined by element-wise summation, and the channel attention feature map is obtained through a sigmoid function:

[0012]

[0013] wherein F is the feature map output by the encoder network, M c (F) is the channel attention feature map, is the average pooling feature, For max pooling features, AvgPool represents the average pooling layer, MaxPool represents the max pooling layer, MLP represents the multi-layer perceptron, sigma represents the sigmoid function, w0 and w1 represent the MLP weights, w0 e R C / r×C , w1 e R C×C / r , r is the reduction ratio, and C is the number of lane lines.

[0014] The channel attention feature map is multiplied by the feature map output by the encoder network to obtain a channel correction feature map.

[0015] In the spatial attention module, the channel correction feature map is input into three 1x1 convolution blocks respectively to output a first feature map F1, a second feature map F2 and a third feature map F3, wherein {F1, F2, F3} e R H×W×C , the correlation feature C mn between the mth position in the first feature map and the nth position in the second feature map is calculated.

[0016] C mn = F1(x m ) T F2(x n );

[0017] The correlation feature C mn is normalized to S mn by a Softmax function:

[0018]

[0019] Wherein S mn e [0, 1];

[0020] The matrix multiplication between S mn and the transpose matrix of F3(x m ) is performed to obtain the feature map of the nth position:

[0021]

[0022] The feature map of each position is reshaped to R H×W×C , denoted as:

[0023] X = (X1, X2,..., X n ,..., X H×W ).

[0024] As preferred, in the decoder network, the feature map obtained by the CBAM attention mechanism module is first dimensionally expanded, and then sequentially passes through two fully connected layers and once dimensionally reorganized to obtain a position feature map, the reorganized feature map is divided into a plurality of row anchor frames based on a row anchor detection mode, and each row anchor frame is divided into a plurality of grids, the probability of the existence of a lane line in each grid in each row anchor frame is predicted, and the position of the lane line pixel point is output.

[0025] As preferred, the probability of the existence of a lane line in each grid in each row anchor frame is predicted, and the position of the lane line pixel point is output, specifically including:

[0026] Classification is performed on each row anchor frame to predict the probability of the existence of a lane line in each grid:

[0027]

[0028] where i represents the i-th row anchor frame, P i represents the probability of the existence of a lane line in the i-th row anchor frame, represents a feature vector in the position feature map, and the expected value of the horizontal coordinate is used to represent the position of the lane line pixel point:

[0029]

[0030] wherein, is the expected value of the horizontal coordinate of the lane line for each row anchor frame, which is calculated by weighted summation, p ij is the probability of the lane line pixel point at the grid (j, i), and p ij is the probability of the existence of a lane line in all grids on the i-th row anchor frame. i

[0031] As preferred, the instance segmentation network in the auxiliary branch specifically includes:

[0032] The output features of the third unit and the output features of the fourth unit sequentially pass through convolution and upsampling, and then perform Concat operation with the output features of the fifth unit to obtain auxiliary segmentation features, and the auxiliary segmentation features pass through convolution to predict the lane line segmentation result.

[0033] As preferred, the loss function of the training process of the encoder-decoder based neural network is:

[0034] L total =L cls +αL str +βL seg ;

[0035] wherein, L cls is a classification loss: ​

[0036]

[0037] where L CE is the cross-entropy loss, P c,j,: represents the probability that the grid on the jth row anchor frame is the cth lane line,

[0038] T c,j,: is the one-hot label of the correct lane line position;

[0039] L str is the structural loss, including the similarity loss L sim and the shape loss L shp , L str = L sim + γL shp ;

[0040] The similarity loss L sim is:

[0041]

[0042] where P c,j,: represents the probability that the grid on the jth row anchor frame is the cth lane line, P c,j+1,: is the probability that the adjacent grid on the j+1th row anchor frame is the cth lane line, the distance between the two adjacent grids in the same lane is represented by the L1 norm ||·||1, and γ is a hyperparameter;

[0043] The shape loss L shp is:

[0044]

[0045] where C is the number of lane lines, h is the number of row anchor frames, Loc i,j is the position of the cth lane line in the jth row anchor frame, Loc c,j+1 is the position of the cth lane line in the j+1th row anchor frame, and Loc c,j+2 is the position of the cth lane line in the j+2th row anchor frame;

[0046] L seg is the lane pixel segmentation loss provided by the auxiliary branch:

[0047] L seg = -∑[y seg *log(p seg )+(1-y seg )*log(1-p seg )];

[0048] where y segis a real lane line segmentation label, taking a value of 0 or 1, indicating whether each pixel belongs to a lane line; p seg is a lane line segmentation result predicted by the instance segmentation network, taking a value in the range of 0 to 1, indicating the probability that each pixel belongs to a lane line.

[0049] In a second aspect, the present application provides a lane line rapid detection device, characterized in that it comprises:

[0050] an image acquisition module configured to acquire a road condition image;

[0051] a model construction module configured to construct and train an encoder-decoder based neural network to obtain a lane line detection model, the encoder-decoder based neural network comprising a main branch and an auxiliary branch, the main branch comprising an encoder network, a CBAM attention mechanism module and a decoder network connected in sequence, the encoder network comprising an improved residual structure, the improved residual structure comprising a first convolutional layer and a first branch and a second branch connected to the first convolutional layer respectively, the first branch comprising a second convolutional layer, the second branch comprising a third convolutional layer and a fourth convolutional layer connected in sequence, the first branch, the second branch and the input of the first convolutional layer being connected through an adder, and the auxiliary branch being an instance segmentation network and being used only in the training process;

[0052] an output module configured to input the road condition image into the lane line detection model to obtain a lane line image.

[0053] In a third aspect, the present application provides an electronic device comprising one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any of the implementation manners of the first aspect.

[0054] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in any of the implementation manners of the first aspect.

[0055] Compared with the prior art, the present application has the following beneficial effects:

[0056] (1) The improved ResNet structure and the embedded CBAM attention mechanism module are used to extract lane line local features in the present application, which improves the ability to extract multi-scale and slender lane line features in different driving scenes.

[0057] (2) The application changes the lane line detection task from pixel-level classification to lane anchor box-based lane line position prediction, greatly reducing the calculation cost required for lane line detection, and improves the general uniform row anchor box method in the row position selection formula and the lane line offset, solves the problems of single lane line modeling constraint, lack of continuity and consistency between rows, and further improves the detection speed of lane lines under the condition of ensuring high accuracy.

[0058] (3) The application proposes a new loss function to utilize lane line prior knowledge to improve detection accuracy in various environments. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0060] Figure 1 is an exemplary device architecture diagram to which an embodiment of the present application can be applied;

[0061] Figure 2 is a flowchart of the lane line rapid detection method of the embodiment of the present application;

[0062] Figure 3 is a structure diagram of the encoder-decoder-based neural network of the lane line rapid detection method of the embodiment of the present application;

[0063] Figure 4 is a diagram of the improved ResNet structure of the lane line rapid detection method of the embodiment of the present application;

[0064] Figure 5 is a structure diagram of the CBAM attention mechanism module of the lane line rapid detection method of the embodiment of the present application, wherein Figure a is a structure diagram of the channel attention module, and Figure b is a structure diagram of the spatial attention;

[0065] Figure 6 is a schematic diagram of the lane line rapid detection device of the embodiment of the present application;

[0066] Figure 7 is a structure diagram of a computer device of an electronic device suitable for implementing the embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the present application with reference to the accompanying drawings. Obviously, the described embodiments are only a part rather than all of the embodiments of the present application. Based upon the embodiments of the present application, all other embodiments obtained by those ordinarily skilled in the art without creative effort belong to the scope of the present application.

[0068] Figure 1 An exemplary device architecture 100 to which the lane line rapid detection method or the lane line rapid detection apparatus of embodiments of the present application can be applied is shown.

[0069] As shown in Figure 1 , the device architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is a medium to provide a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0070] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 can be installed with various applications, such as data processing applications, file processing applications, etc.

[0071] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned listed electronic devices. They can be implemented as multiple software or software modules (such as software or software modules used to provide distributed services) or as a single software or software module. No specific limitation is made herein.

[0072] The server 105 can be a server providing various services, such as a background data processing server processing files or data uploaded by the terminal devices 101, 102, 103. The background data processing server can process the obtained files or data to generate processing results.

[0073] It should be noted that the lane line rapid detection method provided by embodiments of the present application can be executed by the server 105 or the terminal devices 101, 102, 103, and correspondingly, the lane line rapid detection apparatus can be arranged in the server 105 or the terminal devices 101, 102, 103.

[0074] It should be understood that,Figure 1 The number of terminal devices, networks and servers in the above-mentioned apparatus architecture is only illustrative. Any number of terminal devices, networks and servers can be provided according to implementation needs. In the case that the data processed does not need to be acquired from a remote place, the above-mentioned apparatus architecture can not include a network, but only a server or a terminal device.

[0075] Figure 2 A lane line rapid detection method provided by an embodiment of the present application is shown, which comprises the following steps:

[0076] S1, acquiring a road condition image.

[0077] Specifically, the road condition image can be an image obtained after being collected in different scenes and preprocessed. The preprocessing includes intercepting in height to remove a non-interest region irrelevant to lane line detection, avoid interference of other factors, and improve the accuracy of lane line detection.

[0078] S2, constructing an encoder-decoder based neural network and training to obtain a lane line detection model. The encoder-decoder based neural network comprises a main branch and an auxiliary branch. The main branch comprises an encoder network, a CBAM attention mechanism module and a decoder network connected in sequence. The encoder network comprises an improved residual structure. The improved residual structure comprises a first convolutional layer and a first branch and a second branch connected with the first convolutional layer respectively. The first branch comprises a second convolutional layer, and the second branch comprises a third convolutional layer and a fourth convolutional layer connected in sequence. The first branch, the second branch and the input of the first convolutional layer are connected through an adder. The auxiliary branch is an instance segmentation network and is used only in the training process.

[0079] In a specific embodiment, the encoder network comprises a first unit, a second unit, a third unit, a fourth unit and a fifth unit connected in sequence. The first unit comprises a fifth convolutional layer and a max-pooling layer connected in sequence. The second unit and the fifth unit comprise an improved residual structure. The third unit comprises two improved residual structures connected in sequence. The fourth unit comprises three improved residual structures connected in sequence.

[0080] Specifically, a structural diagram of the encoder-decoder based neural network is as shown in Figure 3As shown, the overall network structure is based on an encoder-decoder structure, wherein the encoder network is a residual backbone network adopting an improved ResNet structure. For the continuity target of lane lines and the like elongated structures, a new residual structure improved to fuse different size receptive fields is adopted as the backbone network to extract features of the preprocessed pictures. Through aggregation of the target shallow and deep features obtained by the encoder network, a multi-layer feature map used to extract lane lines is obtained. In order to reduce the network depth, the encoder network is composed of a first unit, a second unit, a third unit, a fourth unit and a fifth unit. As shown in Figure 4 As shown, the improved ResNet structure is composed of two residual branches, i.e. Res1 and Res2. Compared with the conventional residual module (only Res1), the improved residual structure is equivalent to adding one branch (Res2), so that the feature extraction performance of the network is better improved through weighted fusion of the outputs of each residual branch. In addition, the sum of the weight parameters of the two residual branches is 1, and the weight parameter of each residual branch can be trained to tend to be optimal, so as to enhance the generalization performance of the network.

[0081] In a specific embodiment, the CBAM attention mechanism module includes a channel attention module and a spatial attention module. In the channel attention module, the feature map output by the encoder network is input into an average pooling layer and a maximum pooling layer respectively to obtain average pooling features and maximum pooling features, and the average pooling features and the maximum pooling features are input into a multi-layer perceptron respectively to obtain two output feature vectors. The two output feature vectors are combined by element-wise summation, and the channel attention feature map is obtained through a sigmoid function:

[0082]

[0083] wherein F is the feature map output by the encoder network, M c (F) is the channel attention feature map, is the average pooling feature, is the maximum pooling feature, AvgPool represents the average pooling layer, MaxPool represents the maximum pooling layer, MLP represents the multi-layer perceptron, σ represents the sigmoid function, w0 and w1 represent the MLP weights, w0∈R C / r×C , w1∈R C×C / r , r is the reduction ratio, and C is the number of lane lines.

[0084] The channel attention feature map is multiplied by the feature map output by the encoder network to obtain a channel correction feature map.

[0085] In the spatial attention module, the channel correction feature map is input into three 1x1 convolution blocks respectively to output a first feature map F1, a second feature map F2 and a third feature map F3, wherein {F1, F2, F3}∈RH×W×C Calculate the correlation feature C between the m-th position in the first feature map and the n-th position in the second feature map. mn :

[0086] C mn =F1(x m ) T F2(x n );

[0087] The relevant features C are extracted using the Softmax function. mn Normalized to S mn S mn To measure the influence of the m-th position on the n-th position, the more similar the feature representations of the two positions are, the higher their correlation.

[0088]

[0089] Among them, S mn ∈[0,1];

[0090] S mn and F3(x m Perform matrix multiplication between the transposes of the given matrix to obtain the feature map at position n:

[0091]

[0092] Reshape the feature map at each location into R. H×W×C , is represented as:

[0093] X = (X1, X2, ..., X n ,...,X H×W ).

[0094] Specifically, due to the unique, elongated shape of lane lines, discontinuities arise when the road changes, such as occlusion and wear, leading to poor feature extraction performance of the encoding network for effective targets. To better highlight effective features and suppress background information, a CBAM attention mechanism module is embedded at the end of the encoder network, referencing... Figure 5 In (a) and 5(b), the CBAM attention mechanism module combines the channel attention module and the spatial attention module to improve feature representation capabilities and further enhance the accuracy of network model detection. Specifically, the channel attention module assigns different weights to each channel to enhance the ability to identify important feature channels. The spatial attention module is responsible for capturing global contextual information, combining the positional information of all lane line pixels with pixels on the feature map, which helps infer blurred pixels.

[0095] In specific embodiments, in the decoder network, the feature map obtained by the CBAM attention mechanism module is first dimensionally expanded, and then sequentially passes through two fully connected layers and once dimensionally reorganized to obtain a position feature map. The reorganized feature map is divided into a plurality of row anchor frames based on a row anchor detection manner, and each row anchor frame is divided into a plurality of grids. The probability of the existence of a lane line in each grid in each row anchor frame is predicted, and the position of the lane line pixel point is output.

[0096] In specific embodiments, the probability of the existence of a lane line in each grid in each row anchor frame is predicted, and the position of the lane line pixel point is output, specifically including:

[0097] Classification is performed on each row anchor frame to predict the probability of the existence of a lane line in each grid:

[0098]

[0099] where i represents the i-th row anchor frame, P i represents the probability of the existence of a lane line in the i-th row anchor frame, represents a feature vector in the position feature map, and the expected value of the horizontal coordinate is used to represent the position of the lane line pixel point:

[0100]

[0101] wherein, is the expected value of the horizontal coordinate of the lane line of each row anchor frame, which is calculated by weighted summation, p ij is the probability of the lane line pixel point at the grid (j, i), and the probability p ij of the existence of a lane line in all grids on the i-th row anchor frame is a combination of p i .

[0102] Specifically, first, the feature map obtained by the CBAM attention mechanism module is FLatten (dimensionally expanded), and after passing through two FC (fully connected layers) and once ReShape (dimensional reorganization), the image area of the lane line is divided into a row anchor frame with a predefined number of row anchor frames based on a row anchor detection manner. Each row anchor frame is divided into a grid with a predefined number of grids. The predefined number of grids and the number of row anchors should be much smaller than the size of the image. The lane pixel instance segmentation method requires a calculation cost of (C+1) x W x H. In the row anchor detection manner, the predefined number of row anchors h << H, and the number of grids divided on the row anchor w << W. At this time, the overall calculation cost of the network model is C x h x (w+1), and C is the number of lanes. Therefore, the row anchor detection manner greatly reduces the calculation cost of the forward reasoning of the network, and realizes the improvement of the detection speed.

[0103] For the detection method of the lane anchor frame, there are problems of single lane line modeling constraint, lack of continuous lines and consistency between rows, i.e. lane lines in the near field of view occupy multiple grids, resulting in a high false positive value. By using a row-by-row position selection formula and lane line offset, the above problems can be avoided. In addition, in order to avoid lane line pixels occupying multiple grids, the expected value of the horizontal coordinate is used to represent the position of the lane line pixel. Since the horizontal coordinate of the grid is at the upper left corner, the description is not accurate, and the use of lane bias information in the horizontal direction of each row can better improve the prediction accuracy of the lane line position. After obtaining the lane line positioning of the feature map dimension, an offset map is added to predict the offset of each row in the horizontal direction close to the row position.

[0104] In specific embodiments, the auxiliary branch specifically includes an instance segmentation network.

[0105] The output features of the third unit and the output features of the fourth unit are sequentially subjected to convolution and upsampling, and then subjected to Concat operation with the output features of the fifth unit to obtain auxiliary segmentation features. The auxiliary segmentation features are subjected to convolution to predict lane line segmentation results.

[0106] Specifically, the auxiliary branch is an instance segmentation network. To further enhance the semantic analysis capability of the detection network, an instance segmentation network is added as an auxiliary branch in addition to the main branch composed of an encoder-decoder. The global context semantic information and local feature information extracted by the network are aggregated, and the last three layers of the output of the improved ResNet structure under the downsampling stage are extracted separately. Convolution, upsampling and Concat (splicing) operations are performed on the separately extracted output to obtain auxiliary segmentation features, which fully utilize multi-scale features to assist lane position modeling, and are only used in the training stage.

[0107] Specifically, in the training process, a lane line dataset needs to be obtained first. The lane line dataset includes simple scenes with clear lane lines and complex scenes with blurred and dirty lane lines. The simple scene is an image dataset Tusimple taken on a highway in sunny weather with clear lane line markings, and the complex scene is an image dataset Culane taken on an urban road, which contains nine scenes including normal, congestion, turning, glare, night, no lane, shadow and road arrow markings.

[0108] In order to achieve better training results, the obtained dataset needs to be preprocessed. First, the lane line images in the dataset are cut in height to remove the non-interest area irrelevant to lane line detection. Then, the lane line images in the dataset are subjected to random horizontal flipping, random rotation and random scaling operation to expand the dataset, and finally the lane line dataset is obtained. The lane line dataset is divided into a training set and a test set in proportion.

[0109] In specific embodiments, the loss function of the training process of the encoder-decoder based neural network is:

[0110] L total = L cls + aL str + bL seg ;

[0111] wherein L cls is a classification loss:

[0112]

[0113] wherein L CE is a cross-entropy loss, P c,j,: represents the probability that the grid on the jth row anchor frame is the cth lane line,

[0114] T c,j,: is a one-hot label of the correct lane line position;

[0115] L str is a structural loss, including a similarity loss L sim and a shape loss L shp , L str = L sim + gL shp ;

[0116] The similarity loss L sim is:

[0117]

[0118] wherein P c,j,: represents the probability that the grid on the jth row anchor frame is the cth lane line, P c,j+1,: is the probability that the adjacent grid on the j+1th row anchor frame is the cth lane line, the distance between the two adjacent grids in the same lane is represented by the L1 norm ||·||1, and g is a hyperparameter;

[0119] The shape loss L shp is:

[0120]

[0121] wherein C is the number of lane lines, h is the number of row anchors, Loc i,j is the position of the cth lane line in the jth row anchor frame, Loc c,j+1 is the position of the cth lane line in the j+1th row anchor frame, and Loc c,j+2 is the position of the cth lane line in the j+2th row anchor frame;

[0122] L seg is a lane pixel segmentation loss provided by the auxiliary branch:

[0123] L seg = -∑[y seg *log(p seg )+(1-y seg )*log(1-p seg )];

[0124] wherein y seg is a true lane line segmentation label, taking a value of 0 or 1, indicating whether each pixel belongs to a lane line; p seg is a lane line segmentation result predicted by the instance segmentation network, taking a value in the range of 0 to 1, indicating the probability that each pixel belongs to a lane line.

[0125] Specifically, the structural loss is to solve the problem of no visual clues by using the prior knowledge of lane lines, including similarity loss and shape loss. Since the lane line is continuous, that is, the same lane line should be adjacent at the positions selected by adjacent row anchor boxes, therefore, when calculating the similarity loss, the probability that the adjacent grids on the adjacent two row anchor boxes are lane lines is used for calculation, and since the distance between the adjacent two points should be as close as possible, the smaller the L1 norm is, the better. In addition, the lane line is generally a straight line, although it is a curve, due to the effect of perspective, most of them are straight lines, and the shape loss can be obtained by calculating the second-order variance of the position expectation value. The lane pixel segmentation loss provided by the auxiliary branch uses a pixel-level cross-entropy loss function to measure the accuracy of the lane line segmentation result, which is only used during training and does not affect the detection speed.

[0126] After setting the loss function, learning rate and other parameters, the lane line detection model based on the encoder-decoder neural network is trained using the lane line dataset.

[0127] S3, inputting the road condition image into the lane line detection model to obtain a lane line image detected.

[0128] Specifically, the lane line detection model can be used to detect the road condition image to obtain a lane line image in which the lane line result is predicted.

[0129] Further referring to Figure 6 , as an implementation of the method shown in the above figures, an embodiment of a lane line rapid detection device is provided, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied in various electronic devices.

[0130] The embodiment of the present application provides a lane line rapid detection device, which comprises:

[0131] The image acquisition module 1 is configured to acquire a road condition image.

[0132] The model construction module 2 is configured to construct an encoder-decoder based neural network and train to obtain a lane line detection model, the encoder-decoder based neural network comprising a main branch and an auxiliary branch, the main branch comprising an encoder network, a CBAM attention mechanism module and a decoder network connected in sequence, the encoder network comprising an improved residual structure, the improved residual structure comprising a first convolutional layer and a first branch and a second branch connected with the first convolutional layer respectively, the first branch comprising a second convolutional layer, the second branch comprising a third convolutional layer and a fourth convolutional layer connected in sequence, the first branch, the second branch and the input of the first convolutional layer being connected through an adder, and the auxiliary branch being an instance segmentation network and being used only in the training process.

[0133] The output module 3 is configured to input the road condition image into the lane line detection model to obtain a lane line image.

[0134] Reference will be made to the following description Figure 7 which shows a structural schematic diagram of a computer device 700 of an electronic device (for example Figure 1 a server or a terminal device) suitable for implementing the embodiments of the present application. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0135] As Figure 7 shown, the computer device 700 comprises a central processing unit (CPU) 701 and a graphics processor (GPU) 702, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 703 or programs loaded into a random access memory (RAM) 704 from a storage portion 709. In the RAM 704, various programs and data required for the operation of the device 700 are also stored. The CPU 701, the GPU 702, the ROM 703 and the RAM 704 are connected to each other through a bus 705. An input / output (I / O) interface 706 is also connected to the bus 705.

[0136] The following components are connected to the I / O interface 706: an input part 707 including a keyboard, a mouse, etc.; an output part 708 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 709 including a hard disk, etc.; and a communication part 710 including a network interface card such as a LAN card, a modem, etc. The communication part 710 performs communication processing via a network such as the Internet. A drive 711 can also be connected to the I / O interface 706 as necessary. A removable media 712 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 711 as necessary, so that a computer program read out therefrom is installed in the storage part 709 as necessary.

[0137] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 710, and / or installed from the removable media 712. When the computer program is executed by the central processing unit (CPU) 701 and the graphics processor (GPU) 702, the above-described functions defined in the methods of the present application are performed.

[0138] Note that the computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer-readable medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present context, a computer-readable medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present context, a computer-readable signal medium can include a computer-readable program code in a baseband or propagated as carrier waves in a propagated data signal associating with a carrier wave. Such a propagated signal can take a wide variety of forms including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The program contained in the computer-readable medium can be transmitted in any suitable format including, but not limited to, wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0139] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0140] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may be executed in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0141] The modules involved in the embodiments of the present application can be implemented in software or in hardware. The described modules can also be arranged in a processor.

[0142] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The above computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire a road condition image; construct and train an encoder-decoder based neural network to obtain a lane line detection model, the encoder-decoder based neural network comprising a main branch and an auxiliary branch, the main branch comprising an encoder network, a CBAM attention mechanism module and a decoder network connected in sequence, the encoder network comprising an improved residual structure, the improved residual structure comprising a first convolutional layer and a first branch and a second branch connected to the first convolutional layer respectively, the first branch comprising a second convolutional layer, the second branch comprising a third convolutional layer and a fourth convolutional layer connected in sequence, the first branch, the second branch and the input of the first convolutional layer being connected through an adder, the auxiliary branch being an instance segmentation network and being used only in the training process; input the road condition image into the lane line detection model to obtain a lane line image.

[0143] The above description is only the preferred embodiment of the present application and the explanation of the technical principles. It should be understood by those skilled in the art that the scope of the protection of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features. It should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the concept of the present application. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed (but not limited to) in the present application.

Claims

1. A lane line rapid detection method, characterized in that, The method comprises the following steps: obtaining a road condition image; constructing and training an encoder-decoder-based neural network to obtain a lane line detection model, the encoder-decoder-based neural network comprising a main branch and an auxiliary branch, the main branch comprising an encoder network, a CBAM attention mechanism module and a decoder network connected in sequence, the encoder network comprising an improved residual structure, the improved residual structure comprising a first convolutional layer and a first branch and a second branch connected to the first convolutional layer respectively, the first branch comprising a second convolutional layer, the second branch comprising a third convolutional layer and a fourth convolutional layer connected in sequence, the first branch, the second branch and the input of the first convolutional layer being connected through an adder; the CBAM attention mechanism module comprising a channel attention module and a spatial attention module, in the channel attention module, feature maps output by the encoder network are input into an average pooling layer and a maximum pooling layer respectively to obtain average pooling features and maximum pooling features, and the average pooling features and the maximum pooling features are input into multilayer perceptrons respectively to obtain two output feature vectors, the two output feature vectors are combined by element-wise summation, and a channel attention feature map is obtained through a sigmoid function: wherein F is the feature map output by the encoder network, M c (F) is a channel attention feature map, is an average-pooled feature, is a max-pooled feature, AvgPool denotes an average pooling layer, MaxPool denotes a max pooling layer, MLP denotes a multi-layer perceptron, σ denotes a sigmoid function, w0and w1denote MLP weights, w0∈ R C / r×C , w1∈ R C×C / r , r is a reduction ratio, and C is a number of lane lines. the channel attention feature map is multiplied by the feature maps output by the encoder network to obtain a channel correction feature map; In the spatial attention module, the channel correction feature map is respectively input into three 1x1 convolution blocks, and first feature map F1, second feature map F2 and third feature map F3 are output, wherein {F1, F2, F3} ∈ R H×W×C , the correlation feature C between the mth position in the first feature map and the nth position in the second feature map is calculated mn : C mn = F1(x m ) T F2(x n ) The relevant features C are extracted using the Softmax function. mn Normalized to S mn : where S mn ∈ [0, 1]; S mn and the transpose matrix of F3(x m ) are performed to obtain the feature map of the nth position: Reshape the feature map of each position to R H×W×C is represented as: X = (X1, X2, ..., X...) n ,...,X H×W The decoder network first unfolds the feature map obtained by the CBAM attention mechanism module, and then passes it through two fully connected layers and one dimension reshaping to obtain a position feature map. Based on row anchor detection, the reshaping feature map is divided into several row anchor boxes, and each row anchor box is divided into several grids. The probability of lane lines existing in each grid of each row anchor box is predicted, and the position of the lane line pixel is output. The auxiliary branch is an instance segmentation network, which is only used during training. inputting the road condition image into the lane line detection model to obtain a lane line image.

2. The method of claim 1, wherein, The encoder network comprises a first unit, a second unit, a third unit, a fourth unit and a fifth unit connected in sequence, the first unit comprising a fifth convolutional layer and a maximum pooling layer connected in sequence, the second unit and the fifth unit comprising one improved residual structure, the third unit comprising two improved residual structures connected in sequence, and the fourth unit comprising three improved residual structures connected in sequence.

3. The method of claim 1, wherein, The method further comprises the following steps: performing classification on each row anchor frame to predict the probability of the existence of a lane line in each grid: where i denotes the i-th row anchor box, P i denotes the probability that the i-th row anchor box contains a lane line, denotes the feature vector in the position feature map, which uses the expected value of the horizontal coordinate to represent the position of the lane line pixel point: wherein, is the expected value of the horizontal coordinate of each lane anchor box lane line, which is calculated by weighted summation, p ij is the probability of the lane line pixel point at the grid (j, i), and the probability p ij of the existence of the lane line in all grids on the i-th lane anchor box is combined to form P i .

4. The method of claim 2, wherein, The auxiliary branch comprises the following steps: the output features of the third unit and the output features of the fourth unit are sequentially subjected to convolution and upsampling, and then subjected to a Concat operation with the output features of the fifth unit to obtain auxiliary segmentation features, and the auxiliary segmentation features are subjected to convolution to predict lane line segmentation results.

5. The method of claim 4, wherein, The loss function of the training process of the encoder-decoder-based neural network is: L total = L cls + aL str + bL seg ; where L cls is the classification loss: wherein L CE is the cross-entropy loss, P c,j,: represents the probability that the grid on the jth row anchor box is the cth lane line, T c,j,: is the one-hot label of the correct lane line position; L str L sim L shp L str L sim L shp ; similarity loss L sim is: where P c,j,: represents the probability that the grid on the jth row anchor frame is the cth lane line, P c,j+1,: is the probability that the adjacent grid on the j+1th row anchor frame is the cth lane line, the distance between two adjacent grids in the same lane is represented by the L1 norm ||·||1, and γ is a hyperparameter. Shape loss L shp is: wherein C is the number of lane lines, h is the number of line anchor boxes, Loc i,j is the position of the cthlane line in the jthline anchor box, Loc c,j+1 is the position of the cthlane line in the j+1thline anchor box, Loc c,j+2 is the position of the cthlane line in the j+2thline anchor box. L seg is the lane pixel segmentation loss provided by the auxiliary branch: L seg = -∑[y seg *log(p seg )+(1-y seg )*log(1-p seg )] wherein y seg is a true lane line segmentation label, taking values of 0 or 1, indicating whether each pixel belongs to a lane line; p seg is a lane line segmentation result predicted by the instance segmentation network, taking values in the range of 0 to 1, indicating the probability that each pixel belongs to a lane line.

6. A lane line quick detection device employing the lane line quick detection method according to any one of claims 1 to 5, characterized by The method comprises the following steps: an image acquisition module configured to obtain a road condition image; The model construction module is configured to construct an encoder-decoder based neural network and train to obtain a lane line detection model, the encoder-decoder based neural network comprising a main branch and an auxiliary branch, the main branch comprising an encoder network, a CBAM attention mechanism module and a decoder network connected in sequence, the encoder network comprising an improved residual structure, the improved residual structure comprising a first convolutional layer and a first branch and a second branch connected with the first convolutional layer respectively, the first branch comprising a second convolutional layer, the second branch comprising a third convolutional layer and a fourth convolutional layer connected in sequence, the first branch, the second branch and the input of the first convolutional layer being connected through an adder, and the auxiliary branch being an instance segmentation network and being used only in the training process; The output module is configured to input the road condition image into the lane line detection model to detect a lane line image. 7.An electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-5.

Images