Complex scene structured road detection method based on deep learning

Abstract

The invention relates to a complex scene structured road detection method based on deep learning, and belongs to the field of automatic driving environment perception. The method comprises the steps that a data set is divided and preprocessed; constructing an end-to-end lane detection network based on a B spline curve, converting the learning of curve parameters into the learning of control points, improving a residual network, introducing a mixed attention module and an auxiliary loss branch, introducing an overturning fusion module, and extracting the coordinates of the control points of the lane lines by using a regression branch; matching with the lane line existence probability in the existence branch to obtain a detection result; performing network training by using the training sample; evaluating the performance of the network; and performing lane line detection by using the trained model. The method not only has the lightweight characteristic, but also can maintain high lane detection precision in complex scenes such as vehicle shielding and lane line defect, and meets the requirements of automatic driving on real-time performance and accuracy of complex scene structured road detection.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving environment perception, and in particular relates to a complex scene structured road detection method based on deep learning. Background Art

[0002] In the application of autonomous driving systems, environmental perception is responsible for obtaining key information from the outside world, and the acquisition of lane information is a basic part of environmental perception. At present, the focus of lane detection research has shifted to methods based on deep learning. According to different implementation ideas of the algorithm, it can be roughly divided into segmentation-based methods, point detection-based methods, anchor-based methods, and curve-based methods.

[0003] The segmentation-based method separates lane line pixels from other backgrounds by classifying the pixels in the image one by one; this method spends a lot of computing power in the irrelevant areas of non-lanes, and the detection speed is much slower than other methods. The point detection-based method uses a sequence of points to indirectly represent lane lines; this type of method solves the problem of unclear boundary division in the segmentation method and improves the detection speed, but lacks consideration of global information, and the key point sequence lacks continuity, which easily affects the overall effect of the lane line. The anchor-based method uses a pre-set anchor or similar method as a benchmark to detect lane lines; this type of method has a certain a priori nature, and there will be positioning problems when there are fewer features in the starting part of the lane, resulting in the preset anchor being unable to handle large-angle lanes when the lane inclination changes due to the change in the camera position, and the generalization is poor. The curve-based method detects the curve as a whole during the lane line modeling process, and is usually modeled as a polynomial curve; this method is relatively simple, has a small number of parameters and calculations, does not have a post-processing step, and has a fast inference speed, but the polynomial curve coefficients are abstract and difficult to learn, making the performance of this type of method lag behind other methods.

[0004] The above methods still face many challenges in real-world applications: First, images are prone to large changes in brightness and uneven colors at different times and under the interference of factors such as rain and glare; second, vehicle occlusion, lane line incompleteness, tree occlusion, etc. in the road environment lead to insufficient lane line information; third, lane detection requires high real-time requirements while being accurate when driving at high speeds. Therefore, it is urgent to improve the existing lane line detection network and propose a complex scene structured road detection method based on deep learning to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of imbalance between real-time and accuracy of lane line detection tasks in complex environments in autonomous driving scenarios, and to provide a complex scene structured road detection method based on deep learning. By establishing a lane model based on B-spline curves and designing a deep learning network for detecting B-spline curve control points, the learning of curve parameters is converted into the learning of control points. The road detection method can maintain high performance while having the characteristics of lightweight, thereby improving the speed and accuracy of lane line detection and enhancing the safety and reliability of autonomous driving.

[0006] To achieve the above object, the technical solution of the present invention is: a complex scene structured road detection method based on deep learning, comprising:

[0007] Obtain the lane line dataset and divide it into training samples and test samples in proportion, and perform different preprocessing on the training samples and test samples respectively;

[0008] Construct an end-to-end lane detection network based on B-spline curves, send training samples to the end-to-end lane detection network based on B-spline curves for training, and obtain a trained lane detection network model;

[0009] The test samples are sent to the trained lane detection network model for performance evaluation. If the performance meets the actual application requirements, the trained lane detection network model is used for lane line detection and visualization. Otherwise, the training parameters of the lane detection network model are adjusted and retrained.

[0010] In one embodiment of the present invention, the lane line dataset includes lane line images and annotation files of lane line images; the division ratio of training samples and test samples in the dataset is adjusted as needed, but the training sample size should be larger than the test sample size to ensure that the model can be fully trained; test sample preprocessing includes image size adjustment, data type conversion and data standardization, etc., and training sample preprocessing adopts test sample preprocessing method, and is combined with methods including random affine transformation, random horizontal flipping and color jittering to enhance the generalization ability and robustness of the model.

[0011] In one embodiment of the present invention, the end-to-end lane detection network based on B-spline curves includes an encoding network, an auxiliary loss branch, and a decoding network; wherein,

[0012] The encoding network is composed of an improved ResNet backbone network, a hybrid attention module, a flip fusion module and average pooling; the improved ResNet backbone network is used to extract features of the input lane line image, and the features output by the conv3_x module of the improved ResNet backbone network are passed to subsequent modules and input into the auxiliary loss branch at the same time; the hybrid attention module is used to filter key information in the global features extracted by the improved ResNet backbone network to help the network better understand the context and global semantics in the image; the flip fusion module is used to process the features weighted by the hybrid attention module, and the perception ability of the network is enhanced by fusing the original features and the flipped features, and the processed enhanced features are input into the decoding network after average pooling;

[0013] The auxiliary loss branch is only used in the training stage, and uses the feature map output by the conv3_x module to segment the lane area and the non-lane area, so as to improve the network's attention to detail information during the training process, thereby improving the detection performance of the model in complex scenes.

[0014] The decoding network has two parallel branches, namely a regression branch and an existence detection branch. The regression branch is responsible for extracting the coordinates of the lane control points based on the B-spline curve to accurately define the shape of the lane. The existence detection branch is responsible for judging the existence probability of each lane to ensure that only the actually existing lanes are fitted. The Hungarian algorithm is used to optimally match the outputs of the two parallel branches to obtain the final coordinates of the lane control points and output the lane detection results.

[0015] In one embodiment of the present invention, the improved ResNet backbone network adopts the residual network as the basic architecture, and makes two improvements: first, deformable convolution is used to replace the standard convolution in the conv3_x module and the conv4_x module to enhance the backbone network's adaptive perception ability of different spaces; second, the internal expansion after the residual module is replaced by a hole residual block with expansion rates of 4 and 8 respectively to generate a 16-fold downsampled feature map, thereby enhancing the network's perception ability of long-distance features such as lane lines; the deformable convolution first uses a standard convolution kernel to extract features from the input lane line image to generate an initial feature map; then the feature map is input into an additional convolution layer to calculate the deformation offset required for the deformable convolution; finally, the convolution kernel after offset adjustment is used to perform a convolution operation on the feature map.

[0016] In one embodiment of the present invention, the hybrid attention module is divided into a parallel spatial attention module and a channel attention module; wherein,

[0017] The spatial attention module performs average pooling and maximum pooling on the channel dimension for the input feature map of H×W×C, compresses it to obtain two feature maps of H×W×1, then concatenates the two feature maps to obtain a feature map of H×W×2, and then uses a 7×7 convolution operation to reduce the number of channels to 1. Finally, the spatial attention weight is obtained through the Sigmoid activation function, and the original feature map is weighted and then output as a feature map with enhanced spatial information. The spatial attention mechanism can better capture lane information in the spatial direction, strengthen information alignment in the image, and improve the detection performance of the network.

[0018] The channel attention module performs DCT operation on the input feature map to compress the feature map of size H×W×C to 1×1×C; through adaptively adjusted 1D convolution of size k, the feature map is quickly convolved to capture each channel and its k adjacent channels to achieve weight calibration of each channel feature value and complete information exchange between channels; then the weight is normalized through the Sigmoid activation function, and multiplied channel by channel with the original feature map to obtain a feature map with channel attention; the use of one-dimensional convolution effectively reduces the amount of calculation, and the channel attention mechanism can learn the importance of information in different channels and reduce the interference of irrelevant features on the task;

[0019] The DCT operation divides the input features into N parts in the channel dimension, and processes each two-dimensional DCT component accordingly, and then combines the frequency components to form a total frequency component of the same size as the input tensor, providing more feature information for improving model performance.

[0020] In one embodiment of the present invention, the flip fusion module is used to process the key features selected by the hybrid attention module. First, the flip feature map is obtained by horizontal flipping, and it is spliced ​​with the original feature map. The offset and mask required for the deformable convolution are obtained by convolution operation. The mask is processed by the Sigmoid activation function and combined with the offset and flip feature map. Figure 1 The same input is input into the deformable convolution for operation, and the output flip feature map is obtained. It is added to the original feature map to obtain the flipped fused feature map, and then the flipped fused two-dimensional feature map is pooled into one dimension in the height direction through average pooling, and finally passed through the regression module as the input of the decoding network. The flip fusion module enhances the network's learning and fitting ability for lane symmetry and realizes feature alignment, and finally can assist in reasoning about the blocked lane through the known lane line information on the other side.

[0021] In one embodiment of the present invention, the auxiliary loss branch uses a 3×3 convolution to reduce the dimension of the feature map extracted from the conv3_x module from 256 to 64 in the channel direction to obtain a feature image; then the feature is nonlinearly mapped using a ReLu activation function after normalization; finally, a 1×1 convolution operation is performed to reduce the number of channels to one dimension to obtain a lane segmentation mask for subsequent lane detection task loss calculations to improve the network's ability to recognize distant lane lines.

[0022] In one embodiment of the present invention, in the decoding network, for the existence detection branch, a one-dimensional convolution operation is used to reduce the number of feature map channels to 1, and then the scores therein are converted into probability values ​​through a Sigmoid activation function, and then a maximum pooling operation is used to find the local maximum value in the existence probability to obtain the existence probability of each lane; for the regression branch, a one-dimensional convolution is used to reduce the number of channels to the same as the number of coordinates of the control points of the B-spline curve; for the matching of the predicted index and the target index, the Hungarian algorithm is used for optimal matching, and the loss matrix calculated according to the overall loss function is input into the Hungarian algorithm to find the matching solution with the minimum total loss, thereby realizing end-to-end learning of the model.

[0023] In one embodiment of the present invention, the overall loss function of the lane detection network model includes a regression loss function set according to a regression branch, an existence loss function set according to an existence detection branch, and an auxiliary loss function for optimizing the lane detection network model through an auxiliary loss branch. The overall loss function calculation expression of the lane detection network model is as follows:

[0024] L=λ1L rgs +λ2L ext +λ3L aux

[0025] Among them, L is the overall loss function, L rgs is the regression loss function, L ext There exists a loss function, L aux is the auxiliary loss function, λ1 is the regression loss weight coefficient, λ2 is the existence loss weight coefficient, and λ3 is the auxiliary loss weight coefficient;

[0026] The B-spline curve is expressed as:

[0027]

[0028] Where P i Ci=0,1,…,n) are the control points of the control polygon, B i,k (u) is the B-spline curve basis function, k is the order of the B-spline curve, u i are the nodes of the B-spline curve;

[0029] For the de Boor-Cox recursive definition of the B-spline basis function:

[0030]

[0031] The regression loss function uses B-spline curve sampling loss. To ensure the uniformity of the sampling points, the value range of the node u is evenly divided into N equal parts, and the sampling node parameters can be obtained:

[0032]

[0033] In the formula, △u is the distance between adjacent node values;

[0034] For the true value of the lane line, the B-spline curve is used to calculate the sampling loss of the lane key points of the lane line data set; the B-spline curve is fitted by interpolation, and the given lane line point set is {F0, F1, …, F x} There are x+1 points in total, using the B-spline curve formula The relationship between them is obtained:

[0035]

[0036] According to the chord length accumulation method, let the current type value point F i The parameter is equal to the sum of the lengths of all previous type value points Determine B by the degree and nodes of the B-spline curve i,k (u), and finally add two boundary conditions to constrain the curve equation through the tangent vectors at the beginning and end of the curve. The derivatives at the beginning and end are:

[0037]

[0038] By combining the above two equations, the true value of the lane line B-spline curve control point can be obtained, and the sampling points are obtained at equal intervals. The number of true value sampling points is T, and the number of predicted value sampling points N is greater than the true value sampling points T to ensure sufficient matching coverage. The predicted value sampling points and the true value sampling points are matched using the best dichotomy method:

[0039]

[0040]

[0041] In the formula, Q i,π(i) ∈(0,1) represents the matching quality between the ith true value and the ith predicted value, which is determined by the category score of the ith predicted value. and the true value sampling point o i Sampling points with predicted values The L1 distance between them is obtained. This matching process is evaluated by Hungarian matching, where argmax is used to find the optimal matching strategy to maximize the overall matching quality;

[0042] After completing the matching of the true value and the predicted value, the loss is calculated. The B-spline curve sampling loss performs L1 loss between the true value sampling point and the predicted value sampling point. The loss definition of n sampling points is:

[0043]

[0044] Where n is the number of sampling points, p(u) and Represent the matching probabilities of the true and predicted values ​​respectively;

[0045] The existence loss function adopts the binary cross entropy loss, and the loss function is:

[0046] L ext =-(ylog(p)+w(1-y)log(1-p))

[0047] In the formula, y is the true value, p is the predicted value, and w is the weight of the negative sample;

[0048] The auxiliary loss function uses the segmentation mask to calculate the loss, adopts the binary cross entropy loss, and the loss function is:

[0049] L aux =-(ylog(p)+w(1-y)(1-p))

[0050] In the formula, y is the true value, p is the predicted value, and w is the weight of the negative sample.

[0051] In one embodiment of the present invention, the performance meets the actual application requirements, and takes precision, recall and F1 score as indicators, and the calculation formula is as follows:

[0052]

[0053]

[0054]

[0055] Among them, when the intersection-and-joint ratio of the predicted lane line and the real lane line is greater than the threshold, it is regarded as a true positive (TP), that is, the detected lane is the real lane; when the intersection-and-joint ratio is not greater than the threshold, if it is still predicted as a lane, it is regarded as a false positive (FP), that is, a non-lane is mistakenly predicted as a lane; and if it is actually a lane but is not predicted as a lane, it is regarded as a false negative (FN), that is, the actual lane is not detected.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) A new end-to-end lane detection method based on B-spline curves is proposed. A B-spline curve lane model is established according to the characteristics of lane curves. Different from the polynomial curve model, the control points of the B-spline curve are more conducive to the learning expression of the network.

[0058] (2) In the designed lightweight network, lane features are first extracted by improving the deformable convolution backbone network. Then, the serial feature extraction attention mechanism is improved to a parallel feature extraction attention mechanism to enhance the richness of feature information acquisition. The discrete cosine function (DCT) is used to improve the global average pooling in the attention mechanism, and the irrationality of the global average pooling process in producing the same data enhancement for different features is optimized, thereby improving the detection performance of the proposed method. Then, the flip fusion module is used to obtain more global semantic information, which improves the information missing problem in occlusion and lane line defect scenes. Finally, through the pooling operation, the control point coordinates of the B-spline curve and the existence information of each lane are regressed.

[0059] (3) To address the information loss problem caused by the deep conduction process of the network model, an auxiliary loss branch is added during the network training process. By extracting the feature map from the middle module of the backbone network and entering the auxiliary loss branch, the discriminative lane features are extracted for lane area segmentation. After the auxiliary loss is given a certain weight, it is added to the loss function and the gradient reduction direction is optimized, so that the model's ability to extract features is further improved. It is removed during the inference process without affecting the detection speed of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The figure is an overall flow chart of the lane detection method according to an embodiment of the present invention.

[0061] Figure 2 The overall architecture diagram of the lane detection system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically cited and described in detail as follows.

[0063] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0064] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0065] like Figure 1 As shown, the present invention provides a complex scene structured road detection method based on deep learning, comprising:

[0066] Obtain the lane line dataset and divide it into training samples and test samples in proportion, and perform different preprocessing on the training samples and test samples respectively;

[0067] Construct an end-to-end lane detection network based on B-spline curves, send training samples to the end-to-end lane detection network based on B-spline curves for training, and obtain a trained lane detection network model;

[0068] The test samples are sent to the trained lane detection network model for performance evaluation. If the performance meets the actual application requirements, the trained lane detection network model is used for lane line detection and visualization. Otherwise, the training parameters of the lane detection network model are adjusted and retrained.

[0069] The following is the specific implementation process of the present invention.

[0070] Figure 2 The figure shows the overall architecture of the lane detection system of an embodiment of the present invention. First, the input image is preprocessed to expand the sample size to enhance the generalization ability and robustness of the model. Then, the lane detection network is entered, which consists of an encoding network, an auxiliary loss branch, and a decoding network. The coordinates of the lane control points are extracted through the regression branch and matched with the probability of the lane existence in the existence branch to obtain the final lane control point coordinates, and then the image with lane line annotations is output.

[0071] The encoding network consists of an improved ResNet backbone network, a hybrid attention module, a flip fusion module and average pooling. The improved ResNet backbone network is used to extract features from the input lane line image. The features output by the conv3_x module of the network are not only passed to subsequent modules, but also input into the auxiliary loss branch at the same time to enhance the network's ability to capture detail information. The hybrid attention module is used to further filter the key information in the global features extracted by the backbone network to help the network better understand the context and global semantics in the image. The flip fusion module is used to process the attention-weighted features, and the network's perception ability is enhanced by fusing the original features and the flipped features. The processed enhanced features are input into the decoding network after average pooling.

[0072] The ResNet backbone network of the present invention adopts an 18-layer residual network as the basic architecture, and makes two improvements: first, deformable convolution is used to replace the standard convolution in the conv3_x and conv4_x modules to enhance the backbone network's adaptive perception of different spaces; second, the internal expansion after the residual module is replaced by a hole residual block with expansion rates of 4 and 8, respectively, to generate a 16-fold downsampled feature map, thereby enhancing the network's perception of long-distance features such as lane lines; wherein, the features extracted by the conv3_x module are input into the auxiliary loss branch, and the output features of the backbone network enter the hybrid attention module for further processing;

[0073] Specifically, deformable convolution uses a standard convolution kernel to extract features from the input image and generate an initial feature map. The feature map is then input into an additional convolution layer to calculate the deformation offset required for the deformable convolution. Finally, the convolution kernel with offset adjustment is used to perform a convolution operation on the feature map.

[0074] The hybrid attention module is divided into a parallel spatial attention module and a channel attention module.

[0075] Specifically, the spatial attention module performs average pooling and maximum pooling on the channel dimension for the input feature map of H×W×C, compresses it to obtain two feature maps of H×W×1, then concatenates the two feature maps to obtain a feature map of H×W×2, and then uses a 7×7 convolution operation to reduce the number of channels to 1. Finally, the spatial attention weight is obtained through the Sigmoid activation function, and the original feature map is weightedly calculated to output a feature map with enhanced spatial information. The spatial attention mechanism can better capture lane information in the spatial direction, strengthen information alignment in the image, and improve the detection performance of the network.

[0076] Specifically, the channel attention module performs DCT operation on the input feature map to compress the feature map of size H×W×C to 1×1×C; through adaptively adjusted 1D convolution of size k, the feature map is quickly convolved to capture each channel and its k adjacent channels to achieve weight calibration of the eigenvalues ​​of each channel and complete information exchange between channels; then the weights are normalized through the Sigmoid activation function and multiplied channel by channel with the original feature map to obtain a feature map with channel attention; the use of one-dimensional convolution effectively reduces the amount of computation, and the channel attention mechanism can learn the importance of information in different channels and reduce the interference of irrelevant features on the task.

[0077] Specifically, DCT divides the input features into N parts in the channel dimension, and processes each two-dimensional DCT component accordingly. It then combines the frequency components to form a total frequency component of the same size as the input tensor, providing more feature information for improving model performance.

[0078] The flip fusion module is used to process the key features filtered by the hybrid attention module. First, the flip feature map is obtained by horizontal flipping and then concatenated with the original feature map. The offset and mask required for deformable convolution are obtained through convolution operation. The mask is processed using the Sigmoid activation function and combined with the offset and flip feature map. Figure 1 The two-dimensional feature map is then pooled into one dimension in the height direction through average pooling, and finally passed through the regression module as the input of the decoding network; the flipping fusion module enhances the network's learning and fitting ability for lane symmetry and realizes feature alignment, and finally can assist in reasoning about the blocked lane through the known lane line information on the other side.

[0079] The auxiliary loss branch uses 3×3 convolution to reduce the dimension of the feature map extracted from the conv3_x module from 256 to 64 in the channel direction to obtain the feature image; then the ReLu activation function is used to perform nonlinear mapping on the features after normalization; finally, a 1×1 convolution operation is performed to reduce it to one dimension to obtain the lane segmentation mask, which is used for the subsequent lane detection task loss calculation to improve the network's ability to recognize distant lane lines.

[0080] The decoding network includes an existence branch and a regression branch. For the existence branch, a one-dimensional convolution operation is used to reduce the number of feature map channels to 1, and then the scores are converted into probability values ​​through the Sigmoid activation function. The local maximum value in the existence probability is found through the maximum pooling operation to obtain the existence probability of each lane; for the regression branch, a one-dimensional convolution is used to reduce the number of channels to the same number of coordinates of the control points of the B-spline curve; for the matching of the predicted index and the target index, the Hungarian algorithm is used for optimal matching, and the loss matrix calculated according to the overall loss function is input into the Hungarian algorithm to find the matching solution with the minimum total loss, thus realizing end-to-end learning of the model.

[0081] The B-spline curve is expressed as:

[0082]

[0083] Where P i (i=0,1,…,n) are the control points of the control polygon, B i,k (u) is the B-spline curve basis function, k is the order of the B-spline curve, u i are the nodes of the B-spline curve;

[0084] For the de Boor-Cox recursive definition of the B-spline basis function:

[0085]

[0086] In order to have a good fitting ability, the example of the present invention uses a free parameter cubic B-spline curve with 5 control points, so that P0 is free from the interference of P4 and the flexibility of the curve at the end of the curve is enhanced. The expansion formula is:

[0087] P(u)=P0B 0,4 (u)+P1B 1,4 +P2B 2,4 (u)+P3B 3,4 (u)+P4B 4,4 (u),u∈[u3,u5]

[0088] The overall loss function includes the regression loss function set according to the regression branch, the existence loss function set according to the existence branch, and the auxiliary loss function for optimizing the network through the auxiliary loss branch. The overall loss function calculation expression of the detection model is as follows:

[0089] L=λ1L rgs +λ2L ext +λ3L aux

[0090] Among them, L is the overall loss function, L rgs is the regression loss function, Lext There exists a loss function, L aux is the auxiliary loss function, λ1 is the regression loss weight coefficient, λ2 is the existence loss weight coefficient, and λ3 is the auxiliary loss weight coefficient.

[0091] Specifically, the regression loss function uses the B-spline curve sampling loss. The example of the present invention uses the node u∈(u3,u5) in the cubic B-spline curve with 5 control points. To ensure the uniformity of the sampling points, the value range of the node u is evenly divided into N equal parts, and the sampling node parameters can be obtained:

[0092]

[0093] In the formula, △u is the distance between adjacent node values;

[0094] For the true value of the lane line, the B-spline curve is used to calculate the sampling loss of the lane key points of the data set; the B-spline curve is fitted by interpolation, assuming that the given lane line point set is {F0, F1, …, F x} There are x+1 points in total, using the B-spline curve formula The relationship between them can be obtained:

[0095]

[0096] According to the chord length accumulation method, let the current type value point F i The parameter is equal to the sum of the lengths of all previous type value points Determine B by the degree and nodes of the B-spline curve i,k (u), and finally add two boundary conditions to constrain the curve equation through the tangent vectors at the beginning and end of the curve. The derivatives at the beginning and end are:

[0097]

[0098] By combining the above two equations, the true value of the lane line B-spline curve control point can be obtained. Similarly, equally spaced sampling points are obtained. The number of true value sampling points is T. The number of predicted value sampling points N is greater than the true value sampling points T to ensure sufficient matching coverage. The predicted value sampling points and the true value sampling points are matched using the best dichotomy method:

[0099]

[0100]

[0101] In the formula, Q i,π(i) ∈(0,1) represents the matching quality between the ith true value and the ith predicted value, which is determined by the category score of the ith predicted value. and the true value sampling point o i Sampling points with predicted values The L1 distance between them is obtained. This matching process is evaluated through Hungarian matching, where argmax is used to find the optimal matching strategy to maximize the overall matching quality.

[0102] After completing the matching of the true value and the predicted value, the loss is calculated. The B-spline curve sampling loss performs L1 loss between the true value sampling point and the predicted value sampling point. The loss definition of n sampling points is:

[0103]

[0104] Where n is the number of sampling points, p(u) and They represent the matching probability of the true and predicted values ​​respectively, and the subsequent training can be better converged through a simple and effective B-spline loss function.

[0105] Specifically, there is a loss function that uses a binary cross entropy loss, and the loss function is:

[0106] L ext =-(ylog(p)+w(1-y)log(1-p))

[0107] In the formula, y is the true value, p is the model prediction value, and w is the weight of the negative sample.

[0108] Specifically, the auxiliary loss function uses the segmentation mask to calculate the loss. Since it is a binary image segmentation task, the binary cross entropy loss is used, and the loss formula is:

[0109] L aux =-(ylog(p)+w(1-y)log(1-p))

[0110] In the formula, y is the true value, p is the model prediction value, and w is the weight of the negative sample.

[0111] Figure 1 Shown is an overall flow chart of the lane detection method according to an embodiment of the present invention.

[0112] The present embodiment provides a lane line detection method based on deep learning, including the following specific steps:

[0113] (1) Prepare experimental image data and process the data set

[0114] In order to prove that this method has good lane detection capabilities in various complex scenes, a public dataset is used as the lane line dataset. In addition to normal scenes, there are also test samples in different complex scenes such as congestion, night, no lane lines, shadows, arrows, glare, curves, intersections, etc.; the training and test samples are divided according to the proportion, and the paths of training, test images and annotation information are set; the input data is preprocessed, and the test sample preprocessing includes image size adjustment, data type conversion and data standardization. The training sample preprocessing adopts the test sample preprocessing method, as well as random affine transformation, random horizontal flipping and color jittering.

[0115] (2) Constructing a convolutional neural network

[0116] Build as Figure 2 The convolutional neural network shown in the figure consists of an encoding network, an auxiliary loss branch, and parallel regression branches and existence detection branches. The encoding network consists of an improved ResNet backbone network, a hybrid attention module, a flip fusion module, and average pooling. The improved ResNet backbone network is used to extract the features of the input lane line image. The features output by its conv3_x module are not only passed to subsequent modules, but also input into the auxiliary loss branch at the same time; the hybrid attention module is used to further screen the key features in the global features extracted from the backbone network; the flip fusion module is used to process the features weighted by attention to obtain the enhanced features after flip fusion, and then input into the decoding network after average pooling; the regression branch in the decoding network is used to extract the coordinates of the lane line control points based on the B-spline curve, and match them with the lane line existence probability in the existence branch to obtain the detection result.

[0117] (3) Model training

[0118] The model training adopts the total loss function including regression loss function, existence loss function and auxiliary loss function, sets the training cycle, batch size, optimizer, learning rate, weight decay, etc.; sends the preprocessed training samples to the network for training, and saves the model after training.

[0119] (4) Use the saved model to perform lane line detection on the test image and save the detected image into a folder.

[0120] (5) The acquired lane line position information is compared with the real position information of the marked lane line. The accuracy, recall and F1 score are used as indicators. The calculation formula is as follows:

[0121]

[0122]

[0123]

[0124] Among them, when the intersection-and-joint ratio of the predicted lane line and the real lane line is greater than the threshold, it is regarded as a true positive (TP), that is, the detected lane is the real lane; when the intersection-and-joint ratio is not greater than the threshold, if it is still predicted as a lane, it is regarded as a false positive (FP), that is, a non-lane is mistakenly predicted as a lane; and if it is actually a lane but is not predicted as a lane, it is regarded as a false negative (FN), that is, the actual lane is not detected.

[0125] The above description is only a typical embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0126] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of complex scene structured road detection methods based on deep learning under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A complex scene structured road detection method based on deep learning, characterized in that: include: Obtain the lane line dataset and divide it into training samples and test samples in proportion, and perform different preprocessing on the training samples and test samples respectively; Construct an end-to-end lane detection network based on B-spline curves, send training samples to the end-to-end lane detection network based on B-spline curves for training, and obtain a trained lane detection network model; The test samples are sent to the trained lane detection network model for performance evaluation. If the performance meets the actual application requirements, the trained lane detection network model is used for lane line detection and visualization. Otherwise, the training parameters of the lane detection network model are adjusted and retrained.

2. According to claim 1, a complex scene structured road detection method based on deep learning is characterized in that: The lane line data set includes lane line images and lane line image annotation files; The test sample preprocessing includes image size adjustment, data type conversion and data standardization. The training sample preprocessing adopts the test sample preprocessing method and combines it with methods including random affine transformation, random horizontal flipping and color jittering.

3. The complex scene structured road detection method based on deep learning according to claim 1, characterized in that: The end-to-end lane detection network based on B-spline curves includes an encoding network, an auxiliary loss branch and a decoding network; wherein, The encoding network is composed of an improved ResNet backbone network, a hybrid attention module, a flip fusion module and average pooling; the improved ResNet backbone network is used to extract features of the input lane line image, and the features output by the conv3_x module of the improved ResNet backbone network are passed to subsequent modules and input into the auxiliary loss branch at the same time; the hybrid attention module is used to screen key information in the global features extracted by the improved ResNet backbone network; the flip fusion module is used to process the features weighted by the attention of the hybrid attention module, and the processed enhanced features are input into the decoding network after average pooling; The auxiliary loss branch is only used in the training phase and uses the feature map output by the conv3_x module to segment the lane area and the non-lane area; The decoding network has two parallel branches, namely a regression branch and an existence detection branch. The regression branch is responsible for extracting the coordinates of the lane control points based on the B-spline curve to accurately define the shape of the lane. The existence detection branch is responsible for judging the existence probability of each lane to ensure that only the actually existing lanes are fitted. The Hungarian algorithm is used to optimally match the outputs of the two parallel branches to obtain the final coordinates of the lane control points and output the lane detection results.

4. The complex scene structured road detection method based on deep learning according to claim 3 is characterized in that: The improved ResNet backbone network adopts the residual network as the basic architecture and makes two improvements: first, deformable convolution is used to replace the standard convolution in the conv3_x module and the conv4_x module; second, the internal expansion after the residual module is replaced by the hole residual block with expansion rates of 4 and 8 respectively to generate a 16-fold downsampled feature map; the deformable convolution first uses the standard convolution kernel to extract the features of the input lane line image to generate an initial feature map; then the feature map is input into an additional convolution layer to calculate the deformation offset required for the deformable convolution; finally, the convolution kernel with offset adjustment is used to perform a convolution operation on the feature map.

5. The complex scene structured road detection method based on deep learning according to claim 3 is characterized in that: The hybrid attention module is divided into a parallel spatial attention module and a channel attention module; wherein, The spatial attention module performs average pooling and maximum pooling on the channel dimension of the input feature map H×W×C, compresses it to obtain two feature maps of H×W×1, then concatenates the two feature maps to obtain a feature map of H×W×2, and then uses a 7×7 convolution operation to reduce the number of channels to 1. Finally, the spatial attention weight is obtained through the Sigmoid activation function, and the original feature map is weighted and calculated to output a feature map with enhanced spatial information; The channel attention module performs DCT operation on the input feature map to compress the feature map of size H×W×C to 1×1×C; through adaptively adjusted 1D convolution of size k, the feature map is quickly convolved to capture each channel and its k adjacent channels to achieve weight calibration of each channel feature value and complete information exchange between channels; then the weight is normalized through the Sigmoid activation function, and multiplied channel by channel with the original feature map to obtain a feature map with channel attention; The DCT operation divides the input features into N parts in the channel dimension, and processes each two-dimensional DCT component accordingly, and then combines the frequency components to form a total frequency component of the same size as the input tensor.

6. The complex scene structured road detection method based on deep learning according to claim 3, characterized in that: The flip fusion module is used to process the key features selected by the hybrid attention module. First, a flipped feature map is obtained by horizontal flipping, and it is spliced ​​with the original feature map. The offset and mask required for the deformable convolution are obtained by convolution operation. The mask is processed by Sigmoid activation function, and is input into the deformable convolution together with the offset and flipped feature map for operation to obtain the output flipped feature map. The output flipped feature map is added to the original feature map to obtain the flipped fused feature map. The flipped fused two-dimensional feature map is pooled into one dimension in the height direction by average pooling, and finally passed through the regression module as the input of the decoding network.

7. The complex scene structured road detection method based on deep learning according to claim 3 is characterized in that: The auxiliary loss branch uses 3×3 convolution to reduce the dimension of the feature map extracted from the conv3_x module from 256 to 64 in the channel direction to obtain a feature image; Then, after normalization, the ReLu activation function is used to perform nonlinear mapping on the features; Finally, through a 1×1 convolution operation, the number of channels is reduced to one dimension to obtain the lane segmentation mask for subsequent lane detection task loss calculation.

8. The complex scene structured road detection method based on deep learning according to claim 3, characterized in that: In the decoding network, for the existence detection branch, a one-dimensional convolution operation is used to reduce the number of feature map channels to 1, and then the scores therein are converted into probability values ​​through the Sigmoid activation function, and then the local maximum value in the existence probability is found through the maximum pooling operation to obtain the existence probability of each lane; for the regression branch, a one-dimensional convolution is used to reduce the number of channels to the same as the number of coordinates of the control points of the B-spline curve; for the matching of the predicted index and the target index, the Hungarian algorithm is used for optimal matching, and the loss matrix calculated according to the overall loss function is input into the Hungarian algorithm to find the matching solution with the minimum total loss.

9. The complex scene structured road detection method based on deep learning according to claim 3, characterized in that: The overall loss function of the lane detection network model includes the regression loss function set according to the regression branch, the existence loss function set according to the existence detection branch, and the auxiliary loss function for optimizing the lane detection network model through the auxiliary loss branch. The overall loss function calculation expression of the lane detection network model is as follows: L=λ1L rgs +λ2L ext +λ3L aux Among them, L is the overall loss function, L rgs is the regression loss function, L ext There exists a loss function, L aux is the auxiliary loss function, λ1 is the regression loss weight coefficient, λ2 is the existence loss weight coefficient, and λ3 is the auxiliary loss weight coefficient; The B-spline curve is expressed as: Where P i 9i=0,1,…,n) are the control points of the control polygon, B i,k (u) is the B-spline curve basis function, k is the order of the B-spline curve, u i are the nodes of the B-spline curve; For the de Boor-Cox recursive definition of the B-spline basis function: The regression loss function uses B-spline curve sampling loss. To ensure the uniformity of the sampling points, the value range of the node u is evenly divided into N equal parts, and the sampling node parameters can be obtained: In the formula, △u is the distance between adjacent node values; For the true value of the lane line, the B-spline curve is used to calculate the sampling loss of the lane key points of the lane line data set; the B-spline curve is fitted by interpolation, and the given lane line point set is {F0, F1, …, F x } There are x+1 points in total, using the B-spline curve formula The relationship between them is obtained: According to the chord length accumulation method, let the current type value point F i The parameter is equal to the sum of the lengths of all previous type value points Determine B by the degree and nodes of the B-spline curve i,k (u), and finally add two boundary conditions to constrain the curve equation through the tangent vectors at the beginning and end of the curve. The derivatives at the beginning and end are: By combining the above two equations, the true value of the lane line B-spline curve control point can be obtained, and the sampling points are obtained at equal intervals. The number of true value sampling points is T, and the number of predicted value sampling points N is greater than the true value sampling points T to ensure sufficient matching coverage. The predicted value sampling points and the true value sampling points are matched using the best dichotomy method: In the formula, Q i,π(i) ∈(0,1) represents the matching quality between the ith true value and the ith predicted value, which is determined by the category score of the ith predicted value. and the true value sampling point o i Sampling points with predicted values The L1 distance between them is obtained. This matching process is evaluated by Hungarian matching, where argmax is used to find the optimal matching strategy to maximize the overall matching quality; After completing the matching of the true value and the predicted value, the loss is calculated. The B-spline curve sampling loss performs L1 loss between the true value sampling point and the predicted value sampling point. The loss definition of n sampling points is: Where n is the number of sampling points, p(u) and Represent the matching probability of the true value and the matching probability of the predicted value respectively; The existence loss function adopts the binary cross entropy loss, and the loss function is: L ext =-(ylog(p)+w(1-y)log(1-p)) In the formula, y is the true value, p is the predicted value, and w is the weight of the negative sample; The auxiliary loss function uses the segmentation mask to calculate the loss, adopts the binary cross entropy loss, and the loss function is: L aux =-(ylog(p)+w(1-y)(1-p)) In the formula, y is the true value, p is the predicted value, and w is the weight of the negative sample.

10. The complex scene structured road detection method based on deep learning according to claim 3, characterized in that: The performance meets the actual application requirements. The accuracy, recall and F1 score are used as indicators. The calculation formula is as follows: Among them, when the intersection-and-joint ratio of the predicted lane line and the real lane line is greater than the threshold, it is regarded as a true positive TP, that is, the detected lane is the real lane; when the intersection-and-joint ratio is not greater than the threshold, if it is still predicted as a lane, it is regarded as a false positive FP, that is, the non-lane is mistakenly predicted as a lane; and if it is actually a lane but is not predicted as a lane, it is regarded as a false negative FN, that is, the actual lane is not detected.

Images