Automatic driving target detection method based on improved TwinLiteNet model

Abstract

The invention discloses an automatic driving target detection method based on an improved TwinLiteNet model, and relates to an intelligent driving target detection method. The method comprises the following steps: preprocessing data for constructing a model; dividing the processed data set into a training set, a verification set and a test set according to a preset proportion; an automatic driving road detection network model of an improved TwinLiteNet model is constructed, the model architecture is composed of an encoder, a multi-head self-attention module and decoder modules, two decoders are used for two independent tasks respectively, and then driving scene information is processed. Firstly, an encoder extracts features from an input image, an encoded feature map is further processed by a multi-head self-attention module, and a decoder adopts a multi-output strategy to independently output a drivable area and lane line detection; according to the method, the problems of low accuracy and high omission ratio in automatic driving scene detection are solved by optimizing the model feature map processing method.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent driving target detection methods, and in particular to a lane line detection and drivable area segmentation detection method in a driving scene based on deep learning. Background Art

[0002] With the increase in the number of cars in my country and the world, autonomous driving has received widespread attention in the automotive field. Autonomous driving is a car driving method based on advanced technology, which aims to improve traffic safety, efficiency and comfort, bring revolutionary changes to people's travel, and have a far-reaching impact on promoting road traffic and urban development. The premise for the realization of autonomous driving is the effective perception of the driving environment, mainly the accurate identification of lane lines and drivable areas.

[0003] In computer vision object detection tasks, traditional lane line detection usually relies on image processing and geometric principles. In recent years, deep learning has been introduced into autonomous driving tasks. Semantic segmentation in deep learning has higher accuracy in environmental perception and can classify each pixel in the image and mark it as a semantic category in the image, such as people, vehicles, roads, buildings, etc. The recognition of these environmental information ensures the safe driving of the vehicle. Specifically, the accurate detection of the drivable area and lane lines lays the foundation for autonomous driving decision-making, planning and vehicle control.

[0004] In the previous driving scene detection methods, most of them focused on improving the detection speed of the model during the autonomous driving process, which indeed solved the problem that the network model had a large amount of calculation and was not lightweight enough, but lost the detection accuracy. This resulted in the problems of low accuracy and high missed detection rate. Therefore, the present invention proposes an autonomous driving target detection method based on an improved TwinLiteNet model to solve the problems existing in the prior art. Summary of the invention

[0005] The purpose of the present invention is to provide an automatic driving target detection method based on an improved TwinLiteNet model, which solves the problems of low accuracy and high missed detection rate by improving the quality of road feature representation, strengthens the ability to extract important features of drivable areas and lane lines, solves the problem of poor detection caused by shadow interference, and reduces the computational cost generated by redundant feature extraction, thereby improving the computational efficiency of the model.

[0006] The present invention adopts the following technical solutions:

[0007] An automatic driving target detection method based on an improved TwinLiteNet model is disclosed. The method improves the quality of road feature representation, solves the problems of low accuracy and high missed detection rate, and enhances the ability to extract important features of drivable areas and lane lines. The method specifically includes the following steps.

[0008] 1) Preprocess the data for building the model;

[0009] 2) Build an autonomous driving road detection network model based on the improved TwinLiteNet model. The model architecture consists of an encoder, a multi-head self-attention module, and a decoder module. There are two decoders for two independent tasks, and then process the driving scene information;

[0010] 3) Train the improved TwinLiteNet model for autonomous driving road detection network model, and use the training set constructed in step 1 to train the lane line detection model in step 2;

[0011] 4) Use the validation set to test the effect of improving the TwinLiteNet model on the autonomous driving road detection network model.

[0012] The automatic driving target detection method based on the improved TwinLiteNet model and the automatic driving road detection network model based on the improved TwinLiteNet model, in the step 1, the image data of the driving scene is preprocessed, the key areas of the image are marked, including the drivable area and lane lines, and the corresponding json file is saved. The data set is divided into a training set, a validation set, and a test set according to a preset ratio.

[0013] The method for automatic driving target detection based on the improved TwinLiteNet model, the improvement of automatic driving road detection based on the improved TwinLiteNet model, the encoder part is optimized by using the CBAM module, and the representation ability of the feature map is enhanced by the channel attention mechanism and the spatial attention mechanism; in the channel attention, the feature information of the input feature map is aggregated by the maximum pooling layer and the average pooling layer; the processed feature map is then passed through the spatial attention, the convolution layer and the Sigmoid function to obtain the weighted feature M s (X):

[0014] M s (X) = σ(f 7×7 ([AvgPool(X); MaxPool(X)]))

[0015] This method also optimizes the connection between the encoder and decoder of the model through the multi-head self-attention mechanism; the feature map processed by the encoder stage is input into the multi-head self-attention module, and three linear transformations Q = X·W are used.Q ,K=X·W K ,V=X·W V , get the projection matrices of query Q, key K and value V respectively, and then calculate the attention score of each attention head;

[0016]

[0017] Among them, softmax is used to calculate the attention weight, and the attention score matrix is ​​weighted and summed with the value V. The attention scores of each head are spliced ​​and projected back to the original space to obtain the final output;

[0018] The decoder part adds spatial convolution and transposed convolution to the decoder, so that the model can make full use of the spatial and channel information between features when performing upsampling operations, thereby improving the model performance and reducing the amount of model calculation.

[0019] The method for autonomous driving target detection based on the improved TwinLiteNet model, the autonomous driving road detection model of the improved TwinLiteNet is obtained, the improved model is trained using the constructed training set to obtain its updated model weight file, and the suffix is ​​.pth file.

[0020] The method for automatic driving target detection based on the improved TwinLiteNet model, the improved automatic driving road detection model weight file, the improved model is performance tested using a validation set to obtain a validation result, compared with the original model, the detection accuracy of lane lines in driving scenes is improved by 1.2%, and the interaction ratio is improved by 4.52%.

[0021] The present invention has the following beneficial effects:

[0022] In view of the problem of imbalance between accuracy and real-time performance in current autonomous driving detection tasks, the present invention proposes a high-precision autonomous driving scene detection model, which improves the semantic feature recognition capability by adding MHSA to the TwinLiteNet network, and adds CBAM attention module and Ghost convolution module to the encoder to make the overall model more lightweight while improving the representation capability. Compared with the TwinLiteNet model before the improvement, the improved network has higher detection accuracy, interaction ratio and average interaction ratio in lane line detection and drivable area segmentation, and the detection effect is better than the TwinLiteNet model and other classic target detection networks, and the model shows a good balance between accuracy and high computing speed during the detection process. The present invention can effectively improve the detection effect of autonomous driving detection tasks without losing detection speed, realize efficient and accurate driving scenario detection, and can help improve the safety of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the network framework of the present invention;

[0024] Figure 2 It is a schematic diagram of the MHSA module structure in step 2 of the present invention;

[0025] Figure 3 is a drivable area segmentation map of the data set of the present invention;

[0026] Figure 4 is a lane line detection result graph of the data set of the present invention;

[0027] Figure 5 It is a TotalLoss curve change diagram of the model on the data set of the present invention;

[0028] Figure 6 It is a schematic diagram of an implementation scheme of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0030] The present invention integrates the convolutional block attention module into the encoder to improve the quality of road feature representation and enable the network to generate more discriminative features. Ghost convolution is introduced into the encoder to reduce the parameters of the model to ensure the lightweight of the model. At the same time, multi-head self-attention is combined between the encoder and the decoder to improve the ability to extract important features of the drivable area and lane lines. Long-term dependencies between contextual information are established to achieve better results in the detection of details of driving scenes, solve the problem of poor detection caused by shadow interference in the TwinLiteNet model, and thus improve the accuracy of the model.

[0031] First, in the data preprocessing part, the input and output data are preprocessed, and then the data is input into the network to start formal training. The model first receives a 3×640×360 image as input. The encoding stage combines the ESPNet structure and the convolutional block attention module to enhance the feature extraction capability, and integrates the Ghost module to streamline parameters and reduce computational overhead. After that, the input features are extracted with high efficiency through a two-stage downsampling process.

[0032] The encoded feature map is processed by a dual attention module to enhance the semantic representation capability, and its output is summed and fused with the features previously enhanced by CBAM. This process promotes the integration of multi-scale feature information and improves the model performance and robustness. The obtained feature size is 32×80×45.

[0033] The feature map is then further processed by the MHSA layer, which aims to capture the correlation between features and the global context information. After MHSA processing, the decoder adopts a multi-output strategy to provide two independent outputs for drivable area and lane line detection, and promotes the rapid convergence of the model through normalization and activation functions. Finally, after two steps of inverse process upsampling in the encoding stage, the model provides two image outputs of 2×640×360, representing the detected drivable area and lane line respectively.

[0034] The processing speed of the model of the present invention is also at a good level, without losing too much processing speed. By comparing the training set and the validation set, it can be seen that when the value of the TotalLoss loss function runs on the data set for about 55 rounds, the training set loss function has dropped below 0.1, and the value is close to between 0.1 and 0.05, indicating that the processing of some optimization parameters of the model has achieved good results, ensuring that the model convergence speed and training efficiency are at a good level.

[0035] Example

[0036] The automatic driving target detection method based on the improved TwinLiteNet model of the present invention includes the following:

[0037] 1. Preprocess the data for building the model;

[0038] Self-made driving scene dataset, including driving scenes in shadow and non-shadow. Mark the key areas of the picture, including the drivable area and lane lines, and save the corresponding json file. Divide the dataset into training set, validation set, and test set according to the preset ratio;

[0039] 2. Build an improved TwinLiteNet model for autonomous driving road detection network model. The model architecture consists of an encoder, a multi-head self-attention module, and a decoder module. There are two decoders for two independent tasks, and then process the driving scene information. First, the encoder extracts features from the input image, and the encoded feature map is further processed by the multi-head self-attention module. The decoder adopts a multi-output strategy to independently output the drivable area and lane line detection;

[0040] 3. Train the improved TwinLiteNet model for autonomous driving road detection network model. Use the training set constructed in step 1 to train the lane detection model in step 2. The algorithm treats autonomous driving road detection as a confidence map regression problem. Set training hyperparameters and use the Adam optimizer, whose learning rate decreases over time. Use Focal loss and Tversky loss as loss functions to improve the segmentation effect of the model. Use the Adam optimizer to improve the loss descent path and improve the performance of the training model.

[0041] 4. The validation set is used to test the effect of improving the TwinLiteNet model's autonomous driving road detection network model. The accuracy, intersection over union (IoU) and mean interaction over union (MIoU) are used to analyze the experimental results. The larger the three indicators, the better the model reasoning effect and the higher the accuracy.

[0042] The specific implementation steps of the above step 1 are:

[0043] Use annotation software to annotate the driving scene images and obtain the corresponding label json file. The json file contains information such as the image size, driving scene category, and the spatial coordinate position of the bounding box. Divide the data set into training set, validation set, and test set according to the preset ratio. Among them, the sample ratio of (training set + validation set): test set is 9:1; the sample ratio of training set: validation set is 9:1; at this time, train, val, and test files will be generated to record the images after each division.

[0044] The specific implementation steps of the above step 2 are:

[0045] In the processing flow, the algorithm first receives a 3×640×360 image as input. The process of improving the TwinLiteNet algorithm is:

[0046] (1) Encoder stage:

[0047] The feature map is input to the encoder, the input features are reduced in dimension, and then the low-dimensional features are convolved using multiple convolution kernels with different expansion rates. The input feature map size is c×h×w, where c, h, and w are the number, height, and width of the input channels, respectively. First, it is processed by Reduce.

[0048]

[0049] Here, in order to reduce the M-dimensional input features to the dimension of the number of convolutional layers, the features are then split to obtain a unified output.

[0050]

[0051] Using K convolution kernels with different expansion rates to convolve the feature map output in the previous step in parallel will result in K feature maps of the same size.

[0052] After the first layer of convolution processing, the output feature map enters the CBAM module, and the channel attention mechanism and spatial attention mechanism are combined with the CBAM structure to enhance the representation ability of the feature map. In the channel attention, the input feature map is aggregated through the maximum pooling layer and the average pooling layer to aggregate the spatial information of the feature map, and the size becomes c×1×1. Then, the data of the two different spatial contexts generated are compressed by r times through a multi-layer shared perceptron (ShareMLP) and then expanded back to the original number. After linear transformation and normalization of the two activation functions, the pooling function and the Sigmoid function, the weighted feature M is obtained. c (X).

[0053] M c (X)=σ(MLP(AvgPool(X))+MLP(MaxPool(X)))

[0054] Combine the original feature map X with the weighted feature M c (X) is multiplied to obtain X′, which is the input of spatial attention.

[0055]

[0056] In spatial attention, X′ first passes through the maximum pooling layer and the average pooling layer to obtain two 1×h×w feature maps, which are then concatenated and passed through the convolution layer f and the Sigmoid function to obtain the weighted feature M s (X).

[0057] M s (X) = σ(f 7×7 ([AvgPool(X); MaxPool(X)]))

[0058] (2) MHSA processing stage:

[0059] The MHSA module is located after the encoder. The input here is the feature map processed by the encoder. The size of the feature map after the encoder is (180, 360). Considering that MHSA does not change the input and output sizes, the resolution is set to (45, 80) under the premise that the head is 4.

[0060] In the calculation of the MHSA module, let the input be X, and through three linear transformations Q = X·W Q ,K=X·W K ,V=X·W V , and obtain the projection matrices of query Q, key K, and value V respectively.

[0061] Then, the attention score of each attention head is calculated.

[0062]

[0063] In the formula, softmax is used to calculate the attention weight, and the attention score matrix is ​​weighted and summed with the value V. The attention scores of each head are concatenated and projected back to the original space to obtain the final output.

[0064] (3) Decoder stage:

[0065] Adding spatial convolution and transposed convolution to the decoder allows the model to fully utilize the spatial and channel information between features when performing upsampling operations, thereby improving model performance while reducing the amount of model computation.

[0066] The input features first flow through the spatial reconstruction unit to refine the spatial information.

[0067]

[0068] W=Gate(Sigmoid(W γ (GN(X))))

[0069] Among them, μ, σ are the mean and standard deviation, γ and β are trainable variables, and ε is a very small constant used to maintain stability. A larger γ represents a greater variation between pixels, that is, richer spatial information. Then the reconstruction operation is performed.

[0070]

[0071] In this step, the features with more information and the features with less information are added together to generate features with more information and save space. The weighted two different information features are combined to obtain X w1 and X w2 , after connecting them, we get the spatial refinement feature map X w .

[0072] After reducing the redundant features in the spatial dimension, the feature map is further refined through the channel reconstruction unit. The segmentation operation refines the input spatial features X w It is divided into two parts: one part has a channel number of αC, and the other part has a channel number of (1-α)C, where α is a hyperparameter and 0≤α≤1. Then the channel numbers of the two sets of features are compressed using convolution kernels to obtain X up and X low .

[0073] The optimized features are used as input for transposed convolution, which increases the spatial resolution of the feature map by upsampling the optimized features. After that, the feature map is transformed nonlinearly by batch normalization and activation function to enhance the model's ability to learn complex functions. After three consecutive upsampling stages, the final output of the model is obtained. Upsample to Upsample again to get The final output is a 2×H×W feature map.

[0074] The specific implementation steps of the above step 3 are:

[0075] Preset the batch_size and epochs for model training, and randomly select a batch of images from the training set to input into the model. Then the images are trained and feature extracted by the model in step 2. Preset the learning rate of the model.

[0076] Using Focal loss and Tversky loss as the loss function of the segmentation model, the regression loss is calculated. The purpose of Focal loss is to reduce the classification error between pixels, while addressing the influence of samples that are easy to predict and penalizing samples that are difficult to predict.

[0077]

[0078] N is the number of pixels in the input image, c is the number of classes, one class is the drivable area or lane line, and the rest are background. i (c) is to determine the prediction of pixel i of class c, is the reference truth value of pixel i of type c. γ is the balance correction factor.

[0079] Tversky Loss solves the problem of imbalanced categories in segmentation tasks. Tversky Loss introduces α and β parameters to adjust the importance of false positives and false negatives in the calculation process.

[0080]

[0081] TP is true positive, FN is false negative, FP is false positive, c is the number of classes, one class is the drivable area or lane, and the rest are background. α and β are penalty factors that control the penalty size for FP and FN.

[0082] The total network loss is obtained by weighted summation of the loss function, and then the model weight file is updated with the suffix .pth.

[0083] The specific implementation steps of step 4 are:

[0084] The validation set loss is calculated on the validation set by using the improved TwinliteNet model after parameter update, and the generalization performance of the model is verified based on the accuracy, intersection over union (IoU) and mean interaction ratio (MIoU).

[0085]

[0086] Where TP is true positive, TN is true negative, FN is false negative, FP is false positive; Z is the total number of predictions made by the model.

Claims

1. An automatic driving target detection method based on an improved TwinLiteNet model, characterized in that: The method improves the quality of road feature representation, solves the problems of low accuracy and high missed detection rate, and strengthens the ability to extract important features of drivable areas and lane lines. Specifically, the method includes the following steps: 1) Preprocess the data for building the model; 2) Build an autonomous driving road detection network model based on the improved TwinLiteNet model. The model architecture consists of an encoder, a multi-head self-attention module, and a decoder module. There are two decoders for two independent tasks, and then process the driving scene information; 3) Train the improved TwinLiteNet model for autonomous driving road detection network model, and use the training set constructed in step 1 to train the lane line detection model in step 2; 4) Use the validation set to test the effect of improving the TwinLiteNet model on the autonomous driving road detection network model.

2. The automatic driving target detection method based on the improved TwinLiteNet model according to claim 1, characterized in that: Based on the improved TwinLiteNet model, the autonomous driving road detection network model, in step 1, pre-processes the image data of the driving scene, marks the key areas of the image, including the drivable area and lane lines, and saves the corresponding json file; The data set is divided into training set, validation set, and test set according to the preset ratio.

3. The automatic driving target detection method based on the improved TwinLiteNet model according to claim 1, characterized in that: The improvement of the autonomous driving road detection based on the improved TwinLiteNet model uses the CBAM module to optimize the encoder part, and enhances the representation ability of the feature map through the channel attention mechanism and the spatial attention mechanism; in the channel attention, the feature information of the input feature map is aggregated through the maximum pooling layer and the average pooling layer; the processed feature map is then passed through the spatial attention, the convolution layer and the Sigmoid function to obtain the weighted feature M s (X): M s (X)=σ(f 7×7 ([AvgPool(X);MaxPool(X)])) This method also optimizes the connection between the encoder and decoder of the model through the multi-head self-attention mechanism; the feature map processed by the encoder stage is input into the multi-head self-attention module, and three linear transformations Q = X·W are used. Q ,K=X·W K ,V=X·W V , get the projection matrices of query Q, key K and value V respectively, and then calculate the attention score of each attention head; Among them, soft max is used to calculate the attention weight, and the attention score matrix is ​​weighted and summed with the value V. The attention scores of each head are spliced ​​and projected back to the original space to obtain the final output; The decoder part adds spatial convolution and transposed convolution to the decoder, so that the model can make full use of the spatial and channel information between features when performing upsampling operations, thereby improving the model performance and reducing the amount of model calculation.

4. The automatic driving target detection method based on the improved TwinLiteNet model according to claim 3 is characterized in that: The improved TwinLiteNet autonomous driving road detection model is trained using the constructed training set to obtain an updated model weight file with a .pth file suffix.

5. The automatic driving target detection method based on the improved TwinLiteNet model according to claim 4 is characterized in that: The improved autonomous driving road detection model weight file is used to perform performance testing on the improved model using a validation set to verify the results.

Images