A lane detection method and apparatus combining super-resolution and knowledge distillation

Abstract

This invention discloses a lane detection method combining super-resolution and knowledge distillation, comprising an image information acquisition module, an image preprocessing module, a lane detection module, and a model output postprocessing module. The image information acquisition module acquires image information in front of the vehicle using a camera. The image preprocessing module performs noise reduction on the image information transmitted from the image information acquisition module. The lane detection module performs semantic segmentation on the output of the image preprocessing module. The model output postprocessing module obtains lane information in the image based on the output of the lane detection module. While ensuring accuracy, this method effectively reduces computational load and can run on edge devices.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving. Background Technology

[0002] Autonomous driving perception modules acquire information about the vehicle's surrounding environment, and one of their most important components is lane detection, which is a prerequisite for ensuring the safe operation of autonomous vehicles within lane lines. Therefore, lane detection has become a hot research topic in the field of autonomous driving perception. Existing lane detection methods can be broadly categorized into three types: traditional image processing methods, methods combining traditional image processing with CNNs, and lane detection methods based on semantic segmentation.

[0003] Traditional lane detection methods typically consist of three steps: image preprocessing, local feature extraction, and lane line fitting. Local feature extraction, which captures local lane line information within the region of interest (ROI) of the image using edge, texture, and color features, is a crucial step in traditional lane detection methods. However, it requires analyzing the distribution characteristics of lane lines in the image, manually designing and combining feature extraction algorithms, and tailoring traditional lane detection methods to specific road scenarios. These methods are increasingly ill-suited to the complex scenarios of real-world autonomous driving applications, and the design of these extraction algorithms demands a high level of expertise.

[0004] In methods combining traditional image processing with CNNs, CNNs, relying on large-scale datasets and their powerful nonlinear fitting capabilities, overcome the limitations of traditional lane detection methods and improve the algorithm's generalization ability. Early lane detection methods based on convolutional neural networks simply replaced the local feature extraction step in traditional methods with neural networks, and they still required complex post-processing.

[0005] In semantic segmentation-based lane detection methods, lane detection is defined as a dense classification prediction problem. Some of these methods employ direct upsampling, while others use an encoder-decoder approach. The former uses dilated convolutions to preserve the receptive field size after removing downsampling pooling layers and obtains higher-resolution feature maps with richer spatial information. However, their direct upsampling method cannot effectively recover lane boundary pixels. Therefore, most lane segmentation networks use an encoder-decoder approach. The decoder can progressively recover low-resolution feature maps, refining the lane boundary features, but the decoder incurs significant computational overhead.

[0006] To address the issues that algorithms using direct upsampling cannot effectively recover lane line features and that encoder-decoder networks incur significant computational costs, this paper addresses these problems. Summary of the Invention

[0007] Purpose of the invention: In order to overcome the shortcomings of the prior art, the present invention provides a lane detection method and apparatus that combines super-resolution and knowledge distillation, which effectively reduces the amount of computation and can run on edge devices.

[0008] Technical Solution: To achieve the above objectives, the present invention provides a lane detection method combining super-resolution and knowledge distillation, comprising an image information acquisition module, an image preprocessing module, a lane detection module, and a model output postprocessing module; the image information acquisition module acquires image information in front of the vehicle using a camera; the image preprocessing module performs noise reduction processing on the image information transmitted from the image information acquisition module; the lane detection module performs semantic segmentation on the image preprocessing output; the model output postprocessing module obtains lane information in the image based on the output of the lane detection module.

[0009] Furthermore, firstly, in the image information acquisition module, image data generated during vehicle movement is collected by the camera and transmitted to the image preprocessing module; secondly, in the image preprocessing module, noise reduction processing is performed on the image information.

[0010] Further, the specific operations for denoising image information are as follows:

[0011] The network structure adopted is DnCNN. The first stage of the network uses 3x3 convolutional kernels and ReLU as the activation function. The second stage of the network uses 3×3 convolutional kernels, BN (batch normalization) layers and ReLU. The third stage of the network uses 3×3 convolutional kernels and zero padding in each layer to keep the input and output sizes of each layer consistent.

[0012] The loss function is:

[0013]

[0014] Where x i For a clean image, y i R represents the noisy image, θ represents the residual network, θ represents the parameters of the residual network, and N represents the number of images input in a batch.

[0015] Furthermore, in the lane detection module: semantic segmentation is performed on the image preprocessing output, and its network structure includes a teacher module and a student module.

[0016] Furthermore, in the Teacher module: the input image size is 1024x1024. The Encode part network structure consists of 5 stacked residual networks with 3x3 convolutional kernels, downsampled to a feature map of size 128x128. This 128x128 feature map is then input into the Teacher Decode network, passed through 5 layers of deconvolutional networks, upsampled to 1024x1024, and finally passed through a 1x1 convolutional network to output a lane line prediction map.

[0017] Furthermore, in the Student module: the input image size is 256x256, the Encode part network structure is a stack of 2 residual networks, using 3x3 convolutional kernels, downsampled to a feature map of size 128x128; the 128x128 feature map is input into the Student Decode network; after passing through 5 layers of deconvolutional networks, it is upsampled to a size of 1024x1024, and after passing through a 1x1 convolutional network, the lane line prediction map is output.

[0018] Furthermore, during the training of the Student module, the feature maps of the first, third, and fifth layers in the Decode part of the Teacher network are compared with the feature maps of the first, third, and fifth layers in the Decode part of the Student module to calculate the L1 loss.

[0019] Furthermore, the L1 loss formula is as follows:

[0020]

[0021] In the formula, y i The feature set output by the Teacher. Let n be the feature set output by Student, and n be the set of feature maps.

[0022] Furthermore, during the training of the Teacher and Student networks, the cross-entropy loss function is applied to the output results. The cross-entropy loss function is used to calculate the binary classification loss function, and its formula is as follows:

[0023] L=-[ylogy′+(1-y)log(1-y′)]

[0024] Where y is the true value and y' is the estimated value.

[0025] Furthermore, in the model output post-processing module: the feature map output by the lane detection module is processed by a softmax function to output the final lane detection result, where the softmax function formula is as follows:

[0026]

[0027] Each pixel in the output corresponds to an array Z, where Zi indicates that the i-th element Zj in Z is the sum of array Z;

[0028] After passing through the softmax function, the output is a 1x1024x1024 prediction map, where a pixel value of 1 represents lane lines and a value of 0 represents non-lane lines.

[0029] Beneficial effects: By inputting a smaller image, most of the model's computation is performed at a smaller resolution, greatly reducing the computational load of the model. Then, by borrowing the idea of ​​super-resolution, super-resolution is performed on both the semantic segmentation output and the image, resulting in an output result that is twice the size of the input image. At the same time, a larger teacher semantic segmentation network is used to perform knowledge distillation on the small model, thereby improving the final performance. This invention combines the disadvantages and advantages of the prior art (1) and the prior art (2) related to this invention. While ensuring accuracy, it effectively reduces the computational load and can run on edge devices. Attached Figure Description

[0030] Appendix Figure 1 This is a schematic diagram of the knowledge distillation model;

[0031] Appendix Figure 2 A schematic diagram of the LaneNet+H-Net neural network model for lane line detection;

[0032] Appendix Figure 3 The network model for the SAD module;

[0033] Appendix Figure 4 This is a diagram illustrating the interrelationships of the four modules;

[0034] Appendix Figure 5 The network structure is DnCNN;

[0035] Appendix Figure 6 This is the network structure of the lane line detection module. Detailed Implementation

[0036] The following is in conjunction with the appendix Figures 1 to 6 The present invention will be further described below.

[0037] To obtain high-resolution final feature maps and improve segmentation accuracy, semantic segmentation networks have mainly developed in two directions: the Encoder-Decoder approach. Encoder-Decoder networks use FCNs (Full Cell Networks) as encoders to capture high-level semantic information, and design decoders to gradually recover spatial semantic information. For example, SegNet utilizes pooling layer indices saved during the downsampling stage to recover reduced spatial information; U-Net uses skip connections to fuse features from different levels in the decoder, enriching the final predicted semantic feature map; and RefineNet uses a multi-path decoder to improve segmentation performance. However, Encoder-Decoder networks ignore the global background, and most methods simply fuse features from adjacent stages without considering the differences in their representations. This leads to some pixels belonging to the same category being misclassified.

[0038] When designing semantic segmentation models, there's always a desire to use complex models to achieve better results. However, complex networks mean deeper hidden layers, leading to significant redundancy in training parameters and making the model difficult to deploy on devices with limited computing power. To find a better balance between network accuracy and efficiency, Hinton et al. first proposed the knowledge distillation technique, which trains on the dataset using the original labels, such as... Figure 1 In recent years, deep neural networks have achieved significant success in both industry and academia, particularly in computer vision tasks. This success is largely attributed to their scalability in encoding massive amounts of data (feature extraction) and manipulating billions of model parameters. However, deploying these complex deep models on resource-constrained devices (e.g., mobile phones and embedded devices) presents a challenge: not only is computationally complex high, but storage requirements are also substantial. To address this, various model compression and acceleration techniques have been proposed; among them, knowledge distillation, which can effectively learn small "student" models from large "teacher" models, has rapidly gained attention. The original knowledge distillation framework typically comprises one or more large pre-trained teacher models and a small student model, with the teacher model usually much larger than the student model. The main idea is to train an efficient student model under the guidance of the teacher model to achieve competitive performance. The supervisory signals from the teacher model (often referred to as the "knowledge" learned by the teacher model) can help the student model mimic the behavior of the teacher model.

[0039] Super-resolution (SR) refers to reconstructing a corresponding high-resolution image from an observed low-resolution image. This is inherently an ill-posed problem because for a single low-resolution (LR) image, there are usually many corresponding high-resolution (HR) images. In recent years, with the rapid development of deep learning technology, deep learning-based SR models have been actively explored and have achieved state-of-the-art performance on current SR evaluation metrics. From early methods based on convolutional neural networks to more recent methods based on generative adversarial networks (GANs), various deep learning methods have been widely applied to SR tasks.

[0040] Prior art related to this invention (1)

[0041] In 2018, Davy Neven et al. proposed a lane detection neural network model, LaneNet+H-Net, which can perform end-to-end lane detection. The model structure is as follows: Figure 2 As shown, the main network models include LaneNet and H-Net. LaneNet is a multi-task model. One branch performs binary semantic segmentation of lane lines, distinguishing between lane lines and background. The other branch uses the lane line segmentation mask output by the first branch as input. During training, it learns to cluster the segmented lane line masks using a pre-designed clustering loss function. Combining the two branches, the LaneNet+H-Net network achieves end-to-end lane line instance segmentation. LaneNet only segments lane lines, but curve fitting is generally required. Usually, before curve fitting, the image is projected as a bird's-eye view to reduce the maximum number of curve fitting iterations. A transformation matrix H is used during projection. This transformation matrix H is usually calculated only once and then applied to all images in the dataset. However, since road surfaces are not perfectly smooth, the bird's-eye view obtained after applying the transformation matrix H to images taken in sloping areas will contain errors. To avoid the influence of errors caused by uneven road surfaces, this paper proposes an H-Net to predict the transformation matrix for each image; by combining LaneNet and H-Net, this network achieves excellent lane detection performance.

[0042] The drawback of the above solution is that it requires a large amount of computation, making it unsuitable for deployment on edge devices.

[0043] Prior art (2) related to this invention;

[0044] In 2019, Yuenan Hou et al. proposed a network model based on the SAD (Self Attention Distillation) module, the network structure of which is as follows: Figure 3 As shown. Self-attention modules are considered one of the methods to improve the performance of semantic segmentation models, but the large amount of computation they bring is also undeniable. This paper adopts a self-attention distillation module, using it as a guide for network training. When a simple model is half-trained, self-attention modules are introduced at different layers of the network, and the loss function of the output of the self-attention modules in adjacent layers is calculated. This allows low-level features to learn high-level semantic features, and high-level features to receive help from low-level detailed features, thereby improving the network's feature recognition ability. After training is complete, the self-attention module is not used during prediction. By using self-attention distillation, network performance is improved while the computational cost is significantly reduced.

[0045] The drawback of the above scheme is that although it is fast, the output segmentation map is not accurate enough.

[0046] Current semantic segmentation models require a large feature map size and high-resolution input images to achieve good performance, which greatly increases the computational cost of semantic segmentation and hinders its application on devices with limited computing power. In short, this invention aims to solve the contradiction between model computational cost and model performance.

[0047] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0048] A lane detection method and apparatus combining super-resolution and knowledge distillation is proposed. By inputting a small-sized image, the majority of the model's computation is performed at a small resolution to significantly reduce the computational load. Then, drawing on the idea of ​​super-resolution, super-resolution is applied to both the semantic segmentation output and the image, resulting in an output result that is twice the size of the input image. Simultaneously, a larger teacher semantic segmentation network is used to perform knowledge distillation on this small model, thereby improving the final performance.

[0049] like Figure 4It can be divided into four modules: image information acquisition module, image preprocessing module, lane line detection module, and model output postprocessing module. The image information acquisition module uses a camera to acquire image information in front of the vehicle. The image preprocessing module performs noise reduction on the image information transmitted from the image information acquisition module. The lane line detection module performs semantic segmentation on the output of the image preprocessing module. The model output postprocessing module obtains lane line information in the image based on the output of the lane line detection module.

[0050] First, in the image information acquisition module: the camera collects image data generated during the vehicle's movement and transmits the image information to the image preprocessing module.

[0051] Secondly, in the image preprocessing module: denoising is performed on the image information. Specifically, DnCNN is used, and its network structure is as follows: Figure 5 As shown;

[0052] The first stage of the network uses a 3x3 convolutional kernel and ReLU as the activation function; the second stage of the network uses a 3×3 convolutional kernel, a BN (batch normalization) layer, and ReLU; the third stage of the network uses a 3×3 convolutional kernel, and each layer is zero-padding to ensure that the input and output sizes of each layer remain consistent.

[0053] The loss function is:

[0054]

[0055] Where x i For a clean image, y i R represents the noisy image, θ represents the residual network, θ represents the parameters of the residual network, and N represents the number of images input in a batch.

[0056] Then, in the lane detection module: semantic segmentation is performed on the image preprocessing output, and its network structure is as follows: Figure 6 .

[0057] It includes the teacher module and the student module.

[0058] In the Teacher module: the input image size is 1024x1024. The Encode part network structure consists of 5 stacked residual networks with 3x3 convolutional kernels, downsampled to a feature map of size 128x128. This 128x128 feature map is then input into the Teacher Decode network, passed through 5 layers of deconvolutional networks, upsampled to 1024x1024, and finally passed through a 1x1 convolutional network to output a lane line prediction map.

[0059] In the Student module: the input image size is 256x256. The Encode part network structure consists of two stacked residual networks with 3x3 convolutional kernels, downsampled to a feature map of size 128x128. This 128x128 feature map is then input into the Student Decode network; after passing through five layers of deconvolutional networks, it is upsampled to a size of 1024x1024, and finally output as a lane line prediction map after passing through a 1x1 convolutional network.

[0060] During the training of the Student module, the feature maps of layers 1, 3, and 5 in the Decode part of the Teacher network are compared with the feature maps of layers 1, 3, and 5 in the Decode part of the Student module to calculate the L1 loss. The L1 loss formula is as follows:

[0061]

[0062] In the formula, y i The feature set output by the Teacher. Let n be the feature set output by Student, and n be the set of feature maps.

[0063] Furthermore, during the training of the Teacher and Student networks, the cross-entropy loss function is applied to the output results. The cross-entropy loss function is used to calculate the binary classification loss function, and its formula is as follows:

[0064] L=-[ylogy′+(1-y)log(1-y′)]

[0065] Where y is the true value and y' is the estimated value.

[0066] Finally, in the model output post-processing module: the feature map output by the lane detection module is processed by the softmax function to output the final lane detection result, where the softmax function formula is as follows:

[0067]

[0068] Each pixel in the output corresponds to an array Z, where Zi indicates that the i-th element Zj in Z is the sum of array Z;

[0069] After passing through the softmax function, the output is a 1x1024x1024 prediction map, where a pixel value of 1 represents lane lines and a value of 0 represents non-lane lines.

[0070] This solution effectively reduces computational load while maintaining accuracy, and can run on edge devices.

[0071] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A lane detection method combining super-resolution and knowledge distillation, characterized in that: It includes an image information acquisition module, an image preprocessing module, a lane line detection module, and a model output postprocessing module. The image information acquisition module acquires image information in front of the vehicle using a camera. The image preprocessing module performs noise reduction processing on the image information transmitted from the image information acquisition module. The lane line detection module performs semantic segmentation on the output results of the image preprocessing module. The model output postprocessing module obtains lane line information in the image based on the output results of the lane line detection module. The network structure for semantic segmentation of image preprocessing output includes a teacher module and a student module. In the Teacher module: the input image size is 1024x1024. The Encode part network structure is a stack of 5 residual networks with 3x3 convolutional kernels, downsampled to a feature map of size 128x128. The 128x128 feature map is input into the Teacher Decode network, passed through 5 layers of deconvolutional networks, upsampled to a size of 1024x1024, and then passed through a 1x1 convolutional network to output a lane line prediction map. In the Student module: the input image size is 256x256. The Encode part network structure is a stack of two residual networks with 3x3 convolutional kernels, downsampled to a feature map of size 128x128. The 128x128 feature map is input into the StudentDecode network. After passing through 5 layers of deconvolutional networks, it is upsampled to a size of 1024x1024. After passing through a 1x1 convolutional network, the lane line prediction map is output.

2. The lane detection method combining super-resolution and knowledge distillation according to claim 1, characterized in that: First, in the image information acquisition module, image data generated during vehicle movement is collected by the camera and transmitted to the image preprocessing module; second, in the image preprocessing module, noise reduction processing is performed on the image information.

3. The lane detection method combining super-resolution and knowledge distillation according to claim 2, characterized in that: Specific steps for denoising image information: The network structure adopted is DnCNN. The first stage of the network uses 3x3 convolutional kernels and ReLU as the activation function. The second stage of the network uses 3×3 convolutional kernels, batch normalization layers and ReLU. The third stage of the network uses 3×3 convolutional kernels and zero padding in each layer to keep the input and output sizes of each layer consistent. The loss function is: Where x i For a clean image, y i R represents the noisy image, θ represents the residual network, θ represents the parameters of the residual network, and N represents the number of images input in a batch.

4. The lane detection method combining super-resolution and knowledge distillation according to claim 3, characterized in that: When training the Student module, the feature maps of the first, third, and fifth layers of the Decode part of the Teacher network are compared with the feature maps of the first, third, and fifth layers of the Decode part of the Student module to calculate the L1 loss.

5. The lane detection method combining super-resolution and knowledge distillation according to claim 4, characterized in that: The L1 loss formula is as follows: In the formula, y i The feature set output by the Teacher. Let n be the feature set output by Student, and n be the set of feature maps.

6. The lane detection method combining super-resolution and knowledge distillation according to claim 5, characterized in that: During the training of the Teacher and Student networks, the cross-entropy loss function is applied to the output results. The cross-entropy loss function is used to calculate the binary classification loss function, and its formula is as follows: L=-[ylogy′+(1-y)log(1-y′)] Where y is the true value and y' is the estimated value.

7. The lane detection method combining super-resolution and knowledge distillation according to claim 6, characterized in that: In the model output post-processing module: the feature map output by the lane detection module is processed by the softmax function to output the final lane detection result, where the softmax function formula is as follows: Each pixel in the output corresponds to an array Z, where Zi represents the i-th element in Z, and Zj is the sum of array Z; After passing through the softmax function, the output is a 1x1024x1024 prediction map, where a pixel value of 1 represents lane lines and a value of 0 represents non-lane lines.

Images