A lane detection method to improve lightweight network detection accuracy

Abstract

This invention discloses a lane detection method for the field of autonomous driving that improves the detection accuracy of lightweight networks. The method comprises: Step 1: preprocessing training data; Step 2: creating a training set; Step 3: inputting the images from the training set obtained in Step 2 into two convolutional neural networks for training; and Step 4: after training in Step 3, cropping the real lane line images used for testing and inputting them into a student backbone network to obtain model prediction results. This lane detection method improves the detection performance of lightweight networks in the field of lane detection. It proposes a hybrid fusion module to construct multiple hierarchical student subnetworks with a top-down architecture. It also proposes an evolutionary distillation algorithm that improves the learning performance of the student network by using an evolving teacher and dynamically focusing on intermediate knowledge during the teaching process.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, and specifically relates to a lane line detection method for improving lightweight network detection accuracy. Background Art

[0002] Lane detection has a long research history in computer vision and has a wide range of applications. There are two mainstream approaches for lane detection: traditional image processing methods and deep segmentation methods. Recently, deep segmentation methods have achieved great success in this field due to their strong representation and learning capabilities. However, several important and challenging problems remain to be solved. As a fundamental component of autonomous driving, lane detection algorithms are widely used. This requires extremely low computational cost for lane detection. Furthermore, current autonomous driving solutions are typically equipped with multiple camera inputs, which typically requires low computational cost for each camera input. Therefore, a fast network is crucial for lane detection. Due to the dense prediction nature of segmentation, this method is computationally expensive. Another challenge with lane detection is the lack of visual cues. Challenging scenes with severe occlusion and extreme lighting conditions represent another key challenge for lane detection. In these situations, lane detection urgently requires higher-level semantic analysis of lanes. Deep segmentation methods naturally offer stronger semantic representation capabilities than traditional image processing methods, making them the mainstream approach. However, due to the dense pixel-level communication, this message passing incurs a higher computational cost. Furthermore, lanes are sometimes represented as binary segmented features rather than straight or curved lines. Although deep segmentation methods dominate the field of lane detection, this representation makes it difficult to explicitly utilize prior information such as lane rigidity and smoothness. Therefore, it is necessary to design a lane detection method that improves the detection accuracy of lightweight networks. To this end, we propose a lane detection method that improves the detection accuracy of lightweight networks to address this problem. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a lane line detection method that improves the detection accuracy of a lightweight network, so as to solve the problems raised in the above background technology.

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] A lane line detection method for improving lightweight network detection accuracy, the method comprising:

[0006] Step 1: Preprocess the training data, input the image obtained by the camera, and crop the image size;

[0007] Step 2: Read the training data preprocessed in step 1, process it into the standard format for input to the network, and create labels for the training network to create a training set;

[0008] Step 3: Input the images of the training set obtained in step 2 into two convolutional neural networks for training;

[0009] Step 4: After the training in step 3 is completed, the real lane line image used for testing is cropped and input into the student backbone network to obtain the model prediction result.

[0010] Preferably, in the step 3, the convolutional neural network body uses ResNet as the backbone network, and is composed of a ResNet18 student network, a ResNet50 teacher network and a guidance module. The input data is enhanced, and the enhanced data is input into the teacher network to obtain the feature map of the teacher network, soften the probability of the teacher network, calculate the teacher network's own distillation loss and classification loss, calculate the gradient of the model parameters, and use the SGD optimizer to update, input the data into the student network, obtain the feature map of the student network, soften the probability of the student network, calculate the student network's own distillation loss and classification loss, calculate the distillation loss between the student and teacher guidance layers, calculate the gradient of the student model parameters, use the SGD optimizer to update, and complete the training.

[0011] Preferably, the student network and the teacher network are both composed of 4 blocks and a guide module, and each block includes at least one convolution.

[0012] Preferably, the guiding module takes the last feature of the student network and the teacher network as initial input respectively, and then fuses the features through the hybrid fusion module.

[0013] Preferably, the hybrid fusion module is composed of addition and concatenation operators, with multiple hierarchical student sub-networks in a top-down architecture;

[0014] For features , upsample the spatial resolution by a factor of 2, using 1 1. The convolution operation is used to align the channel dimension between the two input features, and then the upsampled features are combined with the corresponding student features through addition and concatenation operations. Fusion, as shown below:

[0015] .

[0016] Preferably, the classification loss is as follows:

[0017]

[0018] The KL loss is shown below:

[0019]

[0020] The feature loss is as follows:

[0021]

[0022] The overall loss of the distillation framework is shown below:

[0023] .

[0024] Preferably, in step 4, the backbone network learns to generate high-precision labels for prediction and distillation and provide robust intermediate features during distillation.

[0025] Preferably, a lane line detector for improving lightweight network detection accuracy stores a program for running the lane line detection method for improving lightweight network detection accuracy.

[0026] Beneficial effects of the present invention:

[0027] 1. The lane detection method of the present invention improves the detection effect of lightweight networks in the field of lane detection and proposes a hybrid fusion module to construct multiple hierarchical student subnetworks with a top-down architecture;

[0028] 2. The lane line detection method of the present invention is different from the traditional two-stage offline distillation method. It narrows the ability gap through training to improve learning ability. It proposes an evolutionary distillation algorithm, which improves the performance of student network learning by using an evolutionary teacher and dynamically focusing on the intermediate knowledge of the teaching process. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 It is a flow chart of the method of the present invention;

[0031] Figure 2 It is a network structure diagram of the present invention;

[0032] Figure 3 This is a feature mixing and fusion module diagram in the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0034] See also Figures 1 to 3 As shown, a lane line detection method for improving lightweight network detection accuracy includes:

[0035] Step 1: Preprocess the training data, use the image obtained by the camera as input, and crop the image size to ;

[0036] Step 2: Create a training set. The first step in training a network using deep learning is to read the training data, process it into a standard format for input to the network, and create labels for the training network.

[0037] Step 3: Input the training set images into two convolutional neural networks for training:

[0038] After the training set images and labels are completed, the network model training phase can be entered. The convolutional neural network uses ResNet as the backbone network, which is composed of ResNet18 student network, ResNet50 teacher network and guidance module. The input data is enhanced and the enhanced data is input into the teacher network to obtain the feature map of the teacher network, soften the probability of the teacher network, and calculate the teacher network's own distillation loss and classification loss; calculate the gradient of the model parameters and use the SGD optimizer to update them, input the data into the student network to obtain the feature map of the student network, soften the probability of the student network, calculate the student network's own distillation loss and classification loss; calculate the distillation loss between the student and teacher guidance layers, calculate the gradient of the student model parameters, use the SGD optimizer to update, and finally complete the training.

[0039] The network components include:

[0040] Both ResNet18 and ResNet50 networks consist of four blocks and a guidance module, with each block containing several convolutions. The backbone network has two learning tasks during distillation. It not only learns to generate high-precision labels for prediction and distillation, but also attempts to provide robust intermediate features to guide the learning of multiple layered students. To leverage this intermediate knowledge, we refine the bottleneck module to form several guidance modules to assist in knowledge representation and transfer. A guidance module is inserted into each corresponding block between the teacher and student streams, making the distillation process effective and efficient both within and across streams. Intra-stream knowledge distillation aims to enhance knowledge representation, while cross-stream knowledge distillation helps improve knowledge transfer. The proposed evolutionary knowledge distillation employs synchronized online training of the teacher and student models, updating the parameters of the student and teacher models during each batch processing, which can reduce the gap between teacher and student capabilities. The teacher and student streams are divided into B blocks. Each block is followed by a guidance module with fully connected layers to form multiple classifiers. The training process consists of two simultaneous stages: intra-stream distillation and cross-stream distillation. For intra-stream distillation, deeper classifiers provide supervision to help learn shallow classifiers, which can improve the stream's ability to represent knowledge. Cross-stream distillation can improve the knowledge transfer from the evolving teacher to the student. The guidance module can help promote knowledge representation and transfer. Through such optimization, the backbone network can learn more generalizable features and significantly outperform the baseline network. During testing, the auxiliary intermediate sub-network can be simply removed while retaining the trained backbone network for deployment.

[0041] The components of the boot module include:

[0042] The guidance module takes the final features of the student and teacher backbone networks as initial inputs. It iteratively generates higher-resolution features by sampling spatially coarser but semantically stronger feature maps. These features are then fused with semantically weaker but spatially finer features via the hybrid fusion module. In this way, hierarchical block features are obtained. Additional convolutional blocks and fully connected layers are set after each block, with each block acting as an independent student classification model to generate soft probabilities for distillation.

[0043] The components of the hybrid fusion module include:

[0044] The hybrid fusion module consists of addition and concatenation operators with multiple hierarchical student sub-networks in a top-down architecture. Using addition and concatenation operations independently to fuse features from the encoder and decoder networks may hinder the information flow or bring redundancy to the network, resulting in performance degradation. In this work, we mix these two operations together to effectively bridge the information flow between the backbone teacher and multiple levels of students to combine their advantages and avoid possible limitations. Specifically, for spatially coarser features, , we upsample the spatial resolution by a factor of 2. Using 1 1 The convolution operation is used to align the channel dimension between the two input features. The upsampled features are then combined with the corresponding student features through addition and concatenation operations. Fusion.

[0045] It is expressed as the following formula:

[0046]

[0047] Yes 1 1 function of the convolutional block to align the number of channels between two features. is a 1 The convolution block of 1 combines the features after the concatenation operation and reduces the number of feature channels by half. . This fusion process is iterated until the latest feature map is generated We set the channel dimension of all student features to , with the last teacher feature The channel dimensions of are the same. Through our proposed hybrid fusion module, even the shallowest student sub-network can still obtain sufficient spatial and semantic information simultaneously, thus achieving better representation capabilities. Multiple more powerful student networks can provide more diverse and sufficient feedback signals, from which the backbone teacher can learn and be significantly improved.

[0048] The components of the loss calculation include:

[0049] In practice, we use classic cross entropy to calculate classification loss, KL divergence and L2 distance loss. We calculate the KL divergence loss between the deep classifier and the shallow classifier in the stream.

[0050] To train a multi-class classification network, we also minimize the predicted probability of each training sample and the true value label The traditional cross entropy between ) loss, the classification loss is as follows:

[0051]

[0052] Given a labeled dataset, use ( ) Divergence loss is used to minimize the difference between the soft output probabilities of the student network and the teacher network. The KL loss is as follows:

[0053]

[0054] After performing feed-forward computations through the multi-branch distillation framework, we obtain the proportionally sized student feature set S, the teacher feature set T, and the intermediate feature set F of the branches. Specifically, we aim to have the backbone teacher classifier guide the learning process of all auxiliary student classifiers. The guidance layer is chosen as the last layer of the convolutional block in the auxiliary branch. We minimize the L2 loss between the intermediate feature maps of the main teacher classifier and those of the other auxiliary student classifiers. The feature loss can be written as:

[0055]

[0056] The overall loss of our proposed distillation framework is as follows:

[0057]

[0058] in 、 and It is a hyperparameter that controls the impact of each loss term and also satisfies . represents all the predicted knots produced by the entire network. The first loss term is the total cross entropy loss of the ground truth labels of all auxiliary students and the backbone teacher network.

[0059] S4, after the training is completed, the real lane line image used for testing is cropped and input into the student backbone network to obtain the model prediction result.

[0060] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0061] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A lane line detection method for improving lightweight network detection accuracy, characterized by: The method comprises: Step 1: Preprocess the training data, input the image obtained by the camera, and crop the image size; Step 2: Read the training data preprocessed in step 1, process it into the standard format for input to the network, and create labels for the training network to create a training set; Step 3: Input the images of the training set obtained in step 2 into two convolutional neural networks for training; Step 4: After the training in step 3 is completed, the real lane line image used for testing is cropped and input into the student backbone network to obtain the model prediction result; In the step 3, the convolutional neural network body uses ResNet as the backbone network, and is composed of a ResNet18 student network, a ResNet50 teacher network and a guidance module. The input data is enhanced, and the enhanced data is input into the teacher network to obtain the feature map of the teacher network, soften the probability of the teacher network, calculate the teacher network's own distillation loss and classification loss, calculate the gradient of the model parameters, and use the SGD optimizer to update, input the data into the student network, obtain the feature map of the student network, soften the probability of the student network, calculate the student network's own distillation loss and classification loss, calculate the distillation loss between the student and teacher guidance layers, calculate the gradient of the student model parameters, use the SGD optimizer to update, and complete the training.

2. A lane line detection method for improving lightweight network detection accuracy according to claim 1, characterized in that: The student network and the teacher network are both composed of 4 blocks and a guidance module, and each block includes at least one convolution.

3. The lane line detection method for improving lightweight network detection accuracy according to claim 1 is characterized in that: The guiding module takes the last feature of the student network and the teacher network as initial input respectively, and then fuses the features through the hybrid fusion module.

4. The lane line detection method for improving lightweight network detection accuracy according to claim 3 is characterized in that: The hybrid fusion module consists of multiple hierarchical student sub-networks with a top-down architecture through addition and concatenation operators; For features , upsample the spatial resolution by a factor of 2, using 1 1. The convolution operation is used to align the channel dimension between the two input features, and then the upsampled features are combined with the corresponding student features through addition and concatenation operations. Fusion, as shown below: Yes 1 1 convolution block function, used to align the number of channels between two features, is a 1 The convolution block of 1 combines the features after the concatenation operation and reduces the number of feature channels by half. , iterate this fusion process until the latest feature map is generated , set the channel dimension of all student features to , with the last teacher feature The channel dimensions are the same.

5. The lane line detection method for improving lightweight network detection accuracy according to claim 1, characterized in that: The classification loss is as follows: The KL loss is shown below: The feature loss is as follows: The overall loss of the distillation framework is shown below: in 、 and is a hyperparameter that controls the impact of each loss term, satisfying represents all the predictions produced by the entire network, and the first loss term is the total cross entropy loss of the ground truth labels of all auxiliary students and the backbone teacher network.

6. The lane line detection method for improving lightweight network detection accuracy according to claim 1, characterized in that: In step 4, the backbone network learns to generate high-precision labels for prediction and distillation during distillation and provides robust intermediate features.

7. A lane line detector for improving lightweight network detection accuracy, storing a program for running a lane line detection method for improving lightweight network detection accuracy as described in any one of claims 1-6.

Images