A lane detection method based on neural network model

Abstract

A lane line detection method based on a neural network model includes the following four steps: step 1, a data sorting and collection module, which integrates multiple data sets, re-labels and cleans and screens, and constructs a new lane line detection training data set to ensure label consistency and data quality; step 2, an image data preprocessing module, in which the data preprocessing process uses cropping, scaling, flipping, deformation, tone adjustment, noise addition, median filtering and other methods to enhance the training image; step 3, a model training module, which adopts multi-model and multi-step training, and adjusts the data sampling distribution for different training stages; step 4, a detection result generation and fusion module, in which the lane line detection result is generated and fused, and two methods are used to enhance the output result. The method improves the accuracy and robustness of the lane line detection method by integrating multiple models, dynamically adjusting the data distribution and post-processing steps.

Description

Technical Field

[0001] The present invention relates to the technical field of lane line detection, and in particular to a lane line detection method based on a neural network model. Background Art

[0002] In the context of the booming development of driverless technology, lane detection, as a key component of the autonomous driving perception module, has important research significance and application value, and has attracted extensive attention from many scholars at home and abroad. Now scholars at home and abroad have conducted in-depth research on lane detection algorithms and achieved rich results.

[0003] The current lane detection algorithms are mainly divided into two categories: traditional detection algorithms and deep learning algorithms. Traditional methods can be further divided into feature-based and model-based. Feature-based algorithms detect lanes by extracting features such as color, texture, edge, direction, and shape, while model-based algorithms construct lane curve models for detection. Although traditional methods have achieved certain success in some cases, they are relatively poorly adaptable to complex scenes and changing lighting conditions.

[0004] Therefore, a lane line detection method based on a neural network model is proposed to solve the above problems. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a lane line detection method based on a neural network model, which can improve the accuracy and robustness of detection through a deep learning algorithm. By utilizing a deep neural network, high-level features in images can be learned and extracted to better adapt to complex road scenes. Convolutional neural networks are widely used in lane line detection.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] A lane line detection method based on a neural network model is characterized by comprising the following four steps:

[0008] Step 1: Data collection and organization,

[0009] Integrate the data of four original datasets, including BDD100K, DET, CULane, and TuSimple datasets, re-label and clean the annotation content in the datasets, and build a new lane detection training dataset;

[0010] Step 2: Image preprocessing,

[0011] The image preprocessing process uses cropping, image scaling, horizontal flipping, random deformation, Hue space saturation adjustment, noise addition and median filtering. The hyperparameters of each data preprocessing are set to enhance the images used for input model training and adjust the training data set images.

[0012] Step 3: Multi-model training,

[0013] YOLOv8, SegFormer and Deeplabv3p models are used for training to obtain multiple algorithm models suitable for lane line detection. The training data is adjusted to ensure that each model can learn useful features directly from the original input data. Appropriate optimization algorithms, loss functions and hyperparameters are used to optimize each model. Each model focuses on processing different types of features. The advantages of different models are combined to improve the overall performance, maximize the use of information from each model, and share the learned knowledge through transfer learning to improve the overall performance.

[0014] During the training process, different sampling methods were used to strengthen the training at different training stages;

[0015] Step 4: Lane detection result generation and fusion,

[0016] The image is input into the three trained models at the same time, and the output results are enhanced in two ways.

[0017] Specifically, the content of labeling and cleaning in step 1 includes: removal of other segmentation labels, unified identification of lane line labels, label ratio labeling and binning, and deleting or merging some labels marked as "other" or "irrelevant" in the data set to simplify the problem.

[0018] Specifically, the lane lines of the four original data sets in step one are marked in different ways, and need to be uniformly identified to ensure consistency, unify the representation of lane line labels, specify the color, shape, coordinate format, etc. of lane line labels, and mark the label ratio of each image.

[0019] Specifically, the specific contents of the hyperparameter settings of each data in the step 2 are as follows: cropping the upper area of ​​the image, such as setting a one-third ratio to crop the sky and other areas containing irrelevant information, and adjusting the image height and width to achieve scaling to complete the size adjustment to adapt to the input size of the model; horizontal flipping can usually set the flipping probability, increase the number of training images, adjust the saturation in the hue space to set a saturation adjustment range, and randomly adjust the image saturation within this range; noise addition sets the intensity of the noise, and selects the type of noise, which includes Gaussian noise and salt and pepper noise; median filtering sets the size of the appropriate filter kernel to adapt to the noise level and feature size of the image and retain image detail requirements.

[0020] Specifically, the specific contents of adopting different training reinforcement methods in different training stages in the step three are as follows: in the initial stage of training, in order to ensure the smooth progress of training, the data with a relatively large label proportion is resampled in a step-by-step manner according to the size of the label value; as the training progresses, the data sampling distribution is adjusted according to the performance of the model on the validation set, samples are increased for categories with poor performance, and samples are reduced for categories with good performance, so that the model pays more attention to categories that are difficult to learn; the data sampling distribution is dynamically adjusted during the training process, and the data sampling strategy is updated according to the real-time performance of the model; near the end of the training, the training of samples that are still difficult for the model is strengthened to further improve the generalization ability of the model; finally, the data sampling distribution is adjusted again to ensure that the model can learn the characteristics of each category in a balanced manner in the last few training stages; different data expansion methods are used in each stage to introduce diversity to prevent the model from overfitting to a certain category or sample; in the mid-term and final stages, the use of dropout and regularization and other means is strengthened to prevent the model from overfitting during training.

[0021] Specifically, the two methods in step 4 are as follows:

[0022] The first method is to perform intersection processing or weighted fusion on the output results of the model. For each model output, an appropriate threshold is used to determine which predictions are considered lane lines. The threshold setting is obtained from the training set. The model output is post-processed to remove unreasonable predictions, further improve the quality of the output, and obtain the summary results of the model.

[0023] The second method is to make color judgment on the coordinate points around the model results on the original image. If the color difference between the color and the segmentation mark is not significant, it will be marked as a lane line and all the results will be returned. An appropriate color difference threshold is set for the comparison between the color and the segmentation mark to ensure that the color judgment is robust. The coordinate points around the judgment results are combined with geometric information, such as the distance and slope of adjacent points, to further improve the accuracy of the judgment.

[0024] Beneficial effects of the present invention:

[0025] The present invention can improve the accuracy and robustness of detection through deep learning algorithms. By utilizing deep neural networks, high-level features in images can be learned and extracted to better adapt to complex road scenes. Convolutional neural networks are widely used in lane line detection. Through multiple layers of convolution and pooling layers, the network can automatically learn features in images without manually designing feature extractors. This makes the deep learning method more flexible and able to adapt to lane line detection in different scenes and under different lighting conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0027] Figure 2 A schematic diagram of the process for constructing the data set of the present invention;

[0028] Figure 3 It is a schematic diagram of the process of the image preprocessing step of the present invention;

[0029] Figure 4 It is a schematic diagram of a label image of the present invention;

[0030] Figure 5 Schematic diagram of the training process of the lane detection algorithm of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example 1

[0035] A lane line detection method based on a neural network model is characterized by comprising the following four steps:

[0036] Step 1: Data collection and organization, and dataset construction. Integrate the open source datasets BDD100K, DET, CULane, and TuSimple, and annotate and clean the original data, including: removal of other segmentation labels, unified identification of lane line labels, label ratio annotation and binning. Redefine the label content in the dataset, maintain label consistency, and annotate the label ratio in each image.

[0037] Step 2: Image data preprocessing. According to the characteristics of the constructed data set, the image preprocessing process was designed specifically. The process used cropping to remove the area content of irrelevant information such as the sky, increase the signal-to-noise ratio of the image content, fix the model data input size by image scaling, adjust the data by horizontal flipping and hue space saturation adjustment, add Gaussian noise and salt and pepper noise to the final image, and use median filtering to obtain the processed image.

[0038] Step 3: Multi-model training: YOLOv8, SegFormer, and Deeplabv3p models are used to train the constructed training dataset.

[0039] Among them, YOLOv8 is a SOTA model that builds on the success of previous YOLO versions and introduces new features and improvements to further enhance performance and flexibility. Specific innovations include a new backbone network, a new Ancher-Free detection head, and a new loss function that can run on a variety of hardware platforms from CPU to GPU.

[0040] YOLOv8 uses the most advanced backbone and intermediate architectures to provide better feature extraction and object detection performance. YOLOv8 uses the Ultralytics head with anchor-free segmentation, which can provide higher accuracy and more efficient detection process compared to anchor-based methods. YOLOv8 focuses on maintaining the best balance between accuracy and speed, and is suitable for a variety of real-time object detection tasks. YOLOv8 provides a range of pre-trained models to meet various tasks and performance requirements, making it easier to find a model suitable for a specific use case.

[0041] Changes to the YOLOv8 model structure:

[0042] Backbone: The CSP concept remains unchanged. YOLOv8 uses the C2f module. In order to reduce weight, the number of blocks in each stage is changed to [3,6,6,3]. In addition, the depth factor of the x version is still 1.0, the same as the L version. The stem convolution in YOLOv8 is a convolution with k=3. The final SPPF module v8 still uses v5.

[0043] Neck: YOLOv8 also removes the convolution in the top down upsampling stage of PAN-FPN in v5.

[0044] The head part changes from the original coupled head to a decoupled head, and from YOLOv5's Anchor-Based to Anchor-Free.

[0045] The Loss calculation process includes two parts: positive and negative sample allocation strategy and Loss calculation. Due to the superiority of the dynamic allocation strategy, TOOD's TaskAlignedAssigner is referenced in the YOLOv8 algorithm. Loss calculation includes two branches: classification and regression branches, without the previous objectness branch. The classification branch still uses BCE Loss. The regression branch needs to be bound to the integral form representation proposed in Distribution Focal Loss, so Distribution Focal Loss is used, and CIoU Loss is also used.

[0046] The reasoning process of the YOLOv8 model is as follows:

[0047] The bbox integral form is converted to 4D bbox format, the bbox branch output by the Head is converted, and the integral form is converted to 4D bbox format using Softmax and Conv calculations.

[0048] Dimension transformation, YOLOv8 outputs three feature maps with scales of 80x80, 40x40 and 20x20. The head part outputs feature maps of 6 scales for classification and regression. The category prediction branches and bbox prediction branches of 3 different scales are spliced ​​and dimensionally transformed. For the convenience of subsequent processing, the original channel dimension will be replaced to the end, and the shape of the category prediction branch and bbox prediction branch are (b,80x80+40x40+20x20,80)=(b,8400,80) and (b,8400,4) respectively.

[0049] Decode and restore to the original image scale classification prediction branch for Sigmoid calculation, while the bbox prediction branch needs to be decoded and restored to the real original image after decoding xyxy format;

[0050] Threshold filtering, traverse each image in the batch, and use score_thr to perform threshold filtering. In this process, multi_label and nms_pre also need to be considered to ensure that the number of detection boxes after filtering is not more than nms_pre;

[0051] Restore to the original image scale and nms. Based on the pre-processing process, restore the remaining detection boxes to the original image scale before the network output, and then perform nms. The final output detection box cannot be more than max_per_img. SegFormer is a simple, efficient but powerful semantic segmentation framework that takes efficiency, accuracy and robustness into consideration. Compared with previous methods, the framework redesigns the encoder and decoder. It unifies Transformers with a lightweight multi-layer perceptron (MLP) decoder. It consists of two main modules: a hierarchical Transformer encoder that generates high-resolution coarse features and low-resolution fine features; (2) a lightweight All-MLP decoder that fuses these multi-level features to produce the final semantic segmentation mask.

[0052] Hierarchical Feature Representation Unlike ViT, which can only generate single-resolution feature maps, this module aims to generate CNN-like multi-level features given an input image. Given an input image with resolution H*W*3, we perform patch merging to obtain a resolution of The hierarchical feature map F i , where i∈1,2,3,4 and C i+1 Greater than C i. Instead, we use an overlapping patch merging process. To this end, we define K, S, and P, where K is the patch size, S is the stride between two adjacent patches, and SP is the padding size. In our experiments, we set K=7, S=4, P=3 and K=3, P=1 to perform overlapping patch merging and S=2 to generate features of the same size as the non-overlapping process. Efficient self-attention. The main computational bottleneck of the encoder is the self-attention layer. The sequence reduction process introduced in PVT is used. This process uses a reduction ratio R to reduce the length of the sequence as follows:

[0053]

[0054] Where K is the sequence to be simplified, refers to a sequence reshaped into shape K, which refers to Linear(C in , C out )(·) Take the one-dimensional tensor C in Takes input and generates a one-dimensional tensor C out The output of the linear layer. Therefore, the new K has dimension As a result, the complexity of the self-attention mechanism is reduced from O(N 2 ) is reduced to In our experiments, we set R = [64, 16, 4, 1] from stage-1 to stage-4.

[0055] Mix-FFN is introduced, which considers the impact of zero padding on leaked position information and directly uses 3×3Conv in the feedforward network (FFN). Mix-FFN can be formulated as:

[0056] x out =MLP(GELU(Conv 3*3 (MLP(x in ))))+x in (3)

[0057] where x in is the function of the self-attention module. Mix-FFN mixes 3×3 convolution and MLP into each FFN. SegFormer avoids a complex decoder. The proposed MLP decoder aggregates information from different layers, thereby combining local attention and global attention to present a powerful representation. The proposed full MLP decoder consists of four main steps. First, the multi-level features F from the MiT encoder are iPass through an MLP layer to unify the channel dimension. Then, in the second step, the features are upsampled to 1 / 4 and concatenated together. Third, an MLP layer is used to fuse the concatenated features F. Finally, another MLP layer uses the fused features to predict a segmentation mask of resolution M×H / 4×W4×Ncls, where Ncls is the number of categories. This allows us to formulate the decoder as:

[0058]

[0059]

[0060] M=Linear(C,N els )(F), (7)

[0061] Where M refers to the predicted mask and Linear(C in , C out )(·) refers to the C in and as input and C out A linear layer with output vector dimensions.

[0062] DeepLabv3 uses convolution with upsampling filters to extract dense feature maps and capture long-range context. Specifically, in order to encode multi-scale information, the cascade module proposed by the authors gradually doubles the atrous rate, while the proposed atrous spatial pyramid pooling module increases image-level features.

[0063] Atrous convolutions, also known as dilated convolutions, allow reusing pre-trained networks to extract denser feature maps by removing the downsampling operations from the last few layers and upsampling the corresponding filter kernels, which is equivalent to inserting holes between the filter weights.

[0064] The use of atrous convolution is discussed in order to obtain more context information in the cascade module and spatial pyramid framework. Atrous convolution is mainly explored as a context module and tool for spatial pyramid pooling. The framework proposed in this invention is universal because it can be applied to any network.

[0065] Specifically, several copies of the original last block in ResNet are replicated and arranged in cascade, and the ASPP module is revisited, which contains multiple atrous convolutions in parallel. The cascaded module is applied directly to the feature map, rather than the belief map. For the proposed module, it was experimentally found to be important to use batch normalization for training. In order to further capture the global context, it is proposed to enhance ASPP with image-level features. Instead, it is advocated to use "atrous convolution", considering a two-dimensional signal, and for each position on the output and filter, a convolution is applied on the input feature map:

[0066] y[i]=∑ k x[i+r·k]w[k] (8),

[0067] where the atrous rate r corresponds to the stride with which we sample the input signal, which is equivalent to convolving the input x with upsampled filters that are produced by inserting r-1 zeros between two consecutive filter values ​​along each spatial dimension (hence the name atrous convolution, where the French word trous means hole in English). Standard convolution is a special case of the rate, while unconditional convolution allows us to adaptively modify the field of view of the filter by changing the rate r=1 value.

[0068] Atrous convolutions also allow explicit control over the density of feature responses computed in fully convolutional networks. Here, the invention is represented by output_stride, the ratio of the input image spatial resolution to the final output resolution. The stride of the last pooling layer or convolution layer that reduces the resolution is set to 1 to avoid signal extraction. All subsequent convolutional layers are then replaced by atrous convolutional layers with rate r=2. This enables the invention to extract denser feature responses without having to learn any additional parameters.

[0069] The introduced stride can easily capture long-range information in deeper blocks. In this proposed model, cascaded ResNet blocks are experimented, up to block7 (i.e., additional block5, block6, block7 as copies of block4), with atrous convolutions with output_stride=256 if no convolution is applied. Motivated by the multi-grid approach using grid hierarchies of different sizes, the present invention adopts different atrous rates within blocks 4 to 7 in the proposed model. Specifically, the present invention defines the unit rate of the three convolutional layers within block4 to block7 as Multi_Grid=(r1, r2, r3) unit rate. The final atrous rate of the convolutional layer is equal to the product of the unit rate and the corresponding rate.

[0070] Improved ASPP module: ASPP consists of dilated convolutions and BN layers with different dilation rates, and attempts to lay out the modules in a cascaded or parallel manner. Batch normalization is included in ASPP. ASPP with different atrous rates can effectively capture multi-scale information. However, as the sampling rate becomes larger, the number of effective filter weights (i.e., weights applied to valid feature regions instead of filling zeros) becomes smaller. When a 3×3 filter is applied to a feature 65×65 map with different atrous rates, in the extreme case where the rate value is close to the feature map size, the filter does not capture the entire image context, but degenerates into a simple 1×1 filter because only the center 3×3 filter weights are effective.

[0071] To overcome this problem and integrate global context information into the model, image-level features are used. Specifically, the present invention applies global average pooling on the last feature map of the model, feeds the generated image-level features into a 1×1 convolution with 256 filters (and batch normalization), and then bidirectionally upsamples the features to the required spatial dimensions.

[0072] Finally, the improved ASPP consists of a 1×1 convolution and three 3×3 convolutions with rates=(6, 12, 18) when output_stride=16 (all with 256 filters and batch normalization). When output_stride=8, the resulting features of all branches are then concatenated and passed through another 1×1 convolution (also with 256 filters and batch normalization) before generating the final 1×1 convolution for the final logarithm.

[0073] Step 4: Lane detection result generation and fusion:

[0074] Multi-model fusion method: The image is simultaneously input into three trained models: YOLOv8, SegFormer, and Deeplabv3, and the output results of each model are obtained.

[0075] The output of the model is processed by intersection or weighted fusion method, and the output is weighted according to the performance of the model on the validation set. The true labels of the lane lines in the training set and the confidence distribution of the model output are analyzed to set the appropriate threshold. According to the performance and characteristics of each model, the appropriate threshold is determined for each model output to determine the lane line prediction. The model output is post-processed to remove unreasonable predictions, smooth the output distribution, and improve the quality of the output. The results of each model are summarized to obtain the final lane line detection result.

[0076] Color detection method: On the original image, perform color judgment on the coordinate points around the model judgment result. If the color difference between the color and the segmentation mark is not significant, mark it as a lane line and return all the results. Set an appropriate color difference threshold for the comparison between the color and the segmentation mark to ensure the robustness of the color judgment. At the coordinate points around the judgment result, combine geometric information (including the distance and slope of adjacent points) to improve the accuracy of the judgment. Summarize the results of color detection and model fusion to obtain the final lane line detection result. The combination of multi-model fusion and color detection can improve the accuracy and robustness of lane line detection.

[0077] Example 2

[0078] Step 1: Build a dataset. Integrate the content of the open source datasets BDD100K, DET, CULane, and TuSimple, redefine the label content in the dataset, maintain label consistency, and annotate the label ratio in each image. Delete or merge some labels in the dataset that are marked as "other" or "irrelevant" to simplify the problem. Unify the original dataset identifiers to ensure consistency, unify the representation of lane line labels, specify the color, shape, coordinate format, etc. of the lane line labels, and annotate the label ratio of each image. The specific process of step one is as follows: Figure 2 shown.

[0079] Step 2: Image preprocessing. According to the characteristics of the constructed dataset, the image preprocessing process is designed specifically. The process uses cropping to remove the area content of irrelevant information such as the sky, increase the signal-to-noise ratio of the image content, fix the model data input size by image scaling, adjust the data by horizontal flipping and hue space saturation adjustment, add Gaussian noise and salt and pepper noise to the final image, and use median filtering to obtain the processed image. The specific process of step 2 is as follows: Figure 2 shown.

[0080] Step 3: Multi-model training. Use YOLOv8, SegFormer, and Deeplabv3p models to train the constructed training data set. During the training process, adjust the sampling distribution of the data for different training stages, and finally obtain multiple algorithm models suitable for lane line detection. Each model may focus on processing different types of features, and the advantages of different models are combined to improve the overall performance. The specific process of sampling in step 3 is as follows: Figure 4 shown.

[0081] Step 4: Generate and fuse lane detection results. Input the image into the three trained models at the same time to obtain the output results of each model. Perform weighted processing on the output results of the model. On the original image, perform color judgment on the coordinate points around the model judgment result. If the color difference between the color and the segmentation mark is not significant, mark it as a lane line and return all the results. Summarize the results of color detection and model fusion to obtain the final lane detection result.

[0082] Further, the multi-model training process of step three:

[0083] Data sampling method during model training: Before starting training, conduct a detailed analysis of the training data to understand the distribution and importance of each feature, identify features with a large label ratio, and pay attention to the problem of class imbalance.

[0084] In the initial stage of training, in order to ensure the smooth progress of training, the data with a larger proportion of labels are resampled in a step-by-step manner according to the size of the label value. The higher the proportion of data, the greater the probability of resampling. This prevents the model from occasionally reading a larger proportion of categories in the early stage of training, and improves the stability and generalization ability of training.

[0085] In the early stages of training, an average sampling strategy is adopted to enable the model to learn in a balanced manner in the early stages, and not be overly affected by features with a large label ratio. This ensures that the model can fully learn various features and thus establish a comprehensive feature representation.

[0086] In the mid-term, as training progresses, the data sampling distribution is dynamically adjusted based on the performance of the model on the validation set. For categories with poor performance, the strategy of increasing sample sampling is adopted, and for categories with good performance, the sample is reduced, so that the model pays more attention to the categories that are difficult to learn. In order to more fully capture the information of these features and improve the generalization performance of the model,

[0087] As the training progresses, for features that still have poor results, the number of samples is increased in a targeted manner through a strategy of further increasing the number of samples to strengthen the algorithm model's learning of these features, thereby optimizing the overall training effect and improving the accuracy and robustness of the model.

[0088] During the training process, different data augmentation methods are used to introduce diversity and prevent the model from overfitting. In the mid-term and final stages, dropout and regularization are used to further prevent the model from overfitting. Near the end of training, training on samples that are still difficult for the model is strengthened to improve the generalization ability of the model. Finally, the data sampling distribution is adjusted to ensure that the model can learn the characteristics of each category in a balanced manner in the last few training stages.

[0089] Adjust the training data, balance the brightness of the sampled data, adjust the scale, and fine-tune it to adapt to the input of each model, ensuring that each model can learn useful features directly from the original input data. During the training process, the input data is randomly transformed, enhanced, rotated, and horizontally flipped to simulate lane line detection and recognition at different angles. For the three models, use appropriate optimization algorithms, loss functions, and hyperparameters to optimize the model. Finally, multiple algorithm models suitable for lane line detection are obtained, each model focuses on processing different types of features, combining the advantages of different models to improve the overall performance and maximize the information of each model. Through transfer learning, some hierarchical feature representations are shared between different models, and the learned knowledge is shared to enhance the generalization ability and improve the overall performance.

[0090] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. The present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A lane line detection method based on a neural network model, characterized in that It includes the following four steps: Step 1: Data collection and organization, Integrate the data of four original datasets, including BDD100K, DET, CULane, and TuSimple datasets, re-label and clean the annotation content in the datasets, and build a new lane detection training dataset; Step 2: Image preprocessing, The image preprocessing process uses cropping, image scaling, horizontal flipping, random deformation, Hue space saturation adjustment, noise addition and median filtering. The hyperparameters of each data preprocessing are set to enhance the images used for input model training and adjust the training data set images. Step 3: Multi-model training, YOLOv8, SegFormer and Deeplabv3p models are used for training to obtain multiple algorithm models suitable for lane line detection. The training data is adjusted to ensure that each model can learn useful features directly from the original input data. Appropriate optimization algorithms, loss functions and hyperparameters are used to optimize each model. Each model focuses on processing different types of features. The advantages of different models are combined to improve the overall performance, maximize the use of information from each model, and share the learned knowledge through transfer learning to improve the overall performance. During the training process, different sampling methods were used for training reinforcement at different training stages; Step 4: Lane detection result generation and fusion, The image is input into the three trained models at the same time, and the output results are enhanced in two ways.

2. The lane line detection method based on a neural network model according to claim 1, characterized in that The content of labeling and cleaning in step 1 includes: removal of other segmentation labels, unified identification of lane line labels, label ratio labeling and binning, and deleting or merging some labels marked as "other" or "irrelevant" in the data set to simplify the problem.

3. The lane line detection method based on a neural network model according to claim 1, characterized in that The lane lines of the four original data sets in step 1 are marked in different ways, and need to be uniformly identified to ensure consistency. The representation of lane line labels is unified, and the color, shape, coordinate format, etc. of lane line labels are specified, and the label ratio of each image is marked.

4. The lane line detection method based on a neural network model according to claim 1, characterized in that The specific contents of the hyperparameter settings for each data in the step 2 are as follows: cropping the upper area of ​​the image, such as setting a one-third ratio to crop the sky and other areas containing irrelevant information, and adjusting the image height and width to achieve scaling to complete the size adjustment to adapt to the input size of the model; horizontal flipping can usually set the flipping probability, increase the number of training images, adjust the saturation in the hue space to set a saturation adjustment range, and randomly adjust the image saturation within this range; noise addition sets the intensity of the noise, and selects the type of noise, which includes Gaussian noise and salt and pepper noise; median filtering sets the size of the appropriate filter kernel to adapt to the noise level and feature size of the image and retain image detail requirements.

5. The lane line detection method based on a neural network model according to claim 1, characterized in that The specific contents of adopting different training reinforcement methods in different training stages in the step three are as follows: in the initial stage of training, in order to ensure the smooth progress of training, the data with a relatively large proportion of labels are resampled in a step-by-step manner according to the size of the label value. As the training progresses, the data sampling distribution is adjusted according to the performance of the model on the validation set, and samples are increased for categories with poor performance, and samples are reduced for categories with good performance, so that the model pays more attention to categories that are difficult to learn. The data sampling distribution is dynamically adjusted during the training process, and the data sampling strategy is updated according to the real-time performance of the model. Near the end of the training, the training of samples that are still difficult for the model is strengthened to further improve the generalization ability of the model. Finally, the data sampling distribution is adjusted again to ensure that the model can learn the characteristics of each category in a balanced manner in the last few training stages. Different data expansion methods are used in each stage to introduce diversity to prevent the model from overfitting to a certain category or sample. In the mid-term and final stages, the use of dropout and regularization and other means are strengthened to prevent the model from overfitting during training.

6. The lane line detection method based on a neural network model according to claim 1, characterized in that The specific contents of the two methods in step 4 are as follows: The first method is to perform intersection processing or weighted fusion on the output results of the model. For each model output, an appropriate threshold is used to determine which predictions are considered lane lines. The threshold setting is obtained from the training set. The model output is post-processed to remove unreasonable predictions, further improve the quality of the output, and obtain the summary results of the model. The second method is to make color judgment on the coordinate points around the model results on the original image. If the color difference between the color and the segmentation mark is not significant, it will be marked as a lane line and all the results will be returned. An appropriate color difference threshold is set for the comparison between the color and the segmentation mark to ensure that the color judgment is robust. The coordinate points around the judgment results are combined with geometric information, such as the distance and slope of adjacent points, to further improve the accuracy of the judgment.

Images