Road line distinguishing method based on deep learning in row dimension
Through deep learning methods based on row dimensions and structured loss functions, the problems of slow speed and small receptive field in complex scenarios are solved, and efficient and accurate lane detection is achieved, suitable for autonomous driving and vehicle assistance systems.
Patent Information
- Application Number
- CN202410917543.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-19
AI Technical Summary
The existing lane detection technology is slow in handling complex scenes and has a small receptive field, so it is impossible to detect accurately in real time, especially in high-resolution video streams, which are difficult to take into account both speed and accuracy.
Using a deep learning method based on row dimensions, the image is divided by row modules, and a multi-layer convolutional neural network is used to predict the lane line position, and a structured loss function is introduced to increase the receptive field and ensure the continuity and shape of the lane line. consistency.
It realizes real-time lane detection at low computing costs, improves detection accuracy and robustness in complex scenarios, and significantly improves the speed and accuracy of lane detection.
Smart Images

Figure CN118982805A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a road line identification method based on deep learning in a row dimension. Background Art
[0002] Traditional lane detection technology mainly relies on pixel-level segmentation. This type of method does not work well when dealing with complex scenes (such as severe occlusion and extreme lighting conditions), and has high computational costs, making it difficult to meet the needs of real-time processing. Although deep learning methods have made some progress in the field of lane detection in recent years, there is still a problem of difficulty in balancing speed and accuracy, especially in the scenario of real-time detection in high-resolution video streams. Most of today's traditional lane line detection has the following two problems: the first is slow speed, and the second is small receptive field, which cannot be judged based on global information. Summary of the invention
[0003] The purpose of the present invention is to provide a road line identification method based on deep learning in the row dimension to solve the shortcomings of existing lane detection methods in processing speed and accuracy, especially the technical problems of application in complex traffic scenarios (such as when a car blocks the view of the lane line).
[0004] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:
[0005] A road line identification method based on deep learning in the row dimension includes the following steps:
[0006] Step 1: Divide the entire image into h row dimensions, and then divide each row into row modules to obtain w row modules;
[0007] Step 2: Select modules corresponding to the positions of c lane lines on each row, specifically including the following steps: among the w modules obtained in each row, use the trained multi-layer convolutional neural network classification model to predict the existence and precise position of the lane line. The prediction method is mainly based on the multi-layer convolutional neural network classification model. The w row modules obtained are probability values belonging to a certain lane line. The position corresponding to a certain lane line in each row is determined according to the row vector composed of multiple probability values.
[0008] Step 3: Introduce a structured loss function to optimize the spatial continuity of lane positions;
[0009] Since general lane detection does not have a strong receptive field, it is difficult to make judgments based on a few row modules when encountering some visual obstructions. Therefore, we need to increase the receptive field and use the continuous and straight characteristics of lane lines to define structured loss to ensure the correct judgment of lane lines. The structured loss function consists of two parts: similarity loss and shape loss; similarity loss ensures that the lane position predictions on adjacent row anchor points are close in space, while shape loss forces the lane to remain straight or conform to the actual curved shape of the road;
[0010] The similarity loss is implemented by calculating the L1 norm of the first-order difference of the prediction results of adjacent row anchor points, and the shape loss constrains the curvature of the lane by calculating the second-order difference of the predicted position;
[0011] For similarity loss, the continuous characteristics of lane lines are used. The classification vectors of adjacent rows should be similar, which is represented by the first-order norm:
[0012]
[0013] The shape loss constrains the curvature of the lane by calculating the second-order difference of the predicted position;
[0014] Use softmax to process:
[0015] Prob i,j,: =softmaa(Pi, j,1:w ),
[0016]
[0017] Get the continuous value of the position, and then use the second-order differential form to write the expression of shape loss:
[0018]
[0019] This formula indicates that if it is a straight line, then the loss is 0.
[0020] Furthermore, in step 2, the prediction of the row module corresponding to the lane line on each row is completed by a multi-layer convolutional neural network, which inputs the w row modules obtained in step 1 and outputs the probability distribution of each potential lane position; the multi-layer convolutional neural network classification model is based on the classic convolutional neural network model for parameter optimization; for the optimized loss, define P i,j is the probability vector of lane change possibility corresponding to the i-th lane line on the j-th row, T i,j is a one-hot encoding, indicating the vector corresponding to the i-th lane line on the j-th row; when P i,j When it is 1, it means it is a lane line, P i,j When it is 0, it means it is not a lane line, and the loss is defined as:
[0021]
[0022] Where C represents the number of lane lines; L CE It is the cross entropy in mathematics.
[0023] The present invention discloses a road line identification method based on deep learning in the row dimension, which has the following advantages: because the present invention adopts a lower dimensional classification method, it can provide real-time lane detection function at an extremely low computing cost. Compared with many existing technologies, the speed is improved. At the same time, because the structured loss is taken into account, it can still maintain a high accuracy and robustness in complex scenarios, which greatly improves the practicality and reliability of lane detection technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 Schematic diagram of performing module segmentation in the row dimension and finding the modules corresponding to two lane lines at the same time:
[0025] Figure 2 (a) is a schematic diagram of the classification detection method of the present invention;
[0026] Figure 2 (b) is a schematic diagram of the traditional classification detection method;
[0027] Figure 3 This is a schematic diagram of lane line detection results obtained based on the Tusimple dataset of the present invention.
[0028] Figure 4 It is a structural diagram of the neural network used in the training process of the present invention. DETAILED DESCRIPTION
[0029] In order to better understand the purpose, structure and function of the present invention, a lane detection method based on global features and row selection of the present invention is further described in detail below with reference to the accompanying drawings.
[0030] A road line identification method based on deep learning in the row dimension includes the following steps:
[0031] Step 1: Divide the entire image into h row dimensions, and then divide each row into row modules to obtain w row modules;
[0032] Step 2: Select modules corresponding to the positions of c lane lines on each row, specifically including the following steps: among the w modules obtained in each row, use the trained multi-layer convolutional neural network classification model to predict the existence and precise position of the lane line. The prediction method is mainly based on the multi-layer convolutional neural network classification model. The w row modules obtained are probability values belonging to a certain lane line. The position corresponding to a certain lane line in each row is determined according to the row vector composed of multiple probability values.
[0033] The prediction of the row module corresponding to the lane line on each row is completed by a multi-layer convolutional neural network, which inputs the w row modules obtained in step 1 and outputs the probability distribution of each potential lane position; the multi-layer convolutional neural network classification model is based on the classic convolutional neural network model for parameter optimization; for the optimization loss, define P i,j is the probability vector of lane change possibility corresponding to the i-th lane line on the j-th row, T i,j is a one-hot encoding, indicating the vector corresponding to the i-th lane line on the j-th row; when P i,j When it is 1, it means it is a lane line, P i,j When it is 0, it means it is not a lane line, and the loss is defined as:
[0034]
[0035] Where C represents the number of lane lines.
[0036] Step 3: Introduce a structured loss function to optimize the spatial continuity of lane positions;
[0037] Since general lane detection does not have a strong receptive field, it is difficult to make judgments based on a few row modules when encountering some visual obstructions. Therefore, we need to increase the receptive field and use the continuous and straight characteristics of lane lines to define structured loss to ensure the correct judgment of lane lines. The structured loss function consists of two parts: similarity loss and shape loss; similarity loss ensures that the lane position predictions on adjacent row anchor points are close in space, while shape loss forces the lane to remain straight or conform to the actual curved shape of the road;
[0038] The similarity loss is implemented by calculating the L1 norm of the first-order difference of the prediction results of adjacent row anchor points, and the shape loss constrains the curvature of the lane by calculating the second-order difference of the predicted position;
[0039] For similarity loss, the continuous characteristics of lane lines are used. The classification vectors of adjacent rows should be similar, which is represented by the first-order norm:
[0040]
[0041] The shape loss constrains the curvature of the lane by calculating the second-order difference of the predicted positions;
[0042] It is processed using softmax:
[0043] Prob i,j,: = softmaa(P i,j,1:w )
[0044]
[0045] Continuous values of the positions are obtained, and the expression of the shape loss is written in the form of second-order differentiation:
[0046]
[0047] This equation indicates that if it is a straight line, the resulting loss is 0.
[0048] As Figure 1 shown, based on the example picture data of the present invention, L multi-classifications are performed in the row dimension to obtain the pixels corresponding to the road lines (L is the number of road lines), and the entire picture can be vertically divided into H such red lines for multi-classification. At the same time, a certain part at the top of the picture (where road lines will definitely not appear) is discarded, which is equivalent to only performing multi-classification on h samples.
[0049] As Figure 2 (a) shows, the present invention only needs to perform h*L multi-classifications on the picture. As Figure 2 (b) shows, the traditional full-pixel classification method needs to perform H*W classifications. Since h < H and L < W, the method of the present invention has a certain advantage in terms of detection speed.
[0050] As Figure 3 shown, when tested on the Tusimple dataset, good results are achieved, reaching approximately 94% in terms of accuracy, exceeding the accuracy corresponding to most traditional lane detection methods.
[0051] As Figure 4 shown, in terms of the overall architecture, the auxiliary branch is shown at the top and is only effective during training. The feature extractor is shown in the blue box. The classification-based prediction and the auxiliary segmentation task are respectively represented in the green and orange boxes. Group classification is performed on each row anchor.
[0052] The lane detection method adopted by the present invention no longer depends on the traditional pixel-by-pixel segmentation method, but is processed based on the global features of the image. Global features refer to the information extracted from the entire image, and these information can reflect the overall structure and layout of the lanes in the image. By analyzing the global features, this method can more robustly process various complex scenarios, such as partial occlusion of the lanes and poor lighting conditions.
[0053] In terms of technical implementation, the present invention uses a deep learning network to extract features from the input image. The network structure is designed to quickly extract high-level semantic information at a low computational cost. The network first processes the input image through multiple convolutional layers, which can capture the shape and direction of the lane at different scales through different convolution kernel sizes and step size configurations. Subsequently, the feature map is converted into a one-dimensional feature vector through a global average pooling layer to capture the global information of the entire image, providing a decision basis for subsequent lane position selection.
[0054] The core of the invention lies in the introduction of a row selection mechanism, which transforms the lane detection problem into a problem of selecting lane positions on predefined row anchor points. Each row anchor point represents a row in the image, and the lane positions are selected on these row anchor points based on global features. This approach significantly reduces the number of pixels to be processed because it avoids the high computational cost of traditional pixel-by-pixel classification.
[0055] In terms of technical implementation, row selection is achieved by setting fixed row anchors, and then using the trained classification model to predict the presence and precise location of lanes at these anchors. The prediction at each row anchor is done by a small fully connected network, which inputs global features and outputs a probability distribution of each potential lane location. In this way, the model is able to predict the location of lanes at different horizontal positions throughout the image, thereby achieving rapid recognition of the global layout of lanes.
[0056] In order to improve the accuracy and spatial continuity of lane detection, the present invention introduces a structured loss function. This loss function can ensure the continuity and smoothness of the detected lanes in space. The structured loss consists of two parts: similarity loss and shape loss. The similarity loss ensures that the lane position predictions on adjacent row anchor points are close in space, while the shape loss forces the lanes to remain straight or conform to the actual curved shape of the road.
[0057] The similarity loss is implemented by calculating the L1 norm of the first-order difference of the prediction results of adjacent row anchor points, and the shape loss constrains the curvature of the lane by calculating the second-order difference of the predicted position. This combined loss function makes lane detection not only accurate on each individual row anchor point, but also continuous and smooth in the shape of the lane in the entire image.
[0058] Embodiment 1:
[0059] Based on the Tusimple dataset, the neural network is trained with the training set data to obtain a network with low loss (including structural loss). Then, the data images to be tested are input into the neural network. After being processed by multiple layers of convolutional neural networks with set hyperparameters, the location of each lane line is obtained. After marking them, several complete lane lines can be obtained. (See attached) Figure 3 )
[0060] Embodiment 2:
[0061] Taking the dataset collected in real life as an example, the same operations as the existing datasets such as the Tusimple dataset can be performed to train and verify the neural network.
[0062] In practical applications, the present invention can significantly improve the speed and accuracy of lane detection. For example, in a highway environment, the present method can detect lanes in real time with extremely low latency, and can accurately identify lanes even when lane markings are unclear or damaged. In addition, the present invention is also applicable to complex urban traffic scenarios, and can effectively handle situations with dense traffic, multiple lanes, and various road conditions.
[0063] Therefore, the present invention provides an efficient and accurate lane detection method through innovative global feature extraction and row selection mechanism, combined with structured loss optimization, which is particularly suitable for real-time applications in autonomous driving and vehicle assistance systems. This method not only improves the performance of lane detection technology, but also reduces the implementation cost, and has important practical application value and broad market prospects.
[0064] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
Claims
1. A road line identification method based on deep learning in the row dimension, comprising the following steps: Step 1: Divide the entire image into h row dimensions, and then divide each row into row modules to obtain w row modules; Step 2: Select modules corresponding to the positions of c lane lines on each row, specifically including the following steps: among the w modules obtained in each row, use the trained multi-layer convolutional neural network classification model to predict the existence and precise position of the lane line. The prediction method is mainly based on the multi-layer convolutional neural network classification model. The w row modules obtained are probability values belonging to a certain lane line. The position corresponding to a certain lane line in each row is determined according to the row vector composed of multiple probability values. Step 3: Introduce a structured loss function to optimize the spatial continuity of lane positions; The structured loss function consists of two parts: similarity loss and shape loss; the similarity loss ensures that the lane position predictions on adjacent row anchors are spatially close, while the shape loss enforces the lane to remain straight or conform to the actual curved shape of the road; The similarity loss is implemented by calculating the L1 norm of the first-order difference of the prediction results of adjacent row anchor points, and the shape loss constrains the curvature of the lane by calculating the second-order difference of the predicted position; For similarity loss, the continuous characteristics of lane lines are used. The classification vectors of adjacent rows should be similar, which is represented by the first-order norm: The shape loss constrains the curvature of the lane by calculating the second-order difference of the predicted position; Use softmax to process: Prob i,j,: =softmaa(P i,j,1:w ), Get the continuous value of the position, and then use the second-order differential form to write the expression of shape loss: This formula indicates that if it is a straight line, then the loss is 0.
2. The road line identification method based on deep learning in the row dimension according to claim 1 is characterized in that: In step 2, the prediction of the row module corresponding to the lane line on each row is completed by a multi-layer convolutional neural network, which inputs the w row modules obtained in step 1 and outputs the probability distribution of each potential lane position; the multi-layer convolutional neural network classification model is based on the classic convolutional neural network model for parameter optimization; for the optimization loss, define P i,j is the probability vector of lane change possibility corresponding to the i-th lane line on the j-th row, T i,j is a one-hot encoding, indicating the vector corresponding to the i-th lane line on the j-th row; when P i,j When it is 1, it means it is a lane line, P i,j When it is 0, it means it is not a lane line, and the loss is defined as: Where C represents the number of lane lines.
Citation Information
Cited By
Lane line detection method and related device
CN121121685A
