A lane detection method using lidar based on deep segmentation model

Abstract

A LiDAR lane detection method based on a deep segmentation model uses point cloud data collected by intelligent driving devices during driving and employs a multi-scale deep convolutional network to detect lane markings. This method uses deep learning technology in conjunction with the intelligent driving device's LiDAR to iteratively update the model. This method, unlike traditional camera-based recognition methods, reduces the hardware requirements of intelligent driving devices and reduces their cost.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to a method for detecting lane lines using a laser radar based on a depth segmentation model. Background Art

[0002] With China's rapid urbanization, new infrastructure has become a major focus. Among the sub-sectors of new infrastructure, 5G, the Internet of Things (IoT), satellite internet, artificial intelligence (AI), data centers, and intelligent transportation infrastructure are all closely related to intelligent driving technology. Intelligent driving, situated at the intersection of these sectors, will serve as a key implementation scenario to drive the development of new infrastructure. Perception is crucial in intelligent driving technology, driven by a diverse array of sensing devices, diverse data modalities, and large volumes of data, along with complex task outputs. Machine learning technologies, such as computer vision, natural language processing, and recommendation systems, have been widely applied in these areas. The successful application of these machine learning technologies relies on vast amounts of data. However, in practical applications, the increased number of sensing devices inevitably increases terminal costs, making intelligent driving technology more difficult to implement. The trend is to implement intelligent driving technology using a single sensing device. LiDAR, a leading sensing device, has established a firm foothold in the intelligent driving sensing market thanks to its high sensing accuracy, high recognition accuracy, and more direct data processing. This reduces the pressure on intelligent driving system chips and algorithms, resulting in greater safety and reliability. Summary of the Invention

[0003] In order to overcome the shortcomings of the above technologies, the present invention provides a lidar lane line detection method based on a deep segmentation model, which can improve the reliability and efficiency of intelligent driving perception software and reduce the cost of intelligent driving hardware.

[0004] The technical solution adopted by the present invention to overcome the technical problems is:

[0005] A method for detecting lane lines using a laser radar based on a depth segmentation model includes the following steps:

[0006] a) Collect lidar point cloud and camera data with the same timestamp;

[0007] b) Selecting the near-ground data based on the Z-axis value of the laser radar point cloud data;

[0008] c) Map the LiDAR and camera data to the same coordinate system based on the LiDAR and camera extrinsic matrix;

[0009] d) Convert the lidar point cloud data into a bird’s-eye view and make it consistent with the camera dimensions;

[0010] e) Mark the required lane lines based on camera data;

[0011] f) Input the lidar point cloud data into a multi-scale convolutional neural network, and use the camera-annotated label data as the target for regression training of the multi-scale convolutional neural network;

[0012] g) The trained multi-scale convolutional neural network performs model inference and outputs lane lines.

[0013] Further, step g) comprises the following steps:

[0014] g-1) Select the near-ground data based on the Z-axis value of the laser radar point cloud data;

[0015] c) transforming the laser radar data coordinate system according to the laser radar's external parameter matrix;

[0016] d) Convert the lidar point cloud data into a bird's-eye view and input it into the trained multi-scale convolutional neural network for lane line detection.

[0017] Furthermore, the multi-scale convolutional neural network is composed of a high-granularity branch, a medium-granularity branch, and a low-granularity branch. The high-granularity branch is sequentially composed of an input layer IL and X skip connection layers SCL, the medium-granularity branch is sequentially composed of an input layer IL and Y skip connection layers SCL, and the low-granularity branch is sequentially composed of an input layer IL and Z skip connection layers SCL. The skip connection layer SCL is composed of a first convolutional layer CL and a second convolutional layer CL. The features output after inputting the first convolutional layer CL are spliced ​​with the features before inputting the first convolutional layer CL and then input into the second convolutional layer CL. The lidar point cloud data of size M×M is input into the high-granularity branch, the lidar point cloud data of size N×N is input into the medium-granularity branch, and the lidar point cloud data of size O×O is input into the low-granularity branch. The output of the high-granularity branch, the output of the medium-granularity branch, and the output of the low-granularity branch are input into the splicing layer CN for splicing, and the output of the splicing layer CN is used as the output of the multi-scale convolutional neural network.

[0018] Preferably, the value of X is 50, the value of Y is 20, and the value of Z is 10.

[0019] Preferably, the value of M is 640, the value of N is 64, and the value of O is 8.

[0020] The present invention achieves the following benefits: By using intelligent driving terminals to collect point cloud data during driving, lane markings are detected using a multi-scale deep convolutional network. This approach utilizes deep learning technology in conjunction with the intelligent driving terminal's lidar to iteratively update the model, unlike traditional camera-based recognition methods. This reduces the number of intelligent driving terminal hardware devices and reduces the cost of intelligent driving hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1This is a structural diagram of the multi-scale convolutional neural network of the present invention;

[0022] Figure 2 It is a structural diagram of the skip connection layer of the present invention. DETAILED DESCRIPTION

[0023] The following is combined with Figure 1 , Attachment Figure 2 The present invention is further described.

[0024] A method for detecting lane lines using a laser radar based on a depth segmentation model includes the following steps:

[0025] a) Collect lidar point cloud and camera data with the same timestamp.

[0026] b) Select the near-ground data based on the Z-axis value of the lidar point cloud data.

[0027] c) Based on the extrinsic parameter matrices of the lidar and camera, map the lidar and camera data to the same coordinate system.

[0028] d) Convert the LiDAR point cloud data into a bird’s-eye view and make it consistent with the camera dimensions.

[0029] e) Mark the required lane lines based on the camera data.

[0030] f) The lidar point cloud data is input into the multi-scale convolutional neural network, and the camera-annotated label data is used as the target for regression training of the multi-scale convolutional neural network.

[0031] g) The trained multi-scale convolutional neural network performs model inference and outputs lane lines.

[0032] By using intelligent driving terminals to collect point cloud data during driving, lane markings are detected using a multi-scale deep convolutional network. Deep learning technology is combined with the intelligent driving terminal's lidar to iteratively update the model. This approach, unlike traditional camera recognition methods, reduces the number of intelligent driving terminal hardware devices and reduces intelligent driving hardware costs.

[0033] Example 1:

[0034] Step g) comprises the following steps:

[0035] g-1) Select the near-ground data based on the Z-axis value of the laser radar point cloud data;

[0036] c) transforming the laser radar data coordinate system according to the laser radar's external parameter matrix;

[0037] d) Convert the lidar point cloud data into a bird's-eye view and input it into the trained multi-scale convolutional neural network for lane line detection.

[0038] Example 2:

[0039] Deep learning is a machine learning algorithm based on learning representations of data. An observation (e.g., an image) can be represented in a variety of ways, such as as a vector of pixel intensity values ​​or, more abstractly, as a series of edges or regions of specific shapes. Using certain representations makes it easier to learn tasks from examples (e.g., face recognition or facial expression recognition). The benefit of deep learning is that it replaces manual feature generation with efficient algorithms for feature learning and hierarchical feature extraction using unsupervised or semi-supervised learning. Deep learning learns the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly helpful for interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to acquire human-like analytical learning capabilities and recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far surpass previous technologies. To date, several deep learning frameworks, such as deep neural networks, convolutional neural networks, deep belief networks, and recurrent neural networks, have been applied in computer vision, speech recognition, natural language processing, audio recognition, and bioinformatics, and have achieved excellent results.

[0040] Semantic segmentation is a deep learning algorithm that associates a label, or category, with each pixel in an image. It is used to identify sets of pixels that form distinct categories. For example, self-driving cars need to identify vehicles, pedestrians, traffic signs, sidewalks, and other road features. Semantic segmentation is used in a variety of applications, including autonomous driving, medical imaging, and industrial inspection.

[0041] The multi-scale convolutional neural network of the present invention is an improvement on the convolutional neural network. A convolutional neural network (CNN) is primarily composed of an input layer, a hidden layer, a fully connected layer, and an output layer. The input layer can process multidimensional data. Typically, the input layer of a one-dimensional convolutional neural network receives a one-dimensional or two-dimensional array; a two-dimensional array may contain multiple channels; the input layer of a two-dimensional convolutional neural network receives a two-dimensional or three-dimensional array; and the input layer of a three-dimensional convolutional neural network receives a four-dimensional array. The hidden layers primarily include convolutional layers and pooling layers. The convolutional layer extracts features from the input data and contains multiple convolution kernels. Each element of the convolution kernel corresponds to a weight coefficient and a bias. The pooling layer performs feature selection and information filtering on the output feature map after the convolutional layer performs feature extraction to prevent overfitting of the convolutional neural network. The fully connected layer is primarily responsible for converting the two-dimensional feature map output by the convolution into a one-dimensional vector, thereby achieving an end-to-end learning process. The output layer uses a logical function or a normalization function to output the classification label. In object recognition problems, the output layer can be designed to output the center coordinates, size, and classification of the object. In image semantic segmentation, the output layer directly outputs the classification result of each pixel.

[0042] The main idea behind multi-scale convolutional neural networks is to sample the input at different granularities and extract features at different input scales. Generally speaking, denser sampling reveals more details, while larger sampling reveals overall trends.

[0043] While ensuring multi-scale input, in order to build more hidden layers to extract features, the idea of ​​skip connection is also introduced here. The original intention of skip connection is to solve the problem of vanishing gradients. When updating the parameters of a deep neural network, the gradient of each layer is usually calculated from the output layer of the network to the backward input layer. Since the gradient is usually a value less than 1, when there are many layers, the gradient will become smaller and smaller. Eventually, the problem of vanishing gradients arises. When the gradient is infinitely close to 0, the network has no way to update and learn. In order to solve this problem, the idea of ​​skip connection was proposed: in short, additional shallow inputs are added to the middle layer of the deep network so that the "path" of the gradient is no longer so long. It is similar to providing a composite path. On the basis of the original "long path", an additional "shortcut" is now added. Skip connection essentially provides an additional "shortcut" for calculating the gradient. As shown in the attached figure Figure 2 The figure shows how skip-layer connections are implemented. CL stands for convolutional layer. The proposed multi-scale convolutional neural network removes pooling and fully connected layers, adds more convolutional layers to extract features, and finally concatenates feature maps of different scales before outputting the result through the output layer.

[0044] Specifically, in the present invention, the multi-scale convolutional neural network is composed of a high-granularity branch, a medium-granularity branch and a low-granularity branch. The high-granularity branch is sequentially composed of an input layer IL and X skip connection layers SCL, the medium-granularity branch is sequentially composed of an input layer IL and Y skip connection layers SCL, and the low-granularity branch is sequentially composed of an input layer IL and Z skip connection layers SCL. The skip connection layer SCL is composed of a first convolutional layer CL and a second convolutional layer CL. The features output after inputting the first convolutional layer CL are spliced ​​with the features before inputting the first convolutional layer CL and then input into the second convolutional layer CL. The lidar point cloud data of size M×M is input into the high-granularity branch, the lidar point cloud data of size N×N is input into the medium-granularity branch, and the lidar point cloud data of size O×O is input into the low-granularity branch. The output of the high-granularity branch, the output of the medium-granularity branch and the output of the low-granularity branch are input into the splicing layer CN for splicing, and the output of the splicing layer CN is used as the output of the multi-scale convolutional neural network. The value of X is 50, the value of Y is 20, and the value of Z is 10. The value of M is 640, the value of N is 64, and the value of O is 8.

[0045] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A laser radar lane detection method based on a deep segmentation model, characterized in that: The steps include: a) Collect lidar point cloud and camera data with the same timestamp; b) Selecting the near-ground data based on the Z-axis value of the laser radar point cloud data; c) Map the LiDAR and camera data to the same coordinate system based on the LiDAR and camera extrinsic matrix; d) Convert the lidar point cloud data into a bird’s-eye view and make it consistent with the camera dimensions; e) Mark the required lane lines based on camera data; f) Input the lidar point cloud data into a multi-scale convolutional neural network, and use the camera-annotated label data as the target for regression training of the multi-scale convolutional neural network; g) The trained multi-scale convolutional neural network performs model inference and outputs lane lines.

2. The laser radar lane detection method based on the depth segmentation model according to claim 1 is characterized in that: Step g) comprises the following steps: g-1) Select the near-ground data based on the Z-axis value of the laser radar point cloud data; c) transforming the laser radar data coordinate system according to the laser radar's external parameter matrix; d) Convert the lidar point cloud data into a bird's-eye view and input it into the trained multi-scale convolutional neural network for lane line detection.

3. The method for lane line detection using a laser radar based on a depth segmentation model according to claim 1, wherein: The multi-scale convolutional neural network is composed of a high-granularity branch, a medium-granularity branch and a low-granularity branch. The high-granularity branch is composed of an input layer IL and X skip connection layers SCL in sequence. The medium-granularity branch is composed of an input layer IL and Y skip connection layers SCL in sequence. The low-granularity branch is composed of an input layer IL and Z skip connection layers SCL in sequence. The skip connection layer SCL is composed of a first convolutional layer CL and a second convolutional layer CL. The features output after inputting the first convolutional layer CL are spliced ​​with the features before inputting the first convolutional layer CL and then input into the second convolutional layer CL. The lidar point cloud data of size M×M is input into the high-granularity branch, the lidar point cloud data of size N×N is input into the medium-granularity branch, and the lidar point cloud data of size O×O is input into the low-granularity branch. The output of the high-granularity branch, the output of the medium-granularity branch and the output of the low-granularity branch are input into the splicing layer CN for splicing. The output of the splicing layer CN is used as the output of the multi-scale convolutional neural network.

4. The method for lane line detection using a laser radar based on a depth segmentation model according to claim 3, wherein: The X value is 50, the Y value is 20, and the Z value is 10.

5. The method for lane line detection using a laser radar based on a depth segmentation model according to claim 3, wherein: The value of M is 640, the value of N is 64, and the value of O is 8.

Images