A lane line detection method, device, equipment and storage medium

Abstract

The application discloses a lane line detection method, device and equipment and a storage medium. The method comprises the following steps: acquiring a point cloud image, generating a first pseudo image based on the laser reflection intensity of the point cloud image, generating a second pseudo image based on the depth information of the point cloud image, fusing the first pseudo image and the second pseudo image to obtain a fused image, and performing lane line detection on the fused image to obtain a lane line image. The application generates a pseudo image under a bird's-eye view based on two dimensions of laser reflection intensity and depth information, skips a complex three-dimensional convolution, reduces the complexity of calculation in a two-dimensional convolution manner, reduces the amount of calculation, improves the real-time performance of lane line detection, utilizes the advantages of point clouds, avoids errors caused by projecting a two-dimensional lane line to a three-dimensional world, avoids errors caused by external light, road bumps and the like on lane line detection, and improves the accuracy of lane line detection.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a lane line detection method, apparatus, device, and storage medium. Background Technology

[0002] Deep learning-based lane detection is gradually becoming a mainstream method for autonomous driving. However, in the lane detection domain, image-based methods have always been dominant, with very few radar-based methods. Similarly, real-time performance remains a significant obstacle to the implementation of radar-based lane detection. While the rapid development of AI chips in recent years has greatly improved the real-time performance of deep learning models, many real-time issues still remain unresolved. Image-based lane detection suffers from short-term blinding in strong light, low light interference, and tunnel entry / exit scenarios, further reducing its detection accuracy. Summary of the Invention

[0003] To address at least one of the aforementioned technical problems and improve the real-time performance and accuracy of lane line detection, this application proposes a lane line detection method.

[0004] According to one aspect of the disclosure of this application, a lane line detection method is provided, the method comprising:

[0005] Acquire point cloud images, which are images representing road surface information around the vehicle;

[0006] Based on the laser reflection intensity of the point cloud image above, a first pseudo image is generated;

[0007] Based on the depth information of the point cloud image, a second pseudo image is generated, and the second pseudo image has the same size as the first pseudo image.

[0008] The first pseudo-image and the second pseudo-image are fused together to obtain a fused image;

[0009] Lane line detection is performed on the above fused image to obtain a lane line image, which includes at least one of lane line information and curb information.

[0010] In some possible implementations, generating the first pseudo-image based on the laser reflection intensity of the point cloud image includes:

[0011] The above point cloud image is encoded to obtain a 3D point cloud encoded image;

[0012] The above three-dimensional point cloud encoded image is reduced in dimensionality to obtain a two-dimensional point cloud encoded image.

[0013] Feature extraction is performed on the above two-dimensional point cloud encoded image to obtain the above first pseudo image.

[0014] In some possible implementations, generating a second pseudo-image based on the depth information of the point cloud image includes:

[0015] The above point cloud image is subjected to point cloud detection processing to obtain a pseudo point cloud image;

[0016] The image of the pseudo point cloud from a top-down view is obtained and downsampled to obtain the second pseudo image mentioned above.

[0017] In some possible implementations, the above-described lane line detection of the fused image to obtain a lane line image includes:

[0018] Feature extraction is performed on the above fused image to obtain a feature image, which includes road feature information, including at least one of lane line feature information and curb feature information.

[0019] The above feature images are classified based on grid segmentation to obtain lane line images.

[0020] In some possible implementations, the above-described classification processing of the feature image based on grid segmentation to obtain a lane line image includes:

[0021] The above feature images are segmented based on a grid to obtain multiple anchor boxes;

[0022] Classify and predict each of the above anchor frames to obtain the lane line image.

[0023] In some possible implementations, the above-described classification and prediction of each anchor frame to obtain a lane line image includes:

[0024] For each of the above anchor frames, a classification prediction is performed to obtain the classification result and probability estimation result corresponding to each anchor frame. The classification result represents the category information of the corresponding anchor frame, and the probability estimation result represents the probability of each of the above anchor frames belonging to the category.

[0025] Based on the above classification results and the above probability estimation results, each of the above anchor frames is filtered, and the above lane line image is obtained according to the filtering results.

[0026] In some possible implementations, fusing the first pseudo-image and the second pseudo-image to obtain a fused image includes:

[0027] The first pseudo-image and the second pseudo-image are aligned and matched to obtain the matched image.

[0028] The matched images are then stitched together to obtain the fused image.

[0029] According to a second aspect of the present application, a lane line detection device is provided, the device comprising:

[0030] The point cloud image acquisition module is used to acquire point cloud images, which are images representing road surface information around the vehicle.

[0031] The first pseudo-image generation module is used to generate a first pseudo-image based on the laser reflection intensity of the point cloud image mentioned above.

[0032] The second pseudo-image generation module is used to generate a second pseudo-image based on the depth information of the point cloud image. The second pseudo-image has the same size as the first pseudo-image.

[0033] The fusion module is used to fuse the first pseudo-image and the second pseudo-image to obtain a fused image;

[0034] The lane line detection module is used to perform lane line detection on the above-mentioned fused image to obtain a lane line image, wherein the lane line image includes at least one of lane line information and curb information.

[0035] According to a third aspect disclosed in this application, a lane line detection device is provided, the device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the lane line detection method described above.

[0036] According to the fourth aspect disclosed in this application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, characterized in that the computer program instructions, when executed by a processor, implement the lane line detection method described above.

[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application.

[0038] Implementing this application will have the following beneficial effects:

[0039] This application employs a superpixel segmentation method for images viewed from above, which simplifies post-processing of model inference, reduces computational complexity and computational load, thereby improving the real-time performance of lane detection.

[0040] The advantage of this application in utilizing radar point clouds is that when projecting an object onto a top-down view, the physical size of the object is preserved, and the result of detecting the point cloud is the position in the real world. This avoids the errors caused by projecting two-dimensional lane lines into the three-dimensional world based on images, and also avoids the problems caused by external light, road bumps, and other conditions that may lead to errors in lane line detection, thereby improving the accuracy of lane line detection. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of this application, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This diagram illustrates a flow chart of a lane line detection method according to an embodiment of this application.

[0043] Figure 2 This diagram illustrates the lane line detection network structure according to an embodiment of this application.

[0044] Figure 3 This diagram illustrates a flowchart of a first pseudo-image generation method according to an embodiment of this application.

[0045] Figure 4 This application shows a top-view feature map of a two-dimensional point cloud encoded image according to an embodiment of the present application;

[0046] Figure 5 This diagram illustrates a flowchart of a second pseudo-image generation method according to an embodiment of this application.

[0047] Figure 6 This illustration shows a schematic diagram of the process of obtaining lane line images based on fused images according to an embodiment of this application;

[0048] Figure 7 A schematic diagram of the classification process based on grid segmentation according to an embodiment of this application is shown;

[0049] Figure 8 This diagram illustrates the classification prediction method of an embodiment of this application.

[0050] Figure 9 This diagram illustrates the flow chart of the image fusion method according to an embodiment of this application.

[0051] Figure 10 This diagram shows a block diagram of a lane line detection device according to an embodiment of this application;

[0052] Figure 11 This application illustrates a structural block diagram of an electronic device according to an embodiment of the present application. Figure 1 ;

[0053] Figure 12 This application illustrates a structural block diagram of an electronic device according to an embodiment of the present application. Figure 1 . Detailed Implementation

[0054] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0056] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0057] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0058] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0059] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed description. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0060] According to one aspect of the disclosure of this application, a lane line detection method is provided; please refer to [link to relevant documentation]. Figure 1 and Figure 2 , Figure 1 This diagram illustrates a flow chart of a lane line detection method according to an embodiment of this application. Figure 2 This diagram illustrates a lane detection network structure according to an embodiment of the present application. The method described above includes:

[0061] Step S101: Obtain a point cloud image, wherein the point cloud image is an image representing the road surface information around the vehicle;

[0062] In the embodiments of this application, the point cloud image is acquired using a lidar. The lidar can be installed on a vehicle and is used to acquire road surface information around the vehicle when performing lane line detection tasks while the vehicle is in motion.

[0063] Step S102: Generate a first pseudo image based on the laser reflection intensity of the point cloud image above;

[0064] Step S103: Based on the depth information of the point cloud image, generate a second pseudo image, the second pseudo image having the same size as the first pseudo image;

[0065] Step S104: Fuse the first pseudo-image and the second pseudo-image to obtain a fused image;

[0066] Step S105: Perform lane line detection on the above fused image to obtain a lane line image, wherein the lane line image includes at least one of lane line information and road edge information.

[0067] In the embodiments of this application, a first pseudo image and a second pseudo image are generated based on the laser intensity dimension and depth information dimension of the point cloud, respectively. This skips the complex three-dimensional convolution and adopts two-dimensional convolution, which reduces the computational complexity and the amount of computation, improves the real-time performance of lane line detection, and since this application uses point cloud images for processing, it can take advantage of the advantages of point clouds to avoid the errors caused by projecting two-dimensional lane lines into the three-dimensional world, and also avoid the errors caused by external light, road bumps and other conditions to lane line detection, thus improving the accuracy of lane line detection.

[0068] In one specific embodiment, a first pseudo-image is generated based on the laser reflection intensity of the point cloud, using the Pillar algorithm (a 3D target detection algorithm for cylindrical point clouds); a second pseudo-image is generated based on the depth information of the point cloud image, using the Pseudo Img algorithm (a pseudo point cloud image).

[0069] In a specific embodiment, after fusing the first pseudo-image and the second pseudo-image, this application also performs feature extraction and grid segmentation. The feature extraction adopts a two-dimensional convolutional network because the first pseudo-image and the second pseudo-image in this application are both two-dimensional images from a top-down perspective. Therefore, a two-dimensional convolutional network is used for feature extraction, which can greatly reduce the computational complexity.

[0070] In one embodiment, see Figure 3 and Figure 4 , Figure 3 This diagram illustrates a flowchart of the first pseudo-image generation method according to an embodiment of this application. Figure 4 This application illustrates a top-view feature map of a two-dimensional point cloud encoded image, whereby the generation of a first pseudo-image based on the laser reflection intensity of the point cloud image includes:

[0071] Step S301: Encode the above point cloud image to obtain a 3D point cloud encoded image;

[0072] Step S302: Perform dimensionality reduction processing on the above three-dimensional point cloud coded image to obtain a two-dimensional point cloud coded image;

[0073] Step S303: Extract features from the above two-dimensional point cloud encoded image to obtain the above first pseudo image.

[0074] In the embodiments of this application, the generation of the first pseudo image adopts the Pillar algorithm (volume point cloud 3D target detection algorithm). Compared with the traditional Voxel algorithm (volume pixel point cloud target detection algorithm), this method can skip the complex 3D convolution and adopt a 2D convolution method to reduce the amount of computation, improve the efficiency of computation, and ensure the real-time performance of detection.

[0075] In one specific embodiment, the point cloud image is first segmented into multiple cylindrical units. Each cylindrical unit is a small three-dimensional cell obtained by dividing the point cloud in a two-dimensional plane (XY plane) with a certain step size, resulting in a three-dimensional point cloud encoded image. Next, the three-dimensional point cloud encoded image is further processed by Pillar Feature Net for dimensionality reduction and feature extraction to generate a two-dimensional (XY plane) image, which is the image from the top-down view, so as to facilitate lane line recognition by learning lane line features from the top-down view.

[0076] In another specific embodiment, the process of calculating the image size using the Pillar algorithm in this application specifically includes the following:

[0077] The original point cloud detection range format is [x_min, y_min, z_min, x_max, y_max, z_max], and the true values ​​can be [0, -12.8, -3, 51.2, 12.8, 3]. The columnar unit obtained by the Pillar algorithm has the format [p_x, p_y, p_z], and the true values ​​can be [0.32, 0.32, 6]. The size of the 2D point cloud encoded image is calculated using the following formula.

[0078]

[0079] The actual value can be obtained by calculation as [160, 80]. The two-dimensional point cloud encoded image is then subjected to feature extraction through Pill Feature Net to obtain the first pseudo image. The size of the image remains unchanged after feature extraction, still [160, 80].

[0080] In one specific embodiment, please refer to Figure 4 , Figure 4 This illustration shows a top-view feature map of a two-dimensional point cloud encoded image according to an embodiment of this application. Figure 4 In the text, "Pillar column" refers to the column obtained after processing the original point cloud image using the Pillar algorithm, and "BEV top view feature" refers to the two-dimensional coded image obtained after dimensionality reduction processing of the multidimensional image. The two-dimensional coded image is an image from a top view perspective, where "BEV" indicates the top view.

[0081] In one embodiment, see Figure 5 , Figure 5 This diagram illustrates a second pseudo-image generation method according to an embodiment of this application. The method for generating the second pseudo-image based on the depth information of the point cloud image includes:

[0082] Step S501: Perform point cloud detection processing on the above point cloud image to obtain a pseudo point cloud image;

[0083] Step S502: Obtain the image of the pseudo point cloud from a top-down view and downsample it to obtain the second pseudo image mentioned above.

[0084] In the embodiments of this application, the generation of the second pseudo-image uses Pseudo Img (pseudo point cloud image) detection, which combines the original point cloud image with depth information to generate a pseudo point cloud image. The advantage of pseudo point cloud images is that they can bridge the gap between image-based point cloud perception and lidar-based point cloud perception. Pseudo point cloud processing is performed directly on the point cloud, avoiding the errors caused by projecting two-dimensional images onto the three-dimensional world. Pseudo point cloud processing makes the detected point cloud the location in the real world, improving the accuracy of detection.

[0085] In one specific embodiment, the pseudo point cloud image can be obtained by combining the original point cloud image acquired by the lidar with the corresponding depth information. The image of the pseudo point cloud image from the top view is taken, which is the same as the first pseudo image. Both are two-dimensional images and are images from the top view. Finally, the image from the top view is downsampled. The purpose of downsampling is to make the size of the second pseudo image and the first pseudo image the same so as to fuse the two.

[0086] In another specific embodiment, this application obtains a second pseudo-image through Pseudo Img Preprocess and ResNet (residual network), specifically including the following:

[0087] The original point cloud detection range format is [x_min, y_min, z_min, x_max, y_max, z_max], and the true values ​​can be [0, 51.2, -3, -12.8, 12.8, 3]. The pseudo image grid size format is [grid_x, grid_y], and the true values ​​can be [0.04, 0.04]. The image size of the pseudo point cloud from the top-down view is calculated using the following formula:

[0088]

[0089] At this point, the size of the pseudo point cloud image from the top-down view becomes [1280, 640]. Based on ResNet, the pseudo point cloud image from the top-down view is downsampled by 8 times to obtain the second pseudo image. At this time, the size of the second pseudo image becomes [160, 80], which is the same as the size of the first pseudo image, so that the two can be fused.

[0090] It is understood that the 8x downsampling used in this application is designed based on the size of the pseudo point cloud image and the size of the first pseudo image from the top view of this application. The purpose is to make the second pseudo image and the first pseudo image the same size so as to fuse them. This application is not limited to using the downsampling factor in other application instances.

[0091] In one embodiment, see Figure 6 , Figure 6 This illustration shows a flowchart of obtaining a lane line image based on a fused image according to an embodiment of this application. The process of detecting lane lines in the fused image to obtain the lane line image includes:

[0092] Step S601: Extract features from the above fused image to obtain a feature image. The feature image includes road feature information, which includes at least one of lane line feature information and curb feature information.

[0093] Step S602: Perform grid segmentation-based classification processing on the above feature images to obtain lane line images.

[0094] In the embodiments of this application, lane line detection can detect at least one of lane lines and road edges. The fused image is the result of fusing the first pseudo image and the second pseudo image in the channel dimension. The lane line feature information and the road edge feature information in the road feature information are both feature information from a top-down view. This is because the first pseudo image and the second pseudo image are both images from a top-down view. Similarly, the fused image is also an image from a top-down view. Therefore, the feature extraction based on the neural network is the feature extraction of the two-dimensional image, and a simple two-dimensional convolution is used.

[0095] In one specific embodiment, lane line detection can employ a backbone network, where feature extraction uses a VGG (Visual Geometry Group) neural convolutional network structure. VGG is a two-dimensional convolutional network that uses 3*3 convolutional kernels and 2*2 pooling kernels, effectively reducing computational complexity.

[0096] In another specific embodiment, the network used for classification prediction in the backbone network adopts the FPN network structure (Feature Pyramid Networks). This network structure utilizes both the high resolution of low-level features and the high semantic information of high-level features. It achieves the prediction effect by fusing the features of these different layers, and the prediction is performed separately on each fused feature layer. The FPN network structure performs well in small object detection.

[0097] In one embodiment, see Figure 7 , Figure 7 The diagram illustrates a grid-segmentation-based classification process according to an embodiment of this application. The process involves performing grid-segmentation-based classification on the feature image to obtain a lane line image, including:

[0098] Step S701: Perform grid-based segmentation on the above feature image to obtain multiple anchor boxes;

[0099] In one specific embodiment, the feature image can be an image after feature extraction by the VGG neural convolutional network. Because downsampling is performed during feature extraction, the number of anchor boxes is reduced, thereby reducing the computational load.

[0100] Step S702: Classify and predict each of the above anchor frames to obtain lane line images.

[0101] In one specific embodiment, classification prediction can be performed using an FPN network structure.

[0102] In the embodiments of this application, feature image segmentation and classification prediction can be performed in the aforementioned backbone network. The fused image is input into the backbone, and the backbone performs feature extraction and classification prediction on the fused image to output the predicted lane line image.

[0103] In one specific embodiment, regarding the setting of anchor boxes, for a 224*224 image, if three anchor boxes of different sizes are set at each position, then there will be 224*224*3=150528 anchor boxes. This results in a large number of anchor boxes, leading to a large amount of subsequent computation. Therefore, this application performs downsampling processing on the image during pseudo-image generation and feature extraction, finally obtaining a downsampled feature image. Anchor boxes are then generated in the feature image. For example, VGG16 downsamples the 224*224 input image 5 times to obtain a 7*7 feature image, so only 7*7*3=147 anchor boxes are needed, reducing the number of anchor boxes while covering most cases.

[0104] In another specific embodiment, regarding the classification prediction of anchor boxes, the predicted bounding boxes can be obtained based on the machine prediction offset of the anchor boxes. NMS (non-maximum suppression in) can be used to overcome the problem of similar predicted bounding boxes for the same target. For a predicted bounding box, the model outputs its prediction result to determine its category.

[0105] The NMS process includes: arranging the predicted probabilities of all predicted bounding boxes in the same image from highest to lowest; selecting the predicted bounding box with the highest probability as a benchmark; calculating the intersection-union ratio (IU) of the remaining predicted bounding boxes with the highest-probability predicted bounding box; if the IU is greater than a preset threshold, removing the predicted bounding box, thus eliminating bounding boxes similar to the highest-probability predicted bounding box. Then, selecting the predicted bounding box with the highest probability from the remaining predicted bounding boxes and repeating the above process to reduce similar predicted bounding boxes.

[0106] In one embodiment, see Figure 8 , Figure 8 This diagram illustrates a classification prediction method according to an embodiment of the present application. The method involves classifying and predicting each anchor frame to obtain a lane line image, including:

[0107] Step S801: Perform classification prediction on each of the above anchor frames to obtain the classification result and probability estimation result corresponding to each anchor frame. The classification result represents the category information of the corresponding anchor frame, and the probability estimation result represents the probability of the category to which each of the above anchor frames belongs.

[0108] Step S802: Screen each of the above-mentioned anchor boxes based on the above classification results and the above probability estimation results, and obtain the above lane line image according to the screening results.

[0109] In an embodiment of the present application, the head (detection head) is used to obtain the output content of the backbone (main network), and the output content includes classification results and probability estimation results. The head part makes predictions on the lane line image based on the classification results and the probability estimation results.

[0110] In a specific embodiment, the predicted bounding box is obtained according to the anchor box machine predicted offset. The head model calculates the probability that each predicted bounding box belongs to each category, and the category corresponding to the largest probability value is the category of the predicted bounding box. The "probability that each predicted bounding box belongs to each category" is the probability estimation result, and finally determining the category of each predicted bounding box is the classification result.

[0111] In another specific embodiment, the binary cross-entropy loss function is used for probability estimation in the prediction model of the present application. The output result depicts the difference between the actual probability and the expected probability. That is to say, the smaller the cross-entropy value, the closer the two probability distributions are. In the present application, for example, the categories of the predicted bounding box include the lane line category, the road edge category, and the non-linear area category. When making a probability estimation for a certain predicted bounding box, the cross-entropy with the lane line category is A, the cross-entropy with the road edge category is B, and the cross-entropy with the non-linear area category is C. Among them, A < B < C. It can be seen that the content in this predicted bounding box is the closest to the lane line, so the result output by the probability estimation is that the probability that the content in this predicted bounding box belongs to the lane line category is the largest.

[0112] In one embodiment, please refer to Figure 9 , Figure 9 shows a schematic flowchart of the image fusion method according to an embodiment of the present application. The above-mentioned fusion of the above first pseudo-image and the above second pseudo-image to obtain a fused image includes:

[0113] Step S901: Align and match the above first pseudo-image and the above second pseudo-image to obtain a matched image;

[0114] Step S902: Stitch the above matched images to obtain the above fused image.

[0115] In an embodiment of the present application, after unifying the sizes of the first pseudo-image and the second pseudo-image, the first pseudo-image and the second pseudo-image are fused in the channel dimension through a concatenate structure (connection structure). Among them, unifying the sizes of the first pseudo-image and the second pseudo-image is a prerequisite for the concatenate structure.

[0116] In a specific embodiment, the image fusion steps include aligning and matching the first pseudo-image and the second pseudo-image, including edge alignment, then stitching the two images together, using the concatenate structure in Neural Network to complete the fusion of the first pseudo-image and the second pseudo-image, connecting two tensors along the channel dimension, fusing the features in the first pseudo-image and the second pseudo-image, fusing the features extracted by multiple convolutional feature extraction frameworks, or fusing the information from the output layer.

[0117] In another specific embodiment, the concatenate structure includes downsampling and upsampling processes. During downsampling, the number of feature maps is doubled, while during upsampling, the number of feature maps is halved. The upsampled feature maps and the corresponding downsampled clipped feature maps are then fused. Using the concatenate structure can alleviate the gradient vanishing problem to some extent.

[0118] According to a second aspect disclosed in this application, a lane line detection device is provided. Please refer to [link to relevant documentation]. Figure 10 , Figure 10 This diagram illustrates a block diagram of a lane line detection device according to an embodiment of this application. The device includes:

[0119] The point cloud image acquisition module is used to acquire point cloud images, which are images representing road surface information around the vehicle.

[0120] The first pseudo-image generation module is used to generate a first pseudo-image based on the laser reflection intensity of the point cloud image mentioned above.

[0121] In the embodiments of this application, the first pseudo-image generation module includes a point cloud encoding unit, a dimensionality reduction unit, and a feature extraction unit.

[0122] The point cloud encoding unit is used to encode the above point cloud image to obtain a three-dimensional point cloud encoded image;

[0123] The dimension reduction unit is used to reduce the dimension of the above three-dimensional point cloud encoded image to obtain a two-dimensional point cloud encoded image;

[0124] The feature extraction unit is used to extract features from the above two-dimensional point cloud coded image to obtain the above first pseudo image.

[0125] The second pseudo-image generation module is used to generate a second pseudo-image based on the depth information of the point cloud image. The second pseudo-image has the same size as the first pseudo-image.

[0126] In the embodiments of this application, the second pseudo-image generation module includes a point cloud detection unit and a downsampling unit.

[0127] The point cloud detection unit is used to perform point cloud detection processing on the above point cloud image to obtain a pseudo point cloud image;

[0128] The downsampling unit is used to acquire the image of the pseudo point cloud image from a top-down view and downsample it to obtain the second pseudo image mentioned above.

[0129] The fusion module is used to fuse the first pseudo-image and the second pseudo-image to obtain a fused image;

[0130] In embodiments of this application, the fusion module includes an alignment unit and a splicing unit.

[0131] The alignment unit is used to align and match the first pseudo-image and the second pseudo-image to obtain a matched image.

[0132] The stitching unit is used to stitch the matched images together to obtain the fused image.

[0133] The lane line detection module is used to perform lane line detection on the above-mentioned fused image to obtain a lane line image, wherein the lane line image includes at least one of lane line information and curb information.

[0134] In the embodiments of this application, the lane line detection module includes a feature extraction unit and a classification processing unit.

[0135] The feature extraction unit is used to extract features from the above-mentioned fused image to obtain a feature image. The feature image includes road feature information, which includes at least one of lane line feature information and curb feature information.

[0136] The classification processing unit performs grid-based classification processing on the aforementioned feature images to obtain lane line images. The classification processing unit also includes a segmentation unit and a classification prediction unit.

[0137] The segmentation unit is used to perform grid-based segmentation on the above feature image to obtain multiple anchor boxes;

[0138] The classification prediction unit is used to classify and predict each of the above anchor boxes to obtain a lane line image. The classification prediction unit also includes a prediction unit and a filtering unit;

[0139] The prediction unit is used to perform classification prediction for each of the above anchor frames, and obtain the classification result and probability estimation result corresponding to each of the above anchor frames. The classification result represents the category information of the corresponding anchor frame, and the probability estimation result represents the probability of the category to which each of the above anchor frames belongs.

[0140] The filtering unit is used to filter each of the above anchor frames based on the above classification results and the above probability estimation results, and obtain the above lane line image according to the filtering results.

[0141] Please refer to Figure 11 It illustrates the structural block diagram of an electronic device provided in one embodiment of this application. Figure 1 The electronic device can be a terminal. This electronic device is used to implement the lane line detection method provided in the above embodiments. Specifically:

[0142] The electronic device 1100 includes a processor 1101 and a memory 1102.

[0143] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0144] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 is used to store at least one instruction, at least one program, code set, or instruction set, configured to be executed by one or more processors to implement the lane detection method described above.

[0145] In some embodiments, the electronic device 1100 may also optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1104, a touch display screen 1105, a camera assembly 1106, an audio circuit 1107, a positioning assembly 1108, and a power supply 1109.

[0146] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on the electronic device 1100, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0147] Please refer to Figure 12 It illustrates the structural block of an electronic device provided in another embodiment of this application. Figure 2 The electronic device can be a server for executing the lane detection method described above. Specifically:

[0148] Electronic device 1200 includes a central processing unit (CPU) 1201, a system memory 1204 including random access memory (RAM) 1202 and read-only memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the CPU 1201. Electronic device 1200 also includes a basic input / output system (I / O system) 1206 that facilitates the transfer of information between various devices within the computer, and a mass storage device 1207 for storing the operating system 1213, application programs 1214, and other program modules 1211.

[0149] The basic input / output system 1206 includes a display 1208 for displaying information and an input device 1209 for user input, such as a mouse or keyboard. Both the display 1208 and the input device 1209 are connected to the central processing unit 1201 via an input / output controller 1210 connected to the system bus 1205. The basic input / output system 1206 may also include the input / output controller 1210 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1210 also provides output to a display screen, printer, or other types of output devices.

[0150] Mass storage device 1207 is connected to central processing unit 1201 via a mass storage controller (not shown) connected to system bus 1205. Mass storage device 1207 and its associated computer-readable media provide non-volatile storage for electronic device 1200. That is, mass storage device 1207 may include computer-readable media (not shown) such as hard disk or CD-ROM (Compact Disc Read-Only Memory) drive.

[0151] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1204 and mass storage device 1207 described above can be collectively referred to as memory.

[0152] According to various embodiments of this application, the computer device 1200 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1200 can be connected to the network 1212 via the network interface unit 1211 connected to the system bus 1205, or the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).

[0153] The aforementioned memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the lane detection method described above.

[0154] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is executed by a processor to implement the lane line detection method.

[0155] Optionally, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random Access Memory), SSD (Solid State Drive), or optical disc, etc. The random access memory may include ReRAM (Resistant Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0156] In an exemplary embodiment, a computer-readable storage medium including program code is also provided, such as a memory including program code, which can be executed by a processor to perform the video display method described above. Optionally, the computer-readable storage medium may be read-only memory (ROM), random access memory (RAM), compact-disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0157] In an exemplary embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the lane line detection method described above.

[0158] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0159] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A lane line detection method characterized by, The method includes: Acquire a point cloud image, wherein the point cloud image is an image representing road surface information around the vehicle; The point cloud image is encoded to obtain a three-dimensional point cloud encoded image; the three-dimensional point cloud encoded image is then dimensionality-reduced to obtain a two-dimensional point cloud encoded image; features are extracted from the two-dimensional point cloud encoded image to obtain a first pseudo image. The point cloud image is subjected to point cloud detection processing to obtain a pseudo point cloud image; the image of the pseudo point cloud image from a top-down view is obtained and downsampled to obtain a second pseudo image, the second pseudo image having the same size as the first pseudo image; The first pseudo-image and the second pseudo-image are fused to obtain a fused image; Feature extraction is performed on the fused image to obtain a feature image including road feature information, wherein the road feature information includes at least one of lane line feature information and curb feature information; The feature image is subjected to grid segmentation-based classification processing to obtain a lane line image, which includes at least one of lane line information and curb information.

2. The method of claim 1, wherein, The step of performing grid-based segmentation classification on the feature image to obtain a lane line image includes: The feature image is segmented based on a grid to obtain multiple anchor boxes; Each anchor frame is classified and predicted to obtain a lane line image.

3. The method of claim 2, wherein, The process of classifying and predicting each anchor frame to obtain a lane line image includes: For each anchor frame, a classification prediction is performed to obtain the classification result and probability estimation result corresponding to each anchor frame. The classification result represents the category information of the corresponding anchor frame, and the probability estimation result represents the probability of each anchor frame belonging to the category. Based on the classification results and the probability estimation results, each anchor frame is filtered, and the lane line image is obtained according to the filtering results.

4. The method according to any one of claims 1 to 3, characterized in that, The step of fusing the first pseudo-image and the second pseudo-image to obtain a fused image includes: The first pseudo-image and the second pseudo-image are aligned and matched to obtain the matched image. The matched images are stitched together to obtain the fused image.

5. A lane line detection device, characterized in that, The device includes: The point cloud image acquisition module is used to acquire point cloud images, which are images representing road surface information around the vehicle; The first pseudo-image generation module is used to encode the point cloud image to obtain a three-dimensional point cloud encoded image; to perform dimensionality reduction processing on the three-dimensional point cloud encoded image to obtain a two-dimensional point cloud encoded image; and to extract features from the two-dimensional point cloud encoded image to obtain a first pseudo-image. The second pseudo-image generation module is used to perform point cloud detection processing on the point cloud image to obtain a pseudo-point cloud image; to obtain an image of the pseudo-point cloud image from a top-down view and to downsample it to obtain a second pseudo-image, wherein the second pseudo-image has the same size as the first pseudo-image. The fusion module is used to fuse the first pseudo image and the second pseudo image to obtain a fused image; The lane line detection module is used to extract features from the fused image to obtain a feature image including road feature information, wherein the road feature information includes at least one of lane line feature information and curb feature information; and to perform grid segmentation-based classification processing on the feature image to obtain a lane line image, wherein the lane line image includes at least one of lane line information and curb information.

6. A lane line detection device, the device comprising a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement a lane line detection method as described in any one of claims 1 to 4.

7. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, When the computer program instructions are executed by the processor, they implement a lane line detection method according to any one of claims 1 to 4.

Images