Method and system for extracting dashed line blocks from lane printing using laser point cloud based on deep learning

Abstract

The present invention relates to a method and system for extracting lane printed dashed line blocks from a laser point cloud based on deep learning. The method comprises: obtaining a laser point cloud of the lane printed dashed line blocks to be extracted and projecting it into one or more images; extracting one or more dashed line blocks from each projected image using a trained target detection model, and identifying the positions of one or more virtual-real changes; inverting the coordinate information of each extracted dashed line block in the laser point cloud based on the positions of the one or more virtual-real changes; and performing quality inspection on each extracted dashed line block based on the positions of the multiple virtual-real changes, the trajectory information of the laser point cloud, and the geometric properties of each dashed line block. The present invention utilizes a multi-dimensional cross-extraction method to obtain dashed line blocks, thereby improving the coverage of the dashed line blocks and reducing their error rate and missed detection rate. It also provides a new direction for automated extraction methods of other road elements.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle assisted driving technology, and specifically relates to a method and system for extracting lane printed dotted line blocks from a laser point cloud based on deep learning. Background Art

[0002] In the field of autonomous driving, vehicle positioning is based on information from road features. Lane markings are the most crucial road feature, containing both location and regulatory information. They are typically produced in a standardized and complete manner and are the most crucial element in autonomous driving positioning technology. Combining GNSS signals with lane marking information enables high-precision positioning within the lane.

[0003] Lane markings can be categorized as solid or dashed lines based on their continuity. Solid lines, with their less pronounced vertical features, are primarily used to ensure lateral positioning accuracy. Dashed lines, on the other hand, ensure both lateral and vertical positioning accuracy.

[0004] Existing methods for extracting dotted blocks from laser point clouds include traditional methods based on intensity segmentation and laser point cloud semantic segmentation methods based on deep learning. The traditional method based on intensity segmentation is the main method because the direct deep learning method of laser point clouds has extremely high data set annotation costs, and the currently open-source semantic segmentation networks are not yet effective enough for engineering applications for small targets such as dotted blocks. The traditional method based on intensity segmentation relies on the calculation of intensity thresholds, and the most commonly used method is the Otsu algorithm. The main problem is that the data intensity distribution in different scenes is significantly different. For example, in areas with metal guardrails or roads blocked by cars, the proportion of high-intensity point clouds is relatively high, which interferes with the calculation of intensity thresholds and easily leads to mis-extraction and missed extraction. Summary of the Invention

[0005] In order to solve the problems of high cost, low accuracy, and easy misextraction and omission in the existing technology of extracting dotted line blocks from laser point clouds, the first aspect of the present invention provides a deep learning-based laser point cloud lane printing dotted line block extraction method, including: obtaining a laser point cloud of the lane printing dotted line block to be extracted, and projecting it into one or more images; using a trained target detection model to extract one or more dotted line blocks in each projected image, and identifying the positions of one or more virtual-real changes; inverting the coordinate information of each extracted dotted line block in the laser point cloud based on the positions of the one or more virtual-real changes; and performing quality inspection on each extracted dotted line block based on the positions of multiple virtual-real changes, the trajectory information of the laser point cloud, and the geometric properties of each dotted line block.

[0006] In some embodiments of the present invention, the use of a trained target detection model to extract one or more dotted blocks in each projected image and identify the locations of one or more virtual-real changes includes: the target detection network includes a semantic segmentation network and a target detection network, the semantic segmentation network is used to extract one or more dotted blocks in each projected image; the target detection network is used to detect the locations of virtual-real changes.

[0007] Furthermore, the semantic segmentation network and the object detection network are trained using the same dataset.

[0008] In some embodiments of the present invention, the inverting the coordinate information of each dotted line block extracted in the laser point cloud according to the one or more virtual-real change positions includes: inverting the image coordinates of each extracted dotted line block to the local coordinates of the corresponding block laser point cloud; inverting the image coordinates of each virtual-real change position to the local coordinates of the corresponding block laser point cloud; and calculating the global point cloud coordinate information before block cutting based on the serial block cutting information of each block laser point cloud.

[0009] In some embodiments of the present invention, the quality inspection of each extracted dotted line block based on the positions of multiple virtual-real changes, the trajectory information of the laser point cloud and the geometric properties of each dotted line block includes: determining all dotted line intervals of the laser point cloud based on the positions of multiple virtual-real changes, and judging the confidence of each dotted line block based on them; calculating the coverage of the dotted line blocks in each dotted line interval, and judging the confidence of each dotted line block based on it.

[0010] In the above embodiment, obtaining the laser point cloud of the lane printed dotted line block to be extracted and projecting it into one or more images includes: longitudinally cutting the laser point cloud of the lane printed dotted line block to be extracted into blocks along the trajectory direction, and ortho-projecting it into one or more images.

[0011] The second aspect of the present invention provides a deep learning-based laser point cloud lane printing dotted line block extraction system, including: a projection module, used to obtain the laser point cloud of the lane printing dotted line block to be extracted, and project it into one or more images; an extraction module, used to use the trained target detection model to extract one or more dotted line blocks in each projected image, and identify the positions of one or more virtual and real changes; an inversion module, used to invert the coordinate information of each extracted dotted line block in the laser point cloud based on the positions of the one or more virtual and real changes; a quality inspection module, used to perform quality inspection on each extracted dotted line block based on the positions of multiple virtual and real changes, the trajectory information of the laser point cloud and the geometric properties of each dotted line block.

[0012] Furthermore, the inversion module includes a first inversion unit, a second inversion unit and a calculation unit, wherein the first inversion unit is used to invert the image coordinates of each extracted dotted line block to the local coordinates of the corresponding block laser point cloud; the second inversion unit is used to invert the image coordinates of each virtual-real change position to the local coordinates of the corresponding block laser point cloud; the calculation unit is used to calculate the global point cloud coordinate information before block cutting based on the serial block cutting information of each block laser point cloud.

[0013] The third aspect of the present invention provides an electronic device comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the deep learning-based laser point cloud lane printed dotted line block extraction method provided in the first aspect of the present invention.

[0014] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for extracting dotted line blocks from lane printing using laser point cloud based on deep learning provided in the first aspect of the present invention is implemented.

[0015] The beneficial effects of the present invention are:

[0016] 1. Compared with the existing method of solving road elements by simply relying on the reflection intensity of point clouds, this invention obtains dotted line blocks through a multi-dimensional cross-extraction method, thereby improving the coverage rate of dotted line blocks;

[0017] 2. Through cross-validation of inverted coordinates and deep learning methods, the error rate and missed detection rate of dotted line blocks were reduced;

[0018] 3. Since other road features in high-precision maps also have similar properties to dotted blocks, the extraction method can be extended to the automated extraction of other road features, providing new ideas for their extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the basic process of a method for extracting dotted line blocks printed on lanes from laser point clouds based on deep learning in some embodiments of the present invention;

[0020] Figure 2 Schematic diagram of a laser point cloud projection image and its extracted dotted blocks in some embodiments of the present invention;

[0021] Figure 3 Schematic diagram of a laser point cloud projection image and its extracted virtual-real change positions in some embodiments of the present invention;

[0022] Figure 4Schematic diagram of the effect of the extracted dotted block in some embodiments of the present invention;

[0023] Figure 5 Schematic diagram of the structure of a system for extracting dotted line blocks from lane printing using laser point cloud based on deep learning in some embodiments of the present invention;

[0024] Figure 6 Schematic diagram of the structure of an electronic device in some embodiments of the present invention. DETAILED DESCRIPTION

[0025] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0026] refer to Figure 1 、 Figure 2 and Figure 4 In a first aspect of the present invention, a method for extracting lane printing dashed line blocks from a laser point cloud based on deep learning is provided, comprising: S100. Acquiring a laser point cloud of the lane printing dashed line blocks to be extracted and projecting it into one or more images; S200. Extracting one or more dashed line blocks from each projected image using a trained target detection model, and identifying the positions of one or more virtual-real changes; S300. Inverting the coordinate information of each extracted dashed line block in the laser point cloud based on the positions of the one or more virtual-real changes; S400. Performing quality inspection on each extracted dashed line block based on the positions of multiple virtual-real changes, the trajectory information of the laser point cloud, and the geometric properties of each dashed line block.

[0027] refer to Figure 2 and Figure 3 In step S200 of some embodiments of the present invention, the use of the trained target detection model to extract one or more dotted blocks in each projection image and identify the locations of one or more virtual-real changes includes: the target detection network includes a semantic segmentation network and a target detection network, the semantic segmentation network is used to extract one or more dotted blocks in each projection image; the target detection network is used to detect the locations of virtual-real changes. Figure 2 The figure shows the effect of extracting dotted blocks from the projection image of the laser point cloud. The left half represents the projection image and the right half represents the extracted dotted blocks.

[0028] Specifically, the training steps include: S201. Annotating the semantics of the dotted blocks on the projected image dataset, and using the training dataset to train the dotted block semantic segmentation network; S202. Annotating the virtual-real change position target boxes on the projected image dataset, and using the training dataset to train the virtual-real change position target detection network; S203. On the verification dataset, using the dotted block semantic segmentation network to segment the dotted blocks, and using the virtual-real change target detection network to detect the virtual-real change position.

[0029] It can be understood that the above-mentioned semantic segmentation network or target detection network includes RCNN series, Mask-RCNN, R-FCN, YOLO series, SSD, FPN, etc., and the training process can be implemented through deep learning frameworks such as Pytorch, Tesorflow, and Paddle.

[0030] refer to Figure 3 Schematically, the left half of the figure represents the projection image of the laser point cloud or the ordinary lane image, and the part marked with a box in the right half of the figure represents the virtual-real change position, that is: the virtual-real change position in the present invention refers to the position where the dotted block in the projection image changes to the solid block, or the solid block changes to the dotted block.

[0031] In step S300 of some embodiments of the present invention, the inversion of the coordinate information of each dotted line block in the laser point cloud extracted according to the one or more virtual-real change positions includes: S301. inverting the image coordinates of each extracted dotted line block to the local coordinates of the corresponding block laser point cloud; S302. inverting the image coordinates of each virtual-real change position to the local coordinates of the corresponding block laser point cloud; S303. calculating the global point cloud coordinate information before block cutting based on the serial block cutting information of each block laser point cloud.

[0032] In step S400 of some embodiments of the present invention, the quality inspection of each extracted dotted line block based on the positions of multiple virtual-real changes, the trajectory information of the laser point cloud and the geometric properties of each dotted line block includes: S401. Determine all dotted line intervals of the laser point cloud based on the positions of multiple virtual-real changes, and judge the confidence of each dotted line block based on them; S402. Calculate the coverage of the dotted line blocks in each dotted line interval, and judge the confidence of each dotted line block based on it.

[0033] Specifically, in step S401, it includes:

[0034] S4011: For the point P1 that changes from real to virtual in the virtual-real change position, form a buffer zone with a width of 0.5m along the extension direction of the trajectory, and search for the nearest virtual-real change position P2 that changes from virtual to real;

[0035] S4012: A dotted interval with a width of 0.5 m is formed along the direction of the trajectory extending from P1 and P2;

[0036] S4013: traverse all virtual-real change point sets and execute S4011 to S4012 to collect all dotted line intervals in the current data;

[0037] S4014: Traverse the dashed line block data and calculate whether each dashed line block exists in a certain dashed line interval range of S4013. If it exists, the dashed line block data is considered correct and marked with a confidence level of 99. If it does not exist in a certain interval, continue along the direction of the trajectory extension and search for the nearest dashed line block within a range of 20mf on both sides of the forward and backward directions.

[0038] S4015: If there are no other dotted blocks within the range of 20m before and after, the dotted block is considered to be a false detection and its confidence is marked as 0; if there are other dotted blocks, the cumulative span length of all dotted blocks in the range is calculated. If the length is less than 10m, these dotted blocks are considered to be false detections and the confidence is marked as 0; if the length is greater than 10m, the dotted block is considered not to be a false detection and the confidence is marked as 99.

[0039] In step S402, for the dotted line intervals that have been missed, additional dotted line blocks are generated and the confidence levels are marked. The specific implementation is as follows:

[0040] S4021. Traverse the dotted interval in S43 and calculate the coverage of the dotted blocks in the dotted interval;

[0041] S4022. For the dotted line intervals with coverage lower than 80%, interpolate forward and backward based on the dotted line segment data contained therein to generate a new dotted line block with a marking confidence level of 50.

[0042] In step S100 of the above embodiment, obtaining the laser point cloud of the lane printed dotted line block to be extracted and projecting it into one or more images includes: longitudinally cutting the laser point cloud of the lane printed dotted line block to be extracted into blocks along the trajectory direction, and ortho-projecting it into one or more images.

[0043] Specifically, it includes: S101: based on the trajectory data, calculating a polygon sequence with no overlap and no gaps along the trajectory direction; S102: using the polygon sequence to cut the laser point cloud into blocks; S103: using CSF (Cloth Simulation Filter, point cloud ground point filter) to filter the point cloud blocks, eliminating the point cloud above the road surface and retaining the road surface point cloud; S104: ortho-projecting the road surface point cloud block by block into an image.

[0044] Schematically, Figure 4The figure shows the dotted line blocks extracted by the laser point cloud lane printing dotted line block extraction method provided by the present invention. The white part is the lane line, the black part inside the lane line is the dotted line block, and the outside is the track line.

[0045] Example 2

[0046] refer to Figure 5 In a second aspect, the present invention provides a deep learning-based laser point cloud lane printing dotted line block extraction system 1, comprising: a projection module 11, for acquiring a laser point cloud of a lane printing dotted line block to be extracted, and projecting it into one or more images; an extraction module 12, for extracting one or more dotted line blocks in each projected image using a trained target detection model, and identifying the positions of one or more virtual-real changes; an inversion module 13, for inverting the coordinate information of each extracted dotted line block in the laser point cloud according to the positions of the one or more virtual-real changes; a quality inspection module 14, for performing quality inspection on each extracted dotted line block according to the positions of multiple virtual-real changes, the trajectory information of the laser point cloud, and the geometric properties of each dotted line block.

[0047] Furthermore, the inversion module 13 includes a first inversion unit, a second inversion unit and a calculation unit, wherein the first inversion unit is used to invert the image coordinates of each extracted dotted line block to the local coordinates of the corresponding block laser point cloud; the second inversion unit is used to invert the image coordinates of each virtual-real change position to the local coordinates of the corresponding block laser point cloud; the calculation unit is used to calculate the global point cloud coordinate information before block cutting based on the serial block cutting information of each block laser point cloud.

[0048] Example 3

[0049] refer to Figure 6 According to the third aspect of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to the first aspect of the present invention.

[0050] The electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0051] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0052] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wire, optical cable, RF (radio frequency), etc., or any suitable combination thereof.

[0053] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more computer programs, which, when executed by the electronic device, cause the electronic device to:

[0054] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0055] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for extracting dotted line blocks from lane printing using laser point cloud based on deep learning, characterized in that: include: Obtaining a laser point cloud of the printed dashed line blocks of the lane to be extracted, and projecting it into one or more images; Utilizing the trained target detection model, one or more dotted blocks are extracted from each projection image, and one or more locations of virtual-real changes are identified; based on the locations of the one or more virtual-real changes, coordinate information of each extracted dotted block in the laser point cloud is inverted; quality inspection is performed on each extracted dotted block based on the locations of the multiple virtual-real changes, trajectory information of the laser point cloud, and geometric properties of each dotted block; inverting the coordinate information of each extracted dotted block in the laser point cloud based on the locations of the one or more virtual-real changes includes: inverting the image coordinates of each extracted dotted block into local coordinates of the corresponding block laser point cloud; The image coordinates of each virtual-real change position are inverted to the local coordinates of the corresponding block laser point cloud; based on the sequential block cutting information of each block laser point cloud, its global point cloud coordinate information before block cutting is calculated.

2. The method for extracting lane printed dashed line blocks from laser point cloud based on deep learning according to claim 1, characterized in that: The method of using the trained target detection model to extract one or more dotted blocks in each projected image and identify the locations of one or more virtual-real changes includes: the target detection network includes a semantic segmentation network and a target detection network, the semantic segmentation network is used to extract one or more dotted blocks in each projected image; the target detection network is used to detect the locations of virtual-real changes.

3. The method for extracting lane printed dashed line blocks from laser point cloud based on deep learning according to claim 2, characterized in that: The semantic segmentation network and the object detection network are trained using the same dataset.

4. The method for extracting lane printed dashed line blocks from laser point cloud based on deep learning according to claim 1, characterized in that: The quality inspection of each extracted dotted line block based on the positions of multiple virtual-real changes, the trajectory information of the laser point cloud and the geometric properties of each dotted line block includes: determining all dotted line intervals of the laser point cloud according to the positions of multiple virtual-real changes, and judging the confidence of each dotted line block based on them; calculating the coverage rate of the dotted line blocks in each dotted line interval, and judging the confidence of each dotted line block based on it.

5. The method for extracting lane printed dashed line blocks from laser point clouds based on deep learning according to any one of claims 1 to 4, characterized in that: The obtaining of the laser point cloud of the lane printed dotted line block to be extracted and projecting it into one or more images includes: longitudinally cutting the laser point cloud of the lane printed dotted line block to be extracted into blocks along the track direction and ortho-projecting them into one or more images.

6. A laser point cloud lane printing dotted line block extraction system based on deep learning, characterized by: include: A projection module is used to obtain a laser point cloud of the printed dashed line block of the lane to be extracted and project it into one or more images; An extraction module is used to extract one or more dotted blocks in each projection image using the trained object detection model, and to identify the locations of one or more virtual-real changes; an inversion module is used to invert the coordinate information of each extracted dotted block in the laser point cloud based on the locations of the one or more virtual-real changes; A quality inspection module, configured to perform quality inspection on each extracted dotted line block based on the positions of the multiple virtual and real changes, the trajectory information of the laser point cloud, and the geometric properties of each dotted line block; The inverting the coordinate information of each extracted dotted line block in the laser point cloud according to the one or more positions of the virtual and real changes includes: inverting the image coordinates of each extracted dotted line block to the local coordinates of the corresponding block laser point cloud; The image coordinates of each virtual-real change position are inverted to the local coordinates of the corresponding block laser point cloud; based on the sequential block cutting information of each block laser point cloud, its global point cloud coordinate information before block cutting is calculated.

7. The deep learning-based laser point cloud lane printing dashed line block extraction system according to claim 6 is characterized in that: The inversion module includes a first inversion unit, a second inversion unit and a calculation unit. The first inversion unit is used to invert the image coordinates of each extracted dotted line block to the local coordinates of the corresponding block laser point cloud; the second inversion unit is used to invert the image coordinates of each virtual-real change position to the local coordinates of the corresponding block laser point cloud; the calculation unit is used to calculate the global point cloud coordinate information of each block laser point cloud before block cutting based on the sequential block cutting information of the block laser point cloud.

8. An electronic device comprising: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the deep learning-based laser point cloud lane printing dotted line block extraction method as described in any one of claims 1 to 5.

9. A computer-readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method for extracting dotted line blocks from lane printing using laser point cloud based on deep learning is implemented.

Images