Method and device for generating laser radar point cloud

By using self-supervised learning of images and sparse LiDAR point clouds in a deep learning network, high-quality dense LiDAR point clouds are generated, solving the problem of high cost of expensive LiDAR sensors and realizing low-cost, high-precision dense point cloud generation.

CN119027574APending Publication Date: 2024-11-26ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310612006.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-26
Publication Date
2024-11-26

Smart Images

  • Figure CN119027574A_ABST
    Figure CN119027574A_ABST
Patent Text Reader

Abstract

The invention relates to a method for generating a laser radar point cloud. The method comprises the following steps: receiving an image from an image acquisition device; and inputting the image into a trained deep learning network so as to output a laser radar point cloud. The invention also relates to a device for generating the laser radar point cloud, a computer storage medium, a computer program product and a roadside infrastructure-based perception system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of generation of LiDAR point clouds, and more specifically, to a method and device for generating LiDAR point clouds, a computer storage medium, a computer program product, and a perception system based on roadside infrastructure. Background Art

[0002] Currently, many applications depend on sensing their surroundings and use depth cues to reason and react accordingly. Autonomous driving and similar machine systems have an increasing need for robust 3D vision. For example, roadside (or infrastructure-based) perception tasks require high-quality raw data. Cameras are the primary sensor, but lack 3D information.

[0003] One existing solution is to use expensive LiDAR sensors (64 or 128 channels) to obtain dense 3D point clouds. However, purchasing and maintaining such LiDAR sensors is very expensive. In addition, due to government regulations, it is time-consuming and challenging to install LiDARs at each target location. In this case, it is not reasonable to use LiDAR sensors for all infrastructure-based perception because it costs too much money and time. Therefore, an improved method for generating dense LiDAR point clouds is urgently needed. Summary of the invention

[0004] According to one aspect of the present invention, a method for generating a lidar point cloud is provided, the method comprising: receiving an image from an image acquisition device; and inputting the image into a trained deep learning network to output a lidar point cloud.

[0005] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned method, the output lidar point cloud is a dense lidar point cloud; during the training stage of the deep learning network, the image and the sparse lidar point cloud are input into the deep learning network for self-supervised learning, and wherein the sparse lidar point cloud is acquired using a lidar with a non-repetitive scanning modality.

[0006] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned method, the image and the sparse lidar point cloud both contain perception data for the same relatively constant scene; the relatively constant scene includes a perception scene based on roadside infrastructure, wherein the roadside infrastructure includes: the image acquisition device installed at a fixed position on the roadside and the lidar with non-repetitive scanning mode, so as to respectively collect images and sparse lidar point clouds for the same relatively constant scene.

[0007] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned method, in the training stage of the deep learning network, the image and the sparse lidar point cloud are used as input for self-supervised learning, including: using the sparse lidar point cloud as a reference for calculating the loss to train the deep learning network without annotation, so that the deep learning network can generate a dense lidar point cloud based on the image.

[0008] As a supplement or alternative to the above-mentioned solution, in the above-mentioned method, the area scanned by the laser radar with non-repetitive scanning mode within a certain period of time includes the entire field of view of the image acquisition device.

[0009] As a supplement or alternative to the above solution, in the above method, the image is a 2D image.

[0010] According to another aspect of the present invention, a device for generating a dense lidar point cloud is provided, the device comprising: a receiving device for receiving an image from an image acquisition device; and a generating device for inputting the image into a trained deep learning network to output a lidar point cloud.

[0011] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned device, the lidar point cloud output by the generating device is a dense lidar point cloud, and the generating device includes: the trained deep learning network; a training unit, which is used to use the image and the sparse lidar point cloud as input for self-supervised learning during the training stage of the deep learning network to obtain the trained deep learning network; and an acquisition unit, which is used to acquire the sparse lidar point cloud using a lidar with a non-repetitive scanning modality.

[0012] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned device, the image and the sparse lidar point cloud both contain perception data for the same relatively constant scene; the relatively constant scene includes a perception scene based on roadside infrastructure, wherein the roadside infrastructure includes: the image acquisition device installed at a fixed position on the roadside and the lidar with non-repetitive scanning mode, so as to respectively collect images and sparse lidar point clouds for the same relatively constant scene.

[0013] As a supplement or alternative to the above-mentioned scheme, in the above-mentioned device, the training unit is configured to: train the deep learning network using the sparse lidar point cloud as a reference for calculating loss without annotation, so that the deep learning network can generate a dense lidar point cloud based on the image.

[0014] As a supplement or alternative to the above-mentioned solution, in the above-mentioned device, the area scanned by the laser radar with non-repetitive scanning mode within a certain period of time includes the entire field of view of the image acquisition device.

[0015] As a supplement or alternative to the above solution, in the above device, the image is a 2D image.

[0016] According to yet another aspect of the present invention, a computer storage medium is provided, wherein the medium includes instructions, and the instructions execute the above method when executed.

[0017] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, and when the computer program is executed by a processor, the method as described above is implemented.

[0018] According to another aspect of the present invention, a perception system based on roadside infrastructure is provided, the system comprising: a camera; and a perception control device, wherein the device comprises a device for generating a dense lidar point cloud in a relatively constant scene as described above.

[0019] The scheme for generating a LiDAR point cloud in an embodiment of the present invention uses images and sparse LiDAR point clouds as inputs for self-supervised learning during the training phase of a deep learning network. Compared to a scheme for generating a dense point cloud (i.e., a pseudo LiDAR point cloud) based on camera images and supervised methods, the scheme in an embodiment of the present invention does not require a high-quality, accurate depth map as the ground truth in the training phase, and thus does not require time-consuming and laborious annotation information. In addition, the sparse LiDAR point cloud is acquired using a LiDAR with a non-repetitive scanning mode that can scan the entire field of view of the camera over time. Therefore, after the sparse point cloud acquired by the LiDAR with a non-repetitive scanning mode (e.g., a lower-cost LiDAR) is combined with the image obtained by the camera and applied to the learning phase, the overall training quality will be close to that of a fully supervised training scheme (when the training data set is sufficiently large and extensive). In other words, the dense lidar point cloud generated based on the image by utilizing the solution of the present invention is a high-quality dense point cloud, the quality of which is close to that of the dense point cloud obtained directly using expensive LiDAR laser radars, thereby greatly saving the costs of hardware procurement, debugging and maintenance; and the accuracy of perception based on the dense lidar point cloud generated based on the image is significantly better than the accuracy of perception based on 2D images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects and advantages of the present invention will become more fully apparent from the following detailed description taken in conjunction with the accompanying drawings, wherein the same or similar elements are denoted by the same reference numerals.

[0021] Figure 1 A schematic diagram of a process for generating a laser radar point cloud according to an embodiment of the present invention is shown;

[0022] Figure 2 A schematic diagram of the structure of a device for generating a laser radar point cloud according to an embodiment of the present invention is shown;

[0023] Figure 3 A schematic diagram showing self-supervised learning of a deep learning network using images and sparse LiDAR point clouds as inputs according to an embodiment of the present invention is shown; and

[0024] Figure 4 A comparison chart of a lidar with a non-repetitive scanning modality and a lidar with a traditional repetitive scanning modality is shown. DETAILED DESCRIPTION

[0025] Hereinafter, a scheme for generating a lidar point cloud according to various exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0026] Figure 1 FIG. 1 is a flow chart of a method 1000 for generating a laser radar point cloud according to an embodiment of the present invention. Figure 1 As shown, the method 1000 for generating a laser radar point cloud comprises the following steps:

[0027] In step 110, an image is received from an image acquisition device; and

[0028] In step 120, the image is input into a trained deep learning network to output a lidar point cloud.

[0029] In some embodiments, the outputted LiDAR point cloud is a dense LiDAR point cloud. Furthermore, during the training phase of the deep learning network, the image and the sparse LiDAR point cloud are input into the deep learning network for self-supervised learning, and wherein the sparse LiDAR point cloud is acquired using a LiDAR with a non-repetitive scanning mode. For example, an inexpensive LiDAR with a non-repetitive scanning mode can be mounted on a movable platform to obtain a sparse point cloud. In some embodiments, the area scanned by the LiDAR with a non-repetitive scanning mode within a certain period of time includes the entire field of view of the image acquisition device.

[0030] In some embodiments, the image and the sparse lidar point cloud both contain perception data for the same relatively constant scene; the relatively constant scene includes a perception scene based on roadside infrastructure, wherein the roadside infrastructure includes: the image acquisition device installed at a fixed position on the roadside and the lidar with a non-repetitive scanning modality, so as to respectively acquire images and sparse lidar point clouds for the same relatively constant scene.

[0031] In some embodiments, the image acquisition device includes a camera or a camera. The image may be a 2D image.

[0032] In the context of the present invention, the term "relatively constant scene" means that the background scene is basically unchanged. For example, the image obtained from a sensor with a fixed observation point (sensor position) and field of view is a relatively constant scene. It is worth noting that in a relatively constant scene, for a laser radar with a non-repetitive scanning mode, as long as enough training data is obtained, the training data or training set can almost cover all encountered situations, which is conducive to robustly obtaining high-quality three-dimensional data for perception tasks. In one or more embodiments, the relatively constant scene includes a perception scene based on roadside infrastructure, a perception scene of VR / AR applications, a robot sensing scene, a factory monitoring scene, etc. Taking the perception scene based on roadside infrastructure as an example, in one embodiment, a camera and a laser radar with a non-repetitive scanning mode are installed at a fixed position on the roadside, so that (1) the observation point and field of view of the camera are fixed; (2) the background scene is fixed (for example, roads, trees, etc.); (3) the location area and size of the presence of traffic participants are repeated and relatively unchanged, for example, highly similar pedestrians always appear in the sidewalk area, and cars of similar size appear in the lane area, etc.

[0033] A method 1000 for generating a lidar point cloud according to an embodiment of the present invention generates a dense lidar point cloud by inputting an image received from an image acquisition device (e.g., a camera) into a trained deep learning network. That is, there is no need to use an expensive lidar (e.g., a 64-channel or 128-channel lidar) to generate a dense lidar point cloud. The embodiment of the present invention only needs to input the image obtained by the camera into a trained deep learning network to generate a dense lidar point cloud of similar quality (compared to directly using an expensive lidar).

[0034] Figure 3 FIG. 1 is a schematic diagram showing a self-supervised learning of a deep learning network using an image and a sparse LiDAR point cloud as input according to an embodiment of the present invention. Figure 3 As shown, during the training phase, images 310 of relatively constant scenes received from a camera and sparse lidar point clouds 315 are provided as inputs to a deep learning network 320 , and ultimately a dense lidar point cloud 330 is obtained.

[0035] In one embodiment, during the training phase of a deep learning network, using images and sparse lidar point clouds as input for self-supervised learning includes: using sparse lidar point clouds to train the deep learning network without annotation, so that the deep learning network can generate dense lidar point clouds based on the images.

[0036] In the context of the present invention, the sparse lidar point cloud is acquired using a lidar with a non-repetitive scanning modality, which may be a low-cost lidar.

[0037] That is, during the training phase of the deep learning network, the sparse point cloud of the LiDAR is used as a reference for calculating the loss. During the testing / inference phase, the input of the deep learning network can be only the camera image. The method of the above embodiment can convert the camera output into a high-precision pseudo LiDAR point cloud by temporarily using the sparse LiDAR point cloud obtained by the low-cost LiDAR for self-supervision.

[0038] A major problem with existing point cloud completion methods is "annotation". Fully supervised methods require manual annotation of depth maps, which is too expensive for industrial use. In contrast, self-supervised methods directly use sparse lidar point clouds as semi-annotations. However, using traditional lidars cannot cover the entire field of view. Figure 4 As shown, the traditional laser radar uses a fixed and repetitive scanning mode, as shown in 420, so the traditional laser radar can only cover a limited area, regardless of the scanning time. The scanning data based on the traditional laser radar can only generate partial annotations for the limited area it covers, and no annotation data can be generated for the other uncovered area, that is, the annotation is incomplete. In contrast, in the method 1000 for generating a laser radar point cloud in an embodiment of the present invention, the sparse laser radar point cloud is acquired using a laser radar with a non-repetitive scanning mode, as shown in 410. The entire scanning area of ​​the laser radar with the non-repetitive scanning mode will cover the entire field of view as the scanning time increases. In this way, when the training data set is large enough and extensive enough, the quality of semi-annotation and self-supervision will be close to that of the fully supervised training scheme.

[0039] In one embodiment, the method 1000 is applied to a perception scene based on roadside infrastructure. In this relatively constant scene, since the camera and the LiDAR with non-repetitive scanning mode are installed at fixed positions on the roadside, (1) the camera's observation point and field of view are fixed; (2) the background scene is fixed (e.g., roads, trees, etc.); (3) the location and size of traffic participants are repeated and unchanged, for example, highly similar pedestrians always appear in the sidewalk area. Due to these characteristics of the perception scene of roadside infrastructure, the method 1000 is more suitable for generating high-quality pseudo LiDAR point clouds.

[0040] In other words, although the above method 1000 is low-cost and self-supervised (no manual labeling is required), the overall supervision quality is still competitive with existing fully supervised methods.

[0041] In addition, it is easy for a person skilled in the art to understand that the method 1000 for generating a laser radar point cloud provided by one or more embodiments of the present invention can be implemented by a computer program. For example, the computer program is included in a computer program product, and when the computer program is executed by a processor, the method 1000 for generating a laser radar point cloud according to one or more embodiments of the present invention is implemented. For another example, when a computer storage medium (such as a USB flash drive) storing the computer program is connected to a computer, the method 1000 for generating a laser radar point cloud according to one or more embodiments of the present invention can be executed by running the computer program.

[0042] refer to Figure 2 , Figure 2 FIG. 2 shows a schematic diagram of a structure of a device 2000 for generating a laser radar point cloud according to an embodiment of the present invention. Figure 2 As shown, the device 2000 for generating a laser radar point cloud includes: a receiving device 210 and a generating device 220. The receiving device 210 is used to receive an image from an image acquisition device; the generating device 220 is used to input the image into a trained deep learning network to output a laser radar point cloud.

[0043] although Figure 2 Not shown, in some embodiments, the generating device 220 may include: the trained deep learning network. In other embodiments, the generating device 220 may also include: a training unit, used to perform self-supervised learning using the image and the sparse lidar point cloud as inputs during the training phase of the deep learning network to obtain the trained deep learning network; and an acquisition unit, used to acquire the sparse lidar point cloud using a lidar with a non-repetitive scanning modality.

[0044] In the context of the present invention, the term "relatively constant scene" means that the background scene is basically unchanged. For example, the image obtained from a sensor with a fixed observation point (sensor position) and field of view is a relatively constant scene. It is worth noting that in a relatively constant scene, for a laser radar with a non-repetitive scanning mode, as long as enough training data is obtained, the training data or training set can almost cover all encountered situations, which is conducive to robustly obtaining high-quality three-dimensional data for perception tasks. In one or more embodiments, the relatively constant scene includes a perception scene based on roadside infrastructure, a perception scene of VR / AR applications, a robot sensing scene, a factory monitoring scene, etc. Taking the perception scene based on roadside infrastructure as an example, in one embodiment, a camera and a laser radar with a non-repetitive scanning mode are installed at a fixed position on the roadside, so that (1) the observation point and field of view of the camera are fixed; (2) the background scene is fixed (for example, roads, trees, etc.); (3) the location area and size of the presence of traffic participants are repeated and relatively unchanged, for example, highly similar pedestrians always appear in the sidewalk area, and cars of similar size appear in the lane area, etc.

[0045] In some embodiments, the image and the sparse lidar point cloud both contain perception data for the same relatively constant scene; the relatively constant scene includes a perception scene based on roadside infrastructure, wherein the roadside infrastructure includes: the image acquisition device installed at a fixed position on the roadside and the lidar with a non-repetitive scanning modality, so as to respectively acquire images and sparse lidar point clouds for the same relatively constant scene.

[0046] In one or more embodiments, the training unit is configured to: train the deep learning network using the sparse lidar point cloud without annotation, so that the deep learning network can generate a dense lidar point cloud based on the image. In one embodiment, the area scanned by the lidar with a non-repetitive scanning mode within a certain period of time includes the entire field of view of the image acquisition device (e.g., a camera).

[0047] In one or more embodiments, the above-mentioned image may be a 2D image.

[0048] The device 2000 for generating a lidar point cloud is self-supervised and does not require a high-quality dense map as ground truth. More importantly, during the inference phase (the application phase of the deep learning network), the device 2000 is able to generate a corresponding high-quality pseudo lidar point cloud using only camera images as input.

[0049] In one or more embodiments, the device 2000 takes full advantage of the perception scene of roadside infrastructure. First, the camera and its field of view are fixed. Second, during the training phase, the training unit can use an inexpensive lidar (e.g., with a non-repetitive horizontal scanning pattern and mounted on a movable platform) as supervision to "teach" the camera to generate a pseudo lidar point cloud. Due to the unique working principle of the lidar, the lidar will scan every pixel in the camera over time. Therefore, although the device 2000 is low-cost and self-supervised (no manual annotation is required), the supervision quality is still competitive with fully-supervised methods.

[0050] Those skilled in the art will appreciate that the device 2000 for generating a laser radar point cloud can be integrated into various types of perception systems (including perception systems based on roadside infrastructure). In one embodiment, the perception system based on roadside infrastructure includes: a camera; and a perception control device, wherein the perception control device includes the device 2000 for generating a laser radar point cloud as described above.

[0051] Those skilled in the art can understand the definition and difference between "dense lidar point cloud" and "sparse lidar point cloud", which will not be elaborated here.

[0052] In summary, the scheme for generating a LiDAR point cloud in an embodiment of the present invention uses images and sparse LiDAR point clouds as inputs for self-supervised learning during the training phase of a deep learning network. Compared to a scheme for generating a dense point cloud (i.e., a pseudo LiDAR point cloud) based on camera images and supervised methods, the scheme in an embodiment of the present invention does not require a high-quality, accurate depth map as the ground truth in the training phase, and thus does not require time-consuming and laborious annotation information. In addition, the sparse LiDAR point cloud is acquired using a LiDAR with a non-repetitive scanning mode, which can scan the entire field of view of the camera over time. Therefore, after the sparse point cloud acquired by the LiDAR with a non-repetitive scanning mode (e.g., a lower-cost LiDAR) is combined with the image obtained by the camera and applied to the learning phase, (when the training data set is sufficiently large and extensive) the overall training quality will be close to that of a fully supervised training scheme. In other words, the dense lidar point cloud generated based on the image by utilizing the solution of the present invention is a high-quality dense point cloud, the quality of which is close to that of the dense point cloud obtained directly using expensive LiDAR laser radars, thereby greatly saving the costs of hardware procurement, debugging and maintenance; and the accuracy of perception based on the dense lidar point cloud generated based on the image is significantly better than the accuracy of perception based on 2D images.

[0053] Although the above description only describes some of the embodiments of the present invention, it should be understood by those skilled in the art that the present invention can be implemented in many other forms without departing from its spirit and scope. Therefore, the examples and embodiments shown are considered to be illustrative rather than restrictive, and the present invention may include various modifications and substitutions without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A method for generating a laser radar point cloud, characterized in that: The method comprises: receiving an image from an image acquisition device; and The image is input into a trained deep learning network to output a lidar point cloud.

2. The method of claim 1, wherein: The outputted lidar point cloud is a dense lidar point cloud; During the training phase of the deep learning network, the image and the sparse lidar point cloud are input into the deep learning network for self-supervised learning, and the sparse lidar point cloud is acquired using a lidar with a non-repetitive scanning modality.

3. The method of claim 2, wherein: The image and the sparse lidar point cloud both contain perception data for the same relatively constant scene; The relatively constant scene includes a perception scene based on roadside infrastructure, wherein the roadside infrastructure includes: the image acquisition device installed at a fixed position on the roadside and the lidar with non-repetitive scanning mode, so as to respectively acquire images and sparse lidar point clouds for the same relatively constant scene.

4. The method of claim 2, wherein: In the training phase of the deep learning network, the image and the sparse lidar point cloud are used as input for self-supervised learning, including: The deep learning network is trained using the sparse lidar point cloud as a reference for calculating loss without annotation, so that the deep learning network can generate a dense lidar point cloud based on the image.

5. The method of claim 3, wherein: The area scanned by the laser radar with non-repetitive scanning mode within a certain period of time includes the entire field of view of the image acquisition device.

6. The method of claim 1, wherein: The image is a 2D image.

7. A device for generating a dense laser radar point cloud, characterized in that: The device comprises: A receiving device, configured to receive an image from an image acquisition device; and Generating means for inputting the image into a trained deep learning network to output a lidar point cloud.

8. The device of claim 7, wherein: The laser radar point cloud output by the generating device is a dense laser radar point cloud, and the generating device comprises: The trained deep learning network; A training unit, configured to use the image and the sparse lidar point cloud as inputs to perform self-supervised learning during a training phase of the deep learning network, so as to obtain the trained deep learning network; and An acquisition unit is used to acquire the sparse lidar point cloud using a lidar with a non-repetitive scanning mode.

9. The device of claim 8, wherein: The image and the sparse lidar point cloud both contain perception data for the same relatively constant scene; The relatively constant scene includes a perception scene based on roadside infrastructure, wherein the roadside infrastructure includes: the image acquisition device installed at a fixed position on the roadside and the lidar with non-repetitive scanning mode, so as to respectively acquire images and sparse lidar point clouds for the same relatively constant scene.

10. The device of claim 8, wherein: The training unit is configured as follows: The deep learning network is trained using the sparse lidar point cloud as a reference for calculating loss without annotation, so that the deep learning network can generate a dense lidar point cloud based on the image.

11. The device of claim 9, wherein: The area scanned by the laser radar with non-repetitive scanning mode within a certain period of time includes the entire field of view of the image acquisition device.

12. The device of claim 7, wherein: The image is a 2D image.

13. A computer storage medium, the medium comprising instructions, characterized in that: The instructions execute the method according to any one of claims 1 to 6 when executed.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

15. A perception system based on roadside infrastructure, characterized in that: The system comprises: Camera; and A perception control device, wherein the device comprises a device as claimed in any one of claims 7 to 12.