Render-based lidar and camera alignment

Abstract

A method for aligning a lidar and a camera of a vehicle includes aggregating a plurality of lidar scans performed by a lidar of a vehicle while the vehicle is in motion to generate an aggregated point cloud, rendering the aggregated point cloud onto a camera image to generate a rendered image, comparing the rendered image to the camera image to determine a difference between the rendered image and the camera image, wherein a difference value indicates the difference between the rendered image and the camera image, and in response to determining that the difference value is less than or equal to a predetermined threshold, determining that the camera is aligned with the lidar.

Description

Technical Field

[0001] introduction This disclosure relates to vehicles, and more particularly to rendering-based LiDAR and camera alignment. Summary of the Invention

[0002] This method and system enable accurate alignment between LiDAR and vehicle cameras in targetless environments. In this targetless, data-driven alignment between aggregated LiDAR point clouds and camera images, the LiDAR point clouds are aggregated using a high-precision navigation system. The relative pose between the LiDAR and camera is calculated via an end-to-end differentiable optimization process: rendering the aggregated point cloud on the image plane and comparing it with the camera image. This method and system offer the following advantages: high-precision alignment through temporal LiDAR aggregation; object filtering allowing calibration in dynamic scenes; direct comparison of two color images avoiding multimodal problems; rendering and comparison processes eliminating the need for feature extraction; and an end-to-end optimization process for alignment.

[0003] In this method, a deep neural network object detector is used to filter dynamic objects from both raw LiDAR and camera data. The filtered LiDAR data is then aggregated using a high-precision navigation system that takes into account the motion of the main vehicle. Two optimization steps are then performed until convergence: the aggregated point cloud is rendered onto the camera plane using the estimated alignment and color; the rendered image is directly compared with the camera image; and then, the pose and color are updated simultaneously.

[0004] This disclosure describes a method for aligning a vehicle's LiDAR with a camera. In one aspect of this disclosure, the method includes: aggregating multiple LiDAR scans performed by the vehicle's LiDAR while the vehicle is in motion to generate an aggregated point cloud; receiving a camera image captured by the vehicle's camera while the vehicle is in motion; rendering the aggregated point cloud onto the camera image to generate a rendered image; comparing the rendered image with the camera image to determine a pose difference value between the rendered image and the camera image, and a color difference value between the rendered image and the camera image; and determining camera-LiDAR alignment in response to determining that the pose difference value between the rendered image and the camera image is less than or equal to a predetermined pose threshold, and that the color difference value between the rendered image and the camera image is less than or equal to a predetermined color threshold.

[0005] In this disclosure, the method further includes removing dynamic objects from multiple LiDAR scans using a deep neural network object detector before rendering the aggregated point cloud onto a camera image to generate a rendered image.

[0006] In this disclosure, the method further includes removing dynamic objects from the camera image using a deep neural network object detector before rendering the aggregated point cloud onto the camera image to generate a rendered image.

[0007] In this disclosure, the aggregated point cloud comprises multiple lidar points. Each lidar point has a location and color.

[0008] In this disclosure, the position and color of each of the plurality of lidar points are used to generate a rendered image.

[0009] In this disclosure, the method further includes: in response to determining that the difference value is greater than a predetermined threshold, using the difference value to update the color and pose of the aggregated point cloud to generate updated color and updated pose, respectively.

[0010] In this disclosure, the method further includes: in response to determining that the difference value is greater than a predetermined threshold, re-rendering the aggregated point cloud onto the camera image using updated colors and updated poses to generate an updated rendered image.

[0011] In this disclosure, the method does not include a feature extraction process.

[0012] In this disclosure, the method further includes receiving multiple lidar scans from the vehicle's lidar, wherein the multiple lidar scans are performed at different times while the vehicle is in motion.

[0013] In this disclosure, multiple LiDAR scans are aggregated using the vehicle's navigation system. Therefore, the vehicle's navigation system can be used to perform the aggregation of multiple LiDAR scans.

[0014] This disclosure also describes a control system for aligning a lidar sensor with a camera in a vehicle system. The control system includes a controller that communicates with the vehicle system. The controller is programmed to perform the methods described above.

[0015] Solution 1. A method for aligning a vehicle's lidar with a camera, the method comprising: Multiple lidar scans performed by the vehicle's lidar while the vehicle is in motion are aggregated to generate an aggregated point cloud; Receive camera images captured by the vehicle's camera while the vehicle is in motion; The aggregated point cloud is rendered onto the camera image to generate a rendered image; The rendered image is compared with the camera image to determine the difference between the rendered image and the camera image, wherein the difference value represents an indication of the difference between the rendered image and the camera image; and In response to determining that the difference value is less than or equal to a predetermined threshold, the camera is aligned with the lidar.

[0016] Option 2. According to the method of Option 1, the method further includes: removing dynamic objects from the multiple LiDAR scans using a deep neural network object detector before rendering the aggregated point cloud onto the camera image to generate the rendered image.

[0017] Solution 3. According to the method of Solution 1, the method further includes: removing dynamic objects from the camera image using a deep neural network object detector before rendering the aggregated point cloud onto the camera image to generate a rendered image.

[0018] Option 4. According to the method of Option 1, wherein the aggregated point cloud includes multiple lidar points, and each of the multiple lidar points has a position and a color.

[0019] Option 5. The method according to Option 4, wherein the position and color of each of the plurality of lidar points are used to generate the rendered image.

[0020] Solution 6. According to the method of Solution 5, the method further includes: in response to determining that the difference value is greater than a predetermined color threshold, using the difference value to update the color and pose of the aggregated point cloud to generate updated color and updated pose, respectively.

[0021] Solution 7. According to the method of Solution 6, the method further includes: in response to determining that the difference value is greater than the predetermined threshold, using the updated color and the updated pose to render the aggregated point cloud onto the camera image again to generate an updated rendered image.

[0022] Option 8. The method according to Option 1, wherein the method does not include a feature extraction process.

[0023] Option 9. The method according to Option 1, the method further comprising: receiving the multiple lidar scans from the lidar of the vehicle, wherein the multiple lidar scans are performed at different times while the vehicle is in motion.

[0024] Option 10. The method according to Option 1, wherein the aggregation of the multiple lidar scans is performed using the vehicle's navigation system.

[0025] Option 11. A control system for aligning a lidar sensor with a camera in a vehicle system, the control system comprising: A controller that communicates with the vehicle system, wherein the controller is programmed to: Receive multiple lidar scans from the lidar of the vehicle system; The multiple lidar scans performed by the vehicle's lidar while the vehicle is in motion are aggregated to generate an aggregated point cloud; Receive camera images captured by the vehicle's camera while the vehicle is in motion; The aggregated point cloud is rendered onto the camera image to generate a rendered image; The rendered image is compared with the camera image to determine the difference between the rendered image and the camera image, wherein the difference value represents an indication of the difference between the rendered image and the camera image; and In response to determining that the difference between the rendered image and the camera image is less than or equal to a predetermined threshold, the camera is aligned with the lidar.

[0026] Option 12. The control system according to Option 11, wherein the controller is further programmed to remove dynamic objects from the multiple LiDAR scans using a deep neural network object detector before rendering the aggregated point cloud onto the camera image to generate the rendered image.

[0027] Option 13. The control system according to Option 11, wherein the controller is further programmed to remove dynamic objects from the camera image using a deep neural network object detector before rendering the aggregated point cloud onto the camera image to generate a rendered image.

[0028] Option 14. The control system according to Option 11, wherein the aggregated point cloud includes a plurality of lidar points, each of the plurality of lidar points having a position and a color.

[0029] Option 15. The control system according to Option 14, wherein the position and color of each of the plurality of lidar points are used to generate the rendered image, and the camera image is greater than a predetermined pose threshold.

[0030] Option 16. The control system according to Option 15, wherein the controller is further programmed to: in response to determining that the difference value is greater than a predetermined color threshold, use the difference value to update the color and pose of the aggregated point cloud to generate updated color and updated pose, respectively.

[0031] Option 17. The control system according to Option 16, wherein the controller is further programmed to: in response to determining that the difference value is greater than the predetermined threshold, render the aggregated point cloud onto the camera image again using the updated color and the updated pose to generate an updated rendered image.

[0032] Option 18. The control system according to Option 11, wherein the controller is not programmed with a feature extraction process.

[0033] Option 19. The control system according to Option 11, wherein the multiple lidar scans are performed at different times while the vehicle is in motion.

[0034] Option 20. The control system according to Option 11, wherein the multiple lidar scans are aggregated using the vehicle system's navigation system.

[0035] The above features and advantages of this teaching, as well as other features and advantages, will readily become apparent from the following detailed description of some of the best modes and other embodiments for carrying out this teaching as defined in the appended claims when understood in conjunction with the accompanying drawings. Attached Figure Description

[0036] This disclosure will be described in conjunction with the following accompanying drawings, wherein similar reference numerals denote similar elements.

[0037] Figure 1 This is a schematic side view of the vehicle.

[0038] Figure 2 This is a flowchart of a method for aligning rendering-based LiDAR and cameras.

[0039] Figure 3 yes Figure 2 The diagram illustrates a schematic representation of the multiscan aggregation process of the method shown in the figure. Detailed Implementation

[0040] The following detailed description is merely exemplary in nature and is not intended to limit application or use. Furthermore, it is not intended to be bound by the express or implied theories presented in the foregoing technical field, background art, summary of the invention, or the following detailed description. As used herein, the term "module" refers individually or in combination to hardware, software, firmware, electronic control components, processing logic, and / or processor devices, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped) and memories executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality.

[0041] Embodiments of this disclosure may be described herein in terms of functional and / or logical block components and various processing steps. It should be understood that such block components may be implemented by a number of hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of this disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which may implement various functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that embodiments of this disclosure may be practiced in conjunction with many systems, and the systems described herein are merely exemplary embodiments of this disclosure.

[0042] For the sake of brevity, techniques relating to signal processing, data fusion, signaling, control, and other functional aspects of the system (and its various operating components) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures included herein are intended to represent exemplary functional relationships and / or physical connections between various elements. It should be noted that alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0043] refer to Figure 1 The vehicle 10 includes a body 12 and multiple wheels 14. In addition to the wheels 14, the vehicle 10 includes one or more lidar sensors 16 (i.e., a light detection and ranging system), one or more cameras 18, and a navigation system 20. The lidar sensor 16 is a detection system that operates based on radar principles, but uses light from a laser. The lidar sensor 16 uses light in the form of pulsed laser light to measure the range (variable distance) from an object. Each of the cameras 18 is configured to capture an image of the object. The navigation system 20 includes a transceiver or other device configured to communicate with a Global Navigation Satellite System (GNSS).

[0044] Camera 18, LiDAR 16, and navigation system 20 communicate with controller 22. Controller 22 is part of control system 24 and includes at least one processor 26 and a non-transitory computer-readable storage device or medium 28. Processor 26 may be a custom processor, a central processing unit (CPU), an auxiliary processor among several processors associated with controller 22, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a means for executing instructions. For example, computer-readable storage device or medium 28 may include volatile and non-volatile storage devices in the form of read-only memory (ROM), random access memory (RAM), and non-fail-to-recovery memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operational variables when processor 26 is powered off. The computer-readable storage device or medium 28 may be implemented using a variety of memory devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or another electrical, magnetic, optical, or combined memory device capable of storing data used by the controller 22 (some of which represents executable instructions). Although shown inside the vehicle 10, the controller 22 may be outside the vehicle 10. In any case, the controller 22 communicates with the camera 18, the lidar 16, and the navigation system 20. The vehicle 10 may be referred to as a vehicle system.

[0045] Figure 2 This is a flowchart of a method 100 for rendering-based LiDAR and camera alignment. In other words, method 100 is used to align cameras 18 with LiDAR 16. Each camera 18 is processed individually and optimized with the corresponding LiDAR 16. Method 100 begins at block 102, where the LiDAR 16 performs multiple LiDAR scans at different times while the vehicle 10 is in motion. The controller 22 then receives the multiple LiDAR scans from the LiDAR 16 of the vehicle 10. Because the LiDAR 16 has a specific range, the multiple LiDAR scans from the LiDAR 16 are aggregated using, for example, a navigation system 20 to virtually extend the range of the LiDAR 16 to generate a three-dimensional (3D) aggregated point cloud.

[0046] Figure 3A schematic illustration shows multiple lidar scans from lidar 16 aggregated at different times. LiDAR 16 scans along the path of vehicle 10 and identifies one or more static objects 30 and dynamic objects (not shown) (such as a moving car). In this disclosure, the static objects 30 are stationary relative to vehicle 10, and the dynamic objects are in motion relative to the static scene. Block 102 may be performed by controller 22 in conjunction with navigation system 20. After aggregating the multiple lidar scans, method 100 proceeds to block 104. Method 100 then proceeds to block 104.

[0047] The aggregated point cloud comprises multiple lidar points. Each of these lidar points has a position [R|T] schematically illustrated at block 110 and a color (RGB) schematically illustrated at block 112. The relative position [R|T] includes a rotation component R and a translation component T. The color (RGB) can be expressed using the RGB color model and therefore can have red / R values, green / G values, and blue / B values. An auxiliary task can be used to colorize the lidar scans.

[0048] At block 104, controller 22 uses a deep neural network object detector to remove (i.e., mask) dynamic objects from multiple LiDAR scans. This allows dynamic objects to be removed from the aggregated point cloud.

[0049] Method 100 also includes block 106. At block 106, camera 18 captures two-dimensional camera images at different times while vehicle 10 is in motion. The raw camera images captured by camera 18 include moving and static objects 30 (…). Figure 3 The controller 22 then receives the camera image captured by the camera 18. Then, method 100 proceeds to block 108.

[0050] At block 108, controller 22 uses a deep neural network object detector to remove (i.e., mask) dynamic objects from the camera images captured by camera 18. At this point, the camera images do not include dynamic objects. After blocks 104 and 108, method 100 proceeds to block 114.

[0051] At block 114, controller 22 renders the point cloud of the image aggregation onto the camera image captured by camera 18, thereby generating a rendered image at block 116. Therefore, the rendered image includes LiDAR points superimposed on the camera image captured by camera 18. To create the rendered image, controller 22 uses the relative position [R|T] from block 110 and the RGB color from block 112. After creating the rendered image, method 100 proceeds to block 118.

[0052] At block 118, controller 22 compares the rendered image with the camera image to determine the difference between the rendered image and the camera image. The difference value indicates the difference between the rendered image and the camera image. This difference value may include: 1) the pose difference value between the rendered image and the camera image. R, T, and 2) The color difference value between the rendered image and the camera image. The rendering performed at block 114 and the comparison performed at block 118 avoid the need for feature extraction in method 100. Therefore, method 100 is characterized by not including feature extraction. If the difference value is less than or equal to a predetermined threshold, controller 22 determines that camera 18 is aligned with LiDAR 16 and method 100 ends. However, if the difference value is greater than the predetermined threshold, method 200 returns to blocks 110 and 112.

[0053] At block 110, if the difference value is greater than a predetermined threshold, the position of the aggregated point cloud is updated to generate the updated position [R|T]. At block 112, if the difference value is greater than a predetermined threshold, the color red-green-blue (RGB) is updated to generate the updated color RGB. Method 100 then returns to block 116. At block 116, in response to determining that the pose difference value between the rendered image and the camera image is greater than a predetermined pose threshold, and the color difference value between the rendered image and the camera image is greater than a predetermined color threshold, controller 22 uses the updated color and the updated pose to render the aggregated point cloud onto the camera image again to generate the updated rendered image.

[0054] The detailed description and accompanying drawings or figures are supporting descriptions of this teaching, but the scope of this teaching is defined only by the claims. While some of the best modes and other embodiments for carrying out this teaching have been described in detail, various alternative designs and embodiments exist for practicing this teaching as defined in the appended claims.

Claims

1. A method for aligning a vehicle's lidar with a camera, the method comprising: Multiple lidar scans performed by the vehicle's lidar while the vehicle is in motion are aggregated to generate an aggregated point cloud; Receive camera images captured by the vehicle's camera while the vehicle is in motion; The aggregated point cloud is rendered onto the camera image to generate a rendered image; The rendered image is compared with the camera image to determine the difference between the rendered image and the camera image, wherein the difference value represents an indication of the difference between the rendered image and the camera image; The aggregated point cloud includes multiple LiDAR points, each of which has a position and a color. The position and color of each of the multiple LiDAR points are used to generate the rendered image. The method further includes, in response to determining that the difference value is greater than a predetermined color threshold, using the difference value to update the color and pose of the aggregated point cloud to generate updated color and updated pose, respectively. In response to determining that the difference value is greater than the predetermined color threshold, the aggregated point cloud is rendered onto the camera image again using the updated color and the updated pose to generate an updated rendered image.

2. The method according to claim 1, further comprising: Before rendering the aggregated point cloud onto the camera image to generate the rendered image, a deep neural network object detector is used to remove dynamic objects from the multiple LiDAR scans.

3. The method according to claim 1, wherein the method further comprises: Before rendering the aggregated point cloud onto the camera image to generate a rendered image, dynamic objects are removed from the camera image using a deep neural network object detector.

4. The method according to claim 1, wherein, The method does not include a feature extraction process.

5. The method according to claim 1, wherein the method further comprises: The multiple lidar scans are received from the vehicle's lidar, and the multiple lidar scans are performed at different times while the vehicle is in motion.

6. The method according to claim 1, wherein, The aggregation of the multiple lidar scans is performed using the vehicle's navigation system.

7. A control system for aligning a lidar sensor with a camera in a vehicle system, the control system comprising: A controller that communicates with the vehicle system, wherein the controller is programmed to: Receive multiple lidar scans from the lidar of the vehicle system; The multiple lidar scans performed by the vehicle's lidar while the vehicle is in motion are aggregated to generate an aggregated point cloud; Receive camera images captured by the vehicle's camera while the vehicle is in motion; The aggregated point cloud is rendered onto the camera image to generate a rendered image; The rendered image is compared with the camera image to determine the difference between the rendered image and the camera image, wherein the difference value represents an indication of the difference between the rendered image and the camera image; The aggregated point cloud includes multiple lidar points, each of which has a position and a color. The position and color of each of the multiple lidar points are used to generate the rendered image, and the camera image is greater than a predetermined pose threshold. In response to determining that the difference value is greater than a predetermined color threshold, the controller uses the difference value to update the color and pose of the aggregated point cloud to generate updated color and updated pose, respectively. In response to determining that the difference value is greater than the predetermined color threshold, the aggregated point cloud is rendered onto the camera image again using the updated color and the updated pose to generate an updated rendered image.

8. The control system according to claim 7, wherein, The controller is further programmed to remove dynamic objects from the multiple LiDAR scans using a deep neural network object detector before rendering the aggregated point cloud onto the camera image to generate the rendered image.

9. The control system according to claim 7, wherein, The controller is further programmed to remove dynamic objects from the camera image using a deep neural network object detector before rendering the aggregated point cloud onto the camera image to generate a rendered image.

10. The control system according to claim 7, wherein, The controller was not programmed with a feature extraction process.

11. The control system according to claim 7, wherein, The multiple lidar scans are performed at different times while the vehicle is in motion.

12. The control system according to claim 7, wherein, The multiple lidar scans are aggregated using the vehicle system's navigation system.

Images