Point cloud and camera image matching method and device and point cloud coloring method and system

Abstract

The invention relates to the technical field of image processing, and provides a laser radar point cloud and camera image matching method and device, and a point cloud coloring method and system, and the matching method comprises the following steps: calibrating the external parameters of a camera relative to a laser radar; converting the point cloud from a laser radar coordinate system to a camera coordinate system according to the external parameters of the camera relative to the laser radar; screening out effective point clouds in the view field range of the camera; and projecting the effective point cloud into a pixel coordinate system. According to the scheme, correct point cloud coloring can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a laser radar point cloud and camera image matching method and device, and a point cloud coloring method and system. BACKGROUND

[0002] With the development of society, intelligent driving, machine vision, photogrammetry and other technologies are more and more widely used in various industries. The types of sensors used to collect data are also diverse, including laser radar, camera, millimeter wave radar, etc. These sensors have their own advantages and disadvantages, and also complement each other, so how to efficiently fuse multi-sensor data is also of great significance to the improvement of the effect of perception algorithms.

[0003] The data generated by the camera is 2D data, which can obtain the color, shape, etc. of the object, and the data is dense. However, the camera is greatly affected by external light conditions and cannot be applied to various weather conditions, and a single camera cannot obtain the depth information of the scene. Laser radar can accurately obtain the distance of the object and is less affected by the environment. However, it cannot obtain the color information of the target, and the data is relatively sparse.

[0004] The camera and laser radar data are complementary, and the image data of the camera and the point cloud data of the laser radar are fused to obtain color point cloud, which can obtain the distance and color information of the object in the scene at the same time.

[0005] The existing point cloud coloring method is to first calculate the camera's intrinsic and extrinsic parameters to obtain the projection matrix, and then directly project the point cloud into the image coordinate system. The point cloud falls within which pixel, then the point cloud is colored as the color of the pixel.

[0006] However, the existing technical method does not consider the relative relationship between the camera and the field of view of the laser radar. When the field of view of the laser radar is much larger than that of the camera, the laser radar may see the rear of the camera. Currently, some models of laser radar have a horizontal field of view angle of 360°. Directly using the projection matrix to project will project the point cloud behind the camera into the image coordinate system, causing point cloud coloring error.

[0007] Therefore, it is necessary to provide a laser radar point cloud and camera image matching method and device, and a point cloud coloring method and system, which can realize correct point cloud coloring.

[0008] The above information disclosed in the background section is only used to strengthen the understanding of the background of the present application, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0009] The main purpose of the present application is to overcome the problem that correct point cloud coloring cannot be realized when the field of view of the laser radar is greater than the field of view of the camera, and to provide a laser radar point cloud and camera image matching method and device, and a point cloud coloring method and system, which can realize correct point cloud coloring.

[0010] To achieve the above purpose, the present application provides a laser radar point cloud and camera image matching method in the first aspect, comprising the following steps: Calibrating the external parameter of the camera relative to the laser radar; Converting the point cloud from the laser radar coordinate system to the camera coordinate system according to the external parameter of the camera relative to the laser radar; Screening out effective point clouds within the field of view of the camera; Projecting the effective point clouds into the pixel coordinate system.

[0011] According to an example embodiment of the present application, the external parameter of the camera relative to the laser radar comprises: Using the laser radar to obtain the point cloud of a static scene within a predetermined time; Using the camera to take pictures of the static scene to obtain a calibration image; Selecting a plurality of feature points on the calibration image; Selecting corresponding feature points in the point cloud; Based on the plurality of feature points on the image and the corresponding feature points, the external parameter of the camera relative to the laser radar is obtained.

[0012] According to an example embodiment of the present application, the plurality of feature points are not coplanar, and the number of feature points is more than 6.

[0013] According to an example embodiment of the present application, the external parameter of the camera relative to the laser radar is calculated using the pnp algorithm.

[0014] According to an example embodiment of the present application, the conversion of the point cloud from the laser radar coordinate system to the camera coordinate system according to the external parameter of the camera relative to the laser radar uses the following formula: ; Wherein, R represents the rotation matrix in the external parameter; T represents the translation vector in the external parameter; (x c , y c , z c ) is the coordinate of the point cloud in the camera coordinate system, and (x L , y L , z L ) is the coordinate of the point cloud in the laser radar coordinate system.

[0015] According to an example embodiment of the present application, the screening out of effective point clouds within the field of view of the camera comprises: acquire the position of the normalized epipolar plane of the midpoints of the four edges of the image; calculate the horizontal and vertical field of view angles of the camera according to the four points; mark the point cloud in the opposite direction of the camera shooting direction as invalid points; calculate the angle of the point cloud with the horizontal direction of the camera optical axis and the angle of the point cloud with the vertical direction of the camera optical axis, mark the point cloud with the horizontal angle greater than half of the horizontal field of view angle as invalid points, and mark the point cloud with the vertical angle greater than half of the vertical field of view angle as invalid points; mark the remaining point cloud other than the invalid points as valid point cloud.

[0016] According to an example embodiment of the present application, the projecting the valid point cloud into the pixel coordinate system comprises: projecting the point in the three-dimensional space in the lidar into the two-dimensional image plane of the camera according to the camera intrinsic parameters, distortion parameters and extrinsic parameters.

[0017] As a second aspect of the present application, the present application provides a laser radar point cloud and camera image matching device, comprising: a calibration module, a mapping module, a screening module and a projection module; The calibration module is used to calibrate the extrinsic parameters of the camera relative to the laser radar. The mapping module is used to convert the point cloud from the laser radar coordinate system to the camera coordinate system according to the extrinsic parameters of the camera relative to the laser radar. The screening module is used to screen out the valid point cloud within the field of view of the camera. The projection module is used to project the valid point cloud into the pixel coordinate system.

[0018] As a third aspect of the present application, the present application provides a point cloud coloring method, comprising the following steps: using the laser radar point cloud and camera image matching method; coloring the point cloud.

[0019] According to an example embodiment of the present application, the coloring the point cloud comprises: copying the color of all pixel points of the image to the corresponding point cloud to complete the coloring of all point clouds.

[0020] As a fourth aspect of the present application, the present application provides a point cloud coloring system, comprising: the laser radar point cloud and camera image matching device and the coloring module; The coloring module is used to color the point cloud.

[0021] The advantage of the present application is: The scheme first calibrates the external parameter relationship of the camera and the laser radar, under the condition that the relative relationship between the camera and the laser radar is not changed, the effective point cloud is screened out, and then projection is carried out, so that correct point cloud coloring can be realized. By fusing the data of the camera and the laser radar, from the fused color point cloud, not only the object position can be obtained, but also the object color can be obtained, and better visual effect can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which example embodiments of the present application are shown. The following drawings described below are only some embodiments of the present application, and other drawings can be obtained according to these drawings without creative labor for those skilled in the art.

[0023] Figure 1 The structure diagram of the point cloud coloring system is schematically shown.

[0024] Figure 2 The step diagram of the point cloud coloring method is schematically shown.

[0025] Figure 3 The step diagram of the matching method of the laser radar point cloud and the camera image is schematically shown.

[0026] Figure 4 The schematic diagram of point cloud screening is schematically shown.

[0027] Figure 5 The original image is schematically shown.

[0028] Figure 6 The image of point cloud coloring by the prior art method is schematically shown.

[0029] Figure 7 The image of point cloud coloring by the present scheme is schematically shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. Like reference numerals refer to like elements throughout the drawings, and thus repeated description of them will be omitted.

[0031] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, implementations, and operations have not been shown or described in detail to avoid obscuring aspects of the application.

[0032] The block diagrams in the drawings show only the functionality of the features and can not imply that the functions must be implemented in the particular order presented or by independent circuits or integrated circuits. These are functional block diagrams, and implementation of the described functionality can vary from device to device depending on the capabilities of the machine or circuit or integrated circuit.

[0033] The flow diagrams shown in the figures are examples only and are not necessarily to be construed as having any prior or subsequent steps than those illustrated and described and are not necessarily performed in the order shown. In some instances, steps can be combined or further partitioned into additional steps. In some instances, steps can be performed concurrently or in different orders that those depicted. Moreover, the steps can be implemented in hardware, software, or both.

[0034] It should be understood that, although terms such as first, second, third, etc. can be used herein to describe various components, these components should not be limited by these terms. These terms are used only to distinguish one component from another. Thus, a first component discussed below could be termed a second component without departing from the teachings of the present disclosure. As used herein, the term "and / or" includes any and all combinations of associated items.

[0035] Those skilled in the art will understand that the modules or flows in the drawings are only schematic diagrams of the example embodiments, and are not necessarily the modules or flows that must be implemented in order to implement the present application, and therefore cannot be used to limit the protection scope of the present application.

[0036] According to the first specific embodiment of the present application, the present application provides a point cloud coloring system, as shown in Figure 1 including a matching device of laser radar point cloud and camera image connected with each other and a coloring module.

[0037] As shown in Figure 1 the matching device of laser radar point cloud and camera image includes a calibration module, a mapping module, a screening module and a projection module connected in sequence.

[0038] The calibration module is used for calibrating the external parameter of the camera relative to the laser radar.

[0039] The mapping module is used to transform the point cloud from the lidar coordinate system to the camera coordinate system based on the camera's extrinsic parameters relative to the lidar.

[0040] The filtering module is used to filter out valid point clouds within the camera's field of view.

[0041] The projection module is used to project the effective point cloud onto the pixel coordinate system.

[0042] The coloring module is connected to the projection module and is used to color the point cloud.

[0043] According to a second specific embodiment of the present invention, the present invention provides a point cloud coloring method, such as... Figure 2 As shown, it includes the following steps: S1: A method of matching LiDAR point clouds with camera images is adopted.

[0044] like Figure 3 As shown, the matching method between lidar point clouds and camera images includes: S11: Calibrate the extrinsic parameters of the camera relative to the lidar.

[0045] The extrinsic parameters of the calibrated camera relative to the lidar include: S111: Use LiDAR to acquire point cloud data of a static scene within a predetermined time period.

[0046] This static scene is rich in features. The preset time is 20-50 seconds, preferably 30 seconds.

[0047] S112: Use a camera to capture the static scene and obtain a calibration image.

[0048] The field of view of a camera is smaller than that of a lidar. Generally, lidar and cameras are placed together, either side-by-side or top-bottom, but their fields of view are different.

[0049] Figure 4 In this setup, the LiDAR and camera can be placed side-by-side, stacked vertically, or separated. However, the fields of view of the camera and LiDAR are different. The camera looks directly in front of it, while some LiDARs only look in front of them. Some LiDARs, on the other hand, have a circular field of view that can see in all directions (but not up or down). In this case, the LiDAR can see objects behind the camera.

[0050] S113: Select multiple feature points on the calibration image.

[0051] Multiple feature points are not coplanar, and the number of feature points is more than 6.

[0052] Multiple feature points are either manually selected or automatically calculated using an algorithm. The feature points are evenly distributed, and the distance between each pair of feature points is relatively far.

[0053] S114: Select the corresponding feature points in the point cloud.

[0054] The feature points are also manually selected or automatically calculated using an algorithm.

[0055] S115: Obtain the extrinsic parameters of the camera relative to the lidar based on the multiple feature points on the image and the corresponding feature points.

[0056] Based on the multiple feature points on the image and the corresponding feature points, the pnp algorithm is used to calculate the extrinsic parameters of the camera relative to the lidar.

[0057] In addition to feature points, calibration can also be based on lines, surfaces, and other features in the image and point cloud.

[0058] As long as the relative relationship between the camera and the lidar does not change, there is no need to recalibrate in other scenes.

[0059] Step S11 also includes calibrating the intrinsic parameters and distortion of the camera. The camera intrinsic parameter calibration uses a checkerboard to calibrate, fixes the camera, and captures images of the calibration board at different positions and attitudes. The Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters and distortion of the camera.

[0060] S12: Convert the point cloud from the lidar coordinate system to the camera coordinate system according to the extrinsic parameters of the camera relative to the lidar.

[0061] The point cloud obtained by the lidar is in the lidar coordinate system. In order to be able to filter the point cloud, the extrinsic parameters of the camera are used to transform the point cloud from the lidar coordinate system to the camera coordinate system. The following formula is used to convert the point cloud from the lidar coordinate system to the camera coordinate system according to the extrinsic parameters of the camera relative to the lidar: ; Where R represents the rotation matrix in the extrinsic parameters; T represents the translation vector in the extrinsic parameters; (x c , y c , z c ) are the coordinates of the point cloud in the camera coordinate system, and (x L , y L , z L ) are the coordinates of the point cloud in the lidar coordinate system.

[0062] S13: Filter out the effective point cloud within the camera field of view.

[0063] Filtering out the effective point cloud within the camera field of view includes: S131: Obtain the position of the normalized image plane of the midpoint of the four edges of the image.

[0064] First, the field of view of the camera is calculated, which is step S131 and step S132.

[0065] First, the midpoints of the four edges of the image are selected, and then the pixel positions of the four points before distortion are obtained based on the distortion coefficients in the intrinsic calibration. Then, the positions of the four points on the normalized image plane are calculated based on the pixel positions of the four points.

[0066] S132: Calculate the horizontal field of view and the vertical field of view of the camera according to the four points.

[0067] Connecting the four points with the camera optical center, we can get four rays. The horizontal field of view θ H of the camera is calculated by the angle of the left and right rays, and the vertical field of view θ V of the camera is calculated by the angle of the upper and lower rays. These two field of views represent the field of view range of the camera.

[0068] S133: Mark the point cloud in the opposite direction of the camera shooting direction as invalid points.

[0069] (x c , y c , z c ) are the coordinates of the point cloud in the camera coordinate system.

[0070] The x-axis is the horizontal direction of the camera, the y-axis is the vertical direction of the camera, and the z-axis is the optical axis direction of the camera.

[0071] z c is less than 0, which is the opposite direction of the camera shooting direction. When the z-coordinate of the point cloud is less than 0, this point cloud is marked as invalid points. Figure 4 The 3rd point in the figure is the point with z-coordinate less than 0, which is the invalid point.

[0072] S134: Calculate the angle of the point cloud with the camera optical axis in the horizontal direction and the angle of the point cloud with the camera optical axis in the vertical direction, and mark the point cloud with the horizontal angle greater than half of the horizontal field of view as invalid points, and mark the point cloud with the vertical angle greater than half of the vertical field of view as invalid points.

[0073] ; .

[0074] In the above formula, θ x is the angle of the point cloud with the camera optical axis in the horizontal direction, θ y is the angle of the point cloud with the camera optical axis in the vertical direction, and (x c , y c , z c ) is the coordinate of the point cloud in the camera coordinate system.

[0075] As shown in Figure 4 A, B, C, D are four points in the image edge, and 0 is the camera optical center. ∠A0C is the horizontal field of view of the camera, and ∠BOD is the vertical field of view of the camera. No. 1, No. 2, No. 3 are three points in the radar point cloud, wherein No. 1 is an effective point, No. 2 is an invalid point outside the field of view, and No. 3 is an invalid point behind the camera.

[0076] If the point cloud is colored according to the ordinary projection relationship, the No. 3 point can also be projected into the image, so the No. 3 point will also be colored, but in fact the camera cannot see the No. 3 point. Therefore, only the effective point cloud in the camera field of view is determined first, and then projected, the correct point cloud coloring can be realized.

[0077] S135: The rest of the point cloud except the invalid point is marked as the effective point cloud.

[0078] S14: Project the effective point cloud into the pixel coordinate system.

[0079] Projecting the effective point cloud into the pixel coordinate system includes: Projecting the point in the three-dimensional space in the laser radar into the camera two-dimensional image plane according to the camera intrinsic parameter, distortion parameter and extrinsic parameter.

[0080] In actual use, the function of opencv: projectPoints() can be directly called, the camera intrinsic parameter, extrinsic parameter, distortion coefficient and 3d coordinates of the radar point cloud are input, and the image coordinates of the point cloud projected into the pixel coordinates can be obtained.

[0081] The formula principle of projection is as follows: ; Wherein, x L , y L , z L are three-dimensional coordinates of the point cloud, R is the rotation matrix in the extrinsic parameter, T is the translation vector in the extrinsic parameter, K is the intrinsic parameter of the camera, which can be determined by the focal length, pixel size and principal point position of the camera. In this scheme, the intrinsic parameter K of the camera is calibrated in S11 (when the extrinsic parameter is calculated by using pnp algorithm, the intrinsic parameter needs to be calibrated first). is the scale coefficient, and u, v are the pixel horizontal coordinates and vertical coordinates of the radar point cloud projected into the image.

[0082] S2: Color the point cloud.

[0083] Coloring the point cloud includes copying the color of all pixel points of the image to the corresponding point cloud, and completing the coloring of all point clouds.

[0084] The scheme first calibrates the external parameter relationship of the camera and the laser radar, under the condition that the relative relationship between the camera and the laser radar is fixed, the effective point cloud is screened out, and then projection is carried out, so that correct point cloud coloring can be realized. By fusing the data of the camera and the laser radar two sensors, from the fused color point cloud, not only the object position can be obtained, but also the object color can be obtained, and better visual effect can be realized.

[0085] Figure 5 The original image photographed by the camera is shown, Figure 6 The point cloud obtained by the laser radar, Figure 6 There is a brightness difference in the middle, which is determined by the reflection intensity of the point, not the true color. Figure 7 The colored image of the scheme, that is, the color in Figure 5 is given to the point cloud in Figure 6 , to obtain the colored point cloud in Figure 7 . Comparing the original image and the colored point cloud, it can be found that the points in the point cloud are all given correct colors, such as the lights inside the tunnel, the walls above the tunnel and the billboards, etc. When the camera field of view and the laser radar field of view do not match, especially when the laser radar field of view is very large and the rear of the camera can be seen, the scheme can prevent the point cloud outside the field of view from being colored incorrectly, and ensure the accuracy of the coloring.

[0086] The above specifically shows and describes the exemplary embodiments of the present application. It should be understood that the present application is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A method for matching a lidar point cloud with a camera image, the method comprising: The method comprises the following steps: Calibrating the extrinsic parameters of the camera relative to the lidar; Converting the point cloud from the lidar coordinate system to the camera coordinate system according to the extrinsic parameters of the camera relative to the lidar; Screening the effective point cloud within the camera field of view; Projecting the effective point cloud into the pixel coordinate system.

2. The method of matching lidar point clouds to camera images of claim 1, wherein, The calibration of the extrinsic parameters of the camera relative to the lidar comprises: Using the lidar to obtain the point cloud of a static scene within a predetermined time; Using the camera to capture the static scene to obtain a calibration image; Selecting a plurality of feature points on the calibration image; Selecting corresponding feature points in the point cloud; Based on the plurality of feature points on the image and the corresponding feature points, the extrinsic parameters of the camera relative to the lidar are obtained.

3. The method of matching lidar point clouds to camera images of claim 2, wherein, The plurality of feature points are not coplanar, and the number of the feature points is more than 6.

4. The method of matching lidar point clouds to camera images of claim 1, wherein, The conversion of the point cloud from the lidar coordinate system to the camera coordinate system according to the extrinsic parameters of the camera relative to the lidar adopts the following formula: ； Where R represents the rotation matrix in the extrinsic parameters; T represents the translation vector in the extrinsic parameters; (x c y c , z c (x) represents the coordinates of the point cloud in the camera coordinate system. L y L , z L () represents the coordinates of the point cloud in the lidar coordinate system.

5. The method of matching lidar point clouds to camera images of claim 1, wherein, The screening of the effective point cloud within the camera field of view comprises: Obtaining the position of the normalized phase plane of the midpoints of the four edges of the image; Calculating the horizontal field of view angle and the vertical field of view angle of the camera according to the four points; Marking the point cloud in the opposite direction of the camera shooting direction as invalid points; Calculating the angle of the point cloud with the horizontal direction of the camera optical axis and the angle of the point cloud with the vertical direction of the camera optical axis, marking the point cloud with the horizontal angle greater than half of the horizontal field of view angle as invalid points, and marking the point cloud with the vertical angle greater than half of the vertical field of view angle as invalid points; Marking the remaining point cloud other than the invalid points as effective point cloud.

6. The method of matching lidar point clouds to camera images of claim 1, wherein, The projection of the effective point cloud into the pixel coordinate system comprises: Projecting the point in the three-dimensional space in the lidar into the two-dimensional image plane of the camera according to the camera intrinsic parameters, distortion parameters and extrinsic parameters.

7. A matching device for lidar point clouds and camera images, characterized in that, It comprises: A calibration module, a mapping module, a screening module and a projection module; The calibration module is used for calibrating the extrinsic parameters of the camera relative to the lidar; The mapping module is used for converting the point cloud from the lidar coordinate system to the camera coordinate system according to the extrinsic parameters of the camera relative to the lidar; The screening module is used for screening the effective point cloud within the camera field of view; The projection module is used for projecting the effective point cloud into the pixel coordinate system.

8. A method of point cloud shading, the method comprising: The method comprises the following steps: The matching method for the lidar point cloud and the camera image according to any one of claims 1-6 is adopted; The point cloud is colored.

9. The method of claim 8, wherein, The coloring of the point cloud comprises: The color of all pixel points of the image is copied to the corresponding point cloud, and the coloring of all point clouds is completed.

10. A point cloud shading system, comprising: It comprises: The matching device for the lidar point cloud and the camera image and the coloring module according to claim 6; The coloring module is used for coloring the point cloud.

Images