Laser radar point cloud projection method, device, equipment and storage medium

Abstract

The present invention relates to the field of autonomous driving technology and discloses a laser radar point cloud projection method, apparatus, device, and storage medium for improving the accuracy of laser radar and camera information fusion. The laser radar point cloud projection method comprises: obtaining an original point cloud scanned by the laser radar and removing motion distortion from the original point cloud to obtain a first point cloud; converting the first point cloud to a camera coordinate system based on the target exposure time of the original camera image to obtain a second point cloud; adding camera motion distortion to the second point cloud to obtain a target point cloud, and projecting the target point cloud onto the original camera image.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a laser radar point cloud projection method, device, equipment and storage medium. Background Art

[0002] In the field of autonomous driving technology, in order to improve the accuracy of environmental perception, it is usually necessary to fuse the information collected by different sensors. Among them, the fusion of information from lidar sensors and image sensors is the focus of sensor information fusion, and ensuring the accuracy of sensor information fusion is the basis for ensuring the accuracy of environmental perception.

[0003] The existing projection method of lidar and camera is generally to dedistort the point cloud and image collected by the two separately, and then project and superimpose the dedistorted point cloud and image. However, due to the different dedistortion principles, the projection superposition after dedistortion still has large errors, which reduces the accuracy of the lidar and camera information fusion. Summary of the Invention

[0004] The present invention provides a laser radar point cloud projection method, device, equipment and storage medium for improving the accuracy of laser radar and camera information fusion.

[0005] A first aspect of the present invention provides a laser radar point cloud projection method, comprising:

[0006] Obtaining an original point cloud scanned by a laser radar, and removing motion distortion from the original point cloud to obtain a first point cloud;

[0007] Based on the target exposure time of the original camera image, converting the first point cloud to a camera coordinate system to obtain a second point cloud;

[0008] Camera motion distortion is added to the second point cloud to obtain a target point cloud, and the target point cloud is projected onto the original camera image.

[0009] Optionally, obtaining an original point cloud scanned by a laser radar and removing motion distortion from the original point cloud to obtain a first point cloud includes:

[0010] Acquire a first extrinsic parameter of the laser radar, where the first extrinsic parameter is used to indicate a relative position parameter between the laser radar and the vehicle coordinate system;

[0011] Obtaining an original point cloud scanned by the laser radar, and determining a starting scanning time of the original point cloud;

[0012] A first ego-vehicle pose change between the start scanning moment and the scanning moment of each lidar point is obtained, and each lidar point in the original point cloud is converted to an ego-vehicle coordinate system using the first ego-vehicle pose change and the first extrinsic parameter to obtain a first point cloud.

[0013] Optionally, converting the first point cloud to a camera coordinate system based on the target exposure moment of the original camera image to obtain a second point cloud includes:

[0014] Acquire a second extrinsic parameter of the camera, where the second extrinsic parameter is used to indicate a relative position parameter between the camera and the vehicle coordinate system;

[0015] Obtaining a target exposure time of an original camera image captured by the camera, and obtaining a second vehicle posture change between the start scanning time and the target exposure time, wherein the target exposure time is used to indicate the exposure time of a middle row or middle column of the original camera image;

[0016] Each lidar point in the first point cloud is converted to a camera coordinate system using the second ego-vehicle pose change and the second extrinsic parameter to obtain a second point cloud.

[0017] Optionally, performing camera motion distortion on the second point cloud to obtain a target point cloud includes:

[0018] Determining a target projection time difference, the target projection time difference being used to indicate a difference between an exposure time of each lidar point at a target projection position and the target exposure time;

[0019] Based on the preset camera distortion internal parameters and the target projection time difference, the second point cloud is subjected to camera motion distortion addition to obtain a target point cloud, wherein the target point cloud includes coordinate information of each lidar point at the target projection position.

[0020] Optionally, determining the target projection time difference includes:

[0021] Obtaining a unit exposure time difference of the original camera image and a vehicle motion parameter within a scanning period of the original point cloud, wherein the unit exposure time difference is used to indicate a difference in exposure times of adjacent rows or adjacent columns;

[0022] The target projection time difference is calculated by the unit exposure time difference, the vehicle motion parameter, the preset camera distortion internal parameter and the preset synchronization time error.

[0023] Optionally, the performing camera motion distortion addition on the second point cloud based on a preset camera distortion intrinsic parameter and the target projection time difference to obtain a target point cloud includes:

[0024] Calculating a third vehicle posture change within a target projection time difference period for each lidar point in the second point cloud using the target projection time difference;

[0025] The target projection position of each lidar point is calculated by the third ego-vehicle posture change and the preset camera distortion internal parameter to obtain a target point cloud. The target projection position is used to indicate the coordinate information of each lidar point in the target point cloud.

[0026] Optionally, the calculation formula of the target point cloud is:

[0027] P0=R*P1+v*t i

[0028] Where P0 represents the coordinate information of a lidar point in the target point cloud, R represents the rotation matrix of the ego vehicle, P1 represents the coordinate information of the corresponding lidar point in the second point cloud, v represents the average speed of the ego vehicle during the scanning period of the original point cloud, and t i Represents the unit exposure time difference.

[0029] A second aspect of the present invention provides a laser radar point cloud projection device, comprising:

[0030] An acquisition module is used to acquire an original point cloud scanned by the laser radar and remove motion distortion from the original point cloud to obtain a first point cloud;

[0031] A conversion module, configured to convert the first point cloud into a camera coordinate system based on a target exposure moment of the original camera image to obtain a second point cloud;

[0032] A projection module is used to add camera motion distortion to the second point cloud to obtain a target point cloud, and project the target point cloud onto the original camera image.

[0033] Optionally, the acquisition module is specifically configured to:

[0034] Acquire a first extrinsic parameter of the laser radar, where the first extrinsic parameter is used to indicate a relative position parameter between the laser radar and the vehicle coordinate system;

[0035] Obtaining an original point cloud scanned by the laser radar, and determining a starting scanning time of the original point cloud;

[0036] A first ego-vehicle pose change between the start scanning moment and the scanning moment of each lidar point is obtained, and each lidar point in the original point cloud is converted to an ego-vehicle coordinate system using the first ego-vehicle pose change and the first extrinsic parameter to obtain a first point cloud.

[0037] Optionally, the conversion module is specifically configured to:

[0038] Acquire a second extrinsic parameter of the camera, where the second extrinsic parameter is used to indicate a relative position parameter between the camera and the vehicle coordinate system;

[0039] Obtaining a target exposure time of an original camera image captured by the camera, and obtaining a second vehicle posture change between the start scanning time and the target exposure time, wherein the target exposure time is used to indicate the exposure time of a middle row or middle column of the original camera image;

[0040] Each lidar point in the first point cloud is converted to a camera coordinate system using the second ego-vehicle pose change and the second extrinsic parameter to obtain a second point cloud.

[0041] Optionally, the projection module includes:

[0042] a determination unit, configured to determine a target projection time difference, wherein the target projection time difference indicates a difference between an exposure time of each laser radar point at a target projection position and the target exposure time;

[0043] An adding unit is used to perform camera motion distortion addition on the second point cloud based on a preset camera distortion internal parameter and the target projection time difference to obtain a target point cloud, wherein the target point cloud includes coordinate information of each lidar point at the target projection position.

[0044] Optionally, the determining unit is specifically configured to:

[0045] Obtaining a unit exposure time difference of the original camera image and a vehicle motion parameter within a scanning period of the original point cloud, wherein the unit exposure time difference is used to indicate a difference in exposure times of adjacent rows or adjacent columns;

[0046] The target projection time difference is calculated by the unit exposure time difference, the vehicle motion parameter, the preset camera distortion internal parameter and the preset synchronization time error.

[0047] Optionally, the adding unit is specifically configured to:

[0048] Calculating a third vehicle posture change within a target projection time difference period for each lidar point in the second point cloud using the target projection time difference;

[0049] The target projection position of each lidar point is calculated by the third ego-vehicle posture change and the preset camera distortion internal parameter to obtain a target point cloud. The target projection position is used to indicate the coordinate information of each lidar point in the target point cloud.

[0050] Optionally, the calculation formula of the target point cloud is:

[0051] P0=R*P1+v*t i

[0052] Where P0 represents the coordinate information of a lidar point in the target point cloud, R represents the rotation matrix of the ego vehicle, P1 represents the coordinate information of the corresponding lidar point in the second point cloud, v represents the average speed of the ego vehicle during the scanning period of the original point cloud, and t i Represents the unit exposure time difference.

[0053] The third aspect of the present invention provides a laser radar point cloud projection device, comprising: a memory and at least one processor, wherein a computer program is stored in the memory; the at least one processor calls the computer program in the memory so that the laser radar point cloud projection device executes the above-mentioned laser radar point cloud projection method.

[0054] A fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned laser radar point cloud projection method.

[0055] In the technical solution provided by the present invention, an original point cloud scanned by a laser radar is obtained, and motion distortion is removed from the original point cloud to obtain a first point cloud; based on the target exposure moment of the original camera image, the first point cloud is converted to a camera coordinate system to obtain a second point cloud; camera motion distortion is added to the second point cloud to obtain a target point cloud, and the target point cloud is projected onto the original camera image. In an embodiment of the present invention, in order to avoid projection errors caused by different distortion removal principles between the lidar point cloud and the camera image, after obtaining the original point cloud scanned by the lidar, only the original point cloud is subjected to motion distortion removal to obtain a first point cloud based on the target reference object coordinate system, and then the first point cloud is converted to the camera coordinate system to obtain a second point cloud in the same coordinate system as the original camera image. Finally, based on the motion distortion principle of the camera, the same motion distortion as the original camera image is added to the second point cloud to obtain a target point cloud, and the target point cloud is projected onto the original camera image to obtain a target point cloud image in which the lidar and camera information are fused, so that the lidar point cloud has the same motion distortion as the camera image. Subsequently, the same image dedistortion principle is used to simultaneously remove motion distortion from the lidar point cloud and the camera image in the target point cloud image. Therefore, the present invention can improve the accuracy of the fusion of lidar and camera information. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic diagram of an embodiment of a projection method of a laser radar point cloud according to an embodiment of the present invention;

[0057] Figure 2Schematic diagram of another embodiment of a laser radar point cloud projection method according to an embodiment of the present invention;

[0058] Figure 3 Schematic diagram of an embodiment of a projection device for a laser radar point cloud according to an embodiment of the present invention;

[0059] Figure 4 Schematic diagram of another embodiment of a projection device for a laser radar point cloud according to an embodiment of the present invention;

[0060] Figure 5 Schematic diagram of an embodiment of a projection device for a laser radar point cloud in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] Embodiments of the present invention provide a laser radar point cloud projection method, apparatus, device, and storage medium for improving the accuracy of laser radar and camera information fusion.

[0062] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0063] It is understood that the execution subject of the present invention can be a projection device for the laser radar point cloud, or a terminal or server. The terminal can be an autonomous driving terminal, and the specific implementation is not limited here. The embodiments of the present invention are described using the terminal as the execution subject as an example.

[0064] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 An embodiment of a laser radar point cloud projection method according to an embodiment of the present invention includes:

[0065] 101. Obtain an original point cloud scanned by the laser radar, and remove motion distortion from the original point cloud to obtain a first point cloud;

[0066] It should be noted that the raw point cloud scanned by the LiDAR refers to the point cloud scanned by the LiDAR based on its internal principles. The raw point cloud is a collection of LiDAR point data based on the LiDAR coordinate system. It is understandable that the raw point cloud is not scanned instantly, but is obtained when the LiDAR completes a circle scan. For example, assuming that the time required for the LiDAR to complete a circle scan is 100ms, then the raw point cloud is the collection of point data scanned by the LiDAR within this 100ms. If the vehicle is moving, the raw point cloud scanned by the LiDAR will have motion distortion. Therefore, in order to improve the accuracy of the LiDAR and camera information fusion, by removing motion distortion from the raw point cloud, a first point cloud without motion distortion can be obtained for subsequent further calculations.

[0067] In one embodiment, in order to remove motion distortion of the original point cloud, the original point cloud can be converted to the target coordinate system at the same time. Specifically, the terminal obtains the relative position parameters (i.e., external parameters) between the laser radar and the target coordinate system, and then determines the target scanning time to be converted, wherein the target scanning time can be the scanning time of any laser radar point in the original point cloud. Based on the relative position parameters between the laser radar and the target coordinate system, the laser radar posture change between the target scanning time and the scanning time of each laser radar point, the terminal converts each laser radar point in the original point cloud to the target coordinate system at the target scanning time to obtain a first point cloud, wherein the target coordinate system can be any reference object coordinate system that has a mapping relationship with the camera to be projected, such as the world coordinate system, the vehicle coordinate system (i.e., the baselink coordinate system), a virtual camera or other cameras, etc., which are not limited here.

[0068] 102. Based on the target exposure time of the original camera image, convert the first point cloud into a camera coordinate system to obtain a second point cloud;

[0069] It should be noted that the original camera image is the image captured by the camera to be projected, i.e., the image of the original point cloud to be projected. The camera coordinate system is the coordinate system of the camera to be projected. To fuse the second point cloud with the original camera image, the terminal converts the first point cloud to the camera coordinate system based on the target exposure time of the original camera image to obtain the second point cloud. The camera to be projected uses a row-by-row or column-by-column exposure method to scan and obtain the original camera image. The camera to be projected is, for example, a rolling shutter camera. It is understandable that, taking the camera to be projected that uses a row-by-row exposure method as an example, when the vehicle is in motion, the original camera image obtained by scanning using a row-by-row exposure method exhibits a rolling effect. That is, the exposure time of each row of pixels in the original camera image is different, resulting in motion distortion in the original camera image.

[0070] In one embodiment, to place the second point cloud in the same coordinate system as the original camera image, the first point cloud is subjected to a camera coordinate system transformation based on the target exposure time of the original camera image to obtain the second point cloud. The target exposure time is used to indicate the exposure time of the target row or target column of the original camera image. For example, the target exposure time can be the earliest exposure time of the row or column of the original camera image, or the latest exposure time of the row or column of the original camera image, and the specific details are not limited here. Specifically, the terminal transforms the first point cloud to the camera coordinate system at the target exposure time by using the camera pose change between the target exposure time and the target scanning time and the relative position parameters (i.e., extrinsic parameters) between the camera to be projected and the target coordinate system to obtain the second point cloud.

[0071] 103. Perform camera motion distortion on the second point cloud to obtain a target point cloud, and project the target point cloud onto the original camera image.

[0072] It is understandable that to improve the accuracy of the fusion of LiDAR and camera information, the terminal adds the same motion distortion as the original camera image to the second point cloud to obtain the target point cloud, and then projects the target point cloud onto the original camera image to obtain the fused point cloud image. This allows the point cloud and image in the fused point cloud image to be dedistorted simultaneously according to the same image dedistortion algorithm, thereby accurately fusing the LiDAR point cloud and camera image. Compared to dedistorting the LiDAR point cloud and camera image separately before fusing, this approach can reduce data loss caused by the fusion process and improve the synchronization between the LiDAR point cloud and camera image.

[0073] It should be noted that since the camera image is scanned by row-by-row or column-by-column exposure, the exposure time of the image cannot be ignored and is about 50ms. If only a single exposure moment is considered for processing, the target point cloud after adding camera motion distortion will still deviate from the original camera image with actual motion distortion. Therefore, in order to further improve the accuracy of adding camera motion distortion, in one embodiment, the terminal first calculates the difference between the exposure time of the target projection position of each lidar point in the target point cloud and the target exposure time, wherein the exposure time of the target projection position of each lidar point is an unknown variable, but the difference between the two can be obtained through known data, and then the target projection time of each lidar point is calculated based on the difference between the two. The target projection time is used to indicate the actual projection time of the lidar point. Finally, the target point cloud is determined based on the camera pose change of each lidar point between the actual projection time and the target exposure time, and the target point cloud is projected onto the original camera image.

[0074] In an embodiment of the present invention, in order to avoid projection errors caused by different distortion removal principles between the lidar point cloud and the camera image, after obtaining the original point cloud scanned by the lidar, only the original point cloud is subjected to motion distortion removal to obtain a first point cloud based on the target reference object coordinate system, and then the first point cloud is converted to the camera coordinate system to obtain a second point cloud in the same coordinate system as the original camera image. Finally, based on the motion distortion principle of the camera, the same motion distortion as the original camera image is added to the second point cloud to obtain a target point cloud, and the target point cloud is projected onto the original camera image to obtain a target point cloud image in which the lidar and camera information are fused, so that the lidar point cloud has the same motion distortion as the camera image. Subsequently, the same image dedistortion principle is used to simultaneously remove motion distortion from the lidar point cloud and the camera image in the target point cloud image. Therefore, the present invention can improve the accuracy of the fusion of lidar and camera information.

[0075] See also Figure 2 Another embodiment of the laser radar point cloud projection method in the embodiment of the present invention includes:

[0076] 201. Obtain an original point cloud scanned by a laser radar, and remove motion distortion from the original point cloud to obtain a first point cloud;

[0077] Specifically, step 201 includes: obtaining a first extrinsic parameter of the laser radar, where the first extrinsic parameter is used to indicate the relative position parameter between the laser radar and the vehicle coordinate system; obtaining an original point cloud scanned by the laser radar, and determining the starting scanning time of the original point cloud; obtaining a first vehicle posture change between the starting scanning time and the scanning time of each laser radar point, and converting each laser radar point in the original point cloud to the vehicle coordinate system through the first vehicle posture change and the first extrinsic parameter to obtain a first point cloud.

[0078] In this embodiment, in order to improve the accuracy of the fusion of laser radar and camera information, the vehicle coordinate system is used as a bridge for the coordinate system conversion between the laser radar and the camera. Therefore, the terminal obtains the first external parameter of the laser radar, wherein the first external parameter is used to indicate the relative position parameter between the laser radar and the vehicle coordinate system. The terminal then determines the laser radar point with the earliest scanning time through the scanning time of each laser radar point in the original point cloud, and determines the earliest scanning time as the starting scanning time of the original point cloud. Then, the terminal calculates the first vehicle posture change from the starting scanning time to the scanning time of each laser radar point in the original point cloud. Finally, based on the first external parameter and the first vehicle posture change, the terminal converts each laser radar point in the original point cloud to the vehicle coordinate system at the starting scanning time to obtain the first point cloud. The first point cloud includes the vehicle coordinate information of each laser radar point at the starting scanning time.

[0079] 202. Based on the target exposure time of the original camera image, convert the first point cloud into a camera coordinate system to obtain a second point cloud;

[0080] Specifically, step 202 includes: obtaining a second extrinsic parameter of the camera, where the second extrinsic parameter is used to indicate a relative position parameter between the camera and the ego-vehicle coordinate system; obtaining a target exposure time of the original camera image captured by the camera, and obtaining a second ego-vehicle pose change between the start scanning time and the target exposure time, where the target exposure time is used to indicate the exposure time of the middle row or middle column of the original camera image; and converting each lidar point in the first point cloud to the camera coordinate system using the second ego-vehicle pose change and the second extrinsic parameter to obtain a second point cloud.

[0081] In this embodiment, similar to step 201, after obtaining the first point cloud in the ego-vehicle coordinate system, the terminal obtains the second extrinsic parameter of the camera to be projected. The second extrinsic parameter indicates the relative position parameter between the camera to be projected and the ego-vehicle coordinate system. The terminal then calculates the second ego-vehicle pose change from the start scan time to the target exposure time. Using the second ego-vehicle pose change and the second extrinsic parameter, each lidar point in the first point cloud is converted to the camera coordinate system at the target exposure time, resulting in a second point cloud. The second point cloud includes the camera coordinate information for each lidar point at the target exposure time. It should be noted that the target exposure time is the exposure time of the middle row or middle column of the original camera image.

[0082] 203. Determine a target projection time difference, where the target projection time difference indicates the difference between the exposure time of each laser radar point at the target projection position and the target exposure time;

[0083] It should be noted that the target projection position is used to indicate the true projection position of the corresponding LiDAR point, that is, the projection position of the corresponding LiDAR point in the original camera image. Since the target projection position is an unknown parameter, but the target projection time difference can be calculated using known parameters, the terminal first calculates the target projection time difference and then performs subsequent processing based on the target projection time difference to determine the target projection position of each LiDAR point in the target point cloud. In this embodiment, since the true projection position of a LiDAR point is difficult to predict directly, the true projection position is indirectly predicted by predicting the time difference between the exposure time of the true projection position and the target exposure time, thereby improving the accuracy of the true projection position calculation and, in turn, improving the accuracy of the LiDAR and camera information fusion.

[0084] Specifically, step 203 includes: obtaining the unit exposure time difference of the original camera image and the ego-vehicle motion parameters within the scanning period of the original point cloud, where the unit exposure time difference is used to indicate the exposure time difference between adjacent rows or columns; and calculating the target projection time difference using the unit exposure time difference, the ego-vehicle motion parameters, a preset camera distortion internal parameter, and a preset synchronization time error.

[0085] In this embodiment, in order to determine the target projection time difference, the terminal calculates the target projection time difference through parameters such as the unit exposure time difference of the original camera image, the vehicle motion parameters, the preset camera distortion internal parameters, and the preset synchronization time error. Among them, the vehicle motion parameters include the average rotation angular velocity and the average speed, the camera distortion internal parameters include the camera focal length, and the preset synchronization time error is the synchronization timestamp error value of the middle row or middle column of the original camera image.

[0086] Specifically, for a camera that scans in a line-by-line exposure mode, to solve the target projection time difference, first solve the following quadratic equation to obtain

[0087]

[0088] Among them, Z2, Z1, Y2, and Y1 in the above quadratic equation are known intermediate variables, and the calculation formula is as follows:

[0089] Z2=[ω*f*t i ] × *Z c +v*f*t i

[0090] Z1=Z c +[ω*t mse ] × +v*t mse

[0091] Y2=[ω*f*t i ] × *Y c +v*f*t i

[0092] Y1=Y c +[ω*t mse ] × +v*t mse

[0093] Based on Obtain the target projection time difference. The calculation formula of the target projection time difference Δt is:

[0094]

[0095] In the above, ω represents the average angular velocity of the vehicle during the scanning period of the original point cloud, f represents the focal length of the camera, and t i Indicates the unit exposure time difference, Z c represents the Z coordinate of the target lidar point in the second point cloud, v represents the average speed of the vehicle during the scanning period of the original point cloud, and t mse Indicates the preset synchronization time error, Y c Indicates the Y coordinate of the target lidar point in the second point cloud, Y p Indicates the Y coordinate of the target lidar point in the target point cloud, Z p Represents the Z coordinate of the target lidar point in the target point cloud.

[0096] Furthermore, after determining the target projection time difference, the coordinates of each lidar point in the target point cloud can be determined. The calculation formula for the coordinates of each lidar point in the target point cloud is:

[0097] P0=R*P1+v*t i

[0098] Among them, P0 represents the coordinate information of a lidar point in the target point cloud, R represents the rotation matrix of the ego vehicle, P1 represents the coordinate information of the corresponding lidar point in the second point cloud, v represents the average speed of the ego vehicle during the scanning period of the original point cloud, and t i Indicates the unit exposure time difference.

[0099] 204. Based on the preset camera distortion internal parameter and the target projection time difference, perform camera motion distortion addition on the second point cloud to obtain a target point cloud, where the target point cloud includes coordinate information of each lidar point at the target projection position.

[0100] Specifically, step 204 includes: calculating the third ego-vehicle posture change of each lidar point in the second point cloud within the target projection time difference period through the target projection time difference; calculating the target projection position of each lidar point through the third ego-vehicle posture change and the preset camera distortion internal parameter to obtain the target point cloud, where the target projection position is used to indicate the coordinate information of each lidar point in the target point cloud.

[0101] In this embodiment, after the terminal determines the target projection time difference, it can determine the true projection moment of each lidar point through the projection moment of each lidar point in the second point cloud and the target projection time difference, that is, the exposure moment of each lidar point at the target projection position, and then calculate the third self-vehicle posture change between the projection moment and the true projection moment of each lidar point in the second point cloud. Finally, according to the third self-vehicle posture change and the camera distortion internal parameter, the target projection position of each lidar point is determined to obtain the target point cloud, wherein the target projection position is used to indicate the coordinate information of each lidar point in the target point cloud.

[0102] In an embodiment of the present invention, in order to avoid projection errors caused by different distortion removal principles between the lidar point cloud and the camera image, after obtaining the original point cloud scanned by the lidar, only the original point cloud is subjected to motion distortion removal to obtain a first point cloud based on the target reference object coordinate system, and then the first point cloud is converted to the camera coordinate system to obtain a second point cloud in the same coordinate system as the original camera image. Finally, based on the motion distortion principle of the camera, the target projection time difference is predicted, and the same motion distortion as the original camera image is added to the second point cloud through the target projection time difference to obtain the target point cloud, and the target point cloud is projected onto the original camera image to obtain a target point cloud image in which the lidar and camera information are fused, so that the lidar point cloud has the same motion distortion as the camera image. Subsequently, the same image dedistortion principle is used to simultaneously remove motion distortion from the lidar point cloud and the camera image in the target point cloud image. Therefore, the present invention can improve the accuracy of the fusion of lidar and camera information.

[0103] The above describes the projection method of the laser radar point cloud in the embodiment of the present invention. The following describes the projection device of the laser radar point cloud in the embodiment of the present invention. Figure 3 In one embodiment of the present invention, a projection device for a laser radar point cloud includes:

[0104] An acquisition module 301 is configured to acquire an original point cloud scanned by a laser radar and remove motion distortion from the original point cloud to obtain a first point cloud.

[0105] A conversion module 302 is configured to convert the first point cloud into a camera coordinate system based on a target exposure time of the original camera image to obtain a second point cloud;

[0106] The projection module 303 is configured to perform camera motion distortion on the second point cloud to obtain a target point cloud, and project the target point cloud onto the original camera image.

[0107] In an embodiment of the present invention, in order to avoid projection errors caused by different distortion removal principles between the lidar point cloud and the camera image, after obtaining the original point cloud scanned by the lidar, only the original point cloud is subjected to motion distortion removal to obtain a first point cloud based on the target reference object coordinate system, and then the first point cloud is converted to the camera coordinate system to obtain a second point cloud in the same coordinate system as the original camera image. Finally, based on the motion distortion principle of the camera, the same motion distortion as the original camera image is added to the second point cloud to obtain a target point cloud, and the target point cloud is projected onto the original camera image to obtain a target point cloud image in which the lidar and camera information are fused, so that the lidar point cloud has the same motion distortion as the camera image. Subsequently, the same image dedistortion principle is used to simultaneously remove motion distortion from the lidar point cloud and the camera image in the target point cloud image. Therefore, the present invention can improve the accuracy of the fusion of lidar and camera information.

[0108] See also Figure 4 Another embodiment of the projection device for a laser radar point cloud according to the embodiment of the present invention includes:

[0109] An acquisition module 301 is configured to acquire an original point cloud scanned by a laser radar and remove motion distortion from the original point cloud to obtain a first point cloud.

[0110] A conversion module 302 is configured to convert the first point cloud into a camera coordinate system based on a target exposure time of the original camera image to obtain a second point cloud;

[0111] The projection module 303 is configured to perform camera motion distortion on the second point cloud to obtain a target point cloud, and project the target point cloud onto the original camera image.

[0112] Optionally, the acquisition module 301 is specifically configured to:

[0113] Acquire a first extrinsic parameter of the laser radar, where the first extrinsic parameter is used to indicate a relative position parameter between the laser radar and the vehicle coordinate system;

[0114] Obtaining an original point cloud scanned by the laser radar, and determining a starting scanning time of the original point cloud;

[0115] A first ego-vehicle pose change between the start scanning moment and the scanning moment of each lidar point is obtained, and each lidar point in the original point cloud is converted to an ego-vehicle coordinate system using the first ego-vehicle pose change and the first extrinsic parameter to obtain a first point cloud.

[0116] Optionally, the conversion module 302 is specifically configured to:

[0117] Acquire a second extrinsic parameter of the camera, where the second extrinsic parameter is used to indicate a relative position parameter between the camera and the vehicle coordinate system;

[0118] Obtaining a target exposure time of an original camera image captured by the camera, and obtaining a second vehicle posture change between the start scanning time and the target exposure time, wherein the target exposure time is used to indicate the exposure time of a middle row or middle column of the original camera image;

[0119] Each lidar point in the first point cloud is converted to a camera coordinate system using the second ego-vehicle pose change and the second extrinsic parameter to obtain a second point cloud.

[0120] Optionally, the projection module 303 includes:

[0121] a determining unit 3031, configured to determine a target projection time difference, wherein the target projection time difference indicates a difference between an exposure time of each lidar point at a target projection position and the target exposure time;

[0122] The adding unit 3032 is used to perform camera motion distortion addition on the second point cloud based on the preset camera distortion internal parameter and the target projection time difference to obtain a target point cloud, wherein the target point cloud includes the coordinate information of each lidar point at the target projection position.

[0123] Optionally, the determining unit 3031 is specifically configured to:

[0124] Obtaining a unit exposure time difference of the original camera image and a vehicle motion parameter within a scanning period of the original point cloud, wherein the unit exposure time difference is used to indicate a difference in exposure times of adjacent rows or adjacent columns;

[0125] The target projection time difference is calculated by the unit exposure time difference, the vehicle motion parameter, the preset camera distortion internal parameter and the preset synchronization time error.

[0126] Optionally, the adding unit 3032 is specifically configured to:

[0127] Calculating a third vehicle posture change within a target projection time difference period for each lidar point in the second point cloud using the target projection time difference;

[0128] The target projection position of each lidar point is calculated by the third ego-vehicle posture change and the preset camera distortion internal parameter to obtain a target point cloud. The target projection position is used to indicate the coordinate information of each lidar point in the target point cloud.

[0129] Optionally, the calculation formula of the target point cloud is:

[0130] P0=R*P1+v*t i

[0131] Where P0 represents the coordinate information of a lidar point in the target point cloud, R represents the rotation matrix of the ego vehicle, P1 represents the coordinate information of the corresponding lidar point in the second point cloud, v represents the average speed of the ego vehicle during the scanning period of the original point cloud, and t i Represents the unit exposure time difference.

[0132] In an embodiment of the present invention, in order to avoid projection errors caused by different distortion removal principles between the lidar point cloud and the camera image, after obtaining the original point cloud scanned by the lidar, only the original point cloud is subjected to motion distortion removal to obtain a first point cloud based on the target reference object coordinate system, and then the first point cloud is converted to the camera coordinate system to obtain a second point cloud in the same coordinate system as the original camera image. Finally, based on the motion distortion principle of the camera, the target projection time difference is predicted, and the same motion distortion as the original camera image is added to the second point cloud through the target projection time difference to obtain the target point cloud, and the target point cloud is projected onto the original camera image to obtain a target point cloud image in which the lidar and camera information are fused, so that the lidar point cloud has the same motion distortion as the camera image. Subsequently, the same image dedistortion principle is used to simultaneously remove motion distortion from the lidar point cloud and the camera image in the target point cloud image. Therefore, the present invention can improve the accuracy of the fusion of lidar and camera information.

[0133] above Figure 3 and Figure 4 The projection device of the laser radar point cloud in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The projection device of the laser radar point cloud in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0134] Figure 5: is a schematic structural diagram of a laser radar point cloud projection device provided in an embodiment of the present invention. The laser radar point cloud projection device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more massive storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), each module may include a series of computer program operations in the laser radar point cloud projection device 500. Furthermore, the processor 510 can be configured to communicate with the storage medium 530 to execute a series of computer program operations in the storage medium 530 on the laser radar point cloud projection device 500.

[0135] The laser radar point cloud projection device 500 may further include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 5 The structure of the laser radar point cloud projection device shown does not constitute a limitation on the laser radar point cloud projection device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0136] The present invention also provides a computer device, which includes a memory and a processor, wherein a computer-readable computer program is stored in the memory. When the computer-readable computer program is executed by the processor, the processor executes the steps of the laser radar point cloud projection method in the above-mentioned embodiments.

[0137] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program runs on a computer, the computer executes the steps of the laser radar point cloud projection method.

[0138] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several computer programs for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program codes.

[0140] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A laser radar point cloud projection method, characterized in that: The projection method of the laser radar point cloud includes: Obtaining an original point cloud scanned by a laser radar, and removing motion distortion from the original point cloud to obtain a first point cloud; Based on a target exposure time of the original camera image, converting the first point cloud into a camera coordinate system to obtain a second point cloud, wherein the target exposure time is used to indicate the exposure time of a target row or a target column of the original camera image; Adding camera motion distortion to the second point cloud to obtain a target point cloud, and projecting the target point cloud onto the original camera image; Adding camera motion distortion to the second point cloud to obtain a target point cloud includes: Determining a target projection time difference, the target projection time difference being used to indicate a difference between an exposure time of each lidar point at a target projection position and the target exposure time; Based on the preset camera distortion internal parameters and the target projection time difference, the second point cloud is subjected to camera motion distortion addition to obtain a target point cloud, wherein the target point cloud includes coordinate information of each lidar point at the target projection position.

2. The laser radar point cloud projection method according to claim 1, characterized in that: The step of obtaining an original point cloud scanned by a laser radar and removing motion distortion from the original point cloud to obtain a first point cloud includes: Acquire a first extrinsic parameter of the laser radar, where the first extrinsic parameter is used to indicate a relative position parameter between the laser radar and the vehicle coordinate system; Obtaining an original point cloud scanned by the laser radar, and determining a starting scanning time of the original point cloud; A first ego-vehicle pose change between the start scanning moment and the scanning moment of each lidar point is obtained, and each lidar point in the original point cloud is converted to an ego-vehicle coordinate system using the first ego-vehicle pose change and the first extrinsic parameter to obtain a first point cloud.

3. The laser radar point cloud projection method according to claim 2, characterized in that: The step of converting the first point cloud into a camera coordinate system based on the target exposure time of the original camera image to obtain a second point cloud includes: Acquire a second extrinsic parameter of the camera, where the second extrinsic parameter is used to indicate a relative position parameter between the camera and the vehicle coordinate system; Obtaining a target exposure time of an original camera image captured by the camera, and obtaining a second vehicle posture change between the start scanning time and the target exposure time, wherein the target exposure time is used to indicate the exposure time of a middle row or middle column of the original camera image; Each lidar point in the first point cloud is converted to a camera coordinate system using the second ego-vehicle pose change and the second extrinsic parameter to obtain a second point cloud.

4. The laser radar point cloud projection method according to claim 1, characterized in that: Determining the target projection time difference includes: Obtaining a unit exposure time difference of the original camera image and a vehicle motion parameter within a scanning period of the original point cloud, wherein the unit exposure time difference is used to indicate a difference in exposure times between adjacent rows or adjacent columns; The target projection time difference is calculated by the unit exposure time difference, the vehicle motion parameter, the preset camera distortion internal parameter and the preset synchronization time error.

5. The laser radar point cloud projection method according to claim 1, characterized in that: The step of adding camera motion distortion to the second point cloud based on a preset camera distortion internal parameter and the target projection time difference to obtain a target point cloud includes: Calculating a third vehicle posture change within a target projection time difference period for each lidar point in the second point cloud using the target projection time difference; The target projection position of each lidar point is calculated by the third ego-vehicle posture change and the preset camera distortion internal parameter to obtain a target point cloud. The target projection position is used to indicate the coordinate information of each lidar point in the target point cloud.

6. The laser radar point cloud projection method according to claim 4, characterized in that: The calculation formula of the target point cloud is: P0=R*P1+v*t i Where P0 represents the coordinate information of a lidar point in the target point cloud, R represents the rotation matrix of the ego vehicle, P1 represents the coordinate information of the corresponding lidar point in the second point cloud, v represents the average speed of the ego vehicle during the scanning period of the original point cloud, and t i Represents the unit exposure time difference.

7. A laser radar point cloud projection device, characterized in that: The projection device of the laser radar point cloud includes: An acquisition module is used to acquire an original point cloud scanned by the laser radar and remove motion distortion from the original point cloud to obtain a first point cloud; a conversion module, configured to convert the first point cloud into a camera coordinate system based on a target exposure time of the original camera image to obtain a second point cloud, wherein the target exposure time is used to indicate the exposure time of a target row or target column of the original camera image; a projection module, configured to perform camera motion distortion addition on the second point cloud to obtain a target point cloud, and project the target point cloud onto the original camera image; The projection module is also used to: determine a target projection time difference, which is used to indicate the difference between the exposure time of each lidar point at the target projection position and the target exposure time; based on a preset camera distortion internal parameter and the target projection time difference, perform camera motion distortion on the second point cloud to obtain a target point cloud, and project the target point cloud onto the original camera image, wherein the target point cloud includes coordinate information of each lidar point at the target projection position.

8. A laser radar point cloud projection device, characterized in that: The laser radar point cloud projection device includes: a memory and at least one processor, wherein the memory stores a computer program; The at least one processor calls the computer program in the memory so that the laser radar point cloud projection device executes the laser radar point cloud projection method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the laser radar point cloud projection method according to any one of claims 1 to 6 is implemented.

Images