Laser radar point cloud projection method, device, equipment and storage medium

Abstract

The present invention relates to the field of autonomous driving technology, and discloses a laser radar point cloud projection method, device, equipment, and storage medium for improving the effect of laser radar point cloud projection onto camera images. The laser radar point cloud projection method includes: obtaining an original point cloud, original point cloud motion parameters, and original vehicle motion parameters scanned by the laser radar, and performing standardization processing on the original point cloud, original point cloud motion parameters, and original vehicle motion parameters in a unified coordinate system to obtain a target point cloud, target point cloud motion parameters, and target vehicle motion parameters; performing camera coordinate system conversion on the target point cloud at different exposure times to obtain a first point cloud at the first exposure time and a second point cloud at the second exposure time; projecting the first point cloud and the second point cloud onto the original camera image to obtain a first image point cloud and a second image point cloud, and interpolating the first image point cloud and the second image point cloud to obtain a target image point cloud.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a laser radar point cloud projection method, device, equipment and storage medium. Background Art

[0002] With the development of autonomous driving technology, the lidar and cameras installed on autonomous vehicles can improve their respective perception capabilities through information fusion. Therefore, accurately projecting the lidar point cloud onto the camera image is the basis for accurate fusion of the two information.

[0003] Existing technologies for projecting lidar point clouds onto camera images typically dedistort both or one of them before projection. This approach performs poorly in scenarios where both the vehicle and environmental obstacles are moving. Summary of the Invention

[0004] The present invention provides a laser radar point cloud projection method, device, equipment and storage medium for improving the effect of projecting the laser radar point cloud onto a camera image.

[0005] A first aspect of the present invention provides a laser radar point cloud projection method, comprising:

[0006] Obtaining an original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters scanned by the laser radar, and performing standardization processing on the original point cloud, the original point cloud motion parameters, and the original ego vehicle motion parameters in a unified coordinate system to obtain a target point cloud, target point cloud motion parameters, and target ego vehicle motion parameters;

[0007] Based on the target point cloud motion parameters and the target vehicle motion parameters, performing camera coordinate system transformation on the target point cloud at different exposure times to obtain a first point cloud at a first exposure time and a second point cloud at a second exposure time;

[0008] The first point cloud and the second point cloud are respectively projected onto the original camera image to obtain a first image point cloud and a second image point cloud, and the first image point cloud and the second image point cloud are interpolated to obtain a target image point cloud.

[0009] Optionally, projecting the first point cloud and the second point cloud onto an original camera image to obtain a first image point cloud and a second image point cloud, and interpolating the first image point cloud and the second image point cloud to obtain a target image point cloud includes:

[0010] Based on preset camera parameters, projecting the first point cloud and the second point cloud onto an original camera image to obtain a first image point cloud and a second image point cloud;

[0011] Connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain a straight line equation, and solving the straight line equation based on the proportion of the target projection moment within the camera exposure period to obtain an interpolated value for each lidar point, where the interpolated value indicates the coordinate corresponding to the projection of the lidar point into the original camera image;

[0012] Generate target image point cloud by interpolation of all lidar points.

[0013] Optionally, connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain a straight line equation, and solving the straight line equation based on a proportion of the target projection moment in the camera exposure period to obtain an interpolated value of each lidar point includes:

[0014] Connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain a straight line equation, and constructing a target interpolation equation based on the straight line equation and the proportion of the target projection moment within the camera exposure period;

[0015] The target interpolation equation is interpolated and solved for each laser radar point to obtain the interpolation value of each laser radar point.

[0016] Optionally, the target interpolation equation includes:

[0017]

[0018] Among them, (x, y) represents the coordinates of a lidar point in the target image point cloud, (x1, y1) represents the coordinates of the corresponding lidar point in the first image point cloud, (x2, y2) represents the coordinates of the corresponding lidar point in the second image point cloud, and h represents the image height of the target image point cloud.

[0019] Optionally, the step of acquiring an original point cloud, original point cloud motion parameters, and original ego-vehicle motion parameters scanned by a laser radar, and performing standardization processing on the original point cloud, the original point cloud motion parameters, and the original ego-vehicle motion parameters in a unified coordinate system to obtain a target point cloud, target point cloud motion parameters, and target ego-vehicle motion parameters includes:

[0020] Obtaining the original point cloud scanned by the laser radar, the original point cloud motion parameters, and the original vehicle motion parameters, wherein the original point cloud motion parameters include the motion speed of each laser radar point, and the original vehicle motion parameters include the vehicle posture and vehicle speed;

[0021] Performing a world coordinate system conversion on the original point cloud and the original point cloud motion parameters to obtain a target point cloud and the target point cloud motion parameters;

[0022] Based on the conversion relationship between the ego-vehicle coordinate system and the world coordinate system, the ego-vehicle posture and the ego-vehicle speed are converted into the world coordinate system to obtain target ego-vehicle motion parameters.

[0023] Optionally, performing a world coordinate system conversion on the original point cloud and the original point cloud motion parameters to obtain a target point cloud and target point cloud motion parameters includes:

[0024] Based on the conversion relationship between the laser radar coordinate system and the vehicle coordinate system, the original point cloud and the original point cloud motion parameters are converted into the vehicle coordinate system to obtain a third point cloud and the first point cloud motion parameters;

[0025] Based on the conversion relationship between the vehicle coordinate system and the world coordinate system, the third point cloud and the motion parameters of the first point cloud are converted into the world coordinate system to obtain the target point cloud and the target point cloud motion parameters.

[0026] Optionally, performing camera coordinate system conversion on the target point cloud at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters to obtain a first point cloud at a first exposure time and a second point cloud at a second exposure time includes:

[0027] Determining a first exposure time and a second exposure time, wherein the first exposure time is used to indicate a start exposure time of the original camera image, and the second exposure time is used to indicate an end exposure time of the original camera image;

[0028] Based on the target point cloud motion parameters and the target vehicle motion parameters at the first exposure time, projecting the target point cloud into the camera coordinate system to obtain a first point cloud at the first exposure time;

[0029] Based on the target point cloud motion parameters and the target vehicle motion parameters at the second exposure moment, the target point cloud is projected into the camera coordinate system to obtain a second point cloud at the second exposure moment.

[0030] A second aspect of the present invention provides a laser radar point cloud projection device, comprising:

[0031] an acquisition module, configured to acquire an original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters scanned by the laser radar, and to perform standardization processing on the original point cloud, the original point cloud motion parameters, and the original ego vehicle motion parameters in a unified coordinate system to obtain a target point cloud, target point cloud motion parameters, and target ego vehicle motion parameters;

[0032] A conversion module is configured to perform camera coordinate system conversion of the target point cloud at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters, to obtain a first point cloud at a first exposure time and a second point cloud at a second exposure time;

[0033] An interpolation module is used to project the first point cloud and the second point cloud onto the original camera image respectively to obtain a first image point cloud and a second image point cloud, and to interpolate the first image point cloud and the second image point cloud to obtain a target image point cloud.

[0034] Optionally, the interpolation module includes:

[0035] A projection unit, configured to project the first point cloud and the second point cloud onto an original camera image based on preset camera parameters, to obtain a first image point cloud and a second image point cloud;

[0036] a solving unit, configured to connect the same lidar point in the first image point cloud and the second image point cloud by a straight line to obtain a straight line equation, and solve the straight line equation based on a proportion of the target projection moment within the camera exposure period to obtain an interpolated value for each lidar point, the interpolated value being used to indicate the coordinates corresponding to the projection of the lidar point into the original camera image;

[0037] The generation unit is used to generate the target image point cloud by interpolating all lidar points.

[0038] Optionally, the solving unit is specifically configured to:

[0039] Connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain a straight line equation, and constructing a target interpolation equation based on the straight line equation and the proportion of the target projection moment within the camera exposure period;

[0040] The target interpolation equation is interpolated and solved for each laser radar point to obtain the interpolation value of each laser radar point.

[0041] Optionally, the target interpolation equation includes:

[0042]

[0043] Among them, (x, y) represents the coordinates of a lidar point in the target image point cloud, (x1, y1) represents the coordinates of the corresponding lidar point in the first image point cloud, (x2, y2) represents the coordinates of the corresponding lidar point in the second image point cloud, and h represents the image height of the target image point cloud.

[0044] Optionally, the acquisition module includes:

[0045] a parameter acquisition unit, configured to acquire an original point cloud scanned by the laser radar, original point cloud motion parameters, and original ego-vehicle motion parameters, wherein the original point cloud motion parameters include the motion speed of each laser radar point, and the original ego-vehicle motion parameters include the ego-vehicle position and velocity;

[0046] A first conversion unit is configured to perform a world coordinate system conversion on the original point cloud and the original point cloud motion parameters to obtain a target point cloud and the target point cloud motion parameters;

[0047] The second conversion unit is configured to perform a world coordinate system conversion on the ego-vehicle posture and the ego-vehicle speed based on a conversion relationship between the ego-vehicle coordinate system and the world coordinate system to obtain target ego-vehicle motion parameters.

[0048] Optionally, the first conversion unit is specifically configured to:

[0049] Based on the conversion relationship between the laser radar coordinate system and the vehicle coordinate system, the original point cloud and the original point cloud motion parameters are converted into the vehicle coordinate system to obtain a third point cloud and the first point cloud motion parameters;

[0050] Based on the conversion relationship between the vehicle coordinate system and the world coordinate system, the third point cloud and the motion parameters of the first point cloud are converted into the world coordinate system to obtain the target point cloud and the target point cloud motion parameters.

[0051] Optionally, the conversion module is specifically configured to:

[0052] Determining a first exposure time and a second exposure time, wherein the first exposure time is used to indicate a start exposure time of the original camera image, and the second exposure time is used to indicate an end exposure time of the original camera image;

[0053] Projecting the target point cloud into a camera coordinate system based on the target point cloud motion parameters and the target vehicle motion parameters at the first exposure time to obtain a first point cloud at the first exposure time;

[0054] Based on the target point cloud motion parameters and the target vehicle motion parameters at the second exposure moment, the target point cloud is projected into the camera coordinate system to obtain a second point cloud at the second exposure moment.

[0055] The third aspect of the present invention provides a laser radar point cloud projection device, comprising: a memory and at least one processor, wherein a computer program is stored in the memory; the at least one processor calls the computer program in the memory so that the laser radar point cloud projection device executes the above-mentioned laser radar point cloud projection method.

[0056] A fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned laser radar point cloud projection method.

[0057] In the technical solution provided by the present invention, an original point cloud, original point cloud motion parameters, and original ego-vehicle motion parameters of a laser radar scan are obtained, and the original point cloud, the original point cloud motion parameters, and the original ego-vehicle motion parameters are respectively standardized in a unified coordinate system to obtain a target point cloud, target point cloud motion parameters, and target ego-vehicle motion parameters; based on the target point cloud motion parameters and the target ego-vehicle motion parameters, the target point cloud is transformed into a camera coordinate system at different exposure times to obtain a first point cloud at a first exposure time and a second point cloud at a second exposure time; the first point cloud and the second point cloud are respectively projected onto an original camera image to obtain a first image point cloud and a second image point cloud, and the first image point cloud and the second image point cloud are interpolated to obtain a target image point cloud. In the embodiment of the present invention, since the information collected by different sensors is all based on their own coordinate systems, by standardizing the acquired original point clouds, original point cloud motion parameters, and original ego-vehicle motion parameters in a unified coordinate system, the information collected by different sensors can be operated on. By converting the target point cloud into a camera coordinate system at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters and projecting it onto the original camera image to be fused, the influence of the movement of environmental objects on the ego-vehicle motion can be eliminated, thereby improving the projection effect. By interpolating the first image point cloud and the second image point cloud based on the proportion of different exposure times in the total exposure times, the interpolation is fast and accurate, thereby improving the effect of the laser radar point cloud being projected onto the camera image. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram of an embodiment of a projection method of a laser radar point cloud according to an embodiment of the present invention;

[0059] Figure 2 Schematic diagram of another embodiment of a laser radar point cloud projection method according to an embodiment of the present invention;

[0060] Figure 3 Schematic diagram of an embodiment of a projection device for a laser radar point cloud according to an embodiment of the present invention;

[0061] Figure 4 Schematic diagram of another embodiment of a projection device for a laser radar point cloud according to an embodiment of the present invention;

[0062] Figure 5 Schematic diagram of an embodiment of a projection device for a laser radar point cloud in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] Embodiments of the present invention provide a laser radar point cloud projection method, apparatus, device, and storage medium for improving the effect of projecting a laser radar point cloud onto a camera image.

[0064] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0065] It is understandable that the execution subject of the present invention can be a projection device of the laser radar point cloud, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0066] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the laser radar point cloud projection method in an embodiment of the present invention includes:

[0067] 101. Obtaining the original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters scanned by the laser radar, and performing standardization processing on the original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters in a unified coordinate system to obtain the target point cloud, target point cloud motion parameters, and target ego vehicle motion parameters;

[0068] It should be noted that the raw point cloud scanned by the LiDAR indicates the point cloud scanned by the LiDAR within a scanning cycle. For example, a scanning cycle of a 360-degree rotating LiDAR is one rotation. The raw point cloud includes the coordinate information of each LiDAR point. The raw point cloud motion parameters indicate the motion parameters of each LiDAR point in the raw point cloud, including motion parameters such as the speed and direction of each LiDAR point. In one embodiment, the raw point cloud also includes the motion parameters of each LiDAR point, that is, the raw point cloud includes the raw point cloud motion parameters. For example, for each LiDAR point output by a frequency modulated continuous wave (FMCW) LiDAR, the speed and direction of each LiDAR point are included. The raw point cloud motion parameters can describe the motion parameters of environmental moving objects, such as the motion parameters of vehicle obstacles. In contrast to the raw vehicle motion parameters, the raw vehicle motion parameters can describe the motion parameters of the vehicle itself. This allows the subsequent projection of the LiDAR point cloud by combining the point cloud motion parameters with the vehicle's motion speed to eliminate the impact of environmental moving objects on the LiDAR projection, thereby improving the projection effect of the LiDAR point cloud.

[0069] It can be understood that the original data collected by different sensors are all established based on their own coordinate systems, among which the original point cloud and the original point cloud motion parameters are data based on the lidar coordinate system, and the original ego-vehicle motion parameters are data based on the ego-vehicle coordinate system. Therefore, in order to make these data in the same coordinate system and be able to perform operations of the same dimension, the original point cloud, the original point cloud motion parameters and the original ego-vehicle motion parameters are respectively converted to a unified coordinate system to obtain the target point cloud corresponding to the original point cloud, the target point cloud motion parameters corresponding to the original point cloud motion parameters, and the target ego-vehicle motion parameters corresponding to the original ego-vehicle motion parameters. Among them, the target point cloud, the target point cloud motion parameters and the target ego-vehicle motion parameters are all data based on the target coordinate system. The target coordinate system can be the ego-vehicle coordinate system, the world coordinate system, the bird's-eye view camera coordinate system, etc., which is not limited here. Specifically, in one embodiment, in order to reduce the data loss caused by the coordinate system conversion operation, the original point cloud and the original point cloud motion parameters are converted to the self-vehicle coordinate system with the self-vehicle coordinate system as the same coordinate system to obtain the target point cloud and the target point cloud motion parameters. Since the original self-vehicle motion parameters are already data based on the self-vehicle coordinate system, the original self-vehicle motion parameters can be determined as the target self-vehicle motion parameters, thereby obtaining standardized data of a unified coordinate system, thereby improving the accuracy of the lidar point cloud projection.

[0070] In one embodiment, the original point cloud motion parameters can also be obtained by motion parameter prediction through machine learning algorithms / models, such as self-supervised point cloud motion estimation models, normal distribution transform (NDT) algorithms, etc., which are not limited here.

[0071] 102. Based on the target point cloud motion parameters and the target ego-vehicle motion parameters, perform camera coordinate system transformation on the target point cloud at different exposure times to obtain a first point cloud at the first exposure time and a second point cloud at the second exposure time;

[0072] It should be noted that, to improve the effect and fusion of the LiDAR point cloud projection onto the camera image and provide point cloud information at different times for the subsequent interpolation process, the target point cloud is transformed into the camera coordinate system at different exposure times based on the target point cloud motion parameters and the target vehicle motion parameters, thereby obtaining a first point cloud at a first exposure time and a second point cloud at a second exposure time. The first exposure time can be any exposure time of the original camera image, such as the starting exposure time, the middle exposure time, and the ending exposure time. For cameras that scan in a row-by-row exposure manner, such as rolling shutter cameras, the first exposure time can be the exposure time of any row of the original camera image, such as the exposure time of the starting row, the exposure time of the middle row, and the exposure time of the last row. Similarly, the second exposure time can be any exposure time of the original camera image that is different from the first exposure time. The difference between the first and second exposure times determines the interpolation equations subsequently constructed but does not affect the accuracy of the LiDAR point cloud projection. Therefore, the projection time is not specifically limited here.

[0073] In one embodiment, after obtaining the target point cloud, target point cloud motion parameters, and target ego-vehicle motion parameters in the target coordinate system based on the normalization processing performed in step 101, calibration parameters (i.e., extrinsic parameters) and sensor time offset values ​​between the target coordinate system and the camera coordinate system are obtained. Using the calibration parameters and sensor time offset values, each lidar point in the target point cloud is converted to the camera coordinate system at different exposure times based on the target point cloud motion parameters and target ego-vehicle motion parameters, resulting in a first point cloud at the first exposure time and a second point cloud at the second exposure time. This embodiment can accurately project the point cloud into the camera coordinate system by combining the calibration parameters and time offset between sensors, thereby improving the accuracy of lidar point cloud projection.

[0074] 103. Project the first point cloud and the second point cloud onto the original camera image respectively to obtain a first image point cloud and a second image point cloud, and interpolate the first image point cloud and the second image point cloud to obtain a target image point cloud.

[0075] It should be noted that the original camera image is an image taken by the original camera to be fused with the original point cloud. The process of projecting the first point cloud and the second point cloud onto the original camera image is the same as the principle and process of projecting the image by the original camera, and will not be repeated here. Projecting the point cloud based on the principle of projecting the image inside the camera can retain the projection effect to the greatest extent and ensure the consistency of the fusion of the point cloud and the image. It is understandable that the first point cloud and the second point cloud can be projected onto a single original camera image to obtain a single image point cloud, wherein the single image point cloud includes the first point cloud, the second point cloud and the original camera image, or the first point cloud and the second point cloud can be projected onto the same original camera image to obtain the first image point cloud and the second image point cloud, wherein the first image point cloud includes the first point cloud and the original camera image, and the second image point cloud includes the second point cloud and the original camera image. It should be noted that the first image point cloud includes each lidar point in the first point cloud, the second image point cloud includes each lidar point in the second point cloud, and the first image point cloud and the second image point cloud both include each lidar point in the original point cloud, and they are in a one-to-one correspondence. The corresponding lidar points have different coordinate information in different point clouds. For example, there is a point M(m1,m2) in the original point cloud, which is A(a1,a1) in the first point cloud, B(b1,b1) in the first image point cloud, A'(a2,a2) in the second point cloud, and B'(b2,b2) in the second image point cloud. The specific details are not limited here.

[0076] In one embodiment, a target interpolation equation is constructed based on the ratio of the actual projection time of the interpolation point to the time span from the first exposure time to the second exposure time. The first and second image point clouds are interpolated according to the target interpolation equation to obtain the target image point cloud. Specifically, the corresponding lidar points in the first and second image point clouds are segmented proportionally based on the ratio of the actual projection time to the time span from the first exposure time to the second exposure time, obtaining segmentation points. The target interpolation equation is then constructed based on the segmentation points and the ratio. Finally, the interpolation and actual projection time of the same lidar point in the first and second image point clouds are calculated according to the target interpolation equation to obtain the target image point cloud. For example, assuming the first exposure time is the starting exposure time and the second exposure time is the middle exposure time, it can be seen that the time span from the first exposure time to the second exposure time is 1 / 2 of the total exposure time of the original camera image. Therefore, the corresponding lidar points in the first and second image point clouds are segmented proportionally based on the ratio of 1 / 2 to obtain 1 / 2 segmentation points. The target interpolation equation is then constructed based on the segmentation points and the 1 / 2 ratio. This embodiment can construct an interpolation equation based on the proportion of the actual projection moment to the total camera exposure time, thereby improving the accuracy of the interpolation and thus improving the effect of the lidar projection.

[0077] In the embodiment of the present invention, since the information collected by different sensors is all based on their own coordinate systems, by standardizing the acquired original point clouds, original point cloud motion parameters, and original ego-vehicle motion parameters in a unified coordinate system, the information collected by different sensors can be operated on. By converting the target point cloud into a camera coordinate system at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters and projecting it onto the original camera image to be fused, the influence of the movement of environmental objects on the ego-vehicle motion can be eliminated, thereby improving the projection effect. By interpolating the first image point cloud and the second image point cloud based on the proportion of different exposure times in the total exposure times, the interpolation is fast and accurate, thereby improving the effect of the laser radar point cloud being projected onto the camera image.

[0078] See also Figure 2 Another embodiment of the laser radar point cloud projection method in the embodiment of the present invention includes:

[0079] 201. Obtaining the original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters scanned by the laser radar, and performing standardization processing on the original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters in a unified coordinate system to obtain the target point cloud, target point cloud motion parameters, and target ego vehicle motion parameters;

[0080] Specifically, step 201 includes: obtaining the original point cloud, original point cloud motion parameters, and original ego-vehicle motion parameters scanned by the laser radar, where the original point cloud motion parameters include the motion speed of each laser radar point, and the original ego-vehicle motion parameters include the ego-vehicle posture and ego-vehicle speed; performing a world coordinate system conversion on the original point cloud and the original point cloud motion parameters to obtain a target point cloud and target point cloud motion parameters; and performing a world coordinate system conversion on the ego-vehicle posture and ego-vehicle speed based on the conversion relationship between the ego-vehicle coordinate system and the world coordinate system to obtain the target ego-vehicle motion parameters.

[0081] In this embodiment, in order to improve the accuracy of the unified coordinate system conversion processing, the world coordinate system is used as the unified target coordinate system. Specifically, after obtaining the original point cloud, original point cloud motion parameters, and original ego-vehicle motion parameters of the laser radar scan, the coordinates of each laser radar point in the original point cloud and the motion speed and motion direction of each laser radar point in the original point cloud motion parameters are converted to the world coordinate system to obtain the target point cloud and target point cloud motion parameters, wherein the target point cloud includes the coordinates of each laser radar point in the world coordinate system, and the target point cloud motion parameters include the target point cloud motion speed and target point cloud motion direction in the world coordinate system. Based on the conversion relationship between the ego-vehicle coordinate system and the world coordinate system (such as the global satellite positioning system network), the ego-vehicle posture, ego-vehicle speed, and ego-vehicle motion direction in the original ego-vehicle motion parameters are converted to the world coordinate system to obtain the target ego-vehicle motion parameters, wherein the target ego-vehicle motion parameters include the target ego-vehicle posture, target ego-vehicle speed, and target ego-vehicle motion direction.

[0082] Furthermore, the original point cloud and the original point cloud motion parameters are converted to the world coordinate system to obtain the target point cloud and the target point cloud motion parameters, including: based on the conversion relationship between the lidar coordinate system and the vehicle coordinate system, the original point cloud and the original point cloud motion parameters are converted to the vehicle coordinate system to obtain the third point cloud and the first point cloud motion parameters; based on the conversion relationship between the vehicle coordinate system and the world coordinate system, the third point cloud and the first point cloud motion parameters are converted to the world coordinate system to obtain the target point cloud and the target point cloud motion parameters.

[0083] It is understandable that since the conversion relationship between the lidar coordinate system and the world coordinate system is difficult to determine, in order to improve the accuracy of the point cloud conversion to the world coordinate system, the original point cloud and the original point cloud motion parameters are first converted to the vehicle coordinate system to obtain the third point cloud and the first point cloud motion parameters in the vehicle coordinate system. Then, based on the conversion relationship between the vehicle coordinate system and the world coordinate system, the third point cloud and the first point cloud motion parameters in the vehicle coordinate system are converted to the world coordinate system to obtain the target point cloud and the target point cloud motion parameters.

[0084] 202. Based on the target point cloud motion parameters and the target ego-vehicle motion parameters, perform camera coordinate system transformation on the target point cloud at different exposure times to obtain a first point cloud at the first exposure time and a second point cloud at the second exposure time;

[0085] Specifically, step 202 includes: determining a first exposure time and a second exposure time, where the first exposure time is used to indicate the start exposure time of the original camera image, and the second exposure time is used to indicate the end exposure time of the original camera image; projecting the target point cloud onto the camera coordinate system based on the target point cloud motion parameters and the target ego-vehicle motion parameters at the first exposure time to obtain a first point cloud at the first exposure time; and projecting the target point cloud onto the camera coordinate system based on the target point cloud motion parameters and the target ego-vehicle motion parameters at the second exposure time to obtain a second point cloud at the second exposure time.

[0086] In this embodiment, in order to accurately obtain point cloud data at different exposure times and improve the accuracy of subsequent interpolation calculations, the first exposure time is first determined as the starting exposure time of the original camera image, and the second exposure time is determined as the ending exposure time of the original camera image. Then, based on the target point cloud motion parameters and the target ego-vehicle motion parameters at the starting exposure time, the target point cloud is projected to the camera coordinate system to obtain the first point cloud at the starting exposure time. Based on the target point cloud motion parameters and the target ego-vehicle motion parameters at the ending exposure time, the target point cloud is projected to the camera coordinate system to obtain the second point cloud at the ending exposure time.

[0087] 203. Based on preset camera parameters, project the first point cloud and the second point cloud onto the original camera image to obtain a first image point cloud and a second image point cloud;

[0088] In this embodiment, the preset camera parameters include the camera's focal length, image size, aperture and other parameters required for the image projection process. The principle and process of projecting the first point cloud and the second point cloud onto the original camera image based on the preset camera parameters are the same as the camera projection image, and will not be repeated here.

[0089] 204. Connect the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain a straight line equation. Solve the straight line equation based on the proportion of the target projection moment within the camera exposure period to obtain an interpolated value for each lidar point. The interpolated value is used to indicate the coordinate corresponding to the projection of the lidar point in the original camera image.

[0090] Specifically, step 204 includes: connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain the equation of the straight line, and constructing a target interpolation equation based on the straight line equation and the proportion of the target projection moment in the camera exposure period; interpolating and solving the target interpolation equation for each lidar point to obtain the interpolation value of each lidar point.

[0091] It can be understood that interpolation is an approximate calculation method that approximates unknown points from known points. The process involves constructing a polynomial function, passing the constructed polynomial function through all known points, and then obtaining the predicted unknown point using the polynomial function. In this embodiment, the same LiDAR point in the first and second image point clouds is a known point. The goal is to accurately predict the unknown point in the target image point cloud using these two known points, so that the target image point cloud and the original camera image are highly integrated. Based on this, the terminal first connects the same LiDAR point in the first and second image point clouds with a straight line to obtain the line equation. The terminal then constructs a target interpolation equation based on the line equation and the proportion of the target projection moment within the camera exposure period. The target projection moment indicates the actual projection moment of the LiDAR point. Finally, the terminal interpolates and solves the target interpolation equation for each LiDAR point to obtain the interpolated value of each LiDAR point. This interpolation indicates the corresponding coordinate of the LiDAR point projected into the original camera image, i.e., the coordinate of the LiDAR point in the target image point cloud.

[0092] In this embodiment, the terminal connects the same lidar point in the first image point cloud and the second image point cloud with a straight line, and the resulting straight line equation is:

[0093]

[0094] Furthermore, since the above straight line equation can be used to indicate the cutting ratio of the point (x, y) to the straight line from the point (x1, y1) to the point (x2, y2), and since the cutting ratio is the same as the ratio of the target projection time to the total camera exposure time, the ratio of the target projection time to the total camera exposure time can be expressed as:

[0095]

[0096] Therefore, the constructed target interpolation equation includes:

[0097]

[0098] Where (x, y) represents the coordinates of a lidar point in the target image point cloud, (x1, y1) represents the coordinates of the corresponding lidar point in the first image point cloud, (x2, y2) represents the coordinates of the corresponding lidar point in the second image point cloud, and h represents the image height of the target image point cloud.

[0099] In this embodiment, based on the principle that the ratio of the actual projection moment to the total exposure time of the camera is the same as the cutting ratio of the projection position straight line of the actual projection position to the total exposure time, an interpolation equation is cleverly constructed to make the calculation of the laser radar projection point both fast and accurate, thereby improving the projection effect of the laser radar.

[0100] 205. Generate target image point cloud by interpolating all lidar points.

[0101] In this embodiment, the point (x, y) obtained based on 204 is the interpolation of the lidar point, which is equivalent to the target projection position of the lidar point. Therefore, by interpolating all lidar points, the target projection positions of all lidar points can be determined, thereby generating a target image point cloud. The target image point cloud is used to indicate the image point cloud information of the fusion of the original point cloud and the original camera image.

[0102] In the embodiment of the present invention, since the information collected by different sensors is all based on their own coordinate systems, the information collected by different sensors can be made operable by standardizing the acquired original point clouds, original point cloud motion parameters, and original ego-vehicle motion parameters in a unified coordinate system. By converting the target point cloud into a camera coordinate system at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters and projecting it onto the original camera image to be fused, the influence of the movement of environmental objects on the ego-vehicle motion can be eliminated, thereby improving the projection effect. By connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line based on the proportion of different exposure times in the total exposure times, the straight line equation is obtained, and the straight line equation is solved to obtain the interpolation value of each lidar point, thereby reducing the complexity of the interpolation and improving the interpolation efficiency, thereby improving the effect of the lidar point cloud projected onto the camera image.

[0103] The above describes the projection method of the laser radar point cloud in the embodiment of the present invention. The following describes the projection device of the laser radar point cloud in the embodiment of the present invention. Figure 3 In one embodiment of the present invention, a projection device for a laser radar point cloud includes:

[0104] An acquisition module 301 is configured to acquire an original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters scanned by the laser radar, and to perform normalization processing on the original point cloud, the original point cloud motion parameters, and the original ego vehicle motion parameters in a unified coordinate system to obtain a target point cloud, target point cloud motion parameters, and target ego vehicle motion parameters.

[0105] A conversion module 302 is configured to perform camera coordinate system conversion on the target point cloud at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters, to obtain a first point cloud at a first exposure time and a second point cloud at a second exposure time;

[0106] The interpolation module 303 is used to project the first point cloud and the second point cloud onto the original camera image respectively to obtain a first image point cloud and a second image point cloud, and interpolate the first image point cloud and the second image point cloud to obtain a target image point cloud.

[0107] In the embodiment of the present invention, since the information collected by different sensors is all based on their own coordinate systems, by standardizing the acquired original point clouds, original point cloud motion parameters, and original ego-vehicle motion parameters in a unified coordinate system, the information collected by different sensors can be operated on. By converting the target point cloud into a camera coordinate system at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters and projecting it onto the original camera image to be fused, the influence of the movement of environmental objects on the ego-vehicle motion can be eliminated, thereby improving the projection effect. By interpolating the first image point cloud and the second image point cloud based on the proportion of different exposure times in the total exposure times, the interpolation is fast and accurate, thereby improving the effect of the laser radar point cloud being projected onto the camera image.

[0108] See also Figure 4 Another embodiment of the projection device for a laser radar point cloud according to the embodiment of the present invention includes:

[0109] An acquisition module 301 is configured to acquire an original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters scanned by the laser radar, and to perform normalization processing on the original point cloud, the original point cloud motion parameters, and the original ego vehicle motion parameters in a unified coordinate system to obtain a target point cloud, target point cloud motion parameters, and target ego vehicle motion parameters.

[0110] A conversion module 302 is configured to perform camera coordinate system conversion on the target point cloud at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters, to obtain a first point cloud at a first exposure time and a second point cloud at a second exposure time;

[0111] The interpolation module 303 is used to project the first point cloud and the second point cloud onto the original camera image respectively to obtain a first image point cloud and a second image point cloud, and interpolate the first image point cloud and the second image point cloud to obtain a target image point cloud.

[0112] Optionally, the interpolation module 303 includes:

[0113] A projection unit 3031 is configured to project the first point cloud and the second point cloud onto an original camera image based on preset camera parameters to obtain a first image point cloud and a second image point cloud;

[0114] A solving unit 3032 is configured to connect the same lidar point in the first image point cloud and the second image point cloud by a straight line to obtain a straight line equation, and solve the straight line equation based on the proportion of the target projection moment within the camera exposure period to obtain an interpolated value for each lidar point, where the interpolated value indicates the coordinate corresponding to the projection of the lidar point into the original camera image;

[0115] The generating unit 3033 is used to generate a target image point cloud by interpolating all the lidar points.

[0116] Optionally, the solving unit 3032 is specifically configured to:

[0117] Connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain a straight line equation, and constructing a target interpolation equation based on the straight line equation and the proportion of the target projection moment within the camera exposure period;

[0118] The target interpolation equation is interpolated and solved for each laser radar point to obtain the interpolation value of each laser radar point.

[0119] Optionally, the target interpolation equation includes:

[0120]

[0121] Among them, (x, y) represents the coordinates of a lidar point in the target image point cloud, (x1, y1) represents the coordinates of the corresponding lidar point in the first image point cloud, (x2, y2) represents the coordinates of the corresponding lidar point in the second image point cloud, and h represents the image height of the target image point cloud.

[0122] Optionally, the acquisition module 301 includes:

[0123] A parameter acquisition unit 3011 is configured to acquire an original point cloud scanned by the laser radar, original point cloud motion parameters, and original ego vehicle motion parameters. The original point cloud motion parameters include the motion speed of each laser radar point, and the original ego vehicle motion parameters include the ego vehicle posture and ego vehicle speed.

[0124] A first conversion unit 3012 is configured to perform a world coordinate system conversion on the original point cloud and the original point cloud motion parameters to obtain a target point cloud and target point cloud motion parameters;

[0125] The second conversion unit 3013 is configured to perform a world coordinate system conversion on the ego-vehicle posture and the ego-vehicle velocity based on a conversion relationship between the ego-vehicle coordinate system and the world coordinate system to obtain target ego-vehicle motion parameters.

[0126] Optionally, the first conversion unit 3012 is specifically configured to:

[0127] Based on the conversion relationship between the laser radar coordinate system and the vehicle coordinate system, the original point cloud and the original point cloud motion parameters are converted into the vehicle coordinate system to obtain a third point cloud and the first point cloud motion parameters;

[0128] Based on the conversion relationship between the vehicle coordinate system and the world coordinate system, the third point cloud and the motion parameters of the first point cloud are converted into the world coordinate system to obtain the target point cloud and the target point cloud motion parameters.

[0129] Optionally, the conversion module 302 is specifically configured to:

[0130] Determining a first exposure time and a second exposure time, wherein the first exposure time is used to indicate a start exposure time of the original camera image, and the second exposure time is used to indicate an end exposure time of the original camera image;

[0131] Based on the target point cloud motion parameters and the target vehicle motion parameters at the first exposure time, projecting the target point cloud into the camera coordinate system to obtain a first point cloud at the first exposure time;

[0132] Based on the target point cloud motion parameters and the target vehicle motion parameters at the second exposure moment, the target point cloud is projected into the camera coordinate system to obtain a second point cloud at the second exposure moment.

[0133] In the embodiment of the present invention, since the information collected by different sensors is all based on their own coordinate systems, the information collected by different sensors can be made operable by standardizing the acquired original point clouds, original point cloud motion parameters, and original ego-vehicle motion parameters in a unified coordinate system. By converting the target point cloud into a camera coordinate system at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters and projecting it onto the original camera image to be fused, the influence of the movement of environmental objects on the ego-vehicle motion can be eliminated, thereby improving the projection effect. By connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line based on the proportion of different exposure times in the total exposure times, the straight line equation is obtained, and the straight line equation is solved to obtain the interpolation value of each lidar point, thereby reducing the complexity of the interpolation and improving the interpolation efficiency, thereby improving the effect of the lidar point cloud projected onto the camera image.

[0134] above Figure 3 and Figure 4 The projection device of the laser radar point cloud in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The projection device of the laser radar point cloud in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0135] Figure 5: is a schematic structural diagram of a laser radar point cloud projection device provided in an embodiment of the present invention. The laser radar point cloud projection device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more massive storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), each module may include a series of computer program operations in the laser radar point cloud projection device 500. Furthermore, the processor 510 can be configured to communicate with the storage medium 530 to execute a series of computer program operations in the storage medium 530 on the laser radar point cloud projection device 500.

[0136] The laser radar point cloud projection device 500 may further include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 5 The structure of the laser radar point cloud projection device shown does not constitute a limitation on the laser radar point cloud projection device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0137] The present invention also provides a computer device, which includes a memory and a processor, wherein a computer-readable computer program is stored in the memory. When the computer-readable computer program is executed by the processor, the processor executes the steps of the laser radar point cloud projection method in the above-mentioned embodiments.

[0138] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program runs on a computer, the computer executes the steps of the laser radar point cloud projection method.

[0139] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several computer programs for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store program code.

[0141] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A laser radar point cloud projection method, characterized in that: The projection method of the laser radar point cloud includes: Obtaining an original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters scanned by the laser radar, and performing standardization processing on the original point cloud, the original point cloud motion parameters, and the original ego vehicle motion parameters in a unified coordinate system to obtain a target point cloud, target point cloud motion parameters, and target ego vehicle motion parameters; Based on the target point cloud motion parameters and the target vehicle motion parameters, performing camera coordinate system transformation on the target point cloud at different exposure times to obtain a first point cloud at a first exposure time and a second point cloud at a second exposure time; Projecting the first point cloud and the second point cloud onto an original camera image to obtain a first image point cloud and a second image point cloud, respectively, and interpolating the first image point cloud and the second image point cloud to obtain a target image point cloud; The method of projecting the first point cloud and the second point cloud onto the original camera image to obtain a first image point cloud and a second image point cloud, and interpolating the first image point cloud and the second image point cloud to obtain a target image point cloud comprises: Based on preset camera parameters, projecting the first point cloud and the second point cloud onto an original camera image to obtain a first image point cloud and a second image point cloud; Connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain a straight line equation, and solving the straight line equation based on the proportion of the target projection moment within the camera exposure period to obtain an interpolated value for each lidar point, where the interpolated value indicates the coordinate corresponding to the projection of the lidar point into the original camera image; Generate target image point cloud by interpolation of all lidar points.

2. The laser radar point cloud projection method according to claim 1, characterized in that: Connecting the same lidar point in the first image point cloud and the second image point cloud by a straight line to obtain a straight line equation, and solving the straight line equation based on the proportion of the target projection moment in the camera exposure period to obtain an interpolation value of each lidar point includes: Connecting the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain a straight line equation, and constructing a target interpolation equation based on the straight line equation and the proportion of the target projection moment within the camera exposure period; The target interpolation equation is interpolated and solved for each laser radar point to obtain the interpolation value of each laser radar point.

3. The laser radar point cloud projection method according to claim 2, characterized in that: The target interpolation equation includes: Among them, (x, y) represents the coordinates of a lidar point in the target image point cloud, (x1, y1) represents the coordinates of the corresponding lidar point in the first image point cloud, (x2, y2) represents the coordinates of the corresponding lidar point in the second image point cloud, and h represents the image height of the target image point cloud.

4. The laser radar point cloud projection method according to claim 1, characterized in that: The step of obtaining the original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters scanned by the laser radar, and performing standardization processing on the original point cloud, the original point cloud motion parameters, and the original ego vehicle motion parameters in a unified coordinate system to obtain the target point cloud, the target point cloud motion parameters, and the target ego vehicle motion parameters includes: Obtaining the original point cloud scanned by the laser radar, the original point cloud motion parameters, and the original vehicle motion parameters, wherein the original point cloud motion parameters include the motion speed of each laser radar point, and the original vehicle motion parameters include the vehicle posture and vehicle speed; Performing a world coordinate system conversion on the original point cloud and the original point cloud motion parameters to obtain a target point cloud and the target point cloud motion parameters; Based on the conversion relationship between the ego-vehicle coordinate system and the world coordinate system, the ego-vehicle posture and the ego-vehicle speed are converted into the world coordinate system to obtain target ego-vehicle motion parameters.

5. The laser radar point cloud projection method according to claim 4, characterized in that: The performing world coordinate system conversion on the original point cloud and the original point cloud motion parameters to obtain the target point cloud and the target point cloud motion parameters includes: Based on the conversion relationship between the laser radar coordinate system and the vehicle coordinate system, the original point cloud and the original point cloud motion parameters are converted into the vehicle coordinate system to obtain a third point cloud and the first point cloud motion parameters; Based on the conversion relationship between the vehicle coordinate system and the world coordinate system, the third point cloud and the motion parameters of the first point cloud are converted into the world coordinate system to obtain the target point cloud and the target point cloud motion parameters.

6. The laser radar point cloud projection method according to claim 1, characterized in that: The step of performing camera coordinate system conversion at different exposure times on the target point cloud based on the target point cloud motion parameters and the target vehicle motion parameters to obtain a first point cloud at a first exposure time and a second point cloud at a second exposure time includes: Determining a first exposure time and a second exposure time, wherein the first exposure time is used to indicate a start exposure time of the original camera image, and the second exposure time is used to indicate an end exposure time of the original camera image; Based on the target point cloud motion parameters and the target vehicle motion parameters at the first exposure time, projecting the target point cloud into the camera coordinate system to obtain a first point cloud at the first exposure time; Based on the target point cloud motion parameters and the target vehicle motion parameters at the second exposure moment, the target point cloud is projected into the camera coordinate system to obtain a second point cloud at the second exposure moment.

7. A laser radar point cloud projection device, characterized in that: The projection device of the laser radar point cloud includes: an acquisition module, configured to acquire an original point cloud, original point cloud motion parameters, and original ego vehicle motion parameters scanned by the laser radar, and to perform standardization processing on the original point cloud, the original point cloud motion parameters, and the original ego vehicle motion parameters in a unified coordinate system to obtain a target point cloud, target point cloud motion parameters, and target ego vehicle motion parameters; A conversion module is configured to perform camera coordinate system conversion of the target point cloud at different exposure times based on the target point cloud motion parameters and the target ego-vehicle motion parameters, to obtain a first point cloud at a first exposure time and a second point cloud at a second exposure time; an interpolation module, configured to project the first point cloud and the second point cloud onto an original camera image, respectively, to obtain a first image point cloud and a second image point cloud, and to interpolate the first image point cloud and the second image point cloud, to obtain a target image point cloud; The interpolation module is also used to: project the first point cloud and the second point cloud onto the original camera image based on preset camera parameters to obtain a first image point cloud and a second image point cloud; connect the same lidar point in the first image point cloud and the second image point cloud with a straight line to obtain a straight line equation, and solve the straight line equation based on the proportion of the target projection moment in the camera exposure period to obtain the interpolation of each lidar point, and the interpolation is used to indicate the corresponding coordinates of the lidar point projected onto the original camera image; and generate a target image point cloud by interpolation of all lidar points.

8. A laser radar point cloud projection device, characterized in that: The laser radar point cloud projection device includes: a memory and at least one processor, wherein the memory stores a computer program; The at least one processor calls the computer program in the memory so that the laser radar point cloud projection device executes the laser radar point cloud projection method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the laser radar point cloud projection method according to any one of claims 1 to 6 is implemented.

Images