Color point cloud generation and registration method and device, and storage medium

Abstract

The invention discloses a color point cloud generation and registration method and device, and a storage medium. The method comprises the steps of collecting laser radar data, IMU data and image data; distortion correction preprocessing is carried out on the laser radar data based on the IMU data, and denoising preprocessing is carried out on the image data; based on the preprocessed laser radar data and image data, performing external parameter calibration on the laser radar and the camera to generate a color point cloud; and registering the color point clouds to obtain a pose transformation relation between two adjacent frames. According to the invention, the color point cloud with excellent quality can be obtained through a common sensor, and the problems that the existing color point cloud is difficult to obtain in an outdoor large scene, the registration precision of the point cloud is relatively low, and the registration time is relatively long are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional point cloud data processing, and in particular relates to a color point cloud generation and registration method, device and storage medium. Background Art

[0002] With the development of lidar technology, it has occupied an important position in the field of robotics and autonomous driving. The point cloud data it outputs has become another major data format for describing the three-dimensional world after camera pixels. Inter-frame point cloud registration technology has become an indispensable part of high-precision positioning and three-dimensional reconstruction of autonomous driving.

[0003] At present, the classic point cloud registration method is the ICP (Iterative Closest Point) algorithm. ICP and its various variants use geometric features such as points and surfaces in the environment to perform iterative optimization and obtain a more accurate inter-frame pose transformation relationship. However, its registration application in scenes with degraded geometric structures such as tunnels and corridors has great limitations.

[0004] As an intuitive description of the environment and with low price, cameras can use color information to make up for the shortcomings of point cloud registration in scenarios with degraded geometric information and enhance the robustness of registration.

[0005] Although depth cameras can obtain geometric and color information in the environment and perform well indoors, their short viewing distance prevents them from providing reliable environmental information in large outdoor scenes. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a color point cloud generation and registration method, device and storage medium.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] In a first aspect, the present invention discloses a color point cloud generation and registration method, comprising:

[0009] Step S1: Collecting lidar data, IMU data and image data;

[0010] Step S2: Preprocessing the laser radar data for distortion correction based on the IMU data, and preprocessing the image data for denoising;

[0011] Step S3: Based on the preprocessed lidar data and image data, the lidar and camera are calibrated with external parameters to generate a color point cloud;

[0012] Step S4: align the color point cloud to obtain the pose transformation relationship between two adjacent frames.

[0013] On the basis of the above technical solution, the following improvements can be made:

[0014] As a preferred solution, step S2 includes:

[0015] Step S2.1: Use IMU data to estimate the vehicle posture in real time, compensate for the point cloud distortion of the lidar data during movement, and achieve distortion correction;

[0016] Step S2.2: Use Gaussian filtering and histogram equalization to preprocess the image data to remove image noise and enhance image contrast.

[0017] As a preferred solution, step S3 includes:

[0018] Step S3.1: superimpose the preprocessed lidar data within a certain time range to form a dense point cloud;

[0019] and storing the image data within the time range to form an image set;

[0020] Step S3.2: Based on the dense point cloud and image set, the SuperGlue network is used to find the correspondence between the 2D image and the 3D dense point cloud, and the external parameters of the lidar and camera are roughly estimated using the random sampling consistency algorithm and minimization of the reprojection error;

[0021] Step S3.3: Based on the current vehicle pose estimation, the dense point cloud is projected into the image, the point cloud closest to the pixel is retained, and the point cloud that is not visible in the image is removed;

[0022] Step S3.4: transform the dense point cloud into the camera coordinate system and map it to the image space;

[0023] Step S3.5: extracting line feature point cloud and surface feature point cloud according to depth consistency and geometric characteristics;

[0024] Step S3.6: Minimize the error terms of the line feature point cloud and the surface feature point cloud to obtain accurate external parameters of the laser radar and camera;

[0025] Step S3.7: After the lidar and camera are calibrated according to accurate external parameters, the point cloud is mapped to the image, the color information is obtained, and a color point cloud is generated.

[0026] As a preferred solution, step S4 includes:

[0027] Step S4.1: Convert the generated color point cloud from RGB space to HSV space to obtain the photometric error E(T) based on the HSV space. G And geometric error E(T) C ;

[0028] Step S4.2: Use the ENIQA algorithm to calculate the image, evaluate the image quality, and obtain an image quality score α;

[0029] Use CNN convolutional neural network to judge weather conditions and obtain weather score β;

[0030] Step S4.3: Based on the image quality score α and the weather score β, obtain the confidence η of the photometric error;

[0031] Step S4.4: Based on the photometric error E(T) G And geometric error E(T) C , obtain the joint error E(T), and minimize the joint error E(T) for iterative optimization to obtain the final pose;

[0032] Where: E(T) = η*E(T) C +(1-η)*E(T) G .

[0033] In a second aspect, the present invention discloses a color point cloud generation and registration device, comprising:

[0034] Data acquisition module, used to collect lidar data, IMU data and image data;

[0035] A preprocessing module is used to perform distortion correction preprocessing on the lidar data based on the IMU data, and to perform denoising preprocessing on the image data;

[0036] The color point cloud generation module is used to calibrate the laser radar and camera based on the preprocessed laser radar data and image data to generate a color point cloud;

[0037] The color point cloud registration module is used to register the color point cloud and obtain the pose transformation relationship between two adjacent frames.

[0038] As a preferred solution, the preprocessing module includes:

[0039] The LiDAR data preprocessing unit is used to estimate the vehicle posture in real time using the IMU data, compensate for the point cloud distortion of the LiDAR data during motion, and achieve distortion correction;

[0040] The image data preprocessing unit is used to preprocess the image data using Gaussian filtering and histogram equalization to remove image noise and enhance image contrast.

[0041] As a preferred solution, the color point cloud generation module includes:

[0042] A preliminary processing unit, used to superimpose the pre-processed lidar data within a certain time range to form a dense point cloud;

[0043] and storing the image data within the time range to form an image set;

[0044] The external parameter rough estimation unit is used to find the correspondence between 2D images and 3D dense point clouds based on dense point clouds and image sets using the SuperGlue network, and to roughly estimate the external parameters of the lidar and camera using the random sampling consistency algorithm and minimizing the reprojection error;

[0045] The culling unit is used to project the dense point cloud into the image based on the current vehicle pose estimation, retain the point cloud closest to the pixel, and cull the point cloud that is not visible in the image;

[0046] A mapping unit, used to transform the dense point cloud into the camera coordinate system and map it to the image space;

[0047] An extraction unit, used for extracting line feature point clouds and surface feature point clouds according to depth consistency and geometric characteristics;

[0048] An accurate external parameter acquisition unit is used to minimize the error terms of line feature point clouds and surface feature points to obtain accurate external parameters of the laser radar and camera;

[0049] The color point cloud generation unit is used to map the point cloud to the image after the laser radar and camera are calibrated according to accurate external parameters, obtain color information, and generate a color point cloud.

[0050] As a preferred solution, the color point cloud registration module includes:

[0051] The error acquisition unit is used to convert the generated color point cloud from RGB space to HSV space and obtain the photometric error E(T) based on the HSV space. G And geometric error E(T) C ;

[0052] A score calculation unit, used to calculate the image using the ENIQA algorithm, evaluate the image quality, and obtain an image quality score α;

[0053] Use CNN convolutional neural network to judge weather conditions and obtain weather score β;

[0054] A confidence acquisition unit, used for obtaining a confidence η of a photometric error based on an image quality score α and a weather score β;

[0055] Iterative optimization unit for photometric error E(T) G And geometric error E(T) C , obtain the joint error E(T), and minimize the joint error E(T) for iterative optimization to obtain the final pose;

[0056] Where: E(T) = η*E(T)C +(1-η)*E(T) G .

[0057] In a third aspect, the present invention discloses a storage medium, which includes a computer program. When the computer program is executed by one or more processors, any of the above-mentioned color point cloud generation and registration methods is implemented.

[0058] The present invention discloses a color point cloud generation and registration method, device and storage medium, which solve the problems of difficulty in obtaining existing color point clouds in large outdoor scenes, low point cloud registration accuracy and long registration time.

[0059] The present invention has the following beneficial effects:

[0060] First, the present invention can obtain high-quality color point clouds through common sensors.

[0061] Second, the present invention can automatically estimate the external parameters of the lidar and camera without the need for additional calibration plates and fixed objects, thereby improving the universality of the method.

[0062] Third, the present invention performs color point cloud registration based on the HSV space, thereby improving the registration accuracy and reducing the registration time. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 A flowchart of a color point cloud generation and registration method provided in an embodiment of the present invention.

[0065] Figure 2 A flowchart of generating a color point cloud provided by an embodiment of the present invention.

[0066] Figure 3 (a) and (b) are two example images after the color point cloud is generated according to the embodiments of the present invention. DETAILED DESCRIPTION

[0067] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0070] In order to achieve the purpose of the present invention, in some embodiments of the color point cloud generation and registration method, as Figure 1 As shown, the color point cloud generation and registration method includes:

[0071] Step S1: Collect 3D laser radar data, IMU (Inertial Measurement Unit) data and image data;

[0072] Step S2: Preprocessing the 3D laser radar data for distortion correction based on the IMU data, and preprocessing the image data for denoising;

[0073] Step S3: Based on the preprocessed 3D laser radar data and image data, the laser radar and camera are calibrated with external parameters to generate a color point cloud;

[0074] Step S4: align the color point cloud to obtain the pose transformation relationship between two adjacent frames.

[0075] This embodiment uses 3D laser radar, which is compatible with solid-state laser radar and mechanical laser radar, and can improve the universality of the application of the present invention.

[0076] Each step is described in detail below.

[0077] Step S2 mainly pre-processes the data, including:

[0078] Step S2.1: Use IMU data to estimate the vehicle posture in real time, compensate for the point cloud distortion of 3D lidar data during movement, and achieve distortion correction;

[0079] Step S2.2: Use Gaussian filtering and histogram equalization to preprocess the image data to remove image noise and enhance image contrast.

[0080] The present invention uses laser radar data and IMU data to remove point cloud distortion. First, the IMU data is integrated to obtain the rotation angle between frames, and the attitude of the radar is subjected to attitude quaternion spherical linear interpolation, and the corresponding point cloud is converted to the final state attitude to achieve point cloud motion distortion removal.

[0081] The coordinate system of each point in each frame of point cloud is different, which makes each frame of point cloud of LiDAR have certain differences from the real environment. The correction of self-motion distortion is to convert the coordinate system of all points in this frame to the same coordinate system.

[0082] The distortion correction formula is:

[0083]

[0084] Where: L represents the radar coordinate system, i represents the coordinate system corresponding to the point cloud, j represents the index of the point cloud, and T represents the coordinate transformation of the i coordinate system relative to the L radar coordinate system.

[0085] Furthermore, the present invention uses Gaussian filtering and histogram equalization to pre-process the image, remove image noise, enhance image contrast, and improve image quality.

[0086] First, the camera internal parameters are calibrated, and the image distortion is corrected using the calibrated camera internal parameters. The camera distortion only considers radial distortion k and tangential distortion p.

[0087] The specific distortion correction formula is:

[0088] x distorted =x(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 1 xy+p 2 (r 2 +2x 2 );

[0089] y distorted =y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+2p 2 xy+p 1 (r 2 +2y 2 );

[0090] r 2 =x 2 +y2 ;

[0091] Where: r is the distance between the pixel point and the pixel center;

[0092] x distorted ,y distorted are the pixel values ​​after distortion correction respectively;

[0093] x and y are the pixel values ​​before distortion correction;

[0094] k 1 , k 2 , k 3 is the radial distortion coefficient, p 1 、p 2 is the tangential distortion coefficient.

[0095] Then, Gaussian low-pass filtering is used to remove noise in the image. The Gaussian filtering formula is:

[0096]

[0097] Among them, σ is the standard deviation, x is the horizontal coordinate of the pixel in the image, and y is the vertical coordinate of the pixel in the image.

[0098] Finally, the image is histogram equalized to weaken the impact of light intensity and weather on image color and enhance local contrast while ensuring overall consistency.

[0099] When using lidar and cameras to generate color point clouds, they need to be calibrated with external parameters. Conventional external parameter calibration algorithms require specific scenes, such as calibration rooms, calibration platforms, etc.; or specific calibration objects, including chessboards, spheres, etc., which greatly affect the calibration work efficiency and cost.

[0100] The invention proposes a fully automated calibration method that does not require fixed calibration objects and calibration scenes. Specifically, step S3 includes:

[0101] Step S3.1: Use the STD algorithm to perform inter-frame registration, and superimpose the pre-processed lidar data within a certain time range (e.g., 5s) to form a dense point cloud to make up for the sparseness of a single-frame laser point cloud;

[0102] and storing the image data within the time range to form an image set;

[0103] Step S3.2: Based on the dense point cloud and image set, the SuperGlue network is used to find the correspondence between the 2D image and the 3D dense point cloud, and the following is obtained:

[0104] 2D keypoint collection

[0105] 3D Point Cloud Collection

[0106] Where: M represents the number of data in the dense point cloud and image set.

[0107] For sets I and P, the random sampling consensus (RANSAC) algorithm is used to remove mismatched relationship pairs;

[0108] The Levenberg-Marquardt method is used to minimize the reprojection error to roughly estimate the external parameters of the lidar and camera, and obtain a rough registration of the image and point cloud, where the projection error is:

[0109]

[0110] Where: ρ is the Cauchy robust kernel function;

[0111] P K j is the jth point in the Kth point cloud in the point cloud set P in the radar coordinate system;

[0112] T C L is the coordinate transformation of the radar coordinate system relative to the camera coordinate system;

[0113] π() is the mapping function from 3D point cloud to 2D key points;

[0114] X k j is the j-th feature point of the K-th image in the image set.

[0115] Step S3.3: Based on the current vehicle pose estimation, the dense point cloud is projected into the image, the point cloud closest to the pixel is retained, and the point cloud that is not visible in the image is removed;

[0116] Step S3.4: transform the dense point cloud into the camera coordinate system and map it to the image space;

[0117]

[0118] Step S3.5: extracting line feature point cloud and surface feature point cloud according to depth consistency and geometric characteristics;

[0119] Step S3.6: Minimize the error terms of the line feature point cloud and the surface feature point cloud to obtain accurate external parameters of the laser radar and camera;

[0120] Step S3.7: After the lidar and camera are calibrated according to accurate external parameters, the point cloud is mapped to the image, the color information is obtained, and a color point cloud is generated.

[0121] The dense color point cloud obtained after projection is as follows Figure 3 As shown in (a) and (b), the color information of lane lines, flower beds, trees, etc. on the road can be clearly represented.

[0122] The present invention superimposes point clouds within a certain time range into dense point clouds to make up for the deficiency of sparse single-frame laser point clouds, and stores images within this time interval. SuperGlue is used to find the correspondence between 2D images and 3D dense point clouds, and the laser camera extrinsic parameters are roughly estimated using random sampling consistency (RANSAC) and minimizing reprojection errors. The view-based hidden point removal method filters out points that are invisible from the camera's perspective, extracts line and surface features based on depth consistency and point cloud curvature, minimizes line and surface errors, and obtains accurate extrinsic parameters.

[0123] After obtaining the color point cloud, the present invention performs color point cloud registration to obtain the inter-frame pose relationship. The present invention proposes a relatively novel color point cloud inter-frame registration method based on HSV space. Specifically, step S4 includes:

[0124] Step S4.1: Convert the generated color point cloud from RGB space to HSV space to obtain the photometric error E(T) based on the HSV space. G And geometric error E(T) C ;

[0125] Step S4.2: Use the ENIQA algorithm to calculate the image, evaluate the image quality, and obtain an image quality score α, where α=Eniqa(I), I is the input image;

[0126] Use CNN convolutional neural network to process the image, judge the weather conditions (such as sunny, cloudy, haze, etc.), and obtain the weather score β, where β = CNN (I), I is the input image;

[0127] Step S4.3: Based on the image quality score α and the weather score β, obtain the confidence η of the photometric error;

[0128] η=α+β;

[0129] Step S4.4: Based on the photometric error E(T) G And geometric error E(T) C , obtain the joint error E(T), and minimize the joint error E(T) for iterative optimization to obtain the final pose;

[0130] Where: E(T) = η*E(T) C +(1-η)*E(T) G .

[0131] The confidence η of the above photometric error is closely related to the image quality and weather conditions. When the image quality is higher and the weather conditions are better, the confidence η of the photometric error is higher; when the image quality is lower and the weather conditions are worse, the confidence η of the photometric error is lower. Through these two evaluation criteria, the confidence of the photometric error in the joint optimization can be dynamically adjusted to improve the robustness of the algorithm.

[0132] Color point cloud registration comprehensively considers the position information and color information in the point cloud, and converts the color information RGB (red, green, blue) in the color point cloud P into HSV (hue, saturation, brightness). i ) represents the hue information of each point in the point cloud. To ensure continuity and spatial consistency, the sin function is used to map it and obtain H(p i ).

[0133] The specific formula is as follows:

[0134] H(p i )=sin(h(p i ));

[0135] The first-order Taylor expansion of H(p) around point p is H(u) p It can be approximated as:

[0136]

[0137] Among them, H(u) p Represents the hue value of the smallest range u at point p;

[0138] H(p) represents the hue value at point p;

[0139] is the gradient value at point p;

[0140] o(u 2 ) represents the quadratic minimum of u.

[0141] The photometric error E(T) based on the HSV space is C for:

[0142]

[0143] Among them, K is the set of all points in the color point cloud;

[0144] p and q are points in the color point cloud set;

[0145] is the projection point of point q on the tangent plane of point p;

[0146] The projection point The hue value of

[0147] H(q) is the hue value of point q.

[0148] Similarly, the geometric error E(T) G To calculate the distance between points and surfaces in space, the specific formula is:

[0149] E(T) G =∑ (p,q∈K) ((s(q, T)-p) T n p ) 2 .

[0150] Among them, K is the set of all points in the color point cloud;

[0151] p and q are points in the color point cloud set;

[0152] s(q, T) represents the point q after the current coordinate transformation T;

[0153] n p Represents the gradient value at point p.

[0154] In summary, the present invention mainly includes: data collection, data preprocessing, color point cloud generation and color point cloud registration steps, and outputs the optimized vehicle posture.

[0155] The dedistorted LiDAR data is stacked to generate a dense point cloud, and the LiDAR camera is automatically calibrated for external parameters. The projection transformation and consistency evaluation are completed to obtain a color point cloud.

[0156] After the color point cloud is generated, the accurate pose transformation relationship between two adjacent frames can be obtained through color point cloud registration, where the pose transformation includes 6 degrees of freedom, displacement xyz and rotation rol, pitch, and yaw.

[0157] Further, in some embodiments of the color point cloud generation and registration device, the color point cloud generation and registration device includes:

[0158] Data acquisition module, used to collect lidar data, IMU data and image data;

[0159] A preprocessing module is used to perform distortion correction preprocessing on the lidar data based on the IMU data, and to perform denoising preprocessing on the image data;

[0160] The color point cloud generation module is used to calibrate the laser radar and camera based on the preprocessed laser radar data and image data to generate a color point cloud;

[0161] The color point cloud registration module is used to register the color point cloud and obtain the pose transformation relationship between two adjacent frames.

[0162] Furthermore, the preprocessing module includes:

[0163] The LiDAR data preprocessing unit is used to estimate the vehicle posture in real time using the IMU data, compensate for the point cloud distortion of the LiDAR data during motion, and achieve distortion correction;

[0164] The image data preprocessing unit is used to preprocess the image data using Gaussian filtering and histogram equalization to remove image noise and enhance image contrast.

[0165] Furthermore, the color point cloud generation module includes:

[0166] A preliminary processing unit, used to superimpose the pre-processed lidar data within a certain time range to form a dense point cloud;

[0167] and storing the image data within the time range to form an image set;

[0168] The external parameter rough estimation unit is used to find the correspondence between 2D images and 3D dense point clouds based on dense point clouds and image sets using the SuperGlue network, and to roughly estimate the external parameters of the lidar and camera using the random sampling consistency algorithm and minimizing the reprojection error;

[0169] The culling unit is used to project the dense point cloud into the image based on the current vehicle pose estimation, retain the point cloud closest to the pixel, and cull the point cloud that is not visible in the image;

[0170] A mapping unit, used to transform the dense point cloud into the camera coordinate system and map it to the image space;

[0171] An extraction unit, used for extracting line feature point clouds and surface feature point clouds according to depth consistency and geometric characteristics;

[0172] An accurate external parameter acquisition unit is used to minimize the error terms of line feature point clouds and surface feature points to obtain accurate external parameters of the laser radar and camera;

[0173] The color point cloud generation unit is used to map the point cloud to the image after the laser radar and camera are calibrated according to accurate external parameters, obtain color information, and generate a color point cloud.

[0174] Furthermore, the color point cloud registration module includes:

[0175] The error acquisition unit is used to convert the generated color point cloud from RGB space to HSV space and obtain the photometric error E(T) based on the HSV space. G And geometric error E(T) C ;

[0176] A score calculation unit, used to calculate the image using the ENIQA algorithm, evaluate the image quality, and obtain an image quality score α;

[0177] Use CNN convolutional neural network to judge weather conditions and obtain weather score β;

[0178] A confidence acquisition unit, used for obtaining a confidence η of a photometric error based on an image quality score α and a weather score β;

[0179] Iterative optimization unit for photometric error E(T) G And geometric error E(T) C , obtain the joint error E(T), and minimize the joint error E(T) for iterative optimization to obtain the final pose;

[0180] Where: E(T) = η*E(T) C +(1-η)*E(T) G .

[0181] In this embodiment, the specific content of the color point cloud generation and registration device is similar to the content of the color point cloud generation and registration method disclosed in the above embodiment, and will not be repeated here.

[0182] In addition, an embodiment of the present invention further discloses a storage medium, which includes a computer program. When the computer program is executed by one or more processors, the color point cloud generation and registration method disclosed in any of the above embodiments is implemented.

[0183] The present invention discloses a color point cloud generation and registration method, device and storage medium, which solve the problems of difficulty in obtaining existing color point clouds in large outdoor scenes, low point cloud registration accuracy and long registration time.

[0184] The present invention has the following beneficial effects:

[0185] First, the present invention can obtain high-quality color point clouds through common sensors (such as 3D lidar, visual camera, IMU).

[0186] Second, the present invention can automatically estimate the external parameters of the lidar and camera without the need for additional calibration plates and fixed objects, thereby improving the universality of the method.

[0187] Third, the present invention performs color point cloud registration based on the HSV space, thereby improving the registration accuracy and reducing the registration time.

[0188] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or portions of the method and apparatus of the present invention, can take the form of program codes (i.e., instructions) embedded in a tangible medium, such as a floppy disk, a CD-ROM, a hard disk drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0189] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A color point cloud generation and registration method, characterized in that: include: Step S1: Collecting lidar data, IMU data and image data; Step S2: Preprocessing the laser radar data for distortion correction based on the IMU data, and preprocessing the image data for denoising; Step S3: Based on the preprocessed lidar data and image data, the lidar and camera are calibrated with external parameters to generate a color point cloud; Step S4: align the color point cloud to obtain the pose transformation relationship between two adjacent frames.

2. The color point cloud generation and registration method according to claim 1, characterized in that: The step S2 comprises: Step S2.1: Use IMU data to estimate the vehicle posture in real time, compensate for the point cloud distortion of the lidar data during movement, and achieve distortion correction; Step S2.2: Use Gaussian filtering and histogram equalization to preprocess the image data to remove image noise and enhance image contrast.

3. The color point cloud generation and registration method according to claim 1, characterized in that: The step S3 comprises: Step S3.1: superimpose the preprocessed lidar data within a certain time range to form a dense point cloud; and storing the image data within the time range to form an image set; Step S3.2: Based on the dense point cloud and image set, the SuperGlue network is used to find the correspondence between the 2D image and the 3D dense point cloud, and the external parameters of the lidar and camera are roughly estimated using the random sampling consistency algorithm and minimization of the reprojection error; Step S3.3: Based on the current vehicle pose estimation, the dense point cloud is projected into the image, the point cloud closest to the pixel is retained, and the point cloud that is not visible in the image is removed; Step S3.4: Convert the dense point cloud to the camera coordinate system and map it to the image space: Step S3.5: extracting line feature point cloud and surface feature point cloud according to depth consistency and geometric characteristics; Step S3.6: Minimize the error terms of the line feature point cloud and the surface feature point cloud to obtain accurate external parameters of the laser radar and camera; Step S3.7: After the lidar and camera are calibrated according to accurate external parameters, the point cloud is mapped to the image, the color information is obtained, and a color point cloud is generated.

4. The color point cloud generation and registration method according to claim 1, characterized in that: The step S4 comprises: Step S4.1: Convert the generated color point cloud from RGB space to HSV space to obtain the photometric error E(T) based on the HSV space. G And geometric error E(T) C ; Step S4.2: Use the ENIQA algorithm to calculate the image, evaluate the image quality, and obtain an image quality score α; Use CNN convolutional neural network to judge weather conditions and obtain weather score β; Step S4.3: Based on the image quality score α and the weather score β, obtain the confidence η of the photometric error; Step S4.4: Based on the photometric error E(T) G And geometric error E(T) C , obtain the joint error E(T), and minimize the joint error E(T) for iterative optimization to obtain the final pose; Where: E(T)=η*E(T) C +(1-n)*E(T) G 。 5. Color point cloud generation and registration device, characterized in that: include: Data acquisition module, used to collect lidar data, IMU data and image data; A preprocessing module is used to perform distortion correction preprocessing on the lidar data based on the IMU data, and to perform denoising preprocessing on the image data; The color point cloud generation module is used to calibrate the laser radar and camera based on the preprocessed laser radar data and image data to generate a color point cloud; The color point cloud registration module is used to register the color point cloud and obtain the pose transformation relationship between two adjacent frames.

6. The color point cloud generation and registration device according to claim 5, characterized in that: The preprocessing module comprises: The LiDAR data preprocessing unit is used to estimate the vehicle posture in real time using the IMU data, compensate for the point cloud distortion of the LiDAR data during motion, and achieve distortion correction; The image data preprocessing unit is used to preprocess the image data using Gaussian filtering and histogram equalization to remove image noise and enhance image contrast.

7. The color point cloud generation and registration device according to claim 5, characterized in that: The color point cloud generation module comprises: A preliminary processing unit, used to superimpose the pre-processed lidar data within a certain time range to form a dense point cloud; and storing the image data within the time range to form an image set; The external parameter rough estimation unit is used to find the correspondence between 2D images and 3D dense point clouds based on dense point clouds and image sets using the SuperGlue network, and to roughly estimate the external parameters of the lidar and camera using the random sampling consistency algorithm and minimizing the reprojection error; The culling unit is used to project the dense point cloud into the image based on the current vehicle pose estimation, retain the point cloud closest to the pixel, and cull the point cloud that is not visible in the image; A mapping unit, used to transform the dense point cloud into the camera coordinate system and map it to the image space; An extraction unit, used for extracting line feature point clouds and surface feature point clouds according to depth consistency and geometric characteristics; An accurate external parameter acquisition unit is used to minimize the error terms of line feature point clouds and surface feature points to obtain accurate external parameters of the laser radar and camera; The color point cloud generation unit is used to map the point cloud to the image after the laser radar and camera are calibrated according to accurate external parameters, obtain color information, and generate a color point cloud.

8. The color point cloud generation and registration device according to claim 5, characterized in that: The color point cloud registration module comprises: The error acquisition unit is used to convert the generated color point cloud from RGB space to HSV space and obtain the photometric error E(T) based on the HSV space. G And geometric error E(T) C ; A score calculation unit, used to calculate the image using the ENIQA algorithm, evaluate the image quality, and obtain an image quality score α; Use CNN convolutional neural network to judge weather conditions and obtain weather score β; A confidence acquisition unit, used for obtaining a confidence η of a photometric error based on an image quality score α and a weather score β; Iterative optimization unit for photometric error E(T) G And geometric error E(T) C , obtain the joint error E(T), and minimize the joint error E(T) for iterative optimization to obtain the final pose; Where: E(T)=η*E(T) C +(1-n)*E(T) G 。 9. A storage medium, characterized in that The storage medium contains a computer program, and when the computer program is executed by one or more processors, the color point cloud generation and registration method according to any one of claims 1 to 4 is implemented.

Images