Cross-modal registration method and system based on laser radar and camera images from Patch to pixel

Abstract

The invention relates to a cross-modal registration method and system for laser radar and camera images based on Patch-to-pixel, and the method comprises the steps: 1, carrying out the data preprocessing, and projecting a point cloud into two projection images, including a reflectivity image and a distance image; step 2, acquiring Patch-level features and pixel-level features of the projection drawing and the camera image; 3, performing feature matching by using a matching network from Patch to pixels, and estimating a pixel matching pair of the projection drawing and the camera image; and 4, converting the pixel matching point pair into a three-dimensional-two-dimensional point pair of the point cloud and the image, and estimating a transformation matrix from the given three-dimensional-two-dimensional point pair by using a PnP algorithm to complete the registration process. According to the invention, cross-modal registration between the laser radar point cloud and the camera image is realized, and the registration accuracy on the KITTI data set can reach more than 99%.

Description

Technical Field

[0001] The present invention relates to a cross-modal registration method and system for laser radar and camera images based on patch to pixel, and belongs to the technical field of image processing. Background Art

[0002] Advanced perception systems are needed in autonomous driving and robotic systems to obtain accurate and comprehensive information about the surrounding environment. The research and development of perception systems is crucial to promoting the development of autonomous driving technology. By continuously improving the accuracy, stability, and adaptability of perception systems, safer and more efficient intelligent driving cars can be achieved, bringing huge potential benefits to society. Sensors are the most reliable data source in perception systems. Through data fusion from different sensors, road conditions, obstacle locations, and motion states can be understood in real time. Data fusion collected by different sensors plays an important role in many downstream tasks, such as target detection and segmentation. By utilizing these fused data, accuracy and efficiency can be improved, thereby ensuring safety and robustness in complex environments. Among these methods, the most common processing process is to fuse the laser radar (such as solid-state laser radar or rotating laser radar) point cloud with the camera (such as high-definition camera or infrared camera) image.

[0003] However, a prerequisite for successful fusion is accurate alignment between these data. A common approach to solving the alignment task between sensors is to use a checkerboard calibration method to calculate the external parameters between sensors, which is called offline calibration. By using these methods, high-precision alignment between sensors can be achieved, but at the cost of an unavoidable time cost. In addition, some online calibration methods have been proposed to correct external parameter errors caused by collisions or vibrations. These online calibration methods can correct deviations within a limited error range, but the effect is lost when the error is too large. Although these methods can achieve high-precision alignment, the complex calibration process makes the preprocessing of data fusion time-consuming and inefficient. These processes must be repeated whenever the sensor deployment changes.

[0004] Therefore, how to propose a method to bypass these calibration processes and directly perform registration between collected data has become an urgent problem to be solved. For this reason, cross-modal registration methods have emerged. Recent cross-modal registration methods have solved this cross-modal data alignment task by using specific deep learning networks. However, the cross-modal registration process usually involves comparing the high-dimensional feature similarities between point clouds and images to establish correspondences, and the domain gap between 3D and 2D data always affects this process. Specifically, there are dimensionality and information differences between the 3D spatial data of point clouds and the 2D texture information of images. Although some existing methods attempt to reduce this effect by exchanging feature information or using feature extraction networks with more similar structures, the difference in data dimensions still affects the accuracy of registration, especially leading to a higher matching failure rate.

[0005] In addition, point cloud feature extraction networks commonly used in cross-modal registration tasks (such as PointNet++, sparse convolutional networks) usually simply regard lidar point clouds as a series of unordered points or voxels, failing to effectively utilize the spatial and geometric features of point clouds. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an efficient cross-modal registration method for lidar and camera images based on patch to pixel. The point cloud is projected into two types of projection images, namely distance map and reflectivity map, through the geometric information and reflectivity information of the lidar point cloud, and the cross-modal registration task is converted into an image registration task. The features of the projection image and the camera image are respectively extracted by a deep learning network, and the matching relationship between the lidar point cloud and the camera image is found by matching them in a high-dimensional feature space through feature similarity, and finally the transformation matrix between the two is estimated.

[0007] In order to better utilize the scanning characteristics of LiDAR to bridge the field gap, the present invention proposes a method for projecting point clouds into 2D representations: distance maps and reflectivity maps. At the same time, the present invention designs a multi-scale feature extraction network to alleviate the cross-modal differences between images and projection maps and effectively extract features. In addition, the present invention introduces a feature matching framework based on patch to pixel. Patch-level search solves the supervision and success rate problems caused by the scarcity of positive samples, while pixel-level matching improves the registration accuracy. Experimental results show that the network design of the present invention can achieve high-precision registration while maintaining low computational complexity.

[0008] In the present invention, first, data preprocessing is performed to project the point cloud. This method of mapping the point cloud to pixels in the 2D representation provides a data format that implicitly contains more geometric information, which is more efficient and accurate than using an unordered sparse point cloud. After projection, the cross-modal registration task is converted into an image registration task between the camera image and the projection image. Common image registration methods are usually completed by key point matching. Due to the cross-modal differences between the projected 2D image and the camera image, the traditional key point registration method cannot be directly applied to this task. In addition, the overlapping area between the 2D image and the camera image is usually very limited in actual scenes, so the proportion of truly matched point pairs is extremely small, resulting in a scarcity of positive samples. For existing image matching methods based on deep learning, this scarcity may lead to supervision difficulties, making it difficult for the network to converge, and increasing the number of outliers, thereby increasing the possibility of registration failure. To solve this problem, the present invention proposes a patch-to-pixel matching framework. Patch-level matching solves the supervision problem while ensuring the robustness of the network, and pixel-level matching improves the accuracy of registration.

[0009] The present invention also provides a cross-modal registration system of laser radar and camera images based on patch to pixel.

[0010] Terminology explanation:

[0011] 1. LaserID refers to the unique identifier assigned to each laser transmitter-receiver unit (laser head) in a rotating LiDAR system. For example, a rotating LiDAR may have 64 laser heads, and LaserIDs are assigned from 0 to 64, with each LaserID corresponding to the scan data of a specific laser head. Each laser identifier corresponds to a fixed pitch angle position within the LiDAR scan range and generates a single line of point cloud data during active imaging.

[0012] 2. PnP (Perspective-n-Point), PnP is a method for solving the correspondence between 3D and 2D points. It describes how to estimate the camera pose when n 3D space points and their positions are known. PnP is based on the pinhole imaging model of the camera and solves the pose parameters by minimizing the error of projecting 3D points onto the image plane. If the 3D position of a feature point in two images is known, at least 3 point pairs (and at least one additional verification point to verify the result) are required to calculate the camera motion. There are many algorithms for solving the PnP problem, such as the EPnP algorithm (see the paper "EPnP: Efficient Perspective-n-Point Camera Pose Estimation" for details).

[0013] 3. Softmax,Softmax is a commonly used mathematical function, mainly used in multi-classification tasks to convert a set of real numbers into probability distribution. It expands the input value to a positive number through an exponential function and normalizes it to a value between 0 and 1, so that the sum of all outputs is 1, which meets the definition of probability. The expression of the Softmax function is:

[0014]

[0015] Among them, z i is the i-th element in the input vector, and N is the dimension of the input vector.

[0016] 4. Top-k, Top-k refers to the operation of selecting the top k elements in a set of data, usually used for sorting, screening or evaluation tasks. It is particularly common in deep learning and information retrieval. For example, in classification problems, Top-k Accuracy is used to evaluate whether the top k categories predicted by the model contain the correct answer. The Top-k method determines the top k highest values ​​or rankings by comparing the scores (such as probability or similarity) of each element.

[0017] The technical solution of the present invention is:

[0018] An efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel, including:

[0019] Step 1: Perform data preprocessing and project the point cloud into two projection images, including reflectivity image and distance image;

[0020] Step 2: Input the projection image and camera image into the convolutional neural network built in Python; the convolutional neural network outputs the patch-level features and pixel-level features of the projection image and camera image;

[0021] Step 3: Use the patch-to-pixel matching network to perform feature matching. The patch-level features first estimate the patch matching pairs, and then use the estimated patch matching pairs to extract the pixel-level features corresponding to each patch from the pixel-level features; the extracted pixel-level features are used for pixel-level matching to estimate the pixel matching pairs of the projection image and the camera image;

[0022] Step 4: Convert the pixel matching point pairs into 3D-2D point pairs of point cloud and image, and use the PnP (Perspective-n-Point) algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process.

[0023] Preferably, according to the present invention, the point cloud is projected into two projection images, including a reflectivity image and a distance image; including:

[0024] 1) Use the spherical projection principle to transform the point cloud coordinates; transform the point cloud from the Cartesian coordinate system to the spherical coordinate system, and calculate the azimuth according to the coordinates (x, y, z) of the point cloud Azimuth The calculation formula is: The value range of is [-π, π];

[0025] 2) After obtaining the LaserID of the point cloud, generate a projection map and record the mapping. The pixel coordinates of the projection map corresponding to the point cloud are: For the reflectivity map, the reflectivity of the point cloud is recorded in the pixel, that is, the gray value of each pixel represents the reflectivity of the point cloud falling on the pixel; for the distance map, the radius r of the point cloud in the spherical coordinate system is recorded in the pixel, and the radius x, y, and z are the coordinate values ​​of the point cloud on the X, Y, and Z axes in the Cartesian coordinate system.

[0026] Preferably, according to the present invention, the projection image and the camera image are input into a convolutional neural network constructed by Python language; the convolutional neural network outputs patch-level features and pixel-level features of the projection image and the camera image; including:

[0027] The feature extraction network, i.e., the convolutional neural network built in Python, is divided into two branches. The first branch is used to process the reflectivity map and the distance map, and the second branch is used to process the camera image. Each branch will extract patch-level features with a scale of 1 / 4 of the original image and pixel-level features with the same scale as the original image.

[0028] The first half of the feature extraction network is the image encoder part, and the second half of the feature extraction network is the image decoder part. The scale of the features in the image encoder part is gradually downsampled, and the image encoder part finally generates a high-dimensional feature at 1 / 32 of the original image scale; the high-dimensional feature is passed to the image decoder part, and in the image decoder part, the scale of the feature is gradually upsampled; specifically, it includes:

[0029] First, obtain the patch-level features at 1 / 4 the size of the original image. Among them, D Patch is the number of Patch-level feature channels, H and W are the height and width of the original image, Refers to the set of real numbers in mathematics; then, through two layers of deconvolution, the pixel-level features at the original image resolution are obtained Among them, D pixel is the number of pixel-level feature channels.

[0030] In the second branch, the RGB three-channel camera image is input into the convolutional neural network. After five layers of full convolutional networks, 128-channel small-scale features are obtained. Then, the small-scale features are upsampled by deconvolution. After three layers of deconvolution, 256-channel patch-level features with a scale of 1 / 4 of the original camera image are obtained. The patch-level features are further upsampled by deconvolution. After two layers of deconvolution, the scale is restored to the original image scale. At this time, a pixel-level feature with a channel number of 256 and the same scale as the original camera image is obtained.

[0031] In the first branch, first, the reflectivity map and the distance map are subjected to feature extraction by a five-layer full convolutional network to obtain a small-scale feature of 128 channels. Then, the two small-scale features obtained are spliced, and the number of channels of the spliced ​​feature is 256, and the scale remains unchanged. Thirdly, the spliced ​​feature is subjected to a full convolution operation again, and the obtained feature is deconvoluted and upsampled. After three layers of deconvolution, the number of channels is 256, and the scale is the original projection. Figure 1 / 4 scale of Patch-level features; the Patch-level features are deconvolved and upsampled twice again to obtain pixel-level features with the same scale as the original projection image and 256 channels;

[0032] At this point, the patch-level features and pixel-level features of the camera image and projection image are obtained.

[0033] Preferably, according to the present invention, feature matching is performed using a patch-to-pixel matching network; comprising:

[0034] First, the patch-level features extracted by the feature extraction network are input into the patch matching module for patch-level matching; including:

[0035] For two patch-level features that need to be matched, first flatten the two patch-level features and input them into a linear layer neural network with the same number of channels and output channels. The number and scale of the output feature channels are the same as the input patch-level features. Then, the two output features are matrix multiplied to obtain the patch similarity matrix S. Finally, the matrix S is subjected to row-by-row and column-by-column Softmax operations to obtain the matrix after row Softmax processing and the matrix after column Softmax processing, respectively. The Hadamard product of the two matrices is calculated to obtain the distribution matrix P, which is described by the following formula:

[0036]

[0037] Among them, P ij represents the value of the i-th row and j-th column in the allocation matrix P, S ij Represents the value of the i-th row and j-th column in the allocation matrix S, N2D is the total number of patches in the camera image, that is, the total number of rows in the S matrix; N 3D is the total number of patches in the projected image, that is, the total number of columns in the S matrix;

[0038] Select Top-k from the allocation matrix P to obtain the matching relationship between the projection image and the camera image patch-level features, i.e., the patch matching pair; then, the pixel-level features selected from the patch matching pair are used for pixel-level matching; including:

[0039] For each patch matching pair, all pixels in it are sent to the patch matching module for matching, and only the highest similarity value is retained, that is, only the highest item in Top-k is taken; by performing pixel-level matching on each patch matching pair, the final pixel matching pair is obtained.

[0040] Further preferably, the calculation formula of the Patch similarity matrix S is as follows:

[0041]

[0042] Among them, f C and f L These are the two matrices output by the two linear layers respectively. Faltten is the flattening operation and Linear is the linear layer operation.

[0043] Further preferably, the pixel matching point pairs are converted into 3D-2D point pairs of point clouds and images, and the transformation matrix is ​​estimated from the given 3D-2D point pairs using the PnP algorithm to complete the registration process; including:

[0044] Through pixel matching pairs, find the points in the point cloud corresponding to the pixels on the projection image, and then establish the 3D-2D correspondence between the projected 3D point cloud and the camera image pixels, that is, the 3D-2D point pairs of the point cloud and the image;

[0045] The EPnP algorithm is used to solve the pose and finally estimate the transformation matrix.

[0046] Preferably, according to the present invention, in the convolutional neural network and the matching network, the total loss function is:

[0047] L total =L patch +L pixel ;

[0048] Assuming that there is an actual correspondence It contains M corresponding items, and the actual corresponding relationship Each value in is divided by 4 and rounded to get the true Patch correspondence The first part of the loss function L patchThe expression is as follows:

[0049]

[0050] Among them, (i, j) refers to the value of each corresponding item in the corresponding relationship, P ij It refers to the value corresponding to the i-th row and j-th column in the allocation matrix P;

[0051] For each Patch correspondence, the matching network calculates M allocation matrices p; by Divide by 4 and take the remainder to obtain the true pixel correspondence in each assignment matrix p Thus we get the second part of the loss function L Pixel , the second part of the loss function L pixel The expression is as follows:

[0052]

[0053] Among them, p i represents the allocation matrix calculated by the matching module for the i-th pair of patches, represents the i-th true pixel correspondence, Indicates p i The value at the corresponding position in the matrix.

[0054] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a cross-modal registration method of a laser radar and a camera image based on patch to pixel are implemented.

[0055] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a cross-modal registration method of a laser radar and camera image based on patch to pixel.

[0056] An efficient cross-modal registration system for LiDAR and camera images based on patch-to-pixel, including:

[0057] The data preprocessing module is configured to: perform data preprocessing and project the point cloud into two projection images, including a reflectivity image and a distance image;

[0058] The patch-level feature and pixel-level feature acquisition module is configured to: input the projection image and the camera image into the convolutional neural network constructed by the Python language; the convolutional neural network outputs the patch-level features and pixel-level features of the projection image and the camera image;

[0059] The feature matching module is configured to: use a patch-to-pixel matching network to perform feature matching, first estimate patch matching pairs from patch-level features, then extract pixel-level features corresponding to each patch from pixel-level features using the estimated patch matching pairs; use the extracted pixel-level features for pixel-level matching to estimate pixel matching pairs of the projection image and the camera image;

[0060] The registration module is configured to: convert pixel matching point pairs into 3D-2D point pairs of point clouds and images, estimate the transformation matrix from the given 3D-2D point pairs using the PnP algorithm, and complete the registration process.

[0061] The beneficial effects of the present invention are:

[0062] 1. The cross-modal registration framework proposed in this paper uses distance map and reflectivity map features to reduce domain differences, realizes cross-modal registration between lidar point cloud and camera image, and the registration accuracy can reach more than 99% on the KITTI dataset.

[0063] 2. Aiming at the cross-modal differences between the projection image and the camera image in the registration task, the present invention proposes a multi-scale feature extraction network that can efficiently extract features of the image and two different types of projection images at the same time.

[0064] 3. The feature matching framework from patch to pixel proposed in the present invention not only solves the supervision problem caused by the limited overlapping area between the projection image and the camera image, but also significantly improves the registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flowchart of the implementation of the cross-modal registration method of laser radar and camera images based on patch to pixel in the present invention;

[0066] Figure 2 Schematic diagram of a feature extraction network of the present invention;

[0067] Figure 3 Schematic diagram of the patch-to-pixel matching process of the present invention. DETAILED DESCRIPTION

[0068] The present invention will be further described in detail below with reference to the embodiments given in the accompanying drawings.

[0069] Example 1

[0070] An efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel, such as Figure 1 As shown, including:

[0071] Step 1: Perform data preprocessing and project the point cloud into two projection images, including reflectivity image and distance image;

[0072] Step 2: Input the projection image obtained by the lidar point cloud and the camera image taken by the high-definition camera into the convolutional neural network built in Python; the convolutional neural network outputs the patch-level features and pixel-level features of the projection image and camera image for feature matching;

[0073] Step 3: Use the patch-to-pixel matching network to perform feature matching. The patch-level features first estimate the patch matching pairs, and then use the estimated patch matching pairs to extract the pixel-level features corresponding to each patch from the pixel-level features; the extracted pixel-level features are used for pixel-level matching to estimate the pixel matching pairs of the projection image and the camera image;

[0074] Step 4: Convert the pixel matching point pairs into 3D-2D point pairs of point cloud and image, and use the PnP (Perspective-n-Point) algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process.

[0075] Example 2

[0076] According to the efficient cross-modal registration method of laser radar and camera images based on patch to pixel described in Example 1, the difference is that: in the field of autonomous driving, commonly used laser radar systems usually adopt a multi-line rotation scanning design. The performance of the laser radar is closely related to the number of its beams, and more beams mean better object detection effects. The number of beams is usually expressed in terms of the number of laser heads, including different specifications such as 16 lines, 32 lines, 64 lines, and 128 lines. Its working principle is to obtain data on the surrounding environment by rotating the laser head within a range of 360°. Taking a 64-line laser radar as an example, it will generate 64 circles of point cloud data, which are stored in sequence from top to bottom. Therefore, according to this storage characteristic of the point cloud, it can be divided into 64 rows, where the number of each row is the LaserID of the point cloud in that row.

[0077] The traditional FOV spherical projection method based on point cloud depth will cause thread misalignment during the projection process, that is, the point cloud data of two adjacent lines are too close in the z-axis direction, resulting in the point cloud data of one thread being covered and the data of the other thread being missing during projection. This will cause holes in the projection image, and data needs to be supplemented by interpolation. In contrast, the LaserID-based projection method can effectively eliminate the point cloud position deviation caused by the jitter of the lidar sensor, so there is no need for interpolation and completion, which improves the precision and accuracy of the projection image. At the same time, in terms of imaging principles, the reflectivity information will be closer to the image data obtained by the camera than the point cloud depth information. Finally, the LaserID-based projection method is used to project the reflectivity of the point cloud to obtain a reflectivity map. The point cloud is projected into two projection maps, including a reflectivity map and a distance map; including:

[0078] 1) Use the spherical projection principle to transform the point cloud coordinates; transform the point cloud from the Cartesian coordinate system to the spherical coordinate system, and calculate the azimuth according to the coordinates (x, y, z) of the point cloud Azimuth The calculation formula is: The value range of is [-π, π];

[0079] 2) After obtaining the LaserID of the point cloud, generate a projection map and record the mapping. Generally, the laser radar will record the LaserID information of each point when collecting the point cloud, combined with the azimuth obtained above The projection image pixel coordinates [u, v] corresponding to the point cloud are obtained in the following way: Taking a 64-line laser radar as an example, assuming that the projection image resolution is set to 64×1024, LaserID is used as the projection coordinate of the point in the height direction, that is, u=LaserID, and the width direction is proportionally projected according to the 1024 resolution. The calculation formula is: From the above, the pixel coordinates of the projection image corresponding to the point cloud are: For the reflectivity map, the reflectivity of the point cloud is recorded in the pixel, that is, the gray value of each pixel represents the reflectivity of the point cloud falling on the pixel; for the distance map, the radius r of the point cloud in the spherical coordinate system is recorded in the pixel, and the radius x, y, and z are the coordinate values ​​of the point cloud on the X, Y, and Z axes in the Cartesian coordinate system.

[0080] After completing data preprocessing, two projection images, reflectivity map and distance map, are obtained. The projection images will be sent to the subsequent feature extraction network together with the camera image.

[0081] The projection image and camera image are input into the convolutional neural network built in Python; the convolutional neural network outputs the patch-level features and pixel-level features of the projection image and camera image; including:

[0082] Different from other image registration tasks, due to the cross-modal differences between the projection image and the camera image, this paper designs an image feature extraction network based on deep learning in the feature extraction part to solve this problem. The detailed structure of the network is as follows: Figure 2 The feature extraction network, i.e. the convolutional neural network built in Python, is divided into two branches. The first branch is used to process the reflectivity map and the distance map, and the second branch is used to process the camera image. Each branch will extract patch-level features with a scale of 1 / 4 of the original image and pixel-level features with the same scale as the original image.

[0083] The feature extraction network uses a convolutional neural network (CNN) to process camera images and projection images. The first half of the feature extraction network is the image encoder part, and the second half of the feature extraction network is the image decoder part. The scale of the features in the image encoder part is gradually downsampled, and the image encoder part finally generates a high-dimensional feature at 1 / 32 of the original image scale; the high-dimensional feature is passed to the image decoder part, and in the image decoder part, the scale of the feature is gradually upsampled; specifically, it includes:

[0084] First, obtain the patch-level features at 1 / 4 the size of the original image. Among them, D Patch is the number of Patch-level feature channels, H and W are the height and width of the original image, Refers to the set of real numbers in mathematics; then, through two layers of deconvolution, the pixel-level features at the original image resolution are obtained Among them, D pixel is the number of pixel-level feature channels.

[0085] like Figure 2 As shown in the figure, in the second branch (camera image branch), the RGB three-channel camera image is input into the convolutional neural network. After five layers of full convolutional networks, 128 channels of small-scale (1 / 32 scale of the original camera image) features are obtained. Then, the small-scale features are upsampled by deconvolution. After three layers of deconvolution, 256 channels of Patch-level features with a scale of 1 / 4 of the original camera image are obtained. The Patch-level features are further upsampled by deconvolution. After two layers of deconvolution, the scale is restored to the original image scale. At this time, a pixel-level feature with a channel number of 256 and the same scale as the original camera image is obtained.

[0086] In the first branch (reflectivity map & distance map branch), the same CNN architecture is used for the projection map feature extraction branch. However, unlike the camera image feature extraction, two identical encoder branches are used separately for the distance map and the reflectivity map. The features extracted by the two identical encoder branches are connected together and passed to the projection map decoder network for upsampling, and a patch-level feature and a pixel-level feature are also obtained. Specifically, for the two inputs of the projection map branch, the reflectivity map and the distance map, first, the reflectivity map and the distance map are subjected to feature extraction by a five-layer fully convolutional network to obtain a small-scale feature of 128 channels. Then, the two small-scale features are spliced, and the number of channels of the spliced ​​feature is 256, and the scale remains unchanged. Thirdly, the spliced ​​feature is subjected to a full convolution operation again to obtain a feature with a scale that remains unchanged and the number of channels is 128. The obtained features are deconvolved and upsampled. After three layers of deconvolution, the number of channels is 256 and the scale is the same as the original projection. Figure 1 / 4 scale of Patch-level features; the Patch-level features are deconvolved and upsampled twice again to obtain pixel-level features with the same scale as the original projection image and 256 channels;

[0087] At this point, the patch-level features and pixel-level features of the camera image and projection image are obtained.

[0088] The next step is to use these features of different scales for feature matching.

[0089] In order to solve the problem of scarce positive samples due to the limited overlap between camera images and point clouds, we first search at the patch level, and then perform pixel-level matching within the matched patches. This matching strategy not only increases the proportion of positive samples through patch-level search and solves the network supervision problem, but also significantly improves the registration accuracy through pixel-level matching.

[0090] The structure of the matching network is as follows: Figure 3 shown.

[0091] Use the patch-to-pixel matching network to perform feature matching; including:

[0092] First, the patch-level features extracted by the feature extraction network are input into the patch matching module for patch-level matching; the patch matching module is similar to the optimized network structure in Lightglue (see the paper 'LightGlue: Local Feature Matching at Light Speed' for details). It includes:

[0093] For two patch-level features that need to be matched, first flatten the two patch-level features and input them into a linear layer neural network with the same number of channels and output channels. The number and scale of the output feature channels are the same as the input patch-level features. Then, the two output features are matrix multiplied (the multiplied matrix is ​​transposed first) to obtain the patch similarity matrix S. Finally, the matrix S is subjected to row-by-row and column-by-column Softmax operations to obtain the matrix after row Softmax processing and the matrix after column Softmax processing, respectively. The Hadamard product of the two matrices is calculated to obtain the distribution matrix P, which is described by the following formula:

[0094]

[0095] Among them, P ij represents the value of the i-th row and j-th column in the allocation matrix P, S ij Represents the value of the i-th row and j-th column in the allocation matrix S, N 2D is the total number of patches in the camera image, that is, the total number of rows in the S matrix; N 3D is the total number of patches in the projected image, that is, the total number of columns in the S matrix;

[0096] Select Top-k from the allocation matrix P to obtain the matching relationship between the projection image and the camera image patch-level features, namely the patch matching pair; specifically, for all the values ​​in the allocation matrix, we arrange them from large to small and select the largest k. The specific value of this k is defined by ourselves, and the rows and columns in the matrix corresponding to the selected values ​​are the patch matching pairs. The patch feature scale is 1 / 4 of the original image, and each patch corresponds to a 4×4 image area, containing 16 pixels. Therefore, each patch of the selected patch matching pair contains the features of 16 pixels in the actual image area, and we have previously extracted pixel-level features with the same scale as the original image in the feature extraction network. For pixel-level features, we separately extract the pixel-level features of the corresponding area in the patch in each matching pair, and the pixel-level features of the area that is not selected as a matching patch matching pair are discarded. This step is Figure 3 We call it feature screening.

[0097] Next, the pixel-level features selected from the patch matching pairs are used for pixel-level matching, including:

[0098] For each patch matching pair, all pixels in it are sent to the patch matching module for matching, and only the highest similarity value is retained, that is, only the highest item in Top-k is taken; that is, in each matching patch pair, a pair of matching pixels is selected by the patch matching module. By performing pixel-level matching on each patch matching pair, the final pixel matching pair is obtained.

[0099] The calculation formula of the patch similarity matrix S is as follows:

[0100] The projection image patch-level features and the camera image patch-level features are first flattened and sent to the linear layer. The two matrices are multiplied by matrix multiplication to obtain the score matrix S.

[0101]

[0102] Among them, f C and f L These are the two matrices output by the two linear layers respectively. Faltten is the flattening operation and Linear is the linear layer operation.

[0103] Convert pixel matching point pairs into 3D-2D point pairs of point clouds and images, and use the PnP algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process; including:

[0104] Through pixel matching pairs, find the points in the point cloud corresponding to the pixels on the projection image, and then establish the 3D-2D correspondence between the projected 3D point cloud and the camera image pixels, that is, the 3D-2D point pairs of the point cloud and the image;

[0105] The EPnP algorithm is used to solve the pose and finally estimate the transformation matrix. It includes: finding the point in the point cloud corresponding to each pixel through the projection graph, so the pixel matching pairs obtained by pixel matching can obtain the 3D-2D point pairs between the point cloud and the image. For the EPnP algorithm, in the specific implementation, first, the minimum point set is randomly sampled from the 3D-2D correspondence pair through the RANSAC framework; then, the EPnP algorithm is used to represent the 3D point as the weighted sum of 4 control points, and a linear equation system is constructed according to the 2D projection relationship, and the coordinates of the control points in the camera coordinate system are solved by singular value decomposition (SVD); then, the rotation matrix R and the translation vector t are calculated by combining the coordinates of the control points in the world coordinate system to obtain the transformation matrix T; finally, the matching accuracy of the point pair is evaluated by the reprojection error, the internal points are screened and iteratively optimized, and the optimal transformation matrix T is output.

[0106] In convolutional neural networks and matching networks, the total loss function is:

[0107] L total =L patch+L pixel ;

[0108] In the design of the loss function, the LiDAR point cloud is projected onto the camera image using the real external parameters provided in the dataset to obtain the actual 3D-2D correspondence. Assuming that we have the actual correspondence It contains M corresponding items, and the actual corresponding relationship Each value in is divided by 4 and rounded to get the true Patch correspondence For each true patch correspondence, the corresponding position value in the patch assignment matrix P should be as large as possible. Based on this idea, the first part of the loss function L is designed. Patch , the first part of the loss function L patch The expression is as follows:

[0109]

[0110] Among them, (i, j) refers to the value of each corresponding item in the corresponding relationship, P ij It refers to the value corresponding to the i-th row and j-th column in the allocation matrix P;

[0111] During the training process, the matching network of the present invention does not use top-k, but directly uses the actual patch correspondence for subsequent pixel-level matching. For each patch correspondence, the matching network calculates M allocation matrices p; Divide by 4 and take the remainder to obtain the true pixel correspondence in each assignment matrix p Thus we get the second part of the loss function L Pixel , the second part of the loss function L pixel The expression is as follows:

[0112]

[0113] Among them, p i represents the allocation matrix calculated by the matching module for the i-th pair of patches, represents the i-th true pixel correspondence, Indicates p i The value at the corresponding position in the matrix.

[0114] The present invention conducts experiments on KITTI odometry (see the paper 'Are we ready for autonomous driving? The KITTI vision benchmark suite' for details) and nuScenes (see the paper 'nuScenes: A multimodal dataset for autonomous driving' for details) datasets.

[0115] KITTIOdometry: We generate image-point cloud pairs from the same data frame of a 2D / 3D sensor. Following the settings of other related work, sequences 00-08 are used for training and sequences 09-10 are used for testing. The camera image resolution is downsampled to 512×160 and the reflectivity map resolution is 1024×64. A 2D translation of ±10m (along the x and y axes) and an arbitrary rotation around the z axis are applied to the point cloud during training and testing. To ensure authenticity, we first apply a rotation around the z axis to the point cloud, and then apply a translation on the x and y axes.

[0116] nuScenes: We selected 850 scenes for training and 150 scenes for testing. The camera image resolution was downsampled to 320×160 and the reflectivity map resolution was 1024×32. We applied a 2D translation of ±10m (along the x and y axes) and an arbitrary rotation around the z axis to the point cloud, in the same way as KITTI.

[0117] The present invention is compared with the following three methods: DeepI2P, CorrI2P and VP2P-Match.

[0118] DeepI2P (see the paper 'DeepI2P: Image-to-point cloud registration via deepclassification' for details), this method includes two strategies: Grid Cls.+PnP and Frus.Cls.+Inv.Proj. GridCls.+PnP first divides the input image into a 32×32 grid, trains a neural network to classify 3D points into 2D grid cells, and estimates the transformation matrix through EPnP and RANSAC. Frus.Cls.+Inv.Proj introduces cone-based classification, combined with inverse camera projection, and is divided into DeepI2P(2D) and DeepI2P(3D).

[0119] CorrI2P (see the paper 'CorrI2P: Deep image-to-point cloud registration via dense correspondence') is based on DeepI2P. It improves the registration accuracy between point clouds and image pairs by introducing feature exchange modules and overlapping area supervision.

[0120] VP2P-Match (see the paper 'Differentiable registration of images and lidarpoint clouds with voxelpoint-to-pixel matching' for details) uses sparse convolution to enhance the similarity between point cloud features and image CNN extracted features, while combining a differentiable PnP solver in an end-to-end framework to improve matching accuracy and speed.

[0121] The transformation matrix can usually be decomposed into two separate parts: the rotation matrix R and the translation matrix T. Therefore, the standard for measuring the registration accuracy is the relative translation error (RTE) E T and relative rotation error (RRE)E R , and then evaluate the accuracy of the registration:

[0122]

[0123] E T =||T gt -T E ||;

[0124] Here γ is the inverse matrix of the true value of the rotation matrix The predicted rotation matrix predicted value R is obtained by right multiplying The corresponding value after conversion to Euler angle. Similarly, the true value of the translation matrix T …f Subtract the predicted value T and calculate the L2 norm to get E T This embodiment counts the mean and standard deviation of all data. At the same time, the registration success rate Acc is counted, and the standard for successful registration is defined as RTE < 2m and RRE < 5°. The results are shown in Table 1:

[0125] Table 1

[0126]

[0127] It can be seen from Table 1 that the method of the present invention is superior to the existing methods in both registration accuracy and precise registration success rate.

[0128] Example 3

[0129] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the cross-modal registration method of laser radar and camera images based on patch to pixel described in embodiment 1 or 2 are implemented.

[0130] Example 4

[0131] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the cross-modal registration method of laser radar and camera images based on patch to pixel as described in Example 1 or 2.

[0132] Example 5

[0133] An efficient cross-modal registration system for LiDAR and camera images based on patch-to-pixel, including:

[0134] The data preprocessing module is configured to: perform data preprocessing and project the point cloud into two projection images, including a reflectivity image and a distance image;

[0135] The patch-level feature and pixel-level feature acquisition module is configured to: input the projection image obtained by the lidar point cloud and the camera image taken by the high-definition camera into the convolutional neural network built by the Python language; the convolutional neural network outputs the patch-level features and pixel-level features of the projection image and the camera image for feature matching;

[0136] The feature matching module is configured to: use a patch-to-pixel matching network to perform feature matching, first estimate patch matching pairs from patch-level features, then extract pixel-level features corresponding to each patch from pixel-level features using the estimated patch matching pairs; use the extracted pixel-level features for pixel-level matching to estimate pixel matching pairs of the projection image and the camera image;

[0137] The registration module is configured to: convert pixel matching point pairs into 3D-2D point pairs of point clouds and images, estimate the transformation matrix from the given 3D-2D point pairs using the PnP (Perspective-n-Point) algorithm, and complete the registration process.

[0138] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An efficient cross-modal registration method for LiDAR and camera images based on Patch to Pixel, characterized in that: include: Step 1: Perform data preprocessing and project the point cloud into two projection images, including reflectivity image and distance image; Step 2: Input the projection image and camera image into the convolutional neural network built in Python; the convolutional neural network outputs the patch-level features and pixel-level features of the projection image and camera image; Step 3: Use the patch-to-pixel matching network to perform feature matching. The patch-level features first estimate the patch matching pairs, and then use the estimated patch matching pairs to extract the pixel-level features corresponding to each patch from the pixel-level features; The extracted pixel-level features are used for pixel-level matching to estimate the pixel matching pairs of the projection image and the camera image; Step 4: Convert the pixel matching point pairs into 3D-2D point pairs of point cloud and image, and use the PnP algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process.

2. According to claim 1, an efficient cross-modal registration method for laser radar and camera images based on patch to pixel, characterized in that: Project the point cloud into two projection images, including reflectivity map and distance map; including: 1) Use the spherical projection principle to transform the point cloud coordinates; transform the point cloud from the Cartesian coordinate system to the spherical coordinate system, and calculate the azimuth according to the coordinates (x, y, z) of the point cloud Azimuth The calculation formula is: The value range of is [-π, π]; 2) After obtaining the LaserID of the point cloud, generate a projection map and record the mapping. The pixel coordinates of the projection map corresponding to the point cloud are: For the reflectivity map, the reflectivity of the point cloud is recorded in the pixel, that is, the gray value of each pixel represents the reflectivity of the point cloud falling on the pixel; for the distance map, the radius r of the point cloud in the spherical coordinate system is recorded in the pixel, and the radius x, y, and z are the coordinate values ​​of the point cloud on the X, Y, and Z axes in the Cartesian coordinate system.

3. According to claim 1, an efficient cross-modal registration method for laser radar and camera images based on patch to pixel, characterized in that: The projection image and camera image are input into the convolutional neural network built in Python language; the convolutional neural network outputs the patch-level features and pixel-level features of the projection image and camera image; including: The feature extraction network, i.e., the convolutional neural network built in Python, is divided into two branches. The first branch is used to process the reflectivity map and the distance map, and the second branch is used to process the camera image. Each branch will extract patch-level features with a scale of 1 / 4 of the original image and pixel-level features with the same scale as the original image. The first half of the feature extraction network is the image encoder part, and the second half of the feature extraction network is the image decoder part. The scale of the features in the image encoder part is gradually downsampled, and the image encoder part finally generates a high-dimensional feature at 1 / 32 of the original image scale; the high-dimensional feature is passed to the image decoder part, and in the image decoder part, the scale of the feature is gradually upsampled; specifically, it includes: First, obtain the patch-level features at 1 / 4 the size of the original image Among them, D Patch is the number of Patch-level feature channels, H and W are the height and width of the original image, Refers to the set of real numbers in mathematics; then, through two layers of deconvolution, the pixel-level features at the original image resolution are obtained Among them, D pixel is the number of pixel-level feature channels; In the second branch, the RGB three-channel camera image is input into the convolutional neural network. After five layers of full convolutional networks, 128-channel small-scale features are obtained. Then, the small-scale features are upsampled by deconvolution. After three layers of deconvolution, 256-channel patch-level features with a scale of 1 / 4 of the original camera image are obtained. The patch-level features are further upsampled by deconvolution. After two layers of deconvolution, the scale is restored to the original image scale. At this time, a pixel-level feature with a channel number of 256 and the same scale as the original camera image is obtained. In the first branch, firstly, the reflectivity map and the distance map are subjected to feature extraction of a five-layer full convolutional network to obtain a small-scale feature of 128 channels. Then, the two small-scale features obtained are spliced, and the number of channels of the spliced ​​feature is 256, and the scale remains unchanged. Then, the spliced ​​feature is subjected to a full convolution operation again, and the obtained feature is deconvolved and upsampled. After three layers of deconvolution, a patch-level feature with a channel number of 256 and a scale of 1 / 4 of the original projection image is obtained. The patch-level feature is deconvolved and upsampled twice again to obtain a pixel-level feature with the same scale as the original projection image and a channel number of 256. At this point, the patch-level features and pixel-level features of the camera image and projection image are obtained.

4. The efficient cross-modal registration method of laser radar and camera images based on patch-to-pixel according to claim 1, characterized in that: Use the patch-to-pixel matching network to perform feature matching; including: First, the patch-level features extracted by the feature extraction network are input into the patch matching module for patch-level matching; including: For two patch-level features that need to be matched, first flatten the two patch-level features and input them into a linear layer neural network with the same number of channels and output channels. The number and scale of the output feature channels are the same as the input patch-level features. Then, the two output features are matrix multiplied to obtain the patch similarity matrix S. Finally, the matrix S is subjected to row-by-row and column-by-column Softmax operations to obtain the matrix after row Softmax processing and the matrix after column Softmax processing, respectively. The Hadamard product of the two matrices is calculated to obtain the distribution matrix P, which is described by the following formula: Among them, P <＝ represents the value of the i-th row and j-th column in the allocation matrix P, D <＝ Represents the value of the i-th row and j-th column in the allocation matrix S, N 22 is the total number of patches in the camera image, that is, the total number of rows in the S matrix; N R2 is the total number of patches in the projected image, that is, the total number of columns in the S matrix; Select Top-k from the allocation matrix P to obtain the matching relationship between the projection image and the camera image patch-level features, i.e., the patch matching pair; then, the pixel-level features selected from the patch matching pair are used for pixel-level matching; including: For each patch matching pair, all pixels in it are sent to the patch matching module for matching, and only the highest similarity value is retained, that is, only the highest item in Top-k is taken; by performing pixel-level matching on each patch matching pair, the final pixel matching pair is obtained.

5. The efficient cross-modal registration method of laser radar and camera images based on patch-to-pixel according to claim 4, characterized in that: The calculation formula of the patch similarity matrix S is as follows: Among them, f Z and f [ These are the two matrices output by the two linear layers respectively. Faltten is the flattening operation and Linear is the linear layer operation.

6. The efficient cross-modal registration method of laser radar and camera images based on patch-to-pixel according to claim 1, characterized in that: Convert pixel matching point pairs into 3D-2D point pairs of point clouds and images, and use the PnP algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process; including: Through pixel matching pairs, find the points in the point cloud corresponding to the pixels on the projection image, and then establish the 3D-2D correspondence between the projected 3D point cloud and the camera image pixels, that is, the 3D-2D point pairs of the point cloud and the image; The EPnP algorithm is used to solve the pose and finally estimate the transformation matrix.

7. An efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel according to any one of claims 1-6, characterized in that: In convolutional neural networks and matching networks, the total loss function is: L ]^]_` =L a_]bc +L d<ef` ; Assuming that there is an actual correspondence It contains M corresponding items, and the actual corresponding relationship Each value in is divided by 4 and rounded to get the true Patch correspondence The first part of the loss function L a_]bc The expression is as follows: Among them, (i, j) refers to the value of each corresponding item in the corresponding relationship, P <＝ It refers to the value corresponding to the i-th row and j-th column in the allocation matrix P; For each Patch correspondence, the matching network calculates M allocation matrices p; by Divide by 4 and take the remainder to obtain the true pixel correspondence in each assignment matrix p Thus we get the second part of the loss function L a<ef` , the second part of the loss function L d<ef` The expression is as follows: Among them, p i represents the allocation matrix calculated by the matching module for the i-th pair of patches, represents the i-th true pixel correspondence, Indicates p i The value at the corresponding position in the matrix.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the cross-modal registration method of lidar and camera images based on patch to pixel described in claims 1-7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cross-modal registration method of lidar and camera images based on patch to pixel are implemented in claims 1-7.

10. An efficient cross-modal registration system for LiDAR and camera images based on patch-to-pixel, characterized in that: include: The data preprocessing module is configured to: perform data preprocessing and project the point cloud into two projection images, including a reflectivity image and a distance image; The patch-level feature and pixel-level feature acquisition module is configured to: input the projection image and the camera image into the convolutional neural network constructed by the Python language; the convolutional neural network outputs the patch-level features and pixel-level features of the projection image and the camera image; The feature matching module is configured to: perform feature matching using a patch-to-pixel matching network, first estimate patch matching pairs from the patch-level features, and then extract the pixel-level features corresponding to each patch from the pixel-level features using the estimated patch matching pairs; The extracted pixel-level features are used for pixel-level matching to estimate the pixel matching pairs of the projection image and the camera image; The registration module is configured to: convert pixel matching point pairs into 3D-2D point pairs of point clouds and images, estimate the transformation matrix from the given 3D-2D point pairs using the PnP algorithm, and complete the registration process.

Images