A method and system for cross-modal registration of LiDAR and camera images based on patch-to-pixel mapping.

Abstract

This invention relates to a cross-modal registration method and system for LiDAR and camera images based on patch-to-pixel mapping, comprising: Step 1: performing data preprocessing, projecting the point cloud into two projection maps, including a reflectance map and a distance map; Step 2: acquiring patch-level features and pixel-level features of the projection maps and camera images; Step 3: using a patch-to-pixel matching network to perform feature matching, estimating pixel matching pairs between the projection maps and camera images; Step 4: converting the pixel matching point pairs into 3D-2D point pairs of the point cloud and images, estimating the transformation matrix from the given 3D-2D point pairs using the PnP algorithm, and completing the registration process. This invention achieves cross-modal registration between LiDAR point clouds and camera images, with a registration accuracy of over 99% on the KITTI dataset.

Description

Technical Field

[0001] This invention relates to a cross-modal registration method and system for LiDAR and camera images based on patch-to-pixel mapping, belonging to the field of image processing technology. Background Technology

[0002] Advanced perception systems are required in autonomous driving and robotics systems to acquire accurate and comprehensive information about the surrounding environment. Research and development of perception systems are crucial for advancing autonomous driving technology. By continuously improving the accuracy, stability, and adaptability of perception systems, safer and more efficient intelligent driving vehicles can be achieved, bringing enormous potential benefits to society. Sensors, as the most reliable data source in perception systems, can provide real-time information on road conditions, obstacle locations, and motion states through data fusion from different sensors. Data fusion from different sensors plays a vital role in many downstream tasks, such as target detection and segmentation. By utilizing this fused data, accuracy and efficiency can be improved, thereby ensuring safety and robustness in complex environments. Among these methods, the most common process is fusing point clouds from LiDAR (such as solid-state or rotating LiDAR) with images from cameras (such as high-definition or infrared cameras).

[0003] However, successful fusion relies on accurate alignment of these data. A common approach to aligning sensors is to calculate extrinsic parameters using a checkerboard calibration method, known as offline calibration. These methods allow for high-precision alignment between sensors, but at the cost of unavoidable time complexity. Furthermore, some online calibration methods have been proposed to correct extrinsic parameter errors caused by collisions or vibrations. These online methods can correct deviations within a limited error range, but their effectiveness diminishes when the error becomes too large. While these methods can achieve high-precision alignment, the complex calibration process makes data fusion preprocessing time-consuming and inefficient. These processes must be repeated whenever sensor deployment changes.

[0004] Therefore, how to propose a method to bypass these calibration processes and directly register the acquired data has become an urgent problem to be solved. To this end, cross-modal registration methods have emerged. Recent cross-modal registration methods have solved this cross-modal data alignment task by using specific deep learning networks. However, the cross-modal registration process typically involves comparing the high-dimensional feature similarity between point clouds and images to establish correspondences, and the domain gap between 3D and 2D data always affects this process. Specifically, there are dimensional and informational differences between the 3D spatial data of point clouds and the 2D texture information of images. Although some existing methods attempt to reduce this impact by exchanging feature information or using feature extraction networks with more similar structures, the difference in data dimensionality still affects the accuracy of registration, especially leading to a high matching failure rate.

[0005] Furthermore, point cloud feature extraction networks commonly used in cross-modal registration tasks (such as PointNet++ and sparse convolutional networks) typically treat LiDAR point clouds simply as a series of unordered points or voxels, failing to effectively utilize the spatial and geometric features of point clouds. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide an efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel mapping. This method projects the LiDAR point cloud into two types of projection maps: a distance map and a reflectance map, using geometric and reflectance information. This transforms the cross-modal registration task into an image registration task. A deep learning network is used to extract features from both the projection maps and the camera image. Feature similarity is then used to match the LiDAR point cloud and the camera image in a high-dimensional feature space, thus finding the matching relationship between them. Finally, the transformation matrix between the two is estimated.

[0007] To better leverage the scanning characteristics of LiDAR to bridge the domain gap, this invention proposes a method for projecting point clouds onto 2D representations: a distance map and a reflectance map. Simultaneously, this invention designs a multi-scale feature extraction network to mitigate cross-modal differences between the image and the projected map, effectively extracting features. Furthermore, this invention introduces a patch-to-pixel feature matching framework. Patch-level search addresses the supervision and success rate issues caused by the scarcity of positive samples, while pixel-level matching improves registration accuracy. Experimental results show that the network design of this invention achieves high-precision registration while maintaining low computational complexity.

[0008] In this invention, firstly, the point cloud is projected through data preprocessing. This approach of mapping the point cloud to pixels in a 2D representation provides a data format that implicitly contains more geometric information, making it more efficient and accurate compared to using unordered sparse point clouds. After projection, the cross-modal registration task is transformed into image registration between the camera image and the projected image. Common image registration methods typically achieve this through keypoint matching. However, due to the cross-modal differences between the projected 2D image and the camera image, traditional keypoint registration methods cannot be directly applied to this task. Furthermore, the overlap between the 2D image and the camera image is usually very limited in real-world scenarios, resulting in a very small proportion of truly matched point pairs and a scarcity of positive samples. For existing deep learning-based image matching methods, this scarcity can lead to supervision difficulties, making it difficult for the network to converge and increasing the number of outliers, thereby increasing the likelihood of registration failure. To address this issue, this invention proposes a patch-to-pixel matching framework. Patch-level matching solves the supervision problem while ensuring the robustness of the network, while pixel-level matching improves the accuracy of registration.

[0009] The present invention also provides a cross-modal registration system for LiDAR and camera images based on patch to pixel.

[0010] Terminology Explanation:

[0011] 1. LaserID is a unique identifier assigned to each laser transmitter-receiver unit (laser head) in a rotating lidar system. For example, a rotating lidar system may have 64 laser heads, with LaserIDs ranging from 0 to 64. Each LaserID corresponds to the scan data of a specific laser head. Each laser identifier corresponds to a fixed pitch angle position within the lidar's scanning range and generates a single line of point cloud data during active imaging.

[0012] 2. PnP (Perspective-n-Point): PnP is a method for solving the 3D-to-2D point mapping problem. It describes how to estimate the camera pose when n 3D spatial points and their positions are known. PnP is based on the pinhole camera model and solves for pose parameters by minimizing the error of the 3D points projected onto the image plane. If the 3D position of a feature point in one of two images is known, then at least three point pairs (and at least one additional verification point to verify the result) are needed to calculate the camera motion. There are many algorithms for solving the PnP problem, such as the EPnP algorithm (see the paper "EPnP: Efficient Perspective-n-Point Camera Pose Estimation").

[0013] 3. Softmax: Softmax is a commonly used mathematical function, primarily used in multi-class classification tasks, to transform a set of real numbers into a probability distribution. It expands the input values ​​to positive numbers using an exponential function and normalizes them to values ​​between 0 and 1, ensuring that the sum of all outputs equals 1, conforming to the definition of probability. The expression for the Softmax function is:

[0014]

[0015] Among them, z i It is the i-th element in the input vector, and N is the dimension of the input vector.

[0016] 4. Top-k: Top-k refers to the operation of selecting the top k elements in a set of data, typically used for sorting, filtering, or evaluation tasks. It is particularly common in deep learning and information retrieval. For example, in classification problems, Top-k accuracy is used to evaluate whether the model's predictions of the top k classes contain the correct answer. The Top-k method determines the top k highest values ​​or rankings by comparing the scores (such as probability or similarity) of each element.

[0017] The technical solution of this invention is as follows:

[0018] An efficient cross-modal registration method based on patch-to-pixel LiDAR and camera images includes:

[0019] Step 1: Perform data preprocessing, projecting the point cloud into two types of projection maps, including a reflectance map and a distance map;

[0020] Step 2: Input the projection image and camera image into a convolutional neural network built with Python; the convolutional neural network outputs patch-level and pixel-level features of the projection image and camera image.

[0021] Step 3: Use a patch-to-pixel matching network to perform feature matching. First, estimate the patch matching pairs. Then, use the estimated patch matching pairs to extract the pixel-level features corresponding to each patch from the pixel-level features. Use the extracted pixel-level features for pixel-level matching to estimate the pixel matching pairs between the projection map and the camera image.

[0022] Step 4: Convert the pixel matching point pairs into point cloud and image 3D-2D point pairs, and use the PnP (Perspective-n-Point) algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process.

[0023] According to a preferred embodiment of the present invention, the point cloud is projected into two projection maps, including a reflectance map and a distance map; including:

[0024] 1) Perform coordinate transformation on the point cloud using the principle of spherical projection; transform the point cloud from the Cartesian coordinate system to the spherical coordinate system, and calculate the azimuth angle based on the point cloud's coordinates (x, y, z). Azimuth The calculation formula is: The range of values ​​for is [-π, π].

[0025] 2) After obtaining the LaserID of the point cloud, generate a projection map and record the mapping. The pixel coordinates of the projection map corresponding to the point cloud are: For reflectance maps, the reflectance of the point cloud is recorded within pixels; that is, the gray value of each pixel represents the reflectance of the point cloud falling on that pixel. For distance maps, the radius *r* of the point cloud in the spherical coordinate system is recorded within pixels. x, y, and z are the coordinates of the point cloud on the X, Y, and Z axes in the Cartesian coordinate system.

[0026] According to a preferred embodiment of the present invention, the projection image and the camera image are input into a convolutional neural network constructed using the Python language; the convolutional neural network outputs patch-level features and pixel-level features of the projection image and the camera image; including:

[0027] The feature extraction network, a convolutional neural network built with Python, is divided into two branches. The first branch processes reflectance maps and distance maps, while the second branch processes camera images. Each branch extracts patch-level features at 1 / 4 the scale of the original image, as well as pixel-level features at the same scale as the original image.

[0028] The first half of the feature extraction network is the image encoder, and the second half is the image decoder. The image encoder progressively downsamples the features, ultimately generating a high-dimensional feature at 1 / 32 of the original image scale. This high-dimensional feature is then passed to the image decoder, where the features are progressively upsampled. Specifically, this includes:

[0029] First, obtain patch-level features at 1 / 4 the size of the original image. Among them, D Patch This represents the number of feature channels at the patch level, where H and W are the height and width of the original image. The set of real numbers in mathematical notation is used; then, through two layers of deconvolution, pixel-level features at the original image resolution are obtained. Among them, D pixel It is the number of feature channels at the pixel level.

[0030] In the second branch, the RGB three-channel camera image is input into the convolutional neural network. After five layers of fully convolutional network, 128 channels of small-scale features are obtained. Then, deconvolution is used to upsample the small-scale features. After three layers of deconvolution, 256 channels of patch-level features at 1 / 4 scale of the original camera image are obtained. The patch-level features are then upsampled again using deconvolution. After two layers of deconvolution, the scale is restored to the original image scale. At this point, 256 channels of pixel-level features with the same scale as the original camera image are obtained.

[0031] In the first branch, feature extraction is first performed on the reflectance map and distance map using a five-layer fully convolutional network, resulting in 128-channel small-scale features. Then, the two small-scale features are concatenated, resulting in 256 channels with the scale unchanged. Next, the concatenated features are subjected to another fully convolutional operation, followed by deconvolutional upsampling. After three deconvolutional layers, the resulting feature has 256 channels and the scale is the original projection. Figure 1 / 4 scale Patch-level features; the Patch-level features are then deconvolved and upsampled twice to obtain pixel-level features with the same scale as the original projection image and 256 channels.

[0032] At this point, the respective Patch-level and pixel-level features of the camera image and the projection map have been obtained.

[0033] According to a preferred embodiment of the present invention, feature matching is performed using a patch-to-pixel matching network; including:

[0034] First, the patch-level features extracted by the feature extraction network are input into the patch matching module for patch-level matching; this includes:

[0035] For two patch-level features that need to be matched, firstly, the two patch-level features are flattened and then input into a linear neural network with the same number of channels as the output. The number of channels and scale of the output features are the same as the input patch-level features. Then, the two output features are multiplied to obtain the patch similarity matrix S. Finally, softmax operations are performed on matrix S row by row and column by column to obtain the matrix after row softmax processing and the matrix after column softmax processing, respectively. The Hadamard product of the two matrices is then taken to obtain the assignment matrix P, which is described by the following formula:

[0036]

[0037] Among them, P ij S represents the value in the i-th row and j-th column of the allocation matrix P. ij N represents the value in the i-th row and j-th column of the allocation matrix S.2D N represents the total number of camera image patches, which is the total number of rows in matrix S; 3D This represents the total number of patches in the projected image, which is the total number of columns in the S matrix.

[0038] The Top-k features are selected from the allocation matrix P to obtain the matching relationship between the patch-level features of the projection map and the camera image, i.e., the patch matching pairs; then, the pixel-level features selected from the patch matching pairs are used for pixel-level matching; including:

[0039] For each Patch matching pair, all pixels are sent to the Patch matching module for matching, and only the highest similarity value is retained, that is: only the highest one in the Top-k is taken; by performing pixel-level matching on each Patch matching pair, the final pixel matching pair is obtained.

[0040] A further preferred method is to calculate the Patch similarity matrix S using the following formula:

[0041]

[0042] Among them, f C and f L These are two matrices output by the two linear layers, where Flatten is the flattening operation and Linear is the linear layer operation.

[0043] A further preferred embodiment involves converting pixel-matching point pairs into 3D-2D point pairs of point clouds and images, and using the PnP algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process; including:

[0044] By matching pixels, the points in the point cloud corresponding to the pixels on the projection map are found, and then a 3D-2D correspondence between the 3D point cloud of the projection and the pixels of the camera image is established, that is, the three-dimensional-two-dimensional point pairs between the point cloud and the image.

[0045] The pose is solved using the EPnP algorithm, and the transformation matrix is ​​finally estimated.

[0046] According to a preferred embodiment of the present invention, the total loss function in the convolutional neural network and the matching network is:

[0047] L total =L patch +L pixel ;

[0048] Assuming there is an actual correspondence It contains M corresponding items, representing the actual correspondence. Divide each value in the table by 4 and round down to the nearest integer to obtain the actual Patch correspondence. The first part L of the loss function patchThe expression is as follows:

[0049]

[0050] Where (i,j) refers to the value of each corresponding term in the correspondence, P ij This refers to the value corresponding to the i-th row and j-th column in the allocation matrix P;

[0051] For each patch correspondence, the matching network calculates M assignment matrices p; by verifying the actual correspondence... Divide by 4 and take the remainder to obtain the true pixel correspondence in each assignment matrix p. Thus, the second part L of the loss function is obtained. Pixel The second part of the loss function, L pixel The expression is as follows:

[0052]

[0053] Where, p i This indicates that the matching module calculates the assignment matrix for the i-th pair of patches. This represents the correspondence of the i-th real pixel. p i The value at the corresponding position in the matrix.

[0054] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a cross-modal registration method for patch-to-pixel LiDAR and camera images.

[0055] A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of a cross-modal registration method for LiDAR and camera images based on patch-to-pixel.

[0056] An efficient cross-modal registration system based on patch-to-pixel LiDAR and camera images, comprising:

[0057] The data preprocessing module is configured to perform data preprocessing, projecting the point cloud into two types of projection maps, including a reflectance map and a distance map;

[0058] The module for obtaining patch-level and pixel-level features is configured to: input the projection map and camera image into a convolutional neural network constructed using Python; and output the patch-level and pixel-level features of the projection map and camera image.

[0059] The feature matching module is configured to: use a patch-to-pixel matching network to perform feature matching. The patch-level features first estimate patch matching pairs, and then use the estimated patch matching pairs to extract the pixel-level features corresponding to each patch from the pixel-level features; use the extracted pixel-level features for pixel-level matching to estimate the pixel matching pairs between the projection map and the camera image.

[0060] The registration module is configured to convert pixel matching point pairs into 3D-2D point pairs of point clouds and images, and use the PnP algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process.

[0061] The beneficial effects of this invention are as follows:

[0062] 1. The cross-modal registration framework proposed in this invention utilizes the features of distance map and reflectivity map to reduce domain differences, and realizes cross-modal registration between lidar point cloud and camera image. The registration accuracy can reach more than 99% on the KITTI dataset.

[0063] 2. In response to the cross-modal differences between projection maps and camera images in the registration task, this invention proposes a multi-scale feature extraction network that can efficiently extract features from both the image and two different types of projection maps simultaneously.

[0064] 3. The feature matching framework from patch to pixel proposed in this invention not only solves the supervision problem caused by the limited overlap area between the projection map and the camera image, but also significantly improves the registration accuracy. Attached Figure Description

[0065] Figure 1 This is a flowchart illustrating the implementation process of the cross-modal registration method for LiDAR and camera images based on patch-to-pixel data according to the present invention.

[0066] Figure 2 This is a schematic diagram of the feature extraction network of the present invention;

[0067] Figure 3 This is a schematic diagram of the Patch-to-pixel matching process of the present invention. Detailed Implementation

[0068] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0069] Example 1

[0070] An efficient cross-modal registration method based on patch-to-pixel LiDAR and camera images, such as... Figure 1 As shown, it includes:

[0071] Step 1: Perform data preprocessing, projecting the point cloud into two types of projection maps, including a reflectance map and a distance map;

[0072] Step 2: Input the projection map obtained from the LiDAR point cloud and the camera image obtained from the high-definition camera into a convolutional neural network built using Python; the convolutional neural network outputs patch-level features and pixel-level features of the projection map and camera image for feature matching;

[0073] Step 3: Use a patch-to-pixel matching network to perform feature matching. First, estimate the patch matching pairs. Then, use the estimated patch matching pairs to extract the pixel-level features corresponding to each patch from the pixel-level features. Use the extracted pixel-level features for pixel-level matching to estimate the pixel matching pairs between the projection map and the camera image.

[0074] Step 4: Convert the pixel matching point pairs into point cloud and image 3D-2D point pairs, and use the PnP (Perspective-n-Point) algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process.

[0075] Example 2

[0076] The efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel mapping, as described in Example 1, differs in that: in the field of autonomous driving, commonly used LiDAR systems typically employ a multi-line rotating scanning design. The performance of a LiDAR is closely related to the number of laser beams; more beams mean better object detection. The number of beams is usually expressed in terms of the number of laser heads, including different specifications such as 16 lines, 32 lines, 64 lines, and 128 lines. Its working principle involves the laser head rotating within a 360° range to acquire data about the surrounding environment. Taking a 64-line LiDAR as an example, it generates 64 rings of point cloud data, which are stored sequentially from top to bottom. Therefore, based on this storage characteristic of the point cloud, it can be divided into 64 rows, where the number of each row is the LaserID of that row of point cloud data.

[0077] Traditional FOV spherical projection methods based on point cloud depth suffer from thread misalignment during projection. This occurs when point cloud data from adjacent lines are too close together along the z-axis, causing one thread's data to be overwritten while another thread's data is missing. This results in holes in the projected image, requiring interpolation for data completion. In contrast, LaserID-based projection effectively eliminates point cloud positional deviations caused by LiDAR sensor jitter, eliminating the need for interpolation and improving the accuracy and precision of the projected image. Furthermore, in terms of imaging principles, reflectivity information is closer to the image data acquired by the camera than point cloud depth information. Therefore, the LaserID-based projection method projects the reflectivity of the point cloud to obtain a reflectivity map. The point cloud is projected into two types of maps: a reflectivity map and a distance map.

[0078] 1) Perform coordinate transformation on the point cloud using the principle of spherical projection; transform the point cloud from the Cartesian coordinate system to the spherical coordinate system, and calculate the azimuth angle based on the point cloud's coordinates (x, y, z). Azimuth The calculation formula is: The range of values ​​for is [-π, π].

[0079] 2) After obtaining the LaserID of the point cloud, generate a projection map and record the mapping. Generally, when a LiDAR system collects point clouds, it records the LaserID information of each point. Combined with the azimuth angle obtained above... The pixel coordinates [u,v] of the projection map corresponding to the point cloud are obtained as follows: Taking a 64-line LiDAR as an example, assuming the projection map resolution is set to 64×1024, the LaserID is used as the projection coordinate of the point in the height direction, i.e., u = LaserID, and the width direction is projected proportionally according to the 1024 resolution. The calculation formula is: From the above, the pixel coordinates of the projection map corresponding to the point cloud are: For reflectance maps, the reflectance of the point cloud is recorded within pixels; that is, the gray value of each pixel represents the reflectance of the point cloud falling on that pixel. For distance maps, the radius *r* of the point cloud in the spherical coordinate system is recorded within pixels. x, y, and z are the coordinates of the point cloud on the X, Y, and Z axes in the Cartesian coordinate system.

[0080] After data preprocessing, two projection maps, reflectance map and distance map, are obtained. These projection maps, along with the camera images, are then fed into the subsequent feature extraction network.

[0081] The projection map and camera image are input into a convolutional neural network built using Python; the convolutional neural network outputs patch-level and pixel-level features of the projection map and camera image, including:

[0082] Unlike other image registration tasks, this invention addresses the cross-modal differences between the projection image and the camera image by designing a deep learning-based image feature extraction network in the feature extraction section. The detailed structure of the network is as follows: Figure 2 As shown. The feature extraction network, a convolutional neural network built with Python, is divided into two branches. The first branch processes reflectance maps and distance maps, while the second branch processes camera images. Each branch extracts patch-level features at 1 / 4 the scale of the original image, as well as pixel-level features at the same scale as the original image.

[0083] The feature extraction network uses a convolutional neural network (CNN) to process camera images and projection maps. The first half of the feature extraction network is the image encoder, and the second half is the image decoder. The image encoder progressively downsamples the features, ultimately generating a high-dimensional feature at 1 / 32 of the original image scale. This high-dimensional feature is then passed to the image decoder, where the feature scale is progressively upsampled. Specifically, this includes:

[0084] First, obtain patch-level features at 1 / 4 the size of the original image. Among them, D Patch This represents the number of feature channels at the patch level, where H and W are the height and width of the original image. The set of real numbers in mathematical notation is used; then, through two layers of deconvolution, pixel-level features at the original image resolution are obtained. Among them, D pixel It is the number of feature channels at the pixel level.

[0085] like Figure 2 As shown, in the second branch (camera image branch), the RGB three-channel camera image is input into the convolutional neural network. After five layers of fully convolutional network, 128 channels of small-scale features (1 / 32 scale of the original camera image) are obtained. Then, deconvolution is used to upsample the small-scale features. After three layers of deconvolution, 256 channels of patch-level features at 1 / 4 scale of the original camera image are obtained. The patch-level features are further upsampled using deconvolution. After two layers of deconvolution, the scale is restored to the original image scale. At this point, pixel-level features with 256 channels and the same scale as the original camera image are obtained.

[0086] In the first branch (reflectance map & distance map branch), the same CNN architecture is used for the projection map feature extraction branch. However, unlike the camera image feature extraction, two identical encoder branches are used for the distance map and reflectance map respectively. The features extracted by the two identical encoder branches are concatenated and passed to the projection map decoder network for upsampling, resulting in a patch-level feature and a pixel-level feature. Specifically, for the two inputs of the projection map branch—reflectance map and distance map—first, a five-layer fully convolutional network is used to extract features from the reflectance map and distance map respectively, resulting in 128-channel small-scale features. Then, the two small-scale features are concatenated, resulting in 256 channels with the same scale. Next, the concatenated features are fully convolutionally processed again, resulting in 128 channels with the same scale. Finally, the resulting features are deconvolutionally upsampled, resulting in 256 channels at the scale of the original projection map after three deconvolution layers. Figure 1 / 4 scale Patch-level features; the Patch-level features are then deconvolved and upsampled twice to obtain pixel-level features with the same scale as the original projection image and 256 channels.

[0087] At this point, the respective Patch-level and pixel-level features of the camera image and the projection map have been obtained.

[0088] The next step is to use these features at different scales for feature matching.

[0089] To address the scarcity of positive samples due to the limited overlap between camera images and point clouds, a search is first performed at the patch level, followed by pixel-level matching within the matched patches. This matching strategy not only increases the proportion of positive samples through patch-level search, solving the network supervision problem, but also significantly improves registration accuracy through pixel-level matching.

[0090] The structure of the matching network is as follows Figure 3 As shown.

[0091] Feature matching is performed using a patch-to-pixel matching network; including:

[0092] First, the patch-level features extracted by the feature extraction network are input into the patch matching module for patch-level matching; the patch matching module is similar to the optimized network structure in Lightglue (see the paper 'LightGlue: Local Feature Matching at Light Speed'). It includes:

[0093] For two patch-level features that need to be matched, firstly, the two patch-level features are flattened and then input into a linear neural network with the same number of channels as the output. The number of channels and scale of the output features are the same as the input patch-level features. Then, the two output features are multiplied (the multiplicand matrix is ​​first transposed) to obtain the patch similarity matrix S. Finally, softmax operations are performed on matrix S row by row and column by column to obtain the matrix after row softmax processing and the matrix after column softmax processing, respectively. The Hadamard product of the two matrices is then taken to obtain the assignment matrix P, which is described by the following formula:

[0094]

[0095] Among them, P ij S represents the value in the i-th row and j-th column of the allocation matrix P. ij N represents the value in the i-th row and j-th column of the allocation matrix S. 2D N represents the total number of camera image patches, which is the total number of rows in matrix S; 3D This represents the total number of patches in the projected image, which is the total number of columns in the S matrix.

[0096] We select the top-k values ​​from the assignment matrix P to obtain the matching relationship between the patch-level features of the projected image and the camera image, i.e., patch matching pairs. Specifically, for all values ​​in the assignment matrix, we arrange them from largest to smallest and select the k largest values. The specific value of k is defined by us. The rows and columns in the matrix corresponding to the selected values ​​are the patch matching pairs. The patch feature scale is 1 / 4 of the original image, and each patch corresponds to a 4×4 image region containing 16 pixels. Therefore, each patch of the selected patch matching pair contains the features of 16 pixels in the actual image region. We previously extracted pixel-level features with the same scale as the original image in the feature extraction network. For pixel-level features, we extract the pixel-level features of the corresponding region in each matching pair's patch separately. The pixel-level features of the regions not selected as matching patch pairs are discarded. This step is performed in... Figure 3 This is what we call feature filtering.

[0097] Next, pixel-level features selected from the patch matching pairs are used for pixel-level matching; including:

[0098] For each patch matching pair, all pixels are sent to the patch matching module for matching, retaining only the highest similarity value; that is, only the highest value from the Top-k is taken. In other words, within each matching patch pair, the patch matching module selects a pair of matching pixels. By performing pixel-level matching on each patch matching pair, the final pixel matching pair is obtained.

[0099] The formula for calculating the patch similarity matrix S is as follows:

[0100] The projection map patch-level features and the camera image patch-level features are first flattened and fed into a linear layer. The two matrices are then multiplied using matrix multiplication to obtain the score matrix S.

[0101]

[0102] Among them, f C and f L These are two matrices output by the two linear layers, where Flatten is the flattening operation and Linear is the linear layer operation.

[0103] The pixel matching point pairs are converted into 3D-2D point pairs of point clouds and images. The PnP algorithm is used to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process; including:

[0104] By matching pixels, the points in the point cloud corresponding to the pixels on the projection map are found, and then a 3D-2D correspondence between the 3D point cloud of the projection and the pixels of the camera image is established, that is, the three-dimensional-two-dimensional point pairs between the point cloud and the image.

[0105] The EPnP algorithm is used to solve for the pose and finally estimate the transformation matrix. This includes: finding the corresponding point in the point cloud for each pixel using the projection map; therefore, pixel matching pairs obtained through pixel matching can yield 3D-2D point pairs between the point cloud and the image. In the specific implementation of the EPnP algorithm, firstly, the minimum point set is randomly sampled from the 3D-2D correspondence pairs using the RANSAC framework; then, the EPnP algorithm is used to represent the 3D point as a weighted sum of four control points, constructing a system of linear equations based on the 2D projection relationship, and solving for the coordinates of the control points in the camera coordinate system using singular value decomposition (SVD); then, combining the coordinates of the control points in the world coordinate system, the rotation matrix R and translation vector t are calculated to obtain the transformation matrix T; finally, the matching accuracy of the point pairs is evaluated through reprojection error, inlier points are selected and iterative optimization is performed to output the optimal transformation matrix T.

[0106] In convolutional neural networks and matching networks, the overall loss function is:

[0107] L total =L patch+L pixel ;

[0108] In the design of the loss function, the LiDAR point cloud is projected onto the camera image using the true extrinsic parameters provided in the dataset to obtain the actual 3D-2D correspondence. This assumes the existence of the actual correspondence. It contains M corresponding items, representing the actual correspondence. Divide each value in the table by 4 and round down to the nearest integer to obtain the actual Patch correspondence. For each true patch correspondence, the corresponding position value in the patch allocation matrix P should be as large as possible. Based on this idea, the first part L of the loss function is designed. Patch The first part of the loss function L patch The expression is as follows:

[0109]

[0110] Where (i,j) refers to the value of each corresponding term in the correspondence, P ij This refers to the value corresponding to the i-th row and j-th column in the allocation matrix P;

[0111] During training, the matching network of this invention does not use top-k, but directly uses the actual patch correspondences for subsequent pixel-level matching. For each patch correspondence, the matching network calculates M assignment matrices p; by matching the actual correspondences... Divide by 4 and take the remainder to obtain the true pixel correspondence in each assignment matrix p. Thus, the second part L of the loss function is obtained. Pixel The second part of the loss function, L pixel The expression is as follows:

[0112]

[0113] Where, p i This indicates that the matching module calculates the assignment matrix for the i-th pair of patches. This represents the correspondence of the i-th real pixel. p i The value at the corresponding position in the matrix.

[0114] This invention was tested on the KITTIOdometry dataset (see the paper 'Are we ready for autonomous driving? The KITTI vision benchmark suite') and the nuScenes dataset (see the paper 'nuScenes: A multimodal dataset for autonomous driving').

[0115] KITTIOdometry: This invention generates image-point cloud pairs from the same data frame of a 2D / 3D sensor. Following the setup of other related works, sequences 00-08 are used for training, and sequences 09-10 are used for testing. The camera image resolution is downsampled to 512×160, and the reflectance map resolution is 1024×64. During training and testing, a 2D translation (along the x and y axes) of ±10m and an arbitrary rotation about the z axis are applied to the point cloud. To ensure realism, we first apply a rotation about the z axis to the point cloud, and then apply translations along the x and y axes.

[0116] nuScenes: This invention selected 850 scenes for training and 150 scenes for testing. The camera image resolution was downsampled to 320×160, and the reflectance map resolution was 1024×32. Two-dimensional translations (along the x and y axes) of ±10m and arbitrary rotations around the z axis were applied to the point cloud, using the same method as KITTI.

[0117] This invention was compared with the following three methods: DeepI2P, CorrI2P, and VP2P-Match.

[0118] DeepI2P (see the paper 'DeepI2P: Image-to-point cloud registration via deepclassification') includes two strategies: Grid Cls.+PnP and Frus.Cls.+Inv.Proj. GridCls.+PnP first divides the input image into a 32×32 grid, trains a neural network to classify 3D points into 2D grid cells, and estimates the transformation matrix using EPnP and RANSAC. Frus.Cls.+Inv.Proj introduces frustum-based classification, combined with inverse camera projection, resulting in DeepI2P(2D) and DeepI2P(3D).

[0119] CorrI2P (see the paper 'CorrI2P: Deep image-to-point cloud registration via dense correspondence') improves the registration accuracy between point clouds and image pairs by introducing a feature exchange module and overlapping region supervision, based on DeepI2P.

[0120] VP2P-Match (see the paper 'Differentiable registration of images and lidarpoint clouds with voxelpoint-to-pixel matching') enhances the similarity between point cloud features and image CNN-extracted features through sparse convolution. It also incorporates a differentiable PnP solver in an end-to-end framework, thereby improving matching accuracy and speed.

[0121] The transformation matrix can usually be decomposed into two separate parts: the rotation matrix R and the translation matrix T. Therefore, the standard for measuring registration accuracy is the relative translation error (RTE) E. T and relative rotation error (RRE)E R This allows for the assessment of the accuracy of the registration.

[0122]

[0123] E T =||T gt -T E ||;

[0124] Here, γ is the inverse of the true value of the rotation matrix. The predicted value R is obtained by right multiplying the predicted rotation matrix. The value after conversion to Euler angles. Similarly, the true value T of the translation matrix. …f The L2 norm is obtained by subtracting the predicted value T from E. T This embodiment statistically analyzed the mean and standard deviation of all data. The registration success rate (Acc) was also calculated, defined as RTE < 2m and RRE < 5°. The results are shown in Table 1.

[0125] Table 1

[0126]

[0127] As can be seen from Table 1, the method of the present invention is superior to existing methods in both registration accuracy and accurate registration success rate.

[0128] Example 3

[0129] A computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the cross-modal registration method for LiDAR and camera images based on patch to pixel as described in Embodiment 1 or 2.

[0130] Example 4

[0131] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the cross-modal registration method for LiDAR and camera images based on patch-to-pixel as described in Embodiment 1 or 2.

[0132] Example 5

[0133] An efficient cross-modal registration system based on patch-to-pixel LiDAR and camera images, comprising:

[0134] The data preprocessing module is configured to perform data preprocessing, projecting the point cloud into two types of projection maps, including a reflectance map and a distance map;

[0135] The module for obtaining patch-level and pixel-level features is configured to: input the projection map obtained from the LiDAR point cloud and the camera image obtained from the high-definition camera into a convolutional neural network constructed using Python; the convolutional neural network outputs patch-level and pixel-level features of the projection map and the camera image; and is used for feature matching.

[0136] The feature matching module is configured to: use a patch-to-pixel matching network to perform feature matching. The patch-level features first estimate patch matching pairs, and then use the estimated patch matching pairs to extract the pixel-level features corresponding to each patch from the pixel-level features; use the extracted pixel-level features for pixel-level matching to estimate the pixel matching pairs between the projection map and the camera image.

[0137] The registration module is configured to convert pixel matching point pairs into 3D-2D point pairs of point clouds and images, and use the PnP (Perspective-n-Point) algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process.

[0138] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel mapping, characterized in that, include: Step 1: Perform data preprocessing, projecting the point cloud into two types of projection maps, including a reflectance map and a distance map; Step 2: Input the projection image and camera image into a convolutional neural network built with Python; the convolutional neural network outputs patch-level and pixel-level features of the projection image and camera image. Step 3: Use a patch-to-pixel matching network to perform feature matching. First, estimate the patch matching pairs for the patch-level features, and then use the estimated patch matching pairs to extract the pixel-level features corresponding to each patch from the pixel-level features. The extracted pixel-level features are used for pixel-level matching to estimate pixel-matching pairs between the projection map and the camera image; Step 4: Convert the pixel matching point pairs into 3D-2D point pairs of point cloud and image, and use the PnP algorithm to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process; The projection image and camera image are input into a convolutional neural network built using Python; the convolutional neural network outputs patch-level and pixel-level features of the projection image and camera image, including: The feature extraction network, a convolutional neural network built with Python, is divided into two branches. The first branch processes reflectance maps and distance maps, while the second branch processes camera images. Each branch extracts patch-level features at 1 / 4 the scale of the original image, as well as pixel-level features at the same scale as the original image. The first half of the feature extraction network is the image encoder, and the second half is the image decoder. The image encoder progressively downsamples the features, ultimately generating a high-dimensional feature at 1 / 32 of the original image scale. This high-dimensional feature is then passed to the image decoder, where the features are progressively upsampled. Specifically, this includes: First, obtain patch-level features at 1 / 4 the size of the original image. Among them, D Patch This represents the number of feature channels at the patch level, where H and W are the height and width of the original image. The set of real numbers in mathematical notation is used; then, through two layers of deconvolution, pixel-level features at the original image resolution are obtained. Among them, D pixel It is the number of feature channels at the pixel level; In the second branch, the RGB three-channel camera image is input into the convolutional neural network. After five layers of fully convolutional network, 128 channels of small-scale features are obtained. Then, deconvolution is used to upsample the small-scale features. After three layers of deconvolution, 256 channels of patch-level features at 1 / 4 scale of the original camera image are obtained. The patch-level features are then upsampled again using deconvolution. After two layers of deconvolution, the scale is restored to the original image scale. At this point, 256 channels of pixel-level features with the same scale as the original camera image are obtained. In the first branch, features are first extracted from the reflectance map and distance map using a five-layer fully convolutional network, resulting in 128-channel small-scale features. Then, the two small-scale features are concatenated, resulting in 256 channels while maintaining the same scale. Next, the concatenated features are subjected to another fully convolutional operation, followed by deconvolutional upsampling. After three layers of deconvolution, a patch-level feature with 256 channels and a scale of 1 / 4 of the original projection map is obtained. The patch-level feature is then subjected to two more deconvolutional upsampling operations to obtain pixel-level features with the same scale as the original projection map and 256 channels. At this point, the respective Patch-level and pixel-level features of the camera image and the projection map are obtained; In convolutional neural networks and matching networks, the overall loss function is: L total =L Patch +L pixel ; Assuming there is an actual correspondence It contains M corresponding items, representing the actual correspondence. Divide each value in the table by 4 and round down to the nearest integer to obtain the actual Patch correspondence. The first part L of the loss function Patch The expression is as follows: Where (i,j) refers to the value of each corresponding term in the correspondence, P ij This refers to the value corresponding to the i-th row and j-th column in the allocation matrix P; For each patch correspondence, the matching network calculates M assignment matrices p; by verifying the actual correspondence... Divide by 4 and take the remainder to obtain the true pixel correspondence in each assignment matrix p. Thus, the second part L of the loss function is obtained. Pixel The second part of the loss function, L pixel The expression is as follows: Where, p i This indicates that the matching module calculates the assignment matrix for the i-th pair of patches. This represents the correspondence of the i-th real pixel. p i The value at the corresponding position in the matrix.

2. The efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel as described in claim 1, characterized in that, The point cloud is projected into two types of projection maps, including a reflectance map and a distance map; including: 1) Perform coordinate transformation on the point cloud using the principle of spherical projection; transform the point cloud from the Cartesian coordinate system to the spherical coordinate system, and calculate the azimuth angle based on the point cloud's coordinates (x, y, z). Azimuth The calculation formula is: The range of values ​​for is [-π, π]. 2) After obtaining the LaserID of the point cloud, generate a projection map and record the mapping. The pixel coordinates of the projection map corresponding to the point cloud are: For reflectance maps, the reflectance of the point cloud is recorded within pixels; that is, the gray value of each pixel represents the reflectance of the point cloud falling on that pixel. For distance maps, the radius *r* of the point cloud in the spherical coordinate system is recorded within pixels. x, y, and z are the coordinates of the point cloud on the X, Y, and Z axes in the Cartesian coordinate system.

3. The efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel as described in claim 1, characterized in that, Feature matching is performed using a patch-to-pixel matching network; including: First, the patch-level features extracted by the feature extraction network are input into the patch matching module for patch-level matching; this includes: For two patch-level features that need to be matched, firstly, the two patch-level features are flattened and then input into a linear neural network with the same number of channels as the output. The number of channels and scale of the output features are the same as the input patch-level features. Then, the two output features are multiplied to obtain the patch similarity matrix S. Finally, softmax operations are performed on matrix S row by row and column by column to obtain the matrix after row softmax processing and the matrix after column softmax processing, respectively. The Hadamard product of the two matrices is then taken to obtain the assignment matrix P, which is described by the following formula: Among them, P ij S represents the value in the i-th row and j-th column of the allocation matrix P. ij N represents the value in the i-th row and j-th column of the allocation matrix S. 2D This represents the total number of patches in the camera images, which is the total number of rows in the S matrix. This represents the total number of patches in the projected image, which is the total number of columns in the S matrix. The Top-k features are selected from the allocation matrix P to obtain the matching relationship between the patch-level features of the projection map and the camera image, i.e., the patch matching pairs; then, the pixel-level features selected from the patch matching pairs are used for pixel-level matching; including: For each Patch matching pair, all pixels are sent to the Patch matching module for matching, and only the highest similarity value is retained, that is: only the highest one in the Top-k is taken; by performing pixel-level matching on each Patch matching pair, the final pixel matching pair is obtained.

4. The efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel as described in claim 3, characterized in that, The formula for calculating the patch similarity matrix S is as follows: S=Linear(Flatten(f C ))×Linear(Flatten(f L )) T ; Among them, f C and f L These are two matrices output by the two linear layers, with Flatten representing the flattening operation and Linear representing the linear layer operations.

5. The efficient cross-modal registration method for LiDAR and camera images based on patch-to-pixel as described in claim 1, characterized in that, The pixel matching point pairs are converted into 3D-2D point pairs of point clouds and images. The PnP algorithm is used to estimate the transformation matrix from the given 3D-2D point pairs to complete the registration process; including: By matching pixels, the points in the point cloud corresponding to the pixels on the projection map are found, and then a 3D-2D correspondence between the 3D point cloud of the projection and the pixels of the camera image is established, that is, the three-dimensional-two-dimensional point pairs between the point cloud and the image. The pose is solved using the EPnP algorithm, and the transformation matrix is ​​finally estimated.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the cross-modal registration method for LiDAR and camera images based on patch-to-pixel as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the cross-modal registration method for LiDAR and camera images based on patch-to-pixel as described in any one of claims 1-5.

8. A highly efficient cross-modal registration system for LiDAR and camera images based on patch-to-pixel mapping, characterized in that, include: The data preprocessing module is configured to perform data preprocessing, projecting the point cloud into two types of projection maps, including a reflectance map and a distance map; The module for obtaining patch-level and pixel-level features is configured to: input the projection map and camera image into a convolutional neural network constructed using Python; and output the patch-level and pixel-level features of the projection map and camera image. The feature matching module is configured to use a patch-to-pixel matching network to perform feature matching. The patch-level features first estimate the patch matching pairs, and then use the estimated patch matching pairs to extract the pixel-level features corresponding to each patch from the pixel-level features. The extracted pixel-level features are used for pixel-level matching to estimate pixel-matching pairs between the projection map and the camera image; The registration module is configured to convert pixel matching point pairs into 3D-2D point pairs of point clouds and images, estimate the transformation matrix from the given 3D-2D point pairs using the PnP algorithm, and complete the registration process. The projection image and camera image are input into a convolutional neural network built using Python; the convolutional neural network outputs patch-level and pixel-level features of the projection image and camera image, including: The feature extraction network, a convolutional neural network built with Python, is divided into two branches. The first branch processes reflectance maps and distance maps, while the second branch processes camera images. Each branch extracts patch-level features at 1 / 4 the scale of the original image, as well as pixel-level features at the same scale as the original image. The first half of the feature extraction network is the image encoder, and the second half is the image decoder. The image encoder progressively downsamples the features, ultimately generating a high-dimensional feature at 1 / 32 of the original image scale. This high-dimensional feature is then passed to the image decoder, where the features are progressively upsampled. Specifically, this includes: First, obtain patch-level features at 1 / 4 the size of the original image. Among them, D Patch This represents the number of feature channels at the patch level, where H and W are the height and width of the original image. The set of real numbers in mathematical notation is used; then, through two layers of deconvolution, pixel-level features at the original image resolution are obtained. Among them, D pixel It is the number of feature channels at the pixel level; In the second branch, the RGB three-channel camera image is input into the convolutional neural network. After five layers of fully convolutional network, 128 channels of small-scale features are obtained. Then, deconvolution is used to upsample the small-scale features. After three layers of deconvolution, 256 channels of patch-level features at 1 / 4 scale of the original camera image are obtained. The patch-level features are then upsampled again using deconvolution. After two layers of deconvolution, the scale is restored to the original image scale. At this point, 256 channels of pixel-level features with the same scale as the original camera image are obtained. In the first branch, features are first extracted from the reflectance map and distance map using a five-layer fully convolutional network, resulting in 128-channel small-scale features. Then, the two small-scale features are concatenated, resulting in 256 channels while maintaining the same scale. Next, the concatenated features are subjected to another fully convolutional operation, followed by deconvolutional upsampling. After three layers of deconvolution, a patch-level feature with 256 channels and a scale of 1 / 4 of the original projection map is obtained. The patch-level feature is then subjected to two more deconvolutional upsampling operations to obtain pixel-level features with the same scale as the original projection map and 256 channels. At this point, the respective Patch-level and pixel-level features of the camera image and the projection map are obtained; In convolutional neural networks and matching networks, the overall loss function is: L total =L Patch +L pixel ; Assuming there is an actual correspondence It contains M corresponding items, representing the actual correspondence. Divide each value in the table by 4 and round down to the nearest integer to obtain the actual Patch correspondence. The first part L of the loss function Patch The expression is as follows: Where (i,j) refers to the value of each corresponding term in the correspondence, P ij This refers to the value corresponding to the i-th row and j-th column in the allocation matrix P; For each patch correspondence, the matching network calculates M assignment matrices p; by verifying the actual correspondence... Divide by 4 and take the remainder to obtain the true pixel correspondence in each assignment matrix p. Thus, the second part L of the loss function is obtained. Pixel The second part of the loss function, L pixel The expression is as follows: Among them, p i This indicates that the matching module calculates the assignment matrix for the i-th pair of patches. This represents the correspondence of the i-th real pixel. p i The value at the corresponding position in the matrix.

Images