Point cloud and image level registration method based on cross-modal flow estimation

Abstract

The invention relates to a point cloud and image level registration method based on cross-modal flow estimation, and belongs to the field of computer vision and photogrammetry. The method comprises the following steps: acquiring original point cloud data and two-dimensional image data, and converting the point cloud data to obtain a point cloud depth image; respectively extracting depth image features and RGB image features; constructing a global matching cost body and a local matching cost body by adopting different matching cost calculation modes under different scales; repeatedly updating the optical flow by searching the matching cost from the related matching cost body by using the current estimated value so as to obtain a pixel-level matching relationship between the images; and finally, calculating the attitude of the imaging equipment relative to the point cloud reference system to finish registration. According to the method, information provided by different scales is effectively combined while depth feature sparsity is considered, the problem of high memory consumption caused by a network model is relieved, more robust cost body representation is generated, finally, the registration effect is improved, and a more accurate point and pixel corresponding relation is recovered.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and photogrammetry, and relates to a point cloud and image hierarchical registration method based on cross-modal flow estimation. Background Art

[0002] With the rapid development of sensor technology and computer technology, earth observation sensors such as measuring cameras and laser scanners have gradually become miniaturized and lightweight. This progress has promoted the realization of low-cost, lightweight multi-sensor remote sensing systems. UAV and vehicle-mounted mobile measurement systems have been widely used in basic surveying and mapping, environmental monitoring, smart city construction, navigation and location services, etc. The range imaging and visible light imaging data collected by these devices show the characteristics of multi-source, multi-perspective and high resolution, but due to the large differences in data modality and spatial reference, it is difficult to directly perform high-precision registration. This difference not only limits the data fusion effect, but also significantly reduces the practicality of the data. High-precision registration is the pre-link of data fusion and also its core part, which is crucial to improving the data fusion effect.

[0003] In the field of photogrammetry and remote sensing, a large number of studies have shown that the fusion of laser point cloud data and image data can effectively improve the accuracy, robustness and automation of object classification, building extraction and change detection. The pixel-level high-precision registration and fusion of the two are of great significance to related photogrammetry and remote sensing applications. In recent years, the research on data registration methods for mobile measurement platforms has received widespread attention in the academic community. Some researchers use LiDAR maps and RGB images for 2D-3D feature matching. Although there is a large overlap between point clouds and corresponding images, it is still difficult to ensure that the extracted image features and point cloud features have great repeatability and high positioning accuracy. In order to overcome the modality gap, other researchers project 3D data into 2D space or reconstruct 3D point clouds from multi-view images for matching in the same domain. In these ways, the modality gap and appearance differences are alleviated to a certain extent. How to make full use of the characteristics of cross-modal registration and improve the accuracy of cross-modal registration in complex scenes is still a big challenge. To solve this problem, subsequent studies proposed a pixel-level matching method based on optical flow estimation that projects 3D point clouds into 2D space.

[0004] The core idea of ​​optical flow estimation is to find the pixel-level offset from the source image to the target image, establish a matching cost volume between the two images during the calculation process, and establish the pixel-level correspondence between the images by matching cost search. There are two problems with the current registration model based on optical flow estimation. First, due to the large difference in data modality, the depth map obtained by point cloud projection is quite different from the RGB image in data semantics and data sparsity. It is difficult to obtain high matching accuracy using the existing optical flow estimation algorithm. Second, when faced with low overlapping areas, traditional methods find it difficult to establish sufficient correspondences, resulting in erroneous correspondences involved in subsequent pose calculations, thereby reducing the registration accuracy. Therefore, how to efficiently solve the above problems has become the core challenge of three-dimensional point cloud and two-dimensional image registration. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a point cloud and image hierarchical registration method based on cross-modal flow estimation. In view of the differences between different modal data, different feature extraction branches are designed for different modal data to improve the quality of image feature extraction. When facing scenes with extremely low overlapping areas, training models at multiple scales are stacked in a cascade manner, so that the model still maintains a high-precision registration effect in extreme scenes. At the same time, different matching cost body calculation methods are used for different scales, taking into account the memory consumption and matching accuracy at different scales.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A point cloud and image hierarchical registration method based on cross-modal flow estimation, the method comprising the following steps:

[0008] S1, data acquisition, obtaining the original point cloud data and two-dimensional image data, and converting the point cloud data from the initial pose to the camera coordinate reference system and then using the camera internal parameter projection to obtain the point cloud depth map;

[0009] S2, feature extraction, extracting depth image features and RGB image features respectively through a feature extraction module, wherein the feature extraction module includes an RGB camera image feature extraction branch and a point cloud depth image feature extraction branch;

[0010] S3, cost body construction, using different matching cost calculation methods to construct matching cost bodies at different scales, using a full correlation matching strategy to generate a global matching cost body at a low scale, and using a local correlation matching strategy to generate a local matching cost body at a high scale;

[0011] S4, correspondence estimation, in the GRU convolutional layer, the optical flow is repeatedly updated by using the current estimate to find the matching cost from the relevant matching cost volume to obtain the pixel-level matching relationship between images;

[0012] S5. Set the corresponding loss function and network parameters to optimize the above process;

[0013] S6, posture solution, according to the pixel-level correspondence between the camera intrinsic parameters, depth image and RGB image, the PnP mathematical model is used to calculate the posture of the imaging device relative to the 3D point cloud reference system to complete the alignment.

[0014] Furthermore, in step S1, the RGB image and the point cloud depth image are randomly cropped in the training stage, and are center cropped in the inference stage, and are standardized; for the original three-dimensional point cloud data, an initial pose is randomly generated within a preset error range, and a corresponding sparse depth map is generated according to the camera intrinsic parameters and the initial pose, and is also randomly cropped to a preset size.

[0015] Further, in step S2, each branch adopts an encoder-decoder architecture to obtain multi-scale features, wherein a set of sparse convolutional layers is added before the encoder-decoder architecture of the depth branch to encrypt the sparse depth map into a dense depth map;

[0016] In the encoder-decoder architecture, a convolutional network is used to extract features from the input image. The feature encoder is applied to the depth image I1 and the RGB image I2 to encode the input image into a low-resolution dense feature map.

[0017] Encoder g θ Output features at 1 / 32 resolution:

[0018] R H×W×3 →R H / 32×W / 32×D

[0019] Where D represents the number of channels; the feature encoder is composed of multiple residual blocks, and then passes through a decoder with a similar structure to decode the features to 1 / 16, 1 / 8, and 1 / 4 scales to obtain multi-scale features.

[0020] Furthermore, in the sparse convolutional layer of the depth branch, the sparse depth map is encrypted before the depth map feature extraction, and the process is as follows:

[0021] The sparse convolutional layer is used to evaluate only the observed pixels and standardize the output to take into account sparsity. A mask is attached to the output of each layer, which normalizes the output of each convolutional layer and adjusts the result to the visible area of ​​the previous layer. At the same time, the receptive field of the mask is expanded layer by layer, and the dense result is input into a similar encoding and decoding structure. Finally, features at three scales are obtained.

[0022] Let f represent the mapping from the input domain to the output domain achieved by the convolutional neural network. Assume that the input X = {x u,v} is only partially observed, O = {o u,v} represents the corresponding binary variable, indicating whether the input is observed ({o u,v}=1) or not observed ({o u,v}=0), the output of the standard convolutional layer in the convolutional neural network is calculated as follows:

[0023]

[0024] The kernel size is 2k+1, the weight is ω, and the bias is b;

[0025] The output calculation method of the convolutional layer introduces O = {o u,v} explicitly account for sparsity by evaluating only the observed pixels and normalizing the output, i.e.:

[0026]

[0027] Replace ∈ with a constant to avoid the denominator being 0, x u+i,v+j is the input pixel.

[0028] Further, in step S3, a global matching cost volume is generated by adopting a full correlation matching strategy at a low scale, wherein the global matching cost volume is calculated as follows:

[0029] Visual similarity is computed by constructing the complete matching cost between all pixel pairs. The matching cost volume is formed by the dot products between all pairs of eigenvectors; the correlation volume C is calculated as a single matrix multiplication:

[0030] C(g θ (I1),g θ (I2))∈R H*W*H*W

[0031]

[0032] Among them, g θ (I1)∈R H*W*D For a given RGB image feature, g θ (I2)∈R H*W*D Given a deep image feature.

[0033] Further, in step S3, a local correlation matching strategy is adopted at a high scale to generate a local matching cost body, wherein the calculation method of the local matching cost body is based on the PatchMatch algorithm. When calculating the local correlation, the offset value of the neighborhood point of the required point is used to calculate the corresponding position of the current point, and the offset matching cost is calculated;

[0034] In the propagation stage, a point in the feature map is regarded as a block, and several neighbor seed points are selected. Each point is moved by the offset of the neighbor point, and the candidate offset is obtained from its neighbors. Then, a multi-dimensional matching cost volume is calculated based on the neighbor candidate offset and its own offset;

[0035] Given a shift Δp, the correlation calculation of the propagation is defined as:

[0036] Corr=F1·W(F2,S(flow,Δp))

[0037] F1 and F2 represent features respectively. Figure 1 and feature map 2, flow represents the offset map, S(flow,Δp) represents the transformation of the offset map according to Δp, and W(.) represents the result of mapping F2 according to the offset map.

[0038] Further, in step S4, it specifically includes:

[0039] First, regression prediction is performed at a low scale by taking the existing correspondence, global correlation, and potential hidden state as the input of the GRU convolutional layer and outputting the updated correspondence and hidden state;

[0040] Then, the low-scale results are used as initial values ​​to guide the update using the local correlation cost volume at a high scale.

[0041] Finally, accurate pixel-level matching relationship between images is obtained.

[0042] Furthermore, in step S5, the loss function uses a weighted loss function, and the batch size batch_size and the number of iterations of the model are initialized and set, the Adam optimizer and the dynamic learning rate are used, and the data enhancement strategy of random flipping and linear stretching is used in the training set, specifically:

[0043] The weighted loss function includes: using the average endpoint error EPE as the pixel matching error for calculating the RGB image and the LiDAR depth image, which is expressed as:

[0044]

[0045] Among them, F gt (u,v) represents the true offset, F pre (u,v) is the prediction offset, mask uv Record pixels for which the pixel has a valid depth value;

[0046] Secondly, a loss function based on the surface normal vector is used to introduce local geometric constraints to supervise model learning: it obtains a synthetic depth image based on the true value pose and the predicted pose respectively. and Then from and Randomly sample several points from the common points of the effective depth value and calculate the surface normal vector n respectively. gt and n input ; Repeat this process N times, calculating the normal vector each time and The L2 distance between them, finally, the loss function is calculated according to the following formula:

[0047]

[0048] The final loss function consists of the weighted sum of the above loss functions, namely:

[0049] τ=τ epe +α×τ normal

[0050] Among them, α is the weighting coefficient.

[0051] Further, in step S6, after estimating the dense correspondence between the depth image and the RGB image, the correspondence between the 2D pixel positions in the camera image and the 3D points in the LiDAR map is inferred based on the camera intrinsic parameters K, and then the camera pose is estimated using the PnP solver in a RANSAC cycle, where

[0052] After obtaining the dense correspondence between the depth image and the RGB image, the correspondence between the points in the 3D point cloud and the pixels in the 2D image is obtained based on the transitivity of the relationship;

[0053] The PnP mathematical model is used to solve the 3D to 2D point pair motion. The solution process is described as:

[0054]

[0055] Given the coordinates P of n points in the world coordinate system, the coordinates p of the corresponding pixel, and the camera intrinsic parameter K, solve the camera coordinate system (O c X c Y c Z c ) relative to the world coordinate system (O w X w Y w Z w ) is the [RT] matrix in the formula;

[0056] The RANSAC random sampling consensus algorithm is used to estimate the pose parameters in the PnP mathematical model. The specific process includes:

[0057] 1) Given a data set S, select the minimum number of samples required to build a model from it, and denote the selected data set as S1;

[0058] 2) Using the selected data set S1, a mathematical model M1 is calculated;

[0059] 3) Use the calculated model M1 to test the remaining points in the data set. If the tested data point is within the allowable error range, the data point is judged as an inlier, otherwise it is judged as an outlier. The data set composed of all inliers is recorded as The consistent set is called S1;

[0060] 4) Compare the number of inliers of the current model and the best model previously introduced, and record the model parameters and the number of inliers when the number of inliers is the largest;

[0061] 5) Repeat steps 1 to 4 until the iteration ends or the number of internal points of the current model is greater than the set threshold; finally, the relative pose of the transformation between the camera reference system and the world reference system that best conforms to the corresponding relationship distribution is obtained.

[0062] The beneficial effects of the present invention are:

[0063] First, the present invention alleviates the difference between point cloud and image by converting 3D point cloud into 2D depth map, and converts the problem into multi-source image registration problem. In order to establish the pixel-level correspondence between image and point cloud, the offset of pixel points in the depth image on the RGB image is predicted, and the pixel-level correspondence is established between the RGB image and the depth synthesis image.

[0064] Second, the present invention introduces a sparse convolution layer in the depth branch to replace the traditional convolution, which better adapts to the information sparsity of depth image data, improves the feature extraction quality of the depth branch, and solves the technical problem that the existing method has poor matching effect between dense pixels and sparse depth.

[0065] Third, the present invention takes into account the rich and multi-level semantic information provided by multiple scales, and uses the encoder-decoder modality to extract multi-scale features, so that the network can capture richer feature information, which helps to improve the model's adaptability to complex environments.

[0066] Fourth, the present invention adopts a multi-scale, phased approach to gradually refine the matching cost volume, thereby enhancing the model's ability to predict weak texture areas and edge areas. In order to solve the problem of high GPU memory consumption in existing methods, the present invention inputs low-scale information into a high-consumption full-correlation matching module to obtain a rough result to guide the next layer. Applying high resolution to local cost matching not only accepts the initial value guidance of the previous layer, but also avoids the problem of excessive memory consumption caused by high resolution.

[0067] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0069] Figure 1 It is the overall framework of the point cloud and image hierarchical registration method based on cross-modal flow estimation of the present invention. DETAILED DESCRIPTION

[0070] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0071] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0072] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0073] See also Figure 1, which is a point cloud and image hierarchical registration method based on cross-modal flow estimation.

[0074] This invention aims to solve the accuracy problem of establishing cross-modal correlation in point cloud and image registration. After converting the point cloud into a deep synthetic image, a more suitable multi-source image registration method is explored on the basis of existing mature image registration. While ensuring the registration accuracy, the problem of model memory consumption is alleviated, and the generalization of the model is studied so that the model can still maintain a high accuracy when facing unfamiliar scenes that have not been trained.

[0075] Example

[0076] This embodiment proposes a point cloud and image hierarchical registration method based on cross-modal flow estimation, such as Figure 1 The overall framework structure of the method shown includes at least the following steps:

[0077] (1) Data Acquisition

[0078] The original point cloud data and two-dimensional image data are obtained, and the point cloud data is converted from the initial pose to the camera coordinate reference system, and then the point cloud depth map is obtained by using the camera intrinsic parameter projection.

[0079] (2) Feature extraction

[0080] It includes an RGB camera image feature extraction branch and a point cloud depth image feature extraction branch. Each branch adopts an encoder-decoder architecture to obtain multi-scale features. In particular, considering the data sparsity of the depth branch, a set of sparse convolutional layers is added before the encoder-decoder architecture of the depth branch to encrypt the sparse depth map into a dense depth map.

[0081] Encoder-Decoder Architecture

[0082] A convolutional network is used to extract features from the input image. The feature encoder is applied to the depth image I1 and the RGB image I2 to encode the input image into a dense feature map of lower resolution. Encoder g θ Output features at 1 / 32 resolution: R H×W×3 →R H / 32×W / 32×D , where D = 256 is set. The feature encoder consists of 6 residual blocks, which are then decoded to 1 / 16, 1 / 8, and 1 / 4 scales through a decoder with a similar structure to obtain multi-scale features for use in the next module.

[0083] Sparse Convolutional Layer

[0084] For the depth branch, considering the sparsity of the depth image, the sparse depth map is encrypted before the depth map feature extraction. The specific approach is to use a sparse convolution layer to evaluate only the observed pixels and appropriately standardize the output to account for sparsity. A mask is attached to each layer output, which normalizes the output of each convolution layer and adjusts the result to the visible area of ​​the previous layer. At the same time, the receptive field of the mask is expanded layer by layer. The dense result is input into a similar encoding and decoding structure, and the features at three scales are finally obtained for the next module.

[0085] (3) Cost body construction

[0086] Different matching cost calculation methods are used to construct the matching cost volume at different scales. At low scale, the full correlation matching strategy is used to generate the global matching cost volume, and at high scale, the local correlation matching strategy is used to generate the local matching cost volume.

[0087] (4) Correspondence estimation

[0088] First, regression prediction is performed at a low scale, using the existing correspondence, correlation, and potential hidden state as the input of the GRU convolution layer, and outputting the updated correspondence and hidden state. Then, the low-scale results are used as the initial value to guide, and the local correlation cost body is used to continue updating at a high scale, and finally an accurate pixel-level matching relationship between images is obtained.

[0089] (5) Loss function and network settings

[0090] The loss function uses a weighted loss function. The batch size batch_size is set to 2 during model operation, the number of iterations is 100, the Adam optimizer and dynamic learning rate are used, and the random flipping and linear stretching data enhancement strategies are used in the training set.

[0091] (6) Posture calculation

[0092] According to the pixel-level correspondence between the camera's intrinsic parameters, depth images, and RGB images, the PnP mathematical model is used to calculate the posture of the imaging device relative to the 3D point cloud reference system to complete the registration.

[0093] In step (1) of this embodiment, for two-dimensional image data, the input image is randomly cropped into an RGB image of size 320×960, the data is standardized, and input into the feature extraction module for feature extraction. For three-dimensional point cloud data, an initial pose is randomly generated within an error range of [±2m,±10°], and a corresponding depth map is generated based on the initial pose in the camera, which is also randomly cropped to a size of 320×960 and input into the feature extraction module for feature extraction.

[0094] In step (2) of this embodiment, in the sparse convolutional layer, for the depth branch, considering the sparsity of the depth image, the sparse depth map is encrypted before the depth map feature extraction. Let f represent the mapping from the input domain (e.g., intensity, depth) to the output domain (e.g., depth, semantics) achieved by the convolutional neural network. Assume that the input X = {x u,v} is only partially observed, O = {o u,v} represents the corresponding binary variable, indicating whether the input is observed ({o u,v}=1) or not observed ({o u,v}=0), the output of the standard convolutional layer in the convolutional neural network is calculated as follows:

[0095]

[0096] The kernel size is 2k+1, the weight is ω, and the bias is b. In order to evaluate only the observed pixels and properly normalize the output to explicitly account for sparsity, the output calculation of the convolutional layer introduces O = {o u,v},Right now:

[0097]

[0098] Add an ∈ to the denominator to avoid the case where no input pixel x is observed u+i,v+j The filter locations of are divided by zero. This is done by using sparse convolutional layers to only evaluate the observed pixels and appropriately normalize the output to account for sparsity. A mask is appended to each layer output that normalizes the output of each convolutional layer to adjust the result to the visible area of ​​the previous layer.

[0099] In step (2) of this embodiment, the encoder-decoder architecture uses a convolutional network to extract features from the input image. The feature encoder is applied to the depth image I1 and the RGB image I2 to encode the input image into a dense feature map of lower resolution. θ Output features at 1 / 32 resolution: R H×W×3 →R H / 32×W / 32×D , where D = 256 is set. The feature encoder consists of 6 residual blocks, which are then decoded to 1 / 16, 1 / 8, and 1 / 4 scales through a decoder with a similar structure to obtain multi-scale features for use in the next module.

[0100] In step (3) of this embodiment, a global matching cost volume is first generated at a low scale using a full correlation matching strategy, wherein the global matching cost volume is calculated as follows:

[0101] Visual similarity is calculated by constructing the complete matching cost between all pixel pairs, given the RGB image feature g θ (I1)∈RH*W*D and the deep feature g θ (I2)∈R H*W*D , the matching cost volume is formed by the dot products between all pairs of eigenvectors. The correlation volume C can be efficiently computed as a single matrix multiplication:

[0102] C(g θ (I1),g θ (I2))∈R H*W*H*W

[0103]

[0104] In step (3) of this embodiment, a local correlation matching strategy is used at a high scale to generate a local matching cost volume, wherein the calculation method of the local matching cost volume is based on the PatchMatch algorithm. In the propagation stage, a point in the feature map is regarded as a block, and several neighbor seed points are selected. Then, each point is moved by the offset of the neighbor point, and a candidate offset is obtained from its neighbors. Then, a multi-dimensional matching cost volume is calculated based on the neighbor candidate offset and its own offset;

[0105] Given a shift Δp, the correlation calculation of the propagation is defined as:

[0106] Corr=F1·W(F2,S(flow,Δp))

[0107] F1 and F2 represent features respectively. Figure 1 and feature map 2, flow represents the offset map, S(flow,Δp) represents the transformation of the offset map according to Δp, and W(.) represents the result of mapping F2 according to the offset map.

[0108] In step (4) of this embodiment, regression prediction is first performed at the 1 / 16 scale, using the existing correspondence, global correlation, and potential hidden state as the input of the GRU convolution layer, and outputting the updated correspondence and hidden state. At the subsequent 1 / 8 and 1 / 4 scales, the low-scale results are used as initial values ​​to guide the local correlation cost body at the high scale to continue updating, and finally obtain an accurate pixel-level matching relationship between images.

[0109] In step (5) of this embodiment, the loss function adopts a weighted loss function, the batch size batch_size is set to 2 during model operation, the number of iterations is 100, the Adam optimizer and dynamic learning rate are used, and the data enhancement strategy of random flipping and linear stretching is adopted in the training set, specifically:

[0110] The average endpoint error (EPE) is used as the pixel matching error for calculating the RGB image and the LiDAR depth image. Due to the sparsity of the point cloud, it is designed as follows:

[0111]

[0112] Among them, F gt (u,v) represents the true offset, F pre (u,v) is the prediction offset, mask uv Record which pixels have valid depth values ​​and thus real displacements.

[0113] Secondly, the loss function normal based on the surface normal vector introduces local geometric constraints to supervise model learning. First, the synthetic depth image is obtained based on the true value pose and the predicted pose. and Then from and Randomly sample 3 points from the common points of the valid depth value and calculate the surface normal vector n respectively. gt and n input Repeat this process N times, calculating the normal vector each time and Finally, the loss function is calculated according to the following formula:

[0114]

[0115] The final loss function consists of the weighted sum of the above loss functions. In this case, the weighting ratio α is 10, that is:

[0116] τ=τ epe +α×τ normal

[0117] In step (6) of this embodiment, after estimating the dense correspondence between the depth image and the RGB image, the correspondence between the 2D pixel positions in the camera image and the 3D points in the LiDAR map is inferred based on the camera intrinsic parameters K, and then the camera pose is estimated using the PnP solver in a RANSAC loop, where

[0118] Since the depth image is projected from a set of three-dimensional point clouds, after obtaining the dense correspondence between the depth image and the RGB image, the correspondence between the points in the three-dimensional point cloud and the two-dimensional image pixels is obtained based on the transitivity of the relationship. In order to solve the motion of 3D to 2D point pairs, the PnP mathematical model is used. PnP (Perspective-n-Point) is a method for solving the correspondence between 3D and 2D points. It describes how to estimate the camera's pose when n 3D space points and their positions are known. If the 3D position of a feature point in the two images is known, then at least 3 point pairs (and at least one additional verification point to verify the result) are required to calculate the motion of the camera. The solution process can be described as:

[0119]

[0120] Given the coordinates P of n points in the world coordinate system, the coordinates p of the corresponding pixel, and the camera intrinsic parameter K, solve the camera coordinate system (O c X c Y c Z c ) relative to the world coordinate system (O w X w Y w Z w ), that is, the [RT] matrix in the formula. Because the correspondence between the point cloud and the image obtained contains mismatches, which causes outliers in the estimated curve, the RANSAC random sampling consensus algorithm is used to estimate the pose parameters in the PnP mathematical model. The specific process is divided into four steps: 1. Given a data set S, select the minimum number of samples required to build a model from it, and record the selected data set as S1. 2. Use the selected data set S1 to calculate a mathematical model M1. 3. Use the calculated model M1 to test the remaining points in the data set. If the tested data point is within the allowable error range, the data point is judged as an inlier, otherwise it is judged as an outlier. The data set composed of all inliers is recorded as Called the consistency set of S1. 3. Compare the number of inliers between the current model and the best model previously introduced, and record the model parameters and the number of inliers when the number of inliers is the largest. 4. Repeat steps 1-4 until the iteration ends or the current model is good enough (the number of inliers is greater than the set threshold). Each model generated is either discarded because of too few inliers or selected because it is better than the existing model. The relative pose between the camera reference system and the world reference system that best fits the distribution of the corresponding relationship is obtained.

[0121] This example conducts comparative experiments on the KITTI dataset with other existing methods. The experimental results are as follows:

[0122] Method Transl.[cm] Rot.[deg] Fail[%] CMRNet 46 1.6 - CMRNet++ 55 1.46 2.18 HyperMap 48 1.42 - I2D-Loc 18 0.7 1.61 Our 16.25 0.49 1

[0123] Experiments show that advanced results have been achieved on the KITTI dataset. Compared with I2D-Loc, under the same training conditions, Transl.[cm] is improved from 18 to 16.25, and Rot.[deg] is improved from 0.7 to 0.49, which is a significant improvement.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A point cloud and image hierarchical registration method based on cross-modal flow estimation, characterized by: The method comprises the following steps: S1, data acquisition, obtaining the original point cloud data and two-dimensional image data, and converting the point cloud data from the initial pose to the camera coordinate reference system and then using the camera internal parameter projection to obtain the point cloud depth map; S2, feature extraction, extracting depth image features and RGB image features respectively through a feature extraction module, wherein the feature extraction module includes an RGB camera image feature extraction branch and a point cloud depth image feature extraction branch; S3, cost body construction, using different matching cost calculation methods to construct matching cost bodies at different scales, using a full correlation matching strategy to generate a global matching cost body at a low scale, and using a local correlation matching strategy to generate a local matching cost body at a high scale; S4, correspondence estimation, in the GRU convolutional layer, the optical flow is repeatedly updated by using the current estimate to find the matching cost from the relevant matching cost volume to obtain the pixel-level matching relationship between images; S5. Set the corresponding loss function and network parameters to optimize the above process; S6, posture solution, according to the pixel-level correspondence between the camera intrinsic parameters, depth image and RGB image, the PnP mathematical model is used to calculate the posture of the imaging device relative to the 3D point cloud reference system to complete the alignment.

2. The point cloud and image hierarchical registration method based on cross-modal flow estimation according to claim 1, characterized in that: In step S1, the RGB image and the point cloud depth image are randomly cropped in the training stage and centrally cropped in the inference stage, and are standardized. For the original three-dimensional point cloud data, an initial pose is randomly generated within a preset error range, and a corresponding sparse depth map is generated according to the camera intrinsic parameters and the initial pose, which is also randomly cropped to a preset size.

3. The point cloud and image hierarchical registration method based on cross-modal flow estimation according to claim 2, characterized in that: In step S2, each branch adopts an encoder-decoder architecture to obtain multi-scale features, wherein a set of sparse convolutional layers is added before the encoder-decoder architecture of the depth branch to encrypt the sparse depth map into a dense depth map; In the encoder-decoder architecture, a convolutional network is used to extract features from the input image. The feature encoder is applied to the depth image I1 and the RGB image I2 to encode the input image into a low-resolution dense feature map. Encoder g θ Output features at 1 / 32 resolution: R H×W×3 →R H / 32×W / 32×D Where D represents the number of channels; the feature encoder is composed of multiple residual blocks, and then passes through a decoder with a similar structure to decode the features to 1 / 16, 1 / 8, and 1 / 4 scales to obtain multi-scale features.

4. The point cloud and image hierarchical registration method based on cross-modal flow estimation according to claim 3, characterized in that: In the sparse convolutional layer of the depth branch, the sparse depth map is encrypted before the depth map feature extraction. The process is as follows: The sparse convolutional layer is used to evaluate only the observed pixels and standardize the output to take into account sparsity. A mask is attached to the output of each layer, which normalizes the output of each convolutional layer and adjusts the result to the visible area of ​​the previous layer. At the same time, the receptive field of the mask is expanded layer by layer, and the dense result is input into a similar encoding and decoding structure. Finally, features at three scales are obtained. Let f represent the mapping from the input domain to the output domain achieved by the convolutional neural network. Assume that the input X = {x u，v } is only partially observed, O = {o u，v } represents the corresponding binary variable, indicating whether the input is observed ({o u，v }=1) or not observed ({o u，v }=0), the output of the standard convolutional layer in the convolutional neural network is calculated as follows: The kernel size is 2k+1, the weight is ω, and the bias is b; The output calculation method of the convolutional layer introduces O = {o u，v } explicitly account for sparsity by evaluating only the observed pixels and normalizing the output, i.e.: Replace ∈ with a constant to avoid the denominator being 0, x u+i，v+j is the input pixel.

5. The point cloud and image hierarchical registration method based on cross-modal flow estimation according to claim 4, characterized in that: In step S3, a global matching cost volume is generated by adopting a full correlation matching strategy at a low scale, wherein the global matching cost volume is calculated as follows: Visual similarity is computed by constructing the complete matching cost between all pixel pairs. The matching cost volume is formed by the dot products between all pairs of eigenvectors; the correlation volume C is calculated as a single matrix multiplication: C(g θ (I1),g θ (I2))∈R H*W*H*W Among them, g θ (I1)∈R H*W*D For a given RGB image feature, g θ (I2)∈R H*W*D Given a deep image feature.

6. The point cloud and image hierarchical registration method based on cross-modal flow estimation according to claim 5, characterized in that: In step S3, a local correlation matching strategy is used at a high scale to generate a local matching cost body, wherein the calculation method of the local matching cost body is based on the PatchMatch algorithm. When calculating the local correlation, the offset value of the neighborhood point of the required point is used to calculate the corresponding position of the current point, and the offset matching cost is calculated; In the propagation stage, a point in the feature map is regarded as a block, and several neighbor seed points are selected. Each point is moved by the offset of the neighbor point, and the candidate offset is obtained from its neighbors. Then, a multi-dimensional matching cost volume is calculated based on the neighbor candidate offset and its own offset; Given a shift Δp, the correlation calculation of the propagation is defined as: Corr=F1·W(F2,S(flow,△p)) F1 and F2 represent feature map 1 and feature map 2 respectively, flow represents the offset map, S(flow,Δp) represents the transformation of the offset map according to Δp, and W(.) represents the result of mapping F2 according to the offset map.

7. The point cloud and image hierarchical registration method based on cross-modal flow estimation according to claim 6, characterized in that: In step S4, it specifically includes: First, regression prediction is performed at a low scale by taking the existing correspondence, global correlation, and potential hidden state as the input of the GRU convolutional layer and outputting the updated correspondence and hidden state; Then, the low-scale results are used as initial values ​​to guide the update using the local correlation cost volume at a high scale. Finally, accurate pixel-level matching relationship between images is obtained.

8. The point cloud and image hierarchical registration method based on cross-modal flow estimation according to claim 7, characterized in that: In step S5, the loss function uses a weighted loss function, and the batch size and number of iterations of the model are initialized. The Adam optimizer and dynamic learning rate are used, and the data enhancement strategy of random flipping and linear stretching is used in the training set. Specifically: The weighted loss function includes: using the average endpoint error EPE as the pixel matching error for calculating the RGB image and the LiDAR depth image, which is expressed as: where F gt (u, v) represents the true offset, F pre (u, v) is the prediction offset, mask uv Record pixels for which the pixel has a valid depth value; Secondly, a loss function based on the surface normal vector is used to introduce local geometric constraints to supervise model learning: it obtains a synthetic depth image based on the true value pose and the predicted pose respectively. and Then from and Randomly sample several points from the common points of the effective depth value and calculate the surface normal vector n respectively. gt and n input ; Repeat this process N times, calculating the normal vector each time and The L2 distance between them, finally, the loss function is calculated according to the following formula: The final loss function consists of the weighted sum of the above loss functions, namely: τ=τ epe +a×t normal Among them, α is the weighting coefficient.

9. The point cloud and image hierarchical registration method based on cross-modal flow estimation according to claim 8, characterized in that: In step S6, after estimating the dense correspondence between the depth image and the RGB image, the correspondence between the 2D pixel positions in the camera image and the 3D points in the LiDAR map is inferred based on the camera intrinsic parameters K, and then the camera pose is estimated in a RANSAC cycle using a PnP solver, where After obtaining the dense correspondence between the depth image and the RGB image, the correspondence between the points in the 3D point cloud and the pixels in the 2D image is obtained based on the transitivity of the relationship; The PnP mathematical model is used to solve the 3D to 2D point pair motion. The solution process is described as: Given the coordinates P of n points in the world coordinate system, the coordinates p of the corresponding pixel, and the camera intrinsic parameter K, solve the camera coordinate system (O c X c Y c Z c ) relative to the world coordinate system (O w X w Y w Z w ), which is the [RT] matrix in the formula; The RANSAC random sampling consensus algorithm is used to estimate the pose parameters in the PnP mathematical model. The specific process includes: 1) Given a data set S, select the minimum number of samples required to build a model, and denote the selected data set as S1; 2) Using the selected data set S1, a mathematical model M1 is calculated; 3) Use the calculated model M1 to test the remaining points in the data set. If the tested data point is within the allowable error range, the data point is judged as an inlier, otherwise it is judged as an outlier. The data set composed of all inliers is recorded as The consistent set is called S1; 4) Compare the number of inliers of the current model and the best model previously introduced, and record the model parameters and the number of inliers when the number of inliers is the largest; 5) Repeat steps 1 to 4 until the iteration ends or the number of internal points of the current model is greater than the set threshold; finally, the relative pose of the transformation between the camera reference system and the world reference system that best conforms to the corresponding relationship distribution is obtained.

Images