An unsupervised RGB-D point cloud registration method and device

By combining iterative methods of two-dimensional and three-dimensional convolution, strong discriminative point cloud features are extracted, and high-quality pseudo-poses are generated by spatial consistency screening and bidirectional rendering, the insufficient information fusion and pseudo-bits in unsupervised point cloud registration are solved. The problem of low posture accuracy achieves efficient and accurate RGB-D point cloud registration and effective training of network models.

CN117765040BActive Publication Date: 2024-12-03NAT UNIV OF DEFENSE TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202311788415.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-22
Publication Date
2024-12-03
Estimated Expiration
2043-12-22

AI Technical Summary

Technical Problem

The existing unsupervised point cloud registration method has shortcomings in feature extraction and pseudo-relative pose accuracy, resulting in insufficient information fusion and serious ambiguity in matching, affecting the training effect of network models.

Method used

It is an iterative way to combine two-dimensional image convolution and three-dimensional point convolution to fuse photometric and geometric information, and extract strong discriminant point cloud features through the back projection and splicing of visual center encoding and geometric center encoding features, and use it based on Subset sampling and bidirectional rendering of spatial consistency filtering generate high-quality pseudo-poses for unsupervised training of network models.

Benefits of technology

Improve the representation ability of point cloud features and the accuracy of pseudo-poses, enhance the generalization and computing efficiency of the network model across data sets, provide more effective supervision signals, reduce low-quality pose assumptions, and improve point correspondence. the internal point rate and discriminant characteristics of the characteristics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117765040B_ABST
    Figure CN117765040B_ABST
Patent Text Reader

Abstract

The present invention provides an unsupervised RGB-D point cloud registration method and device, the method comprising: obtaining geometric information of a point cloud and photometric information of an RGB image corresponding to the point cloud, fusing the geometric information and the photometric information, and extracting point cloud features; establishing point correspondences based on the point cloud features to obtain a point correspondence set; filtering the point correspondence set to obtain a valid subset and generate a pseudo pose; and re-rendering the point cloud geometrically and photometrically based on the pseudo pose to achieve unsupervised training of a network model. The present invention can achieve accurate unsupervised RGB-D point cloud registration, and has strong cross-dataset generalization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to an unsupervised RGB-D point cloud registration method and device. Background Art

[0002] Unsupervised point cloud registration refers to obtaining dense point-to-point correspondences between point clouds without manual annotation. Due to the lack of pose information between point clouds, unsupervised point cloud registration cannot supervise the point cloud feature representation obtained by the network model through the real point correspondence between the two frames of point clouds. Therefore, the unsupervised registration method needs to construct point correspondences between the two frames of point clouds, and calculate the pseudo-relative pose between the two frames of point clouds based on this, and use the pseudo-relative pose to supervise the point cloud features. This requires the network model to obtain point cloud feature representations with strong discriminability even without training. However, there is often serious ambiguity in obtaining point cloud features only through the consistency of the geometric structure between the two frames of point clouds. It is difficult to effectively distinguish different objects with similar geometric structures or points in areas with unclear geometric structures, which easily leads to a large number of outlier correspondences, which seriously affects the quality of the pseudo-pose.

[0003] In practical application scenarios, existing unsupervised point cloud registration methods train network models by simultaneously examining the consistency of two modalities. Although impressive results have been achieved, there are still many problems and challenges.

[0004] Specifically, on the one hand, in the process of extracting point cloud features, existing methods generally extract the features of the two modalities independently and simply splice them, which results in insufficient information fusion of the two modalities, thus affecting the discriminative ability of the point cloud features. For example, the direct back-projection model method, although theoretically able to simultaneously encode the geometric information and photometric information of the scene, has very limited performance improvements. For example, the decoupled splicing model method, although it can significantly improve the performance of the model, processes the data of the two modalities completely independently during feature extraction, ignoring the intrinsic correlation between the two modalities, resulting in the features of the two modalities still being independent, and the information fusion between the two modalities is not sufficient, which limits the descriptive ability and discriminativeness of the finally extracted point features, and there is still a relatively serious matching ambiguity.

[0005] On the other hand, the accuracy of the pseudo-relative pose generated by the existing methods is low, making it difficult to effectively supervise the unsupervised training of the network model. For example, differentiable random optimization is usually used to solve the pseudo-relative pose and align the two frames of point clouds, and then differentiable rendering is used to perform depth and photometric re-rendering of the point cloud. Finally, the network model is supervised by the geometric and photometric consistency between the re-rendered image and the input image. However, this method is difficult to achieve an effective balance between pose accuracy, computational overhead, and supervisory signal strength, which ultimately leads to the relatively low accuracy of the pseudo-relative pose generated by the existing methods. In particular, depth and photometric re-rendering have high requirements for the accuracy of the pseudo-relative pose. Inaccurate poses will cause serious deviations in the re-rendering results, causing the supervisory signal to tend to random noise, which cannot provide effective supervision for the network model and affects the convergence of the network model. Summary of the invention

[0006] The present invention provides an unsupervised RGB-D point cloud registration method and device, which are used to solve the technical problems of how to extract reliable point cloud features using data of two modalities and how to effectively supervise the point cloud features.

[0007] In one aspect, the present invention provides an unsupervised RGB-D point cloud registration method, the method comprising:

[0008] Acquire geometric information of a point cloud and photometric information of an RGB image corresponding to the point cloud, fuse the geometric information with the photometric information, and extract point cloud features;

[0009] Establishing point correspondence based on the point cloud features to obtain a point correspondence set;

[0010] Filtering the corresponding set of points to obtain a valid subset and generate a pseudo pose;

[0011] The point cloud is geometrically and photometrically re-rendered based on the pseudo-pose to achieve unsupervised training of the network model.

[0012] Furthermore, the point cloud feature extraction method includes:

[0013] The photometric information and the geometric information are fused in an iterative manner by using two-dimensional image convolution and three-dimensional point convolution; wherein the geometric features are projected into the image space, and the feature correlation in each two-dimensional local window is learned by two-dimensional convolution to obtain the visual center encoding feature; the visual features are back-projected into the three-dimensional space, and enhanced by three-dimensional point convolution, and the geometric information of the point cloud is explicitly embedded into the visual features to obtain the geometric center encoding feature;

[0014] The visual center coding feature is back-projected into the three-dimensional space again, and is concatenated with the geometric center coding feature to obtain the point cloud feature.

[0015] Furthermore, point correspondence is established based on the point cloud features, and the obtained point correspondence set includes:

[0016] for Every point p in i , find the point at The first matching point q with the largest feature similarity in mi And the second matching point q with the second largest feature similarity mi′ , and calculate the difference between the feature similarity of the point and the first matching point and the second matching point, and obtain the weight corresponding to the point; select the k points with the largest weight as the corresponding points from arrive The point corresponding to

[0017] In this way, for Each point q in i , and similarly select the k points with the largest weights as arrive The point corresponds to

[0018] Merge the point correspondences in the two directions together to generate the point correspondence set including 2k point correspondences

[0019] Where k is a positive integer, Representation based on depth image The resulting point cloud; Representation based on depth image The resulting point cloud.

[0020] Furthermore, at least one subset in the point correspondence set is a valid subset;

[0021] The point correspondences in the valid subset are all interior point correspondences.

[0022] Furthermore, by increasing the number of collected subsets, it is ensured that at least one subset in the point correspondence set is a valid subset;

[0023] Among them, the number of collected subsets n is: p is the inlier rate of the point.

[0024] Furthermore, the filtering includes:

[0025] For a given three point correspondences in a subset, the distance between any two points in c1=(x1, y1), c2=(x2, y2) and c3=(x3, y3) in the two frame point clouds is less than a preset distance threshold;

[0026] That is: c1, c2 and c3 are all interior point correspondences |||x i , x j ||-||y i ,y j |||<δ, where δ is the preset distance threshold.

[0027] Furthermore, a weighted SVD algorithm is used to calculate all interior point correspondences in the valid subset to generate the pseudo pose.

[0028] Further, geometrically and photometrically re-rendering the point cloud based on the pseudo pose includes:

[0029] Will The corresponding RGB image Back-projected onto the point cloud, the colored point cloud is recorded as in, For point p i The color vector of

[0030] Based on the pseudo pose T = {R, t}, Align to In the coordinate system, it is recorded as And use differentiable rendering to Rendered into depth image and RGB image; where p i * =Rp i +t; R is the three-dimensional rotation matrix, t is the three-dimensional translation vector;

[0031] In this way, based on the pseudo pose Align to In the coordinate system of And use differentiable rendering to Rendered into depth image and RGB image; where q i * =R -1 (q i -t);

[0032] When the two point clouds are aligned, unsupervised training of the network model is achieved by supervising the geometric and photometric consistency between the rendered depth and RGB images and the corresponding input images.

[0033] Furthermore, the overall loss function of the unsupervised training is for:

[0034]

[0035] in, is the corresponding loss function, is the geometric consistency loss function; is the photometric consistency loss function; corr is the balance coefficient of the corresponding loss function; geo is the balance coefficient of the geometric consistency loss function; photo is the balance coefficient of the photometric consistency loss function.

[0036] On the other hand, the present invention also provides an unsupervised RGB-D point cloud registration device, which includes at least a stacked vision and geometry encoding module and a re-rendering based robust unsupervised registration module, and is used to execute the steps of any of the above methods.

[0037] In general, the technical solution conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0038] (1) The present invention provides an unsupervised RGB-D point cloud registration method and device, which uses two-dimensional and three-dimensional convolutions to fuse photometric and geometric information in an iterative manner, so as to simultaneously utilize the inductive bias of the two convolutions and improve the representation ability of features. At the same time, a subset sampling method based on spatial consistency screening is used to achieve efficient and accurate pseudo-relative pose estimation, and a bidirectional rendering method is used to supervise the network model, which can achieve accurate unsupervised RGB-D point cloud registration and has strong cross-dataset generalization.

[0039] (2) The present invention provides an unsupervised RGB-D point cloud registration method and device, which samples a large number of corresponding subsets of three-element points and screens them through spatial consistency conditions. According to the spatial consistency of rigid transformation, the distances between any two corresponding internal points in the two frames of point clouds should be approximately equal, which can greatly reduce the number of low-quality pose assumptions, and ensure the computational efficiency of the network model while improving the accuracy of pose assumptions. At the same time, a bidirectional rendering method is used to supervise the network model, so that each frame of point cloud is aligned with another frame of point cloud through the obtained pseudo-relative pose, and after alignment, differentiable rendering is used for depth and photometric re-rendering; finally, by supervising the geometric and photometric consistency between the two frames of re-rendered images and the two frames of input images, unsupervised training of the network is achieved, which can provide more effective supervision signals for the training of the network model and solve the problem of how to effectively supervise point cloud features.

[0040] (3) The present invention provides an unsupervised RGB-D point cloud registration method and device, which can effectively filter invalid subsets by removing subsets that do not meet the conditions, thereby greatly reducing the number of subsets that need to be verified and significantly improving the computational efficiency. At the same time, as the training progresses, the discriminability of point cloud features gradually increases, and the internal point rate of the corresponding points also gradually increases. By reducing the number of sampled subsets accordingly, the computational efficiency is further improved.

[0041] (4) The present invention provides an unsupervised RGB-D point cloud registration method and device, which can utilize more point correspondences in training by first screening and then obtaining pseudo poses, thereby enhancing the gradient propagation of the network model and ensuring high-quality pseudo poses. In training, the pseudo poses are used to re-render the point cloud to achieve unsupervised training of the network model; in testing, the pseudo poses are directly used as the registration result between two frames of point clouds.

[0042] (5) The present invention provides an unsupervised RGB-D point cloud registration method and device, which simultaneously utilizes the inductive bias of two convolutions to fuse the information of the two modalities, thereby improving the feature representation capability, more fully fusing the information of different modalities, and thus extracting more discriminative point features, thus solving the problem of how to extract reliable point cloud features using data from two modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is a schematic diagram of the method steps of an unsupervised RGB-D point cloud registration method and device provided by the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings and embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0046] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method or device including the element.

[0047] like Figure 1As shown, the present invention provides an unsupervised RGB-D point cloud registration method, the method comprising:

[0048] Step 101: obtain geometric information of the point cloud and photometric information of the RGB image corresponding to the point cloud, fuse the geometric information and the photometric information, and extract point cloud features.

[0049] It should be noted that if two frames of RGB-D images are given and but are two frames of RGB images, is the corresponding depth image; and Two frames of depth images are and By camera parameters The three-dimensional point cloud obtained by back-projection can directly obtain the geometric information of the point cloud and the photometric information of the RGB image corresponding to the point cloud.

[0050] Due to the problems of insufficient information fusion and weak feature discriminability in existing geometric and photometric multimodal feature learning methods, the geometric information from the point cloud and the photometric information from the RGB image are fused to extract point cloud features with strong discriminability.

[0051] In the RGB-D point cloud registration task, point clouds and RGB images provide complementary geometric information and photometric information respectively. In order to fuse the information of the two modalities and extract point cloud features that perceive both geometric and photometric information, the direct back-projection model back-projects each pixel in the RGB image onto the point cloud, and then uses a point cloud feature learning network to extract features from the colored point cloud.

[0052] Specifically, the point cloud feature extraction method includes: iteratively using two-dimensional image convolution and three-dimensional point convolution to fuse photometric information and geometric information; wherein the fusion includes: projecting the geometric features into the image space, learning the feature correlation in each two-dimensional local window through two-dimensional convolution, and obtaining the visual center coding feature; back-projecting the visual features into the three-dimensional space, and enhancing them through three-dimensional point convolution, explicitly embedding the geometric information of the point cloud into the visual features, and obtaining the geometric center coding feature; finally, back-projecting the visual center coding feature into the three-dimensional space again, and splicing it with the geometric center coding feature to obtain the point cloud feature.

[0053] As a specific preferred embodiment of the present invention, for a given two-frame input point cloud and And their corresponding RGB images and First, two shallower modality-specific feature extraction branches are used to extract the initial point cloud geometric features and image visual features. Then, the features from the two modalities are fused and enhanced within each point cloud through the geometric center encoding module and the visual center encoding module. Finally, the features enhanced by the visual center encoding module are back-projected back to the three-dimensional space again and combined with the corresponding features extracted by the geometric center encoding module to obtain the final point cloud features that fully integrate photometric and geometric information.

[0054] For geometric features, KPConv is preferred, but it should be noted that geometric features are independent of the choice of point convolution and can be combined with any point convolution operation; for visual features, shallow 2DCNN is preferred.

[0055] Since each point cloud is calculated independently, the point cloud features are The calculation on The calculations above can be done in a similar way.

[0056] Specifically, after extracting the initial modal features, the geometric center encoding module is used to fuse the visual features into the geometric features and enhance the mixed features in three-dimensional space. The corresponding geometric characteristic matrix And its corresponding RGB image Visual feature map of The calculation of the geometric center encoding module includes two stages: feature back-projection and feature enhancement.

[0057] In the feature back-projection stage, the geometric features of each point in the point cloud are combined with its corresponding visual features in the RGB image. For example, for each p in the point cloud i , first calculate its The pixel coordinates after projection are then calculated based on the pixel coordinates after projection u i Retrieve the 4 nearest pixels and their characteristics, and calculate u by inverse distance interpolation i The corresponding visual features are obtained i Finally, the visual features of the point cloud are concatenated with its geometric features as the mixed features of the point cloud.

[0058] In the feature enhancement stage, the combined features are further enhanced through three-dimensional point convolution, so that it can better perceive the geometric structure of the point cloud. The point cloud features enhanced in this stage are recorded as For example, a fully connected layer is used to perform feature projection on the input features, and then two KPConv-based residual modules are used to enhance the features, and finally another fully connected layer is used to perform feature projection again to obtain the final output features.

[0059] Afterwards, the visual center encoding module is used to fuse the features enhanced by the geometric center encoding module into the visual features and enhance them in two-dimensional space. Similarly, the visual center encoding module also includes two stages: feature projection stage and feature enhancement stage.

[0060] In the feature projection stage, the mixed feature H from the geometric center encoding module is first 3D Scattered On the top, we get the corresponding two-dimensional projection feature, recorded as For example, the point p can be directly transformed into i The mixed feature h i 3D Projection to pixel u i On; you can also use differentiable rendering to 3D Scattered in a smoother way superior.

[0061] Preferably, the differentiable rendering includes: Each pixel v in j =(x, y), maintaining a bucket Used to store the points that contribute to the characteristics of the pixel. Every point p in i , put it into the distance u i The two-dimensional pixel distance does not exceed σ r According to the depth from near to far Sort the points in and then enumerate them in order Each point in the image is composited using Alpha to calculate v j Finally, the scattered mixed features With the initial visual feature F j 2D Spliced ​​together as a new mixed feature, recorded as

[0062] In the feature enhancement stage, G is convolved in the image space through two-dimensional image convolution. 2D Further enhancement is performed, and the enhanced features are recorded as For example, a structure symmetrical to that in the geometric center encoding module is used, including a 1×1 2D convolution layer, two 3×3 2D residual convolution modules, and another 1×1 2D convolution layer.

[0063] Through this iterative feature enhancement, the inductive bias of two-dimensional and three-dimensional convolutions is used to fuse the information from the two modalities, which more fully considers the correlation between the two modalities and extracts more discriminative point features.

[0064] The feature H obtained by the visual center encoding module is transformed into 2D Back-projected into three-dimensional space and combined with the feature H obtained by the geometric center encoding module 3D After that, the concatenated features are reduced in dimension through a linear projection and normalized to a unit hypersphere as the final output point cloud feature, denoted as

[0065] F=Norm(Concat(H 3D ,BackProject(H 2D ; K))W);

[0066] Among them, H 3D is the feature obtained by the geometric center encoding module; H 2D is the feature obtained by the visual center encoding module; K is the camera parameter; is the linear projection weight.

[0067] Step 102: Establish point correspondence based on point cloud features to obtain a point correspondence set.

[0068] That is, the point cloud features extracted from each frame of point cloud and its corresponding RGB image are extracted. and The points correspond to each other

[0069] Specifically, point correspondence is established based on point cloud features, and the point correspondence set obtained includes:

[0070] for Every point p in i , find the point at The first matching point q with the largest feature similarity in mi And the second matching point q with the second largest feature similarity mi′ , and calculate the difference between the feature similarity of the point and the first matching point and the second matching point to obtain the weight corresponding to the point.

[0071] It should be noted that the matching point q mi or q mi′ The calculation formula is: Weight ω i The calculation method is It can be seen that ω i The larger the value, the greater the pi With q mi There is a strong feature similarity between them, and mi′ The stronger the characteristic difference between them, the stronger the p i With q mi The ambiguity of the match between them is weaker, and they are more likely to be a pair of internal points; on the contrary, p i With q mi and q mi′ The features of p are similar, so i With q mi There is a strong matching ambiguity between them, and it is more likely that they are outlier correspondences.

[0072] Therefore, the present invention selects the k points with the largest weights as the corresponding points from arrive The point corresponds to in,

[0073] In this way, for Each point q in i , and similarly select the k points with the largest weights as arrive The point corresponds to

[0074] Finally, the point correspondences in the two directions are merged together to generate a point correspondence set including 2k point correspondences Where k is a positive integer, Representation based on depth image The resulting point cloud; Representation based on depth image The resulting point cloud.

[0075] It should be noted that for each point, its weight ω in the Lowe test i It is used to weight point correspondences in the subsequent pose calculation process.

[0076] Step 103: Filter the point correspondence set to obtain a valid subset and generate a pseudo pose.

[0077] After obtaining the point correspondence set, the pose is solved based on the extracted point correspondence for subsequent supervision of the network model. Align to The relative pose T∈Sε(3) contains a three-dimensional rotation matrix R∈Sε(3) and a three-dimensional translation vector

[0078] However, due to the lack of accurate pose information to supervise the training of the network model, the point correspondences will still contain a large number of outlier correspondences, which is particularly serious in the initial stage of training. This leads to inaccurate pseudo poses obtained, affecting the training of the network.

[0079] In order to obtain a more accurate pseudo-pose, existing methods use a "hypothesis-verification" approach to perform differentiable random optimization on the pseudo-pose. Specifically. First, several smaller subsets are randomly sampled from all point correspondences, and a pose is calculated for each subset using a weighted SVD algorithm. Here, each pose is called a hypothesis. Next, these pose hypotheses are verified on all point correspondences, and the pose that retains the most inliers is selected as the final pseudo-pose. However, the point correspondence subset sampled in this way will inevitably contain outlier correspondences, which will result in the accuracy of the pose being unsatisfactory even if the optimal pose hypothesis is selected through "hypothesis-verification".

[0080] In order to solve the above problems, a pseudo-pose solution method based on spatial consistency screening is proposed to obtain a more robust and accurate pseudo-pose. The core idea is to reduce the impact of outlier correspondences on pose solution by reducing the number of point correspondences sampled each time.

[0081] It should be noted that at least one subset in the point correspondence set is a valid subset; the point correspondences in the valid subset are all interior point correspondences.

[0082] In order to ensure that at least one subset of sampled points is all inliers, the number of sampled subsets is increased accordingly. Assuming that the inlier rate of point correspondence is p, then the confidence that at least one subset is all inliers is at least 0.999. The number of sampled subsets required can be obtained by solving 1-(1-p 3 )n>0.999, that is

[0083] That is, by increasing the number of collected subsets, it is ensured that at least one subset in the point correspondence set is a valid subset; wherein the number of collected subsets n is: p is the inlier rate of the point.

[0084] However, at the beginning of training, the point cloud features are relatively indiscriminate and the inlier rate of point correspondences is low. At this time, a large number of subsets need to be sampled to obtain accurate pseudo-poses. For example, when the inlier rate of point correspondences is p = 0.05, more than 55,000 subsets need to be sampled and verified, which will significantly increase the computational overhead of the model. In fact, only when a subset is full of inlier correspondences can it obtain a valid pseudo-pose; the other subsets are all invalid samples. Therefore, it is necessary to filter and screen the potential valid subsets.

[0085] Further, filtering includes: for a given three point correspondences in a subset, the distance between any two point correspondences in the two frame point clouds is less than a preset distance threshold; that is, c1, c2 and c3 are all inner point correspondences |||x i , x j ||-||y i ,y j |||<δ, where δ is the preset distance threshold.

[0086] By removing the subsets that do not meet this necessary condition, we can effectively filter out invalid subsets, thereby greatly reducing the number of subsets that need to be verified and significantly improving computational efficiency. As training progresses, the discriminability of point cloud features gradually increases, and the internal point rate of the corresponding points also gradually increases, which also reduces the number of sampled subsets and further improves computational efficiency.

[0087] Since the 6-DOF pose between two frames of point clouds can be uniquely determined by at least 3 pairs of non-collinear point correspondences, as a preferred embodiment of the present invention, only 3 pairs of point correspondences are sampled for each subset to solve a corresponding pose hypothesis. During back propagation, the gradient of the pose can only be transferred back to these 3 point correspondences, and a large number of point correspondences are not effectively supervised, which in turn affects the training of the network model.

[0088] Therefore, the present invention uses a weighted SVD algorithm to calculate all internal point correspondences in the valid subset to generate a pseudo pose, denoted as T = {R, t}. This method can utilize more point correspondences in training, thereby enhancing the gradient propagation of the network model while ensuring high-quality pseudo poses. In training, the pseudo pose is used to re-render the point cloud to achieve unsupervised training of the network model. In testing, the pseudo pose is directly used as the registration result between the two frames of point clouds.

[0089] Step 104: geometrically and photometrically re-render the point cloud based on the pseudo-pose to achieve unsupervised training of the network model.

[0090] That is, the two frames of point clouds are geometrically and photometrically re-rendered based on the acquired pseudo-pose, and the network model is trained by supervising the geometric and photometric consistency between the rendered depth and RGB images and the corresponding input images.

[0091] It is worth noting that this section adopts the cross-rendering method, which aligns a frame of point cloud to the coordinate system of another frame of point cloud for rendering, while the other frame of point cloud does not participate in the rendering process, thereby avoiding the problem of supervision signal degradation.

[0092] As a preferred embodiment of the present invention, geometric and photometric re-rendering of a point cloud based on a pseudo pose includes:

[0093] First, The corresponding RGB image Back-projected onto the point cloud, the colored point cloud is recorded as in, For point p i Color vector.

[0094] Then, based on the pseudo pose T = {R, t}, Align to In the coordinate system, it is recorded as And use differentiable rendering to Rendering to Depth Image and RGB images Among them, p i * =Rp i +t; R is the three-dimensional rotation matrix, and t is the three-dimensional translation vector.

[0095] It should be noted that the rendering method is the same as the above feature rendering method, using Alpha synthesis to calculate each pixel v j =(x, y) corresponding to the depth and color

[0096] Specific, deep The calculation formula is:

[0097]

[0098] depth The calculation formula is:

[0099]

[0100] in, For point The Z-axis coordinate of The subscript numbers of the points in the order are but Every point in In pixel v j Transparency For point In pixel v j The transmittance in .

[0101] In this way, based on the pseudo pose Align to In the coordinate system of And using differentiable rendering Rendering to Depth Image and RGB images Among them, q i * =R -1 (q i -t).

[0102] Since both the rendering process and the pseudo-pose solution process are differentiable, the entire network can be trained end-to-end.

[0103] When the two point clouds are aligned, unsupervised training of the network model is achieved by supervising the geometric and photometric consistency between the rendered depth and RGB images and the corresponding input images.

[0104] Depend on Rendered depth image Should be with The corresponding depth image is geometrically consistent, and Rendered RGB image of Should be with The corresponding RGB image The depth and RGB images are photometrically consistent and vice versa. Therefore, by maximizing the geometric and photometric consistency between the rendered depth and RGB images and the corresponding input images, unsupervised training of the network model can be achieved.

[0105] Furthermore, the overall loss function for unsupervised training is for:

[0106]

[0107] in, is the corresponding loss function, is the geometric consistency loss function; is the photometric consistency loss function; corr is the balance coefficient of the corresponding loss function; geo is the balance coefficient of the geometric consistency loss function; photo is the balance coefficient of the photometric consistency loss function.

[0108] A point correspondence loss is used to directly supervise the weights of the point correspondences extracted by the network. Specifically, given a set of point correspondences This section first uses the pseudo pose generated by the network model to align all point correspondences, and calculates the weighted squared offset error between all point correspondences as the point correspondence loss:

[0109]

[0110] For a point corresponding to (x i ,y i ), if it is an outlier correspondence, then its offset error is relatively large, and this loss will make its weight ω i becomes smaller; if it is an internal point correspondence, its cheap error is close to 0, and the loss is equal to its weight ω i Imposing weaker constraints.

[0111] The present invention supervises the geometric consistency between the rendered depth image and the input depth image through an l1 loss. However, due to the change in viewing angle, the rendered depth image may only cover a part of the image area; at the same time, due to factors such as occlusion and specular reflection, the input depth image usually has a part of the depth data missing. For example, the loss can be calculated in the area where both images have depth values; in addition, only The rendered image is The supervision is performed between the input images.

[0112] However, the integrity of the depth data under different perspectives may be different. Therefore, as a preferred embodiment of the present invention, the network model is supervised from two directions at the same time to improve the stability of training.

[0113] Specifically, by rendering the depth image With the input depth image The weighted l1 loss between them is the geometric consistency loss function To supervise, the calculation formula is:

[0114]

[0115] in, is the mask matrix.

[0116] Similarly, by rendering the RGB image With the input depth image The weighted l1 loss between them is the photometric consistency loss function To supervise, the calculation formula is:

[0117]

[0118] On the other hand, the present invention also provides an unsupervised RGB-D point cloud registration device, which includes at least a stacked vision and geometry encoding module and a robust unsupervised registration module based on re-rendering, and is used to execute the steps of any of the above methods. The technical features of the device are consistent with those of the above methods and will not be described one by one here.

[0119] In summary, the present invention proposes an unsupervised RGB-D point cloud registration method based on joint geometric and photometric consistency. On the one hand, a stacked visual and geometric encoding module is proposed, which uses two-dimensional and three-dimensional convolutions in an iterative manner to fuse photometric and geometric information, so that the inductive bias of the two convolutions can be used at the same time to improve the representation ability of features. On the other hand, a robust unsupervised registration module based on re-rendering is proposed, which realizes efficient and accurate pseudo-relative pose estimation through a subset sampling method based on spatial consistency screening, and uses a bidirectional rendering method to supervise the network model. The present invention can achieve accurate unsupervised RGB-D point cloud registration, and has strong generalization across data sets.

[0120] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. If there is no detailed description in a certain embodiment, please refer to the relevant description of other embodiments.

[0121] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0122] In addition, the technical solution of the present application, or the part that contributes to the prior art, or the whole or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, magnetic disk or optical disk, etc., various media that can store program codes.

[0123] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0124] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An unsupervised RGB-D point cloud registration method, characterized in that: The method includes: Acquire geometric information of a point cloud and photometric information of an RGB image corresponding to the point cloud, fuse the geometric information with the photometric information, and extract point cloud features; Establishing point correspondence based on the point cloud features to obtain a point correspondence set; including: Each point in , find the point at The first matching point with the largest feature similarity in And the second matching point with the second largest feature similarity , and calculate the difference between the feature similarity of the point and the first matching point and the second matching point, and get the weight corresponding to the point; select the one with the largest weight The points correspond to arrive The point corresponding to In this way, for Each point in , and also choose the one with the largest weight The points correspond to arrive The point corresponding to ; Merge the points in the two directions together to generate The points corresponding to the set of points ;in, is a positive integer, Representation based on depth image The resulting point cloud; Representation based on depth image The resulting point cloud; Filtering the corresponding set of points to obtain a valid subset and generate a pseudo pose; The point cloud is geometrically and photometrically re-rendered based on the pseudo-pose to achieve unsupervised training of the network model.

2. The unsupervised RGB-D point cloud registration method according to claim 1, characterized in that: The point cloud feature extraction method includes: The photometric information and the geometric information are fused in an iterative manner by using two-dimensional image convolution and three-dimensional point convolution; wherein the geometric features are projected into the image space, and the feature correlation in each two-dimensional local window is learned by two-dimensional convolution to obtain the visual center encoding feature; the visual features are back-projected into the three-dimensional space, and enhanced by three-dimensional point convolution, and the geometric information of the point cloud is explicitly embedded into the visual features to obtain the geometric center encoding feature; The visual center coding feature is back-projected into the three-dimensional space again, and is concatenated with the geometric center coding feature to obtain the point cloud feature.

3. The unsupervised RGB-D point cloud registration method according to claim 1, characterized in that: At least one subset in the point correspondence set is a valid subset; The point correspondences in the valid subset are all interior point correspondences.

4. The unsupervised RGB-D point cloud registration method according to claim 3, characterized in that: By increasing the number of collected subsets, it is ensured that at least one subset in the point correspondence set is a valid subset; Among them, the number of subsets collected for: ; is the inlier rate of the corresponding point.

5. The unsupervised RGB-D point cloud registration method according to claim 3, characterized in that: The filtering includes: For three points in a subset, , and The distance between any two points in the two frame point clouds is less than the preset distance threshold; Right now: ,in, is the preset distance threshold.

6. The unsupervised RGB-D point cloud registration method according to claim 3, characterized in that: The weighted SVD algorithm is used to calculate all interior point correspondences in the valid subset to generate the pseudo pose.

7. The unsupervised RGB-D point cloud registration method according to claim 1, characterized in that: Performing geometric and photometric re-rendering of the point cloud based on the pseudo pose includes: Will The corresponding RGB image Back-projected onto the point cloud, the colored point cloud is recorded as ,in, , for point The color vector of Based on pseudo pose ,Will Align to In the coordinate system, it is recorded as , and use differentiable rendering to Rendered into depth image and RGB image; among them, ; is the three-dimensional rotation matrix, is the three-dimensional translation vector; In this way, based on the pseudo pose Align to In the coordinate system of , and use differentiable rendering to Rendered into depth image and RGB image; among them, ; When the two point clouds are aligned, unsupervised training of the network model is achieved by supervising the geometric and photometric consistency between the rendered depth and RGB images and the corresponding input images.

8. The unsupervised RGB-D point cloud registration method according to claim 1, characterized in that: The overall loss function of the unsupervised training for: ; in, is the corresponding loss function, is the geometric consistency loss function; is the photometric consistency loss function; is the balance coefficient of the corresponding loss function; is the balance coefficient of the geometric consistency loss function; is the balance coefficient of the photometric consistency loss function.

Citation Information

Patent Citations

  • Three-dimensional point cloud environment real-time reconstruction method based on Kinect V2 sensor

    CN113256796A

  • Monocular depth and pose joint estimation method based on epipolar geometry

    CN113450410A

  • Rendering-based three-dimensional object detector training method

    CN114022602A

  • Systems and methods for point cloud registration

    US20220414821A1

  • KR20220041498A