Dual-view point cloud reconstruction method based on rotation-invariant region consistency

Abstract

A double-view point cloud reconstruction method based on rotation invariant region consistency, by introducing a point cloud initialization network PINet, using PINet to generate point cloud x1 and point cloud x2 corresponding to the input RGB image of different views; using R2Net to divide the point cloud into N regions, and extracting the N region-level rotation invariant features of the point cloud, and using DCM to match the features and obtain the matching result E through the original cross attention mechanism; the AoA model is introduced into the original cross attention mechanism, and the optimized matching result pair is obtained and spliced, and the aggregated features F are obtained by aggregating the point clouds x1 and x2; using the aggregated features F and the point cloud x1 to generate a point cloud O with fine structure; feeding the point cloud O into the local folding network to obtain a point cloud Y with high smoothness, the point cloud Y contains structure information from different views, which is used to capture the complex topological structure of the input object.

Description

Technical fields:

[0001] The proposed dual-view point cloud reconstruction method based on rotation-invariant region consistency belongs to multi-view... Figure Three In the field of dimensional reconstruction, this involves exploring the rotation invariance of object regions, learning the region consistency of cross-view objects, and using multi-view feature aggregation to enhance reconstruction performance. Background technology:

[0002] Current Multi-view Figure Three The mainstream strategy for 3D reconstruction methods is to aggregate views with different features from an object using different methods, and then reconstruct the object's 3D shape by decoding the aggregated features. Therefore, feature aggregation is crucial in multi-view reconstruction. Figure Three It plays an important role in the reconstruction.

[0003] Early polyvision Figure Three 3D reconstruction methods use recurrent neural networks (RNNs) to aggregate features from multiple views and decode the aggregated features into a 3D shape. For example, 3D-R2N2 first uses a 3D convolutional neural network (3D-CNN) to extract features from multiple input views, aggregates these features using a 3D long short-term memory neural network (3D-LSTM), and then introduces a deconvolutional neural network to decode the aggregated features and generate a voxel representation of the 2D image. Kar et al. proposed a multi-view... Figure Three The 3D reconstruction method is called LSM. This architecture first uses a U-Net network to capture grid features from different views, then uses a recurrent neural network to calculate the relationships between these grid features to guide feature aggregation, and finally uses a 3D U-Net network to decode the aggregated features into voxel representations. R-MVSNet extracts features from multiple views using a convolutional neural network, while simultaneously employing gated recurrent units to selectively retain and further aggregate the extracted features to reconstruct the point cloud representation of the input object. However, due to the lack of long-term memory in recurrent neural networks, information from the input image is easily forgotten, making accurate multi-view feature aggregation impossible.

[0004] In addition, some works aggregate features from different views by pooling techniques or bilinear interpolation algorithms. For example, DeepMVS and 3DensiNet simply employ max pooling to aggregate multiple features from a set of images. 3D-RETR extracts features for each input image using ViT, and then aggregates these features by adaptive pooling to generate the point cloud representation of the input object. Pixel2Mesh++ series of methods fuse features from different views by bilinear interpolation, and use the fused features to guide the deformation of the template mesh to generate a three-dimensional shape describing the object surface. Although the above methods avoid the limitations of recurrent neural networks, they cannot model the relationship between multiple views and have relatively poor reconstruction performance.

[0005] To this end, some researches attempt to exploit the correlation between different views using attention mechanisms to guide multi-view feature aggregation. DVPC predicts a coarse point cloud for each view and has mutual information between coarse point clouds from different views captured by attention mechanisms, thus performing feature aggregation to reconstruct the point cloud representation. UMIFormer develops a Transformer network for multi-view reconstruction, which divides each image into fixed-size sub-image blocks, and then aggregates features from different views by attention mechanisms to exploit the correlation between sub-image blocks to reconstruct the three-dimensional shape of the object. Yang et al. proposed a long-range grouping based attention mechanism for multi-view reconstruction. It alternately processes the correlation within a single view and between different views to aggregate features, and then decodes the aggregated features using deconvolution and self-attention mechanisms to generate the voxel representation of the object. In fact, an object usually has regions with similar structures under different views, while there is a certain degree of rotation between these regions (as shown in FIG. 1). Therefore, exploring the rotational invariance of the object regions and learning the regional consistency of the object across views should help to aggregate multi-view features. However, existing methods have not explored this idea, resulting in inaccurate aggregation of features from different views and affecting the reconstruction performance. Figure Three

[18] Figure Three Figure 1

[0006] Therefore, this paper proposes a dual-view point cloud reconstruction method based on rotation-invariant regional consistency (DPR2), which consists of a view encoder and a point cloud decoder, and uses a pair of RGB images from different views as input to gradually recover the high-quality point cloud representation of a given object. SUMMARY

[0007] ​​​​A double-view point cloud reconstruction method based on rotation-invariant region consistency, comprising the following steps: 1: introducing a point cloud initialization network PINet, using the PINet network to generate a point cloud x1 and a point cloud x2 corresponding to the input RGB images of different views;

[0008] 2: R2Net adopts farthest point sampling and K nearest neighbors to divide the point cloud generated by PINet into N regions;

[0009] 3: R2Net extracts N region-level rotation-invariant features of the point cloud, so that the region features have rotation and translation invariance, and the region-level rotation-invariant features of the point cloud x1 and the point cloud x2 are respectively and wherein represents the rotation-invariant feature of the tth region of the point cloud x1, represents the rotation-invariant feature from the tth region of the point cloud x2;

[0010] 4: DCM matches the features by the original cross-attention mechanism and obtains a matching result E=[e1,...,e N ], represents the feature matching result of each region of x2 and the tth region of x1; through region feature matching, the region consistency of different view point clouds (x1 and x2) can be preliminarily obtained, that is and e n is a pair of region-consistent features;

[0011] 5: The AoA model is introduced to the original cross-attention mechanism to eliminate the error information in E and obtain an optimized feature matching result

[0012] 6: The features and the matching result are spliced to realize accurate aggregation of the rotation-invariant features of different regions of the point cloud x1 and the point cloud x2, and the aggregated region-level features are denoted as F, which is the feature aggregation result of the point cloud x1 and the point cloud x2;

[0013] 7: The PRNet is used to establish the relationship between the aggregated features F and the point cloud x1 to generate a second point cloud data with fine structure, denoted as point cloud O;

[0014] 8: The output O of the point cloud deformation module is fed into the local folding network to obtain a third point cloud Y with high smoothness, which contains structure information from different views and is used to capture the complex topological structure of the input object.

[0015] Further, the PINet network comprises a PSG unit and a convolutional gated recurrent neural network unit, the PINet network repeatedly performs convolution operation and deconvolution operation on the input RGB image by using an hourglass network, obtains shape attributes and local details of the RGB image, models the relationship between the shape attributes and the local details by using a convolutional gated recurrent neural network unit, and selectively updates the shape attributes and the local details in each iteration, and combines the updated shape attributes and the local details.

[0016] Further, the step 3 comprises: S301: calculating the Euclidean distance between the centroid point in the region and the neighborhood point, and taking the distance as the "edge" of the region; S302: extracting all "edges" in the region by using a multilayer perceptron, and combining all "edges" in the region to describe the basic structural information of the "edges"; and S303: improving the combination of the "edges" and explicitly depicting the structural information of the point cloud region.

[0017] Further, according to the original cross-attention, when the region feature matching of x1 and x2 is performed, is regarded as a query Q, is regarded as a key K and a value V; the DCM calculates the attention score between Q and the key K.

[0018] Further, in order to optimize the matching result E, the AoA model is introduced into the original cross-attention mechanism, only the relevant information between Q and K is retained, and the error information in E is eliminated.

[0019] Further, the point cloud deformation module is constructed in advance, the point cloud deformation model is used to improve the fineness of the point cloud, the point cloud deformation module adopts MLP and reshaping operation to decode F into a guide matrix T which has the same size as the point cloud x1, and is used to guide the deformation of each point in the point cloud x1 to generate the point cloud O.

[0020] This application takes two RGB images from different views as input and promotes multi-view feature aggregation by learning the rotation invariance and region consistency of the object region to reconstruct a high-quality point cloud. The rotation invariance of the object region is obtained using a region-level rotation-invariant feature extraction network in the view encoder. A two-stage cross-attention mechanism in the point cloud decoder is used to initially model the region consistency between two coarse point clouds. The point cloud refiner in the decoder explores the region consistency between point clouds from different views, thereby achieving accurate feature aggregation. Furthermore, the point cloud decoder uses the aggregated features to reconstruct a fine point cloud of the input object, generating a point cloud with fine-grained details and a smooth surface, significantly improving the quality of the reconstructed point cloud. Quantitative and qualitative comparisons with existing state-of-the-art methods will be presented below. Experiments on the ShapeNet and Pix3D datasets show that the proposed method outperforms existing state-of-the-art methods in terms of numerical comparisons and visualization results. Attached image description:

[0021] Figure 1 There is a certain degree of rotation between the regions.

[0022] Figure 2 Overall structure of the model method

[0023] Figure 3 Rotation-invariant feature extraction process for the t-th region of a point cloud

[0024] Figure 4 Two-stage attention mechanism structure diagram

[0025] Figure 5 For the ShapeNet dataset, compared with existing multi-view datasets... Figure Three Qualitative Comparison of 3D Reconstruction Methods

[0026] Figure 6 For the Pix3D dataset, compared with existing multi-view... Figure Three Qualitative Comparison of 3D Reconstruction Methods Detailed implementation method:

[0027] The following example, illustrated with diagrams, will provide a detailed explanation of this method.

[0028] This invention includes:

[0029] Step 1: Introduce a point cloud initialization network called PINet. It follows the structure of the point cloud generation network in the DN-Net method, and can process RGB images from different viewpoints in parallel, initializing a point cloud for each image without requiring camera parameters (see...). Figure 2The midpoint cloud x1 and the point cloud x2) are outputted and oriented according to the perspective of the given object. Specifically, PINet is composed of a PSG unit and a convolution gated recurrent neural network unit. It repeatedly performs convolution and deconvolution operations on the input RGB image using an hourglass network to obtain its shape attributes and local details, respectively. The convolution gated recurrent neural network unit models the relationship between the shape attributes and the local details and selectively updates them in each iteration to more effectively combine the two sets of information. Therefore, PINet has high flexibility in capturing the structure of the point cloud.

[0030] Step 2: R2Net uses Farthest Point Sampling and K-Nearest Neighbors to divide the rough point cloud generated by PINet into N regions, where N is set to 4.

[0031] Step 3: R2Net extracts features of the N regions of the point cloud while ensuring that these features are rotationally and translationally invariant. Figure 3 The rotationally invariant feature extraction process for the tth region of the point cloud is shown.

[0032] Specifically, S301: Calculate the Euclidean distance between the centroid point and the neighborhood points in the region, and consider this distance as the "edge" of the region. S302: Combine all "edges" in the region to describe the basic structural information of the "edges". S303: Further improve the combination of "edges" to clearly depict the structural information of the point cloud region.

[0033] S301 includes: assuming that the centroid point in the tth region of the point cloud is C t , the lth neighborhood point of point C t and point B tl constitute an "edge" J tl , which is calculated as follows:

[0034]

[0035] S302 includes: using a Multilayer Perceptron (MLP) to extract features of all edges in the tth region:

[0036] P t = [p t1 , p t2 ,..., p tL ] (2)

[0037] p tl = MLP(J tl ) (3)

[0038] wherein, represents the features of the l-th edge in the t-th region of the point cloud, represents the features of all edges in the t-th region of the point cloud, L represents that the region has L edges, and d is the dimension of the features. The MLP consists of three linear transformation layers.

[0039] In addition, inspired by the Transformer model, R2Net learns the correlation between the edges within a region by using a self-attention mechanism. According to the correlation, the edges within a region are further combined to describe the basic geometric structure of the region. Technically, P t is mapped to the query Q = P t W Q , the key K = P t W K , and the value V = P t W V wherein is a linear transformation matrix. The correlation w t between all edges in the t-th region can be calculated as:

[0040]

[0041] wherein d k represents a constant scaling factor.

[0042] Subsequently, w t is used to calculate the weighting of each edge with other edges, and the combination of all edges is calculated through a reshaping operation and a fully connected operation.

[0043] r t = reshape(MLP(w t V)) (5)

[0044] In the formula, the MLP consists of three linear transformation layers; reshape(·) represents a reshaping operation; is a rotation-invariant feature of the t-th region of the point cloud, and can describe the basic geometric structure of the region.

[0045] S303 comprises: based on the research finding that, regardless of whether the query and the key have relevant elements, the self-attention will always output a weighted average, which leads to the existence of false information in the combination of the edges. Therefore, step 5: an AoA model is introduced to measure the correlation between the combination result r t and the query Q and the key K, and filter the false information in r t . The AoA model consists of an information matrix and an attention gate matrix. In the t-th region, r t and Q are linearly transformed to obtain the information matrix I t , and r tand Q are further linearly transformed and sigmoid activated to construct the attention gate matrix B t .

[0046]

[0047] where, denotes the mapping matrix for the t-th region; d is the dimension of r t and Q; and σ denotes the sigmoid activation function.

[0048] B t The value of each channel in B t can be regarded as the correlation of the values on the corresponding channel of I t Therefore, performing Hadamard product with I t can improve the error information in r t .

[0049]

[0050] where, ⊙ denotes Hadamard product. denotes the improved feature of the t-th region of the point cloud, which can explicitly depict the geometric structure of the region.

[0051] Based on step 3, the region-level rotation-invariant features of point cloud x1 and point cloud x2 are extracted as and where denotes the rotation-invariant feature of the t-th region of point cloud x1, denotes the rotation-invariant feature from the t-th region of point cloud x2, t takes values from 1 to N.

[0052] Step 4: DCM matches features through original cross-attention mechanism and Preliminary modeling of region consistency between two coarse point clouds.

[0053] The feature matching process of step 4 is as follows: according to the original cross-attention, when matching the region features of x1 and x2, is regarded as the query Q, is regarded as the key K and the value V. DCM calculates the attention score between Q and the key K.

[0054]

[0055] where, denotes the linear transformation matrix, a nt describes the correlation between the n-th region of x1 and the t-th region of x2.

[0056] matrix represents the correlation between all regions of two point clouds, which is used to weight V to achieve feature matching between different regions of two point clouds.

[0057]

[0058] wherein, represents the feature matching result of N regions. represents the feature matching result of each region of x2 with the t-th region of x1.

[0059] Step 5: After the original cross-attention mechanism, the AoA model is introduced to optimize the matching result to obtain high-quality region consistency.

[0060] Specifically, through region feature matching, the region consistency of different view rough point clouds is preliminarily obtained, that is, and e n is a pair of features with region consistency. It is further observed that no matter whether the query Q and the key K are related, the original cross-attention mechanism will always calculate the attention score of Q and K. When Q and K are completely irrelevant, there will be error information in the matching result E. In order to further optimize E, the AoA model is introduced to the original cross-attention mechanism, only the relevant information between Q and K is retained, and the error information in E is eliminated.

[0061] First, information matrix Z and attention gate matrix Gate are constructed using E and Q, as shown in formula (11) and formula (12). Then, Z and Gate are Hadamard multiplied to eliminate misleading information in E, as shown in formula (13).

[0062]

[0063] wherein, represents the optimized feature matching result. Z represents the information matrix, and Gate represents the attention gate matrix. is a linear transformation matrix, d represents the dimension of E and Q. σ represents the sigmoid activation function, and represents the Hadamard product.

[0064] Through the above operation, DCM can learn the high-quality region consistency between cross-view rough point clouds, that is, and is a pair of features with high-quality region consistency.

[0065] Step 6: By splicing and the rotation-invariant features of different regions of rough point clouds can be accurately aggregated.

[0066]

[0067] where, denotes the characteristic aggregation result of x1 and x2.

[0068] Step 7: After DCM, PRNet is proposed. It can establish the connection between the aggregated features F and the coarse point cloud to produce a point cloud with fine structures and smooth surfaces. PRNet includes a point cloud deformation module and a local folding network.

[0069] First, a point cloud deformation module is constructed to improve the fineness of the coarse point cloud. Specifically, the point cloud deformation module uses MLP and reshaping operation to decode F into a guide matrix T with the same size as x1, which is used to guide the deformation of each point in x1.

[0070] T = reshape(MLP(F)) (15)

[0071] where, T represents the guide signal, which is a matrix with the same size as the point cloud x1. reshape(·) represents the reshaping operation. MLP consists of two linear transformation layers.

[0072] To achieve stable deformation of x1, this paper hopes to find a gating matrix c related to and to control the information flow of x1. A direct way to define c is to concatenate and and then map it to a weight matrix with the same size as x1. The calculation process of the gating matrix c is shown in equation (16), and the calculation process of the point cloud deformation is shown in equation (17).

[0073]

[0074] O = T + c ⊙ x1 (17)

[0075] where the values in the matrix c consist of real numbers between 0 and 1, σ represents the sigmoid activation function, and O represents the result of the point cloud deformation module.

[0076] Step 8: Previous studies have found that local folding networks are good at approximating smooth surfaces. Therefore, the output O of the point cloud deformation module is fed into this network to further improve its smoothness and generate a point cloud Y with a size of 8192x3. This point cloud contains structure information from different views, so it can capture the complex topological structure of the input object.

[0077] Quantitative and qualitative comparison with existing advanced methods:

[0078] Three-dimensional reconstruction on ShapeNet dataset:

[0079] To evaluate the reconstruction performance of DPR2, we quantitatively and qualitatively compared our proposed method with existing state-of-the-art methods 3D-R2N2, LSM, DV-Net, DVPC, P2M++, and MVP2M++ on the ShapeNet dataset.

[0080] Quantitative Comparison. To ensure fair comparison, the proposed method used the same training and testing data as all other methods. All results are derived from the published papers of methods P2M++, DVPC, and MVP2M++. As shown in Tables 1 and 2, for the average CD metric, DPR2 improves upon methods 3D-R2N2, LSM, DV-Net, DVPC, P2M++, and MVP2M++ by 80.00%, 57.92%, 30.22%, and 30.22%, respectively.

[0081] 23.02%, 23.62%, and 9.06%. For the average F-Score(λ), DPR2 improves upon 3D-R2N2, LSM, DV-Net, DVPC, P2M++, and MVP2M++ by 62.84%, 51.84%, 19.48%, 16.56%, 12.32%, and 7.72%, respectively. For the average F-Score(2λ), DPR2 improves upon 3D-R2N2, LSM, DV-Net, DVPC, P2M++, and MVP2M++ by 36.67%, 31.66%, 7.59%, 5.31%, 5.22%, and 1.42%, respectively. Although the above methods propose some advanced feature aggregation strategies, they do not explore object rotation invariance and neglect exploring regional consistency across view objects, making it difficult to accurately aggregate features from multiple views and resulting in poor reconstruction results.

[0082] Qualitative comparison. To more clearly illustrate the superiority of our proposed method, we conducted a qualitative comparison with existing state-of-the-art methods. For example... Figure 5 As shown, the voxels generated by 3D-R2N2 have low resolution and incomplete structures. DV-Net generates point clouds with many outliers and lacks detail. While DVPC can reconstruct relatively accurate point clouds, it still struggles to reconstruct objects with complex topological structures, such as the hollowed-out structure of the chair in row 2 of the figure. MVP2M++ can recover the 3D mesh of a given object, but the reconstruction using this method lacks reliability; for example, the reconstruction of the speaker in row 5 of the figure is not smooth enough. In contrast, DPR2's reconstruction is more visually expressive. It can clearly recover the slender structure of the chair legs and the hollowed-out structure of the armrests in rows 1, 2, and 5 of the figure. For the car in row 4, DPR2 can reconstruct the fine-grained details of the rear wing. Furthermore, it can effectively capture the smooth surfaces of the sofa and speaker.

[0083] 3D reconstruction on Pix3D dataset:

[0084] To verify the generalization ability of DPR2, we qualitatively compare the proposed method with DV-Net, DVPC and MVP2M++ on the real dataset Pix3D. For each input image, we use the mask provided by Pix3D to remove the background. To make a fair comparison, all methods are not pre-trained on Pix3D.

[0085] As shown in Figure 6 , the proposed method captures the fine-grained details of the object more clearly. The existing advanced methods are difficult to accurately aggregate multiple view features, resulting in insufficient generalization ability. DPR2 explores the regional consistency of cross-view objects from the perspective of object region rotational invariance, which can accurately realize multi-view feature aggregation. Even if the real-world images are not pre-trained, the proposed method can still reconstruct high-quality point clouds. For example, the proposed method captures the slender structure of the chair legs and restores the smooth surface of the bed and cabinet. Experiments show that DPR2 has good generalization ability for real-world objects.

[0086] Table 1 Quantitative comparison results on the ShapeNet dataset in the CD index with existing advanced methods

[0087]

[0088]

[0089] Table 2 Quantitative comparison results on the ShapeNet dataset in the F-Score index with existing advanced methods

[0090]

Claims

1. A method for dual-view point cloud reconstruction based on rotation-invariant region consistency, characterized in that: Step 1: introducing a point cloud initialization network PINet, using the PINet network to generate a point cloud x1 and a point cloud x2 corresponding to the input RGB images of different perspectives; Step 2: R2Net uses farthest point sampling and K-nearest neighbors to divide the point cloud generated by PINet into N regions; Step 3: R2Net extracts N region-level rotation-invariant features from the point cloud, so that the region features have rotation and translation invariance, and the region-level rotation-invariant features of the point cloud x1 and the point cloud x2 are respectively and wherein represents the rotation-invariant feature of the t-th region of the point cloud x1, represents the rotation-invariant feature from the t-th region of the point cloud x2, and t takes values from 1 to N; Step 4: DCM matches features through original cross-attention mechanism and Obtain matching results E = [e1,...,e N ], denote the matching results of the features of each region of x2 and the t-th region of x1; where a nt describes the correlation between the n-th region of x1 and the t-th region of x2; through region feature matching, the region consistency performance of different view point clouds x1 and x2 is preliminarily obtained, that is and e n is a pair of features with region consistency; Step 5: AoA model is introduced to the original cross-attention mechanism to eliminate the error information in E and obtain an optimized feature matching result Step 6: performing stitching on the features and matching results to achieve accurate aggregation of the rotation-invariant features of different regions of the point cloud x1 and the point cloud x2, and the aggregated region-level features are denoted as F, which are used for the feature aggregation results of the point cloud x1 and the point cloud x2. Step 7: using PRNet to establish a connection between the aggregated feature F and the point cloud x1 to generate a second point cloud data with fine structure, denoted as point cloud O; Step 8: feeding the output O of the point cloud deformation module into the local folding network to obtain a third point cloud Y with high smoothness, which contains structure information from different views for capturing the complex topological structure of the input object; The PINet network includes a PSG unit and a convolutional gated recurrent neural network unit, R2Net is used to calculate the Euclidean distance between the centroid point and the neighborhood point in the region, and the distance is regarded as the "edge" of the region. All "edges" in the region are combined to describe the basic structure information of the "edge", and the combination of the "edge" is further improved to clearly depict the structure information of the point cloud region. The AoA model is composed of an information matrix and an attention gate matrix; PRNet includes a point cloud deformation module and a local folding network.

2. The method of claim 1, wherein: The PINet network includes a PSG unit and a convolutional gated recurrent neural network unit. The PINet network uses an hourglass network to repeatedly perform convolution and deconvolution operations on the input RGB images, obtains the shape attributes and local details of the RGB images, uses the convolutional gated recurrent neural network unit to model the relationship between the shape attributes and the local details, and selectively updates the shape attributes and the local details in each iteration, and combines the updated shape attributes and the local details.

3. The method of claim 1, wherein: The step 3 includes: S301: calculating the Euclidean distance between the centroid point and the neighborhood point in the region, and regarding the distance as the "edge" of the region; S302: using a multilayer perceptron to extract all "edges" in the region, combining all "edges" in the region to describe the basic structure information of the "edge"; S303: improving the combination of the "edge" and clearly depicting the structure information of the point cloud region.

4. The method of claim 1, wherein: According to the original cross-attention, when region feature matching is performed on x1 and x2, is considered as a query Q, is considered as a key K and a value V; the DCM calculates an attention score between Q and the key K.

5. The method of claim 4, wherein: In order to optimize the matching result E, the AoA model is introduced into the original cross-attention mechanism, only the relevant information between Q and K is retained, and the error information in E is eliminated.

6. The method of claim 1, wherein: The point cloud deformation module is constructed in advance, which is used to improve the fineness of the point cloud. The point cloud deformation module uses MLP and reshaping operation to decode F into a guide matrix T with the same size as the point cloud x1, which is used to guide the deformation of each point in the point cloud x1 to generate the point cloud O.

Images