Sparse view angle surface reconstruction method based on cross-view angle information complementation

Abstract

The invention discloses a sparse view angle surface reconstruction method based on cross-view angle information complementation. The method comprises a training stage and an inference stage. The training stage comprises the following steps of: firstly, inputting an input sparse view angle image into a network to obtain a multi-scale feature and construct a cost body; inputting the multi-view-angle cost body into a cross-view-angle geometric complementation module to obtain an optimized cost body; inputting the optimized cost body into a cross-view-angle geometry Transform, obtaining an image and a depth map of a new view angle through a body rendering technology, and using a truth value depth to supervise the training of a network; in the inference stage, the multi-view-angle images and the corresponding poses are input into the trained cross-view-angle information complementary sparse view angle surface reconstruction network, the depth of each view angle is obtained, and a final surface is obtained by using a depth image fusion algorithm. According to the invention, through explicitly constructing a cross-view complementary information fusion mode, the scene reconstruction accuracy and robustness under large view change are improved.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional reconstruction of computer vision technology, and specifically, is a sparse view surface reconstruction method based on cross-view information complementarity. Background Art

[0002] The 3D surface reconstruction technology aims to reconstruct the 3D surface of the entire scene from a set of calibrated images describing a scene. This task is of great significance to the practical application and development of mobile robots and virtual reality. It can assist mobile robots or virtual reality devices to capture images to reconstruct the 3D surface of the entire scene, thereby realizing the editing of 3D scenes or the accurate positioning of devices.

[0003] Three-dimensional reconstruction is a long-standing task in the field of computer vision. The current mainstream methods all use deep neural networks to infer the depth of each perspective, and then fuse them through the depth map fusion method to obtain the surface representation of the scene. Although these methods perform well on small baseline data sets, in practical applications, it may be difficult for users to capture such a small baseline image set. However, these methods perform poorly on more challenging large baseline image sets. In order to improve the practicality of these three-dimensional reconstruction methods, UFORecon, the method closest to the method of the present invention, introduces a feature matching Transformer to capture the correlation between pixel features of different perspectives, thereby improving the matching robustness under large perspective changes. However, due to the large perspective changes, there is still a lot of noise in the feature matching cost volume constructed by this method, and it is difficult for existing regularizers to infer accurate geometric representations from it, resulting in incorrect reconstruction results. Summary of the invention

[0004] The purpose of the present invention is to provide a sparse view surface reconstruction method based on cross-view information complementarity to solve the problems existing in the existing methods.

[0005] The technical solution for achieving the purpose of the present invention is: a sparse view surface reconstruction method based on cross-view information complementarity, including two processes: a training stage and an inference stage.

[0006] The training phase includes the following steps:

[0007] Step 1: Input the input sparse view image into the feature pyramid network to obtain multi-scale features;

[0008] Step 2: Use the features of multiple views and their corresponding poses to construct the cost volume, and regularize it using a 3D network;

[0009] Step 3: Input the multi-view cost volume into the cross-view geometry complementation module to obtain the optimized cost volume;

[0010] Step 4: Input the optimized cost volume into the cross-view geometry transformer to fully integrate the cost volume geometry under different viewpoints, and obtain the image and depth map of the new viewpoint through volume rendering technology;

[0011] Step 5: Supervise the training of the network by comparing the difference between the rendered depth and color and the true depth and color as the loss function.

[0012] The inference phase includes the following steps:

[0013] Step 1: Input a small set of images with large perspective transformation into the existing pose estimation framework to obtain the pose of each image relative to the world coordinate system.

[0014] Step 2: Input the multi-view images and their corresponding poses into the trained cross-view information complementary sparse view surface reconstruction network to obtain the depth of each view;

[0015] Step 3: Input each perspective depth map and the corresponding pose into the existing TSDF fusion framework to obtain the final surface reconstruction mesh.

[0016] Furthermore, in step 1 of the training phase, the input N sparse view images Input into the feature pyramid network to obtain multi-scale features Where l represents the subscript of the scale, and there are L scales in total.

[0017] Furthermore, in step 2 of the training phase, all source view images are aligned to the reference view through differentiable deformation, and the cost volume is constructed using feature dot product. The specific implementation process is:

[0018] Under the depth hypothesis d, the pixel p in the reference view is compared to the corresponding pixel in the source view The deformation between is defined as:

[0019]

[0020] Among them, R and t represent the rotation and translation transformation between the two views, K0 and K represent the intrinsic parameter matrices of the reference image and the source image; then, the feature correlation of the pixels between the reference image and the source image is calculated as follows:

[0021]

[0022] in, is the feature matching cost volume between the reference image and the i-th source image, D, H, W are the number of depth hypotheses and the height and width of the image, and <·> is the dot product between vectors; the feature matching cost volume is input into a 3D U-Net to regularize it and obtain a smooth geometric volume feature and probability volume features The estimated intermediate depth is obtained by weighting the probability volume feature and the depth hypothesis.

[0023] Furthermore, the specific implementation process of the cross-view geometry complementation module in step 3 of the training phase is as follows:

[0024] In the cross-view geometric complementation module, a confidence map is first estimated for each view. Based on this confidence map, the geometric information of different viewpoints can be adaptively fused. The cross-view geometric complementation module constructs a cross-view geometric volume and a multi-view based on image blocks. Figure 1 Finally, the cross-view geometric information captured by these two volumes is added to the feature volume of each view to obtain the optimized cost volume. The specific process is as follows:

[0025] Step 3.1: Estimate the confidence map for each view; use entropy to measure the depth probability distribution along the depth direction as the prior for the depth confidence; the entropy prior is finally input into a 2D network ψ c The final confidence map is obtained The process is described as follows:

[0026]

[0027] Step 3.2: Construct a cross-view geometry. The cross-view geometry uses the previously calculated confidence map as a clue to adaptively fuse the geometric volume features under different viewpoints. The cross-view geometry G is defined as a sparse volume with a resolution of R×R×R in the world coordinate system, that is, it only stores voxels with geometric volume features, but not empty voxels. For the element at position p in the cross-view geometric volume, it will be projected to each viewpoint to obtain the confidence and geometric volume features of the corresponding position. The confidence is converted into a weighted expression as:

[0028]

[0029] where π i (·) represents the projection function of the i-th view; finally, the geometric volume element across viewing angles at position p is calculated as:

[0030]

[0031] Step 3.3: Construct multi-view Figure 1First, use the intermediate depth map D obtained in step 2 i Estimate the normal vector map for each view

[0032]

[0033] Where A is the matrix composed of all pixel coordinates in the local window, 1 is a vector with all elements equal to 1; then, the aggregation weight w calculated in step 3.2 is i The normal vectors from different perspectives are combined to obtain the aggregated normal vector:

[0034] According to the normal vector, the coordinate representation of a local plane block in space is obtained; the two basis vectors b1 and b2 perpendicular to the normal vector in space are calculated as:

[0035] b1=b a ×n,b2=b1×n

[0036] Where × represents the outer product of the vector, b a =(1,0,0) T is a predefined anchor vector; next, the world coordinate system coordinates p of each grid point of the k×k local plane block uv is calculated as:

[0037] p uv =c+τ(ub1+vb2)

[0038] Where c is the coordinate of the element center, τ is the scale factor that controls the size of the local plane block in the world coordinate system. After obtaining the coordinates of the local plane block, it is projected onto the image of each perspective to obtain the features corresponding to the image block.

[0039] The consistency score s is calculated using the two perspectives with the largest corresponding confidence. The calculation process is expressed as:

[0040]

[0041] Step 3.4: Adaptive geometry fusion; after obtaining the cross-view geometry and multi-view according to steps 3.2 and 3.3, Figure 1 After the integration, they are connected together along the channel direction and input into a 3DU-Netψ u To fuse the geometric information of different cross-views, we finally get a unified geometric body U. This process is expressed as:

[0042] U=ψ i ([G,S])

[0043] The unified geometric body U is projected into each view to obtain a geometrically consistent body Then the geometrically consistent and the original geometry V i Combined to get the final optimized geometry

[0044]

[0045] where ψ f is a set of 3D convolutional layers; through the optimized geometry The optimized intermediate depth map is used as the prior for the subsequent construction of the feature matching cost volume.

[0046] Furthermore, the specific implementation process of the cross-view geometry Transformer in step 4 of the training phase is as follows:

[0047] First, the coordinates of the sampling point under the new perspective in the world coordinate system are obtained: p = o + td, where o represents the coordinates of the camera center of the new perspective in the world coordinate system, d represents the direction of the ray corresponding to the pixel point in the world coordinate system, and t represents the depth value of the sampling point; the jth sampling point is projected onto the multi-scale geometric body under different perspectives to obtain the corresponding geometric features.

[0048] Then according to its corresponding confidence They are fused through a multi-layer perceptron network MLP. The specific process is as follows:

[0049]

[0050] Next, the fused geometric features corresponding to the sampling points on the ray are input into a self-attention network layer SelfAttn to fully evaluate the reliability of each sampling point. The specific process is as follows:

[0051]

[0052] where γ j represents the position code of the jth sampling point, M represents the total number of sampling points on the ray; then the reliability score r is converted into ij Convert to weights to get the fused geometric features

[0053]

[0054] These fused geometric features are input into a self-attention network layer to capture the geometric features along the ray, and then encoded into SRDF values ​​through a multi-layer perceptron network MLP.

[0055]

[0056] Finally, the SRDF values ​​of the sampling points are obtained through the volume rendering method in NeuS to obtain the rendered image and depth map.

[0057] Furthermore, the specific process of loss function and training in step 5 of the training phase is:

[0058] The overall loss function is defined as:

[0059]

[0060] Where α and β are balance factors respectively; color loss function Defined as:

[0061]

[0062] Where S represents the number of effective pixels, Represents the rendered color map, C s Represents the true value of the color image; renders the depth loss Defined as:

[0063]

[0064] in, and D s Represents the rendered depth map and the depth truth respectively; multi-view depth loss Defined as:

[0065]

[0066]

[0067] where λ l represents the balance weight at each stage, and Represents the predicted depth map and true depth of each layer; calculates the above loss function value during the training process, uses the gradient descent method to update the network parameters in the model until the model converges, and obtains the trained sparse perspective reconstruction model.

[0068] Compared with the prior art, the present invention has the following significant advantages:

[0069] (1) The present invention constructs a cross-view geometry complementation module, which adaptively integrates reliable geometric information from different viewpoints by constructing geometric bodies in a unified space, optimizes the inaccurate geometry in each viewpoint, and thus improves the geometric consistency and accuracy in each viewpoint. Compared with previous methods, the present invention uses reliable geometry from different viewpoints to mutually optimize and complement the geometric accuracy of each viewpoint, thereby improving the accuracy of the final scene reconstruction.

[0070] (2) The present invention constructs a cross-view geometry transformer that can robustly fuse reliable geometry from different viewpoints during depth map rendering. Compared with previous methods, the present invention considers geometric information along the ray, fully evaluates the reliability of the sampling point at each viewpoint geometry, reduces the impact of occlusion or geometric inaccuracy on cross-view geometry fusion, and thus improves the accuracy of reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is the overall framework diagram of the cross-view information complementary reconstruction network in the method of the present invention.

[0072] Figure 2 It is a framework diagram of the cross-view geometry complementarity module in the method of the present invention.

[0073] Figure 3 It is a framework diagram of the cross-view geometry Transformer in the method of the present invention.

[0074] Figure 4 It is a flow chart of the training phase of the method of the present invention.

[0075] Figure 5 It is a flow chart of the reasoning stage of the method of the present invention. DETAILED DESCRIPTION

[0076] In order to enhance the accuracy and completeness of reconstruction of existing methods under large baseline image sets, the method of the present invention uses cross-view geometric information to complement the geometric representation of each view. Specifically, the method constructs a cross-view geometric body for fusing geometric information from different views and a multi-view geometric body for providing accurate Figure 1 Multi-view of consistent information Figure 1 The two bodies together provide accurate geometric information. In addition, in the rendering stage, this method uses an explicit Transformer to comprehensively evaluate the reliability of rays at each perspective, thereby reducing the impact of occlusion or geometric inaccurate perspective on cross-perspective geometric fusion.

[0077] The present invention will be further described below in conjunction with the accompanying drawings.

[0078] like Figure 4 As shown, the present invention provides a technical solution: a sparse view surface reconstruction method based on cross-view information complementarity, including two processes: a training stage and an inference stage.

[0079] The specific steps of the training phase process are:

[0080] Step 1: Input N sparse view images Get multi-scale features from the feature pyramid network Where l represents the subscript of the scale, and there are L scales in total.

[0081] Step 2: Utilize the above multi-scale features Construct the feature matching cost volume. Here, the process of constructing the feature matching cost volume is described using the features of one scale. For simplicity, the scale subscript l is omitted in the following description. Differentiable deformation is used to align all source view images to the reference view. Under the depth assumption d, the pixel p in the reference view is aligned with the corresponding pixel in the source view. The deformation between is defined as:

[0082]

[0083] Where R and t represent the rotation and translation transformation between the two views, and K0 and K represent the intrinsic parameter matrices of the reference image and the source image. Then, the feature correlation between the pixels of the reference image and the source image can be calculated as follows:

[0084]

[0085] in is the feature matching cost volume between the reference image and the i-th source image, D, H, W are the number of depth hypotheses and the height and width of the image, and <·> is the dot product between the vectors. The feature matching cost volume is input into a 3D U-Net to regularize it and obtain a smooth geometric volume feature. and probability volume features The estimated intermediate depth can be obtained by weighting the probability volume feature and the depth hypothesis.

[0086] Step 3: Combine Figure 2 , the multi-view cost volume is input into the cross-view geometric complementation module to obtain the optimized cost volume. In the cross-view geometric complementation module, the confidence map is first estimated for each view. According to this confidence map, the geometric information of different views can be adaptively fused. The cross-view geometric complementation module constructs the cross-view geometric volume and the multi-view based on image blocks. Figure 1 Finally, the cross-view geometric information captured by these two volumes is supplemented into the feature volume of each view to obtain the optimized cost volume.

[0087] Step 3.1: Estimate the confidence map for each view. Here, entropy is used to measure the depth probability distribution along the depth direction as a priori for the depth confidence. The entropy prior is finally input into a simple 2D network ψ c The final confidence map is obtained The process is described as follows:

[0088]

[0089] Step 3.2: Construct cross-view geometry. The cross-view geometry uses the previously calculated confidence map as a clue to adaptively fuse the geometric volume features under different viewpoints. The cross-view geometry G is defined as a sparse volume with a resolution of R×R×R in the world coordinate system, that is, it only stores voxels with geometric volume features, but not empty voxels. For the element at position p in the cross-view geometric volume, it will be projected to each viewpoint to obtain the confidence and geometric volume features of the corresponding position. The confidence converted to weight can be expressed as:

[0090]

[0091] where π i (·) represents the projection function of the i-th view. Finally, the geometric volume element across viewing angles at position p can be calculated as:

[0092]

[0093] Step 3.3: Construct multi-view Figure 1 In order to better handle a wide range of viewing angle changes, the present invention adopts a block-based multi-view Figure 1 Consistency score. Figure 1 The resolution of the volume S is the same as that of the cross-view geometry G and both are sparse volumes. First, use the intermediate depth map D obtained in step 2 i Estimate the normal vector map for each view:

[0094]

[0095] Where A is the matrix of all pixel coordinates in the local window, and 1 is a vector whose elements are all 1. Then, the aggregation weight w calculated in step 3.2 is i The normal vectors from different perspectives can be fused to get the aggregated normal vector:

[0096]

[0097] According to the normal vector, the coordinate representation of a local plane block in space can be obtained. The two basis vectors b1 and b2 perpendicular to the normal vector in space can be calculated as:

[0098] b1=b a ×n,b2=b1×n

[0099] Where × represents the outer product of the vector, b a =(1,0,0) T is a predefined anchor vector. Next, the world coordinate system coordinates of each grid point of the k×k local plane block p uv can be calculated as:

[0100] p uv =c+τ(ub1+vb2)

[0101] Where c is the coordinate of the element center, τ is the scale factor that controls the size of the local plane block in the world coordinate system. After obtaining the coordinates of the local plane block, it can be projected onto the image of each perspective to obtain the features corresponding to the image block. Due to the large difference in perspectives, there may be serious occlusions. Here, we simply use the two perspectives with the largest corresponding confidence to calculate the consistency score s. The calculation process is expressed as:

[0102]

[0103] Step 3.4: Adaptive geometry fusion. After obtaining the cross-view geometry and multi-view according to steps 3.2 and 3.3, Figure 1 After the transformation, they are concatenated along the channel direction and input into a 3DU-Netψ u To fuse the geometric information of different cross-views, we finally get a unified geometric body U. This process can be expressed as:

[0104] U=ψ i ([G,S])

[0105] In order to obtain a more accurate geometry with fine edges, the unified geometry U is projected onto each view to obtain a geometrically consistent geometry. Then the geometrically consistent and the original geometry V i Combined to get the final optimized geometry

[0106]

[0107] where ψ f It is a set of 3D convolutional layers. Through the optimized geometry The optimized intermediate depth map can be obtained as a priori for the subsequent construction of the feature matching cost volume.

[0108] Step 4: Combine Figure 3 , the optimized cost volume is input into the cross-view geometry Transformer, the cost volume geometry under different viewpoints is fully integrated, and the image and depth map of the new viewpoint are obtained through volume rendering technology. The specific implementation process is as follows: First, the coordinates of the sampling points under the new viewpoint in the world coordinate system are obtained:

[0109] p = o + td, where o represents the coordinates of the camera center of the new perspective in the world coordinate system, d represents the direction of the ray corresponding to the pixel point in the world coordinate system, and t represents the depth value of the sampling point. Project the jth sampling point onto the multi-scale geometric body at different perspectives to obtain the corresponding geometric features. Then according to its corresponding confidence They are fused through a multi-layer perceptron network (MLP), the specific process is as follows:

[0110]

[0111] Next, the fused geometric features corresponding to the sampling points on the ray are input into a self-attention network layer (SelfAttn) to fully evaluate the reliability of each sampling point. The specific process is as follows:

[0112]

[0113] where γ j represents the position code of the jth sampling point, and M represents the total number of sampling points on the ray. Then the reliability score r is converted into ij Converted into weights, we can get the fused geometric features:

[0114]

[0115]

[0116] These fused geometric features are input into a self-attention network layer to capture the geometric features along the ray, and then encoded into SRDF values ​​through a multi-layer perceptron network (MLP):

[0117]

[0118] Finally, the SRDF values ​​of the sampling points can be used to obtain the rendered image and depth map through the volume rendering method in NeuS.

[0119] Step 5: Use the difference between the rendered depth and color and the true depth and color as the loss function to supervise the training of the network. The specific implementation process is: The total loss function is defined as:

[0120]

[0121] Where α and β are balance factors. Color loss function Defined as:

[0122]

[0123] Where S represents the number of effective pixels, Represents the rendered color map, C s Represents the true value of the color image. Rendering depth loss Defined as:

[0124]

[0125] in, and D s Represent the rendered depth map and the depth truth respectively. Multi-view depth loss Defined as:

[0126]

[0127] where λ l represents the balance weight at each stage, and The predicted depth map and true depth of each layer are represented. During the training process, the above loss function value is calculated, and the network parameters in the model are updated using the gradient descent method until the model converges to obtain a trained sparse perspective reconstruction model.

[0128] The specific steps of the testing phase process are:

[0129] Step 1: Input a small number of images with large perspective transformations into an existing pose estimation framework, such as Colmap, to obtain the pose of each image relative to the world coordinate system.

[0130] Step 2: Input the multi-view images and their corresponding poses into the trained cross-view information complementary sparse view surface reconstruction network to obtain the depth of each view;

[0131] Step 3: Input each view depth map and the corresponding pose into the existing TSDF depth map fusion framework to obtain the final surface reconstruction mesh.

[0132] The present invention is further described below in conjunction with the accompanying drawings and embodiments of the specification. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0133] like Figure 4 and Figure 5 As shown, the present invention provides a technical solution: a sparse view surface reconstruction method based on cross-view information complementarity, including two processes: training stage and reasoning stage. The overall network architecture is shown in Figure 1 shown.

[0134] The specific steps of the training phase process are:

[0135] Step 1: Input N sparse view images Get multi-scale features from the feature pyramid network Where l represents the subscript of the scale, which includes L scales in total, and this method includes 3 scales.

[0136] Step 2: Utilize the above multi-scale features Construct the feature matching cost volume. Here, the process of constructing the feature matching cost volume is described using the features of one scale. For simplicity, the scale subscript l is omitted in the following description. Differentiable deformation is used to align all source view images to the reference view. Under the depth assumption d, the pixel p in the reference view is aligned with the corresponding pixel in the source view. The deformation between is defined as:

[0137]

[0138] Where R and t represent the rotation and translation transformation between the two views, and K0 and K represent the intrinsic parameter matrices of the reference image and the source image. Then, the feature correlation between the pixels of the reference image and the source image can be calculated as follows:

[0139]

[0140] in is the feature matching cost volume between the reference image and the i-th source image, D, N, W are the number of depth hypotheses and the height and width of the image, and <·> is the dot product between vectors. In this method, the depth hypothesis of each stage is set to 48, 32, and 8, and the height and width of the image are set to 512 and 640 respectively. The feature matching cost volume is input into a 3D U-Net to regularize it and obtain a smooth geometric volume feature and probability volume features The estimated intermediate depth can be obtained by weighting the probability volume feature and the depth hypothesis.

[0141] Step 3: Input the multi-view cost volume into the cross-view geometric complementation module to obtain the optimized cost volume. In the cross-view geometric complementation module, a confidence map is first estimated for each view. Based on this confidence map, the geometric information of different views can be adaptively fused. The cross-view geometric complementation module constructs a cross-view geometric volume and a multi-view based on image blocks. Figure 1 Finally, the cross-view geometric information captured by these two volumes is supplemented into the feature volume of each view to obtain the optimized cost volume.

[0142] Step 3.1: Estimate the confidence map for each view. Here, entropy is used to measure the depth probability distribution along the depth direction as a priori for the depth confidence. The entropy prior is finally input into a simple 2D network ψ c The final confidence map is obtained The process is described as follows:

[0143]

[0144] Step 3.2: Construct cross-view geometry. The cross-view geometry uses the previously calculated confidence map as a clue to adaptively fuse the geometric volume features under different viewpoints. The cross-view geometry G is defined as a sparse volume with a resolution of R×R×R in the world coordinate system, that is, it only stores voxels with geometric volume features, but not empty voxels. In this method, the resolution R is set to 96. For the element at position p in the cross-view geometric volume, it will be projected to each viewpoint to obtain the confidence and geometric volume features of the corresponding position. The confidence converted to weight can be expressed as:

[0145]

[0146] where π i (·) represents the projection function of the i-th view. Finally, the geometric volume element across viewing angles at position p can be calculated as:

[0147]

[0148] Step 3.3: Construct multi-view Figure 1 In order to better handle a wide range of viewing angle changes, the present invention adopts a block-based multi-view Figure 1 Consistency score. Figure 1 The resolution of the volume S is the same as that of the cross-view geometry G and both are sparse volumes. First, use the intermediate depth map D obtained in step 2 i Estimate the normal vector map for each view:

[0149]

[0150] Where A is the matrix of all pixel coordinates in the local window, and 1 is a vector whose elements are all 1. The local window size in this method is 7×7. Then, the aggregation weight w calculated in step 3.2 is i The normal vectors from different perspectives can be fused to get the aggregated normal vector:

[0151]

[0152] According to the normal vector, the coordinate representation of a local plane block in space can be obtained. The two basis vectors b1 and b2 perpendicular to the normal vector in space can be calculated as:

[0153] b1=b a ×n,b2=b1×n

[0154] Where × represents the outer product of the vector, b a =(1,0,0) T is a predefined anchor vector. Next, the world coordinate system coordinates p of each grid point of the k×k local plane block are calculated uv , in this method, k is set to 3. The specific process is:

[0155] p uv =c+τ(ub1+vb2)

[0156] Where c is the coordinate of the element center, τ is the scale factor that controls the size of the local plane block in the world coordinate system. After obtaining the coordinates of the local plane block, it can be projected onto the image of each perspective to obtain the features corresponding to the image block. Due to the large difference in perspectives, there may be serious occlusions. Here, we simply use the two perspectives with the largest corresponding confidence to calculate the consistency score s. The calculation process is expressed as:

[0157]

[0158] Step 3.4: Adaptive geometry fusion. After obtaining the cross-view geometry and multi-view according to steps 3.2 and 3.3, Figure 1 After the transformation, they are concatenated along the channel direction and input into a 3DU-Netψ u To fuse the geometric information of different cross-views, we finally get a unified geometric body U. This process can be expressed as:

[0159] U=ψ u ([G,S])

[0160] where [·] denotes the concatenation operation along the channel dimension.

[0161] In order to obtain a more accurate geometry with fine edges, the unified geometry U is projected onto each view to obtain a geometrically consistent geometry. Then the geometrically consistent and the original geometry V i Combined to get the final optimized geometry

[0162]

[0163] where ψ f It is a set of 3D convolutional layers. Through the optimized geometry The optimized intermediate depth map can be obtained as a priori for the subsequent construction of the feature matching cost volume.

[0164] Step 4: Input the optimized cost volume into the cross-view geometry Transformer, fully integrate the cost volume geometry under different viewpoints, and obtain the image and depth map of the new viewpoint through volume rendering technology. The specific implementation process is: first obtain the coordinates of the sampling point under the new viewpoint in the world coordinate system: p = o + td, where o represents the coordinates of the camera center of the new viewpoint in the world coordinate system, d represents the direction of the ray corresponding to the pixel point in the world coordinate system, and t represents the depth value of the sampling point. Project the jth sampling point into the multi-scale geometric body under different viewpoints to obtain the corresponding geometric features. Then according to its corresponding confidence They are fused through a multi-layer perceptron network (MLP), the specific process is as follows:

[0165]

[0166] Next, the fused geometric features corresponding to the sampling points on the ray are input into a self-attention network layer (SelfAttn) to fully evaluate the reliability of each sampling point. The specific process is as follows:

[0167]

[0168] where γ j represents the position code of the jth sampling point, M represents the total number of sampling points on the ray, which is set to 64 in this method. Then the reliability score r is converted into ij Converted into weights, we can get the fused geometric features:

[0169]

[0170] These fused geometric features are input into a self-attention network layer to capture the geometric features along the ray, and then encoded into SRDF values ​​through a multi-layer perceptron network (MLP):

[0171]

[0172] Finally, the SRDF values ​​of the sampling points can be used to obtain the rendered image and depth map through the volume rendering method in NeuS.

[0173] Step 5: Use the difference between the rendered depth and color and the true depth and color as the loss function to supervise the training of the network. The specific implementation process is: The total loss function is defined as:

[0174]

[0175] Where α and β are balance factors, which are both set to 1 in this method. Color loss function Defined as:

[0176]

[0177] Where S represents the number of effective pixels, Represents the rendered color map, C s Represents the true value of the color image. Rendering depth loss Defined as:

[0178]

[0179] in, and D s Represent the rendered depth map and the depth truth respectively. Multi-view depth loss Defined as:

[0180]

[0181] where λ l Indicates the balance weight of each stage. In this method, the balance weights of the three stages are set to 0.5, 0.3, and 0.2 respectively. and The predicted depth map and true depth of each layer are represented. During the training process, the above loss function value is calculated, and the network parameters in the model are updated using the gradient descent method until the model converges to obtain a trained sparse perspective reconstruction model.

[0182] The specific steps of the testing phase process are:

[0183] Step 1: Input a small number of images with large perspective transformations into an existing pose estimation framework, such as Colmap, to obtain the pose of each image relative to the world coordinate system.

[0184] Step 2: Input the multi-view images and their corresponding poses into the trained cross-view information complementary sparse view surface reconstruction network to obtain the depth map of each view;

[0185] Step 3: Input each view depth map and the corresponding pose into the existing TSDF depth map fusion framework to obtain the final surface reconstruction mesh.

[0186] The above embodiments should be regarded as illustrative and non-restrictive, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the invention and within the spirit and principles of the present invention should be covered by the protection scope of the present invention.

Claims

1. A sparse view surface reconstruction method based on cross-view information complementarity, characterized by: It includes two processes: training phase and inference phase; The training phase includes the following steps: Step 1: Input the input sparse view image into the feature pyramid network to obtain multi-scale features; Step 2: Use the features of multiple views and their corresponding poses to construct the cost volume, and regularize it using a 3D network; Step 3: Input the multi-view cost volume into the cross-view geometry complementation module to obtain the optimized cost volume; Step 4: Input the optimized cost volume into the cross-view geometry transformer to fully integrate the cost volume geometry under different viewpoints, and obtain the image and depth map of the new viewpoint through volume rendering technology; Step 5: Supervise the training of the network by comparing the difference between the rendered depth and color and the true depth and color as the loss function; The inference phase includes the following steps: Step 1: Input a small number of images with large perspective transformations into the existing pose estimation framework to obtain the pose of each image relative to the world coordinate system; Step 2: Input the multi-view images and their corresponding poses into the trained cross-view information complementary sparse view surface reconstruction network to obtain the depth of each view; Step 3: Input each view depth map and the corresponding pose into the existing TSDF fusion framework to obtain the final surface reconstruction mesh.

2. The sparse view surface reconstruction method based on cross-view information complementarity according to claim 1, characterized in that: In step 1 of the training phase, the input N sparse view images are Input into the feature pyramid network to obtain multi-scale features Where l represents the subscript of the scale, and there are L scales in total.

3. The sparse view surface reconstruction method based on cross-view information complementarity according to claim 1, characterized in that: In step 2 of the training phase, all source view images are aligned to the reference view through differentiable deformation, and the cost volume is constructed using feature dot products. The specific implementation process is: Under the depth hypothesis d, the pixel p in the reference view is compared to the corresponding pixel in the source view The deformation between is defined as: Among them, R and t represent the rotation and translation transformation between the two views, K0 and K represent the intrinsic parameter matrices of the reference image and the source image; then, the feature correlation of the pixels between the reference image and the source image is calculated as follows: in, is the feature matching cost volume between the reference image and the i-th source image, D, H, W are the number of depth hypotheses and the height and width of the image, and <·> is the dot product between vectors; the feature matching cost volume is input into a 3D U-Net to regularize it and obtain a smooth geometric volume feature and probability volume features The estimated intermediate depth is obtained by weighted summing of the probability volume feature and the depth hypothesis 4. The sparse view surface reconstruction method based on cross-view information complementarity according to claim 1, characterized in that: The specific implementation process of the cross-view geometry complementation module in step 3 of the training phase is: In the cross-view geometric complementation module, a confidence map is first estimated for each view. Based on this confidence map, the geometric information of different views can be adaptively fused. The cross-view geometric complementation module constructs a cross-view geometric volume and a multi-view consistency volume based on image blocks. Finally, the cross-view geometric information captured by these two volumes is supplemented to the feature volume of each view to obtain the optimized cost volume. The specific process is as follows: Step 3.1: Estimate the confidence map for each view; use entropy to measure the depth probability distribution along the depth direction as the prior for the depth confidence; the entropy prior is finally input into a 2D network ψ c The final confidence map is obtained The process is described as follows: Step 3.2: Construct cross-view geometry; the cross-view geometry uses the previously calculated confidence map as a clue to adaptively fuse geometric volume features under different viewpoints; The cross-view geometry G is defined as a sparse volume with a resolution of R×R×R in the world coordinate system, that is, it only stores voxels with geometric volume features, but not empty voxels; for the element at position p in the cross-view geometry volume, it will be projected to each view to obtain the confidence and geometric volume features of the corresponding position; The confidence is converted into weight as: where π i (·) represents the projection function of the i-th view; finally, the geometric volume element across viewing angles at position p is calculated as: Step 3.3: Construct a multi-view consistent volume; first use the intermediate depth map D obtained in step 2 i Estimate the normal vector map for each view Where A is the matrix composed of all pixel coordinates in the local window, 1 is a vector with all elements equal to 1; then, the aggregation weight w calculated in step 3.2 is i The normal vectors from different perspectives are combined to obtain the aggregated normal vector: According to the normal vector, the coordinate representation of a local plane block in space is obtained; the two basis vectors b1 and b2 perpendicular to the normal vector in space are calculated as: <h2 style=";text-align:left;direction:ltr">b1=b<h2 style=";text-align:left;direction:ltr"> a <h2 style=";text-align:left;direction:ltr"> ×n,b2=b1×n Where × represents the outer product of the vector, b a =(1,0,0) T is a predefined anchor vector; next, the world coordinate system coordinates p of each grid point of the k×k local plane block vv is calculated as: p uv =c+τ(ub1+vb2) Where c is the coordinate of the element center, τ is the scale factor that controls the size of the local plane patch in the world coordinate system; After obtaining the coordinates of the local plane block, project it onto the image of each perspective to obtain the features corresponding to the image block; The consistency score s is calculated using the two perspectives with the largest corresponding confidence. The calculation process is expressed as: Step 3.4: Adaptive geometry fusion; After obtaining the cross-view geometry and multi-view consistent volume according to steps 3.2 and 3.3, they are concatenated along the channel direction and input into a 3D U-Netψ u To fuse the geometric information of different cross-views, we finally get a unified geometric body U. This process is expressed as: U=ψ u ([G,S]) The unified geometric body U is projected into each view to obtain a geometrically consistent body Then the geometrically consistent and the original geometry V i Combined to get the final optimized geometry where ψ f is a set of 3D convolutional layers; through the optimized geometry The optimized intermediate depth map is used as the prior for the subsequent construction of the feature matching cost volume.

5. The sparse view surface reconstruction method based on cross-view information complementarity according to claim 1, characterized in that: The specific implementation process of the cross-view geometry Transformer in step 4 of the training phase is: First, the coordinates of the sampling point under the new perspective in the world coordinate system are obtained: p = o + td, where o represents the coordinates of the camera center of the new perspective in the world coordinate system, d represents the direction of the ray corresponding to the pixel point in the world coordinate system, and t represents the depth value of the sampling point; the jth sampling point is projected onto the multi-scale geometric body under different perspectives to obtain the corresponding geometric features. Then according to its corresponding confidence They are fused through a multi-layer perceptron network MLP. The specific process is as follows: Next, the fused geometric features corresponding to the sampling points on the ray are input into a self-attention network layer SelfAttn to fully evaluate the reliability of each sampling point. The specific process is as follows: where γ j represents the position code of the jth sampling point, M represents the total number of sampling points on the ray; then the reliability score r is converted into ij Convert to weights to get the fused geometric features These fused geometric features are input into a self-attention network layer to capture the geometric features along the ray, and then encoded into SRDF values ​​through a multi-layer perceptron network MLP. Finally, the SRDF values ​​of the sampling points are obtained through the volume rendering method in NeuS to obtain the rendered image and depth map.

6. The sparse view surface reconstruction method based on cross-view information complementarity according to claim 1, characterized in that: The specific process of loss function and training in step 5 of the training phase is: The overall loss function is defined as: Where α and β are balance factors respectively; color loss function Defined as: Where S represents the number of effective pixels, Represents the rendered color map, C s Represents the true value of the color image; renders the depth loss Defined as: in, and D s Represents the rendered depth map and the depth truth respectively; multi-view depth loss Defined as: where λ l represents the balance weight at each stage, and Represents the predicted depth map and true depth of each layer; calculates the above loss function value during the training process, uses the gradient descent method to update the network parameters in the model until the model converges, and obtains the trained sparse perspective reconstruction model.

Images