Sparse view surface reconstruction method based on cross-view information complement

Abstract

The application discloses a sparse view surface reconstruction method based on cross-view information complementarity, which comprises a training stage and an inference stage. The training stage comprises the following steps: firstly, inputting an input sparse view image into a network to obtain multi-scale features and constructing a cost volume; next, inputting the multi-view cost volume into a cross-view geometric complementarity module to obtain an optimized cost volume; then, inputting the optimized cost volume into a cross-view geometric Transformer to obtain a new view image and a depth map through a volume rendering technology, and using a true value depth supervision network for training; in the inference stage, inputting multi-view images and corresponding poses into the trained cross-view information complementary sparse view surface reconstruction network to obtain the depth of each view, and using a depth map fusion algorithm to obtain a final surface. The application improves the scene reconstruction accuracy and robustness under large view changes by explicitly constructing a cross-view complementary information fusion mode.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of three-dimensional reconstruction of computer vision technology, in particular to a sparse view surface reconstruction method based on cross-view information complementation. BACKGROUND

[0002] Three-dimensional surface reconstruction technology aims to reconstruct the three-dimensional surface of the entire scene from a given set of calibrated images describing a scene. This task is of great significance to the practical application and development of mobile robots and virtual reality. It can assist mobile robots or virtual reality devices to capture images and reconstruct the three-dimensional surface of the entire scene, thereby achieving three-dimensional scene editing or accurate positioning of the device.

[0003] Three-dimensional reconstruction is a long-term task in the field of computer vision. The mainstream method currently used is to first infer the depth of each view using a deep neural network, and then fuse the depth maps to obtain the surface representation of the scene. Although these methods perform well on small baseline datasets, in practical applications, users may have difficulty capturing such small baseline image sets. However, these methods perform poorly on more challenging large baseline image sets. To improve the practicality of these three-dimensional reconstruction methods, UFORecon, the method closest to the present application, introduces a feature matching Transformer to capture the correlation between pixel features of different views, thereby improving the robustness of matching under large view changes. However, due to the large view change, there is still a lot of noise in the feature matching cost volume constructed by this method, and the existing regularizer cannot infer an accurate geometric representation from it, resulting in incorrect reconstruction results. SUMMARY

[0004] The present application aims to provide a sparse view surface reconstruction method based on cross-view information complementation to solve the problems existing in the prior art.

[0005] The technical solution to achieve the present application is a sparse view surface reconstruction method based on cross-view information complementation, including a training phase and an inference phase.

[0006] The training phase includes the following steps:

[0007] Step 1: input the input sparse view image into the feature pyramid network to obtain multi-scale features;

[0008] Step 2: construct a cost volume using multi-view features and their corresponding poses, and regularize it using a 3D network;

[0009] Step 3: input the multi-view cost volume into the cross-view geometry complementary module to obtain the optimized cost volume;

[0010] Step 4: input the optimized cost volume into the cross-view geometry Transformer to fully fuse the cost volume geometry under different views, and obtain the image and depth map of the new view through volume rendering technology;

[0011] Step 5: compare the difference between the rendered depth and color and the true value depth and color as a loss function to supervise the training of the network.

[0012] The inference stage includes the following steps:

[0013] Step 1: input the input set of a small number of images with large view angle transformation into the existing pose estimation framework to obtain the pose of each image relative to the world coordinate system.

[0014] Step 2: input the multi-view image and the corresponding pose into the trained cross-view information complementary sparse view surface reconstruction network to obtain the depth of each view.

[0015] Step 3: input each view depth map and the corresponding pose into the existing TSDF fusion framework to obtain the final surface reconstruction mesh.

[0016] Further, in step 1 of the training stage, the input N sparse view images are input into the feature pyramid network to obtain multi-scale features where l represents the subscript of the scale, and a total of L scales are included.

[0017] Further, in step 2 of the training stage, all source view images are aligned to the reference view through differentiable deformation, and a cost volume is constructed using feature point multiplication, and the specific implementation process is as follows:

[0018] Under the depth assumption d, the deformation between the pixel p in the reference view and the corresponding pixel in the source view is defined as:

[0019]

[0020] where R and t represent the rotation and translation transformation between the two views, and K0 and K represent the intrinsic matrices of the reference image and the source image; then, the feature correlation calculation between the pixels of the reference image and the source image is as follows:

[0021]

[0022] where, For the feature matching cost volume between the reference image and the i-th source image, D, H, W are the number of depth hypotheses and the height and width of the image respectively, and <·> is the dot product between vectors; the feature matching cost volume is input into a 3D U-Net to regularize it and obtain a smooth geometric volume feature and a probability volume feature The probability volume feature is weighted and summed with the depth hypotheses to obtain the estimated intermediate depth

[0023] Further, the specific implementation process of the cross-view geometric complementary module in step 3 of the training phase is as follows:

[0024] In the cross-view geometric complementary module, first, a confidence map is estimated for each view; according to this confidence map, the geometric information of different views can be adaptively fused; the cross-view geometric volume and the multi-view consistency volume based on image blocks are constructed in the cross-view geometric complementary module Figure One Finally, the cross-view geometric information captured by the two volumes is supplemented to the feature volume of each view to obtain the optimized cost volume, and the specific process is as follows:

[0025] Step 3.1: Estimate the confidence map of each view; use the entropy of the depth probability distribution along the depth direction as the prior of the depth confidence; the entropy prior is finally input into a 2D network ψ c to obtain the final confidence map The process is described as follows:

[0026]

[0027] Step 3.2: Construct the cross-view geometric body; the cross-view geometric body adaptively fuses the geometric volume features of different views based on the previously calculated confidence map; the cross-view geometric body G is defined as a sparse volume in the world coordinate system with a resolution of R×R×R, that is, it only saves voxels with geometric volume features, but does not save empty voxels; for the element located at position p in the cross-view geometric volume, it is projected into each view to obtain the confidence and geometric volume feature at the corresponding position; the confidence is converted into a weight and represented as:

[0028]

[0029] where π i (·) represents the projection function of the i-th view; finally, the cross-view geometric volume element at position p is calculated as:

[0030]

[0031] Step 3.3: Construct a multi-view Figure OneBody; first, the intermediate depth map D obtained in step 2 is used i Estimating the normal map of each view

[0032]

[0033] where A is a matrix composed of all pixel coordinates in the local window, and 1 is a vector with all elements being 1; then, the aggregated weight w calculated in step 3.2 is used to calculate the aggregated normal map N as follows: i Fusing the normals of different views to obtain the aggregated normal map:

[0034] According to the normal map, the coordinate representation of a local planar patch in space is obtained; two basis vectors b1, b2 in space perpendicular to the normal vector are calculated as follows:

[0035] b1 = b a × n, b2 = b1 × n

[0036] where × represents the outer product of vectors, and b a = (1, 0, 0) T is a predefined anchor vector; next, the coordinates p uv of the world coordinate system of each grid point of the k × k local planar patch are calculated as follows:

[0037] p uv = c + τ (ub1 + vb2)

[0038] where c is the center coordinate of the element, τ is a scale factor that controls the size of the local planar patch in the world coordinate system; after obtaining the coordinates of the local planar patch, the image block corresponding to the feature is obtained by projecting it onto each view image;

[0039] The two views with the highest confidence are used to calculate the consistency score s, and the calculation process is described as follows:

[0040]

[0041] Step 3.4: Adaptive geometric body fusion; after obtaining the cross-view geometry and multi-view Figure One body according to steps 3.2 and 3.3, respectively, they are connected together along the channel direction and input into a 3D U-Net ψ u to fuse different cross-view geometric information, and finally obtain a unified geometry U; this process is represented as follows:

[0042] U = ψ i ([G, S])

[0043] The unified geometry U is projected onto each view to obtain a geometric consistent body Then the geometric consistent volume and the original geometric volume V i Fusion to get the final optimized geometric volume

[0044]

[0045] where ψ f is a set of 3D convolutional layers; through the optimized geometric volume get the optimized intermediate depth map as the prior of subsequent construction of feature matching cost volume.

[0046] Further, the specific implementation process of the cross-view geometry Transformer in step 4 of the training stage is as follows:

[0047] First, get the coordinates of the sampling points in the world coordinate system under the new view: p = o + td, where o represents the coordinates of the camera center of the new view in the world coordinate system, d represents the direction of the pixel corresponding to the ray in the world coordinate system, and t represents the depth value of the sampling point; Project the jth sampling point into the multi-scale geometric volume under different views to obtain the corresponding geometric features

[0048] Then, according to the corresponding confidence Fuse them through a multi-layer perception network MLP, the specific process is as follows:

[0049]

[0050] Next, input the fused geometric features corresponding to the sampling points on the ray into a self-attention network layer SelfAttn to fully evaluate the reliability of each sampling point, the specific process is as follows:

[0051]

[0052] where γ j represents the position encoding of the jth sampling point, and M represents the total number of sampling points on the ray; Then use the softmax function to convert the reliability score r ij into a weight to get the fused geometric features

[0053]

[0054] These fused geometric features are input into a self-attention network layer to capture the geometric features along the ray, and a multi-layer perception network MLP is used to encode the geometric features into SRDF values

[0055]

[0056] Finally, the SRDF values of the obtained sampling points are used to obtain a rendered image and a depth map by volume rendering in NeuS.

[0057] Further, the specific process of the loss function and the training in step 5 of the training stage is as follows:

[0058] The total loss function is defined as:

[0059]

[0060] Wherein, alpha and beta are balance factors; the color loss function is defined as:

[0061]

[0062] Wherein, S represents the number of effective pixels, represents a rendered color image, and C s represents a true value of a color image; the rendered depth loss is defined as:

[0063]

[0064] Wherein, and D s respectively represent a rendered depth image and a depth true value; the multi-view depth loss is defined as:

[0065]

[0066]

[0067] Wherein, lambda l represents a balance weight of each stage, and represent a predicted depth image and a true value depth of each level; in the training process, the loss function values are calculated, the network parameters in the model are updated by using a gradient descent method, until the model converges, and a trained sparse view reconstruction model is obtained.

[0068] Compared with the prior art, the present application has the following advantages:

[0069] (1) The present application constructs a cross-view geometry complementary module, and reliably geometric information of different views is adaptively fused by constructing geometric bodies in a unified space, and inaccurate geometry in respective views is optimized, so that the geometric consistency and accuracy in each view are improved. Compared with the previous method, the present application optimizes and supplements the geometric accuracy of respective views by reliable geometry in different views, so that the accuracy of the final scene reconstruction is improved.

[0070] (2) The present application constructs a cross-view geometry Transformer, which can robustly fuse reliable geometry of different views in the process of rendering depth maps. Compared with previous methods, the present application fully evaluates the reliability of sampling points in each view geometry by considering the geometry information along the ray, reduces the influence of occlusion or inaccurate geometry view on cross-view geometry fusion, and thus improves the accuracy of reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is the overall framework diagram of the cross-view information complementary reconstruction network in the method of the present application.

[0072] Figure 2 is the framework diagram of the cross-view geometry complementary module in the method of the present application.

[0073] Figure 3 is the framework diagram of the cross-view geometry Transformer in the method of the present application.

[0074] Figure 4 is the flowchart of the training stage in the method of the present application.

[0075] Figure 5 is the flowchart of the inference stage in the method of the present application. DETAILED DESCRIPTION

[0076] In order to enhance the accuracy and completeness of the existing method in the reconstruction of a large baseline image set, the method of the present application uses cross-view geometry information to complement each other's geometry representation in different views. Specifically, the method constructs a cross-view geometry body for fusing geometry information of different views and a multi-view consistency body for providing accurate multi-view consistency information. The two bodies together provide accurate geometry information. In addition, in the rendering stage, the method uses an explicit Transformer to comprehensively evaluate the reliability of the ray in each view, thereby reducing the influence of occlusion or inaccurate geometry view on cross-view geometry fusion. Figure One Figure One

[0077] The present application will be further described below in conjunction with the accompanying drawings of the specification.

[0078] As shown in the drawings, Figure 4 the present application provides a technical solution: a sparse view surface reconstruction method based on cross-view information complementarity, including a training stage and an inference stage.

[0079] The specific steps of the training stage process are:

[0080] Step 1: input the input N sparse view images into the feature pyramid network to obtain multi-scale features ​​​where l denotes the scale index, and there are L scales in total.

[0081] Step 2: Utilize the multi-scale features described above Construct the feature matching cost volume. Here we describe the process of constructing the feature matching cost volume with the features of one scale, for simplicity, the scale index l is omitted in the following description. A differentiable warping is used to align all source views to the reference view. Under the depth hypothesis d, the warping between the pixel p in the reference view and the corresponding pixel is defined as:

[0082]

[0083] where R and t denote the rotation and translation transformation between two views, and K0 and K denote the intrinsic matrix of the reference image and the source image. Then the feature correlation of the pixels between the reference image and the source image can be calculated as follows:

[0084]

[0085] where is the feature matching cost volume between the reference image and the i-th source image, D, H, W are the number of depth hypotheses and the height and width of the image respectively, and <·> is the dot product between vectors. The feature matching cost volume will be input into a 3D U-Net to regularize it, obtaining the smooth geometric volume feature and the probability volume feature The estimated intermediate depth can be obtained by weighting the probability volume feature with the depth hypotheses

[0086] Step 3: Combine Figure 2 The multi-view cost volumes are input into the cross-view geometry complementary module to obtain the optimized cost volume. In the cross-view geometry complementary module, first, the confidence map for each view is estimated. According to this confidence map, the geometric information of different views can be adaptively fused. The cross-view geometry complementary module constructs the cross-view geometric volume and the multi-view consistency volume based on image blocks, and finally the cross-view geometric information captured by these two volumes is supplemented to the feature volume of each view to obtain the optimized cost volume. Figure One

[0087] Step 3.1: Estimate the confidence map for each view. Here the depth probability distribution along the depth direction is measured by entropy as the prior of the depth confidence. The entropy prior will be finally input into a simple 2D network ψ c to obtain the final confidence map The process is described as follows:

[0088] ​

[0089] Step 3.2: Constructing the cross-view geometry. The cross-view geometry adaptively fuses geometric volume features from different viewpoints, using the previously calculated confidence map as a guide. The cross-view geometry G is defined as a sparse volume with a resolution of R×R×R in the world coordinate system; that is, it only stores voxels with geometric volume features and does not store empty voxels. For an element at position p in the cross-view geometry, it is projected onto each viewpoint to obtain the confidence score and geometric volume features at the corresponding position. The confidence score converted to weights can be expressed as:

[0090]

[0091] Where π i (·) represents the projection function of the i-th view. Finally, the cross-view geometric volume element at position p can be calculated as:

[0092]

[0093] Step 3.3: Constructing Multiple Views Figure One To better handle large-scale viewpoint changes, this invention employs a block-based multi-view approach. Figure One Consistency score. Multiple views Figure One The volume S and the cross-view geometry G have the same resolution and are both sparse volumes. First, the intermediate depth map D obtained in step 2 is used. i Estimate the normal vector map for each viewpoint:

[0094]

[0095] Where A is a matrix consisting of the coordinates of all pixels within the local window, and I is a vector with all elements equal to 1. Then, the aggregation weight w calculated in step 3.2 is used... i Normal vectors from different perspectives can be fused to obtain aggregated normal vectors:

[0096]

[0097] Based on this normal vector, the coordinate representation of a local planar block in space can be obtained. The two basis vectors b1 and b2 perpendicular to the normal vector in space can be calculated as follows:

[0098] b1 = b a ×n,b2=b1×n

[0099] Where × represents the outer product of vectors, b a =(1,0,0) T This is a predefined anchor vector. Next, the world coordinates p of each grid point in the k×k local planar block. uv It can be calculated as:

[0100] p uv =c + τ(ub1 + vb2)

[0101] Where c is the coordinate of the center of the element. τ is a scale factor controlling the size of the local planar patch in the world coordinate system. After obtaining the coordinates of the local planar patch, it can be projected onto the image from each viewpoint to obtain the features corresponding to the image patch. Due to the large differences in viewpoints, there may be severe occlusion. Here, we simply use the two viewpoints with the highest confidence to calculate the consistency score s. The calculation process is described as follows:

[0102]

[0103] Step 3.4: Adaptive geometry fusion. Following steps 3.2 and 3.3, cross-view geometry and multi-view geometry are obtained respectively. Figure One After being assembled, they are connected together along the channel direction and input into a 3DU-Netψ u In this process, geometric information from different cross-perspectives is integrated to ultimately obtain a unified geometric body U. This process can be represented as:

[0104] U = ψ i ([G,S])

[0105] To obtain a more accurate geometry with fine edges, the unified geometry U is projected onto various views to obtain a geometrically consistent body. Then geometrically consistent body And the original geometry V i The final optimized geometry is obtained by combining them.

[0106]

[0107] Where ψ f It is a set of 3D convolutional layers. Through optimized geometry... The optimized intermediate depth map can be used as a priori for constructing the feature matching cost body.

[0108] Step 4: Combining Figure 3 The optimized cost volume is input into a cross-view geometry Transformer to fully integrate the cost volume geometry from different viewpoints, and volume rendering techniques are used to obtain the image and depth map of the new viewpoint. The specific implementation process is as follows: First, the coordinates of the sampling points in the world coordinate system under the new viewpoint are obtained:

[0109] p = o + td, where o denotes the camera center of the new view in the world coordinate system, d denotes the direction of the pixel corresponding to the ray in the world coordinate system, and t denotes the depth value of the sampling point. The jth sampling point is projected into the multi-scale geometry under different views to obtain the corresponding geometric features Then according to its corresponding confidence Fuse them through a multi-layer perception network (MLP), the specific process is as follows:

[0110]

[0111] Next, the fused geometric features corresponding to the sampling points on the ray are input into a self-attention network layer (SelfAttn) to fully evaluate the reliability of each sampling point, and the specific process is as follows:

[0112]

[0113] Where γ j represents the position encoding of the jth sampling point, and M represents the total number of sampling points on the ray. Then use the softmax function to convert the reliability score r ij into a weight, and the fused geometric features can be obtained:

[0114]

[0115]

[0116] These fused geometric features are input into a self-attention network layer to capture the geometric features along the ray, and a multi-layer perception network (MLP) is used to encode the geometric features into SRDF values:

[0117]

[0118] Finally, the SRDF values of the sampling points obtained can be rendered into a rendered image and a depth map through the volume rendering method in NeuS.

[0119] Step 5: Supervise the training of the network by comparing the difference between the rendered depth and color and the true value depth and color as the loss function. The specific implementation process is as follows: the total loss function is defined as:

[0120]

[0121] Where α, β are balance factors. The color loss function is defined as:

[0122]

[0123] Where S represents the number of valid pixels, denotes the rendered color map, C s denotes the ground truth of the color image. The rendered depth loss is defined as:

[0124]

[0125] where, and D s denote the rendered depth map and the ground truth depth, respectively. The multi-view depth loss is defined as:

[0126]

[0127] where λ l denotes the balancing weight of each stage, and denote the predicted depth map and the ground truth depth of each level. The above loss function value is calculated during the training process, and the network parameters in the model are updated by using the gradient descent method until the model converges, thereby obtaining the trained sparse view surface reconstruction model.

[0128] The specific steps of the test stage process are:

[0129] Step 1: input the input set of a small number of images with large view angle transformation into an existing pose estimation framework, such as Colmap, to obtain the pose of each image relative to the world coordinate system.

[0130] Step 2: input the multi-view images and the corresponding poses into the trained sparse view surface reconstruction network complementary to the cross-view information to obtain the depth of each view.

[0131] Step 3: input each view depth map and the corresponding pose into an existing TSDF depth map fusion framework to obtain the final surface reconstruction mesh.

[0132] The present application will be further described below in conjunction with the drawings and embodiments of the present application. The described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0133] As shown in Figure 4 and Figure 5 , the present application provides a technical solution: a sparse view surface reconstruction method based on cross-view information complementarity, including a training stage and an inference stage. The overall network architecture is as shown in Figure 1 .

[0134] The specific steps of the training stage process are:

[0135] Step 1: input N sparse view images into the feature pyramid network to get multi-scale features where l denotes the index of scale, and there are L scales in total, including 3 scales in this method.

[0136] Step 2: construct the feature matching cost volume using the above multi-scale features Here, the process of constructing the feature matching cost volume is described using the feature description of one of the scales, and for simplicity, the scale index l is omitted in the following description. A differentiable deformation is used to align all source views to the reference view. Under the depth hypothesis d, the deformation between the pixel p in the reference view and the corresponding pixel in the source view is defined as:

[0137]

[0138] where R and t represent the rotation and translation transformation between the two views, and K0 and K represent the intrinsic matrix of the reference image and the source image. Then, the feature correlation between the pixels of the reference image and the source image can be calculated as follows:

[0139]

[0140] where is the feature matching cost volume between the reference image and the i-th source image, D, N, and W are the number of depth hypotheses and the height and width of the image, respectively, and <·> is the dot product between vectors. In this method, the depth hypotheses of each stage are set to 48, 32, and 8, respectively, and the height and width of the image are set to 512 and 640, respectively. The feature matching cost volume is input into a 3D U-Net to regularize it, obtaining a smooth geometric volume feature and a probability volume feature The estimated intermediate depth

[0141] Step 3: input the multi-view cost volume into the cross-view geometric complementary module to obtain the optimized cost volume. In the cross-view geometric complementary module, first, the confidence map is estimated for each view. According to this confidence map, the geometric information of different views can be adaptively fused. The cross-view geometric complementary module constructs a cross-view geometric volume and a multi-view consistency volume based on image blocks, and finally, the cross-view geometric information captured by these two volumes is supplemented into the feature volume of each view to obtain the optimized cost volume. Figure One

[0142] ​​Step 3.1: Estimate the confidence map per viewpoint. Here, entropy is used to measure the depth probability distribution along the depth direction as the prior for depth confidence. The entropy prior is finally input into a simple 2D network ψ. c In this process, the final confidence map is obtained. The process is described as follows:

[0143]

[0144] Step 3.2: Constructing the cross-view geometry. The cross-view geometry adaptively fuses geometric volume features from different viewpoints, using the previously calculated confidence map as a guide. The cross-view geometry G is defined as a sparse volume with a resolution of R×R×R in the world coordinate system; that is, it only stores voxels with geometric volume features and does not store empty voxels. In this method, the resolution R is set to 96. For an element at position p in the cross-view geometry, it is projected onto each viewpoint to obtain the confidence score and geometric volume features at the corresponding position. The confidence score converted to weights can be expressed as:

[0145]

[0146] Where π i (·) represents the projection function of the i-th view. Finally, the cross-view geometric volume element at position p can be calculated as:

[0147]

[0148] Step 3.3: Constructing Multiple Views Figure One To better handle large-scale viewpoint changes, this invention employs a block-based multi-view approach. Figure One Consistency score. Multiple views Figure One The volume S and the cross-view geometry G have the same resolution and are both sparse volumes. First, the intermediate depth map D obtained in step 2 is used. i Estimate the normal vector map for each viewpoint:

[0149]

[0150] Where A is a matrix consisting of the coordinates of all pixels within the local window, and I is a vector with all elements equal to 1. In this method, the local window size is 7×7. Then, the aggregation weight w calculated in step 3.2 is used... i Normal vectors from different perspectives can be fused to obtain aggregated normal vectors:

[0151]

[0152] Based on this normal vector, the coordinate representation of a local planar block in space can be obtained. The two basis vectors b1 and b2 perpendicular to the normal vector in space can be calculated as follows:

[0153] b1 = b a ×n,b2=b1×n

[0154] Where × represents the outer product of vectors, b a =(1,0,0) T These are predefined anchor vectors. Next, the world coordinates p of each grid point in the k×k local planar block are calculated. uv In this method, k is set to 3. The specific process is as follows:

[0155] p uv =c + τ(ub1 + vb2)

[0156] Where c is the coordinate of the center of the element. τ is a scale factor controlling the size of the local planar patch in the world coordinate system. After obtaining the coordinates of the local planar patch, it can be projected onto the image from each viewpoint to obtain the features corresponding to the image patch. Due to the large differences in viewpoints, there may be severe occlusion. Here, we simply use the two viewpoints with the highest confidence to calculate the consistency score s. The calculation process is described as follows:

[0157]

[0158] Step 3.4: Adaptive geometry fusion. Following steps 3.2 and 3.3, cross-view geometry and multi-view geometry are obtained respectively. Figure One After being assembled, they are connected together along the channel direction and input into a 3DU-Netψ u In this process, geometric information from different cross-perspectives is integrated to ultimately obtain a unified geometric body U. This process can be represented as:

[0159] U = ψ u ([G,S])

[0160] Where [·] represents a connection operation along the channel dimension.

[0161] To obtain a more accurate geometry with fine edges, the unified geometry U is projected onto various views to obtain a geometrically consistent body. Then geometrically consistent body And the original geometry V i The final optimized geometry is obtained by combining them.

[0162]

[0163] Where ψ f It is a set of 3D convolutional layers. Through optimized geometry... The optimized intermediate depth map can be used as a priori for constructing the feature matching cost body.

[0164] Step 4: The optimized cost volume is input into a cross-view geometry Transformer, which fully fuses the cost volume geometry under different views and obtains the image and depth map of the new view through volume rendering technology. The specific implementation process is as follows: first, the coordinates of the sampling points in the world coordinate system under the new view are obtained: p = o + td, where o represents the coordinates of the camera center of the new view in the world coordinate system, d represents the direction of the pixel point corresponding to the ray in the world coordinate system, and t represents the depth value of the sampling point. The jth sampling point is projected into the multi-scale geometry under different views to obtain the corresponding geometric features Then, according to the corresponding confidence The fusion is performed through a multi-layer perception network (MLP), and the specific process is as follows:

[0165]

[0166] Next, the fused geometric features corresponding to the sampling points on the ray are input into a self-attention network layer (SelfAttn) to fully evaluate the reliability of each sampling point, and the specific process is as follows:

[0167]

[0168] where γ j represents the position encoding of the jth sampling point, and M represents the total number of sampling points on the ray, which is set to 64 in this method. Then, the reliability score r ij is converted into a weight using the softmax function, and the fused geometric features can be obtained:

[0169]

[0170] These fused geometric features are input into a self-attention network layer to capture the geometric features along the ray, and a multi-layer perception network (MLP) is used to encode the geometric features into SRDF values:

[0171]

[0172] Finally, the obtained SRDF values of the sampling points can be rendered into an image and a depth map through the volume rendering method in NeuS.

[0173] Step 5: The difference between the rendered depth and color and the true depth and color is used as a loss function to supervise the training of the network. The specific implementation process is as follows: the total loss function is defined as:

[0174]

[0175] where a, b are balance factors, which are both set as 1 in this method. Color loss function is defined as:

[0176]

[0177] where S represents the number of valid pixels, represents the rendered color map, C s represents the ground truth of the color image. Rendered depth loss is defined as:

[0178]

[0179] where, and D s represent the rendered depth map and the ground truth depth, respectively. Multi-view depth loss is defined as:

[0180]

[0181] where λ l represents the balance weight of each stage, which are set as 0.5, 0.3, 0.2 for the three stages in this method, and represent the predicted depth map and the ground truth depth of each level. In the training process, the above loss function values are calculated, the network parameters in the model are updated by using the gradient descent method, until the model converges, and the trained sparse view surface reconstruction model is obtained.

[0182] The specific steps of the test stage process are:

[0183] Step 1: input the input set of a small number of images with large view angle transformation into an existing pose estimation framework, such as Colmap, to obtain the pose of each image relative to the world coordinate system.

[0184] Step 2: input the multi-view images and the corresponding poses into the trained sparse view surface reconstruction network complementary to the cross-view information to obtain the depth map of each view.

[0185] Step 3: input each view depth map and the corresponding pose into an existing TSDF depth map fusion framework to obtain the final surface reconstruction mesh.

[0186] The above embodiments should be regarded as exemplary and non-limiting, but the protection scope of the present application is not limited thereto, and any modification, equivalent replacement and improvement made by any person skilled in the art within the technical range disclosed by the present application, within the spirit and principles of the present application, should be covered within the protection scope of the present application.

Claims

1. A sparse view surface reconstruction method based on complementary cross-view information, characterized in that: The method comprises a training phase and an inference phase; The training phase comprises the following steps: Step 1: input the input sparse view image into a feature pyramid network to obtain multi-scale features; Step 2: construct a cost volume using multi-view features and their corresponding poses, and regularize the cost volume using a 3D network; Step 3: input the multi-view cost volume into a cross-view geometric complementary module to obtain an optimized cost volume; Step 4: input the optimized cost volume into a cross-view geometric Transformer to fully fuse the cost volume geometry under different views, and obtain a new view image and depth map through volume rendering technology; Step 5: compare the rendered depth and color with the true depth and color to obtain a loss function, and supervise the training of the network; The inference phase comprises the following steps: Step 1: input a small set of images with large view transformation into an existing pose estimation framework to obtain the pose of each image relative to the world coordinate system; Step 2: input the multi-view images and the corresponding poses into the trained cross-view information complementary sparse view surface reconstruction network to obtain the depth of each view; Step 3: input each view depth map and the corresponding pose into an existing TSDF fusion framework to obtain the final surface reconstruction mesh; The specific implementation process of the cross-view geometric complementary module in step 3 of the training phase is as follows: In the cross-view geometric complementary module, first, a confidence map is estimated for each view; according to the confidence map, the geometric information of different views can be adaptively fused; the cross-view geometric complementary module constructs a cross-view geometric volume and a multi-view consistency volume based on image blocks, and finally the cross-view geometric information captured by the two volumes is supplemented to the feature volume of each view to obtain an optimized cost volume, the specific process is as follows: Step 3.1: Estimate per-view confidence map; measure the depth probability distribution along the depth direction by entropy as the prior of depth confidence; the entropy prior is finally input to a 2D network ψ c to get the final confidence map The procedure is described as follows: Step 3.2: construct a cross-view geometric volume; the cross-view geometric volume adaptively fuses the geometric volume features of different views according to the previously calculated confidence map; The cross-view geometric volume G is defined as a sparse volume with a resolution of R x R x R in the world coordinate system, that is, it only saves voxels with geometric volume features, but does not save empty voxels; for the elements in the cross-view geometric volume located at position p, the confidence and geometric volume features of the corresponding position in each view are obtained by projection; The confidence is converted into a weight as follows: where π i (·) denotes the projection function of the i-th view; finally, the cross-view geometric volume element at position p is computed as: Step 3.3: Constructing the multi-view consistent volume; first use the intermediate depth map D obtained in step 2 i Estimating the normal vector map for each view where A is a matrix of all pixel coordinates in the local window, and 1 is a vector with all elements being 1; then, the aggregated normal vector is obtained by fusing the normal vectors of different views through the aggregated weights w calculated in step 3.2 i The normal vectors of different views are fused to obtain an aggregated normal vector: According to the normal vector, the coordinate representation of a local planar block in space is obtained; two basis vectors b1 and b2 perpendicular to the normal vector in space are calculated as follows: b1 = b a x n, b2 = b1 x n where x denotes the outer product of vectors, b a = (1, 0, 0) T is a predefined anchor vector; next, the coordinates p uv of the world coordinate system of each grid point of the k x k local planar patch are computed as: p uv = c + τ(ub1+vb2) where c is the element center coordinate, τ is a scale factor that controls the size of the local planar patch in the world coordinate system; After obtaining the coordinates of the local planar block, the image block corresponding features are obtained by projecting onto each view image; The consistency score s is calculated using the two views with the highest confidence, and the calculation process is as follows: Step 3.4: Adaptive geometry fusion; after obtaining the geometry across views and multi-view consistent volume respectively according to step 3.2 and 3.3, they are concatenated along the channel direction and input into a 3D U-Net ψ u to fuse different geometry information across views, and finally get the unified geometry U; this process is represented as: U = ψ u ([G, S]) The unified geometry U is projected to get geometry consistent bodies under each view The geometry consistent bodies are then fused to get the final optimized geometry and the original geometry V i to get the final optimized geometry where ψ f is a set of 3D convolutional layers; the optimized geometry is used to obtain an optimized intermediate depth map as a prior for the subsequent construction of the feature matching cost volume; The specific implementation process of the cross-view geometric Transformer in step 4 of the training phase is as follows: First, the coordinates of the sampling points in the world coordinate system under the new perspective are obtained: p = o + td, where o represents the coordinates of the camera center of the new perspective in the world coordinate system, d represents the direction of the ray corresponding to the pixel point in the world coordinate system, and t represents the depth value of the sampling point; the jth sampling point is projected into the multi-scale geometric body under different perspectives to obtain the corresponding geometric features Then according to its corresponding confidence It is fused by a multi-layer perception network MLP, and the specific process is Next, the fused geometric features of the sampling points on the ray are input into a self-attention network layer SelfAttn to fully evaluate the reliability of each sampling point, and the specific process is as follows: where γ j represents the position encoding of the j-th sampling point, M represents the total number of sampling points on the ray; then the reliability score r ij is converted into weight by using the softmax function, and the fused geometric features are obtained These fused geometry features are input into a self-attention network layer to capture geometry features along the ray and encode the geometry features into SRDF values through a multi-layer perceptron network (MLP) Finally, the SRDF values of the obtained sampling points are used to obtain the rendered image and depth map through volume rendering in NeuS.

2. The method of claim 1, wherein: In step 1 of the training phase, the input N sparse view images are input into the feature pyramid network to obtain multi-scale features where l represents the subscript of the scale, and a total of L scales are included.

3. The method of claim 1, wherein: In step 2 of the training phase, all source views are aligned to the reference view through differentiable deformation, and a cost volume is constructed using feature point multiplication, the specific implementation process is as follows: Under the depth assumption d, the pixel p in the reference view and the corresponding pixel in the source view The deformation between them is defined as: Where R and t represent the rotation and translation transformation between two views, K0 and K represent the intrinsic matrix of the reference image and the source image; Then, the feature correlation of the pixels between the reference image and the source image is calculated as follows: where, is the feature matching cost volume between the reference image and the i-th source image, D, H, W are the number of depth hypotheses and the height and width of the image respectively, and is the dot product between vectors; the feature matching cost volume is input into a 3D U-Net to regularize it and obtain smooth geometric volume features and probability volume features The probability volume features are weighted-summed with the depth hypotheses to obtain the estimated intermediate depth 4. The method of claim 1, wherein, In step 5 of the training phase, the specific process of the loss function and training is as follows: The total loss function is defined as follows: where a, b are the balance factors, respectively; color loss function is defined as: where S denotes the number of valid pixels, denotes the rendered color map, C s denotes the ground truth of the color image; rendered depth loss is defined as: where, and D s denote the rendered depth map and ground truth depth, respectively; multi-view depth loss is defined as: where λ l denotes the balance weight of each stage, and denotes the predicted depth map and the ground truth depth of each level; the loss function value is calculated during the training process, the network parameters in the model are updated by using the gradient descent method until the model converges, and a trained sparse view reconstruction model is obtained.

Images