Multi-view modeling method, device and modeling equipment based on light implicit field

Abstract

The present application relates to a multi-view modeling method, apparatus and modeling equipment based on the implicit field of light. The initial depth map of the reference view in the multi-view is obtained by constructing the three-dimensional features of the cost volume through camera parameters and multi-view features, and then a group of rays are projected from the camera framing direction of the reference view, and the initial depth of each ray is obtained according to the initial depth map. The initial depth is uniformly sampled within the preset range of each initial depth to obtain a number of sampling points corresponding to each ray. Then, the multi-view fusion view features of the sampling points are obtained by the correlation between the multi-view features of each sampling point obtained by the self-attention mechanism of the polar line perceptron. The fusion features of each sampling point are obtained by superimposing the multi-view fusion view features and the three-dimensional features of the cost volume. The fusion features are input into the sequence model to predict the depth value of the corresponding ray implicit field, and then multi-view modeling is performed based on the accurate depth map obtained according to the depth value of each ray. The depth estimation based on light in this method is simpler and lighter.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-perspective modeling method, apparatus and modeling equipment based on a light implicit field. Background Art

[0002] Since the advent of the seminal work MVSNet, learning-based multi-view reconstruction has received widespread attention. The core idea of MVSNet and most subsequent works is to construct a 3D cost volume in the frustum of the reference view by transforming the image features of multiple source views onto a set of forward parallel scanning planes with assumed depths, and then apply 3D convolution to the cost volume to extract 3D geometric features and regress the final depth map of the reference view.

[0003] Since 3D convolutions typically have large computational and memory consumption, most existing methods are limited to low-resolution cost volumes. Several recent works have proposed increasing the sampling or refining the cost volume to improve the resolution of the output depth map. However, such improvements still require a trade-off between depth and spatial (image) resolution. For example, CasMVSNet chooses to narrow the depth range to allow high-resolution depth maps to match the spatial resolution of the input RGB image. At the same time, it also limits the 3D convolution to a narrow bandwidth, resulting in reduced efficiency in 3D feature learning. Summary of the Invention

[0004] Based on this, it is necessary to provide a multi-perspective modeling method, device and modeling equipment based on the implicit field of light to address the above technical problems, so as to improve the speed and accuracy of multi-perspective modeling.

[0005] A multi-view modeling method based on a light implicit field, the method comprising:

[0006] Constructing a cost volume based on camera parameters and two-dimensional features of multiple views, and obtaining an initial depth map of a reference view based on the three-dimensional features of the cost volume; wherein the multiple views include a reference view and multiple source views;

[0007] Projecting a set of rays from the camera viewing direction of the reference view, obtaining an initial depth of each ray according to the initial depth map, and uniformly sampling within a preset range of the initial depth of each ray to obtain a number of sampling points corresponding to each ray;

[0008] Obtaining matching correlations between the two-dimensional features of multiple views of each sampling point through the self-attention mechanism layer of the epipolar perceptron, obtaining a multi-view fusion feature of each sampling point based on the matching correlations, and superimposing the multi-view fusion feature with the three-dimensional features of the cost volume to obtain a fusion feature of each sampling point;

[0009] The fusion features of each sampling point are sequentially input into a pre-trained sequence model to obtain the sequence features of each sampling point and the corresponding entire light feature, and the depth value of the corresponding light implicit field is predicted based on the light feature;

[0010] An accurate depth map of the reference view is obtained according to the depth values of all the light implicit fields, and multi-view modeling is performed according to the accurate depth map.

[0011] In one embodiment, the matching correlation between the two-dimensional features of multiple views of each sampling point obtained by the self-attention mechanism layer of the epipolar perceptron is:

[0012] S=SelfAttention(Q,K,V)=Softmax(QK T )V

[0013] Q=XW Q

[0014] K=XW k

[0015] V=XW v

[0016]

[0017] Among them, S is the matching relevance score, Q is the query vector, K is the key vector, V is the value vector, and X is the multi-view vector of the input. Figure 2 Dimensional features, W Q 、W k 、W v are the weights of the query vector, key vector, and value vector learned from the self-attention mechanism layer, is the two-dimensional feature of the multi-view at the Pth sampling point, N is the number of sampling points, and I is the number of views in the multi-view.

[0018] The multi-view fusion features of each sampling point obtained according to the matching correlation are:

[0019] Z=AddNorm(X)=LayerNorm(X+S)

[0020] Where LayerNorm(·) is the layer normalization function.

[0021] In one embodiment, the polar perceptron includes four self-attention mechanism layers; each of the self-attention mechanism layers includes two AddNorm layers and one feedforward layer.

[0022] In one embodiment, the fusion feature of each sampling point obtained by superimposing the multi-view fusion feature and the three-dimensional feature of the cost volume is:

[0023]

[0024] in, is the multi-view fusion feature of the sampling point, is the three-dimensional feature of the cost volume, F p is the fusion feature of the sampling point.

[0025] In one embodiment, the fusion features of each sampling point are sequentially input into a pre-trained sequence model to obtain the corresponding entire light feature:

[0026]

[0027]

[0028]

[0029]

[0030] c k =z f ○c k-1 +z i ○z

[0031] h k =z o ○tanh(c k )

[0032] Among them, F k is the sequence feature of the sampling point, h k-1 is the k-1th hidden node, z is the unit input activation vector, z f is the forget gate activation vector, z u is the update gate activation vector, z o is the output gate activation vector, c k is the light feature prediction value output at the kth moment, W, W f 、W u 、W O are the weight matrices of the unit input gate, forget gate, update gate, and output gate, respectively. b, b f 、b u 、b o are the bias vectors of the unit input gate, forget gate, update gate and output gate respectively, and ○ is the dot multiplication operator symbol.

[0033] In one embodiment, the step of training the sequence model includes:

[0034] Taking light features as input, a multi-layer perceptron is used to predict the depth value of the light implicit field:

[0035] l = MLPl (c K )

[0036] Among them, MLP is a multi-layer perceptron, c K is the output light feature prediction value, and l is the depth value of the light implicit field.

[0037] Taking the light feature prediction value output at the current moment k, the sequence feature of the sampling point, and the depth value predicted at the current moment k as input, a multi-layer perceptron is used to predict the signed distance of the sampling point on the light:

[0038]

[0039]

[0040] in, is the normalized depth value, is the normalized signed distance, s max is the maximum symbol distance on the light;

[0041] The loss function of the sequence model is constructed based on the predicted depth value and signed distance:

[0042] L=w s L s +w l L l +w sl L sl

[0043]

[0044]

[0045] Among them, L is the loss function of the sequence model, L s is the loss function of the signed distance, L l is the loss function of the depth value, L sl is the consistency penalty loss function, L1 is the L1 norm, s k is the true value of the signed distance, is the predicted value of the signed distance, l is the true value of the signed distance, is the predicted value of the signed distance;

[0046] By optimizing the loss function, a trained sequence model is obtained.

[0047] A multi-view modeling device based on a light implicit field, the device comprising:

[0048] An initial depth map acquisition module is configured to construct a cost volume based on camera parameters and two-dimensional features of multiple views, and obtain an initial depth map of a reference view based on the three-dimensional features of the cost volume; wherein the multiple views include a reference view and multiple source views;

[0049] a sampling module, configured to project a set of rays from the camera viewing direction of the reference view, obtain an initial depth of each ray according to the initial depth map, and uniformly sample within a preset range of the initial depth of each ray to obtain a number of sampling points corresponding to each ray;

[0050] A fusion feature acquisition module is configured to obtain matching correlations between the two-dimensional features of multiple views of each sampling point through a self-attention mechanism layer of an epipolar perceptron, obtain a multi-view fusion feature of each sampling point based on the matching correlations, and superimpose the multi-view fusion feature with the three-dimensional features of the cost volume to obtain a fusion feature of each sampling point;

[0051] A prediction module is used to input the fusion features of each sampling point into a pre-trained sequence model in sequence to obtain the sequence features of each sampling point and the corresponding entire light feature, and predict the depth value of the corresponding light implicit field based on the light feature;

[0052] The multi-view modeling module is used to obtain an accurate depth map of the reference view according to the depth values of all light implicit fields, and perform multi-view modeling according to the accurate depth map.

[0053] A modeling device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0054] Constructing a cost volume based on camera parameters and two-dimensional features of multiple views, and obtaining an initial depth map of a reference view based on the three-dimensional features of the cost volume; wherein the multiple views include a reference view and multiple source views;

[0055] Projecting a set of rays from the camera viewing direction of the reference view, obtaining an initial depth of each ray according to the initial depth map, and uniformly sampling within a preset range of the initial depth of each ray to obtain a number of sampling points corresponding to each ray;

[0056] Obtaining matching correlations between the two-dimensional features of multiple views of each sampling point through the self-attention mechanism layer of the epipolar perceptron, obtaining a multi-view fusion feature of each sampling point based on the matching correlations, and superimposing the multi-view fusion feature with the three-dimensional features of the cost volume to obtain a fusion feature of each sampling point;

[0057] The fusion features of each sampling point are sequentially input into a pre-trained sequence model to obtain the sequence features of each sampling point and the corresponding entire light feature, and the depth value of the corresponding light implicit field is predicted based on the light feature;

[0058] An accurate depth map of the reference view is obtained according to the depth values of all the light implicit fields, and multi-view modeling is performed according to the accurate depth map.

[0059] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0060] Constructing a cost volume based on camera parameters and two-dimensional features of multiple views, and obtaining an initial depth map of a reference view based on the three-dimensional features of the cost volume; wherein the multiple views include a reference view and multiple source views;

[0061] Projecting a set of rays from the camera viewing direction of the reference view, obtaining an initial depth of each ray according to the initial depth map, and uniformly sampling within a preset range of the initial depth of each ray to obtain a number of sampling points corresponding to each ray;

[0062] Obtaining matching correlations between the two-dimensional features of multiple views of each sampling point through the self-attention mechanism layer of the epipolar perceptron, obtaining a multi-view fusion feature of each sampling point based on the matching correlations, and superimposing the multi-view fusion feature with the three-dimensional features of the cost volume to obtain a fusion feature of each sampling point;

[0063] The fusion features of each sampling point are sequentially input into a pre-trained sequence model to obtain the sequence features of each sampling point and the corresponding entire light feature, and the depth value of the corresponding light implicit field is predicted based on the light feature;

[0064] An accurate depth map of the reference view is obtained according to the depth values of all the light implicit fields, and multi-view modeling is performed according to the accurate depth map.

[0065] Prior art efforts focused on adaptive optimization of cost volumes have resulted in limited output depth resolution. In reality, depth maps are view-dependent, but cost volumes are not. Since the target is a depth map, refining the cost volume seems neither cost-effective nor necessary. From this perspective, a large portion of the cost volume may be invisible. Therefore, this application primarily provides a solution for directly optimizing depth values along camera rays. Specifically, this solution involves first obtaining an initial depth map for a reference view in a multi-view image using the 3D features of the cost volume constructed using camera parameters and multi-view features. A set of rays are then cast from the camera's viewing direction of the reference view. The initial depth of each ray is then obtained based on the initial depth map. Samples are then uniformly sampled within a preset range of each initial depth to obtain a number of sampling points corresponding to each ray. The matching correlation between the multi-view features of each sampling point is then obtained through the self-attention mechanism layer of the epipolar perceptron to obtain a multi-view fused view feature for that sampling point. The fused features of each sampling point are then superimposed on the multi-view fused view features and the 3D features of the cost volume. The fused features are then input into a trained sequence model to predict the depth value of the corresponding ray's implicit field. Finally, multi-view modeling can be performed based on the accurate depth map obtained from the depth values of each ray.

[0066] The present invention obtains an accurate depth map of the reference view by estimating the depth value of each ray. Compared with the method of estimating depth on a three-dimensional cost volume, it has the following advantages:

[0067] 1. Since the depth map depends on the view, ray-based depth optimization is simpler and lighter;

[0068] 2. The one-dimensional implicit fields of all rays have the same spatial characteristics, and the learning of the sequence model is simplified and standardized, thereby achieving efficient network training and accurate depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 1 is a flow chart of a multi-view modeling method based on a light implicit field in one embodiment;

[0070] Figure 2 A schematic diagram of generating sampling points in one embodiment;

[0071] Figure 3 FIG. 1 is a schematic diagram of calculating denoising view features in one embodiment;

[0072] Figure 4 Schematic diagram of the network architecture of a recurrent neural network LSTM in one embodiment;

[0073] Figure 5 FIG1 is a schematic diagram of a preferred execution process of a multi-view modeling method based on a light implicit field in one embodiment;

[0074] Figure 6 Schematic diagram of the fusion of transformation features in one embodiment;

[0075] Figure 7 Schematic diagram comparing visual results of surface reconstruction using the present method and a comparative method in one embodiment.

[0076] Figure 8 FIG1 is a structural block diagram of a multi-view modeling device based on a light implicit field in one embodiment;

[0077] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0079] In one embodiment, Figure 1 As shown, a multi-view modeling method based on light implicit field is provided, which includes the following steps:

[0080] Step 102 : constructing a cost volume according to camera parameters and two-dimensional features of multiple views, and obtaining an initial depth map of the reference image according to the three-dimensional features of the cost volume.

[0081] The multi-view includes a reference view and multiple source views. This method obtains a reference image I1 and N-1 source images each time. as input.

[0082] The depth map contains the distance between each point in the scene and the camera. This method starts by constructing a 3D cost volume and estimating the initial depth map. The initial depth map is a rough and inaccurate depth map that needs to be refined through subsequent operations.

[0083] Step 104 : Project a group of rays from the camera viewing direction of the reference view, obtain the initial depth of each ray according to the initial depth map, and uniformly sample within a preset range of the initial depth of each ray to obtain a number of sampling points corresponding to each ray.

[0084] The number of rays cast can be determined according to the resolution required for the accurate depth map finally restored by optimization.

[0085] like Figure 2As shown, we select any ray for detailed description. Ideally, we can sample on the ray to generate as many sampling points as possible, but considering that most points are far away from the surface, they provide some less important information for depth estimation. In order to make the subsequent recursive neural network easier to train, this method obtains the initial depth of the selected ray according to the initial depth map and uniformly samples K points within the range of ±δ of the initial depth / rough depth estimated along the ray. Where K is the number of sampling points, for example Figure 2 In the example, K=6.

[0086] That is, this method limits the modeling of each ray to a fixed length range ±δ centered on the estimated surface intersection point calculated by the existing multi-view modeling method.

[0087] In step 106, the matching correlation between the two-dimensional features of the multiple views of each sampling point is obtained through the self-attention mechanism layer of the epipolar perceptron. The multi-view fusion feature of each sampling point is obtained based on the matching correlation. The multi-view fusion feature and the three-dimensional feature of the cost volume are superimposed to obtain the fusion feature of each sampling point.

[0088] A simple method to aggregate features at sampling point P is to extract features from multi-view images based on view projection and obtain variance, but image features are easily affected by image defects such as specular reflection and lighting changes. Naive variance averaging considers multi-view image features, which may lead to unreliable features and inaccurate cross-view feature correlation.

[0089] To address this problem, this method proposes an epipolar perception method that uses a self-attention mechanism to learn cross-view feature correlations. The epipolar perceptron can learn the importance of features from multiple views, thereby obtaining denoised multi-view fusion features of each sampling point to mitigate the impact of image defects.

[0090] In order to further improve the quality of features and make ray search easier to learn, this method obtains the fused features of each sampling point by connecting the denoised multi-view features and the three-dimensional features of the cost volume. The fused features are then incorporated into the one-dimensional implicit field input prediction model along the corresponding ray to obtain the accurate ray depth value.

[0091] In step 108 , the fusion features of each sampling point are sequentially input into a pre-trained sequence model to obtain the sequence features of each sampling point and the corresponding entire light feature, and the depth value of the corresponding light implicit field is predicted based on the light feature.

[0092] This method transforms the "range measurement" formula for each ray into learning a one-dimensional implicit field along the ray. First, because the depth map is view-dependent, ray-based depth optimization is simpler and more lightweight. Second, because the one-dimensional implicit fields of all rays have the same spatial characteristics, learning is simplified and normalized, resulting in efficient network training and accurate prediction results.

[0093] The sequence features of each sampling point are used as input to obtain the light features at the corresponding moment until the final light features are output.

[0094] Step 110 : Obtain an accurate depth map of the reference view according to the depth values of all light implicit fields, and perform multi-view modeling according to the accurate depth map.

[0095] In the above-mentioned multi-view modeling method based on the implicit field of light, the initial depth map of the reference view in the multi-view is first obtained by the three-dimensional features of the cost volume constructed by the camera parameters and multi-view features, and then a set of rays are projected from the camera viewing direction of the reference view. Then, the initial depth of each ray is obtained according to the initial depth map, and uniformly sampled within the preset range of each initial depth to obtain several sampling points corresponding to each ray. Then, the matching correlation between the multi-view features of each sampling point obtained by the self-attention mechanism layer of the polar line perceptron is used to obtain the multi-view fusion view features of the sampling point. The fusion features of each sampling point are obtained by superimposing the multi-view fusion view features and the three-dimensional features of the cost volume. The fusion features are input into the trained sequence model to predict the corresponding depth value of the implicit field of light. Finally, multi-view modeling can be performed based on the accurate depth map obtained from the depth value of each ray. The present invention obtains an accurate depth map of the reference view by estimating the depth value of each ray. Compared with the method of estimating depth on a three-dimensional cost volume, it has the following advantages: since the depth map depends on the view, ray-based depth optimization is simpler and lighter; the one-dimensional implicit fields of all rays have the same spatial characteristics, and the learning of the sequence model is simplified and standardized, thereby achieving efficient network training and accurate depth estimation.

[0096] In one embodiment, before constructing the cost volume according to the camera parameters and the two-dimensional features of multiple views, a two-dimensional convolutional network is used to extract the two-dimensional features of the reference view and the source view respectively. Here, the two-dimensional convolutional network can be 2D-UNet.

[0097] In one embodiment, before obtaining the initial depth map of the reference image according to the three-dimensional features of the cost volume, the three-dimensional features of the cost volume are extracted through a three-dimensional convolutional network, where the two-dimensional convolutional network can be 3D-UNet.

[0098] Preferably, if Figure 3 As shown in Figure 2, the matching correlation between the two-dimensional features of multiple views of each sampling point obtained through the self-attention mechanism layer of the epipolar perceptron is:

[0099] S=SelfAttention(Q,K,V)=Softmax(QK T )V

[0100] Q=XW Q

[0101] K=XW k

[0102] V=XW v

[0103]

[0104] Among them, S is the matching relevance score, Q is the query vector, K is the key vector, V is the value vector, and X is the multi-view vector of the input. Figure 2 Dimensional features, W Q 、W k 、W v are the weights of the query vector, key vector, and value vector learned from the self-attention mechanism layer, is the two-dimensional feature of the multi-view at the Pth sampling point, N is the number of sampling points, and I is the number of views in the multi-view.

[0105] The multi-view fusion features of each sampling point obtained based on the matching correlation are:

[0106] Z=AddNorm(X)=LayerNorm(X+S)

[0107] Where LayerNorm(·) is the layer normalization function.

[0108] Specifically, the network architecture of the epipolar transformer contains four self-attention layers, each followed by two AddNorm layers and a feedforward layer.

[0109] In one embodiment, the fusion feature of each sampling point is obtained by superimposing the multi-view fusion feature and the three-dimensional feature of the cost volume:

[0110]

[0111] in, is the multi-view fusion feature of the sampling point, is the three-dimensional feature of the cost volume, F p is the fusion feature of the sampling point.

[0112] Preferably, the fusion features of each sampling point are sequentially input into the pre-trained sequence model to obtain the corresponding entire light feature:

[0113]

[0114]

[0115]

[0116]

[0117] c k =z f ○c k-1 +z i ○z

[0118] h k =z o ○tanh(c k )

[0119] Among them, F k is the sequence feature of the sampling point, h k-1 is the k-1th hidden node, z is the unit input activation vector, z f is the forget gate activation vector, z u is the update gate activation vector, z o is the output gate activation vector, c k is the light feature prediction value output at the kth moment, W, W f 、W u 、W o are the weight matrices of the unit input gate, forget gate, update gate, and output gate, respectively. b, b f 、b u 、b o are the bias vectors of the unit input gate, forget gate, update gate and output gate respectively, and ○ is the dot multiplication operator symbol.

[0120] Preferably, the step of training the sequence model comprises:

[0121] l = MLP l (c K )

[0122] Among them, MLP is a multi-layer perceptron, c K is the output light feature prediction value, and l is the depth value of the light implicit field.

[0123] Taking the light feature prediction value output at the current moment k, the sequence feature of the sampling point, and the depth value predicted at the current moment k as input, a multi-layer perceptron is used to predict the signed distance of the sampling point on the light:

[0124]

[0125]

[0126] in, is the normalized depth value, is the normalized signed distance, s ma x is the maximum symbol distance on the ray;

[0127] The loss function of the sequence model is constructed based on the predicted depth value and signed distance:

[0128] L=w s L s +w l L l +w sl L sl

[0129]

[0130]

[0131] Among them, L is the loss function of the sequence model, L s is the loss function of the signed distance, L l is the loss function of the depth value, L sl is the consistency penalty loss function, L1 is the L1 norm, s k is the true value of the signed distance, is the predicted value of the signed distance, l is the true value of the signed distance, is the predicted value of the signed distance;

[0132] By optimizing the loss function, a trained sequence model is obtained.

[0133] This method involves two learning tasks: 1) predicting the signed distances of a fixed-length sequence of points (i.e., sampling points) and 2) regressing the depth of a ray. A carefully designed loss function links these two tasks, allowing this multi-task learning approach to achieve highly accurate estimates of the intersection point between each ray and the scene's surface.

[0134] like Figure 4 As shown, a network architecture of a recursive neural network LSTM (Long Short-Term Memory Network) is provided. Each sampling point on the light is sequentially input into the LSTM, and the position of the zero-crossing point (i.e., the intersection of the light and the modeled object) and the SDF (Signed Distance) of the point on the light are estimated. The one-dimensional implicit fields of all rays have the same spatial characteristics, i.e., the monotonicity of the SDF along the direction of the light. Therefore, learning will be simplified and normalized, thereby achieving efficient network training and accurate results.

[0135] like Figure 5 As shown, a schematic diagram of the preferred execution process of the present method is provided.

[0136] The first part is multi-view feature extraction. Multi-view includes a reference image and multiple source images. 2D U-Net is used to extract two-dimensional features of multi-view.

[0137] The second part is the 3D cost volume convolution, which uses 3D U-Net to extract the 3D features of the cost volume and obtain a rough depth map of the reference view through the extracted 3D features.

[0138] The third part is the matching of two-dimensional features, and multi-view fusion features are obtained through the self-attention mechanism (Self-Attention) layer and Add Norm layer in the feature sensor.

[0139] The fourth part is the depth value estimation based on the one-dimensional implicit field of light. The fusion feature obtained by superimposing the three-dimensional features of the cost volume obtained in the second part and the multi-view fusion features obtained in the third part is input into the trained sequence model to estimate the depth value of the corresponding light within the previously determined sampling point range (Determined point sampling range) ±δ. Figure 6 As shown in Figure 1, a schematic diagram of multi-view feature fusion is provided. The prediction of the 1D implicit field is lightweight, and the monotonicity of the ray-based distance field (SDF) around the intersection with the target surface facilitates robust learning. Compared with existing purely cost-volume-based methods, the depth estimation of this method is more accurate.

[0140] The fifth part is the optimization of the recurrent neural network, which uses a multi-task learning strategy to optimize the ray implicit field multi-view reconstruction method. These two tasks, namely SDF estimation of the sampling points on the ray and zero-crossing position estimation, are essentially related.

[0141] To validate this method, we provide implementation details for training and inference. During inference, three input images are of size 640×512, and the output feature size is 640×512×8. The 2D convolutional network consists of 6 convolutional layers and 6 deconvolutional layers, and each layer is followed by a ReLu layer except the last layer. The 3D cost volume is fed into a 3D convolutional network, which consists of three 3D convolutional layers and three 3D deconvolutional layers. On each ray, the number of sampling points K is 16. The point sampling δ range of DTU is 20mm, and bilinear interpolation and trilinear interpolation are used to implement feature extraction of view and cost volume, respectively. The fused feature F k The length of z is 32. f , z u , z o , c k , h k The hidden dimension of MLP is 50. l and MLP s Both contain 4 fully convolutional layers. The weight w of the multi-task learning loss function s 、w l 、w sl The perceptron and LSTM are trained jointly in an end-to-end fashion. We use the Adam optimizer with an initial learning rate of 0.0005, which is reduced by 0.9 every two epochs. Training takes 48 hours.

[0142] To evaluate the proposed method on the DTU dataset, we compare the accuracy and completeness of the final reconstruction using the distance metric in . The quantitative results are shown in Table 1.

[0143] All methods are compared using a distance metric. Numbers are reported in millimeters.

[0144] It can be seen that our method not only produces competitive results in terms of accuracy and completeness, but also achieves the most advanced overall score performance. This demonstrates the effectiveness of our proposed light implicit field, especially in balancing accuracy and completeness. Qualitative comparisons are shown in Figure 2. Figure 7 As shown, Figure 7 Figure 3 compares the visual results of surface reconstruction using our method and comparison methods. The results demonstrate that our method achieves high-quality shape reconstruction in a variety of scenarios. In particular, our method outperforms other methods in textureless regions, heavy occlusions, and scenes with complex geometry. Note the results in the challenging areas highlighted in the figure.

[0145] Table 1 Quantitative results on the DTU dataset

[0146]

[0147] Our method is compared with a baseline on the Tanks & Templates dataset. A network trained on the DTU dataset, without any fine-tuning, is used for testing on the Tanks & Templates scene. The F-score is used as the evaluation metric. Quantitative results are shown in Table 2. The results demonstrate that our method achieves the best overall performance, demonstrating good versatility across large scenes using both epipolar perception and zero-crossing estimation based on the light implicit field.

[0148] Table 2 Performance results on the Tanks&Templates dataset

[0149]

[0150] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0151] In one embodiment, Figure 8 As shown, a multi-view modeling device based on a light implicit field is provided, comprising: an initial depth map acquisition module, a sampling module, a fusion feature acquisition module, a prediction module and a multi-view modeling module, wherein:

[0152] An initial depth map acquisition module is used to construct a cost volume based on camera parameters and two-dimensional features of multiple views, and obtain an initial depth map of a reference view based on the three-dimensional features of the cost volume; wherein the multiple views include a reference view and multiple source views;

[0153] A sampling module is used to project a set of rays from the camera view direction of the reference view, obtain the initial depth of each ray according to the initial depth map, and uniformly sample within a preset range of the initial depth of each ray to obtain a number of sampling points corresponding to each ray;

[0154] The fusion feature acquisition module is used to obtain the matching correlation between the two-dimensional features of multiple views of each sampling point through the self-attention mechanism layer of the epipolar perceptron, obtain the multi-view fusion feature of each sampling point based on the matching correlation, and superimpose the multi-view fusion feature with the three-dimensional feature of the cost volume to obtain the fusion feature of each sampling point;

[0155] The prediction module is used to input the fusion features of each sampling point into the pre-trained sequence model in sequence to obtain the sequence features of each sampling point and the corresponding entire light feature, and predict the depth value of the corresponding light implicit field based on the light feature;

[0156] The multi-view modeling module is used to obtain an accurate depth map of the reference view according to the depth values of the implicit fields of all rays, and perform multi-view modeling according to the accurate depth map.

[0157] In one embodiment, the fusion feature acquisition module is further configured to obtain the matching correlation between the two-dimensional features of the multiple views of each sampling point through the self-attention mechanism layer of the epipolar perceptron:

[0158] S=SelfAttention(Q,K,V)=Softmax(QK T )V

[0159] Q=XW Q

[0160] K=XW k

[0161] V=XW v

[0162]

[0163] Among them, S is the matching relevance score, Q is the query vector, K is the key vector, V is the value vector, and X is the multi-view vector of the input. Figure 2 Dimensional features, W Q 、W k 、W v are the weights of the query vector, key vector, and value vector learned from the self-attention mechanism layer, is the two-dimensional feature of the multi-view at the Pth sampling point, N is the number of sampling points, and I is the number of views in the multi-view.

[0164] In one embodiment, the prediction module is further configured to concatenate the denoised view features and the three-dimensional features of the cost volume to obtain the transformed features of each source view:

[0165]

[0166] in, is the multi-view fusion feature of the sampling point, is the three-dimensional feature of the cost volume, F p is the fusion feature of the sampling point.

[0167] In one embodiment, the prediction module is further configured to sequentially input the fusion features of each sampling point into a pre-trained sequence model to obtain the corresponding entire light feature:

[0168]

[0169]

[0170]

[0171]

[0172] c k =z f ○c k-1 +z i ○z

[0173] h k =z o ○tanh(c k )

[0174] Among them, F k is the sequence feature of the sampling point, h k-1 is the k-1th hidden node, z is the unit input activation vector, z f is the forget gate activation vector, z u is the update gate activation vector, z o is the output gate activation vector, c k is the light feature prediction value output at the kth moment, W, W f 、W u 、W o are the weight matrices of the unit input gate, forget gate, update gate, and output gate, respectively. b, b f 、b u 、b o are the bias vectors of the unit input gate, forget gate, update gate and output gate respectively, and ○ is the dot multiplication operator symbol.

[0175] Regarding the specific limitations of the multi-perspective modeling device based on the light implicit field, please refer to the limitations of the multi-perspective modeling method based on the light implicit field above, which will not be repeated here. The various modules in the above-mentioned multi-perspective modeling device based on the light implicit field can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0176] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a multi-perspective modeling method based on a light implicit field is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0177] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0178] In one embodiment, a modeling device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0179] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0180] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0181] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multi-view modeling method based on light implicit field, characterized in that: The method comprises: Constructing a cost volume based on camera parameters and two-dimensional features of multiple views, and obtaining an initial depth map of a reference view based on the three-dimensional features of the cost volume; wherein the multiple views include a reference view and multiple source views; Projecting a set of rays from the camera viewing direction of the reference view, obtaining an initial depth of each ray according to the initial depth map, and uniformly sampling within a preset range of the initial depth of each ray to obtain a number of sampling points corresponding to each ray; Obtaining matching correlations between the two-dimensional features of multiple views of each sampling point through the self-attention mechanism layer of the epipolar perceptron, obtaining a multi-view fusion feature of each sampling point based on the matching correlations, and superimposing the multi-view fusion feature with the three-dimensional features of the cost volume to obtain a fusion feature of each sampling point; The fusion features of each sampling point are sequentially input into a pre-trained sequence model to obtain the sequence features of each sampling point and the corresponding entire light feature, and the depth value of the corresponding light implicit field is predicted based on the light feature; An accurate depth map of the reference view is obtained according to the depth values of all the light implicit fields, and multi-view modeling is performed according to the accurate depth map.

2. The method according to claim 1, characterized in that The step of obtaining matching correlations between two-dimensional features of multiple views of each sampling point through the self-attention mechanism layer of the epipolar perceptron and obtaining multi-view fusion features of each sampling point based on the matching correlations includes: The matching correlation between the two-dimensional features of multiple views of each sampling point obtained through the self-attention mechanism layer of the epipolar perceptron is: S=SelfAttention(Q,K,V)=Softmax(QK T )V Q=XW Q K=XW k V=XW v Among them, S is the matching relevance score, Q is the query vector, K is the key vector, V is the value vector, X is the input multi-view two-dimensional feature, W Q 、W k 、W v are the weights of the query vector, key vector, and value vector learned from the self-attention mechanism layer, is the two-dimensional feature of the multi-view at the P-th sampling point, N is the number of sampling points, and I is the number of views in the multi-view; The multi-view fusion features of each sampling point obtained according to the matching correlation are: Z=RddNorm(X)=LayerNorm(X+S) Where LayerNorm(·) is the layer normalization function.

3. The method according to claim 1, characterized in that The polar line perceptron includes 4 self-attention mechanism layers; each of the self-attention mechanism layers includes 2 AddNorm layers and 1 feedforward layer.

4. The method according to claim 1, wherein The step of superimposing the multi-view fusion feature and the three-dimensional feature of the cost volume to obtain the fusion feature of each sampling point includes: The fusion feature of each sampling point is obtained by superimposing the multi-view fusion feature and the three-dimensional feature of the cost volume: in, is the multi-view fusion feature of the sampling point, is the three-dimensional feature of the cost volume, F p is the fusion feature of the sampling point.

5. The method according to claim 1, wherein The step of sequentially inputting the fusion features of each sampling point into a pre-trained sequence model to obtain the sequence features of each sampling point and the corresponding features of the entire light line includes: The fusion features of each sampling point are sequentially input into the pre-trained sequence model to obtain the corresponding features of the entire light: c k =z f ○c k-1 +with i ○with h k =z o ○tanh(c k ) Among them, F k is the sequence feature of the sampling point, h k-1 is the k-1th hidden node, z is the unit input activation vector, z f is the forget gate activation vector, z u is the update gate activation vector, z o is the output gate activation vector, c k is the light feature prediction value output at the kth moment, W, W f 、W u 、W o are the weight matrices of the unit input gate, forget gate, update gate, and output gate, respectively. b, b f 、b u 、b o are the bias vectors of the unit input gate, forget gate, update gate and output gate respectively, and ○ is the dot multiplication operator symbol.

6. The method according to claim 1, characterized in that The steps of training the sequence model include: Taking light features as input, a multi-layer perceptron is used to predict the depth value of the light implicit field: ι=MLP l (c K ) Among them, MLP is a multi-layer perceptron, c K is the output light feature prediction value, ι is the depth value of the light implicit field; Taking the light feature prediction value output at the current moment k, the sequence feature of the sampling point, and the depth value predicted at the current moment k as input, a multi-layer perceptron is used to predict the signed distance of the sampling point on the light: in, is the normalized depth value, is the normalized signed distance, s max is the maximum symbol distance on the light; The loss function of the sequence model is constructed based on the predicted depth value and signed distance: L=w s L s +w l L l +w sl L sl Among them, L is the loss function of the sequence model, L s is the loss function of the signed distance, L l is the loss function of the depth value, L sl is the consistency penalty loss function, L1 is the L1 norm, s k is the true value of the signed distance, is the predicted value of the signed distance, l is the true value of the signed distance, is the predicted value of the signed distance; By optimizing the loss function, a trained sequence model is obtained.

7. The method according to any one of claims 1 to 6, characterized in that The number of the rays is determined according to the resolution required for the ultimately restored accurate depth map.

8. A multi-view modeling device based on light implicit field, characterized in that: The device comprises: An initial depth map acquisition module is configured to construct a cost volume based on camera parameters and two-dimensional features of multiple views, and obtain an initial depth map of a reference view based on the three-dimensional features of the cost volume; wherein the multiple views include a reference view and multiple source views; a sampling module, configured to project a set of rays from the camera viewing direction of the reference view, obtain an initial depth of each ray according to the initial depth map, and uniformly sample within a preset range of the initial depth of each ray to obtain a number of sampling points corresponding to each ray; A fusion feature acquisition module is configured to obtain matching correlations between the two-dimensional features of multiple views of each sampling point through a self-attention mechanism layer of an epipolar perceptron, obtain a multi-view fusion feature of each sampling point based on the matching correlations, and superimpose the multi-view fusion feature with the three-dimensional features of the cost volume to obtain a fusion feature of each sampling point; A prediction module is used to sequentially input the fusion features of each sampling point into a pre-trained sequence model to obtain the sequence features of each sampling point and the corresponding entire light feature, and predict the depth value of the corresponding light implicit field based on the light feature; The multi-view modeling module is used to obtain an accurate depth map of the reference view according to the depth values of all light implicit fields, and perform multi-view modeling according to the accurate depth map.

9. A modeling device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images