Method for predicting depth map through multi-view stereoscopic vision system, electronic device, and storage medium

Abstract

A method for predicting a depth map through a multi-view stereoscopic vision system, an electronic device, and a storage medium are provided. A method performed on an electronic device for predicting a depth map through a multi-view stereoscopic vision system includes: extracting source features from a source image and extracting aggregated reference features from a reference image; constructing a cost body based on the source features and the aggregated reference features; performing iterative estimation with multiple iterations based on the cost body to obtain a series of index fields, the iterative estimation comprising: for each iteration, using the last index field to predict a residual value through a gated recurrent neural network, and obtaining a current index field according to the residual value and the last index field; and estimating a first depth map based on the final index field.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 343,910 filed on May 19, 2022. The entire contents of the above patent application are incorporated herein by reference and are made a part of this specification. Technical Field

[0001] The present disclosure relates to three-dimensional (3D) reconstruction technology, and more particularly to a method, an electronic device, and a non-transitory computer-readable storage medium for predicting a depth map through a multi-view stereo vision system. Background Art

[0002] Multi-view stereo (MVS) aims to recover dense 3D geometry from multiple images captured from different viewpoints with a calibrated camera. It is a fundamental problem in computer vision with a wide range of applications from autonomous driving, remote sensing, augmented reality to robotics. Following the seminal MVSNet, many learning-based methods have been proposed that have achieved great improvements in accuracy or efficiency compared to traditional methods.

[0003] Most learning-based MVS methods rely on traditional plane scanning methods to generate a cost volume by comparing the convolutional neural network (CNN) features of the reference and source images in multiple depth planes, and then apply a 2D or 3D convolutional encoder-decoder architecture to aggregate and regularize the cost volume.

[0004] 2D CNN methods use multi-layer features as skip connections to help decode the cost volume for depth regression. Although skip connections improve the depth map, they weaken the role of the cost volume and the geometric knowledge in the cost volume to some extent. Therefore, when tested in unknown domains, the generalization ability of 2D CNN methods will decrease.

[0005] 3D CNN methods use soft-argmin to regress the depth map to the expected value from the cost volume distribution, so they cannot predict the best candidate but only the average candidate when dealing with flat or multimodal distributions caused by textureless, repeated, or occluded areas. Therefore, when this happens, soft-argmin cannot classify well to find the best candidate, but gives an average prediction by aggregating the best candidate, the worst candidate, and the middle candidate that have not been seen.

[0006] Therefore, how to improve the accuracy and generalization ability of multi-view stereo vision is one of the urgent problems to be solved. Summary of the invention

[0007] The present disclosure provides a method, an electronic device, and a storage medium for predicting a depth map through a multi-view stereo vision system to accurately predict depth.

[0008] According to the present disclosure, a method for predicting a depth map by a multi-view stereoscopic vision system executed on an electronic device includes: extracting source features from multiple source images and extracting aggregated reference features from a reference image, wherein the reference image is one of multiple captured images of the same scene with different camera poses, and the source image is the remaining captured image; constructing a cost body based on the source features and the aggregated reference features; performing an iterative estimation with multiple iterations based on the cost body to obtain a series of index fields, the iterative estimation including: for each iteration, predicting a residual value through a recursive neural network using the index field obtained last, and obtaining an index field of the current iteration based on the residual value and the index field obtained last; and estimating a first depth map based on the index field finally obtained in the series of index fields.

[0009] An electronic device includes: a memory for storing a program; and a processor, coupled to the memory, and the processor is used to execute the program to: extract source features from multiple source images and extract aggregated reference features from a reference image, wherein the reference image is one of multiple captured images of the same scene with different camera poses, and the source image is the remaining captured image; construct a cost body based on the source features and the aggregated reference features; perform iterative estimation with multiple iterations based on the cost body to obtain a series of index fields, the iterative estimation comprising: for each iteration, using the last obtained index field to predict a residual value through a recursive neural network, and obtaining the index field of the current iteration based on the residual value and the last obtained index field; and estimate a first depth map based on the index field finally obtained in the series of index fields.

[0010] Based on the above description, the present disclosure utilizes a recursive neural network to iteratively optimize the final index field obtained by the cost volume based on depth regression. Therefore, the depth can be predicted by learning to recursively index the asymmetric cost volume, thereby improving accuracy and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are included to provide a further understanding of the present invention, and are incorporated in and constitute a part of this specification. The accompanying drawings illustrate embodiments of the present invention and together with the description serve to explain the principles of the present invention.

[0012] Figure 1 is a block diagram of a user equipment according to an embodiment of the present disclosure.

[0013] Figure 2is a flowchart of a method for predicting a depth map by a multi-view stereo vision system according to an embodiment of the present disclosure.

[0014] Figure 3 is a schematic diagram illustrating feature extraction and cost volume construction according to an embodiment of the present disclosure.

[0015] Figure 4 is a schematic diagram illustrating depth prediction using iterative estimation according to an embodiment of the present disclosure.

[0016] Figure 5 is a schematic diagram illustrating depth prediction using iterative estimation and residual pose estimation according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] Figure 1 is a block diagram of a user device according to an embodiment of the present disclosure. Figure 1 The electronic device 100 is a device with computing capabilities. The electronic device 100 may be, for example, a server, a desktop computer, a laptop computer, a network computer, a workstation, a personal digital assistant (PDA), a tablet personal computer (PC), etc.

[0018] The electronic device 100 includes a processor 110 and a memory 120. The processor 110 is coupled to the memory 120. The processor 110 is, for example, a central processing unit (CPU), a physics processing unit (PPU), a programmable microprocessor, an embedded control chip, a digital signal processor (DSP), an application specific integrated circuit (ASIC), etc.

[0019] The memory 120 is, for example, any type of fixed or removable random-access memory (RAM), read-only memory (ROM), flash memory, hard disk drive, other similar devices, or a combination of these devices. The memory 120 stores a plurality of code segments, which are executed by the processor 110 after being installed by the processor 110 to perform a method for predicting a depth map through a multi-view stereo vision system.

[0020] In the following embodiments, depth prediction is achieved by learning to directly optimize the cost volume to iteratively estimate a series of index fields. For ease of optimization, the cost volumes at the pixel level and frame level are improved separately. At the pixel level, the transformer layer is applied asymmetrically to the reference view (but not to the source view) to construct an asymmetric cost volume to store more accurate matching similarity cues. At the frame level, a residual pose network is provided to correct the camera pose (usually obtained by visual SLAM and inevitably contains noise) to more accurately backward warp the reference features to match the corresponding features in the source view.

[0021] Figure 2 1 is a flow chart of a method for predicting a depth map by a multi-view stereo vision system according to an embodiment of the present disclosure. In this embodiment, the method for predicting a depth map is roughly divided into feature extraction, cost volume construction, and depth estimation. The following will be discussed in detail.

[0022] Please refer to Figure 1 and Figure 2 In step S205, the processor 110 extracts source features from the plurality of source images and extracts aggregated reference features from the reference image. In one embodiment, at least one imaging sensor (e.g., a calibration camera) is used to capture a set of captured images from different viewpoints of the same scene at a known camera pose, the set of captured images being represented by I = {I0, I1, I2, ..., I N-2 ,I N-1}. One of the captured images in the group is used as a reference image, and the remaining captured images are used as source images. For example, the reference image is the first captured image represented by I0, and the source image is The other captured images I1 to I N-1 Then, feature extraction is performed to extract S Extract source features (by f S ), and extract aggregated reference features from the reference image I0 (denoted by express).

[0023] Next, in step S210, the processor 110 constructs a cost volume based on the source features and the aggregated reference features. In one embodiment, the cost volume is defined on a 3D viewing cone connected to a camera in perspective projection, and the cost volume is generated by plane scanning stereo vision.

[0024] Figure 3 is a schematic diagram showing feature extraction and cost volume construction according to an embodiment of the present disclosure. Figure 3, feature extractor 310 is used for source features, and feature extractor 320 and transformer layer 330 are used for reference images. In one embodiment, feature extractors 310 and 320 are based on PairNet. PairNet is a lightweight feature pyramid network (FPN) on top of the first fourteen layers of MnasNet. Transformer layer 330 can be implemented by four-head self-attention with position encoding. Feature extractor 310 is used to extract features from source image I S Extract source features That is, {f1,f2,…,f N-1 The feature extractor 320 is used to extract the matching feature f0 from the reference image I0, and the transformer layer 330 is used to construct an aggregated reference feature based on the matching feature f0.

[0025] Specifically, in one embodiment, the reference image It is reduced to 1 / 32 scale in space and restored to 1 / 2 scale to obtain multi-scale features (For feature channels, s = 2, 4, 8, 16 and F0 = 32). In the feature extractor 320, a fusion layer G is provided to combine the 1 / 2 scale features f 0,2 and other features (f 0,4 ,f 0,8 ,f 0,16 ) is aggregated into a 1 / 4 scale matching feature f0, such as: f0 = G( <f 0,2 ↓2,f 0,4 ,f 0,8 ↑2,f 0,16 ↑4>). The fusion layer G is Conv 3×3 function, batch normalization, rectified linear unit (ReLU), and Conv 1×1 Here, ↓ x and ↑ x are downsampling and upsampling by scale x, respectively, <·> is the concatenation along the channel dimension, and where F1 = 128. Similarly, feature extractor 310 (with the same shared weights as I0) is also applied to the source image I S To extract a set of source features f S .

[0026] In addition to the local pixel-level features (matching features f0) extracted from the feature extractor 320, the transformer layer 330 is used to exploit global long-range information to better guide feature matching. The transformer layer 330 is applied to the matching features (local features) f0 of the reference image I0 to construct the aggregated reference feature as shown in equation (1): where σ(·) is the softmax operator, ω α is the learned scalar weight initialized to zero, W Q , W K , and W V is the projection matrix of query, key, and value, where h = 4 for multi-head attention. Q and

[0027] The final output of the transformer layer 330 (reference feature ) contains local information and global information, which are represented by the scalar weight ω α Balance to enhance subsequent cost body construction.

[0028] The transformer layer 330 is applied only to the reference image I0, while the source features Still have the local representation from the feature extractor 310. The asymmetric use of the transformer layer 330 provides the ability to better balance high frequency features (via high pass CNN) and low frequency features (via self-attention). High frequency features are beneficial for image matching in local and structural regions, while low frequency features (with noise information suppressed by spatial smoothing (acting as a low pass filter) of the transformer layer 330) provide more global context clues for robust matching, especially for areas full of low texture, repeating patterns, occlusions, etc. Therefore, the network architecture containing the feature extractor 320 and the transformer layer 330 can learn where to rely on global features rather than local features, and vice versa.

[0029] Next, the source feature f S and the reference features of the aggregation In one embodiment, the cost volume C0 is generated by running conventional plane scanning stereo vision, where the plane scanning stereo vision uniformly samples M0=64 depth hypotheses in the inverse depth space, with the constraint that 1 / d~U(d min ,d max ). Here, d min and d max They are the near plane and far plane of the 3D frustum.

[0030] For a given depth hypothesis d and a known camera intrinsic parameter matrix K = {Ki}(i=0,1,2,…,N-1) and reference image I0 and source image I i The original relative pose between Here, R 0.i Represents the reference image I0 and the source image I i The rotation matrix between 0,i Represents the reference image I0 and the source image I i Θ0 is the translation vector between The set of, that is, θ1 represents the relative pose of the reference image I0 and the source image I1, θ2 represents the relative pose of the reference image I0 and the source image I2, ..., θ N-1 Represents the reference image I0 and the source image I N-1 By transforming the source feature f i Warp to the reference image I0 and compute the aggregated reference features and distorted features The similarity between , the cost map is calculated. The homography H(B0, Θ0) is used as a reverse 2D grid sampling to generate the warped features Here, B0 represents a set of depth planes and θ0 represents the original set of relative poses. Specifically, the pixel p in the reference image I0 is p = (u, v, 1) T It will be distorted into the source image I as shown in equation (2) i The corresponding pixel in

[0031] Then the distorted features From i Bilinear sampling is Given the twisted features and the reference features of the aggregation The cost volume is formulated as equation (3):

[0032] For all depth planes Construct the cost volume C0 and get a 3D tensor, represented as

[0033] After constructing the cost body C0, in step S215, the processor 110 performs an iterative estimation with multiple iterations based on the cost body C0 to obtain a series of index fields. Then, in step S220, the processor 110 estimates the first depth map based on the index field finally obtained in the series of index fields. In the iterative estimation, for each iteration, the residual value is predicted by a recursive neural network (e.g., a gated recurrent unit (GRU)) using the index field obtained last time, and the index field of the current iteration is obtained according to the residual value and the index field obtained last time. For example, in the first iteration, the updated index field of the current iteration is obtained using the predicted residual value and the initial index field. In each iteration except the first iteration, the updated index field of the current iteration is obtained using the predicted residual value and the index field from the last iteration. In one embodiment, the index field of the current iteration is obtained by adding the residual value to the index field obtained last time.

[0034] In an embodiment, the processor 110 learns to index the cost volume C0 by approximating the correct depth plane pixel by pixel via an index field. The proposed iterative estimation of the index field (the index grid used to identify the depth plane) enables learning to anchor in the cost volume domain. Specifically, the iterative estimation recursively predicts residual values ​​to retrieve the cost features for the next iteration in the descending direction of the cost volume, and also directly indexes (i.e., samples by linear interpolation) the depth plane to render a depth map, where the depth map is iteratively optimized to approach the true depth, making the system end-to-end trainable.

[0035] Figure 4 is a schematic diagram showing depth prediction using iterative estimation according to an embodiment of the present disclosure. In an embodiment, in order to optimize the dense stereo matching problem, semi-global matching (SGM) is adopted. In SGM, the cost volume C0 is iteratively aggregated by adding the costs of all 1D minimum cost paths terminating at pixel p when the disparity is d when traversing from pixel pr to pixel p along direction r (out of sixteen directions), and the best disparity of each pixel p is given by d * (p) = argmin d (C'(p,d)) is given, where C' is the aggregated cost volume. Similar to SGM, in an embodiment, when predicting disparity from the cost volume C0, argmin is replaced by softargmin, but it still i) needs to predefine the update direction r when traversing from pixel pr to pixel p, and ii) softargmin focuses on measuring the distance between the expected value of the disparity map and the true value disparity, and therefore cannot handle the multimodal distribution in C0 well.

[0036] Please refer to Figure 4 , the architecture for predicting the depth map also includes a context feature network 410 and a GRU-based module 420. The context feature network 410 is used to extract context features from the reference image I0 For example, the context feature network 410 is composed of a plurality of residual blocks. And the GRU-based module 420 is used to implicitly optimize the cost volume C0. By expanding the optimization problem to iterative updates, the index field sequence is estimated.

[0037] The GRU-based module 420 starts with an initial index field φ0 and estimates a series of index fields in multiple iterations. (i.e., φ1,…,φ T ). Specifically, before performing the iteration, the initial value is determined from the cost body C0 by the sof-argmin function as the initial index field φ0. Then, at each iteration t (assuming T iterations, t = 1, 2, ..., T), the index field φ t is estimated as an index grid to iteratively better approximate (i.e., closer to the ground truth) depth planes with lower matching costs.

[0038] In one embodiment, a four-level matching pyramid It is constructed by repeatedly pooling the cost volume C0 along the depth dimension using a kernel of size 2. In order to index the matching pyramid, a lookup operator L is defined, which is used to calculate the cost feature from the cost volume C0 based on the last obtained index field For example, in iteration 1, the lookup operator L retrieves the cost feature from the cost volume C0 according to the index field φ0 In iteration 2, the lookup operator L retrieves the cost feature from the cost volume C0 according to the index field φ1 …and in iteration T, the search operator L is based on the index field φ T-1 Retrieve cost features from cost volume C0

[0039] Index field φ t , retrieved cost features And context features concatenate and with the latent hidden state h t The GRU module 420 outputs the residual value Δφ. t-1 and the new hidden state h t+1 : φ t =φ t-1 +Δφ t-1 .

[0040] By giving the final index field φT In the case of , the depth hypothesis is sampled via linear interpolation and the first depth map D of iteration T is estimated T In one embodiment, assume that the index field φ T At 1 / 4 resolution, the index field φ is upsampled using an upsampling operator U (e.g., a convex combination of 3×3 neighbors). T Upsample to full resolution. For example, use two convolutional layers to extract the hidden state h t Prediction weight mask And perform softmax on the weights of these 9 adjacent elements. The final high-resolution index field (the upsampled index field ) is obtained by weighted combination of 9 adjacent elements and reshaped to resolution H × W. Convex combination can be implemented using the einsum function in PyTorch.

[0041] In one embodiment, a set of depth planes Used to construct the cost volume C0. A small M0 helps reduce computation and space. Perform spatial upsampling on the depth plane B0 to obtain the depth plane The depth index field is indexed by the scale s D =M1 / M0=4 adjustment. To simulate the convex combination mentioned above, a similar weighted summation is applied along the depth dimension when sampling the depth from the depth plane B1. Specifically, three convolutional layers are used to predict another mask from the hidden state And further reshaped to H×W×M1. Given a pixel p, and the upsampling index field Final depth (first depth map D T ) is estimated as shown in equation (4): Among them, for a given pixel p, the neighboring elements are indexed in the field The radius is r = 4, and The largest integer less than or equal to i is given, and [i] represents indexing the depth plane B1 by linear interpolation since the index i is a real number.

[0042] The iterative estimation of the index field enables learning to be directly anchored in the cost volume domain. This indexing paradigm distinguishes our method from other depth estimation methods, such as convolution-based depth regression by fusing cost volume and skipped multi-layer features via 2D CNN, or soft-argmin-based depth regression after cost volume aggregation and regularization via 3D CNN.

[0043] In another embodiment, an accurate cost volume is beneficial for GRU-based iterative optimization. The quality of the generated cost volume C0 is not only determined by the aggregated reference features and source feature f S The camera pose is determined by a skewed method (for which an asymmetric use of transformer layers is proposed) and also by homography warping. In practice, the camera pose is usually obtained by visual SLAM (Simultaneous Localization and Mapping) algorithms and inevitably contains noise. Therefore, a residual pose network is provided to correct the camera pose to accurately back-warp the reference features to match the corresponding features in the source image.

[0044] Figure 5 is a schematic diagram illustrating depth prediction using iterative estimation and residual pose estimation according to an embodiment of the present disclosure. Figure 5 The embodiment shown is based on Figure 4 The system architecture (including feature extractors 310 and 320, transformer layer 330, context feature extractor 410, GRU-based iterative optimization layer 420, and residual pose network 520, etc.) is trained to minimize the loss between the predicted depth map and the true depth.

[0045] Here, in view of the inaccuracy of the pose between the reference image and the source image, the residual pose network 520 is combined to correct the relative pose. This basically corrects the cost volume at the frame level. Another round of GRU-based iterative updates will be performed based on the new cost volume, and the depth map is predicted as the final output.

[0046] In estimating the first depth map D T Afterwards (see Figure 4 ), based on the first depth map D T The adjusted residual pose constructs an updated cost volume C1. Then, the iterative estimation based on the updated cost volume C1 is re-executed to obtain another series of index fields (φ"1,…,φ" T ), and based on the final index field obtained in another series of index fields (φ" T ) Estimate the second depth map D′ T '. The second depth map D' T The evaluation of ′ is similar to the first depth map D T assessment.

[0047] Before constructing the update cost volume C1, the source image I S Warp to obtain multiple distorted source images (i.e., warped S |D t ,D GT ,Θ0)), the residual pose network 520 is used to use the first depth map D T , camera pose (i.e., original relative pose Θ0), and true depth D gtAt least one pair of reference image I0 and distorted source image Encode and obtain the residual pose Δθ i Here, Θ0 represents the relationship between the reference image I0 and the source image I1 to the source image I N-1 In an embodiment, the residual pose network 520 may be implemented using a residual neural network (ResNet) backbone pre-trained with an image network.

[0048] Specifically, given the (currently estimated) depth map D at iteration t in the first iteration estimation T and the true depth D gt , source image I i By using Eq. (2) with the (noisy) ground-truth camera pose (i.e., the original relative pose θ0) and D T or D gt is warped into the reference image I0. During network training, the probability prob(D T )=0.6 at D T and D gt During network inference, the depth map D is always selected. T , that is, during network inference, the probability prob(D T )=1. The input of the residual pose network 520 is obtained by concatenating the reference image I0 and the distorted source image The result of the connection obtained And the output is an axis-angle representation, which is further converted into a residual rotation matrix (residual pose) Δθ i , to obtain the updated relative pose θ' i =Δθ i ·θ i Here, Δθ i is the relative pose θ i For each pair of source and reference images, the set of residual poses is And perform correction as Θ1 = ΔΘ·Θ0. The set of updated poses is The updated homography H(B0,θ1) is used to implement the updated homography H(B0,θ1), and a more accurate cost volume, namely, an updated cost volume C1, is obtained based on equations (2) and (3).

[0049] Then, the iterative estimation is re-performed based on the updated cost volume C1 to obtain another series of index fields. Figure 4As shown in the description, the initial value is determined from the updated cost body C1 by the soft-argmin function as the initial index field φ"0. Then, at each iteration t (assuming T iterations of updates, t = 1, 2, ..., T), the GRU-based module 420 is used to input the index field φ" obtained last time. t-1 Cost features and context features retrieved from the updated cost volume C1 Get the residual value Δφ" t-1 Then, by converting the residual value Δφ" t-1 and the last obtained index field φ" t-1 Add, estimate the index field φ" t .

[0050] Extensive experiments on the real-world MVS dataset demonstrate that our method achieves state-of-the-art performance in both in-dataset evaluation and cross-dataset generalization.

[0051] The present disclosure is implemented in PyTorch and trained end-to-end with a mini-batch size of 8 per NVIDIA RTX A6000 GPU. During training, the AdamW optimizer is used and the gradient is clipped to the range [-1, 1]. When the cost volume is generated by plane scanning stereo vision, the number of plane hypotheses is set to M1 = 256. The context network and the 3-level GRU architecture adopt the same hyperparameters. In the training schedule, the network architecture of the present disclosure is trained for 20 epochs, with an initial learning rate of 1e-4 and decayed by half at the 4th and 8th epochs, respectively. For fair comparison, the baseline PairNet and IterMVS are trained for 20 epochs on the same training samples of ScanNet, using the official code and Adam optimizer. For the baseline PairNet, the learning rate scheduler recommended in its paper is used. For the baseline Iter-MVS, a learning rate of 1e-4 is used, decaying by half at the 4th and 8th epochs. The present disclosure can make more accurate and clear depth predictions, especially for areas close to boundaries and edges. For both near and far objects, our method outperforms the baselines IterMVS and Pair-Net.

[0052] In summary, embodiments of the present disclosure provide a method, electronic device, and storage medium for predicting a depth map through a multi-view stereo vision system. The present disclosure estimates a depth map from a set of multi-view images with poses (one as a reference image and the others as source images) and utilizes a learning optimization paradigm that iteratively indexes a plane-scanned cost volume and regresses the depth map through a convolutional GRU. The cost volume construction is further improved by combining a transformer block with a reference image and a residual pose network to correct the relative pose.

[0053] Since the cost volume plays a vital role in encoding multi-view geometry, the present disclosure aims to improve the cost volume construction at the pixel level and the frame level. At the pixel level, a transformer layer is introduced into the reference image (but not the source image) to break the symmetry of the siamese network (commonly used in MVS to extract image features). This asymmetric body allows the network to extract global features from the reference image to predict the depth map of the reference image. In addition, at the frame level, given the pose inaccuracy between the reference image and the source image, a residual pose network is combined to correct the relative pose. This essentially corrects the cost volume at the frame level.

[0054] Extensive experiments on ScanNet, DTU, 7-Scenes, and RGB-D Scenes V2 demonstrate the excellent accuracy and cross-dataset generalization ability of our method.

[0055] It will be apparent to those skilled in the art that various modifications and variations may be made to the disclosed embodiments without departing from the scope of the present disclosure. In view of the foregoing, the present disclosure is intended to cover modifications and variations as long as they fall within the scope of the following claims and their equivalents.

Claims

1. A method for predicting a depth map by a multi-view stereo vision system performed on an electronic device, comprising: extracting source features from a plurality of source images and extracting aggregated reference features from a reference image, wherein the reference image is one of a plurality of captured images of the same scene with different camera poses and the source image is the remaining captured image; constructing a cost volume based on the source features and the aggregated reference features; Performing iterative estimation with multiple iterations based on the cost volume to obtain a series of index fields, the iterative estimation comprising: for each iteration, using the index field obtained last time to predict a residual value through a recursive neural network, and obtaining an index field of a current iteration according to the residual value and the index field obtained last time; and A first depth map is estimated based on the index field finally obtained from the series of index fields.

2. The method for predicting a depth map by a multi-view stereo vision system according to claim 1, further comprising: extracting contextual features from the reference image using a contextual feature network; Wherein, the iterative estimation further comprises: Before executing the iteration, determining an initial value from the cost volume by a soft-argmin function as an initial index field of the series of index fields; and For each iteration, include: Retrieving a cost feature from the cost volume according to the last obtained index field; Inputting the cost feature and the context feature into the recurrent neural network and obtaining the residual value; and The index field of the current iteration is obtained by adding the residual value to the index field obtained last time.

3. The method for predicting a depth map by a multi-view stereo vision system according to claim 1, wherein: After estimating the first depth map, the method further comprises: constructing an updated cost volume based on the residual pose adjusted by the first depth map; re-performing the iterative estimation based on the updated cost volume to obtain another series of index fields; and A second depth map is estimated based on the finally obtained index field in the another series of index fields.

4. The method for predicting a depth map by a multi-view stereo vision system according to claim 3, wherein: Before calculating the updated cost volume, the method further includes: warping the source image to obtain a plurality of warped source images; and The reference image and the warped source image are encoded using a residual pose network with at least one of the first depth map, a camera pose, and a true depth, and the residual pose is obtained.

5. The method for predicting a depth map by a multi-view stereo vision system according to claim 1, wherein: Extracting the aggregated reference features from the reference image comprises: extracting matching features from the reference image by a feature extractor; and The aggregated reference features are constructed based on the matching features through a transformer layer.

6. The method for predicting a depth map by a multi-view stereo vision system according to claim 1, wherein: Estimating the first depth map comprises: The first depth map is estimated by sampling a plurality of depth planes through linear interpolation using the index field finally obtained from the series of index fields.

7. An electronic device comprising: Memory, used to store programs; as well as a processor coupled to the memory, and configured to execute the program to: extracting source features from a plurality of source images and extracting aggregated reference features from a reference image, wherein the reference image is one of a plurality of captured images of the same scene with different camera poses and the source image is the remaining captured image; constructing a cost volume based on the source features and the aggregated reference features; Performing iterative estimation with multiple iterations based on the cost volume to obtain a series of index fields, the iterative estimation comprising: for each iteration, using the index field obtained last time to predict a residual value through a recursive neural network, and obtaining an index field of a current iteration according to the residual value and the index field obtained last time; and A first depth map is estimated based on the index field finally obtained from the series of index fields.

8. The electronic device according to claim 7, wherein: The processor is configured to execute the program to: extracting contextual features from the reference image using a contextual feature network; Wherein, the iterative estimation further comprises: Before executing the iteration, determining an initial value from the cost volume by a soft-argmin function as an initial index field of the series of index fields; and For each iteration, include: Retrieving a cost feature from the cost volume according to the last obtained index field; Inputting the cost feature and the context feature into the recurrent neural network and obtaining the residual value; and The index field of the current iteration is obtained by adding the residual value to the index field obtained last time.

9. The electronic device according to claim 7, wherein: The processor is configured to execute the program to: constructing an updated cost volume based on the residual pose adjusted by the first depth map; re-performing the iterative estimation based on the updated cost volume to obtain another series of index fields; as well as A second depth map is estimated based on the finally obtained index field in the another series of index fields.

10. The electronic device according to claim 9, wherein: The processor is configured to execute the program to: distorting the source image to obtain a plurality of distorted source images; as well as The reference image and the warped source image are encoded using a residual pose network with at least one of the first depth map, a camera pose, and a true depth, and the residual pose is obtained.

11. The electronic device according to claim 7, wherein: The processor is configured to execute the program to: extracting matching features from the reference image; as well as The aggregated reference features are constructed based on the matching features through a transformer layer.

12. The electronic device according to claim 7, wherein: The processor is configured to execute the program to: The first depth map is estimated by sampling a plurality of depth planes through linear interpolation using the index field finally obtained from the series of index fields.

13. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium records at least one programming instruction, and after being loaded into the electronic device, the at least one programming instruction performs the following steps: extracting source features from a plurality of source images and extracting aggregated reference features from a reference image, wherein the reference image is one of a plurality of captured images of the same scene with different camera poses and the source image is the remaining captured image; constructing a cost volume based on the source features and the aggregated reference features; Performing iterative estimation with multiple iterations based on the cost volume to obtain a series of index fields, the iterative estimation comprising: for each iteration, using the index field obtained last time to predict a residual value through a recursive neural network, and obtaining an index field of a current iteration according to the residual value and the index field obtained last time; and A first depth map is estimated based on the index field finally obtained from the series of index fields.

Images