Neural network structure and estimation method for multi-view depth based on confidence fusion

Abstract

The application relates to a neural network structure and an estimating method for multi-view depth based on confidence fusion, aiming at the confidence jump and confidence low problem existing in the multi-view depth estimation task, and proposes a new multi-view depth estimation method and network based on confidence fusion. Including a four-scale feature extraction network and a four-level depth sampling module. Make full use of the training advantages in the multi-round training process, propose a neighborhood average pooling strategy, combine a CL joint loss function, and significantly improve the integrity and accuracy of target view depth estimation.

Description

Technical Field

[0001] This invention belongs to the field of machine learning and computer vision, specifically 3D reconstruction technology. It relates to a neural network structure and estimation method for multi-view depth based on confidence fusion. Background Technology

[0002] In recent years, with the rapid development of machine learning and artificial intelligence, significant achievements have been made in the field of computer vision, with vision-based 3D reconstruction being one of the most representative technologies. Multi-view depth estimation is the most critical technical aspect of vision-based 3D reconstruction. Due to its advantages such as convenient data acquisition and low cost, multi-view depth estimation technology is widely used in scenarios such as autonomous driving, virtual reality, and ancient building restoration, becoming one of the key approaches for artificial intelligence to perceive and analyze the real world.

[0003] From the perspective of methodology, multi-view depth estimation techniques can be divided into geometry-based models and learning-based models. Geometry-based models obtain a depth map of the reference viewpoint through random initialization, photometric consistency calculation, and iterative propagation optimization. They offer strong interpretability but struggle with repetitive textures, reflective regions, and low-texture areas, resulting in high accuracy but low completeness. Learning-based models, on the other hand, construct a dataset, design and train a neural network model, and then estimate the depth map of the reference viewpoint based on the trained neural network. Because deep neural networks can adaptively perceive larger image regions and more complex texture relationships, learning-based models can significantly improve the accuracy and completeness of the depth estimation map.

[0004] While learning-based models offer significant advantages, several insurmountable challenges remain in improving the quality of depth estimation. First, real-world multi-view depth estimation datasets often consist of images captured from different angles by a single camera within short time intervals. Due to object motion and lighting variations, these datasets inherently contain considerable noise, causing the model to oscillate between various approximate solutions later in the training process. Second, while classification-based multi-scale, multi-view depth estimation models, as the mainstream framework, offer performance improvements, the uniform division of the reference view's ray distribution leads to a depth confidence gap, resulting in regular confidence gaps in previously accurately estimated, smooth depth regions. Summary of the Invention

[0005] Technical problems to be solved

[0006] To avoid the shortcomings of existing technologies, this invention proposes a neural network structure and estimation method for multi-view depth based on confidence fusion, which solves the problems of confidence oscillation and confidence loss in learning-based multi-scale multi-view depth estimation methods, and significantly improves the completeness and accuracy of multi-view depth estimation.

[0007] Technical solution

[0008] A multi-view depth estimation method based on confidence fusion is characterized by the following steps:

[0009] Step 1: Take photos of the same scene from N different perspectives to obtain N RGB images {I1, I2, ..., I...} n} Input these N images into the Colmap algorithm to obtain the camera intrinsic and extrinsic parameters corresponding to these N images, as well as the sparse point cloud of the entire scene, which contains K three-dimensional points;

[0010] Choose any RGB image as a reference image and calculate the overall confidence value G between the reference image and other viewpoint images pairwise. j The confidence value G of the j-th image compared to the reference image and the K sparse point clouds. j,k Summing yields:

[0011]

[0012] G j Let j be the overall confidence value between the j-th image and the reference image.

[0013] According to G j Select M images as source images in descending order of value;

[0014] Step 2: Extract features at four scales from the reference image and the M source images using a 4-scale feature extraction network, resulting in a deep feature set F = {F_{i} ... R F1, ..., F M};

[0015] Each deep feature set contains deep features at four scales:

[0016]

[0017] The dimensions of the deep features at different scales satisfy:

[0018]

[0019] Where s represents the scale number, C s Indicates the number of channels at the current scale;

[0020] Step 3: At scale 1, based on the relative positions of the intrinsic and extrinsic parameters of the selected reference frame and the sparse point cloud generated by Colmap, calculate the depth range D0 of the scene = [d min ,d max ], where d min and d maxThese represent the closest and furthest distances of all points in the sparse point cloud relative to the reference map, respectively.

[0021] Then, D0 is divided into 16 equal intervals, and the midpoint of each interval is taken as the initial sampling depth value. In scales 2 to 4, the depth value generated by the previous scale is used as the intermediate value for sampling the depth range of the current scale. The depth range size of each scale satisfies the following:

[0022]

[0023] Similar to the generation method of scale 1, D s Divide the sample into 16 equal intervals, and take the midpoint of each interval as the depth value for the current scale.

[0024] Step 4: For each scale s, take each deep feature at scale s from the M source images obtained in Step 2. By transforming the homography transformation matrix H, the projected data is projected onto the 16 depth samples obtained in step 3 to obtain the projected deep features.

[0025] The H

[0026] H = [d l K i R i R -1 K -1 K i t i -K i R i R -1 t] 3×4

[0027] Where K, R, and t represent the intrinsic parameter matrix, rotation matrix, and translation matrix of the reference graph, respectively; K i R i and t i Let represent the intrinsic parameter matrix, rotation matrix, and translation matrix of the i-th source image, respectively;

[0028] Step 5: Through the projection transformation in Step 4, the deep features of the M source images are projected at 16 sampling depths, resulting in the projected deep features. l represents the depth sampling sequence number; at the same pixel location, the deep features projected from M source images and the deep features from the reference image are fused without biased variance to obtain the cost volume. The cost body is generated in the following way:

[0029]

[0030] in,

[0031] Step 6: Transfer the cost body Input is fed into a depth-biased cost volume regularization network, and output is a probability volume. Then, along the depth dimension, neighborhood average pooling is performed on the probability volume. That is, for each pixel position and each depth, the probability value is averaged with the two depth values ​​in the two directions before and after it, for a total of five depth values, and this average is used as the new probability of the current pixel position and current depth. Finally, the depth values ​​represented by the 16 positions with the highest probabilities for each pixel are the estimated depth of the current pixel, and the highest probability is used as the confidence of the current estimate.

[0032] Step 7: Repeat steps 3 to 6 from scale 1 to scale 4. The depth map output at scale 4 is the preliminary depth estimation map of the reference image with the same pixel size as the original.

[0033] Step 8: Save the depth map output from training epochs 11 to 15. For the depth value of each pixel, starting from epoch 11, update the depth value corresponding to pixels whose probability value is less than the probability value estimated in the current epoch. The update process is as follows:

[0034]

[0035]

[0036] Where e represents the current training round, and P and D represent the depth value and feasibility, respectively; the depth value after updating all rounds is the final depth value of the reference graph.

[0037] The confidence value G j,k :

[0038]

[0039] Where, θ k θ0 represents the angle between the k-th point in the sparse point cloud and the reference image and image j; θ0, These represent experimental hyperparameters.

[0040] The experimental hyperparameters θ0, In the experiment, the values ​​were set to 5, 1, and 100 respectively.

[0041] The number of channels C at the current scale s In the experiment, scales 1 to 4 were set to 64, 32, 16 and 8 respectively.

[0042] A neural network structure for the aforementioned confidence-based multi-view depth estimation method is characterized by comprising a 4-scale feature extraction network and a four-level depth sampling module; the output of the 4-scale feature extraction network is connected to the input of an initial depth sampling module and a third-level depth resampling module, respectively; the depth sampling module and the third-level depth resampling module are connected to a cost body regularization module for feature fusion and cost body generation, and then connected to a neighborhood average pooling module as the output of the module; the output of each level of the neighborhood average pooling module is connected to the feature fusion and cost body generation module at the next level input; the four scale features of the 4-scale feature extraction network at the input are respectively input to the input of the initial depth sampling module and the third-level depth resampling module, and the output of the third-level depth resampling module obtains the multi-view depth estimation data; wherein F 1 ={F 1 R F 1 1, ..., F 1 M} Input to the initial depth sampling module, F 2 ={F 2 R F 2 1, ..., F 2 M The data input to the first-level depth resampling module is fused with the data from the initial depth sampling module, F. 3 ={F 3 R F 3 1, ..., F 3 M The data input to the second-level depth resampling module is fused with the output of the first-level depth sampling module. 4 ={F 4 R F 4 1, ..., F 4 M The data input to the third-level depth resampling module is fused with the output of the second-level depth sampling module.

[0043] The 4-scale feature extraction network is one or multiple 4-scale feature extraction networks in parallel, serving as a feature extraction network for the reference image and M source images in series or in parallel.

[0044] The four-scale feature extraction network uses the F4N feature extraction network.

[0045] The cost volume regularization network employs a deep paranoid cost volume 3D convolutional neural network.

[0046] The depth values ​​obtained by the initial depth sampling module, the second-level depth resampling module, the third-level depth resampling module, and the fourth-level depth resampling module are used as the feature mapping spatial positions of the four scales.

[0047] The reference depth map is generated by fusing depth estimation maps from multiple training rounds; the probability volume is obtained by averaging the probabilities of the original probability volume at the two positions before and after each pixel in the depth dimension, for a total of 5 positions.

[0048] Beneficial effects

[0049] This invention proposes a neural network structure and estimation method for multi-view depth based on confidence fusion. Addressing the issues of confidence fluctuations and confidence dips in multi-view depth estimation tasks, this invention proposes a novel multi-view depth estimation method based on confidence fusion. It fully utilizes the training advantages of multiple training rounds, proposes a neighborhood average pooling strategy, and combines it with the CL joint loss function to significantly improve the completeness and accuracy of target view depth estimation. Attached Figure Description

[0050] Figure 1 : Neural network structure diagram of the present invention

[0051] Figure 2 Examples of the input reference image IR, output confidence map C, and depth map D of this invention.

[0052] Figure 3 Flowchart of the method of this invention Detailed Implementation

[0053] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:

[0054] This disclosure provides a perspective depth estimation method based on confidence fusion, the method comprising:

[0055] Multi-view depth estimation network structure such as Figure 1 As shown;

[0056] It includes a 4-scale feature extraction network and a four-level depth sampling module; the output of the 4-scale feature extraction network is connected to the input of the initial depth sampling module and the third-level depth resampling module, respectively; the depth sampling module and the third-level depth resampling module are connected to the cost body regularization module as a feature fusion and cost body generation module, and then connected to a neighborhood average pooling module as the output of the module; the output of each level of the neighborhood average pooling module is connected to the feature fusion and cost body generation module at the next level input; the four scale features of the 4-scale feature extraction network at the input are respectively input to the input of the initial depth sampling module and the third-level depth resampling module, and the output of the third-level depth resampling module obtains multi-view depth estimation data; where F 1 ={F 1 R F 1 1, ..., F 1 M} Input to the initial depth sampling module, F 2 ={F 2 R F 2 1, ..., F 2 M The data input to the first-level depth resampling module is fused with the data from the initial depth sampling module, F. 3 ={F 3 R F 3 1, ..., F 3 M The data input to the second-level depth resampling module is fused with the output of the first-level depth sampling module. 4 ={F 4 R F 4 1, ..., F 4 M The data input to the third-level depth resampling module is fused with the output of the second-level depth sampling module.

[0057] The 4-scale feature extraction network consists of one or multiple parallel 4-scale feature extraction networks, used as a reference image and M source images for cascaded or parallel feature extraction. The 4-scale feature extraction network employs the F4N feature extraction network.

[0058] The cost volume regularization network employs a deep paranoid cost volume 3D convolutional neural network.

[0059] The depth values ​​obtained by the initial depth sampling module, the second-level depth resampling module, the third-level depth resampling module, and the fourth-level depth resampling module are used as the feature mapping spatial positions of the four scales.

[0060] The reference depth map is generated by fusing depth estimation maps from multiple training rounds.

[0061] The probability volume is obtained by averaging the probabilities of the original probability volume at five positions, two before and two after each pixel in the depth dimension.

[0062] The perspective depth estimation method based on confidence fusion has the following steps:

[0063] 1) Selection of reference perspective

[0064] Input images of the same scene from multiple perspectives {I1, I2, ..., I...} n Based on the Colmap algorithm, the camera intrinsic parameter matrix K corresponding to n images is obtained. 3x3 and extrinsic parameter matrix T 3x4 = [R|t], and the depth range {D1, D2, ..., D} corresponding to each image. n The entire scene is accurately estimated as a sparse point cloud {P1, P2, ..., P}. m}

[0065] Based on the viewpoint selection algorithm proposed in MVSNet, the confidence values ​​between each pair of n images are calculated, G. ij,k This indicates that the i-th and j-th images represent the sparse point cloud P. k The confidence values ​​between P and P k This refers to sparse point clouds visible in both images, calculated as follows:

[0066]

[0067] Where, θ k Represents sparse point cloud P k The angle between reference figure i and reference figure j; θ0, These represent experimental hyperparameters, which were set to 5, 1, and 100 respectively in the experiment.

[0068] Calculate each G ij,k Then, the overall feasibility of the i-th and j-th images is calculated as follows, where K is the set of indexes of the coefficient point clouds visible in both images:

[0069] G ij =∑ k∈K G ij,k

[0070] After calculating the overall confidence level between all pairs of images, for each image, the four viewpoint images with the highest overall feasibility are selected to form a five-image set Imeigetun={I nR I n1 I n2 In3 I n4}

[0071] 2) The extraction process of the 4-scale feature extraction module:

[0072] Five images from the image set In are sequentially input into a 4-scale feature extraction network F4N to obtain a deep feature set F = {F_n}. R FR = {F1, F2, F3, F4}, where each element in the set represents the deep feature set of the corresponding image. Each deep feature set contains deep features at four scales. R1 F R2 F R3 F R4 The dimensions of deep features at different scales satisfy... Where s represents the scale number, C s This indicates the number of channels at the current scale. (C) s In the experiment, scales 1 to 4 were set to 64, 32, 16 and 8 respectively.

[0073] 3) Uniform depth sampling module steps:

[0074] The uniform depth sampling module is divided into two sub-modules: initial depth sampling and estimation-based depth sampling. Initial depth sampling refers to sampling the depth of deep features F at a scale of s=1. R1 For each pixel x in I, nR The corresponding depth range D R Divide the image into 16 uniform intervals, and use the midpoint of each interval as the depth sample of pixel x {d1, d2, ..., d...}. 16 Depth sampling based on depth estimation means that for each pixel x, except for scale s=1, the depth map estimate D is obtained from the previous scale. s-1 Then, D s-1 After bilinear interpolation and upsampling by a factor of 2, the upsampled D s-1 The depth value is used as the midpoint, DR / 2 s As a new interval, this new interval is evenly divided into 16 sub-intervals. The midpoint of all intervals is used as the depth sampling point {d1, d2, ..., dn} of pixel x. 16}

[0075] 4) Steps of the feature fusion and cost volume generation module:

[0076] For each scale s, divide the input by I R The deep features of the other four images at scale s are projected onto the image at scale I using the homography transformation matrix H. R In different depth spaces with the same orientation, the calculation method for H is as follows:

[0077]

[0078] By performing the above projection transformation, that is, multiplying the corresponding features by the homography transformation matrix H, we obtain the deep features F of the source image projected onto the reference image viewpoint at each sampling depth. s,dl n The projection of the reference image at each depth is the deep feature of the reference image itself.

[0079] At the same depth d l For the same pixel x, the cost is calculated based on unbiased variance fusion, as shown below:

[0080]

[0081] The cost of all depth and pixel location is used to construct the cost volume V. s .

[0082] 5) Steps of the cost body regularization module:

[0083] Cost body V s Input the deep paranoia cost regularization network to obtain the probability volume P of the current stage. s Unlike typical 3DCNN network structures, the depth bias cost volume regularization network proposed in this invention maintains the depth dimension unchanged during downsampling.

[0084] 2. The differences between the network proposed in this invention and CasMVSNet:

[0085] The network proposed in this invention differs from CasMVSNet in the following ways: 1) CasMVSNet's feature extraction module contains two downsampling convolutional layers, while the method proposed in this invention contains three downsampling convolutional layers; 2) CasMVSNet's depth sampling divides the original depth space into 48, 32, and 8 depths sequentially, while the depth partitioning strategy proposed in this invention uses 16, 16, 16, and 16 depths; 3) The 3D convolutional kernels of the cost regularization network in CasMVSNet are 3*3*3 and 5*5*5, while the 3D convolutional kernels of the regularization network in the method proposed in this invention are 1*3*3 and 1*5*5; 4) CasMVSNet's training loss function is based on multi-class weighted regression loss, while the method proposed in this invention is based on a joint loss function.

[0086] 3. Loss Function

[0087] 1) Cross-entropy loss

[0088] For each scale, the probability volume P sThe cross-entropy loss for each pixel x is calculated as follows:

[0089]

[0090] Where π1 represents the depth range estimated in the current stage, which can include the set of pixels corresponding to the actual pixels. The cross-entropy loss for all stages is:

[0091]

[0092] 2) Regression loss:

[0093] For each scale, the probability volume P s The regression loss for each pixel x is calculated as follows:

[0094]

[0095] Where π1 represents the set of pixels whose estimated depth range at the current stage does not include the actual pixel set, and D(x) represents the probability volume P along the direction of depth D. s The value after weighted summation.

[0096] The regression loss for all scales is:

[0097]

[0098] 3) CL Joint Loss

[0099] United L c and L r The joint loss of CL is obtained, and the calculation method is as follows:

[0100] CL=L c +λ·L r

[0101] Where λ represents the hyperparameter, set to 0.001.

[0102] 4. Fusion of training and confidence scores

[0103] 1) Training dataset

[0104] This invention uses the DTU dataset as the training dataset, which contains 128 scenes. This invention selects 79 scenes containing only 49 viewpoints from this dataset as the training set.

[0105] 2) Neighborhood average pooling

[0106] For the probability volume output P4 at scale s=4, this invention optimizes the probability volume in the depth dimension using neighborhood average pooling. Specifically, for each depth d... lThe probability of the corresponding pixel position x is summed with the probability values ​​of the previous two depths and the probability values ​​of the next two depths, and then divided by 5 to obtain the new probability of the current position. All elements of the probability volume P4 are subjected to neighborhood average pooling to obtain the pooled probability volume P.

[0107] Then, the depth value of each pixel is estimated based on the WTA strategy. That is, along the direction of the depth dimension, the depth value corresponding to the depth with the highest probability in the probability volume P is selected as the depth estimate D of each pixel, and the corresponding maximum probability is used as the confidence level C of each pixel.

[0108] 3) Confidence fusion

[0109] For the DTU dataset, this invention trains for a total of 16 epochs. Starting from the 11th training epoch, confidence and depth map fusion is performed. Using the depth estimates D and C output from the 11th training epoch as benchmarks, the depth estimates D and confidence C are iteratively updated in each training epoch. For each pixel x, if the confidence value of the current epoch is higher than the benchmark confidence value, the depth of the pixel corresponding to the benchmark depth estimate D is updated to the depth value output in the current epoch; otherwise, it remains unchanged. The specific update method is as follows:

[0110]

[0111]

[0112] Where P i P and D represent the confidence plot for the current round and the baseline confidence plot, respectively; i D and D represent the depth map and baseline depth map for the current round, respectively.

[0113] The updated baseline depth map in the last round is the depth map for the viewpoint where depth needs to be estimated.

[0114] 5. Experimental Comparison

[0115] Gipuma 0.283 0.873 0.578 Colmap 0.400 0.664 0.532 CVP-MVSNet 0.296 0.406 0.351 CasMVSNet 0.325 0.385 0.355 This invention 0.330 0.314 0.322

Claims

1. A multi-view depth estimation method based on confidence fusion, characterized in that... The steps are as follows: Step 1: Take photos of the same scene from N different perspectives to obtain N RGB images {I1, I2, ..., I...} n } Input these N images into the Colmap algorithm to obtain the camera intrinsic and extrinsic parameters corresponding to these N images, as well as the sparse point cloud of the entire scene, which contains K three-dimensional points; Choose any RGB image as a reference image and calculate the overall confidence value between the reference image and other viewpoint images pairwise. The confidence value of the j-th image compared to the reference image and the K sparse point clouds. Summing yields: The overall confidence value between the j-th image and the reference image. according to Select M images as source images in descending order of value; Step 2: Extract features at four scales from the reference image and the M source images using a 4-scale feature extraction network, resulting in a deep feature set F={F_reference_image ... R F1, ..., F M }; Each deep feature set contains deep features at four scales: The dimensions of the deep features at different scales satisfy: Where s represents the scale number, C s Indicates the number of channels at the current scale; Step 3: At scale 1, calculate the depth range of the scene based on the relative positions of the intrinsic and extrinsic parameters of the selected reference frame and the sparse point cloud generated by Colmap. ,in, and These represent the closest and furthest distances of all points in the sparse point cloud relative to the reference map, respectively. Then, The sample is divided into 16 equal intervals, and the midpoint of each interval is taken as the initial sampling depth value. In scales 2 to 4, the depth value generated at the previous scale is used as the intermediate value for sampling the depth range at the current scale. The depth range size at each scale satisfies the following: Similar to the generation method of scale 1, Divide the sample into 16 equal intervals, and take the midpoint of each interval as the depth value for the current scale. ; Step 4: For each scale s, take each deep feature at scale s from the M source images obtained in Step 2. The projection of the homography transformation matrix H onto the 16 depth samples obtained in step 3 yields the projected deep features. ; The H in, R and t represent the intrinsic parameter matrix, rotation matrix, and translation matrix of the reference graph, respectively; , and Let represent the intrinsic parameter matrix, rotation matrix, and translation matrix of the i-th source image, respectively; Step 5: Through the projection transformation in Step 4, the deep features of the M source images are projected at 16 sampling depths, resulting in the projected deep features. 'l' represents the depth sampling sequence number; at the same pixel location, the deep features projected from M source images and the deep features from the reference image are fused without biased variance to obtain the cost volume. The cost body is generated as follows: in, ; Step 6: Transfer the cost body Input is fed into a depth-biased cost volume regularization network, and output is a probability volume. Then, along the depth dimension, neighborhood average pooling is performed on the probability volume. That is, for each pixel position and each depth, the probability value is averaged with the two depth values ​​in the two directions before and after, for a total of five depth values, and this average is used as the new probability of the current pixel position and current depth. Finally, the depth values ​​represented by the 16 positions with the highest probabilities for each pixel are the estimated depth of the current pixel, and the highest probability is used as the confidence of the current estimate. Step 7: Repeat steps 3 to 6 from scale 1 to scale 4. The depth map output at scale 4 is the preliminary depth estimation map of the reference image with the same pixel size as the original. Step 8: Save the depth map output from training epochs 11 to 15. For the depth value of each pixel, starting from epoch 11, update the depth value corresponding to pixels whose probability value is less than the probability value estimated in the current epoch. The update process is as follows: Where e represents the current training round, and P and D represent the depth value and feasibility, respectively; the depth value after updating all rounds is the final depth value of the reference graph.

2. The multi-view depth estimation method based on confidence fusion according to claim 1, characterized in that: The confidence value : in, This represents the angle between the k-th point in the sparse point cloud and the reference image and image j; , , These represent experimental hyperparameters.

3. The multi-view depth estimation method based on confidence fusion according to claim 2, characterized in that: The experimental hyperparameters , , 4. The multi-view depth estimation method based on confidence fusion according to claim 1, characterized in that: The number of channels C at the current scale s In the experiment, scales 1 to 4 were set to 64, 32, 16 and 8 respectively.

5. A neural network system for the multi-view depth estimation method based on confidence fusion as described in any one of claims 1 to 4, characterized in that... It includes a 4-scale feature extraction network and a four-level depth sampling module; The output of the 4-scale feature extraction network is connected to the input of the initial depth sampling module and the third-level depth resampling module, respectively. The depth sampling module and the third-level depth resampling module are connected to the cost body regularization module via a feature fusion and cost body generation module, and then to a neighborhood average pooling module as the output. The output of each level's neighborhood average pooling module is connected to the feature fusion and cost body generation module at the next level's input. The four scale features of the 4-scale feature extraction network are input to the input of the initial depth sampling module and the third-level depth resampling module, respectively. The output of the third-level depth resampling module yields multi-view depth estimation data; where F... 1 ={F 1 R F 1 1, ..., F 1 M } Input to the initial depth sampling module, F 2 ={F 2 R F 2 1, ..., F 2 M The data input to the first-level depth resampling module is fused with the data from the initial depth sampling module, F. 3 ={F 3 R F 3 1, ..., F 3 M The data input to the second-level depth resampling module is fused with the output of the first-level depth sampling module. 4 ={F 4 R F 4 1, ..., F 4 M The data input to the third-level depth resampling module is fused with the output of the second-level depth sampling module.

6. The neural network system according to claim 5, characterized in that: The 4-scale feature extraction network is one or multiple 4-scale feature extraction networks in parallel, serving as a feature extraction network for the reference image and M source images in series or in parallel.

7. The neural network system according to claim 5, characterized in that: The four-scale feature extraction network uses the F4N feature extraction network.

8. The neural network system according to claim 5, characterized in that: The cost volume regularization network employs a deep paranoid cost volume 3D convolutional neural network.

9. The neural network system according to claim 5, characterized in that: The depth values ​​obtained by the initial depth sampling module, the second-level depth resampling module, the third-level depth resampling module, and the fourth-level depth resampling module are used as the feature mapping spatial positions of the four scales.

10. The neural network system according to claim 5, characterized in that: The reference depth map is generated by fusing depth estimation maps from multiple training rounds; the probability volume is obtained by averaging the probabilities of the original probability volume at the two positions before and after each pixel in the depth dimension, for a total of 5 positions.

Images