Multi-view three-dimensional reconstruction method

Abstract

The invention relates to the technical field of multi-view three-dimensional reconstruction, and discloses a multi-view three-dimensional reconstruction method, which comprises the following steps: constructing a multi-view image training set, and constructing a neural network which is mainly composed of a feature pyramid, a multi-scale feature fusion module, a cost volume construction module and an integrated depth estimation module, inputting the multiple view angle images of the same object in the training set under different illumination conditions into a neural network for multiple rounds of network training, and obtaining a neural network training model after the network training is finished; reconstructing the test image pair by using a neural network training model to obtain a three-dimensional point cloud of the test image pair; the method has the advantage that the accuracy and integrity of three-dimensional reconstruction can be effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of multi-view stereoscopic reconstruction, and in particular, relates to a multi-view stereoscopic three-dimensional reconstruction method. Background Art

[0002] Multi-view stereo (MVS) technology aims to restore the 3D structure of a scene through images from multiple perspectives. As a core technology in 3D reconstruction, MVS has been widely used in computer vision and image processing tasks such as virtual reality, autonomous driving, robot navigation, and digital modeling. In recent years, convolutional neural networks (CNNs) and deep learning technologies have promoted the advancement of MVS technology in various applications due to their powerful feature learning capabilities and accurate 3D reconstruction performance.

[0003] The goal of MVS technology is to restore the three-dimensional structure as accurately and completely as possible by synthesizing image data from different perspectives. The main challenge is how to accurately extract depth information from different perspectives. The reconstruction accuracy and completeness of MVS are particularly important, especially in complex scenes, environments with scarce textures or blurred edges. With the continuous development of deep learning methods, MVS technology has made significant progress in detail recovery and accurate reconstruction of complex structures.

[0004] At present, the MVS method based on deep learning can restore relatively detailed three-dimensional structures under high-quality images and sufficient viewing angles. However, how to maintain high accuracy and integrity of the reconstruction results in more complex and changing environments remains a key challenge in MVS research.

[0005] Currently, there are still several problems with multi-view stereo 3D reconstruction:

[0006] First, how to achieve a balance in multi-scale feature extraction and fusion? In multi-view stereo (MVS) technology, multi-scale feature extraction can effectively capture details of different scales and improve the accuracy of depth estimation. However, multi-scale extraction of features is usually accompanied by higher computational overhead, especially in the process of feature fusion, which may lead to information redundancy and affect computational efficiency. In order to reduce unnecessary computation while improving detection accuracy, reasonable strategies must be adopted to ensure that multi-scale features can be effectively exchanged and fused without increasing excessive computational burden. Therefore, how to design a suitable feature fusion method to balance detail recovery and computational cost has become the key to improving the accuracy and practicality of MVS.

[0007] Second, how to optimize the cost volume construction to improve accuracy without increasing the computational burden? The cost volume plays a vital role in MVS reconstruction and can improve the accuracy of depth estimation. However, how to fully consider the importance of each perspective and weight the cost volume of each perspective is a feasible method. However, excessive weighting may make the cost volume calculation too complicated, thereby increasing the computational overhead of the model. How to avoid overly complex calculation operations while ensuring the accuracy of the cost volume has become a difficult problem in designing an efficient MVS model. By reasonably selecting a weighting strategy and weighting only key feature areas, the accuracy of depth estimation can be effectively improved, while reducing invalid calculations and ensuring that the model can run stably in different environments.

[0008] Third, how to combine classification and regression tasks in depth estimation? The depth estimation method that combines classification and regression can improve the accuracy of MVS, but it also increases the complexity and computational burden of the model. How to avoid excessive complexity of the model while improving accuracy has become an urgent problem to be solved. By carefully designing the network architecture so that classification and regression tasks share important features, the advantages of both can be maximized while reducing redundant calculations and memory consumption. In this way, not only can the high accuracy of depth estimation be guaranteed, but also the low computational overhead can be maintained in practical applications to achieve efficient 3D reconstruction.

[0009] In view of this, the present invention is proposed. Summary of the invention

[0010] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0011] A multi-view stereoscopic 3D reconstruction method, step S1: construct a multi-view training set and build a neural network. The neural network is mainly composed of a feature pyramid, a multi-scale feature fusion module, a cost volume construction module, and an integrated depth estimation module; wherein the input end of the first layer of the feature pyramid first receives a plurality of color images of size 100 taken from multiple perspectives, the first color image is recorded as a reference image, and the images other than the first color image are recorded as source images; for the reference image, the feature map output by the output end of the first layer of the feature pyramid is recorded as FR1, the input end of the second layer of the feature pyramid receives FR1, the feature map output by the output end of the second layer of the feature pyramid is recorded as FR2, the input end of the third layer of the feature pyramid receives FR2, the feature map output by the output end of the third layer of the feature pyramid is recorded as FR3, the input end of the fourth layer of the feature pyramid receives FR3, and the feature map output by the output end of the fourth layer of the feature pyramid is recorded as FR4;

[0012] The size of FR1 is 8×H×W, and the size of FR2 is The size of FR3 is The size of FR4 is

[0013] The input end of the multi-scale feature enhancement module receives FR1, FR2, FR3 and FR4, and the feature graphs at the output end of the multi-scale feature enhancement module are F1, F2, F3 and F4, such as Figure 4 As shown; the size of F1 is 8×H×W, and the size of F2 is The size of F3 is The size of F4 is For the i-th source image, the input of the multi-scale feature enhancement module receives and The feature map at the output of the multi-scale feature enhancement module is and The size is 8×H×W, The size is The size is The size is

[0014]

[0015] Step S2: inputting several images of different perspectives in the training set into the neural network for multiple rounds of network training, and obtaining a neural network training model after the network training is completed;

[0016] Step S3: Use the neural network training model to predict the test image pair to obtain a depth map of the test image pair;

[0017] Step S2 and step S3 are performed in four stages:

[0018] In the first stage, the cost volume construction module includes perspective transformation, feature difference calculation module and weighted sum fusion module; for each view angle i, the input end of the perspective transformation receives the camera parameters C of the reference image view angle, the camera parameters C of the source image view angle i , reference image feature F4 and source image feature The transformation feature map output by the perspective transformation output end is The size is D is the number of assumed depths;

[0019] The input of the feature difference calculation module receives the reference cost volume RV4 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size is G is the number of groups; the input of the weighted and fusion module receives the intra-group correlation feature of each source view as The first-stage cost volume output at the output of the weighted and fusion module is V4, and the size of V4 is

[0020] In the first stage, the depth estimation module includes a 3D regularization module, a classification depth module, a regression depth module and a depth fusion module. The input end of the regularization module receives the first stage cost volume V4, and the first stage probability map output by the output end of the regularization module is P4. The size of P4 is The input end of the classification depth module receives the first stage probability map P4 and the reference image feature F4. The output end of the classification depth module outputs the classification depth map CD4. The size of CD4 is The input end of the regression depth module receives the first stage probability map P4, and the output end of the regression depth module outputs the regression depth map RD4, the size of RD4 is The input end of the deep fusion module receives the classification depth map CD4 and the regression depth map RD4. The output end of the deep fusion module outputs the first stage depth map D4, and the size of D4 is

[0021] In the second stage, for each view i, the perspective transform input receives the camera parameters C of the reference image view and the camera parameters C of the source image view i , reference image feature F3, source image feature The first stage depth map is D4, and the transformation feature map output at the perspective transformation output end is The size is The input of the feature difference calculation module receives the reference cost volume RV3 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size is The input of the weighted and fusion module receives the intra-group correlation feature of each source view as The second-stage cost volume output at the output of the weighted and fusion module is V3, and the size of V3 is

[0022] In the second stage, the input end of the regularization module receives the second stage cost volume V3, and the second stage probability map output by the output end of the regularization module is P3, and the size of P3 is The input end of the classification depth module receives the second stage probability map P3 and the reference image feature F3. The output end of the classification depth module outputs the classification depth map CD3. The size of CD3 is The input end of the regression depth module receives the second stage probability map P3, and the output end of the regression depth module outputs the regression depth map RD3, the size of RD3 is The input end of the deep fusion module receives the classification depth map CD3 and the regression depth map RD3. The output end of the deep fusion module outputs the second stage depth map D3, and the size of D3 is

[0023] In the third stage, for each view i, the input of the feature map alignment module receives the camera parameters C of the reference image view and the camera parameters C of the source image view i , reference image feature F2, source image feature The second stage depth map is D3, and the transformed feature map output at the output of the feature map alignment module is The size is The input of the feature difference calculation module receives the reference cost volume RV2 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size is The input of the weighted and fusion module receives the intra-group correlation feature of each source view as The third stage cost volume output at the output of the weighted and fusion module is V2, and the size of V2 is

[0024] In the third stage, the input end of the regularization module receives the third stage cost volume V2, and the third stage probability map output by the output end of the regularization module is P2, and the size of P2 is The input end of the classification depth module receives the third stage probability map P2 and the reference image feature F2. The output end of the classification depth module outputs the classification depth map CD2. The size of CD2 is The input end of the regression depth module receives the third stage probability map P2, and the output end of the regression depth module outputs the regression depth map RD2, the size of RD2 is The input end of the deep fusion module receives the classification depth map CD2 and the regression depth map RD2. The output end of the deep fusion module outputs the third stage depth map D2, and the size of D2 is

[0025] In the fourth stage, the cost volume construction module includes perspective transformation, feature difference calculation module and weighted sum fusion module; for each view angle i, the input end of the perspective transformation receives the camera parameters C of the reference image view angle, the camera parameters C of the source image view angle i , reference image feature F1, source image feature The depth map of the third stage is D2, and the transformation feature map output at the perspective transformation output end is The size of is 8×D×H×W; the input end of the feature difference calculation module receives the reference cost volume RV1 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size of is G×D×H×W; the input of the weighted and fusion module receives the intra-group correlation feature of each source view as Cor_F1 i , the fourth stage cost volume output at the output of the weighted and fusion module is V1, and the size of V1 is 8×D×H×W;

[0026] In the fourth stage, the input end of the regularization module receives the fourth stage cost volume V1, and the output end of the regularization module outputs the fourth stage probability map P1, and the size of P1 is D×H×W; the input end of the classification depth module receives the fourth stage probability map P1 and the reference image feature F1, and the output end of the classification depth module outputs the classification depth map CD1, and the size of CD1 is H×W; the input end of the regression depth module receives the fourth stage probability map P1, and the output end of the regression depth module outputs the regression depth map RD1, and the size of RD1 is H×W; the input end of the depth fusion module receives the classification depth map CD1 and the regression depth map RD1, and the output end of the depth fusion module outputs the fourth stage depth map D final , D final The size is H×W.

[0027] As a preferred implementation of the present invention, in a specific embodiment, the training set construction process is: select 128 scenes, images taken from 49 angles and 7 different lighting conditions as the training set. In this embodiment, for each scene, each time training, 5 images are selected from 343 images taken from 49 angles and 7 different lighting conditions for training. The image size is H×W, H=512, W=640.

[0028] As a preferred implementation of the present invention, in a specific embodiment, the process of obtaining the neural network training model is as follows: inputting images shot from multiple perspectives in the training set into the neural network for network training, calculating the loss function L before each round of network training to optimize the neural network, and obtaining the neural network training model after a total of 16 rounds of network training; wherein, in the network training, D G Denotes the label depth image, D k represents the predicted depth image obtained at the kth stage, λ k represents the weight of the loss in the kth stage, represents the cross entropy (CE) loss.

[0029] As a preferred implementation mode of the present invention, in a specific embodiment, a neural network training model is used to predict a test image, and the process of predicting a depth image is as follows: arbitrarily select seven images taken from seven perspectives as test images; then input the test image into the neural network training model to predict a depth image of the test image.

[0030] As a preferred embodiment of the present invention, the second, third and fourth layers of the feature pyramid have the same structure, but the input and output are different. In a specific embodiment, Figure 3 As shown, the first layer of the feature pyramid consists of two customized two-dimensional convolution blocks, and the second, third, and fourth layers of the feature pyramid consist of three customized two-dimensional convolution blocks. Each customized two-dimensional convolution block consists of a convolution layer, a Group Normalization layer, and a ReLU activation layer. For the first layer of the feature pyramid, the input end of the first convolution layer inputs the RGB image, the input end of the first GN layer receives the feature map output by the output end of the first convolution layer, the input end of the first ReLU activation layer receives the feature map output by the output end of the first GN layer, the input end of the second convolution layer inputs the feature map output by the output end of the first ReLU activation layer, the input end of the second GN layer receives the feature map output by the output end of the second convolution layer, the input end of the second ReLU activation layer receives the feature map output by the output end of the second GN layer, and the feature map output by the output end of the second ReLU activation layer is denoted as FR1. Among them, the first convolution layer is a 2D convolution layer, the input size of the first convolution layer is 3×H×W, the output size is 8×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the second convolution layer is a 2D convolution layer, the input size of the second convolution layer is 8×H×W, the output size is 8×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1. For the i-th layer of the feature pyramid, the input end of the (3i-3)th convolution layer is FR i-1, the input end of the (3i-3)th GN layer receives the feature map output by the output end of the (3i-3)th convolutional layer, the input end of the (3i-3)th ReLU activation layer receives the feature map output by the output end of the (3i-3)th GN layer, the input end of the (3i-2)th convolutional layer receives the feature map output by the output end of the (3i-3)th ReLU activation layer, the input end of the (3i-2)th GN layer receives the feature map output by the output end of the (3i-2)th convolutional layer, and the input end of the (3i-2)th ReLU activation layer receives the feature map output by the output end of the (3i-2)th The input end receives the feature map output by the output end of the (3i-2)th GN layer, the input end of the (3i-1)th convolutional layer receives the feature map output by the output end of the (3i-2)th ReLU activation layer, the input end of the (3i-1)th GN layer receives the feature map output by the output end of the (3i-1)th convolutional layer, the input end of the (3i-1)th ReLU activation layer receives the feature map output by the output end of the (3i-1)th GN layer, and the feature map output by the output end of the (3i-1)th ReLU activation layer is denoted as FR i Among them, the (3i-3)th convolution layer is a 2D convolution layer, and the input size of the (3i-3)th convolution layer is (8·2 i-1 )×H×W, the output size is (8·2 i )×H×W, the convolution kernel size is 5×5, the sliding step is 1, and the automatic padding is 1; the (3i-2)th convolution layer is a 2D convolution layer, and the input size of the (3i-2)th convolution layer is (8·2 i )×H×W, the output size is (8·2 i )×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the (3i-1)th convolution layer is a 2D convolution layer, and the input size of the (3i-1)th convolution layer is (8·2 i )×H×W, the output size is (8·2 i )×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1.

[0031] As a preferred implementation of the present invention, the multi-scale feature fusion module, in a specific embodiment, is as follows: Figure 4As shown in FIG. 1 , the multi-scale feature fusion module is mainly composed of the 12th to 30th convolutional layers, the 12th to 30th ReLU activation layers, the 12th to 30th GN layers, and the 1st to 4th average pooling layers; the feature map FR4 outputted from the output of the 11th ReLU activation layer is subjected to a bilinear 2-fold upsampling operation to obtain the feature map s1. The input of the 12th convolutional layer receives the feature map FR4 outputted from the output of the 11th ReLU activation layer, the input of the 12th GN layer receives the feature map outputted from the output of the 12th convolutional layer, the input of the 12th ReLU activation layer receives the feature map outputted from the output of the 12th GN layer, and the feature map outputted from the output of the 12th ReLU activation layer is denoted as FR 44 . The input end of the 13th convolutional layer receives the feature map FR3 output by the output end of the 10th ReLU activation layer, the input end of the 13th GN layer receives the feature map output by the output end of the 13th convolutional layer, the input end of the 13th ReLU activation layer receives the feature map output by the output end of the 13th GN layer, and the feature map output by the output end of the 13th ReLU activation layer is denoted as e1. The feature map s1 and the feature map e1 are element-wise added to obtain the feature map e1′. The input end of the 14th convolutional layer receives the feature map e1′, the input end of the 14th GN layer receives the feature map output by the output end of the 14th convolutional layer, the input end of the 14th ReLU activation layer receives the feature map output by the output end of the 14th GN layer, and the feature map output by the output end of the 14th ReLU activation layer is denoted as FR 3_3 . The feature map e1′ is subjected to a bilinear 2-fold upsampling operation to obtain the feature map s2. The input end of the 15th convolutional layer receives the feature map FR2 output by the output end of the 9th ReLU activation layer, the input end of the 15th GN layer receives the feature map output by the output end of the 15th convolutional layer, the input end of the 15th ReLU activation layer receives the feature map output by the output end of the 15th GN layer, and the feature map output by the output end of the 15th ReLU activation layer is denoted as e2. The feature map s2 and the feature map e2 are element-wise added to obtain the feature map e′2. The input end of the 14th convolutional layer receives the feature map e′2, the input end of the 16th GN layer receives the feature map output by the output end of the 16th convolutional layer, the input end of the 16th ReLU activation layer receives the feature map output by the output end of the 16th GN layer, and the feature map output by the output end of the 16th ReLU activation layer is denoted as FR 2_2. The feature map e′2 is subjected to a bilinear 2-fold upsampling operation to obtain the feature map s3. The input end of the 17th convolutional layer receives the feature map FR1 output by the output end of the 8th ReLU activation layer, the input end of the 17th GN layer receives the feature map output by the output end of the 17th convolutional layer, the input end of the 17th ReLU activation layer receives the feature map output by the output end of the 17th GN layer, and the feature map output by the output end of the 17th ReLU activation layer is denoted as e3. The feature map s3 and the feature map e3 are element-wise added to obtain the feature map e′3. The input end of the 18th convolutional layer receives the feature map e′3, the input end of the 18th GN layer receives the feature map output by the output end of the 18th convolutional layer, the input end of the 18th ReLU activation layer receives the feature map output by the output end of the 18th GN layer, and the feature map output by the output end of the 16th ReLU activation layer is denoted as FR 1_1 . For the feature map FR 1_1 The feature map FR is obtained by bilinear 2x downsampling, bilinear 4x downsampling, and bilinear 8x downsampling. 1_2 , FR 1_3 , FR 1_4 , for the feature map FR 2_2 Perform bilinear 2x upsampling, bilinear 2x downsampling, and bilinear 4x downsampling to obtain the feature map FR 2_1 , FR 2_3 , FR 2_4 , for the feature map FR 3_3 The feature map FR is obtained by performing bilinear 4-fold upsampling, bilinear 2-fold upsampling, and bilinear 2-fold downsampling respectively. 3_1 , FR 3_2 , FR 3_4 , for the feature map FR 4_4 The feature map FR is obtained by bilinear 2x downsampling, bilinear 4x downsampling, and bilinear 8x downsampling. 4_1 , FR 4_2 , FR 4_3 . For the feature map FR 1_1 , FR 2_1 , FR 3_1 , FR 4_1 Perform channel connection operation to obtain feature map FF1 i , for the feature map FR 1_2 , FR 2_2 , FR 3_2 , FR 4_2 Perform channel connection operation to obtain feature map FF2 i , for the feature map FR 1_3 , FR 2_3 , FR 3_3 , FR 4_3 Perform channel connection operation to obtain feature map FF3i , for the feature map FR 1_4 , FR 2_4 , FR 3_4 , FR 4_4 Perform channel connection operation to obtain feature map FF4 i The input of the first average pooling layer receives the feature map FF1 i , the input of the 19th convolution layer receives the feature map output by the output of the 1st average pooling layer, the input of the 19th GN layer receives the feature map output by the output of the 19th convolution layer, the input of the 19th ReLU activation layer receives the feature map output by the output of the 19th GN layer, the input of the 20th convolution layer receives the feature map output by the output of the 19th ReLU activation layer, the input of the 20th GN layer receives the feature map output by the output of the 20th convolution layer, the input of the 20th ReLU activation layer receives the feature map output by the output of the 20th GN layer, the input of the 21st convolution layer receives the feature map output by the output of the 20th ReLU activation layer, the input of the 21st GN layer receives the feature map output by the output of the 21st convolution layer, the input of the 21st ReLU activation layer receives the feature map output by the output of the 21st GN layer, and the feature map output by the output of the 21st ReLU activation layer is denoted as F1. The input of the 2nd average pooling layer receives the feature map FF2 i , the input of the 22nd convolutional layer receives the feature map output by the output of the 2nd average pooling layer, the input of the 22nd GN layer receives the feature map output by the output of the 22nd convolutional layer, the input of the 22nd ReLU activation layer receives the feature map output by the output of the 22nd GN layer, the input of the 23rd convolutional layer receives the feature map output by the output of the 22nd ReLU activation layer, the input of the 23rd GN layer receives the feature map output by the output of the 23rd convolutional layer, the input of the 23rd ReLU activation layer receives the feature map output by the output of the 23rd GN layer, the input of the 24th convolutional layer receives the feature map output by the output of the 23rd ReLU activation layer, the input of the 24th GN layer receives the feature map output by the output of the 24th convolutional layer, the input of the 24th ReLU activation layer receives the feature map output by the output of the 24th GN layer, and the feature map output by the output of the 24th ReLU activation layer is denoted as F2. The input of the 3rd average pooling layer receives the feature map FF3 i, the input of the 25th convolution layer receives the feature map output by the output of the 3rd average pooling layer, the input of the 25th GN layer receives the feature map output by the output of the 25th convolution layer, the input of the 25th ReLU activation layer receives the feature map output by the output of the 25th GN layer, the input of the 26th convolution layer receives the feature map output by the output of the 25th ReLU activation layer, the input of the 26th GN layer receives the feature map output by the output of the 26th convolution layer, the input of the 26th ReLU activation layer receives the feature map output by the output of the 26th GN layer, the input of the 27th convolution layer receives the feature map output by the output of the 26th ReLU activation layer, the input of the 27th GN layer receives the feature map output by the output of the 27th convolution layer, the input of the 27th ReLU activation layer receives the feature map output by the output of the 27th GN layer, and the feature map output by the output of the 27th ReLU activation layer is denoted as F3. The input of the 4th average pooling layer receives the feature map FF4 i , the input end of the 28th convolutional layer receives the feature map output by the output end of the 4th average pooling layer, the input end of the 28th GN layer receives the feature map output by the output end of the 28th convolutional layer, the input end of the 28th ReLU activation layer receives the feature map output by the output end of the 28th GN layer, the input end of the 29th convolutional layer receives the feature map output by the output end of the 28th ReLU activation layer, the input end of the 27th GN layer receives the feature map output by the output end of the 29th convolutional layer, the input end of the 29th ReLU activation layer receives the feature map output by the output end of the 29th GN layer, the input end of the 30th convolutional layer receives the feature map output by the output end of the 29th ReLU activation layer, the input end of the 30th GN layer receives the feature map output by the output end of the 30th convolutional layer, the input end of the 30th ReLU activation layer receives the feature map output by the output end of the 30th GN layer, and the feature map output by the output end of the 30th ReLU activation layer is denoted as F4. Among them, the 12th convolution layer is a 2D convolution layer, and the input size of the 12th convolution layer is The output size is The convolution kernel size is 1×1 and the automatic padding is 1; the 13th convolution layer is a 2D convolution layer, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 1×1 and the automatic padding is 1; the 14th convolution layer is a 2D convolution layer, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 3×3 and the automatic padding is 1; the 15th convolution layer is a 2D convolution layer, and the input size of the 15th convolution layer is The output size is The convolution kernel size is 1×1 and the automatic padding is 1; the 16th convolution layer is a 2D convolution layer, and the input size of the 16th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 17th convolution layer is a 2D convolution layer, the input size of the 17th convolution layer is 8×H×W, the output size is 64×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 18th convolution layer is a 2D convolution layer, the input size of the 18th convolution layer is 64×H×W, the output size is 8×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 19th convolution layer is a 2D convolution layer, the input size of the 19th convolution layer is 120× H×W, output size is 15×H×W, convolution kernel size is 1×1, and automatic padding is 0; the 20th convolution layer is a 2D convolution layer, the input size of the 20th convolution layer is 15×H×W, the output size is 120×H×W, the convolution kernel size is 1×1, and the automatic padding is 0; the 21st convolution layer is a 2D convolution layer, the input size of the 21st convolution layer is 120×H×W, the output size is 8×H×W, the convolution kernel size is 1×1, the sliding step is 1, and the automatic padding is 1; the 22nd convolution layer is a 2D convolution layer, the input size of the 22nd convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 23rd convolution layer is a 2D convolution layer, and the input size of the 23rd convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 24th convolution layer is a 2D convolution layer, and the input size of the 24th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic padding is 1; the 25th convolution layer is a 2D convolution layer, and the input size of the 25th convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 26th convolution layer is a 2D convolution layer, and the input size of the 26th convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 27th convolution layer is a 2D convolution layer, and the input size of the 27th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic padding is 1; the 28th convolution layer is a 2D convolution layer, and the input size of the 28th convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 29th convolution layer is a 2D convolution layer, and the input size of the 29th convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 30th convolution layer is a 2D convolution layer, and the input size of the 30th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 1; the feature extraction process of the source view and the reference view is the same, and the feature map is obtained through the feature pyramid and multi-scale feature fusion module. and The size is 8×H×W, The size is The size is The size is

[0032] As a preferred implementation of the present invention, the cost volume construction modules of the second, third and fourth stages are the same, but the input and output are different. The input of the cost volume construction module of the second, third and fourth stages is one more depth map obtained in the previous stage than the cost volume construction module of the first stage. In a specific embodiment, Figure 5 As shown, in the first stage, the input of the cost volume construction module inputs the feature map F4 of the reference view and the feature map F4 of the source view output by the multi-scale feature fusion module Given the assumed depth D, the feature map of the reference view adds a depth dimension to obtain the reference cost volume RV4, the size of RV4 is The feature map of the source image is transformed to the perspective of the reference image through perspective transformation. For each perspective, this process is repeated to align the feature map of the source image to the reference image space. The transformed feature map at the reference perspective can be obtained by calculating the pixel-level matching cost. The size is The channel grouping operation is performed on the reference cost volume RV4 to obtain GRV4, the size of which is Transformation feature map Perform channel grouping operation to obtain The size is Calculate the intra-group correlation to obtain the intra-group correlation characteristics The size is Intra-group correlation characteristics Perform Softmax operation to obtain the weight of the correlation feature within each group The final cost volume V4 of the first stage is obtained by weighting. In the second stage, the input end of the cost volume construction module inputs the feature map F3 of the reference view and the feature map F4 of the source view output by the multi-scale feature fusion module. There is also a depth map D4 obtained in the previous stage. The depth map in the previous stage is responsible for providing the initial depth hypothesis for the current stage. The feature map of the reference view adds a depth dimension to obtain the reference cost volume RV3, the size of RV3 is The feature map of the source image is transformed to the perspective of the reference image through perspective transformation. For each perspective, this process is repeated to align the feature map of the source image to the reference image space. The transformed feature map at the reference perspective can be obtained by calculating the pixel-level matching cost. The size is Refer to the cost volume RV3 to perform channel grouping operation to GRV3, the size of GRV3 is Transformation feature map SV3 i Perform channel grouping operation to obtain The size is Calculate the intra-group correlation to obtain the intra-group correlation characteristics The size is Intra-group correlation characteristics Perform Softmax operation to obtain the weight of the correlation feature within each group The final cost volume V3 of the second stage is obtained by weighting. In the third stage, the input end of the cost volume construction module inputs the feature map F2 of the reference view and the feature map of the source view output by the multi-scale feature fusion module There is also a depth map D3 obtained in the previous stage. The depth map in the previous stage is responsible for providing the initial depth hypothesis for the current stage. The feature map of the reference view adds a depth dimension to obtain the reference cost volume RV2, the size of RV2 is The feature map of the source image is transformed to the perspective of the reference image through perspective transformation. For each perspective, this process is repeated to align the feature map of the source image to the reference image space. The transformed feature map at the reference perspective can be obtained by calculating the pixel-level matching cost. The size is The channel grouping operation is performed on the reference cost volume RV2 to obtain GRV2, the size of which is Transform feature map Perform channel grouping operation to obtain The size is Calculate the intra-group correlation to obtain the intra-group correlation characteristics The size is Intra-group correlation characteristics Perform Softmax operation to obtain the weight of the correlation feature within each group The final cost volume V2 of the third stage is obtained by weighting. In the fourth stage, the input end of the cost volume construction module inputs the feature map F1 of the reference view and the feature map of the source view output by the multi-scale feature fusion module There is also a depth map D2 obtained in the previous stage. The depth map of the previous stage is responsible for providing the initial depth hypothesis for the current stage. The feature map of the reference view adds a depth dimension to obtain the reference cost volume RV1, and the size of RV1 is 8×D×H×W. The feature map of the source image is transformed to the perspective of the reference image through perspective transformation. For each perspective, this process is repeated to align the feature map of the source image to the reference image space. The transformed feature map at the reference perspective can be obtained by calculating the pixel-level matching cost. The size of is 8×D×H×W. The channel grouping operation is performed on the reference cost volume RV1 to obtain GRV1, and the size of GRV1 is Transformation feature map SV1 i Perform channel grouping operation to obtain The size is Calculate the intra-group correlation to obtain the intra-group correlation characteristics The size of is G×D×H×W. For the intra-group correlation feature Perform Softmax operation to obtain the weight of the correlation feature within each group The final cost volume V1 of the fourth stage is obtained by weighting.

[0033] As a preferred embodiment of the present invention, the depth estimation modules of the first, second, third and fourth stages are the same, but the input and output are different. Figure 6As shown in FIG. 1 , the integrated depth estimation module is mainly composed of the 31st to 37th convolutional layers, the 31st to 37th ReLU activation layers, the 31st to 37th GN layers, and the first Sigmoid activation layer. In the first stage, the cost volume V4 of the first stage is input into the input end of the cost volume construction module. The final cost volume V4 of the first stage passes through a 3D regularization network, and the output of the 3D regularization network is subjected to a Softmax operation to obtain a depth probability map P4. The regression depth map RD4 is obtained by a regression method, and the classification depth map CD4 is obtained by a classification method. The input end of the 31st convolutional layer receives the classified depth map CD4 obtained by the classification method, the input end of the 31st GN layer receives the feature map output by the output end of the 31st convolutional layer, the input end of the 31st ReLU activation layer receives the feature map output by the output end of the 31st GN layer, the input end of the 32nd convolutional layer receives the feature map output by the output end of the 31st ReLU activation layer, the input end of the 32nd GN layer receives the feature map output by the output end of the 32nd convolutional layer, the input end of the 32nd ReLU activation layer receives the feature map output by the output end of the 32nd GN layer, and the feature map output by the output end of the 32nd ReLU activation layer is recorded as CD′4. The input of the 33rd convolution layer receives the reference image, the input of the 33rd GN layer receives the feature map output by the output of the 33rd convolution layer, the input of the 33rd ReLU activation layer receives the feature map output by the output of the 33rd GN layer, the input of the 34th convolution layer receives the feature map output by the output of the 33rd ReLU activation layer, the input of the 34th GN layer receives the feature map output by the output of the 34th convolution layer, the input of the 34th ReLU activation layer receives the feature map output by the output of the 34th GN layer, and the feature map output by the output of the 34th ReLU activation layer is recorded as RGB′4. The feature map c1 is obtained by performing channel connection operation on the feature map and the feature map. The input end of the 35th convolutional layer receives the feature map c1, the input end of the 35th GN layer receives the feature map output by the output end of the 35th convolutional layer, the input end of the 35th ReLU activation layer receives the feature map output by the output end of the 35th GN layer, the input end of the 36th convolutional layer receives the feature map output by the output end of the 35th ReLU activation layer, the input end of the 36th GN layer receives the feature map output by the output end of the 36th convolutional layer, the input end of the 36th ReLU activation layer receives the feature map output by the output end of the 36th GN layer, the feature map output by the output end of the 36th ReLU activation layer and the feature map CD′4 are element-wise added to obtain the feature map c2, and the feature map c2 and the feature map RD4 are channel-wise connected to obtain the feature map c3.The input of the 37th convolution layer receives the feature map c3, the input of the 37th GN layer receives the feature map output by the output of the 37th convolution layer, the input of the 37th ReLU activation layer receives the feature map output by the output of the 37th GN layer, and the output of the 37th ReLU activation layer outputs the depth map weight. The feature map RD4 and the feature map c2 are weighted to obtain the depth map D4 of the first stage. The second, third, and fourth stages are operated in the same way to obtain the depth maps D3, D2, and D of the second, third, and fourth stages. final Among them, the 31st convolution layer is a 2D convolution layer, the input size of the 31st convolution layer is 1×H×W, the output size is 64×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 32nd convolution layer is a 2D convolution layer, the input size of the 32nd convolution layer is 64×H×W, the output size is 128×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 33rd convolution layer is a 2D convolution layer, the input size of the 33rd convolution layer is 3×H×W, the output size is 64×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 34th convolution layer is a 2D convolution layer, the input size of the 34th convolution layer is 64×H×W, output size is 128×H×W, convolution kernel size is 3×3, sliding step is 1, and automatic padding is 1; the 35th convolution layer is a 2D convolution layer, the input size of the 35th convolution layer is 256×H×W, the output size is 128×H×W, the convolution kernel size is 3×3, sliding step is 1, and automatic padding is 1; the 36th convolution layer is a 2D convolution layer, the input size of the 36th convolution layer is 128×H×W, the output size is 1×H×W, the convolution kernel size is 3×3, sliding step is 1, and automatic padding is 1; the 37th convolution layer is a 2D convolution layer, the input size of the 37th convolution layer is 2×H×W, the output size is 1×H×W, and the convolution kernel size is 1×1.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1) Based on the characteristics of multi-scale feature extraction and fusion, the method of the present invention designs a multi-level feature fusion module in the neural network. This module enables the network to capture richer and more diverse scene information by extracting features from multiple scales and fusing them. Through this gradual fusion approach, the network can not only make full use of detailed features at different scales, but also avoid information loss caused by a single scale, thereby improving the accuracy of depth estimation and the integrity of three-dimensional reconstruction. This method ensures that the network can retain more details when processing complex or diverse scenes, avoiding the common problem of reduced accuracy in traditional methods.

[0036] 2) The method of the present invention introduces a weighting strategy in the cost volume construction process, so that different areas are treated differently during the construction process. Through this weighted method, important areas in the cost volume can obtain more weights, making depth estimation more accurate, especially in areas with scarce textures or blurred boundaries. Compared with the traditional method of uniformly treating all pixels, the weighted cost volume method of the present invention can effectively reduce the interference of mismatches and noise, and improve the accuracy and stability of three-dimensional reconstruction. This method enables the model to better adapt to different feature areas in complex environments, ensuring the accuracy and completeness of the reconstruction results.

[0037] 3) The method of the present invention combines classification and regression tasks, and makes full use of classification tasks to assist regression prediction in the depth estimation process. Through this joint learning method, the classification task helps the model to more accurately locate different areas of depth information, while the regression task further refines the depth prediction, making the depth estimation more accurate and robust. Compared with the traditional single regression method, the method of the present invention can effectively suppress the influence of noise in complex scenes and improve the ability to restore details. Through the combination of classification and regression, the network can not only accurately predict the depth, but also maintain high stability and accuracy in changing scenes, significantly improving the quality of three-dimensional reconstruction and the ability to restore details.

[0038] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In the attached picture:

[0040] Figure 1 It is the overall implementation framework diagram of the method of the present invention;

[0041] Figure 2 A schematic diagram of the composition structure of a neural network constructed by the method of the present invention;

[0042] Figure 3 A schematic diagram of the composition structure of the feature pyramid in the neural network constructed by the method of the present invention;

[0043] Figure 4 A schematic diagram of the composition structure of a multi-scale feature fusion module in a neural network constructed by the method of the present invention;

[0044] Figure 5 A schematic diagram of the composition structure of the cost volume building module in the neural network constructed by the method of the present invention;

[0045] Figure 6 Schematic diagram of the structure of the integrated depth estimation module in the neural network constructed by the method of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention.

[0047] Embodiment 1:

[0048] A multi-view stereo 3D reconstruction method, the overall implementation framework is shown in the figure Figure 1 As shown in the figure, the method first constructs a multi-view training set and builds a neural network. Then, several images from different perspectives in the training set are input into the neural network for multiple rounds of network training. After the network training is completed, a neural network training model is obtained. The neural network training model is used again to predict the test image pair, and the depth map of the test image pair is obtained, as shown in FIG. Figure 2 As shown, the neural network is mainly composed of a feature pyramid, a multi-scale feature fusion module, a cost volume construction module, and an integrated depth estimation module, wherein:

[0049] The input end of the first layer of the feature pyramid first receives a plurality of color images of size H×W×3 taken from multiple perspectives, the first color image is recorded as a reference image, and the images other than the first image are recorded as source images, such as Figure 3 As shown. For the reference image, the feature map output by the output end of the first layer of the feature pyramid is denoted as FR1, the input end of the second layer of the feature pyramid receives FR1, the feature map output by the output end of the second layer of the feature pyramid is denoted as FR2, the input end of the third layer of the feature pyramid receives FR2, the feature map output by the output end of the third layer of the feature pyramid is denoted as FR3, the input end of the fourth layer of the feature pyramid receives FR3, and the feature map output by the output end of the fourth layer of the feature pyramid is denoted as FR4. The size of FR1 is 8×H×W, and the size of FR2 is The size of FR3 is The size of FR4 is In this embodiment, H×W is set to 512×640. For the i-th source image, the feature map outputted from the first layer output end of the feature pyramid is recorded as The input of the second layer of the feature pyramid receives FS i 1. The feature map output from the second output layer of the feature pyramid is recorded as The input of the third layer of the feature pyramid receives FS i 2. The feature map output by the third output terminal of the feature pyramid is recorded as The input of the fourth layer of the feature pyramid receives The feature map output at the fourth output end of the feature pyramid is recorded as

[0050] The input end of the multi-scale feature enhancement module receives FR1, FR2, FR3 and FR4, and the feature graphs at the output end of the multi-scale feature enhancement module are F1, F2, F3 and F4, such as Figure 4 As shown. The size of F1 is 8×H×W, and the size of F2 is The size of F3 is The size of F4 is For the i-th source image, the input of the multi-scale feature enhancement module receives and The feature map at the output of the multi-scale feature enhancement module is and The size is 8×H×W, The size is The size is The size is

[0051]

[0052] The next steps are divided into four phases:

[0053] In the first stage, the cost volume construction module includes perspective transformation, feature difference calculation module and weighted sum fusion module, such as Figure 5 For each view i, the input of the perspective transform receives the camera parameters C of the reference image view and the camera parameters C of the source image view. i , reference image feature F4 and source image feature The transformation feature map output by the perspective transformation output end is The size is D is the number of assumed depths. The input of the feature difference calculation module receives the reference cost volume RV4 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size is G is the number of groups. The input of the weighted and fusion module receives the intra-group correlation feature of each source view as The first-stage cost volume output at the output of the weighted and fusion module is V4, and the size of V4 is Among them, perspective transformation is an existing method, and its specific implementation process has been made public, as recorded in the document Y.Yao, Z.Luo, S.Li, T.Fang and L.Quan, “Mvsnet: Depthinference for unstructured multi-view stereo,” 2018 IEEE Conference on Europeanconference on computer vision (ECCV), pp.767-783, 2018.

[0054] In the first stage, the depth estimation module includes a 3D regularization module, a classification depth module, a regression depth module and a depth fusion module, such as Figure 6 As shown. The input end of the regularization module receives the first stage cost volume V4, and the first stage probability map output by the output end of the regularization module is P4. The size of P4 is The input end of the classification depth module receives the first stage probability map P4 and the reference image feature F4. The output end of the classification depth module outputs the classification depth map CD4. The size of CD4 is The input end of the regression depth module receives the first stage probability map P4, and the output end of the regression depth module outputs the regression depth map RD4, the size of RD4 is The input end of the deep fusion module receives the classification depth map CD4 and the regression depth map RD4. The output end of the deep fusion module outputs the first stage depth map D4, and the size of D4 is Among them, the 3D regularization module is an existing structural framework, and its network structure has been made public, as recorded in the document Y.Yao, Z.Luo, S.Li, T.Fang and L.Quan, “Mvsnet: Depth inference for unstructured multi-viewstereo,” 2018 IEEE Conference on European conference on computer vision (ECCV), pp.767-783, 2018.

[0055] In the second stage, the cost volume construction module includes perspective transformation, feature difference calculation module and weighted sum fusion module. For each view i, the input end of the perspective transformation receives the camera parameters C of the reference image view, the camera parameters C of the source image view i , reference image feature F3, source image feature The first stage depth map is D4, and the transformation feature map output at the perspective transformation output end is The size is The input of the feature difference calculation module receives the reference cost volume RV3 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size is The input of the weighted and fusion module receives the intra-group correlation feature of each source view as The second-stage cost volume output at the output of the weighted and fusion module is V3, and the size of V3 is Among them, this cascade structure has been made public, as recorded in the document X.Gu, Z.Fan, S.Zhu, Z.Dai, F.Tanand P.Tan, "Cascade cost volume for high-resolution multi-view stereo andstereo matching," 2020IEEE Conference on computer vision and pattern recognition (CVPR), pp.2495-2504, 2020.

[0056] In the second stage, the depth estimation module includes a 3D regularization module, a classification depth module, a regression depth module and a depth fusion module. The input end of the regularization module receives the second stage cost volume V3, and the second stage probability map output by the regularization module output end is P3, and the size of P3 is The input end of the classification depth module receives the second stage probability map P3 and the reference image feature F3. The output end of the classification depth module outputs the classification depth map CD3. The size of CD3 is The input end of the regression depth module receives the second stage probability map P3, and the output end of the regression depth module outputs the regression depth map RD3, the size of RD3 is The input end of the deep fusion module receives the classification depth map CD3 and the regression depth map RD3. The output end of the deep fusion module outputs the second stage depth map D3, and the size of D3 is

[0057] In the third stage, the cost volume construction module includes perspective transformation, feature difference calculation module and weighted sum fusion module. For each view i, the input end of the feature map alignment module receives the camera parameters C of the reference image view, the camera parameters C of the source image view i , reference image feature F2, source image feature The second stage depth map is D3, and the transformed feature map output at the output of the feature map alignment module is The size is The input of the feature difference calculation module receives the reference cost volume RV2 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size is The input of the weighted and fusion module receives the intra-group correlation feature of each source view as The third stage cost volume output at the output of the weighted and fusion module is V2, and the size of V2 is

[0058] In the third stage, the depth estimation module includes a 3D regularization module, a classification depth module, a regression depth module and a depth fusion module. The input end of the regularization module receives the third stage cost volume V2, and the output end of the regularization module outputs the third stage probability map P2, the size of P2 is The input end of the classification depth module receives the third stage probability map P2 and the reference image feature F2. The output end of the classification depth module outputs the classification depth map CD2. The size of CD2 is The input end of the regression depth module receives the third stage probability map P2, and the output end of the regression depth module outputs the regression depth map RD2, the size of RD2 is The input end of the deep fusion module receives the classification depth map CD2 and the regression depth map RD2. The output end of the deep fusion module outputs the third stage depth map D2, and the size of D2 is

[0059] In the fourth stage, the cost volume construction module includes perspective transformation, feature difference calculation module and weighted sum fusion module. For each view i, the input end of the perspective transformation receives the camera parameters C of the reference image view, the camera parameters C of the source image view i , reference image feature F1, source image feature The depth map of the third stage is D2, and the transformation feature map output at the perspective transformation output end is The size of is 8×D×H×W. The input of the feature difference calculation module receives the reference cost volume RV1 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size of is G×D×H×W. The input of the weighted and fusion module receives the intra-group correlation feature of each source view as The fourth-stage cost volume output at the output end of the weighted and fusion module is V1, and the size of V1 is 8×D×H×W.

[0060] In the fourth stage, the depth estimation module includes a 3D regularization module, a classification depth module, a regression depth module and a depth fusion module. The input end of the regularization module receives the fourth stage cost volume V1, and the output end of the regularization module outputs the fourth stage probability map P1, and the size of P1 is D×H×W. The input end of the classification depth module receives the fourth stage probability map P1 and the reference image feature F1, and the output end of the classification depth module outputs the classification depth map CD1, and the size of CD1 is H×W. The input end of the regression depth module receives the fourth stage probability map P1, and the output end of the regression depth module outputs the regression depth map RD1, and the size of RD1 is H×W. The input end of the depth fusion module receives the classification depth map CD1 and the regression depth map RD1, and the output end of the depth fusion module outputs the fourth stage depth map D final , D final The size is H×W.

[0061] Furthermore, in a specific embodiment, the training set is constructed by selecting 128 scenes, images taken from 49 angles and 7 different lighting conditions as the training set. In this embodiment, for each scene, 5 images are selected from 343 images taken from 49 angles and 7 different lighting conditions for each training. The image size is H×W, H=512, W=640.

[0062] Furthermore, in a specific embodiment, the process of obtaining the neural network training model is as follows: inputting images shot from multiple perspectives in the training set into the neural network for network training, calculating the loss function L before each round of network training to optimize the neural network, and obtaining the neural network training model after a total of 16 rounds of network training; wherein, in the network training D G Denotes the label depth image, D k represents the predicted depth image obtained at the kth stage, λ k represents the weight of the loss in the kth stage, represents the cross entropy (CE) loss.

[0063] Furthermore, in a specific embodiment, a neural network training model is used to predict a test image, and a process of predicting a depth image is as follows: arbitrarily selecting seven images taken from different perspectives as test images; then inputting the test image into the neural network training model to predict a depth image of the test image;

[0064] Furthermore, the second, third, and fourth layers of the feature pyramid have the same structure, but the input and output are different. In a specific embodiment, Figure 3As shown, the first layer of the feature pyramid consists of two customized two-dimensional convolution blocks, and the second, third, and fourth layers of the feature pyramid consist of three customized two-dimensional convolution blocks. Each customized two-dimensional convolution block consists of a convolution layer, a Group Normalization layer, and a ReLU activation layer. For the first layer of the feature pyramid, the input end of the first convolution layer inputs the RGB image, the input end of the first GN layer receives the feature map output by the output end of the first convolution layer, the input end of the first ReLU activation layer receives the feature map output by the output end of the first GN layer, the input end of the second convolution layer inputs the feature map output by the output end of the first ReLU activation layer, the input end of the second GN layer receives the feature map output by the output end of the second convolution layer, the input end of the second ReLU activation layer receives the feature map output by the output end of the second GN layer, and the feature map output by the output end of the second ReLU activation layer is denoted as FR1. Among them, the first convolution layer is a 2D convolution layer, the input size of the first convolution layer is 3×H×W, the output size is 8×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the second convolution layer is a 2D convolution layer, the input size of the second convolution layer is 8×H×W, the output size is 8×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1. For the i-th layer of the feature pyramid, the input end of the (3i-3)th convolution layer is FR i-1 , the input end of the (3i-3)th GN layer receives the feature map output by the output end of the (3i-3)th convolutional layer, the input end of the (3i-3)th ReLU activation layer receives the feature map output by the output end of the (3i-3)th GN layer, the input end of the (3i-2)th convolutional layer receives the feature map output by the output end of the (3i-3)th ReLU activation layer, the input end of the (3i-2)th GN layer receives the feature map output by the output end of the (3i-2)th convolutional layer, and the input end of the (3i-2)th ReLU activation layer receives the feature map output by the output end of the (3i-2)th The input end receives the feature map output by the output end of the (3i-2)th GN layer, the input end of the (3i-1)th convolutional layer receives the feature map output by the output end of the (3i-2)th ReLU activation layer, the input end of the (3i-1)th GN layer receives the feature map output by the output end of the (3i-1)th convolutional layer, the input end of the (3i-1)th ReLU activation layer receives the feature map output by the output end of the (3i-1)th GN layer, and the feature map output by the output end of the (3i-1)th ReLU activation layer is denoted as FR i Among them, the (3i-3)th convolution layer is a 2D convolution layer, and the input size of the (3i-3)th convolution layer is (8·2 i-1 )×H×W, the output size is (8·2 i)×H×W, the convolution kernel size is 5×5, the sliding step is 1, and the automatic padding is 1; the (3i-2)th convolution layer is a 2D convolution layer, and the input size of the (3i-2)th convolution layer is (8·2 i )×H×W, the output size is (8·2 i )×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the (3i-1)th convolution layer is a 2D convolution layer, and the input size of the (3i-1)th convolution layer is (8·2 i )×H×W, the output size is (8·2 i )×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1.

[0065] Here, Figure 3 The GN layer is the abbreviation of the Group Normalization layer.

[0066] Furthermore, in a specific embodiment, the multi-scale feature fusion module is as follows: Figure 4 As shown in FIG. 1 , the multi-scale feature fusion module is mainly composed of the 12th to 30th convolutional layers, the 12th to 30th ReLU activation layers, the 12th to 30th GN layers, and the 1st to 4th average pooling layers; the feature map FR4 outputted from the output of the 11th ReLU activation layer is subjected to a bilinear 2-fold upsampling operation to obtain the feature map s1. The input of the 12th convolutional layer receives the feature map FR4 outputted from the output of the 11th ReLU activation layer, the input of the 12th GN layer receives the feature map outputted from the output of the 12th convolutional layer, the input of the 12th ReLU activation layer receives the feature map outputted from the output of the 12th GN layer, and the feature map outputted from the output of the 12th ReLU activation layer is denoted as FR 4_4 . The input end of the 13th convolutional layer receives the feature map FR3 output by the output end of the 10th ReLU activation layer, the input end of the 13th GN layer receives the feature map output by the output end of the 13th convolutional layer, the input end of the 13th ReLU activation layer receives the feature map output by the output end of the 13th GN layer, and the feature map output by the output end of the 13th ReLU activation layer is denoted as e1. The feature map s1 and the feature map e1 are element-wise added to obtain the feature map e1′. The input end of the 14th convolutional layer receives the feature map e1′, the input end of the 14th GN layer receives the feature map output by the output end of the 14th convolutional layer, the input end of the 14th ReLU activation layer receives the feature map output by the output end of the 14th GN layer, and the feature map output by the output end of the 14th ReLU activation layer is denoted as FR 3_3. The feature map e1′ is subjected to a bilinear 2-fold upsampling operation to obtain the feature map s2. The input end of the 15th convolutional layer receives the feature map FR2 output by the output end of the 9th ReLU activation layer, the input end of the 15th GN layer receives the feature map output by the output end of the 15th convolutional layer, the input end of the 15th ReLU activation layer receives the feature map output by the output end of the 15th GN layer, and the feature map output by the output end of the 15th ReLU activation layer is denoted as e2. The feature map s2 and the feature map e2 are element-wise added to obtain the feature map e′2. The input end of the 14th convolutional layer receives the feature map e′2, the input end of the 16th GN layer receives the feature map output by the output end of the 16th convolutional layer, the input end of the 16th ReLU activation layer receives the feature map output by the output end of the 16th GN layer, and the feature map output by the output end of the 16th ReLU activation layer is denoted as FR 2_2 . Feature map s3 is obtained by performing a bilinear 2-fold upsampling operation on feature map e′2. The input end of the 17th convolutional layer receives the feature map FR1 output by the output end of the 8th ReLU activation layer, the input end of the 17th GN layer receives the feature map output by the output end of the 17th convolutional layer, the input end of the 17th ReLU activation layer receives the feature map output by the output end of the 17th GN layer, and the feature map output by the output end of the 17th ReLU activation layer is denoted as e3. Feature map e′3 is obtained by performing element-wise addition operation on feature map s3 and feature map e3. The input end of the 18th convolutional layer receives the feature map e′3, the input end of the 18th GN layer receives the feature map output by the output end of the 18th convolutional layer, the input end of the 18th ReLU activation layer receives the feature map output by the output end of the 18th GN layer, and the feature map output by the output end of the 16th ReLU activation layer is denoted as FR 1_1 . For the feature map FR 1_1 The feature map FR is obtained by performing bilinear 2x downsampling, bilinear 4x downsampling, and bilinear 8x downsampling respectively. 1_2 , FR 1_3 , FR 1_4 , for the feature map FR 2_2 Perform bilinear 2x upsampling, bilinear 2x downsampling, and bilinear 4x downsampling to obtain the feature map FR 2_1 , FR 2_3 , FR 2_4 , for the feature map FR 3_3 Perform bilinear 4x upsampling, bilinear 2x upsampling, and bilinear 2x downsampling to obtain the feature map FR 3_1 , FR 3_2 , FR 3_4 , for the feature map FR 4_4 The feature map FR is obtained by bilinear 2x downsampling, bilinear 4x downsampling, and bilinear 8x downsampling. 4_1 , FR4_2 , FR 4_3 . For the feature map FR 1_1 , FR 2_1 , FR 3_1 , FR 4_1 Perform channel connection operation to obtain feature map For feature map FR 1_2 , FR 2_2 , FR 3_2 , FR 4_2 Perform channel connection operation to obtain feature map For feature map FR 1_3 , FR 2_3 , FR 3_3 , FR 4_3 Perform channel connection operation to obtain feature map For feature map FR 1_4 , FR 2_4 , FR 3_4 , FR 4_4 Perform channel connection operation to obtain feature map The input of the first average pooling layer receives the feature map The input of the 19th convolution layer receives the feature map output by the output of the 1st average pooling layer, the input of the 19th GN layer receives the feature map output by the output of the 19th convolution layer, the input of the 19th ReLU activation layer receives the feature map output by the output of the 19th GN layer, the input of the 20th convolution layer receives the feature map output by the output of the 19th ReLU activation layer, the input of the 20th GN layer receives the feature map output by the output of the 20th convolution layer, the input of the 20th ReLU activation layer receives the feature map output by the output of the 20th GN layer, the input of the 21st convolution layer receives the feature map output by the output of the 20th ReLU activation layer, the input of the 21st GN layer receives the feature map output by the output of the 21st convolution layer, the input of the 21st ReLU activation layer receives the feature map output by the output of the 21st GN layer, and the feature map output by the output of the 21st ReLU activation layer is denoted as F1. The input of the 2nd average pooling layer receives the feature map The input of the 22nd convolutional layer receives the feature map output by the output of the 2nd average pooling layer, the input of the 22nd GN layer receives the feature map output by the output of the 22nd convolutional layer, the input of the 22nd ReLU activation layer receives the feature map output by the output of the 22nd GN layer, the input of the 23rd convolutional layer receives the feature map output by the output of the 22nd ReLU activation layer, the input of the 23rd GN layer receives the feature map output by the output of the 23rd convolutional layer, the input of the 23rd ReLU activation layer receives the feature map output by the output of the 23rd GN layer, the input of the 24th convolutional layer receives the feature map output by the output of the 23rd ReLU activation layer, the input of the 24th GN layer receives the feature map output by the output of the 24th convolutional layer, the input of the 24th ReLU activation layer receives the feature map output by the output of the 24th GN layer, and the feature map output by the output of the 24th ReLU activation layer is denoted as F2. The input of the 3rd average pooling layer receives the feature map FF3 i , the input of the 25th convolution layer receives the feature map output by the output of the 3rd average pooling layer, the input of the 25th GN layer receives the feature map output by the output of the 25th convolution layer, the input of the 25th ReLU activation layer receives the feature map output by the output of the 25th GN layer, the input of the 26th convolution layer receives the feature map output by the output of the 25th ReLU activation layer, the input of the 26th GN layer receives the feature map output by the output of the 26th convolution layer, the input of the 26th ReLU activation layer receives the feature map output by the output of the 26th GN layer, the input of the 27th convolution layer receives the feature map output by the output of the 26th ReLU activation layer, the input of the 27th GN layer receives the feature map output by the output of the 27th convolution layer, the input of the 27th ReLU activation layer receives the feature map output by the output of the 27th GN layer, and the feature map output by the output of the 27th ReLU activation layer is denoted as F3. The input of the 4th average pooling layer receives the feature map The input end of the 28th convolutional layer receives the feature map output by the output end of the 4th average pooling layer, the input end of the 28th GN layer receives the feature map output by the output end of the 28th convolutional layer, the input end of the 28th ReLU activation layer receives the feature map output by the output end of the 28th GN layer, the input end of the 29th convolutional layer receives the feature map output by the output end of the 28th ReLU activation layer, the input end of the 27th GN layer receives the feature map output by the output end of the 29th convolutional layer, the input end of the 29th ReLU activation layer receives the feature map output by the output end of the 29th GN layer, the input end of the 30th convolutional layer receives the feature map output by the output end of the 29th ReLU activation layer, the input end of the 30th GN layer receives the feature map output by the output end of the 30th convolutional layer, the input end of the 30th ReLU activation layer receives the feature map output by the output end of the 30th GN layer, and the feature map output by the output end of the 30th ReLU activation layer is denoted as F4. Among them, the 12th convolution layer is a 2D convolution layer, and the input size of the 12th convolution layer is The output size is The convolution kernel size is 1×1 and the automatic padding is 1; the 13th convolution layer is a 2D convolution layer, and the input size of the 13th convolution layer is The output size is The convolution kernel size is 1×1 and the automatic padding is 1; the 14th convolution layer is a 2D convolution layer, and the input size of the 14th convolution layer is The output size is The convolution kernel size is 3×3 and the automatic padding is 1; the 15th convolution layer is a 2D convolution layer, and the input size of the 15th convolution layer is The output size is The convolution kernel size is 1×1 and the automatic padding is 1; the 16th convolution layer is a 2D convolution layer, and the input size of the 16th convolution layer is The output size is The convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 17th convolution layer is a 2D convolution layer, the input size of the 17th convolution layer is 8×H×W, the output size is 64×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 18th convolution layer is a 2D convolution layer, the input size of the 18th convolution layer is 64×H×W, the output size is 8×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 19th convolution layer is a 2D convolution layer, the input size of the 19th convolution layer is 120× H×W, output size is 15×H×W, convolution kernel size is 1×1, and automatic padding is 0; the 20th convolution layer is a 2D convolution layer, the input size of the 20th convolution layer is 15×H×W, the output size is 120×H×W, the convolution kernel size is 1×1, and the automatic padding is 0; the 21st convolution layer is a 2D convolution layer, the input size of the 21st convolution layer is 120×H×W, the output size is 8×H×W, the convolution kernel size is 1×1, the sliding step is 1, and the automatic padding is 1; the 22nd convolution layer is a 2D convolution layer, the input size of the 22nd convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 23rd convolution layer is a 2D convolution layer, and the input size of the 23rd convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 24th convolution layer is a 2D convolution layer, and the input size of the 24th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic padding is 1; the 25th convolution layer is a 2D convolution layer, and the input size of the 25th convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 26th convolution layer is a 2D convolution layer, and the input size of the 26th convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 27th convolution layer is a 2D convolution layer, and the input size of the 27th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic padding is 1; the 28th convolution layer is a 2D convolution layer, and the input size of the 28th convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 29th convolution layer is a 2D convolution layer, and the input size of the 29th convolution layer is The output size is The convolution kernel size is 1×1 and automatically filled with 0; the 30th convolution layer is a 2D convolution layer, and the input size of the 30th convolution layer is The output size is The convolution kernel size is 1×1, the sliding step is 1, and the automatic filling is 1; the feature extraction process of the source view and the reference view is the same, and the feature map is obtained through the feature pyramid and multi-scale feature fusion module. and The size is 8×H×W, The size is The size is The size is

[0067] Here, element addition, bilinear sampling, and channel concatenation are all common operations in neural networks. Figure 4 The GN layer is the abbreviation of the Group Normalization layer.

[0068] Furthermore, the cost volume construction modules of the second, third, and fourth stages are the same, but the input and output are different. The input of the cost volume construction module of the second, third, and fourth stages is one more depth map obtained in the previous stage than the cost volume construction module of the first stage. In a specific embodiment, Figure 5 As shown, in the first stage, the input of the cost volume construction module inputs the feature map F4 of the reference view and the feature map F4 of the source view output by the multi-scale feature fusion module Given the assumed depth D, the feature map of the reference view adds a depth dimension to obtain the reference cost volume RV4, the size of RV4 is The feature map of the source image is transformed to the perspective of the reference image through perspective transformation. For each perspective, this process is repeated to align the feature map of the source image to the reference image space. The transformed feature map at the reference perspective can be obtained by calculating the pixel-level matching cost. The size is The channel grouping operation is performed on the reference cost volume RV4 to obtain GRV4, the size of which is Transformation feature map Perform channel grouping operation to obtain The size is Calculate the intra-group correlation to obtain the intra-group correlation characteristics The size is Intra-group correlation characteristics Perform Softmax operation to obtain the weight of the correlation feature within each group The final cost volume V4 of the first stage is obtained by weighting. In the second stage, the input end of the cost volume construction module inputs the feature map F3 of the reference view and the feature map F4 of the source view output by the multi-scale feature fusion module. There is also a depth map D4 obtained in the previous stage. The depth map in the previous stage is responsible for providing the initial depth hypothesis for the current stage. The feature map of the reference view adds a depth dimension to obtain the reference cost volume RV3, the size of RV3 is The feature map of the source image is transformed to the perspective of the reference image through perspective transformation. For each perspective, this process is repeated to align the feature map of the source image to the reference image space. The transformed feature map at the reference perspective can be obtained by calculating the pixel-level matching cost. The size is Refer to the cost volume RV3 to perform channel grouping operation to GRV3, the size of GRV3 is Transformation feature map SV3 i Perform channel grouping operation to obtain The size is Calculate the intra-group correlation to obtain the intra-group correlation feature Cor_F3 i ,Cor_F3 i The size is Intra-group correlation characteristics Perform Softmax operation to obtain the weight of the correlation feature within each group The final cost volume V3 of the second stage is obtained by weighting. In the third stage, the input end of the cost volume construction module inputs the feature map F2 of the reference view and the feature map of the source view output by the multi-scale feature fusion module There is also a depth map D3 obtained in the previous stage. The depth map in the previous stage is responsible for providing the initial depth hypothesis for the current stage. The feature map of the reference view adds a depth dimension to obtain the reference cost volume RV2, the size of RV2 is The feature map of the source image is transformed to the perspective of the reference image through perspective transformation. For each perspective, this process is repeated to align the feature map of the source image to the reference image space. The transformed feature map at the reference perspective can be obtained by calculating the pixel-level matching cost. The size is The channel grouping operation is performed on the reference cost volume RV2 to obtain GRV2, the size of which is Transformed feature map SV2 i Perform channel grouping operation to obtain The size is Calculate the intra-group correlation to obtain the intra-group correlation characteristics The size is Intra-group correlation characteristics Perform Softmax operation to obtain the weight of the correlation feature within each group The final cost volume V2 of the third stage is obtained by weighting. In the fourth stage, the input end of the cost volume construction module inputs the feature map F1 of the reference view and the feature map of the source view output by the multi-scale feature fusion module There is also a depth map D2 obtained in the previous stage. The depth map of the previous stage is responsible for providing the initial depth hypothesis for the current stage. The feature map of the reference view adds a depth dimension to obtain the reference cost volume RV1, and the size of RV1 is 8×D×H×W. The feature map of the source image is transformed to the perspective of the reference image through perspective transformation. For each perspective, this process is repeated to align the feature map of the source image to the reference image space. The transformed feature map at the reference perspective can be obtained by calculating the pixel-level matching cost. The size of is 8×D×H×W. The channel grouping operation is performed on the reference cost volume RV1 to obtain GRV1, and the size of GRV1 is Transform feature map Perform channel grouping operation to obtain The size is Calculate the intra-group correlation to obtain the intra-group correlation characteristics The size of is G×D×H×W. For the intra-group correlation feature Perform Softmax operation to obtain the weight of the correlation feature within each group The final cost volume V1 of the fourth stage is obtained by weighting.

[0069] Here, channel grouping operation and Softmax operation are both common operations in neural networks; Figure 5 The GN layer is the abbreviation of the Group Normalization layer.

[0070] Furthermore, the depth estimation modules of the first, second, third and fourth stages are the same, but the input and output are different. Figure 6As shown in FIG. 1 , the integrated depth estimation module is mainly composed of the 31st to 37th convolutional layers, the 31st to 37th ReLU activation layers, the 31st to 37th GN layers, and the first Sigmoid activation layer. In the first stage, the cost volume V4 of the first stage is input into the input end of the cost volume construction module. The final cost volume V4 of the first stage passes through a 3D regularization network, and the output of the 3D regularization network is subjected to a Softmax operation to obtain a depth probability map P4. The regression depth map RD4 is obtained by a regression method, and the classification depth map CD4 is obtained by a classification method. The input end of the 31st convolutional layer receives the classified depth map CD4 obtained by the classification method, the input end of the 31st GN layer receives the feature map output by the output end of the 31st convolutional layer, the input end of the 31st ReLU activation layer receives the feature map output by the output end of the 31st GN layer, the input end of the 32nd convolutional layer receives the feature map output by the output end of the 31st ReLU activation layer, the input end of the 32nd GN layer receives the feature map output by the output end of the 32nd convolutional layer, the input end of the 32nd ReLU activation layer receives the feature map output by the output end of the 32nd GN layer, and the feature map output by the output end of the 32nd ReLU activation layer is recorded as CD′4. The input of the 33rd convolution layer receives the reference image, the input of the 33rd GN layer receives the feature map output by the output of the 33rd convolution layer, the input of the 33rd ReLU activation layer receives the feature map output by the output of the 33rd GN layer, the input of the 34th convolution layer receives the feature map output by the output of the 33rd ReLU activation layer, the input of the 34th GN layer receives the feature map output by the output of the 34th convolution layer, the input of the 34th ReLU activation layer receives the feature map output by the output of the 34th GN layer, and the feature map output by the output of the 34th ReLU activation layer is recorded as RGB′4. The feature map c1 is obtained by performing channel connection operation on the feature map and the feature map. The input end of the 35th convolutional layer receives the feature map c1, the input end of the 35th GN layer receives the feature map output by the output end of the 35th convolutional layer, the input end of the 35th ReLU activation layer receives the feature map output by the output end of the 35th GN layer, the input end of the 36th convolutional layer receives the feature map output by the output end of the 35th ReLU activation layer, the input end of the 36th GN layer receives the feature map output by the output end of the 36th convolutional layer, the input end of the 36th ReLU activation layer receives the feature map output by the output end of the 36th GN layer, the feature map output by the output end of the 36th ReLU activation layer and the feature map CD′4 are element-wise added to obtain the feature map c2, and the feature map c2 and the feature map RD4 are channel-wise connected to obtain the feature map c3.The input of the 37th convolution layer receives the feature map c3, the input of the 37th GN layer receives the feature map output by the output of the 37th convolution layer, the input of the 37th ReLU activation layer receives the feature map output by the output of the 37th GN layer, and the output of the 37th ReLU activation layer outputs the depth map weight. The feature map RD4 and the feature map c2 are weighted to obtain the depth map D4 of the first stage. The second, third, and fourth stages are operated in the same way to obtain the depth maps D3, D2, and D of the second, third, and fourth stages. final Among them, the 31st convolution layer is a 2D convolution layer, the input size of the 31st convolution layer is 1×H×W, the output size is 64×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 32nd convolution layer is a 2D convolution layer, the input size of the 32nd convolution layer is 64×H×W, the output size is 128×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 33rd convolution layer is a 2D convolution layer, the input size of the 33rd convolution layer is 3×H×W, the output size is 64×H×W, the convolution kernel size is 3×3, the sliding step is 1, and the automatic padding is 1; the 34th convolution layer is a 2D convolution layer, the input size of the 34th convolution layer is 64×H×W, output size is 128×H×W, convolution kernel size is 3×3, sliding step is 1, and automatic padding is 1; the 35th convolution layer is a 2D convolution layer, the input size of the 35th convolution layer is 256×H×W, the output size is 128×H×W, the convolution kernel size is 3×3, sliding step is 1, and automatic padding is 1; the 36th convolution layer is a 2D convolution layer, the input size of the 36th convolution layer is 128×H×W, the output size is 1×H×W, the convolution kernel size is 3×3, sliding step is 1, and automatic padding is 1; the 37th convolution layer is a 2D convolution layer, the input size of the 37th convolution layer is 2×H×W, the output size is 1×H×W, and the convolution kernel size is 1×1.

[0071] Experimental example:

[0072] Here, the Softmax operation, channel connection operation, and element addition operation are all common operations in neural networks; Figure 6 The GN layer is the abbreviation of the Group Normalization layer.

[0073] In order to further illustrate the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention.

[0074] In this embodiment, the method of the present invention is used to test a common data set (DTU) for 3D reconstruction and point cloud processing established by the Technical University of Denmark. The training set in DTU includes 79 scenes, the validation set in DTU includes 79 scenes, and the test set in DTU includes 79 scenes.

[0075] In this embodiment, three commonly used objective parameters are selected to evaluate the performance of the method of the present invention, namely Accuracy, Completeness, and Overall. Accuracy measures the degree of proximity between the reconstructed point cloud and the reference point cloud, reflecting the accuracy of the reconstructed model. The smaller the value of Accuracy, the higher the accuracy of the reconstructed model. Completeness measures the coverage of the reconstructed point cloud, that is, to what extent the model captures the surface of the entire scene. The smaller the value of Completeness, the higher the integrity of the reconstructed model. Overall is a comprehensive evaluation of Accuracy and Completeness, usually obtained by the weighted average of these two indicators.

[0076] Table 1 Accuracy, Completeness, and Overall between the point cloud obtained on the DTU dataset using the method of the present invention and the labeled point cloud.

[0077]

[0078]

[0079] From the results given in Table 1, it can be found that the method of the present invention has achieved lower Accuracy, Completeness and Overall on the existing multi-view stereo 3D reconstruction dataset, which means that the point cloud reconstructed by the method of the present invention is relatively close to the label point cloud, and the method of the present invention can effectively realize multi-view stereo 3D reconstruction.

Claims

1. A multi-view stereoscopic 3D reconstruction method, characterized in that: The steps include: Step S1: construct a multi-view training set and build a neural network. The neural network is mainly composed of a feature pyramid, a multi-scale feature fusion module, a cost volume construction module, and an integrated depth estimation module; wherein the input end of the first layer of the feature pyramid first receives a plurality of color images of size 100 taken from multiple perspectives, the first color image is recorded as a reference image, and the images other than the first image are recorded as source images; for the reference image, the feature map output by the output end of the first layer of the feature pyramid is recorded as FR1, the input end of the second layer of the feature pyramid receives FR1, the feature map output by the output end of the second layer of the feature pyramid is recorded as FR2, the input end of the third layer of the feature pyramid receives FR2, the feature map output by the output end of the third layer of the feature pyramid is recorded as FR3, the input end of the fourth layer of the feature pyramid receives FR3, and the feature map output by the output end of the fourth layer of the feature pyramid is recorded as FR4; The size of FR1 is 8×H×W, the size of FR2 is, and the size of FR3 is The size of FR4 is The input end of the multi-scale feature enhancement module receives FR1, FR2, FR3 and FR4, and the feature maps at the output end of the multi-scale feature enhancement module are F1, F2, F3 and F4, as shown in FIG4 ; the size of F1 is 8×H×W, and the size of F2 is The size of F3 is The size of F4 is For the i-th source image, the input of the multi-scale feature enhancement module receives and The feature map at the output of the multi-scale feature enhancement module is and The size is 8×H×W, The size is The size is The size is Step S2: inputting several images of different perspectives in the training set into the neural network for multiple rounds of network training, and obtaining a neural network training model after the network training is completed; Step S3: Use the neural network training model to predict the test image pair to obtain a depth map of the test image pair; Step S2 and step S3 are performed in four stages: In the first stage, the cost volume construction module includes perspective transformation, feature difference calculation module and weighted sum fusion module; for each view angle i, the input end of the perspective transformation receives the camera parameters C of the reference image view angle, the camera parameters C of the source image view angle i , reference image feature F4 and source image feature The transformation feature map output by the perspective transformation output end is The size is D is the number of assumed depths; The input of the feature difference calculation module receives the reference cost volume RV4 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size is G is the number of groups; the input of the weighted and fusion module receives the intra-group correlation feature of each source view as The first-stage cost volume output at the output of the weighted and fusion module is V4, and the size of V4 is In the first stage, the depth estimation module includes a 3D regularization module, a classification depth module, a regression depth module and a depth fusion module. The input end of the regularization module receives the first stage cost volume V4, and the first stage probability map output by the output end of the regularization module is P4. The size of P4 is The input end of the classification depth module receives the first stage probability map P4 and the reference image feature F4. The output end of the classification depth module outputs the classification depth map CD4. The size of CD4 is The input end of the regression depth module receives the first stage probability map P4, and the output end of the regression depth module outputs the regression depth map RD4, the size of RD4 is The input end of the deep fusion module receives the classification depth map CD4 and the regression depth map RD4. The output end of the deep fusion module outputs the first stage depth map D4, and the size of D4 is In the second stage, for each view i, the perspective transform input receives the camera parameters C of the reference image view and the camera parameters C of the source image view i , reference image feature F3, source image feature The first stage depth map is D4, and the transformation feature map output at the perspective transformation output end is The size is The input of the feature difference calculation module receives the reference cost volume RV3 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size is The input of the weighted and fusion module receives the intra-group correlation feature of each source view as The second-stage cost volume output at the output of the weighted and fusion module is V3, and the size of V3 is In the second stage, the input end of the regularization module receives the second stage cost volume V3, and the second stage probability map output by the output end of the regularization module is P3, and the size of P3 is The input end of the classification depth module receives the second stage probability map P3 and the reference image feature F3. The output end of the classification depth module outputs the classification depth map CD3. The size of CD3 is The input end of the regression depth module receives the second stage probability map P3, and the output end of the regression depth module outputs the regression depth map RD3, the size of RD3 is The input end of the deep fusion module receives the classification depth map CD3 and the regression depth map RD3. The output end of the deep fusion module outputs the second stage depth map D3, and the size of D3 is In the third stage, for each view i, the input of the feature map alignment module receives the camera parameters C of the reference image view and the camera parameters C of the source image view i , reference image feature F2, source image feature The second stage depth map is D3, and the transformed feature map output at the output of the feature map alignment module is The size is The input of the feature difference calculation module receives the reference cost volume RV2 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size is The input of the weighted and fusion module receives the intra-group correlation feature of each source view as The third stage cost volume output at the output of the weighted and fusion module is V2, and the size of V2 is In the third stage, the input end of the regularization module receives the third stage cost volume V2, and the third stage probability map output by the output end of the regularization module is P2, and the size of P2 is The input end of the classification depth module receives the third stage probability map P2 and the reference image feature F2. The output end of the classification depth module outputs the classification depth map CD2. The size of CD2 is The input end of the regression depth module receives the third stage probability map P2, and the output end of the regression depth module outputs the regression depth map RD2, the size of RD2 is The input end of the deep fusion module receives the classification depth map CD2 and the regression depth map RD2. The output end of the deep fusion module outputs the third stage depth map D2, and the size of D2 is In the fourth stage, the cost volume construction module includes perspective transformation, feature difference calculation module and weighted sum fusion module; for each view angle i, the input end of the perspective transformation receives the camera parameters C of the reference image view angle, the camera parameters C of the source image view angle i , reference image feature F1, source image feature The depth map of the third stage is D2, and the transformation feature map output at the perspective transformation output end is The size of is 8×D×H×W; the input end of the feature difference calculation module receives the reference cost volume RV1 and the transformed feature map The intra-group correlation feature output at the output end of the feature difference calculation module is The size of is G×D×H×W; the input of the weighted and fusion module receives the intra-group correlation feature of each source view as The fourth stage cost volume output at the output of the weighted and fusion module is V1, and the size of V1 is 8×D×H×W; In the fourth stage, the input end of the regularization module receives the fourth stage cost volume V1, and the output end of the regularization module outputs the fourth stage probability map P1, and the size of P1 is D×H×W; the input end of the classification depth module receives the fourth stage probability map P1 and the reference image feature F1, and the output end of the classification depth module outputs the classification depth map CD1, and the size of CD1 is H×W; the input end of the regression depth module receives the fourth stage probability map P1, and the output end of the regression depth module outputs the regression depth map RD1, and the size of RD1 is H×W; the input end of the depth fusion module receives the classification depth map CD1 and the regression depth map RD1, and the output end of the depth fusion module outputs the fourth stage depth map D final , D final The size is H×W.

2. A multi-view stereoscopic 3D reconstruction method according to claim 1, characterized in that: The process of constructing the training set is as follows: 128 scenes, images shot from 49 angles and under 7 different lighting conditions are selected as the training set.

3. The multi-view stereoscopic 3D reconstruction method according to claim 2, characterized in that: The process of obtaining the neural network training model is as follows: input the images taken from multiple perspectives in the training set into the neural network for network training, calculate the loss function L before the end of each round of network training to optimize the neural network, and obtain the neural network training model after a total of 16 rounds of network training; D G Denotes the label depth image, D k represents the predicted depth image obtained at the kth stage, λ k represents the weight of the loss in the kth stage, represents the cross entropy (CE) loss.

4. The multi-view stereoscopic 3D reconstruction method according to claim 3, characterized in that: The neural network training model is used to predict the test image. The process of predicting the depth image is as follows: randomly select seven images taken from seven perspectives as test images; then input the test image into the neural network training model to predict the depth image of the test image.

5. The multi-view stereoscopic 3D reconstruction method according to claim 1, characterized in that: The second, third, and fourth layers of the feature pyramid have the same structure, but different inputs and outputs. The first layer of the feature pyramid consists of two custom two-dimensional convolution blocks, and the second, third, and fourth layers of the feature pyramid consist of three custom two-dimensional convolution blocks. Each custom two-dimensional convolution block consists of a convolution layer, a Group Normalization layer, and a ReLU activation layer.

6. The multi-view stereoscopic 3D reconstruction method according to claim 4, characterized in that: The multi-scale feature fusion module is mainly composed of the 12th to 30th convolutional layers, the 12th to 30th ReLU activation layers, the 12th to 30th GN layers, and the 1st to 4th average pooling layers; the feature map FR4 output at the output end of the 11th ReLU activation layer is bilinearly up-sampled by 2 times to obtain the feature map s1.

Images