Multi-view stereo matching reconstruction method and system based on perceptual consistency loss

Abstract

A multi-view stereo matching reconstruction method and system based on perceptual consistency loss belongs to the field of three-dimensional reconstruction. In order to solve the problems of serious memory occupation and low integrity in the prior art, the method comprises the following steps: 1. constructing a network model: the entire network includes a feature extraction module, a pyramid volume regularization module and a dual correction layer module; 2. preparing a data set: taking a source image and a reference image as data input, and using the original 3D points from the reference view to estimate a dense depth map D of the three-dimensional structure; 3. training the network model: inputting the data set prepared in step 2 into the network model constructed in step 1 for training; 4. designing a loss function and performing regression; 5. saving the model and testing it; 6. point cloud fusion and reconstruction: fusing the three-dimensional point cloud obtained in step 5, and then using MeshLab software to view the matching reconstruction effect after fusion, and selecting the optimal evaluation index to measure the accuracy of the algorithm and the performance of the system.

Description

Technical Field

[0001] The invention relates to a multi-view stereo matching reconstruction method and system based on perceptual consistency loss, belonging to the technical field of three-dimensional reconstruction. Background Art

[0002] Three-dimensional reconstruction is widely used in industrial measurement, intelligent robots, unmanned driving systems, medical diagnosis, digital city modeling, somatosensory entertainment, etc. Multi-view stereo matching is an end-to-end learning method belonging to three-dimensional reconstruction technology. This method belongs to passive three-dimensional reconstruction and has the characteristics of low cost, simple structure and good practicality. Since each stereo matching method has its own limitations, both in traditional methods and learning methods, the integrity of the matching and the lightweight of the network have high requirements. Therefore, in order to achieve the effect of subsequent point cloud fusion, it is very necessary to optimize the matching depth map. In order to obtain effective depth information in better dense reconstruction, most of the existing multi-view stereo matching reconstruction methods are based on pixels. However, there are two key problems in the reconstruction method, namely, the long operation time cannot effectively achieve lightweight and high integrity while ensuring accuracy.

[0003] The Chinese patent publication number is "CN113963117A", and the name is "A multi-viewing algorithm based on variable convolutional depth network Figure 3 The method first inputs the source image and reference images of multiple perspectives; then, the features of the input image are extracted through a multi-scale feature network constructed by deformable convolution; then, a patch matching iterative model based on learning is used to perform iterative optimization calculations of pixel depth matching and edge processing to obtain an iteratively optimized depth map; finally, the iteratively optimized depth map and the source image are imported into a deep residual network for optimization to obtain the final depth map and perform three-dimensional reconstruction to obtain a stereoscopic visual map. This method uses a supervised learning method, which has a serious memory usage and low integrity. Summary of the invention

[0004] In order to solve the problems of serious memory usage and low integrity in the existing multi-view stereo matching methods, the present invention provides a multi-view stereo matching reconstruction method based on perceptual consistency loss. The obtained depth map is smoother and more complete, has a better reconstruction effect for subsequent point cloud fusion, and is more consistent with the detailed observation of the object. At the same time, the present invention adopts a semi-supervised learning method, which is simple to implement, takes less processing time, and has higher accuracy after reconstruction.

[0005] The solution to the technical problem of the present invention is:

[0006] The multi-view stereo matching reconstruction method based on perceptual consistency loss includes the following steps:

[0007] Step 1, build a network model: the entire network includes a feature extraction module, a pyramid volume regularization module and a dual correction layer module, and each module consists of a convolutional layer, regularization and activation function; the feature extraction module extracts features from the input source image and the reference image, and the pyramid volume regularization module performs downsampling extraction and upsampling integration regularization on the extracted features to construct a 3D cost volume to obtain a dense depth map; finally, the dual correction layer module performs a simple filtering on the obtained depth map to remove redundant information, optimize the depth map combination and retain useful information;

[0008] Step 2, prepare the dataset: take the source image and the reference image as the input of the data, and use the original 3D points from the reference view to estimate the dense depth map D of the 3D structure;

[0009] Step 3, training the network model: input the data set prepared in step 2 into the network model built in step 1 for training;

[0010] Step 4, design loss function and regress: by comparing the depth information obtained by data enhancement consistency loss with the depth information obtained by depth perception loss, the effective surface depth value is obtained as supervision and passed to the depth regression loss. Until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters can be considered to have been trained and saved.

[0011] Step 5, save the model and test: solidify the final model parameters, and when point cloud fusion and reconstruction operations are required, directly input the image into the network to obtain the final 3D point cloud;

[0012] Step 6, point cloud fusion and reconstruction: fuse the 3D point cloud obtained in step 5, and then use MeshLab software to view the matching and reconstruction effect after fusion; at the same time, the quality of the model is not further verified, and the optimal evaluation index is selected to measure the accuracy of the algorithm and the performance of the system.

[0013] In step 1, the feature extraction module includes eight convolutional layers, and convolutional layers 1, 2, and 3 perform downsampling operations to reduce the size of the feature map by half, and convolutional layers 4, 5, and 6 perform downsampling operations on the above feature maps to reduce the size of the feature maps by half again, and finally output by convolutional layers 7 and 8. The output of each convolutional layer of the feature extraction module is subjected to regularization operation; the pyramid volume regularization module includes ten convolutional layers, 3D convolutional layer 1 and 3D convolutional layer 2 perform the first downsampling of the extracted features, 3D convolutional layer 3 and 3D convolutional layer 4 perform the second downsampling of the extracted features, and 3D convolutional layer 5, 3D convolutional layer 6, and 3D convolutional layer 7 output the feature information after convolution. The first 3D deconvolution layer upsamples the input and outputs the deconvolution feature information through the activation function. The second 3D deconvolution layer upsamples the output of the second 3D convolution layer and outputs the deconvolution feature information through the activation function. The third 3D deconvolution layer upsamples the output of the fourth 3D convolution layer and outputs the deconvolution feature information through the activation function. Finally, the above feature information is integrated to construct the 3D cost volume and obtain a dense depth map. The output of each convolution layer in the pyramid volume regularization module is regularized; the double correction layer module contains four convolution layers. Convolution layer 1, convolution layer 2, and convolution layer 3 perform simple filtering on the obtained depth map to remove redundant information. Convolution layer 4 upsamples the input and converts the channels to optimize the depth map combination and retain useful information. The output of each convolution layer in the double correction layer module is regularized; the size of the 2D convolution kernel in all convolution layers is unified to n×n; the size of the 3D convolution kernel is unified to n×n×n.

[0014] In step 3, the data set uses DTU during the training process; semi-supervised training is performed by calculating the internal and external parameters of the camera in the data set and estimating the pose of the reconstructed object from multiple frames of images; multiple frames of images, calibrated camera parameters and depth information are used as inputs of the entire network, and the surface information matched by the loss calculation is used as a label, thereby solving the problem that most of the three-dimensional reconstruction field can only be trained under supervision.

[0015] In the step 4, the loss function during the training process selects a combination of data enhancement consistency loss and depth perception loss; the obtained depth image not only ensures its smoothness and density, but also needs to highlight the details of the authenticity of the object surface, thereby improving the matching and reconstruction effect of the fused point cloud.

[0016] After saving the model in step 5, the ETH3D dataset and the Tank&Temples dataset can be used for testing to evaluate its generalization ability.

[0017] In step 6, the evaluation indicators of accuracy and completeness are selected, which can effectively evaluate the efficiency of the algorithm and measure the role of the matching network.

[0018] The present invention also provides a multi-view stereo matching reconstruction system based on perceptual consistency loss, comprising:

[0019] An image acquisition unit, which is used to acquire a source image and reference images of multiple perspectives, wherein the reference images can be processed to remove some surface information for semi-supervised training;

[0020] The image training unit is used to input the source image and the reference image into the network for training. The semi-supervised training is performed by calculating the internal and external parameters of the camera in the data set and estimating the pose of the reconstructed object from multiple frames of images. The initially calculated depth map is input into the correction layer designed in the method to filter out redundant depth information and obtain a smoother depth map.

[0021] The depth map optimization unit is used to calculate the matching surface information as labels for training through the loss function designed in the method, thereby obtaining a dense depth map.

[0022] The point cloud fusion and reconstruction unit is used to fuse the dense depth map into a three-dimensional point cloud, and perform three-dimensional modeling based on the fused point cloud to obtain a stereoscopic visual image.

[0023] The beneficial effects of the present invention are as follows:

[0024] 1. Using the depth image calculated after loss as a supervision method to achieve semi-supervision can reduce memory usage and computing power, and return the three-dimensional points of the corresponding pixels to the actual encounter position in the three-dimensional world, thereby enhancing the robustness of the network. The calculated smooth depth map makes the point cloud matching reconstruction more complete and the effect is better.

[0025] 2. Using dual correction layers in the backbone network for edge verification can maximize the use of depth information values ​​and effectively solve the problem of incorrect edge occlusion information.

[0026] 3. The entire training network uses a splicing operation on two branches to mix the effective depth information of the correction layer and the loss layer, making the network more capable of calculating the loss of two different depth images; adding depth regression to the loss function reduces the number of network parameters, and ultimately makes the entire network simple in structure and more accurate in reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart of the multi-view stereo matching reconstruction method based on perceptual consistency loss.

[0028] Figure 2 Network structure diagram of the multi-view stereo matching reconstruction method based on perceptual consistency loss.

[0029] Figure 3 This is the specific composition of the feature extraction module described in the present invention.

[0030] Figure 4 It is the specific composition of the pyramid volume regularization module described in the present invention.

[0031] Figure 5 This is the specific composition of the dual correction layer module described in the present invention.

[0032] Figure 6 It is a schematic diagram of the structure of the multi-view stereo matching reconstruction system based on perceptual consistency loss described in the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, a multi-view stereo matching reconstruction method based on perceptual consistency loss specifically includes the following steps:

[0035] Step 1: Build a network model. Figure 2 As shown in , the entire network includes a feature extraction module, a pyramid cost volume regularization module, and a dual correction layer module, and each module consists of a convolutional layer, regularization, and activation function. Figure 3 As shown in the figure, in the feature extraction module, convolutional layers 1, 2, and 3 perform downsampling operations to reduce the size of the feature map by half, and convolutional layers 4, 5, and 6 perform downsampling operations on the above feature maps to reduce the size of the feature map by half again, and finally the convolutional layers 7 and 8 output the output. The output of each convolutional layer of the feature extraction module is subjected to regularization operations; as shown in the figure, Figure 4 As shown, in the pyramid volume regularization module, the three-dimensional convolution layer one and the three-dimensional convolution layer two perform the first down-sampling on the extracted features, the three-dimensional convolution layer three and the three-dimensional convolution layer four perform the second down-sampling on the extracted features, and the three-dimensional convolution layer five, the three-dimensional convolution layer six, and the three-dimensional convolution layer seven output the feature information after convolution. The three-dimensional deconvolution layer one upsamples the input, and outputs the feature information after deconvolution through the activation function. The three-dimensional deconvolution layer two upsamples the output of the three-dimensional convolution layer two, and outputs the feature information after deconvolution through the activation function. The three-dimensional deconvolution layer three upsamples the output of the three-dimensional convolution layer four, and outputs the feature information after deconvolution through the activation function. Finally, the above feature information is integrated to construct the 3D cost volume to obtain a dense depth map. The output of each convolution layer of the pyramid volume regularization module is regularized; as shown in FIG. Figure 5As shown in the figure, in the dual correction layer module, convolutional layers 1, 2, and 3 perform simple filtering on the obtained depth map to remove redundant information, and convolutional layer 4 performs upsampling and channel conversion on the input to optimize the depth map combination and retain useful information. The output of each convolutional layer in the dual correction layer module is regularized; the size of the two-dimensional convolution kernel in all convolutional layers is unified to n×n; the size of the three-dimensional convolution kernel is unified to n×n×n.

[0036] Step 2: Prepare the dataset. The source image and the reference image are taken as the data input, and the dense depth map D of the 3D structure is estimated from the original 3D points in the reference view by semi-supervised training on the dataset.

[0037] Step 3, training the network model. Input the data set prepared in step 2 into the network model built in step 1 for training. In the training process, the data set uses DTU; by calculating the internal and external parameters of the camera in the data set, the pose of the reconstructed object is estimated from multiple frames of images for self-supervised learning training; multiple frames of images, calibrated camera parameters and depth information are used as inputs of the entire network, and the surface information matched by the loss calculation is used as a label to solve the problem that most of the 3D reconstruction field can only be trained with supervision.

[0038] Step 4: Design the loss function and perform regression. Figure 2 The loss designed in the paper compares the parallax of the depth information obtained by the data enhancement consistency loss with the depth information obtained by the depth perception loss, and obtains the effective surface depth value as supervision to pass it to the depth regression loss. Until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters can be considered to have been trained and saved. The obtained depth image can not only ensure its smoothness and considerable density, but also highlight the details of the authenticity of the object surface, thereby improving the matching and reconstruction effect of the fused point cloud.

[0039] Step 5: Save the model and test it. The final model parameters are solidified. When point cloud fusion and reconstruction operations are required, the image can be directly input into the network to obtain the final 3D point cloud. The model is used to test the ETH3D dataset and the Tank&Temples dataset to evaluate its generalization ability.

[0040] Step 6: Point cloud fusion and reconstruction. The 3D point cloud obtained in step 5 is fused, and then the MeshLab software is used to view the matching and reconstruction effect after fusion; at the same time, the accuracy and completeness are selected as the optimal evaluation indicators, which can effectively evaluate the efficiency of the algorithm and measure the role of the matching network.

[0041] Example:

[0042] like Figure 1As shown, a multi-view stereo matching reconstruction method based on perceptual consistency loss specifically includes the following steps:

[0043] Step 1: Build a network model. Figure 2 As shown in , the entire network includes a feature extraction module, a pyramid volume regularization module, and a dual correction layer module, and each module consists of a convolutional layer, regularization, and activation function. Figure 3 As shown in , the feature extraction module contains eight two-dimensional convolutional layers. The convolution kernel size of convolutional layer 3 and convolutional layer 6 is 5×5, and the stride and padding are both 2; the convolution kernel size of convolutional layer 1, convolutional layer 2, convolutional layer 4, convolutional layer 5, convolutional layer 7, and convolutional layer 8 is 3×3, and the stride and padding are both 1; Figure 4 As shown in , the pyramid volume regularization module contains ten 3D convolutional layers, and the convolution kernels of all layers are 3×3×3. The stride of 3D convolutional layer 1, 3D convolutional layer 3, 3D convolutional layer 5, and 3D convolutional layer 7 is 2 and the padding is 1. The stride and padding of 3D convolutional layer 2, 3D convolutional layer 4, 3D convolutional layer 6, 3D deconvolutional layer 1, 3D deconvolutional layer 2, and 3D deconvolutional layer 3 are all 1; as shown in Figure 5 As shown in the figure, the structure of the correction layer 1 and the correction layer 2 in the dual correction layer module is the same, including four two-dimensional convolutional layers. The convolution kernels of convolutional layer 1, convolutional layer 2, convolutional layer 3, and convolutional layer 4 are all 3×3, the step size and padding are both 1, and the splicing dimension is 1. The linear rectification function regularization function is defined as follows:

[0044]

[0045]

[0046] Where X and y are training samples and corresponding labels, ω is the weight coefficient vector; J(.) is the objective function, Ω(ω) is the penalty term; the parameter α controls the strength of regularization, thereby controlling the complexity of the model and reducing overfitting.

[0047] Step 2, prepare the dataset. The dataset uses the DTU dataset. The dataset consists of 124 different objects or scenes. Each object is photographed from 49 perspectives, and each perspective has 7 different brightness levels. Therefore, each object or scene folder contains 343 images. The dataset also contains a training image set of the true depth map. The resolution of each image is 1600×1200.

[0048] Step 3: Train the network model. The network estimates the pose of the reconstructed object from multiple frames of images for semi-supervised training. Semi-supervised training uses multiple frames of images, calibrated camera parameters, and a small amount of depth information as the input of the entire network, and uses the surface information matched by loss calculation as a label to solve the problem that most of the 3D reconstruction field can only be trained with supervision. The system occupies less memory and has higher integrity.

[0049] Step 4: Design the loss function and perform regression. Figure 2 The loss designed in , by comparing the depth information obtained by the data enhancement consistent loss with the depth information obtained by the depth perception loss, obtains the effective surface depth value as supervision and passes it to the depth regression loss to optimize the depth map, thereby achieving a better matching reconstruction effect. The definition of data enhancement consistent loss is as follows:

[0050]

[0051] Among them, M τθ represents the unoccluded mask under τθ, D represents the predicted value of the regular forward channel of the rectified image, It is expressed as the predicted value of the extended image of the corrected image, which is obtained by minimizing D and The difference between them is used to ensure the consistency of data augmentation.

[0052] The depth perception loss is defined as follows:

[0053] p′=K′(R′R -1 (D(p)K -1 pt)+t′)

[0054] P=D(p)K -1 pt

[0055] P′=D(p′)K′ -1 p′-t′

[0056]

[0057] p′ represents the estimated corresponding pixel of p, P and P′ represent the three-dimensional depth information perception points D(p) and C(q) of the reverse projection of p and p′ are the predicted depth and probability values ​​of p pixel, ∈ h represents the threshold of highly reliable depth prediction, ∈ ω Represents the threshold used to filter out false matching pixels. h and ∈ ω The depth-aware predictions obtained by these two thresholds can approximately detect edges, occlusions, and false non-occluded regions, which may form false correspondences.

[0058] The deep regression loss is defined as follows:

[0059] L REG =λ 1 L SSIM +λ 2 L Smooth

[0060] Where L SSIM and L Smooth is the structural similarity loss and the depth smoothness loss, both of which are conditional losses for common depth estimation, so L SSIM The formula can be expressed as:

[0061]

[0062] x and y represent the pixels of the N×N window in the two images, respectively. x and μ y Represent the mean of x and y respectively, which can be used as brightness estimation; σ x and σ y Represent the variance of x and y respectively, which can be used as contrast estimation; σ xy represents the covariance of x and y, which can be used as a structural similarity measure. 1 and c 2 L is a minimum parameter that can avoid the denominator being 0, and is usually taken as 0.01 and 0.03 respectively. Smooth The formula can be expressed as:

[0063]

[0064] in and It is continuous, and x and y are both pixels of the image.

[0065] Based on the above four loss functions, the overall weighted loss function can be constructed as follows:

[0066] L=L R +λ 3 L DA +λ 4 L DPR =λ 1 L SSIM +λ 2 L Smooth +λ 3 L DA +λ 4 L DPR

[0067] After a lot of comparisons and multiple tests, the weights can be assigned as: 1 =0.2,λ 2=0.0067,λ 3 =0.1,λ 4 =0.8.

[0068] The number of training times is set to 16. The upper limit of the number of images input to the network each time is mainly determined by the performance of the computer graphics processor. Generally, the number of images input to the network each time is in the range of 1-4, which can make the network training more stable, the training results are better, and the network can be quickly fitted. During the training process, the learning rate of the parameters is 0.001, which can ensure the network to fit quickly without causing the network to overfit. The algorithm of the parameter optimizer selects the adaptive matrix estimation algorithm. Its main advantage is that after bias correction, the learning rate of each iteration has a certain range, making the parameters relatively stable. The threshold of the loss function is set to about 0.0003. If it is less than 0.0003, it can be considered that the training of the entire network has been basically completed.

[0069] Step 5: Save the model and test it. Use the DTU dataset for training and save the trained model. Use the model to test the ETH3D dataset and the Tank&Temples dataset to evaluate its generalization ability.

[0070] Step 6: point cloud fusion and reconstruction. Subsequent texture enhancement and mesh mapping belong to traditional algorithms, while the present invention can directly use the point cloud fused with the depth map for matching, and the reconstruction effect can be directly viewed using MeshLab software to verify the quality of the matching network; at the same time, as shown in Table 1, the optimal evaluation indicators are accuracy and completeness, which can effectively evaluate the efficiency of the algorithm and measure the role of the matching network. The accuracy formula is as follows:

[0071]

[0072] Among them, G is the surface frame value model, g is the point in the surface frame value model, R is the reconstruction model, and r is the point in the reconstruction model. [.] is the Iverson bracket, which is 1 if the conditions in the Iverson bracket are met, and 0 if they are not met. d is the distance coefficient.

[0073] The formula for completeness is as follows:

[0074]

[0075] Among them, G is the surface frame value model, g is the point in the surface frame value model, R is the reconstruction model, and r is the point in the reconstruction model. [.] is the Iverson bracket, which is 1 if the conditions in the Iverson bracket are met, and 0 if they are not met. d is the distance coefficient.

[0076] like Figure 6 As shown, a multi-view stereo matching reconstruction system based on perceptual consistency loss provided by the present invention includes:

[0077] An image acquisition unit, which is used to acquire a source image and reference images of multiple perspectives, wherein the reference images can be processed to remove some surface information for semi-supervised training;

[0078] The image training unit is used to input the source image and the reference image into the network for training. The semi-supervised training is performed by calculating the internal and external parameters of the camera in the data set and estimating the pose of the reconstructed object from multiple frames of images. The initially calculated depth map is input into the correction layer designed in the method to filter out redundant depth information and obtain a smoother depth map.

[0079] The depth map optimization unit is used to calculate the matching surface information as labels for training through the loss function designed in the method, thereby obtaining a dense depth map.

[0080] The point cloud fusion and reconstruction unit is used to fuse the dense depth map into a three-dimensional point cloud, and perform three-dimensional modeling based on the fused point cloud to obtain a stereoscopic visual image.

[0081] Among them, the implementation of convolution, activation function, splicing operation, regularization operation, and Gaussian filtering are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0082] The present invention constructs a multi-view stereo matching network based on perceptual consistency loss, which can directly obtain a depth map from the source image and the reference image and fuse it into a point cloud without going through other intermediate steps, thus avoiding the manual design of the stereo matching algorithm in the traditional method. The feasibility and superiority of the method are further verified by calculating the relevant indicators of the depth map obtained with the prior art. The comparison of the relevant indicators of the prior art and the method proposed in the present invention is shown in Table 1:

[0083] Table 1 Comparison of relevant indicators of the prior art and the method proposed in the present invention

[0084]

[0085] It is worth noting that the learning method of the above-mentioned method and system is an end-to-end learning method in the semi-supervised field, and the designed loss function is an important implementation function of the present invention.

Claims

1. A multi-view stereo matching reconstruction method based on perceptual consistency loss, characterized by: The method comprises the following steps: Step 1, build a network model: the entire network includes a feature extraction module, a pyramid volume regularization module and a dual correction layer module, and each module consists of a convolutional layer, regularization and activation function; the feature extraction module extracts features from the input source image and the reference image, and the pyramid volume regularization module performs downsampling extraction and upsampling integration regularization on the extracted features to construct a 3D cost volume to obtain a dense depth map; finally, the dual correction layer module performs a simple filtering on the obtained depth map to remove redundant information, optimize the depth map combination and retain useful information; Step 2, prepare the dataset: take the source image and the reference image as the input of the data, and use the original 3D points from the reference view to estimate the dense depth map D of the 3D structure; Step 3, training the network model: input the data set prepared in step 2 into the network model built in step 1 for training; Step 4, design loss function and regress: by comparing the depth information obtained by data enhancement consistency loss with the depth information obtained by depth perception loss, the effective surface depth value is obtained as supervision and passed to the depth regression loss. Until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters can be considered to have been trained and saved. Step 5, save the model and test: solidify the final model parameters, and when point cloud fusion and reconstruction operations are required, directly input the image into the network to obtain the final 3D point cloud; Step 6: Point cloud fusion and reconstruction: fuse the 3D point cloud obtained in step 5, and then use MeshLab software to view the matching and reconstruction effect after fusion; at the same time, in order to further verify the quality of the model, select the optimal evaluation index to measure the accuracy of the algorithm and the performance of the system; The dual correction layer module includes four convolution layers. Convolution layers 1, 2, and 3 perform simple filtering on the obtained depth map to remove redundant information. Convolution layer 4 performs upsampling and channel conversion on the input to optimize the depth map combination and retain useful information. The output of each convolution layer in the dual correction layer module is regularized. The size of the two-dimensional convolution kernel in all convolution layers is unified as ; The size of the three-dimensional convolution kernel is unified as .

2. The multi-view stereo matching reconstruction method based on perceptual consistency loss according to claim 1, characterized in that: In the step 1, the feature extraction module comprises eight convolutional layers, wherein convolutional layer 1, convolutional layer 2, and convolutional layer 3 perform downsampling operations to reduce the size of the feature map by half, convolutional layer 4, convolutional layer 5, and convolutional layer 6 perform downsampling operations on the above feature map to further reduce the size of the feature map by half, and finally the convolutional layer 7 and convolutional layer 8 output the output; the output of each convolutional layer of the feature extraction module is subjected to a regularization operation; the pyramid volume regularization module comprises ten convolutional layers, wherein the three-dimensional convolutional layer 1 and the three-dimensional convolutional layer 2 perform the first downsampling on the extracted features, and the three-dimensional convolutional layer 3 and the three-dimensional convolutional layer 4 perform the second downsampling on the extracted features , three-dimensional convolution layer five, three-dimensional convolution layer six and three-dimensional convolution layer seven to output the feature information after convolution; the three-dimensional deconvolution layer one upsamples the input, and outputs the feature information after deconvolution after the activation function, the three-dimensional deconvolution layer two upsamples the output of the three-dimensional convolution layer two, and outputs the feature information after deconvolution after the activation function, the three-dimensional deconvolution layer three upsamples the output of the three-dimensional convolution layer four, and outputs the feature information after deconvolution after the activation function, and finally integrates the above feature information to construct the 3D cost volume to obtain a dense depth map; the output of each convolution layer of the pyramid volume regularization module is regularized.

3. The multi-view stereo matching reconstruction method based on perceptual consistency loss according to claim 1, characterized in that: In step 3, the data set uses DTU during the training process; semi-supervised training is performed by calculating the internal and external parameters of the camera in the data set and estimating the pose of the reconstructed object from multiple frames of images; multiple frames of images, calibrated camera parameters and depth information are used as inputs of the entire network, and the surface information matched by the loss calculation is used as a label, thereby solving the problem that most of the three-dimensional reconstruction field can only be trained under supervision.

4. The multi-view stereo matching reconstruction method based on perceptual consistency loss according to claim 1, characterized in that: In the step 4, the loss function during the training process selects a combination of data enhancement consistency loss and depth perception loss; the obtained depth image not only ensures its smoothness and density, but also needs to highlight the details of the authenticity of the object surface, thereby improving the matching and reconstruction effect of the fused point cloud.

5. The multi-view stereo matching reconstruction method based on perceptual consistency loss according to claim 1, characterized in that: After saving the model in step 5, the ETH3D dataset and the Tank&Temples dataset can be used for testing to evaluate its generalization ability.

6. The multi-view stereo matching reconstruction method based on perceptual consistency loss according to claim 1, characterized in that: In step 6, the evaluation indicators of accuracy and completeness are selected, which can effectively evaluate the efficiency of the algorithm and measure the role of the matching network.

7. The system of the multi-view stereo matching reconstruction method based on perceptual consistency loss according to claim 1, characterized in that: The system includes: An image acquisition unit, which is used to acquire a source image and reference images of multiple perspectives, wherein the reference images can be processed to remove some surface information for semi-supervised training; The image training unit is used to input the source image and the reference image into the network model for training. The semi-supervised training is performed by estimating the pose of the reconstructed object from multiple frames of images by calculating the internal and external parameters of the camera in the data set. The initially calculated depth map is input into the dual correction layer designed in the method to filter out redundant depth information and obtain a smoother depth map. The depth map optimization unit is used to calculate the matching surface information as labels for training through the loss function designed in the method, thereby obtaining a dense depth map. The point cloud fusion and reconstruction unit is used to fuse the dense depth map into a three-dimensional point cloud, and perform three-dimensional modeling based on the fused point cloud to obtain a stereoscopic visual image.

Images