An Adaptive Cost Volume 3D Reconstruction Method Based on ACVMVS Network

Abstract

This invention discloses an adaptive cost volume 3D reconstruction method based on an ACVMVS network, belonging to the field of 3D spatial reconstruction technology. The steps are as follows: obtaining a training set of 3D reconstruction images; constructing an ACVMVS network; inputting the training set of 3D reconstruction images into the ACVMVS network for training; using the trained ACVMVS network to process the 3D reconstruction images to be reconstructed, obtaining the predicted depth map corresponding to the 3D reconstruction images; and completing the adaptive cost volume 3D reconstruction based on the predicted depth map. This invention solves the problem that existing 3D reconstruction methods do not achieve good reconstruction results for areas with strong illumination, reflection, low texture, no texture, and severe occlusion in image data.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional spatial reconstruction technology, and particularly relates to an adaptive cost volume three-dimensional reconstruction method based on ACVMVS network. Background Technology

[0002] As people's demands for image quality and their requirements for scene recognition continue to rise, 3D spatial vision is becoming increasingly important. Through 3D spatial observation, various scene structures and details missing in the 2D plane can be obtained, making the user experience more realistic and reliable. Conventional 3D reconstruction equipment can reconstruct 3D space from images captured by cameras, but its price is very high, making it insufficient for daily use and unable to be widely adopted by the public.

[0003] Multi-view stereo matching (MVS) aims to recover 3D scene geometry from a set of RGB images using known camera poses, and to obtain a dense 3D model of a real-world scene from multiple images. Traditional methods for recovering dense scene structures from multiple overlapping viewpoints utilize geometric and optical consistency to construct a matching cost, accumulate the matching cost, and then estimate the depth value. In contrast to traditional MVS methods that use handcrafted matching metrics for image consistency checks, deep learning-based MVS methods tend to use frontal plane scanning, evaluating the depth of the same candidate set based on the same image for each pixel, and achieving higher accuracy and completeness on many MVS benchmarks compared to previous state-of-the-art techniques. Recent multi-view stereo matching algorithms typically compute a 3D cost volume based on a set of assumed depths. Applying 3D convolution to the cost volume can then be used to regularize and regress the final scene depth; various variants based on this approach also exist.

[0004] The ACVMVS network takes a reference image and multiple source images as input, extracts depth image features, encodes camera geometry within the network, constructs a 3D cost volume through differentiable homography, and finally obtains the depth map through multiple 3D convolutions. The ACVMVS network significantly improves the performance of MVS reconstruction, but it leaves some unresolved issues: huge memory and runtime consumption, and missing point cloud information. To address these issues, many optimizations have been made, and these variants can be categorized into multi-stage methods and recursive methods. The main drawback of recursive methods is their long runtime, while multi-stage methods improve memory and runtime efficiency. However, most learning-based MVS methods still struggle with handling challenging regions and severe occlusion in MVS. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides an adaptive cost volume 3D reconstruction method based on an ACVMVS network. This method adds the proposed adaptive cost volume (ACV) module to the MVS network, using the relevant volumes of different views to generate attention weights. It learns the motion transition relationships between different views across multiple cost volumes to optimize the reconstruction of object detail structures. This solves the problem that existing 3D reconstruction methods do not achieve good reconstruction results for areas with strong illumination, reflection, low texture, no texture, and severe occlusion in image data.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] This invention provides an adaptive cost volume 3D reconstruction method based on ACVMVS network, comprising the following steps:

[0008] S1. Obtain the training set of 3D reconstructed images;

[0009] S2. Construct the ACVMVS network;

[0010] S3. Input the training set of the three-dimensional reconstructed images into the ACVMVS network for training;

[0011] S4. Use the trained ACVMVS network to process the 3D reconstruction image group to be reconstructed and obtain the predicted depth map corresponding to the 3D reconstruction image group.

[0012] S5. Based on the predicted depth map, complete the adaptive cost volume 3D reconstruction based on the ACVMVS network.

[0013] Furthermore, the training set of the three-dimensional reconstructed image group includes several three-dimensional reconstructed image groups; each of the three-dimensional reconstructed image groups includes three images of the same object from different perspectives.

[0014] The beneficial effects of adopting the above-mentioned further solutions are as follows:

[0015] Furthermore, the ACVMVS network includes a feature extraction module, a first MVS pipeline module, a second MVS pipeline module, an ACV module, and a third MVS pipeline module;

[0016] The input terminal of the feature extraction module serves as the input terminal of the ACVMVS network; the first output terminal of the feature extraction module is connected to the input terminal of the first MVS pipeline module; the second output terminal of the feature extraction module is connected to the output terminal of the first MVS pipeline module and the input terminal of the second MVS pipeline module; the ACV output terminal of the second MVS pipeline module is connected to the input terminal of the ACV module; the ACV input terminal of the second MVS pipeline module is connected to the output terminal of the ACV module; the third output terminal of the feature extraction module is connected to the output terminal of the second MVS pipeline module and the input terminal of the third MVS pipeline module; the output terminal of the third MVS pipeline module serves as the output terminal of the ACVMVS network.

[0017] Furthermore, the feature extraction module includes a deformable 2D convolutional unit, a first 2D convolutional unit, a second 2D convolutional unit, a third 2D convolutional unit, a fourth 2D convolutional unit, a fifth 2D convolutional unit, a first Concat fusion unit, and a second Concat fusion unit;

[0018] The input of the deformable 2D convolutional unit serves as the input of the feature extraction module; the output of the deformable 2D convolutional unit is connected to the input of the first 2D convolutional unit and the second input of the second Concat fusion unit; the output of the first 2D convolutional unit is connected to the input of the second 2D convolutional unit and the second input of the first Concat fusion unit; the output of the second 2D convolutional unit is connected to the input of the third 2D convolutional unit; the first output of the third 2D convolutional unit serves as the first output of the feature extraction module and is connected to the input of the first MVS pipeline module; the second output of the third 2D convolutional unit... The first input terminal of the first Concat fusion unit is connected to the first input terminal of the first Concat fusion unit; the output terminal of the first Concat fusion unit is connected to the input terminal of the fourth 2D convolution unit; the first output terminal of the fourth 2D convolution unit serves as the second output terminal of the feature extraction module and is connected to the input terminal of the second MVS pipeline module; the second output terminal of the fourth 2D convolution unit is connected to the first input terminal of the second Concat fusion unit; the output terminal of the second Concat fusion unit is connected to the input terminal of the fifth 2D convolution unit; the output terminal of the fifth 2D convolution unit serves as the third output terminal of the feature extraction module and is connected to the input terminal of the third MVS pipeline module.

[0019] The first MVS pipeline module includes a first homography variation unit, a first sub-cost body unit, a first cost body construction unit, a first variance cost body measurement unit, a first cost body regression unit, and a first depth regression unit connected in sequence. The input end of the first homography variation unit serves as the input end of the first MVS pipeline module and is connected to the first output end of the third 2D convolution unit. The output end of the first depth regression unit serves as the output end of the first MVS pipeline module and is connected to the input end of the second MVS pipeline module.

[0020] The second MVS pipeline module includes a second homography variation unit, a second sub-cost body unit, an ACV fusion unit, a second cost body construction unit, a second variance cost body measurement unit, a second cost body regression unit, and a second depth regression unit;

[0021] The input of the second homography variation unit is connected to the output of the first depth regression unit and the first output of the fourth 2D convolution unit, respectively; the first output of the second homography variation unit is connected to the input of the second sub-cost body unit; the second output of the second homography variation unit serves as the ACV output of the second MVS pipeline module and is connected to the input of the ACV module; the output of the second sub-cost body unit is connected to the first input of the ACV fusion unit; the second input of the ACV fusion unit serves as the ACV input of the second MVS pipeline module and is connected to the output of the ACV module; the output of the ACV fusion unit is connected to the input of the second cost body construction unit; the output of the second cost body construction unit is connected to the input of the second variance cost body measurement unit; the output of the second variance cost body measurement unit is connected to the input of the second cost body regression unit; the output of the second cost body regression unit is connected to the input of the second depth regression unit; the output of the second depth regression unit serves as the output of the second MVS pipeline module and is connected to the input of the third MVS pipeline module.

[0022] The ACV module includes a first 3D convolutional unit, a second 3D convolutional unit, a third 3D convolutional unit, a fourth 3D convolutional unit, a third Concat fusion unit, an hourglass unit, and a regression unit;

[0023] The input terminal of the first 3D convolutional unit serves as the input terminal of the ACV module and is connected to the second output terminal of the second homography transformation unit; the first output terminal of the first 3D convolutional unit is connected to the input terminal of the second 3D convolutional unit; the second output terminal of the first 3D convolutional unit is connected to the input terminal of the third 3D convolutional unit; the third output terminal of the first 3D convolutional unit is connected to the input terminal of the fourth 3D convolutional unit; the output terminals of the second, third, and fourth 3D convolutional units are all connected to the input terminal of the third Concat fusion unit; the output terminal of the third Concat fusion unit is connected to the input terminal of the hourglass unit; the output terminal of the hourglass unit is connected to the input terminal of the regression unit; the output terminal of the regression unit serves as the output terminal of the ACV module and is connected to the second input terminal of the ACV fusion unit.

[0024] The third MVS pipeline module includes a third homography variation unit, a third sub-cost body unit, a third cost body construction unit, a third difference cost body measurement unit, a third cost body regression unit, and a third depth regression unit connected in sequence. The input end of the third homography variation unit serves as the input end of the third MVS pipeline module and is connected to the output end of the second cost body regression unit and the output end of the fifth 2D convolution unit, respectively. The output end of the third depth regression unit serves as the output end of the third MVS pipeline module.

[0025] Furthermore, the training method for the ACVMVS network includes the following steps:

[0026] A1. Input each group of 3D reconstructed images into the feature extraction module in sequence to obtain the corresponding first feature extraction result, second feature extraction result and third feature extraction result;

[0027] A2. Input the first feature extraction result into the first MVS pipeline module to obtain the first stage depth map;

[0028] A3. Input the first-stage depth map and the second feature extraction results into the second MVS pipeline module, and obtain the second-stage depth map based on the ACV module;

[0029] A4. Input the second-stage depth map and the third feature extraction results into the third MVS pipeline module to obtain the predicted depth map corresponding to each 3D reconstructed image group, and complete the training of the ACVMVS network.

[0030] Further, step A1 includes the following steps:

[0031] A11. Input each group of 3D reconstructed images into the deformable 2D convolutional unit in sequence to obtain the deformable convolutional feature extraction results, and output the deformable convolutional feature extraction results to the first 2D convolutional unit and the second Concat fusion unit.

[0032] A12. The deformable convolution feature extraction result is processed by the first 2D convolution unit to obtain the first 2D convolution feature extraction result, and the first 2D convolution feature extraction result is output to the second 2D convolution unit and the first Concat fusion unit.

[0033] A13. Process the first 2D convolution feature extraction result using the second 2D convolution unit to obtain the second 2D convolution feature extraction result, and output the second convolution feature extraction result to the third 2D convolution unit.

[0034] A14. The second 2D convolution feature extraction result is processed by the third 2D convolution unit to obtain the third 2D convolution feature extraction result. The third 2D convolution feature extraction result is then output to the first Concat fusion unit, and the third 2D convolution feature extraction result is used as the first feature extraction result.

[0035] A15. The first 2D convolutional feature extraction result and the third 2D convolutional feature extraction result are fused using the first Concat fusion unit to obtain the first convolutional feature fusion result, and the first convolutional feature fusion result is output to the fourth 2D convolutional unit.

[0036] A16. The first convolution feature fusion result is processed by the fourth 2D convolution unit to obtain the fourth convolution feature extraction result. The fourth convolution feature extraction result is then output to the second Concat fusion unit, and the fourth convolution feature extraction result is used as the second feature extraction result.

[0037] A17. The deformable convolution feature extraction result and the fourth convolution feature extraction result are fused using the second Concat fusion unit to obtain the second convolution feature fusion result, and the second convolution feature fusion result is output to the fifth 2D convolution unit.

[0038] A18. The second convolution feature fusion result is processed using the fifth 2D convolution unit to obtain the third feature extraction result.

[0039] Further, step A2 includes the following sub-steps:

[0040] A21. Input the first feature extraction result into the first homography transformation unit to obtain the first cost body, and output the first cost body to the first sub-cost body unit;

[0041] A22. Process the first cost body using the first sub-cost body unit to obtain several first sub-cost bodies, and output each first sub-cost body sequentially to the first cost body construction unit;

[0042] A23. Use the first cost body construction unit to splice together each first sub-cost body to obtain the first cost body, and output the first cost body to the first variance cost body measurement unit;

[0043] A24. Process the first cost body using the first variance cost body measurement unit to obtain the first variance measurement result, and output the first variance measurement result to the first cost body regression unit;

[0044] A25. Process the first variance measurement result using the first cost body regression unit to obtain the first cost body regression result, and output the first cost body regression result to the first deep regression unit;

[0045] A26. Process the first cost volume regression results using the first depth regression unit to obtain the first stage depth map.

[0046] Further, step A3 includes the following steps:

[0047] A31. Input the first-stage depth map and the second feature extraction results into the second homography transformation unit to obtain the second cost body, and output the second cost body to the second sub-cost body unit and the ACV module respectively.

[0048] A32. Process the second cost body using the second sub-cost body unit to obtain several second sub-cost bodies, and output each second sub-cost body sequentially to the ACV fusion unit.

[0049] A33. Use the ACV module to process the second cost body, obtain motion feature results from different perspectives, and output the motion feature results from different perspectives to the ACV fusion unit.

[0050] A34. Use the ACV fusion unit to process the motion feature results of each second sub-subbody and different viewpoints to obtain the ACV fusion result, and output the ACV fusion result to the second subbody construction unit.

[0051] A35. Process the ACV fusion results using the second cost body construction unit to obtain the second cost body, and output the second cost body to the second variance cost body measurement unit;

[0052] A36. Process the second cost body using the second variance cost body measurement unit to obtain the second variance measurement result, and output the second variance measurement result to the second cost body regression unit;

[0053] A37. Process the second variance measure result using the second cost body regression unit to obtain the second cost body regression result, and output the second cost body regression result to the second deep regression unit.

[0054] A38. The second cost body regression results are processed using the second depth regression unit to obtain the second-stage depth map.

[0055] Further, step A33 includes the following steps:

[0056] A331. Input the second cost body into the first 3D convolutional unit to obtain the first 3D convolutional feature extraction result, and output the first 3D convolutional feature extraction result to the second 3D convolutional unit, the third 3D convolutional unit and the fourth 3D convolutional unit respectively.

[0057] A332. Process the first 3D convolution feature extraction result using the second 3D convolution unit, the third 3D convolution unit and the fourth 3D convolution unit respectively to obtain the second 3D convolution feature extraction result, the third 3D convolution feature extraction result and the fourth 3D convolution feature extraction result;

[0058] A333. The second 3D convolution feature extraction result, the third 3D convolution feature extraction result, and the fourth 3D convolution feature extraction result are simultaneously output to the third Concat fusion unit for feature fusion to obtain the third convolution feature fusion result, and the third convolution feature fusion result is output to the hourglass unit.

[0059] A333. Use the hourglass unit to process the fusion result of the third convolution feature to obtain the hourglass processing result, and output the hourglass processing result to the regression unit;

[0060] A334. Using regression units to process the hourglass results, we obtain motion characteristic results from different perspectives.

[0061] Further, step A4 includes the following steps:

[0062] A41. Input the second-stage depth map and the third feature extraction result into the third homography transformation unit to obtain the third cost body, and output the third cost body to the third sub-cost body unit;

[0063] A42. Process the third cost body using the third sub-cost body unit to obtain several third sub-cost bodies, and output each third sub-cost body sequentially to the third cost body construction unit;

[0064] A43. Use the third cost body construction unit to splice together each third sub-cost body to obtain the third cost body, and output the third cost body to the third difference cost body measurement unit;

[0065] A44. Process the third cost body using the third-party difference cost body measurement unit to obtain the third-party difference measurement result, and output the third-party difference measurement result to the third cost body regression unit;

[0066] A45. Process the third-party difference measurement results using the third-cost body regression unit to obtain the third-cost body regression results, and output the third-cost body regression results to the third-depth regression unit.

[0067] A46. Use the third depth regression unit to process the regression results of the third cost body to obtain the predicted depth map.

[0068] The beneficial effects of this invention are as follows: This invention provides an adaptive cost volume 3D reconstruction method based on an ACVMVS network. It learns rich information about the cost of 3D reconstructed images by using patches of different sizes at different feature levels. The adaptive cost volume (ACV) module enhances textures at different hypothetical depths, learns sparse and inconspicuous features, and adaptively adjusts the cost. Using predefined sizes and adaptively learned weights, it calculates matching costs in feature maps at different levels, maintains the number of pixels at the center of convolutional blocks, and obtains similarity based on the correlation between patch pixels in different blocks. Simultaneously, it maintains the correlation between the range of convolutional blocks and the feature maps, enabling the understanding of motion structures between different views during multi-view stereo matching. This enriches the detailed features of point cloud reconstruction, improving the completeness and accuracy of point cloud reconstruction. Furthermore, this scheme adds adaptive 2D variable convolutional units to the feature extraction module, allowing the feature extraction network to adaptively learn structural information of different sizes and scenes, improving the network's generalization ability and robustness, resulting in excellent performance on other datasets. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating the steps of the adaptive cost volume 3D reconstruction method based on the ACVMVS network in an embodiment of the present invention.

[0070] Figure 2 This is a network structure block diagram of the ACVMVS network in an embodiment of the present invention.

[0071] Figure 3 This is a structural block diagram of the feature extraction module in an embodiment of the present invention.

[0072] Figure 4 This is a structural block diagram of the first MVS pipeline module in an embodiment of the present invention.

[0073] Figure 5 This is a structural block diagram of the second MVS pipeline module in an embodiment of the present invention.

[0074] Figure 6This is a structural block diagram of the ACV module in an embodiment of the present invention.

[0075] Figure 7 This is a structural block diagram of the third MVS pipeline module in an embodiment of the present invention. Detailed Implementation

[0076] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0077] Example 1

[0078] like Figure 1 As shown, in one embodiment of the present invention, the present invention provides an adaptive cost volume 3D reconstruction method based on ACVMVS network, comprising the following steps:

[0079] S1. Obtain the training set of 3D reconstructed images;

[0080] The training set of the three-dimensional reconstructed image group includes several three-dimensional reconstructed image groups; each three-dimensional reconstructed image group includes three images of the same object from different perspectives.

[0081] S2. Construct the ACVMVS network;

[0082] The ACVMVS network uses a Feature Pyramid Network (FPN) architecture for feature extraction and constructs an initial cost volume through homography of three different feature sizes, similar to CasMVSNet. It employs the original MVSNet pipeline as different levels of propagation – differentiable homography – cost volume regularization – depth map refinement, applying the output of each layer in a cascaded manner to a higher initial depth. This reduces the range of depth prediction and allows the depth map to be predicted from coarse to fine. In the second cascade, this scheme introduces an ACV module for pixel-level similarity learning and adaptive cost volume calculation, making the cost volume features at this level more apparent. This allows the network to understand surface textures at hard-to-focus points and enhances its perception of weak textures.

[0083] like Figure 2 As shown, the ACVMVS network includes a feature extraction module, a first MVS pipeline module, a second MVS pipeline module, an ACV module, and a third MVS pipeline module;

[0084] The input terminal of the feature extraction module serves as the input terminal of the ACVMVS network; the first output terminal of the feature extraction module is connected to the input terminal of the first MVS pipeline module; the second output terminal of the feature extraction module is connected to the output terminal of the first MVS pipeline module and the input terminal of the second MVS pipeline module; the ACV output terminal of the second MVS pipeline module is connected to the input terminal of the ACV module; the ACV input terminal of the second MVS pipeline module is connected to the output terminal of the ACV module; the third output terminal of the feature extraction module is connected to the output terminal of the second MVS pipeline module and the input terminal of the third MVS pipeline module; the output terminal of the third MVS pipeline module serves as the output terminal of the ACVMVS network.

[0085] like Figure 3 As shown, the feature extraction module includes a deformable 2D convolutional unit, a first 2D convolutional unit, a second 2D convolutional unit, a third 2D convolutional unit, a fourth 2D convolutional unit, a fifth 2D convolutional unit, a first Concat fusion unit, and a second Concat fusion unit.

[0086] The input of the deformable 2D convolutional unit serves as the input of the feature extraction module; the output of the deformable 2D convolutional unit is connected to the input of the first 2D convolutional unit and the second input of the second Concat fusion unit; the output of the first 2D convolutional unit is connected to the input of the second 2D convolutional unit and the second input of the first Concat fusion unit; the output of the second 2D convolutional unit is connected to the input of the third 2D convolutional unit; the first output of the third 2D convolutional unit serves as the first output of the feature extraction module and is connected to the input of the first MVS pipeline module; the second output of the third 2D convolutional unit... The first input of the first Concat fusion unit is connected to the first input of the second Concat fusion unit; the output of the first Concat fusion unit is connected to the input of the fourth 2D convolution unit; the first output of the fourth 2D convolution unit serves as the second output of the feature extraction module and is connected to the input of the second MVS pipeline module; the second output of the fourth 2D convolution unit is connected to the first input of the second Concat fusion unit; the output of the second Concat fusion unit is connected to the input of the fifth 2D convolution unit; the output of the fifth 2D convolution unit serves as the third output of the feature extraction module and is connected to the input of the third MVS pipeline module.

[0087] like Figure 4As shown, the first MVS pipeline module includes a first homography variation unit, a first sub-cost body unit, a first cost body construction unit, a first variance cost body measurement unit, a first cost body regression unit, and a first depth regression unit connected in sequence. The input end of the first homography variation unit serves as the input end of the first MVS pipeline module and is connected to the first output end of the third 2D convolution unit. The output end of the first depth regression unit serves as the output end of the first MVS pipeline module and is connected to the input end of the second MVS pipeline module.

[0088] like Figure 5 As shown, the second MVS pipeline module includes a second homography variation unit, a second sub-cost body unit, an ACV fusion unit, a second cost body construction unit, a second variance cost body measurement unit, a second cost body regression unit, and a second depth regression unit;

[0089] The input of the second homography variation unit is connected to the output of the first depth regression unit and the first output of the fourth 2D convolution unit, respectively; the first output of the second homography variation unit is connected to the input of the second sub-cost body unit; the second output of the second homography variation unit serves as the ACV output of the second MVS pipeline module and is connected to the input of the ACV module; the output of the second sub-cost body unit is connected to the first input of the ACV fusion unit; the second input of the ACV fusion unit serves as the ACV input of the second MVS pipeline module and is connected to the output of the ACV module; the output of the ACV fusion unit is connected to the input of the second cost body construction unit; the output of the second cost body construction unit is connected to the input of the second variance cost body measurement unit; the output of the second variance cost body measurement unit is connected to the input of the second cost body regression unit; the output of the second cost body regression unit is connected to the input of the second depth regression unit; the output of the second depth regression unit serves as the output of the second MVS pipeline module and is connected to the input of the third MVS pipeline module.

[0090] like Figure 6 As shown, the ACV module includes a first 3D convolutional unit, a second 3D convolutional unit, a third 3D convolutional unit, a fourth 3D convolutional unit, a third Concat fusion unit, an hourglass unit, and a regression unit;

[0091] The input of the first 3D convolutional unit serves as the input of the ACV module and is connected to the second output of the second homography transformation unit; the first output of the first 3D convolutional unit is connected to the input of the second 3D convolutional unit; the second output of the first 3D convolutional unit is connected to the input of the third 3D convolutional unit; the third output of the first 3D convolutional unit is connected to the input of the fourth 3D convolutional unit; the outputs of the second, third, and fourth 3D convolutional units are all connected to the input of the third Concat fusion unit; the output of the third Concat fusion unit is connected to the input of the hourglass unit; the output of the hourglass unit is connected to the input of the regression unit; the output of the regression unit serves as the output of the ACV module and is connected to the second input of the ACV fusion unit.

[0092] like Figure 7 As shown, the third MVS pipeline module includes a third homography variation unit, a third sub-cost body unit, a third cost body construction unit, a third difference cost body measurement unit, a third cost body regression unit, and a third depth regression unit connected in sequence. The input end of the third homography variation unit serves as the input end of the third MVS pipeline module and is connected to the output end of the second depth regression unit and the output end of the fifth 2D convolution unit, respectively. The output end of the third depth regression unit serves as the output end of the third MVS pipeline module.

[0093] S3. Input the training set of the three-dimensional reconstructed images into the ACVMVS network for training;

[0094] The training method for the ACVMVS network includes the following steps:

[0095] A1. Input each group of 3D reconstructed images into the feature extraction module in sequence to obtain the corresponding first feature extraction result, second feature extraction result and third feature extraction result;

[0096] Step A1 includes the following steps:

[0097] A11. Input each group of 3D reconstructed images into a deformable 2D convolutional unit in sequence to obtain the deformable convolutional feature extraction result, and output the deformable convolutional feature extraction result to the first 2D convolutional unit and the second Concat fusion unit; according to the data format of (channel, size X, size Y), each group of 3D reconstructed images is (3, 512, 640), and the deformable convolutional feature extraction result is (8, 512, 640).

[0098] A12. The deformable convolution feature extraction result is processed by the first 2D convolution unit to obtain the first 2D convolution feature extraction result, and the first 2D convolution feature extraction result is output to the second 2D convolution unit and the first Concat fusion unit; The image data format of the first 2D convolution feature extraction result is (16, 256, 320).

[0099] A13. Process the first 2D convolution feature extraction result using the second 2D convolution unit to obtain the second 2D convolution feature extraction result, and output the second convolution feature extraction result to the third 2D convolution unit; the image data format of the second 2D convolution feature extraction result is (32, 128, 160).

[0100] A14. The second 2D convolution feature extraction result is processed by the third 2D convolution unit to obtain the third 2D convolution feature extraction result, and the third 2D convolution feature extraction result is output to the first Concat fusion unit, and the third 2D convolution feature extraction result is used as the first feature extraction result; the image data format of the third 2D convolution feature extraction result is (32, 128, 160), which is the image data format of the first feature extraction result;

[0101] A15. The first 2D convolutional feature extraction result and the third 2D convolutional feature extraction result are fused using the first Concat fusion unit to obtain the first convolutional feature fusion result, and the first convolutional feature fusion result is output to the fourth 2D convolutional unit; the image data format of the first convolutional feature fusion result is (16,256,320).

[0102] A16. The first convolution feature fusion result is processed by the fourth 2D convolution unit to obtain the fourth convolution feature extraction result, and the fourth convolution feature extraction result is output to the second Concat fusion unit, and the fourth convolution feature extraction result is used as the second feature extraction result; the image data format of the fourth convolution feature extraction result is (16,256,320).

[0103] A17. The deformable convolution feature extraction result and the fourth convolution feature extraction result are fused using the second Concat fusion unit to obtain the second convolution feature fusion result, and the second convolution feature fusion result is output to the fifth 2D convolution unit; the image data format of the second convolution feature fusion result is (8,512,640).

[0104] A18. The second convolution feature fusion result is processed using the fifth 2D convolution unit to obtain the third feature extraction result; the image data format of the third feature extraction result is (8,512,640).

[0105] To extract more learning-friendly features from the original image, this solution introduces deformable 2D convolution in the feature extraction module. Deformable 2D convolution can automatically adjust the scale or receptive field and adjust the shape according to the actual situation, thus better extracting the input features.

[0106] A2. Input the first feature extraction result into the first MVS pipeline module to obtain the first stage depth map;

[0107] Step A2 includes the following sub-steps:

[0108] A21. The first feature extraction result is input into the first homography transformation unit to obtain the first cost body, and the first cost body is output to the first sub-cost body unit. According to the data format of (channel, size X, size Y), there are three images input into the first homography transformation unit based on the first feature extraction result. The image data formats are ref: (32, 128, 160) and src: (2, 32, 128, 160) respectively. The first cost body has 48 depths faked by the homography formula, that is, its image data format is (32, 48, 128, 160).

[0109] In inherently ill-posed regions, such as occluded areas, repeating patterns, textured regions, and reflective surfaces, per-pixel cost calculations are often ambiguous. To address this issue, multi-scale 3D CNNs are typically introduced to aggregate contextual information and adjust costs that may be contaminated by noise. In MVSNet, the 3D cost volume is proposed to use frontal parallel planes at different depths as hypothetical planes, and the depth range is usually determined by sparse reconstruction. The coordinate mapping is determined by homography transformation, the calculation expression of which is as follows:

[0110]

[0111] in, Indicates the first i The feature map and depth of the view are d The homography relationship matrix between the reference feature maps, Indicates the first i The inherent camera property matrix of the view, Indicates the first i The rotation matrix of the view. Indicates the initial position of the first view. Indicates the first i The amount of translation of the view. This represents the principal axis transpose matrix of the reference camera. I Represents a unit vector. d Indicates depth. The transpose of the rotation matrix of the first view. Indicates the first by The inverse of the view's inherent camera property matrix.

[0112] A22. Process the first cost body using the first sub-cost body unit to obtain several first sub-cost bodies, and output each first sub-cost body sequentially to the first cost body construction unit;

[0113] A23. Use the first cost body construction unit to splice together each first sub-cost body to obtain the first cost body, and output the first cost body to the first variance cost body measurement unit;

[0114] A24. Process the first cost body using the first variance cost body measurement unit to obtain the first variance measurement result, and output the first variance measurement result to the first cost body regression unit; the size of the first variance measurement result remains unchanged, only 0 data is removed and the variance is calculated, and the resulting image data format is still (32,48,128,160).

[0115] A25. Process the first variance measurement result using the first cost volume regression unit to obtain the first cost volume regression result, and output the first cost volume regression result to the first depth regression unit; the image data format of the first cost volume regression result is (1,48,128,160).

[0116] A26. The first cost body regression result is processed using the first depth regression unit to obtain the first stage depth map; the image data format of the first stage depth map is (1,128,160).

[0117] In the process of multi-view reconstruction, the construction of 3D cost volume includes at least three main steps: determining discrete hypothetical depth planes; distorting and extracting 2D features of each view in the hypothetical depth plane and constructing several feature volumes; and fusing the feature volumes to construct the 3D cost volume.

[0118] By using differentiable homography, 2D feature maps are distorted into an imaginary plane of a reference camera to form eigenvalues. In order to aggregate multiple eigenvalues ​​into a single cost metric, this scheme provides a variance-based cost metric to fit any number of input features.

[0119] A3. Input the first-stage depth map and the second feature extraction results into the second MVS pipeline module, and obtain the second-stage depth map based on the ACV module;

[0120] Step A3 includes the following steps:

[0121] A31. Input the first-stage depth map and the second feature extraction result into the second homography transformation unit to obtain the second cost body, and output the second cost body to the second sub-cost body unit and the ACV module respectively; Based on the second feature extraction result and the first-stage depth map, three images are input into the second homography transformation unit, and their image data formats are ref: (16,256,320) and src: (2,16,256,320) respectively. 32 pseudo depths are faked by the homography formula, that is, the image data format of the second cost body is (16,32,256,320).

[0122] A32. Process the second cost body using the second sub-cost body unit to obtain several second sub-cost bodies, and output each second sub-cost body sequentially to the ACV fusion unit.

[0123] A33. Use the ACV module to process the second cost body, obtain motion feature results from different perspectives, and output the motion feature results from different perspectives to the ACV fusion unit.

[0124] The cost quantity obtained after homography transformation, i.e. the second cost volume, is input into the ACV module. Patches of different sizes at different feature levels are used to learn rich content information about this cost quantity, which is usually ignored by other MVS networks.

[0125] Step A33 includes the following steps:

[0126] A331. Input the second cost body into the first 3D convolutional unit to obtain the first 3D convolutional feature extraction result, and output the first 3D convolutional feature extraction result to the second 3D convolutional unit, the third 3D convolutional unit and the fourth 3D convolutional unit respectively.

[0127] A332. Process the first 3D convolution feature extraction result using the second 3D convolution unit, the third 3D convolution unit and the fourth 3D convolution unit respectively to obtain the second 3D convolution feature extraction result, the third 3D convolution feature extraction result and the fourth 3D convolution feature extraction result;

[0128] A333. The second, third, and fourth 3D convolutional feature extraction results are simultaneously output to the third Concat fusion unit for feature fusion to obtain the third convolutional feature fusion result, and then output to the hourglass unit; the image data format of the third convolutional feature fusion result is consistent with the image data format of the cost body and each sub-cost body, which is (16, 32, 256, 320).

[0129] Feature maps at different depths exhibit strong similarity, implicitly reflecting the relationships between adjacent pixels in an image; that is, adjacent pixels belonging to the same class tend to have close similarity. To learn more important details on the network, this scheme introduces an adaptive cost volume to enhance texture at different hypothetical depths. To learn scalable features, this scheme maintains the number of pixels at the center of the convolutional block and obtains similarity based on the correlation between patch pixels of different blocks, while also maintaining the correlation between the range of the convolutional block and the feature map. The cost of multi-scale patches is aggregated at the same scale using CONCAT. This scheme concatenates three feature maps of different sizes to the same size, and its calculation expression is as follows:

[0130]

[0131] in, This represents the result of feature fusion from the third convolution. , and These represent the second, third, and fourth 3D convolutional feature extraction results, respectively, which are feature maps of three different sizes.

[0132] A333. Use the hourglass unit to process the fusion result of the third convolution feature to obtain the hourglass processing result, and output the hourglass processing result to the regression unit;

[0133] The hourglass unit consists of a single hourglass network and two 3D deconvolutions, thus obtaining the cost volume of the ACV module. ;

[0134] A334. Using regression units to process the hourglass results, we obtain motion characteristic results from different perspectives.

[0135] A34. Use the ACV fusion unit to process the motion feature results of each second sub-subbody and different viewpoints to obtain the ACV fusion result, and output the ACV fusion result to the second subbody construction unit.

[0136] Enhance the attention-effect-rich ACV fusion results by adding different weights to the first sub-entity and motion feature results from different viewpoints:

[0137]

[0138] in, This indicates the number of channels in the current stage. j This indicates the depth of the assumptions at the current stage. Indicates the first The first under the channel j The amount of homography cost at different depths. This represents the weight set of the two regression results. Indicates the first The first under the channel j The amount of homography cost at different depths. Indicates the first The adaptive cost under the channel, where, Typically, the cost is set to 0.7; through similarity learning, the ACVMVS network can learn sparse and inconspicuous features and adaptively adjust the cost amount.

[0139] A35. Process the ACV fusion results using the second cost body construction unit to obtain the second cost body, and output the second cost body to the second variance cost body measurement unit;

[0140] A36. Process the second cost body using the second variance cost body measurement unit to obtain the second variance measurement result, and output the second variance measurement result to the second cost body regression unit;

[0141] A37. Process the second variance metric result using the second cost body regression unit to obtain the second cost body regression result, and output the second cost body regression result to the second depth regression unit; the image data format of the second cost body regression result is (1,32,256,320).

[0142] A38. The second cost body regression result is processed using the second depth regression unit to obtain the second-stage depth map; the image data format of the second-stage depth map is (1, 256, 320).

[0143] A4. Input the second-stage depth map and the third feature extraction results into the third MVS pipeline module to obtain the predicted depth map corresponding to each 3D reconstructed image group, and complete the training of the ACVMVS network.

[0144] Step A4 includes the following steps:

[0145] A41. Input the second-stage depth map and the third feature extraction result into the third homography transformation unit to obtain the third cost body, and output the third cost body to the third sub-cost body unit; Based on the third feature extraction result and the second-stage depth map, three images are input into the third homography transformation unit, and their image data formats are ref: (8,512,640) and src: (2, 8,512,640) respectively. Eight pseudo depths are faked by the homography formula, that is, the image data format of the second cost body is (8,8,512,640).

[0146] A42. Process the third cost body using the third sub-cost body unit to obtain several third sub-cost bodies, and output each third sub-cost body sequentially to the third cost body construction unit;

[0147] A43. Use the third cost body construction unit to splice together each third sub-cost body to obtain the third cost body, and output the third cost body to the third difference cost body measurement unit;

[0148] A44. Process the third cost body using the third-party difference cost body measurement unit to obtain the third-party difference measurement result, and output the third-party difference measurement result to the third cost body regression unit;

[0149] A45. The third cost volume regression unit processes the third difference measurement result to obtain the third cost volume regression result, and outputs the third cost volume regression result to the third depth regression unit; the image data format of the third cost volume regression result is (1,8,512,640).

[0150] A46. The third depth regression unit is used to process the regression results of the third cost body to obtain the predicted depth map; the image data format of the predicted depth map is (1, 512, 640).

[0151] In this scheme, loss calculations were performed on the first MVS pipeline module, the second MVS pipeline module, the ACV module, and the third MVS pipeline module, respectively. The smoothing loss of each layer was used to calculate the average absolute difference between the actual depth map and the estimated depth map. The final loss is the sum of the three cascaded weights obtained through weight allocation, and its calculation expression is as follows:

[0152]

[0153] in, Indicates the first A set of real pixels that are effectively represented at the level of the surface. This represents the true ground depth value of pixel P. This represents the initial depth estimate. Indicates the first Loss weights corresponding to cascaded features.

[0154] S4. Use the trained ACVMVS network to process the 3D reconstruction image group to be reconstructed and obtain the predicted depth map corresponding to the 3D reconstruction image group.

[0155] S5. Based on the predicted depth map, complete the adaptive cost volume 3D reconstruction based on the ACVMVS network.

[0156] The beneficial effects of this invention are as follows: This invention provides an adaptive cost volume 3D reconstruction method based on an ACVMVS network. It learns rich information about the cost of 3D reconstructed images by using patches of different sizes at different feature levels. The adaptive cost volume (ACV) module enhances textures at different hypothetical depths, learns sparse and inconspicuous features, and adaptively adjusts the cost. Using predefined sizes and adaptively learned weights, it calculates matching costs in feature maps at different levels, maintains the number of pixels at the center of convolutional blocks, and obtains similarity based on the correlation between patch pixels in different blocks. Simultaneously, it maintains the correlation between the range of convolutional blocks and the feature maps, enabling the understanding of motion structures between different views during multi-view stereo matching. This enriches the detailed features of point cloud reconstruction, improving the completeness and accuracy of point cloud reconstruction. Furthermore, this scheme adds adaptive 2D variable convolutional units to the feature extraction module, allowing the feature extraction network to adaptively learn structural information of different sizes and scenes, improving the network's generalization ability and robustness, resulting in excellent performance on other datasets.

[0157] Example 2

[0158] In one embodiment of the invention, the model is trained on a DTU training platform. During the training phase, the number of input images N = 5, and the image resolution is 512 × 640. For coarse-to-fine regularization, the sampling range of the depth assumptions is from 425 mm to 935 mm. The number of planar scan depth assumptions for each stage are 48, 32, and 8, respectively. The corresponding depth intervals decay by 0.25 and 0.5 from the coarsest stage to the most precise stage. The model is trained with Adam for 10 epochs with an initial learning rate of 0.001, decaying by 0.5 times after 6 and 8 epochs, respectively. The batch size on the Nvidia RTX 3090 GPU is 2, and one batch consumes up to 9GB of memory.

[0159] This approach achieves very high-quality point cloud integrity by fine-tuning weak textures through the proposed ACV module and by using adaptive receptive fields to perform meaningful automatic fine-tuning of features.

[0160] Because of the ACV module's ability to learn detailed structural information, it can grasp the motion structure between different views during multi-view stereo matching, thereby enriching the detailed features of point cloud reconstruction and improving the completeness and accuracy of point cloud reconstruction. In addition, an adaptive 2D convolutional layer is added to the feature extraction module in this invention, allowing the network to adaptively learn structural information of different sizes and different scenes, improving the generalization ability and robustness of our network, and enabling it to achieve excellent performance on other datasets.

Claims

1. An adaptive cost volume based 3D reconstruction method of ACVMVS network, characterized in that, The method comprises the following steps: S1, obtaining a three-dimensional reconstruction picture group training set; S2, constructing an ACVMVS network; S3, inputting the three-dimensional reconstruction picture group training set into the ACVMVS network for training; S4, processing a three-dimensional reconstruction picture group to be reconstructed by using the trained ACVMVS network to obtain a predicted depth map corresponding to the three-dimensional reconstruction picture group; S5, completing adaptive cost volume three-dimensional reconstruction based on the ACVMVS network based on the predicted depth map; The ACVMVS network comprises a feature extraction module, a first MVS pipeline module, a second MVS pipeline module, an ACV module and a third MVS pipeline module; An input end of the feature extraction module serves as an input end of the ACVMVS network; a first output end of the feature extraction module is connected with an input end of the first MVS pipeline module; a second output end of the feature extraction module is connected with an output end of the first MVS pipeline module and an input end of the second MVS pipeline module respectively; an ACV output end of the second MVS pipeline module is connected with an input end of the ACV module; an ACV input end of the second MVS pipeline module is connected with an output end of the ACV module; a third output end of the feature extraction module is connected with an output end of the second MVS pipeline module and an input end of the third MVS pipeline module respectively; and an output end of the third MVS pipeline module serves as an output end of the ACVMVS network; The training method of the ACVMVS network comprises the following steps: A1, inputting each three-dimensional reconstruction picture group into the feature extraction module in sequence to obtain corresponding first feature extraction results, second feature extraction results and third feature extraction results respectively; A2, inputting the first feature extraction results into the first MVS pipeline module to obtain first-stage depth maps; A3, inputting the first-stage depth maps and the second feature extraction results into the second MVS pipeline module and obtaining second-stage depth maps based on the ACV module; A4, inputting the second-stage depth maps and the third feature extraction results into the third MVS pipeline module to obtain predicted depth maps corresponding to each three-dimensional reconstruction picture group, and completing training of the ACVMVS network.

2. The adaptive cost volume based 3D reconstruction method of claim 1, wherein, The three-dimensional reconstruction picture group training set comprises a plurality of three-dimensional reconstruction picture groups; each three-dimensional reconstruction picture group comprises three different-view pictures of the same object.

3. The adaptive cost volume based 3D reconstruction method of claim 1, wherein, The feature extraction module comprises a deformable 2D convolution unit, a first 2D convolution unit, a second 2D convolution unit, a third 2D convolution unit, a fourth 2D convolution unit, a fifth 2D convolution unit, a first Concat fusion unit and a second Concat fusion unit. The input end of the deformable 2D convolution unit is an input end of a feature extraction module; the output end of the deformable 2D convolution unit is connected with the input end of a first 2D convolution unit and the second input end of a second Concat fusion unit respectively; the output end of the first 2D convolution unit is connected with the input end of a second 2D convolution unit and the second input end of a first Concat fusion unit respectively; the output end of the second 2D convolution unit is connected with the input end of a third 2D convolution unit; the first output end of the third 2D convolution unit is a first output end of the feature extraction module and is connected with the input end of a first MVS pipeline module; the second output end of the third 2D convolution unit is connected with the first input end of the first Concat fusion unit; the output end of the first Concat fusion unit is connected with the input end of a fourth 2D convolution unit; the first output end of the fourth 2D convolution unit is a second output end of the feature extraction module and is connected with the input end of a second MVS pipeline module; the second output end of the fourth 2D convolution unit is connected with the first input end of a second Concat fusion unit; the output end of the second Concat fusion unit is connected with the input end of a fifth 2D convolution unit; the output end of the fifth 2D convolution unit is a third output end of the feature extraction module and is connected with the input end of a third MVS pipeline module; The first MVS pipeline module comprises a first homography change unit, a first sub-cost volume unit, a first cost volume construction unit, a first variance cost volume measurement unit, a first cost volume regression unit and a first depth regression unit connected in sequence, wherein the input end of the first homography change unit is an input end of the first MVS pipeline module and is connected with the first output end of the third 2D convolution unit, and the output end of the first depth regression unit is an output end of the first MVS pipeline module and is connected with the input end of the second MVS pipeline module; The second MVS pipeline module comprises a second homography change unit, a second sub-cost volume unit, an ACV fusion unit, a second cost volume construction unit, a second variance cost volume measurement unit, a second cost volume regression unit and a second depth regression unit; The input end of the second homography change unit is connected with the output end of the first depth regression unit and the first output end of the fourth 2D convolution unit respectively; the first output end of the second homography change unit is connected with the input end of the second sub-cost volume unit; the second output end of the second homography change unit is the ACV output end of the second MVS pipeline module and is connected with the input end of the ACV module; the output end of the second sub-cost volume unit is connected with the first input end of the ACV fusion unit; the second input end of the ACV fusion unit is the ACV input end of the second MVS pipeline module and is connected with the output end of the ACV module; the output end of the ACV fusion unit is connected with the input end of the second cost volume construction unit; the output end of the second cost volume construction unit is connected with the input end of the second variance cost volume measurement unit; the output end of the second variance cost volume measurement unit is connected with the input end of the second cost volume regression unit; the output end of the second cost volume regression unit is connected with the input end of the second depth regression unit; the output end of the second depth regression unit is the output end of the second MVS pipeline module and is connected with the input end of the third MVS pipeline module; The ACV module comprises a first 3D convolution unit, a second 3D convolution unit, a third 3D convolution unit, a fourth 3D convolution unit, a third Concat fusion unit, an hourglass unit and a regression unit; The input end of the first 3D convolution unit is the input end of the ACV module and is connected with the second output end of the second homography change unit; the first output end of the first 3D convolution unit is connected with the input end of the second 3D convolution unit; the second output end of the first 3D convolution unit is connected with the input end of the third 3D convolution unit; the third output end of the first 3D convolution unit is connected with the input end of the fourth 3D convolution unit; the output end of the second 3D convolution unit, the output end of the third 3D convolution unit and the output end of the fourth 3D convolution unit are all connected with the input end of the third Concat fusion unit; the output end of the third Concat fusion unit is connected with the input end of the hourglass unit; the output end of the hourglass unit is connected with the input end of the regression unit; the output end of the regression unit is the output end of the ACV module and is connected with the second input end of the ACV fusion unit; The third MVS pipeline module comprises a third homography change unit, a third sub-cost volume unit, a third cost volume construction unit, a third variance cost volume measurement unit, a third cost volume regression unit and a third depth regression unit connected in sequence, wherein the input end of the third homography change unit is the input end of the third MVS pipeline module and is connected with the output end of the second depth regression unit and the output end of the fifth 2D convolution unit respectively, and the output end of the third depth regression unit is the output end of the third MVS pipeline module.

4. The adaptive cost volume based 3D reconstruction method of claim 1, wherein, The step A1 comprises the following steps: A11, input each three-dimensional reconstruction picture group into a deformable 2D convolution unit in sequence to obtain a deformable convolution feature extraction result, and output the deformable convolution feature extraction result to a first 2D convolution unit and a second Concat fusion unit; A12, process the deformable convolution feature extraction result by using the first 2D convolution unit to obtain a first 2D convolution feature extraction result, and output the first 2D convolution feature extraction result to a second 2D convolution unit and a first Concat fusion unit; A13, process the first 2D convolution feature extraction result by using the second 2D convolution unit to obtain a second 2D convolution feature extraction result, and output the second convolution feature extraction result to a third 2D convolution unit; A14, process the second 2D convolution feature extraction result by using the third 2D convolution unit to obtain a third 2D convolution feature extraction result, and output the third 2D convolution feature extraction result to the first Concat fusion unit, and take the third 2D convolution feature extraction result as a first feature extraction result; A15, fuse the first 2D convolution feature extraction result and the third 2D convolution feature extraction result by using the first Concat fusion unit to obtain a first convolution feature fusion result, and output the first convolution feature fusion result to a fourth 2D convolution unit; A16, process the first convolution feature fusion result by using the fourth 2D convolution unit to obtain a fourth convolution feature extraction result, and output the fourth convolution feature extraction result to the second Concat fusion unit, and take the fourth convolution feature extraction result as a second feature extraction result; A17, fuse the deformable convolution feature extraction result and the fourth convolution feature extraction result by using the second Concat fusion unit to obtain a second convolution feature fusion result, and output the second convolution feature fusion result to a fifth 2D convolution unit; A18, process the second convolution feature fusion result by using the fifth 2D convolution unit to obtain a third feature extraction result.

5. The adaptive cost volume based 3D reconstruction method of claim 4, wherein, The step A2 includes the following sub-steps: A21, input the first feature extraction result into a first homography transformation unit to obtain a first cost volume, and output the first cost volume to a first sub-cost volume unit; A22, process the first cost volume by using the first sub-cost volume unit to obtain a plurality of first sub-cost volumes, and output each first sub-cost volume to a first cost volume construction unit in sequence; A23, splice each first sub-cost volume by using the first cost volume construction unit to obtain a first cost volume, and output the first cost volume to a first variance cost volume measurement unit; A24, process the first cost volume by using the first variance cost volume measurement unit to obtain a first variance measurement result, and output the first variance measurement result to a first cost volume regression unit; A25, process the first variance measurement result by using the first cost volume regression unit to obtain a first cost volume regression result, and output the first cost volume regression result to a first depth regression unit; A26, process the first cost volume regression result by using the first depth regression unit to obtain a first stage depth map.

6. The adaptive cost volume based 3D reconstruction method of claim 5, wherein, The step A3 includes the following steps: A31, input the first-stage depth map and the second feature extraction result into a second homography change unit to obtain a second cost volume, and output the second cost volume to a second sub-cost volume unit and an ACV module respectively; A32, process the second cost volume by using the second sub-cost volume unit to obtain a plurality of second sub-cost volumes, and output each second sub-cost volume to an ACV fusion unit in sequence; A33, process the second cost volume by using the ACV module to obtain different-view motion feature results, and output the different-view motion feature results to the ACV fusion unit; A34, process each second sub-cost volume and the different-view motion feature results by using the ACV fusion unit to obtain ACV fusion results, and output the ACV fusion results to a second cost volume construction unit; A35, process the ACV fusion results by using the second cost volume construction unit to obtain a second cost volume, and output the second cost volume to a second variance cost volume measurement unit; A36, process the second cost volume by using the second variance cost volume measurement unit to obtain a second variance measurement result, and output the second variance measurement result to a second cost volume regression unit; A37, process the second variance measurement result by using the second cost volume regression unit to obtain a second cost volume regression result, and output the second cost volume regression result to a second depth regression unit; A38, process the second cost volume regression result by using the second depth regression unit to obtain a second-stage depth map.

7. The adaptive cost volume based 3D reconstruction method of claim 6, wherein, The step A33 comprises the following steps: A331, input the second cost volume into a first 3D convolution unit to obtain a first 3D convolution feature extraction result, and output the first 3D convolution feature extraction result to a second 3D convolution unit, a third 3D convolution unit and a fourth 3D convolution unit respectively; A332, process the first 3D convolution feature extraction result by using the second 3D convolution unit, the third 3D convolution unit and the fourth 3D convolution unit respectively to obtain a second 3D convolution feature extraction result, a third 3D convolution feature extraction result and a fourth 3D convolution feature extraction result; A333, output the second 3D convolution feature extraction result, the third 3D convolution feature extraction result and the fourth 3D convolution feature extraction result to a third Concat fusion unit synchronously for feature fusion to obtain a third convolution feature fusion result, and output the third convolution feature fusion result to a sandglass unit; A333, process the third convolution feature fusion result by using the sandglass unit to obtain a sandglass processing result, and output the sandglass processing result to a regression unit; A334, process the sandglass result by using the regression unit to obtain different-view motion feature results.

8. The adaptive cost volume based 3D reconstruction method of claim 7, wherein, The step A4 comprises the following steps: A41, input the second-stage depth map and a third feature extraction result into a third homography change unit to obtain a third cost volume, and output the third cost volume to a third sub-cost volume unit; A42, process the third cost volume by using the third sub-cost volume unit to obtain a plurality of third sub-cost volumes, and output each third sub-cost volume to a third cost volume construction unit in sequence; A43, splicing each third sub-cost volume by using the third cost volume constructing unit to obtain a third cost volume, and outputting the third cost volume to a third variance cost volume measuring unit; A44, processing the third cost volume by using the third variance cost volume measuring unit to obtain a third variance measurement result, and outputting the third variance measurement result to a third cost volume regression unit; A45, processing the third variance measurement result by using the third cost volume regression unit to obtain a third cost volume regression result, and outputting the third cost volume regression result to a third depth regression unit; A46, processing the third cost volume regression result by using the third depth regression unit to obtain a predicted depth map.

Images