A three-dimensional reconstruction method and system based on deep learning
By introducing a cross-attention mechanism into the depth image acquisition model of MVSNet, the problems of low accuracy and efficiency in 3D reconstruction are solved, and higher accuracy and more efficient 3D reconstruction results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG SHUNDE INDUSTRY DESIGN INSTITUTE (GUANGDONG SHUNDE INNOVATIVE DESIGN INSTITUTE)
- Filing Date
- 2022-08-02
- Publication Date
- 2026-07-21
AI Technical Summary
Existing deep learning-based 3D reconstruction methods suffer from low reconstruction accuracy, low efficiency, and the influence of redundant information. In particular, when using MVSNet, key information is easily lost and redundant information is present.
Based on the multi-view stereo vision neural network MVSNet, a cross-attention mechanism is introduced. By improving the feature extraction layer and the cost space regularization layer, and combining the depth image acquisition model, the information capture and matching performance is optimized, thereby improving the accuracy and efficiency of 3D reconstruction.
It improves the accuracy and efficiency of 3D reconstruction models, reduces the loss of key information and the impact of redundant information, generates more accurate depth images, and makes the 3D reconstruction model fit the target object better.
Smart Images
Figure CN115330935B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D reconstruction technology, and in particular to a 3D reconstruction method and system based on deep learning. Background Technology
[0002] 3D reconstruction can reflect complete scene information in an image and has wide applications in industrial automation, virtual reality, and visual navigation. Traditional 3D model reconstruction methods are based on whether the sensor actively illuminates the object, and can be divided into passive and active 3D reconstruction methods. Passive 3D reconstruction methods use the geometric features of the object itself during imaging projection to inversely calculate the object's depth information. Their limitation lies in the need for strict priors and constraints. Active 3D reconstruction methods, on the other hand, require sophisticated hardware, resulting in high costs, and the equipment's performance is easily affected by environmental factors, leading to inaccurate feature extraction and longer processing times. In recent years, with the rapid development of deep learning, Convolutional Neural Networks (CNNs) have gradually been applied to the field of 3D reconstruction. Early neural network methods mainly dealt with the matching problem of corresponding points between two images and calculated the similarity between the two points through the network.
[0003] Currently, common 3D reconstruction methods based on convolutional neural networks include: SurfaceNet and Multi-view Stereo (MVSNet), and the Learned Stereo Machine (LSM) model. SurfaceNet first reconstructs a color voxel cube, then uses the color information of all pixels and camera parameters to construct a 3D cost volume. This 3D cost volume serves as the network's input, and the output is the 3D reconstruction result. This segmentation-based 3D reconstruction method is time-consuming. While LSM directly adopts an end-to-end learning approach, like SurfaceNet, it requires regular grid representation, which suffers from spatial discretization errors and memory consumption. Furthermore, LSM uses manual synthesis, resulting in lower stereo resolution. Additionally, the end-to-end multi-view depth prediction network MVSNet, similar to other depth image-based MVS methods, uses a reference image and multiple source images as input to predict depth images for the reference image. Its key feature is the differentiable homography transformation operation, which allows camera parameters to be implicitly encoded into the network during the construction of the 3D cost volume from the 2D feature map. Compared to other deep learning-based 3D reconstruction methods, MVSNet constructs the 3D cost volume based on the camera's view frustum rather than a regular Euclidean space, and divides the overall 3D reconstruction into a view-by-view depth estimation problem, making large-scale reconstruction possible. However, in the process of acquiring depth information, MVSNet is prone to the loss of key information and the presence of redundant information, which in turn affects the overall reconstruction accuracy and efficiency. Summary of the Invention
[0004] This invention provides a deep learning-based 3D reconstruction method and system, which improves the overall 3D reconstruction accuracy and efficiency when performing 3D reconstruction based on a large number of images.
[0005] To address the aforementioned technical problems, embodiments of the present invention provide a deep learning-based 3D reconstruction method, comprising:
[0006] Acquire multiple images of the target object to be processed;
[0007] All the images to be processed are input into the depth image acquisition model to obtain the depth image corresponding to each image to be processed.
[0008] By using a preset 3D reconstruction algorithm and combining all the depth images, a 3D reconstruction model of the target object is obtained, thereby realizing the 3D reconstruction of the target object.
[0009] The depth image acquisition model is obtained by introducing a cross-attention mechanism on the basis of a preset multi-view stereo vision neural network MVSNet.
[0010] By implementing this embodiment of the invention, the pre-defined multi-view stereo vision neural network MVSNet is improved by introducing a cross-attention mechanism. This allows the depth image acquisition model to better process multiple images corresponding to the target object, thereby optimizing the information capture and matching performance of the depth image acquisition model and improving the accuracy and efficiency of 3D reconstruction of the target object. Furthermore, by acquiring multiple depth information of the target object image by image, the basis for 3D reconstruction of the target object can be enriched, further improving its 3D reconstruction accuracy.
[0011] As a preferred embodiment, the step of inputting all the images to be processed into a depth image acquisition model to obtain the depth image corresponding to each image to be processed specifically involves:
[0012] All the images to be processed are input into the depth image acquisition model. Through the feature extraction layer, the local features and global features corresponding to each image to be processed are extracted respectively, and the local features and the global features are fused to obtain the depth features corresponding to each image to be processed.
[0013] By using a homography transformation layer, differentiable homography transformation is performed on each of the depth features to obtain the homography matrix corresponding to each of the images to be processed, and a first cost space corresponding to each homography matrix is constructed based on each homography matrix.
[0014] By using a cost space regularization layer, each of the first cost spaces is regularized to obtain the second cost space corresponding to each of the first cost spaces and the probability volume corresponding to each of the second cost spaces.
[0015] By acquiring a depth image layer, and according to a preset algorithm, combined with each probability volume, the depth image corresponding to each image to be processed is calculated.
[0016] The depth image acquisition model sequentially includes the feature extraction layer, the homography transformation layer, the cost space regularization layer, and the depth image acquisition layer.
[0017] In a preferred embodiment of the present invention, the feature extraction layer of the depth image acquisition model can capture not only the global features of the image to be processed, but also its local features, and fuse its local and global features to reduce the loss of key information and the occupation of redundant information, thereby improving the feature extraction accuracy and reducing the overall computational load.
[0018] As a preferred embodiment, the acquisition of the depth image acquisition model is specifically as follows:
[0019] In the feature extraction stage of the multi-view stereo vision neural network MVSNet, the cross-attention mechanism is introduced to form the feature extraction layer of the depth image acquisition model. In the cost space regularization stage of the multi-view stereo vision neural network MVSNet, the cross-attention mechanism is introduced to form the cost space regularization layer of the depth image acquisition model, thereby obtaining the depth image acquisition model.
[0020] The cross-attention mechanism is implemented using the Transformer model.
[0021] In a preferred embodiment of the present invention, Transformer technology is introduced in the feature extraction stage and the cost space regularization stage to form a feature extraction layer and a cost space regularization layer of the depth image acquisition model. This enables the depth image acquisition model to better extract key information from the image to be processed, avoiding the negative impact of redundant information on the construction accuracy of the 3D model and excessive data on the processing efficiency of the depth image acquisition model. Furthermore, through the encoder-decoder structure of the cost space regularization layer, adjacent key information is gathered from a large receptive field, thereby improving the accuracy of the generated depth image and further improving the construction accuracy of the 3D reconstruction model.
[0022] As a preferred embodiment, the step of obtaining a 3D reconstruction model of the target object by combining all the depth images using a preset 3D reconstruction algorithm, thereby achieving 3D reconstruction of the target object, specifically involves:
[0023] A preset 3D reconstruction algorithm is used to fuse the depth image and the image to be processed from the same viewpoint to obtain a fused image corresponding to each depth image.
[0024] Based on all the fused images, the three-dimensional point cloud data of the target object is obtained, and the three-dimensional point cloud data is used to construct the three-dimensional reconstruction model of the target object, thereby realizing the three-dimensional reconstruction of the target object.
[0025] A preferred embodiment of the present invention involves fusing a depth image and an image to be processed from the same viewpoint, converting the fused image into three-dimensional point cloud data, and then visualizing the three-dimensional point cloud data to achieve three-dimensional reconstruction of the target object. This can improve the accuracy of the point cloud data and enhance the fit between the constructed three-dimensional reconstruction model and the target object.
[0026] As a preferred embodiment, the acquisition of multiple images to be processed corresponding to the target object specifically includes:
[0027] The target object is photographed from different perspectives using a binocular camera to obtain multiple single-view images;
[0028] All the single-view images are preprocessed to obtain the image to be processed corresponding to each single-view image.
[0029] A preferred embodiment of the present invention involves capturing images of the target object from different perspectives to obtain multiple single-view images, which are then preprocessed to obtain images to be processed corresponding to each single-view image. This provides image data from multiple perspectives for subsequent 3D model construction, avoiding the problem of low 3D reconstruction accuracy caused by constructing a 3D model of the target object based on only a single perspective.
[0030] To address the same technical problem, embodiments of the present invention also provide a deep learning-based 3D reconstruction system, comprising:
[0031] The data acquisition module is used to acquire multiple images of the target object to be processed.
[0032] The data processing module is used to input all the images to be processed into the depth image acquisition model to obtain the depth image corresponding to each image to be processed; wherein, the depth image acquisition model is obtained by introducing a cross attention mechanism on the basis of a preset multi-view stereo vision neural network MVSNet;
[0033] The 3D reconstruction module is used to obtain a 3D reconstruction model of the target object by combining all the depth images using a preset 3D reconstruction algorithm, thereby realizing the 3D reconstruction of the target object.
[0034] As a preferred embodiment, the data processing module specifically includes:
[0035] The feature extraction unit is used to extract local features and global features corresponding to each of the images to be processed through the depth image acquisition model, and fuse the local features and the global features to obtain the depth features corresponding to each of the images to be processed.
[0036] The homography transformation unit is used to perform differentiable homography transformation on each of the depth features to obtain the homography matrix corresponding to each of the images to be processed, and to construct a first cost space corresponding to each homography matrix based on each homography matrix.
[0037] The cost space regularization unit is used to regularize each of the first cost spaces to obtain the second cost space corresponding to each of the first cost spaces and the probability volume corresponding to each of the second cost spaces.
[0038] The depth image acquisition unit is used to calculate the depth image corresponding to each of the images to be processed according to a preset algorithm and in combination with each probability volume.
[0039] As a preferred embodiment, the deep learning-based 3D reconstruction system further includes:
[0040] The model building module is used to introduce the cross-attention mechanism in the feature extraction stage and the cost space regularization stage of the multi-view stereo vision neural network MVSNet, respectively, to obtain the depth image acquisition model; wherein, the cross-attention mechanism is implemented through the Transformer model.
[0041] As a preferred embodiment, the three-dimensional reconstruction module specifically includes:
[0042] The fusion processing unit is used to perform fusion processing on the depth image and the image to be processed from the same viewpoint using a preset three-dimensional reconstruction algorithm, so as to obtain a fused image corresponding to each depth image;
[0043] The three-dimensional reconstruction unit is used to acquire three-dimensional point cloud data of the target object based on all the fused images, and to construct the three-dimensional reconstruction model of the target object using the three-dimensional point cloud data, thereby realizing the three-dimensional reconstruction of the target object.
[0044] As a preferred embodiment, the data acquisition module specifically includes:
[0045] The data acquisition unit is used to capture images of the target object from different perspectives using a binocular camera to obtain multiple single-view images;
[0046] The preprocessing unit is used to preprocess all the single-view images to obtain the image to be processed corresponding to each single-view image. Attached Figure Description
[0047] Figure 1 This is a flowchart illustrating a deep learning-based 3D reconstruction method provided in Embodiment 1 of the present invention.
[0048] Figure 2 : A schematic diagram of the structure of the depth image acquisition model provided in Embodiment 1 of the present invention;
[0049] Figure 3 : A schematic diagram of the structure of the 2D U-Transformer feature extraction layer provided in Embodiment 1 of the present invention;
[0050] Figure 4 : This is a schematic diagram of the structure of a three-dimensional reconstruction system based on deep learning provided in Embodiment 1 of the present invention. Detailed Implementation
[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0052] Example 1:
[0053] Please refer to Figure 1 This invention provides a deep learning-based 3D reconstruction method, which includes steps S1 to S3, as detailed below:
[0054] Step S1: Obtain multiple images to be processed corresponding to the target object.
[0055] As a preferred embodiment, step S1 includes steps S11 to S12, each of which is detailed below:
[0056] Step S11: Using a binocular camera, take pictures of the target object from different perspectives to obtain multiple single-view images.
[0057] Step S12: Preprocess all single-view images to obtain the image to be processed corresponding to each single-view image.
[0058] Step S2: Input all images to be processed into the depth image acquisition model to obtain the depth image corresponding to each image to be processed; wherein, the depth image acquisition model is obtained by introducing a cross-attention mechanism on the basis of the preset multi-view stereo vision neural network MVSNet.
[0059] As a preferred embodiment, step S2 includes steps S21 to S24, each of which is detailed below:
[0060] Step S21, please refer to Figure 2 All images to be processed are input into the depth image acquisition model. The 2D U-Transformer feature extraction layer extracts the local and global features corresponding to each image to be processed, and then fuses the local and global features to obtain the depth features corresponding to each image to be processed. The depth image acquisition model consists of the 2D U-Transformer feature extraction layer, the homography transformation layer, the 3D U-Transformer cost space regularization layer, and the depth image acquisition layer.
[0061] In this embodiment, please refer to Figure 3The feature extraction layer, 2D U-Transformer, consists of an encoder, a bottleneck, a decoder, and a skip connection.
[0062] When the image to be processed is input into the feature extraction layer 2D U-Transformer of the depth image acquisition model, the encoder learns representations of the image while keeping the feature dimension and resolution unchanged. Simultaneously, the Patch Merging layer divides the patches of the image into four parts and concatenates these patches together. At this point, the feature resolution is downsampled by a factor of 2, and the feature dimension increases by a factor of 4 due to the concatenation operation. Therefore, a linear layer is applied to the concatenated features to unify the feature dimension to twice the original dimension. This downsampling process of the Patch Merging layer is repeated three times in the encoder. Next, because the Transformer is too deep to converge, only two consecutive Transformer Blocks are used to construct a bottleneck to learn deep feature representations. In the bottleneck, the feature dimension and resolution remain unchanged. Then, corresponding to the encoder, the Patch Expanding layer in the decoder upsamples the deep features output from the bottleneck. The Patch Expanding layer reshapes the feature maps of adjacent dimensions into higher-resolution feature maps (upsampled by a factor of 2), and correspondingly reduces the feature dimension to half of the original dimension. Finally, the skip connection fuses the multi-scale features from the encoder with the upsampled features from the decoder to obtain the depth features corresponding to the image to be processed. This means connecting the shallow and deep features together to reduce the loss of spatial information caused by downsampling.
[0063] It should be noted that the Transformer Block uses a cross attention mechanism to alternate attention within a patch rather than across the entire image to capture its local features, and applies attention between patches divided from the corresponding single-channel feature maps of the image to capture its global information. This achieves the goal of reducing computation while maintaining feature extraction performance, thereby further improving the overall efficiency of 3D construction.
[0064] Step S22: Through the homography transformation layer, differentiable homography transformation is performed on each depth feature to obtain the homography matrix corresponding to each image to be processed, and the first cost space V′ corresponding to each homography matrix is constructed based on each homography matrix.
[0065] In this embodiment, differentiable homography transformation is used to describe the positional mapping relationship of the object between the world coordinate system and the pixel coordinate system. Each depth feature is transformed into the cone-shaped solid space of the reference camera, forming N+1 cost spaces V′, where N is the patch size in Cross Attention. N+1 feature maps simultaneously generate N+1 cost spaces at the same spatial location in the cone-shaped space of the reference camera.
[0066] It should be noted that the formulas for homography transformation at different depths are given in Equation (1), and the definition of the homography matrix H is given in Equation (2).
[0067] x′~H i (d)·x (1)
[0068]
[0069] Where x is the pixel before projection, x′ is the pixel after projection, ~ represents the projection operation, and H i (d) is the homography matrix of the i-th depth feature map to the depth d plane of the reference image camera frustum, H i Please refer to formula (3) for the calculation formula of (d).
[0070]
[0071] Among them, K i R i and t i Let I be the camera intrinsic parameter matrix, rotation matrix, and translation matrix of the i-th viewpoint image, respectively, where I is the identity matrix and d is the depth value.
[0072] Step S23: Regularize each first cost space through the cost space regularization layer (3D U-Transformer) to obtain the second cost space corresponding to each first cost space and the probability volume corresponding to each second cost space.
[0073] In this embodiment, the first cost space is regularized using a 3D U-Transformer to obtain the second cost space corresponding to each first cost space. Then, along the depth dimension, the softmax operation is used to generate the corresponding probability volume P.
[0074] Step S24: Obtain the layer through the depth image, and calculate the depth image corresponding to each image to be processed according to the preset algorithm and in combination with each probability volume.
[0075] In this embodiment, in order to generate continuous depth estimates, the depth E of each pixel is estimated by using the SoftArgmin operation through the depth image acquisition layer, so as to form the Predicted Depth Map corresponding to each image to be processed. The estimation formula of depth E can be found in Equation (4).
[0076]
[0077] Where P(d) represents the estimated probability of all pixels with a depth assumed to be d, [d min ,d max [ ] represents the range of sampling depth values.
[0078] It should be noted that, referring to equation (5), the average mean pair error of the prediction results for each depth image should be calculated. To reflect the accuracy of depth image prediction results, and to utilize the average pairwise error. The predicted depth maps of each depth image are optimized to form the optimized GT depth maps corresponding to the predicted depth images of each depth image. The optimized results are then used as the final depth maps of each image to be processed, thereby providing accurate depth maps for subsequent 3D reconstruction of the target object.
[0079]
[0080] Where, x valid Let d(x) represent the effective pixel set in the ground reality and These represent the predicted depth of the depth image corresponding to the Predicted Depth Map and the actual ground depth, respectively.
[0081] Step S3: Using a preset 3D reconstruction algorithm, combined with all depth images, a 3D reconstruction model of the target object is obtained, thereby realizing the 3D reconstruction of the target object.
[0082] As a preferred embodiment, step S3 includes steps S31 to S32, each of which is detailed below:
[0083] Step S31: Using a preset 3D reconstruction algorithm, the depth image and the image to be processed from the same viewpoint are fused to obtain the fused image corresponding to each depth image.
[0084] Step S32: Based on all fused images, obtain the three-dimensional point cloud data of the target object, and use the three-dimensional point cloud data to construct a three-dimensional reconstruction model of the target object, thereby realizing the three-dimensional reconstruction of the target object.
[0085] As a preferred solution, the acquisition process of the depth image acquisition model is step S4, as follows:
[0086] Step S4: In the feature extraction stage of the multi-view stereo vision neural network MVSNet, a cross-attention mechanism is introduced to form the feature extraction layer of the depth image acquisition model. In the cost space regularization stage of the multi-view stereo vision neural network MVSNet, a cross-attention mechanism is introduced to form the cost space regularization layer of the depth image acquisition model, thereby obtaining the depth image acquisition model. The cross-attention mechanism is implemented through the Transformer model.
[0087] In this embodiment, the feature extraction layer of the depth image acquisition model is a 2D U-Transformer, and the cost space regularization layer of the depth image acquisition model is a 3D U-Transformer.
[0088] Please refer to Figure 4 The image below is a schematic diagram of a deep learning-based 3D reconstruction system provided in an embodiment of the present invention. The system includes a data acquisition module 1, a data processing module 2, and a 3D reconstruction module 3, as detailed below:
[0089] Data acquisition module 1 is used to acquire multiple images to be processed corresponding to the target object;
[0090] Data processing module 2 is used to input all images to be processed into the depth image acquisition model to obtain the depth image corresponding to each image to be processed; wherein, the depth image acquisition model is obtained by introducing a cross attention mechanism on the basis of the preset multi-view stereo vision neural network MVSNet;
[0091] The 3D reconstruction module 3 is used to obtain a 3D reconstruction model of the target object by combining all depth images through a preset 3D reconstruction algorithm, thereby realizing the 3D reconstruction of the target object.
[0092] As a preferred embodiment, the data processing module 2 specifically includes a feature extraction unit 21, a homography transformation unit 22, a cost space regularization unit 23, and a depth image acquisition unit 24, with each unit described in detail below:
[0093] The feature extraction unit 21 is used to obtain a model through the depth image, extract the local features and global features corresponding to each image to be processed, and fuse the local features and global features to obtain the depth features corresponding to each image to be processed.
[0094] The homography transformation unit 22 is used to perform differentiable homography transformation on each depth feature to obtain the homography matrix corresponding to each image to be processed, and to construct the first cost space corresponding to each homography matrix based on each homography matrix.
[0095] Cost space regularization unit 23 is used to regularize each first cost space to obtain the second cost space corresponding to each first cost space and the probability volume corresponding to each second cost space.
[0096] The depth image acquisition unit 24 is used to calculate the depth image corresponding to each image to be processed according to a preset algorithm and in combination with each probability volume.
[0097] For the preferred option, please refer to Figure 4 The deep learning-based 3D reconstruction system further includes:
[0098] Model building module 4 is used to introduce cross-attention mechanisms in the feature extraction stage and cost space regularization stage of the multi-view stereo vision neural network MVSNet to obtain a depth image acquisition model; the cross-attention mechanism is implemented through the Transformer model.
[0099] As a preferred embodiment, the 3D reconstruction module 3 specifically includes a fusion processing unit 31 and a 3D reconstruction unit 32, with each unit as follows:
[0100] The fusion processing unit 31 is used to fuse depth images and images to be processed from the same viewpoint using a preset 3D reconstruction algorithm to obtain fused images corresponding to each depth image.
[0101] The 3D reconstruction unit 32 is used to acquire the 3D point cloud data of the target object based on all fused images, and to construct the 3D reconstruction model of the target object using the 3D point cloud data, thereby realizing the 3D reconstruction of the target object.
[0102] As a preferred embodiment, the data acquisition module 1 specifically includes a data acquisition unit 11 and a preprocessing unit 12, the details of which are as follows:
[0103] The data acquisition unit 11 is used to capture images of the target object from different perspectives using a binocular camera to obtain multiple single-view images;
[0104] The preprocessing unit 12 is used to preprocess all single-view images to obtain the image to be processed corresponding to each single-view image.
[0105] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0106] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0107] This invention provides a deep learning-based 3D reconstruction method and system. By introducing a cross-attention mechanism on top of a pre-defined multi-view stereo vision neural network (MVSNet), the information capture and matching performance of the depth image acquisition model is optimized, thereby improving the accuracy of the generated depth images. This further enhances the precision and efficiency of the 3D reconstruction model, making it more closely resemble the target object. Furthermore, multiple depth information of the target object is acquired image by image, and a 3D reconstruction model is constructed based on this information. This allows the 3D reconstruction model to fuse multi-view depth information, thus better representing the target object.
[0108] Furthermore, by utilizing the feature extraction layer of the depth image acquisition model, not only can global features of the image to be processed be captured, but also local features can be captured, and these local and global features can be fused to reduce the loss of key information and the occupation of redundant information, thereby improving feature extraction accuracy and reducing overall computational cost. In addition, through the encoder-decoder structure of the cost space regularization layer, adjacent key information is gathered from a large receptive field, thereby improving the accuracy of the generated depth image and further enhancing the construction accuracy of the 3D reconstruction model.
[0109] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.
Claims
1. A 3D reconstruction method based on deep learning, characterized in that, include: Acquire multiple images of the target object to be processed; All the images to be processed are input into a depth image acquisition model to obtain a depth image corresponding to each image to be processed; wherein, the depth image acquisition model is obtained by introducing a cross-attention mechanism on the basis of a preset multi-view stereo vision neural network MVSNet; the depth image acquisition model includes a feature extraction layer, a homography transformation layer, a cost space regularization layer and a depth image acquisition layer in sequence. The step of inputting all the images to be processed into a depth image acquisition model to obtain a depth image corresponding to each image to be processed includes: All the images to be processed are input into the depth image acquisition model. The feature extraction layer extracts the local and global features corresponding to each image to be processed, and then fuses the local and global features to obtain the depth features corresponding to each image to be processed. The feature extraction layer introduces a cross-attention mechanism. The feature extraction layer includes an encoder, a Bottleneck, a decoder, and skip connections. By using a homography transformation layer, differentiable homography transformation is performed on each of the depth features to obtain the homography matrix corresponding to each of the images to be processed, and a first cost space corresponding to each homography matrix is constructed based on each homography matrix. The first cost space is regularized through a cost space regularization layer to obtain the second cost space corresponding to each first cost space and the probability volume corresponding to each second cost space; the cost space regularization layer introduces a cross-attention mechanism; the cost space regularization layer is a U-Net Transformer structure. By acquiring a depth image layer, and according to a preset algorithm, combined with each probability volume, the depth image corresponding to each image to be processed is calculated. By using a preset 3D reconstruction algorithm and combining all the depth images, a 3D reconstruction model of the target object is obtained, thereby realizing the 3D reconstruction of the target object.
2. The deep learning-based 3D reconstruction method as described in claim 1, characterized in that, The step of obtaining a 3D reconstruction model of the target object by combining all the depth images using a preset 3D reconstruction algorithm, thereby achieving 3D reconstruction of the target object, specifically involves: A preset 3D reconstruction algorithm is used to fuse the depth image and the image to be processed from the same viewpoint to obtain a fused image corresponding to each depth image. Based on all the fused images, the three-dimensional point cloud data of the target object is obtained, and the three-dimensional point cloud data is used to construct the three-dimensional reconstruction model of the target object, thereby realizing the three-dimensional reconstruction of the target object.
3. The deep learning-based 3D reconstruction method as described in claim 1, characterized in that, The acquisition of multiple images to be processed corresponding to the target object specifically involves: The target object is photographed from different perspectives using a binocular camera to obtain multiple single-view images; All the single-view images are preprocessed to obtain the image to be processed corresponding to each single-view image.
4. A three-dimensional reconstruction system based on deep learning, characterized in that, include: The data acquisition module is used to acquire multiple images of the target object to be processed. The data processing module is used to input all the images to be processed into the depth image acquisition model to obtain the depth image corresponding to each image to be processed; wherein, the depth image acquisition model is obtained by introducing a cross-attention mechanism on the basis of a preset multi-view stereo vision neural network MVSNet; the depth image acquisition model includes a feature extraction layer, a homography transformation layer, a cost space regularization layer and a depth image acquisition layer in sequence. The step of inputting all the images to be processed into a depth image acquisition model to obtain a depth image corresponding to each image to be processed includes: All the images to be processed are input into the depth image acquisition model. The feature extraction layer extracts the local and global features corresponding to each image to be processed, and then fuses the local and global features to obtain the depth features corresponding to each image to be processed. The feature extraction layer introduces a cross-attention mechanism. The feature extraction layer includes an encoder, a Bottleneck, a decoder, and skip connections. By using a homography transformation layer, differentiable homography transformation is performed on each of the depth features to obtain the homography matrix corresponding to each of the images to be processed, and a first cost space corresponding to each homography matrix is constructed based on each homography matrix. The first cost space is regularized through a cost space regularization layer to obtain the second cost space corresponding to each first cost space and the probability volume corresponding to each second cost space; the cost space regularization layer introduces a cross-attention mechanism; the cost space regularization layer is a U-Net Transformer structure. By acquiring a depth image layer, and according to a preset algorithm, combined with each probability volume, the depth image corresponding to each image to be processed is calculated. The 3D reconstruction module is used to obtain a 3D reconstruction model of the target object by combining all the depth images using a preset 3D reconstruction algorithm, thereby realizing the 3D reconstruction of the target object.
5. The deep learning-based 3D reconstruction system as described in claim 4, characterized in that, The three-dimensional reconstruction module specifically includes: The fusion processing unit is used to perform fusion processing on the depth image and the image to be processed from the same viewpoint using a preset three-dimensional reconstruction algorithm, so as to obtain a fused image corresponding to each depth image; The three-dimensional reconstruction unit is used to acquire three-dimensional point cloud data of the target object based on all the fused images, and to construct the three-dimensional reconstruction model of the target object using the three-dimensional point cloud data, thereby realizing the three-dimensional reconstruction of the target object.
6. The deep learning-based 3D reconstruction system as described in claim 4, characterized in that, The data acquisition module specifically includes: The data acquisition unit is used to capture images of the target object from different perspectives using a binocular camera to obtain multiple single-view images; The preprocessing unit is used to preprocess all the single-view images to obtain the image to be processed corresponding to each single-view image.