Transform cost body aggregation-based multi-view depth map generation system
By adopting the Transformer cost-body aggregation method in the multi-view depth map generation system, combining the local attention mechanism and shift operation of depth perception, the problem of insufficient depth inference accuracy and calculation efficiency in the prior art is solved, and a three-dimensional scene reconstruction with high precision and high integrity is achieved.
Patent Information
- Application Number
- CN202510023479.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-07
- Publication Date
- 2025-05-09
AI Technical Summary
The existing multi-view depth map generation method is difficult to ensure accurate and dense matching in areas such as specular reflections and weak textures, and the calculation efficiency and accuracy of depth inference are insufficient.
A multi-view depth map generation system based on Transformer cost body aggregation is used to aggregate the cost body through the local attention mechanism and shift operation of depth perception, improving the accuracy of depth inference and reducing the computational complexity.
It improves the accuracy of deep inference, enhances the robustness and adaptability of the system, optimizes the computational complexity, and realizes three-dimensional scene reconstruction with high precision and high integrity.
Smart Images
Figure CN119963618A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and three-dimensional reconstruction, and in particular to a multi-view depth map generation system based on Transformer cost volume aggregation, which can reconstruct dense three-dimensional scenes from images of multiple viewpoints and generate high-precision depth maps. Background Art
[0002] With the development of 3D reconstruction technology, multi-view depth map generation has become one of the key steps to achieve high-quality 3D modeling. Multi-view stereo geometry (MVS) is one of the core issues widely studied in the field of computer vision. It aims to reconstruct dense 3D scenes through multiple images with overlapping areas and pre-calibrated camera parameters. Traditional methods recover 3D points through the projection relationship between multiple views, but it is difficult to ensure accurate dense matching in areas such as mirror reflections and weak textures. The depth map-based MVS method uses a 2D convolutional neural network (CNN) to extract features from multiple views, construct a 3D cost volume, and perform regularization to predict the depth map. However, these methods still have shortcomings in terms of video memory usage, computational efficiency, and accuracy of depth inference.
[0003] To this end, a multi-view depth map generation system based on Transformer cost volume aggregation is proposed. Summary of the invention
[0004] The present invention aims to solve the problems raised in the background technology and provides a multi-view depth map generation system based on Transformer cost volume aggregation. The system uses the Transformer structure to aggregate the cost volume, thereby improving the accuracy of depth inference. At the same time, the local attention mechanism and shift operation of depth perception are used to reduce the computational complexity.
[0005] The specific technical solutions are as follows:
[0006] A multi-view depth map generation system based on Transformer cost volume aggregation, comprising:
[0007] Image preprocessing module, used to preprocess the input images of multiple perspectives, including denoising, grayscale, image enhancement, etc.;
[0008] The camera parameter calibration module is used to calibrate the parameters of cameras with multiple viewing angles and obtain the intrinsic and extrinsic parameters of the camera;
[0009] A feature extraction module is used to extract feature points from images of multiple viewpoints and generate feature descriptors;
[0010] The cost volume construction module is used to construct a multi-view cost volume based on the extracted feature points and camera parameters;
[0011] A Transformer cost body aggregation module, which is used to aggregate the cost body using a cost body Transformer-based aggregation method to improve the accuracy of deep inference; the cost body Transformer-based aggregation method includes a depth-aware local attention mechanism and a shift operation to reduce computational complexity and improve the accuracy of deep inference;
[0012] The depth-aware Transformer layer is used to implement the depth-aware local attention mechanism in the Transformer cost volume aggregation module. It uses a depth-aware window extended from a two-dimensional spatial window to perform attention calculations along spatial and depth coordinates. At the same time, it introduces a depth-aware relative position bias to capture the depth information in the cost volume.
[0013] The shifted Transformer layer is used to implement the shift operation of the local window in the Transformer cost volume aggregation module to improve the coverage of the receptive field. Specifically, the local window is shifted along the height, width and depth axes so that different windows can interact with each other, thereby overcoming the limited receptive field problem of the convolutional neural network;
[0014] A regularization module is used to regularize the aggregated cost volume to generate a probability volume;
[0015] A depth map inference module, used to infer the depth map based on the regularized probability volume;
[0016] A depth map optimization module, used to optimize the inferred depth map to improve the accuracy and completeness of the depth map;
[0017] Uncertainty estimation module, used to estimate the uncertainty of the depth map to guide subsequent depth sampling and depth map fusion;
[0018] The depth map fusion module is used to fuse the depth maps of multiple perspectives to generate the final 3D scene reconstruction result.
[0019] The above-mentioned multi-view depth map generation system based on Transformer cost volume aggregation, wherein: the image preprocessing module uses an adaptive filtering algorithm to remove image noise, uses a grayscale algorithm to convert a color image into a grayscale image, and uses an image enhancement algorithm to improve the contrast and clarity of the image.
[0020] The above-mentioned multi-view depth map generation system based on Transformer cost volume aggregation, wherein: the camera parameter calibration module uses Zhang Zhengyou calibration method or laser scanning calibration method to calibrate the camera to obtain the camera's focal length, principal point, distortion coefficient and other internal parameters, as well as the camera's rotation matrix and translation matrix and other external parameters.
[0021] The above-mentioned multi-view depth map generation system based on Transformer cost volume aggregation, wherein: the feature extraction module uses feature extraction algorithms such as SIFT, SURF or ORB to extract feature points from the image and generate corresponding feature descriptors for subsequent image matching.
[0022] The above-mentioned multi-view depth map generation system based on Transformer cost volume aggregation, wherein: the cost volume construction module uses a differentiable homography transformation to map the source image features to the reference image perspective, calculates the matching cost, and forms a cost volume, which is a four-dimensional tensor including spatial coordinates and depth assumptions.
[0023] The above-mentioned multi-view depth map generation system based on Transformer cost volume aggregation, wherein: the regularization module uses a 3D convolution module or a GRU structure to regularize the aggregated cost volume to smooth the noise in the cost volume and generate a probability volume, which is used for subsequent depth map inference.
[0024] The above-mentioned multi-view depth map generation system based on Transformer cost volume aggregation, wherein: the depth map inference module uses the maximum a posteriori probability estimation or depth classification method to infer the depth map from the probability volume. The depth map represents the depth information of each point in the scene.
[0025] The above-mentioned multi-view depth map generation system based on Transformer cost volume aggregation, wherein: the depth map optimization module uses Gauss-Newton layer or iterative closest point (ICP) algorithm to optimize the depth map to eliminate errors in the depth map and improve the integrity of the depth map.
[0026] The above-mentioned multi-view depth map generation system based on Transformer cost body aggregation, wherein: the uncertainty estimation module uses the information entropy of the probability cost body to generate an uncertainty map to represent the uncertainty of each point in the depth map, and the uncertainty map is used to adaptively adjust the depth sampling range and the weight of the depth map fusion.
[0027] The above-mentioned multi-view depth map generation system based on Transformer cost volume aggregation, wherein: the depth map fusion module adopts weighted average or depth map fusion algorithm to fuse the depth maps of multiple perspectives to generate the final three-dimensional scene reconstruction result, and the fused three-dimensional scene reconstruction result has high precision and high integrity.
[0028] The present invention has the following beneficial effects:
[0029] The multi-view depth map generation system based on Transformer cost volume aggregation proposed in the present invention has the following technical effects:
[0030] 1. Improve the accuracy of deep inference: By designing an aggregation method based on the cost body Transformer, including the local attention mechanism and shift operation of depth perception, it is possible to more effectively utilize multi-view information and improve the accuracy of deep inference.
[0031] 2. Enhance the robustness of the system: Through steps such as image preprocessing and camera parameter calibration, it can process images of different qualities and perspectives, enhancing the robustness and adaptability of the system.
[0032] 3. Optimize computational complexity: The use of local attention mechanism and shift operation can reduce computational complexity while maintaining high performance, thereby improving the real-time performance and efficiency of the system.
[0033] 4. Provide uncertainty estimation: Through the uncertainty estimation module, an uncertainty map can be generated to provide guidance for subsequent depth sampling and depth map fusion, further improving the accuracy and completeness of 3D scene reconstruction.
[0034] 5. Achieve high-precision and high-integrity 3D scene reconstruction: Through the depth map fusion module, depth maps from multiple perspectives can be fused to generate 3D scene reconstruction results with high precision and high integrity, providing a reliable foundation for subsequent applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A schematic diagram of the architecture of a multi-view depth map generation system based on Transformer cost volume aggregation provided by an embodiment of the present invention;
[0036] Figure 2 A workflow diagram of a multi-view depth map generation system based on Transformer cost volume aggregation provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.
[0038] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on this patent; in order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0039] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as a limitation on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0040] In the description of the present invention, unless otherwise clearly specified and limited, if the term "connection" or the like appears to indicate the connection relationship between components, the term should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0041] Example
[0042] This embodiment provides a multi-view depth map generation system based on Transformer cost volume aggregation, such as Figure 1-2 As shown, it specifically includes an image preprocessing module, a camera parameter calibration module, a feature extraction module, a cost volume construction module, a Transformer cost volume aggregation module, a depth perception Transformer layer, a shift Transformer layer, a regularization module, a depth map inference module, a depth map optimization module, an uncertainty estimation module, and a depth map fusion module, where:
[0043] The image preprocessing module is used to preprocess the input images of multiple perspectives, including denoising, grayscale, image enhancement, etc. Specifically, an adaptive filtering algorithm is used to remove image noise, a grayscale algorithm is used to convert color images into grayscale images, and an image enhancement algorithm is used to improve the contrast and clarity of the image.
[0044] The camera parameter calibration module is used to calibrate the parameters of cameras with multiple viewing angles and obtain the camera's internal and external parameters. Specifically, the Zhang Zhengyou calibration method or the laser scanning calibration method is used to calibrate the camera to obtain the camera's internal parameters such as focal length, principal point, distortion coefficient, and external parameters such as the camera's rotation matrix and translation matrix.
[0045] The feature extraction module is used to extract feature points from images of multiple perspectives and generate feature descriptors. Specifically, feature extraction algorithms such as SIFT, SURF or ORB are used to extract feature points from images and generate corresponding feature descriptors for subsequent image matching.
[0046] The cost volume construction module is used to construct a multi-view cost volume based on the extracted feature points and camera parameters. It uses a differentiable homography to map the source image features to the reference image view, calculates the matching cost, and forms a cost volume. The cost volume is a four-dimensional tensor that includes spatial coordinates and depth assumptions.
[0047] The Transformer cost body aggregation module aggregates the cost body to improve the accuracy of deep inference. The specific implementation is as follows:
[0048] A cost body Transformer-based aggregation method is designed, including depth-aware local attention mechanism and shift operation. The Transformer cost body aggregation module adopts the cost body Transformer-based aggregation method to aggregate the cost body to reduce the computational complexity and improve the accuracy of depth inference.
[0049] The depth-aware Transformer layer implements the depth-aware local attention mechanism in the Transformer cost volume aggregation module, designs a depth-aware window that extends from a two-dimensional spatial window and performs attention calculations along spatial and depth coordinates. At the same time, a depth-aware relative position bias is introduced to capture the depth information in the cost volume.
[0050] The shifted Transformer layer implements the shift operation of the local window in the Transformer cost body aggregation module to improve the coverage of the receptive field. At the next level, the local window is shifted along the height, width and depth axes so that different windows can interact with each other, thereby overcoming the limited receptive field problem of the convolutional neural network.
[0051] The regularization module regularizes the aggregated cost volume to generate a probability volume. Specifically, a 3D convolution module or a GRU structure is used to regularize the aggregated cost volume to smooth the noise in the cost volume and generate a probability volume. The probability volume is used for subsequent depth map inference.
[0052] The depth map inference module infers the depth map from the regularized probability volume, specifically using the maximum a posteriori probability estimation or depth classification method to infer the depth map from the probability volume. The depth map represents the depth information of each point in the scene.
[0053] The depth map optimization module optimizes the inferred depth map to improve the accuracy and completeness of the depth map. Specifically, the depth map is optimized using a Gauss-Newton layer or an iterative closest point (ICP) algorithm to eliminate errors in the depth map and improve the completeness of the depth map.
[0054] The uncertainty estimation module is used to estimate the uncertainty of the depth map to guide the subsequent depth sampling and depth map fusion. It uses the information entropy of the probability cost volume to generate an uncertainty map to represent the uncertainty of each point in the depth map. The uncertainty map is used to adaptively adjust the depth sampling range and the weight of the depth map fusion.
[0055] The depth map fusion module fuses the depth maps of multiple perspectives to generate the final 3D scene reconstruction result. It uses weighted average or depth map fusion algorithm to fuse the depth maps of multiple perspectives to generate the final 3D scene reconstruction result. The fused 3D scene reconstruction result has high accuracy and high integrity.
[0056] It is worth noting that in the multi-view depth map generation system based on Transformer cost volume aggregation provided in this embodiment, each module works together to complete the entire process from multi-view image input to final 3D scene reconstruction. The following is a detailed introduction to the connection relationship between these modules:
[0057] Image preprocessing module
[0058] Input: Receive raw images from multiple viewpoints.
[0059] Output: The preprocessed image, including the results of denoising, grayscale and image enhancement.
[0060] Connection: Output the processed image to the feature extraction module.
[0061] Camera parameter calibration module
[0062] Input: Raw camera data from multiple viewpoints (may include images and camera position information).
[0063] Output: camera intrinsics and extrinsics.
[0064] Connection: Output the calibrated camera parameters to the cost volume construction module and the depth map fusion module (for perspective transformation and geometric correction during fusion).
[0065] Feature extraction module
[0066] Input: Preprocessed image.
[0067] Output: feature points and corresponding feature descriptors.
[0068] Connection: Output the extracted feature points and descriptors to the cost volume construction module.
[0069] Cost body building module
[0070] Input: feature points, feature descriptors and camera parameters.
[0071] Output: constructed cost volume.
[0072] Connection: Output the cost volume to the Transformer cost volume aggregation module.
[0073] Transformer cost aggregation module
[0074] Input: cost body.
[0075] Internal components: Depth-aware Transformer layer and shifting Transformer layer.
[0076] Depth-aware Transformer layer: processes the depth information in the cost volume and enhances the local attention mechanism.
[0077] Shift Transformer layer: Expand the receptive field and enhance feature interaction through shift operation.
[0078] Output: Aggregated cost volume.
[0079] Connection: Output the aggregated cost volume to the regularization module.
[0080] Regularization module
[0081] Input: Aggregated cost volume.
[0082] Output: Regularized probability volume.
[0083] Connection: Output the regularized probability volume to the depth map inference module.
[0084] Depth map inference module
[0085] Input: Regularized probability volume.
[0086] Output: Inferred depth map.
[0087] Connection: Output the depth map to the depth map optimization module and the uncertainty estimation module.
[0088] Depth map optimization module
[0089] Input: Inferred depth map.
[0090] Output: Optimized depth map.
[0091] Connection: Output the optimized depth map to the depth map fusion module (if multi-view fusion is required).
[0092] Uncertainty Estimation Module
[0093] Input: Regularized probability volume (or inferred depth map, depending on implementation).
[0094] Output: Uncertainty map.
[0095] Connection: The uncertainty map is output to the depth map fusion module (for guiding the fusion weights) and possibly the depth sampling module (for guiding the sampling strategy).
[0096] Depth map fusion module
[0097] Input: Refined depth maps and uncertainty maps (if available) from multiple views.
[0098] Output: The fused 3D scene reconstruction result.
[0099] Connection: Output the final 3D scene reconstruction result for subsequent application use.
[0100] Summarize the connection relationship
[0101] Image preprocessing module → feature extraction module
[0102] Camera parameter calibration module → cost volume construction module, depth map fusion module
[0103] Feature extraction module → cost volume construction module
[0104] Cost body construction module → Transformer cost body aggregation module
[0105] Transformer cost body aggregation module → regularization module
[0106] Regularization module → Depth map inference module
[0107] Depth map inference module → depth map optimization module, uncertainty estimation module
[0108] Depth map optimization module → Depth map fusion module
[0109] Uncertainty estimation module → depth map fusion module (optional, used to guide fusion weights)
[0110] The above connection relationship ensures that each module in the system can process the input data sequentially and effectively, and ultimately generate high-quality three-dimensional scene reconstruction results.
[0111] In order to verify the effectiveness of the multi-view depth map generation system based on Transformer cost volume aggregation provided in this embodiment, the following experimental scheme is designed, and the public datasets DTU and Tanks&Temples are used for testing and compared with the current mainstream methods.
[0112] Experimental protocol
[0113] 1. Data preparation
[0114] DTU dataset: The DTU dataset contains images of multiple objects under different viewpoints and lighting conditions, and is suitable for evaluating the performance of multi-view stereo matching algorithms.
[0115] Tanks&Temples dataset: The Tanks&Temples dataset contains complex scenes and is mainly used to evaluate the performance of 3D reconstruction algorithms in actual scenes.
[0116] 2. Experimental Setup
[0117] Environment configuration: Ensure that the experimental environment has sufficient computing resources (such as GPU) to support the training and inference of deep learning models.
[0118] Model training: Use the training set portion of the DTU dataset to train the system of the present invention and adjust the model parameters to optimize performance.
[0119] Parameter selection: According to experimental needs, adjust the key parameters in modules such as image preprocessing, feature extraction, cost volume construction, and Transformer cost volume aggregation.
[0120] 3. Evaluation indicators
[0121] Depth map accuracy: The accuracy of the depth map is evaluated using indicators such as mean square error (MSE), root mean square error (RMSE), and mean absolute error (MAE).
[0122] 3D reconstruction quality: Use indicators such as completeness, accuracy, and F1 score to evaluate the quality of 3D reconstruction.
[0123] Uncertainty Estimation: The correlation between the uncertainty map and the depth map error is evaluated to verify the effectiveness of the uncertainty estimation.
[0124] 4. Experimental Procedure
[0125] Data preprocessing: Preprocess the images in the DTU and Tanks&Temples datasets, including denoising, grayscale conversion, image enhancement, etc.
[0126] Model training and verification: The system of the present invention is trained on the DTU training set, and the performance is evaluated on the verification set.
[0127] Testing and comparison: The system of this embodiment is tested on the DTU test set and the Tanks & Temples dataset, and compared with mainstream methods. Mainstream methods include but are not limited to depth map generation methods based on convolutional neural networks (CNNs), traditional multi-view stereo matching methods, etc.
[0128] Result analysis: Analyze the experimental results, including the accuracy of the depth map, the quality of the 3D reconstruction, the effectiveness of the uncertainty estimation, etc., and discuss the advantages and disadvantages of the system of the present invention.
[0129] 5. Results and Discussion
[0130] Visualization display: The reconstructed 3D scene is displayed through a 3D visualization tool, and the reconstruction effect of the system of this embodiment is compared with that of the mainstream method.
[0131] Performance analysis: Analyze the performance of the system in this embodiment in terms of depth map accuracy, 3D reconstruction quality, uncertainty estimation, etc., and compare them with mainstream methods.
[0132] Discussion and Outlook: The advantages and disadvantages of the system in this embodiment are discussed, possible improvement directions are proposed, and the future application prospects in the field of 3D reconstruction are proposed.
[0133] Experimental Implementation
[0134] Data download and preprocessing: Download the DTU and Tanks&Temples datasets from public channels and preprocess them according to the experimental plan.
[0135] Model training: Use the preprocessed DTU training set to train the system of the present invention, and record the loss function value during the training process and the performance indicators on the validation set.
[0136] Testing and comparison: The system of the present invention is tested on the DTU test set and the Tanks&Temples dataset, and indicators such as depth map accuracy and 3D reconstruction quality are recorded and compared with mainstream methods.
[0137] Result analysis: Based on the experimental results, the performance advantages and disadvantages of the system of the present invention are analyzed, and improvement suggestions are put forward.
[0138] in conclusion
[0139] Through comparative experiments, the performance advantages of the multi-view depth map generation system based on Transformer cost volume aggregation in the present invention in terms of depth map accuracy, 3D reconstruction quality, etc. can be verified. At the same time, through the uncertainty estimation module, the depth sampling and depth map fusion can be better guided to improve the accuracy and integrity of 3D reconstruction. Finally, the experimental results will be summarized and the practical application prospects and potential improvement directions of the system of the present invention will be discussed.
[0140] In summary: The multi-view depth map generation system based on Transformer cost volume aggregation proposed in this embodiment uses the Transformer structure to aggregate the cost volume, thereby improving the accuracy of depth inference. At the same time, the computational complexity is reduced through the local attention mechanism and shift operation of depth perception. Experimental results show that the multi-view depth map generation system based on Transformer cost volume aggregation proposed in this embodiment has significant advantages in three-dimensional reconstruction and depth map generation, and has broad application prospects.
[0141] Workflow Summary
[0142] The workflow of the multi-view depth map generation system based on Transformer cost volume aggregation provided in this embodiment includes the following steps:
[0143] S1. Image preprocessing: Preprocess the input images of multiple perspectives, including denoising, grayscale and image enhancement, to improve image quality and provide high-quality input for subsequent processing.
[0144] S2. Camera parameter calibration: Calibrate the parameters of cameras with multiple perspectives to obtain the intrinsic and extrinsic parameters of the camera, which is the basis for constructing the cost volume and performing depth inference.
[0145] S3, feature extraction: extract feature points from images of multiple perspectives and generate feature descriptors for subsequent image matching and cost volume construction.
[0146] S4, cost volume construction: According to the extracted feature points and camera parameters, the source image features are mapped to the reference image perspective using a differentiable homography transformation, the matching cost is calculated, and a cost volume is formed.
[0147] S5. Transformer cost body aggregation: The cost body is aggregated using a cost body Transformer-based aggregation method, including depth-aware local attention mechanism and shift operations to improve the accuracy of depth inference.
[0148] S6. Regularization: Regularize the aggregated cost volume to smooth the noise in the cost volume and generate a probability volume.
[0149] S7, depth map inference: infer the depth map based on the regularized probability volume, representing the depth information of each point in the scene.
[0150] S8. Depth map optimization: Optimize the inferred depth map to eliminate errors in the depth map and improve the completeness and accuracy of the depth map.
[0151] S9. Uncertainty estimation: The information entropy of the probability cost body is used to generate an uncertainty map to represent the uncertainty of each point in the depth map, which is used to guide subsequent depth sampling and depth map fusion.
[0152] S10, Depth map fusion: Fusion of depth maps from multiple perspectives to generate the final 3D scene reconstruction result with high accuracy and integrity.
[0153] Beneficial Effects
[0154] Improve the accuracy of deep inference: By designing an aggregation method based on the cost body Transformer, including the depth-aware local attention mechanism and shift operation, it is possible to more effectively utilize multi-view information and improve the accuracy of deep inference.
[0155] Enhance the robustness of the system: Through steps such as image preprocessing and camera parameter calibration, it can process images of different qualities and perspectives, enhancing the robustness and adaptability of the system.
[0156] Optimize computational complexity: The use of local attention mechanism and shift operation can reduce computational complexity while maintaining high performance, thereby improving the real-time performance and efficiency of the system.
[0157] Provide uncertainty estimation: The uncertainty estimation module can generate uncertainty maps to provide guidance for subsequent depth sampling and depth map fusion, further improving the accuracy and completeness of 3D scene reconstruction.
[0158] Achieve high-precision and high-integrity 3D scene reconstruction: Through the depth map fusion module, depth maps from multiple perspectives can be fused to generate high-precision and high-integrity 3D scene reconstruction results, providing a reliable foundation for subsequent applications.
[0159] In summary, the multi-view depth map generation system based on Transformer cost volume aggregation provided in this embodiment has significant beneficial effects and can be widely used in three-dimensional reconstruction, virtual reality, augmented reality and other fields.
[0160] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.
Claims
1. A multi-view depth map generation system based on Transformer cost volume aggregation, characterized in that: include: The image preprocessing module is used to preprocess the input images of multiple perspectives, including denoising, grayscale conversion, and image enhancement; The camera parameter calibration module is used to calibrate the parameters of cameras with multiple viewing angles and obtain the intrinsic and extrinsic parameters of the camera; A feature extraction module is used to extract feature points from images of multiple viewpoints and generate feature descriptors; The cost volume construction module is used to construct a multi-view cost volume based on the extracted feature points and camera parameters; A Transformer cost body aggregation module, for aggregating the cost body using a cost body Transformer-based aggregation method, wherein the cost body Transformer-based aggregation method includes a depth-aware local attention mechanism and a shift operation; The depth-aware Transformer layer is used to implement the depth-aware local attention mechanism in the Transformer cost volume aggregation module. It uses a depth-aware window extended from a two-dimensional spatial window to perform attention calculations along spatial and depth coordinates. At the same time, it introduces a depth-aware relative position bias to capture the depth information in the cost volume. The shifting Transformer layer is used to implement the shifting operation of the local window in the Transformer cost volume aggregation module, including shifting the local window along the height, width and depth axes to make different windows interact with each other; A regularization module is used to regularize the aggregated cost volume to generate a probability volume; A depth map inference module, used to infer the depth map based on the regularized probability volume; A depth map optimization module, used to optimize the inferred depth map; Uncertainty estimation module, used to estimate the uncertainty of the depth map to guide the subsequent depth map fusion; The depth map fusion module is used to fuse the depth maps of multiple perspectives to generate the final 3D scene reconstruction result.
2. The multi-view depth map generation system based on Transformer cost volume aggregation according to claim 1, characterized in that: The image preprocessing module uses an adaptive filtering algorithm to remove image noise, uses a grayscale algorithm to convert a color image into a grayscale image, and uses an image enhancement algorithm to improve the contrast and clarity of the image.
3. The multi-view depth map generation system based on Transformer cost volume aggregation according to claim 2, characterized in that: The camera parameter calibration module uses the Zhang Zhengyou calibration method or the laser scanning calibration method to calibrate the camera to obtain the focal length, principal point, distortion coefficient internal parameters of the camera, and the rotation matrix and translation matrix external parameters of the camera.
4. The multi-view depth map generation system based on Transformer cost volume aggregation according to claim 3, characterized in that: The feature extraction module uses SIFT, SURF or ORB feature extraction algorithm to extract feature points from the image and generate corresponding feature descriptors for subsequent image matching.
5. The multi-view depth map generation system based on Transformer cost volume aggregation according to claim 4, characterized in that: The cost volume construction module uses a differentiable homography transformation to map the source image features to the reference image perspective, calculates the matching cost, and forms a cost volume, which is a four-dimensional tensor including spatial coordinates and depth assumptions.
6. The multi-view depth map generation system based on Transformer cost volume aggregation according to claim 5, characterized in that: The regularization module uses a 3D convolution module or a GRU structure to regularize the aggregated cost volume to smooth the noise in the cost volume and generate a probability volume, which is used for subsequent depth map inference.
7. The multi-view depth map generation system based on Transformer cost volume aggregation according to claim 6, characterized in that: The depth map inference module uses the maximum a posteriori probability estimation or depth classification method to infer the depth map from the probability body, and the depth map represents the depth information of each point in the scene.
8. The multi-view depth map generation system based on Transformer cost volume aggregation according to claim 7, characterized in that: The depth map optimization module optimizes the depth map using Gauss-Newton layers or an iterative closest point algorithm to eliminate errors in the depth map and improve the integrity of the depth map.
9. The multi-view depth map generation system based on Transformer cost volume aggregation according to claim 8, characterized in that: The uncertainty estimation module generates an uncertainty map using the information entropy of the probability cost body to represent the uncertainty of each point in the depth map. The uncertainty map is used to adaptively adjust the depth sampling range and the weight of the depth map fusion.
10. The multi-view depth map generation system based on Transformer cost volume aggregation according to claim 9, characterized in that: The depth map fusion module adopts weighted average or depth map fusion algorithm to fuse the depth maps of multiple perspectives to generate the final 3D scene reconstruction result.