Deep learning three-dimensional reconstruction method and system based on adaptive feature fusion, and electronic equipment
By introducing the CA attention mechanism and ASFF module to the deep learning three-dimensional reconstruction method for adaptive feature fusion, the problems of edge information blurring and feature conflict in the feature extraction process in the prior art are solved, and higher three-dimensional reconstruction accuracy and visual effects are achieved.
Patent Information
- Application Number
- CN202411920929.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-04-22
AI Technical Summary
The existing deep learning three-dimensional reconstruction methods have problems such as blurred edge information, texture distortion or jagged artifacts in the feature extraction process, and the missing information and inconsistency between features of different scales in the depth reasoning task lead to problems such as semantic information imbalance and feature conflict.
The deep learning three-dimensional reconstruction method based on adaptive feature fusion is adopted. By introducing the CA attention mechanism and ASFF module, multi-scale feature extraction and adaptive feature fusion are carried out, the spatial importance weights of feature maps of different levels are learned, important features are highlighted, and the depth map is calculated using the PatchMatch network to perform three-dimensional model reconstruction.
It improves the depth inference effect and three-dimensional reconstruction accuracy, reduces edge information blur and jagged artifacts, and achieves better reconstruction quality and visual effects.
Smart Images
Figure CN119863564A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a deep learning three-dimensional reconstruction method, system and electronic equipment based on adaptive feature fusion, belonging to the technical fields of computer vision and artificial intelligence. Background Art
[0002] 3D reconstruction technology has a wide range of applications in education, entertainment, medical care, transportation and other fields. In the medical field, accurately reconstructed specimens and 3D models of the internal and external structures of the human body are of great significance for medical diagnosis and treatment. In unmanned driving of vehicles, the 3D model of the block scene provides detailed information about environmental elements such as roads, buildings, pedestrians, and vehicles, which is used for environmental perception and path planning, helping vehicles to achieve safe and efficient autonomous navigation. In the fields of virtual reality, augmented reality, movie special effects, game development, etc., 3D models are widely used in scene construction, animation production, visual effects rendering, etc. Users can get an immersive and highly realistic virtual environment experience by rendering and displaying 3D models. The rise of deep learning 3D reconstruction methods provides an idea of using deep learning to complete 3D reconstruction tasks. Among them, the patchmatch network has shown good performance due to its low memory consumption and high reconstruction quality.
[0003] Although the patchmatch network can reconstruct three-dimensional models with complex geometric shapes and appearance textures. However, this method still has many problems: First, in the feature extraction process, the features are mapped to different scales through upsampling or downsampling operations, resulting in blurred edge information, texture distortion or jagged artifacts. Secondly, the problem of missing information in deep reasoning tasks and the inconsistency between features of different scales lead to problems such as unbalanced semantic information, feature conflicts, blurred edges and artifacts, occlusion and discontinuity of parallax. For example, large objects may be more prominent in high-level features, and small objects may only be clearly represented in low-level features; large objects are marked as foregrounds in high-level features, but may be mistaken for backgrounds in low-level features, hindering the network's accurate judgment of object boundaries and depth. Summary of the invention
[0004] The present invention provides a deep learning 3D reconstruction method, system, and electronic device based on adaptive feature fusion, which are used to solve the problems of feature conflict, blurred edge information, texture distortion, or jagged artifacts in existing deep learning 3D reconstruction methods.
[0005] The technical solution of the present invention is: a deep learning 3D reconstruction method based on adaptive feature fusion, the method comprising:
[0006] Step 1: Collect image sequences of the object to be reconstructed from different perspectives, perform SFM reconstruction on the image sequences, and obtain the camera parameters of each image;
[0007] Step 2: Input the image sequence and camera parameters into the feature pyramid network with CA attention mechanism for three-layer multi-scale downsampling and feature extraction;
[0008] Step 3: Send the extracted feature map to the ASFF module for adaptive feature fusion, learn the spatial importance weights of feature maps at different levels, and highlight important features;
[0009] Step 4: Based on the feature map fused in the previous step, the PatchMatch network is used to calculate the depth map of each image and perform coordinate transformation to reconstruct the three-dimensional model.
[0010] Furthermore, the Step 1 includes:
[0011] Step 1.1, build a platform to obtain images of different perspectives of the object to be reconstructed as input images;
[0012] Step 1.2, add a module to the platform to calculate camera parameters using the SFM method;
[0013] Step 1.3, use the module that calculates camera parameters through the SFM method to calculate the machine parameters; the camera parameters include camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters include: focal length, image principal point, distortion coefficient; the camera extrinsic parameters include posture.
[0014] Furthermore, the Step 2 includes:
[0015] Step 2.1, build a feature extraction network, including embedding the CA attention module in the feature pyramid network and introducing channel attention containing position information;
[0016] Step 2.2, extract features through the feature extraction network, in which the CA attention module decomposes the channel attention into one-dimensional feature encoding along two spatial directions respectively, and aggregates the features along the spatial direction, so that the CA attention module captures long-range dependencies along one spatial direction while retaining precise position information in another spatial direction. The obtained feature map is then encoded into a pair of direction-aware and position-sensitive attention maps to obtain enhanced features.
[0017] The feature extraction network extracts features from the image sequence of the object to be reconstructed. It first uses the feature pyramid network to extract features, and then uses the CA attention module to enhance the features.
[0018] Furthermore, the Step 3 includes:
[0019] Step 3.1, add the ASFF module between the feature extraction network and the deep reasoning layer, so that the network can suppress inconsistency by learning the method of filtering conflicting information in space, thereby strengthening the scale invariance of features and improving the deep reasoning performance;
[0020] Step 3.2, perform multi-scale feature fusion on the feature map output in Step 2: Use the ASFF module to adaptively learn the fusion spatial weight of each scale feature map using data-driven, and perform weighted fusion on each scale feature map based on the spatial importance weight.
[0021] Furthermore, the weighted fusion method is expressed as:
[0022]
[0023] in Represents the vector of the inter-channel output feature map at (i, j), coefficient Indicates that the network adaptively learns the spatial importance weights of the three different levels of feature maps in the lth layer, Represents the feature vector at position (i, j) on the feature map adjusted from level n to level l, where n is 1-3.
[0024] The feature extraction network performs feature fusion on the feature maps of different scales obtained; it uses weighted fusion to fuse feature maps of different scales, suppresses the inconsistency between features of different scales, strengthens the scale invariance of features, learns the importance weights of features at different levels, filters conflicting information, and highlights important feature information.
[0025] Furthermore, the Step 4 includes:
[0026] The Patchmatch network is used to perform deep reasoning on the input fused feature map, and the obtained depth map is transformed into a three-dimensional model to complete the three-dimensional reconstruction.
[0027] The present invention also provides a deep learning three-dimensional reconstruction system based on adaptive feature fusion, and the system includes: a module for executing the above-mentioned deep learning three-dimensional reconstruction method based on adaptive feature fusion.
[0028] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned deep learning three-dimensional reconstruction method based on adaptive feature fusion when executing the program.
[0029] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned deep learning three-dimensional reconstruction method based on adaptive feature fusion is implemented.
[0030] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned deep learning three-dimensional reconstruction method based on adaptive feature fusion.
[0031] The present invention uses a feature pyramid network (FPN) that introduces CA attention to encode the position information of input image features, extract features, integrate the CA attention mechanism to supplement the position information, improve the feature extraction capability, and enhance the feature extraction map; uses the ASFF module to perform adaptive feature fusion and learn the spatial importance weights of feature maps at different levels; based on the feature map, the PatchMatch network is then used to calculate the depth map of each image and perform coordinate transformation to reconstruct a three-dimensional model.
[0032] The beneficial effects of the present invention are:
[0033] (1) The present invention proposes a deep learning 3D reconstruction method based on adaptive feature fusion, which introduces the CA (Channel Attention) attention mechanism and the ASFF (Adaptive Spatial Feature Fusion) feature fusion module to enhance the feature scale invariance of the network, thereby improving the deep reasoning effect and 3D reconstruction accuracy.
[0034] (2) The present invention obtains a feature map by extracting features from images of different perspectives of the object to be reconstructed, and then performs feature fusion, filters conflicting information, enhances the expression of important features, improves deep reasoning capabilities, and achieves good reconstruction quality.
[0035] (3) The deep learning 3D reconstruction method based on adaptive feature fusion used in the present invention can provide a 3D data acquisition method for the research of holographic display, provide support for holographic display of complex scenes, and provide a new idea for the 3D reconstruction of objects and the realization of holographic display.
[0036] (4) The adaptive feature fusion deep learning 3D reconstruction method system proposed in the present invention provides a new idea for deep learning methods to complete 3D reconstruction tasks, reduces the blurring of edge information and jagged artifacts in 3D reconstruction, has better reconstruction quality, and provides clearer detail information in visual effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A flowchart of a deep learning 3D reconstruction method based on adaptive feature fusion proposed by the present invention;
[0038] Figure 2 is an overall network structure diagram according to an embodiment of the present invention;
[0039] Figure 3 A diagram of a PatchMatch network structure according to an embodiment of the present invention;
[0040] Figure 4 A CA attention mechanism network structure diagram according to an embodiment of the present invention;
[0041] Figure 5 A schematic diagram comparing the 3D reconstruction method of the present invention and the depth map reconstructed by the PatchMatch network;
[0042] Figure 6 Schematic diagram for comparing the 3D reconstruction method of the present invention with the point cloud reconstructed by the PatchMatch network. DETAILED DESCRIPTION
[0043] Example 1: Figure 1-Figure 6 As shown, a deep learning 3D reconstruction method based on adaptive feature fusion, a deep learning 3D reconstruction method based on adaptive feature fusion, the method comprising:
[0044] Step 1: Collect image sequences of the object to be reconstructed from different perspectives, perform SFM reconstruction on the image sequences, and obtain the camera parameters of each image;
[0045] Furthermore, the Step 1 includes:
[0046] Step 1.1, build a platform to obtain images of different perspectives of the object to be reconstructed as input images;
[0047] Step 1.2, add a module to the platform to calculate camera parameters using the SFM method;
[0048] Step 1.3, use the module that calculates camera parameters through the SFM method to calculate the machine parameters; the camera parameters include camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters include: focal length, image principal point, distortion coefficient; the camera extrinsic parameters include posture.
[0049] Step 2: Input the image sequence and camera parameters into the feature pyramid network with CA attention mechanism for three-layer multi-scale downsampling and feature extraction;
[0050] Furthermore, the Step 2 includes:
[0051] Step 2.1, build a feature extraction network, including embedding the CA attention module in the feature pyramid network and introducing channel attention containing position information;
[0052] Step 2.2, extract features through the feature extraction network, in which the CA attention module decomposes the channel attention into one-dimensional feature encoding along two spatial directions respectively, and aggregates the features along the spatial direction, so that the CA attention module captures long-range dependencies along one spatial direction while retaining precise position information in another spatial direction. The obtained feature map is then encoded into a pair of direction-aware and position-sensitive attention maps to obtain enhanced features.
[0053] The feature extraction network extracts features from the image sequence of the object to be reconstructed. It first uses the feature pyramid network to extract features, and then uses the CA attention module to enhance the features.
[0054] Step 3: Send the extracted feature map to the ASFF module for adaptive feature fusion, learn the spatial importance weights of feature maps at different levels, and highlight important features;
[0055] Furthermore, the Step 3 includes:
[0056] Step 3.1, add the ASFF module between the feature extraction network and the deep reasoning layer, so that the network can suppress inconsistency by learning the method of filtering conflicting information in space, thereby strengthening the scale invariance of features and improving the deep reasoning performance;
[0057] Step 3.2, perform multi-scale feature fusion on the feature map output in Step 2: Use the ASFF module to adaptively learn the fusion spatial weight of each scale feature map using data-driven, and perform weighted fusion on each scale feature map based on the spatial importance weight.
[0058] Furthermore, the weighted fusion method is expressed as:
[0059]
[0060] in Represents the vector of the inter-channel output feature map at (i, j), coefficient Indicates that the network adaptively learns the spatial importance weights of the three different levels of feature maps in the lth layer, Represents the feature vector at position (i, j) on the feature map adjusted from level n to level l, where n is 1-3.
[0061] The feature extraction network performs feature fusion on the obtained feature maps of different scales; it uses weighted fusion to fuse feature maps of different scales, suppress the inconsistency between features of different scales, strengthen the scale invariance of features, learn the importance weights of features at different levels, filter conflicting information, and highlight important feature information;
[0062] Step 4: Based on the feature map fused in the previous step, the PatchMatch network is used to calculate the depth map of each image and perform coordinate transformation to reconstruct the 3D model.
[0063] Furthermore, the Step 4 includes:
[0064] The Patchmatch network is used to perform deep reasoning on the input fused feature map, and the obtained depth map is transformed into a three-dimensional model to complete the three-dimensional reconstruction.
[0065] The present invention also provides a deep learning 3D reconstruction system based on adaptive feature fusion, the system comprising:
[0066] The camera parameter calculation module is used to collect image sequences of the object to be reconstructed at different viewing angles, perform SFM reconstruction on the image sequences, and obtain the camera parameters of each image;
[0067] A three-layer multi-scale downsampling and feature extraction module, which is used to input the image sequence and camera parameters into the feature pyramid network with CA attention mechanism for three-layer multi-scale downsampling and feature extraction;
[0068] The feature fusion module is used to send the extracted feature maps to the ASFF module for adaptive feature fusion, learn the spatial importance weights of feature maps at different levels, and highlight important features;
[0069] The 3D model reconstruction module is used to calculate the depth map of each image based on the feature map fused by the feature fusion module, and perform coordinate transformation to reconstruct the 3D model.
[0070] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned deep learning three-dimensional reconstruction method based on adaptive feature fusion when executing the program.
[0071] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned deep learning three-dimensional reconstruction method based on adaptive feature fusion is implemented.
[0072] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned deep learning three-dimensional reconstruction method based on adaptive feature fusion.
[0073] The present invention uses a feature pyramid network (FPN) that introduces CA attention to encode the position information of input image features, extract features, integrate the CA attention mechanism to supplement the position information, improve the feature extraction capability, and enhance the feature extraction map; uses the ASFF module to perform adaptive feature fusion and learn the spatial importance weights of feature maps at different levels; based on the feature map, the PatchMatch network is then used to calculate the depth map of each image and perform coordinate transformation to reconstruct a three-dimensional model.
[0074] The present invention proposes a deep learning 3D reconstruction method based on adaptive feature fusion, extracts features from images of different perspectives of the object to be reconstructed to obtain a feature map, then performs feature fusion, filters conflicting information, enhances the expression of important features, improves deep reasoning capabilities, and achieves good reconstruction quality. At the same time, it can be combined with existing holographic display technology to achieve holographic display of complex scenes, providing a new idea for 3D reconstruction and holographic display of objects.
[0075] In order to evaluate the reconstruction quality of the present invention, the method of the present invention is tested based on the DTU dataset, and the reconstruction effect is compared with reference Figure 5 and Figure 6 , where (a) is the real data, (b) is the patchmatch network reconnection result, and (c) is the reconstruction result of the present invention. It can be seen that the depth map inferred by the proposed method has smoother and clearer edges, eliminates jagged artifacts, and reconstructs a more accurate point cloud with clearer detail information.
[0076] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A deep learning 3D reconstruction method based on adaptive feature fusion, characterized by: The method comprises: Step 1: Collect image sequences of the object to be reconstructed from different perspectives, perform SFM reconstruction on the image sequences, and obtain the camera parameters of each image; Step 2: Input the image sequence and camera parameters into the feature pyramid network with CA attention mechanism for three-layer multi-scale downsampling and feature extraction; Step 3: Send the extracted feature map to the ASFF module for adaptive feature fusion, learn the spatial importance weights of feature maps at different levels, and highlight important features; Step 4: Based on the feature map fused in the previous step, the PatchMatch network is used to calculate the depth map of each image and perform coordinate transformation to reconstruct the three-dimensional model.
2. The method for deep learning 3D reconstruction based on adaptive feature fusion according to claim 1, characterized in that: The Step 1 includes: Step 1.1, build a platform to obtain images of different perspectives of the object to be reconstructed as input images; Step 1.2, add a module to the platform to calculate camera parameters using the SFM method; Step 1.3, use the module that calculates camera parameters through the SFM method to calculate the machine parameters; the camera parameters include camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters include: focal length, image principal point, distortion coefficient; the camera extrinsic parameters include posture.
3. The method for deep learning 3D reconstruction based on adaptive feature fusion according to claim 1, characterized in that: The Step 2 includes: Step 2.1, build a feature extraction network, including embedding the CA attention module in the feature pyramid network and introducing channel attention containing position information; Step 2.2, extract features through the feature extraction network, in which the CA attention module decomposes the channel attention into one-dimensional feature encoding along two spatial directions respectively, and aggregates the features along the spatial direction, so that the CA attention module captures long-range dependencies along one spatial direction while retaining precise position information in another spatial direction. The obtained feature map is then encoded into a pair of direction-aware and position-sensitive attention maps to obtain enhanced features.
4. The method for deep learning 3D reconstruction based on adaptive feature fusion according to claim 1, characterized in that: The Step 3 includes: Step 3.1, add the ASFF module between the feature extraction network and the deep reasoning layer; Step 3.2, perform multi-scale feature fusion on the feature map output in Step 2: Use the ASFF module to adaptively learn the fusion spatial weight of each scale feature map using data-driven, and perform weighted fusion on each scale feature map based on the spatial importance weight.
5. The method for deep learning 3D reconstruction based on adaptive feature fusion according to claim 4, characterized in that: The weighted fusion method is expressed as: in Represents the vector of the inter-channel output feature map at (i, j), coefficient Indicates that the network adaptively learns the spatial importance weights of the three different levels of feature maps in the lth layer, Represents the feature vector at position (i, j) on the feature map adjusted from level n to level l, where n is 1-3.
6. The method for deep learning 3D reconstruction based on adaptive feature fusion according to claim 1, characterized in that: The Step 4 includes: The Patchmatch network is used to perform deep reasoning on the input fused feature map, and the obtained depth map is transformed into a three-dimensional model to complete the three-dimensional reconstruction.
7. A deep learning 3D reconstruction system based on adaptive feature fusion, characterized in that: The system includes: a module for executing a deep learning three-dimensional reconstruction method based on adaptive feature fusion as described in any one of claims 1 to 6.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, a deep learning three-dimensional reconstruction method based on adaptive feature fusion is implemented as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a deep learning three-dimensional reconstruction method based on adaptive feature fusion as described in any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a deep learning three-dimensional reconstruction method based on adaptive feature fusion as described in any one of claims 1 to 6 is implemented.