A method for three-dimensional reconstruction of an aircraft auxiliary power unit pipeline based on visual measurement
By combining the self-supervised learning and context-aware fusion technology of the DeepMVS module and the Pix2Vox framework, the problem of insufficient restoration of local details in the complex piping system of the aircraft auxiliary power unit was solved, and high-precision and efficient 3D reconstruction effects were achieved.
Patent Information
- Application Number
- CN202411838890.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-10-10
- Estimated Expiration
- 2044-12-13
AI Technical Summary
In the existing technology of 3D reconstruction of complex piping systems of aircraft auxiliary power units, the restoration of local details is not fine enough and the recovery effect of occluded areas is poor, which cannot meet the requirements of high precision and high efficiency.
A visual measurement-based method is combined with the DeepMVS module and the Pix2Vox framework to generate a high-resolution 3D voxel grid through multi-view depth estimation, depth map completion and voxel reconstruction, using self-supervised learning and context-aware fusion technology.
The 3D reconstruction capability of occluded areas has been improved, achieving high-precision and efficient 3D reconstruction of complex piping systems and improving the restoration effect of local details.
Smart Images
Figure CN119762677B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of computer vision measurement, and is a three-dimensional reconstruction method for aircraft auxiliary power unit pipelines based on vision measurement. Background Art
[0002] In the aerospace industry, the aircraft auxiliary power unit (APU) is a critical component in aircraft operation. Its complex ducting system plays a vital role in energy distribution, air circulation, and cooling. These ducts typically have complex geometries, high precision requirements, and are densely distributed. The measurement and 3D reconstruction of these ducts present a major technical challenge in the manufacture and maintenance of APUs.
[0003] Traditional 3D reconstruction methods, such as those based on multi-view geometry (MVS), can, to a certain extent, complete the 3D reconstruction of complex pipelines. However, they suffer from low efficiency, insufficient accuracy, and over-reliance on high-quality annotated data when dealing with local occlusion and complex deformable structures. These methods are unable to fully address the challenges posed by data scarcity and local occlusion in the complex piping systems of aircraft auxiliary power units. In recent years, deep learning-based 3D reconstruction technology has developed rapidly. The Pix2Vox algorithm generates voxelized 3D models from single-view or multi-view images, effectively capturing global geometric features and achieving a good balance between speed and accuracy. However, it is not precise enough in restoring local details, and there is still room for improvement in recovering occluded areas. Further optimization is needed to improve reconstruction accuracy. Summary of the Invention
[0004] The present invention addresses the problem of insufficient local detail restoration in the three-dimensional reconstruction of complex ducts in aircraft auxiliary power units. The present invention provides a three-dimensional reconstruction method for aircraft auxiliary power unit ducts based on visual measurement. The present invention provides the following technical solutions:
[0005] A three-dimensional reconstruction method for aircraft auxiliary power unit pipelines based on visual measurement, the method comprising the following steps:
[0006] Step 1: Perform data preprocessing, take the multi-view RGB images and the corresponding camera poses as input, and use the DeepMVS module to generate a planar scan volume as a preliminary estimate of the depth information;
[0007] Step 2: Generate multi-view depth maps. Use the DeepMVS plane scanning volume generation module to build a candidate set of depth maps for each reference view. Use the convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view.
[0008] Step 3: Perform depth map completion and fusion. Use self-supervised learning technology to complete occluded or missing areas. Based on the context scoring mechanism, generate a complete depth map representation by fusing depth maps from multiple perspectives.
[0009] Step 4: Transcode the multi-view RGB image and depth map into a voxel grid. Using the Pix2Vox convolutional encoder, fuse the multi-view image with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. The depth completion result is introduced as an additional input feature to enhance the ability to capture local details and global geometry of the object.
[0010] Step 5: Refine and optimize the voxel grid. Using the Pix2Vox refinement network, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion. This outputs a high-resolution, complete 3D voxel grid that accurately describes the complex piping in the aircraft auxiliary power unit.
[0011] Preferably, the volume in step 1 includes color projections at different depths for each view, providing a basis for subsequent depth completion, and normalizing and cropping the data to ensure that the input multi-view image is compatible with the network size of Pix2Vox.
[0012] Preferably, the step 2 introduces multi-layer aggregation and cross-view feature aggregation, fuses useful information from different viewpoints through max-pooling, improves the reconstruction capability of occluded areas, and outputs a depth image for each viewpoint.
[0013] Preferably, step 4 further includes decoding, inputting the encoded features into a decoder of Pix2Vox to generate a preliminary voxel grid representation, and selecting high-quality reconstruction areas in the preliminary voxel grids of different perspectives through a context-aware fusion module to remove noise and erroneous reconstruction.
[0014] Preferably, the DeepMVS module combines multi-layer feature aggregation and cross-view feature aggregation, extracts multi-layer feature maps from a single view, generates a plane scan volume, and maps the plane scan volumes of all views into a unified three-dimensional space before performing feature aggregation;
[0015] Within a single view, contextual information is extracted through layer-by-layer feature fusion; between multiple views, complementary information is integrated through maximum pooling to generate a global depth map candidate set. Finally, through further three-dimensional convolution and loss optimization, the final depth map candidate set is generated.
[0016] Preferably, depth map completion uses a self-supervised learning method to construct a customized training objective, i.e., a loss function, to learn depth map completion. The feature extraction and fusion module uses an image neural network to extract multi-pixel features of the depth map, uses a multi-feature pyramid to extract information of different resolutions, and uses a weighted fusion method to fuse the depth map and the RGB image. The depth completion module uses a deconvolution operation to gradually restore the resolution of the depth map, and uses a jump connection to introduce the high-resolution features of the feature extraction and fusion module during the completion process, and outputs the completed depth image. The projection and constraint module projects the completed depth map to other perspectives to generate synthetic views and depths, and uses geometric and photometric consistency constraints to generate target views and calculate losses. The self-supervised optimization module uses a loss function to optimize the network weights, and gradually improves the completion effect through final training.
[0017] Preferably, a self-supervised loss function is constructed, whose core mechanism is mainly based on photometric consistency, parallax consistency and normal consistency. The loss function extracts implicit geometric consistency or physical constraints from the input data itself as a supervision signal for training, helping the model complete the depth map completion.
[0018] A three-dimensional reconstruction system for aircraft auxiliary power unit piping based on visual measurement, the system comprising:
[0019] A data preprocessing module, which takes multi-view RGB images and corresponding camera poses as input and uses the DeepMVS module to generate a planar scan volume as a preliminary estimate of depth information;
[0020] A feature extraction module generates multi-view depth maps. It uses the DeepMVS plane scanning volume generation module to construct a candidate set of depth maps for each reference view. It uses a convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view.
[0021] The completion and fusion module performs depth map completion and fusion. It uses self-supervised learning technology to complete occluded or missing areas and generates a complete depth map representation by fusing depth maps from multiple perspectives based on a contextual scoring mechanism.
[0022] The codec module converts the multi-view RGB images and depth maps into voxel grids. It uses the Pix2Vox convolutional encoder to fuse the multi-view images with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. The depth completion result is introduced as an additional input feature to enhance the ability to capture local details and global geometry of objects.
[0023] An optimization module refines and optimizes the voxel grid. Using the Pix2Vox refinement network, it locally refines the context-fused voxel grid, corrects errors caused by multi-view or depth completion, and outputs a high-resolution, complete 3D voxel grid that accurately describes the pipelines in the aircraft's auxiliary power unit.
[0024] A computer-readable storage medium stores a computer program, which is executed by a processor to implement a three-dimensional reconstruction method for aircraft auxiliary power unit pipelines based on visual measurement.
[0025] A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, a three-dimensional reconstruction method of an aircraft auxiliary power unit pipeline based on visual measurement is implemented.
[0026] The present invention has the following beneficial effects:
[0027] Compared with the prior art, the present invention has the following advantages:
[0028] The present invention integrates multi-view depth estimation, depth map completion and voxel reconstruction. The overall framework is as follows Figure 1 As shown in the figure, the DeepMVS module and the depth map completion module are introduced into the traditional Pix2Vox framework. The multi-view image is processed through the DeepMVS module to generate a preliminary depth image, which is then processed through the depth map completion module to generate a completed depth image. The multi-view image, preliminary depth image, and completed depth image are input into the encoder to obtain high-dimensional features and additional features. The decoder then outputs a preliminary voxel grid. The final 3D reconstruction is obtained using Pix2Vox's context-aware fusion module and refiner.
[0029] This paper combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion results, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; introduces self-supervised learning for depth map completion, further improves the quality of depth map completion, and uses multi-level feature fusion to optimize the depth-to-voxel conversion.
[0030] The present invention integrates multi-view depth estimation, depth map completion and voxelized reconstruction as a whole, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings required to be used in the description of the specific embodiments or prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0032] Figure 1 The figure shows the schematic diagram of the improved algorithm framework of Pix2Vox based on the present application.
[0033] Figure 2 The figure shows the schematic diagram of the algorithm flow of DeepMVS module of the present application.
[0034] Figure 3 The figure shows the schematic diagram of the algorithm flow of the depth map completion module of the present application. DETAILED DESCRIPTION
[0035] The present application is described in detail below in combination with specific embodiments. Embodiment one
[0037] According to Figures 1 to 3 As shown in the figure, the specific optimization technical solution adopted by the present application to solve the above technical problems is: the present application relates to a kind of three-dimensional reconstruction method of aircraft auxiliary power device pipeline based on visual measurement.
[0038] A kind of three-dimensional reconstruction method of aircraft auxiliary power device pipeline based on visual measurement, the method comprises the following steps:
[0039] Step 1: data preprocessing is carried out, multi-view RGB image and corresponding camera pose are used as input, plane scanning volume is generated using DeepMVS module, as preliminary estimation of depth information;
[0040] Step 2: multi-view depth map generation is carried out, the plane scanning volume generation module of DeepMVS is used to construct the depth map candidate set of each reference view, the multi-level features of image are extracted through convolution network, and the accurate depth map of reference view is generated;
[0041] Step 3: depth map completion and fusion are carried out, the occluded or missing area is completed through self-supervised learning technology, based on context score mechanism, a complete depth map representation is generated by fusing depth maps of multiple views;
[0042] Step 4: Transcode the depth map into a voxel grid. Using the Pix2Vox convolutional encoder, the multi-view image is fused with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. The depth completion result is introduced as an additional input feature to enhance the ability to capture local details and global geometry of the object.
[0043] Step 5: Refine and optimize the voxel grid. Using the Pix2Vox refinement network, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion. This outputs a high-resolution, complete 3D voxel grid that accurately describes the complex piping in the aircraft auxiliary power unit.
[0044] This paper combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion results, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; introduces self-supervised learning for depth map completion, further improves the quality of depth map completion, and uses multi-level feature fusion to optimize the depth-to-voxel conversion.
[0045] The present invention integrates multi-view depth estimation, depth map completion and voxelized reconstruction as a whole, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization. Specific embodiment two:
[0047] The difference between the second embodiment of the present application and the first embodiment is that:
[0048] The step 1 further comprises:
[0049] The volume in step 1 includes color projections at different depths for each view, providing a basis for subsequent depth completion, normalizing and cropping the data, and ensuring that the input multi-view image is compatible with the network size of Pix2Vox. Specific embodiment three:
[0051] The only difference between the third embodiment of the present application and the second embodiment is that:
[0052] The step 2 introduces multi-layer aggregation and cross-view feature aggregation, fuses useful information from different viewpoints through the max-pooling method, improves the reconstruction capability of the occluded area, and outputs the depth image of each viewpoint. Specific embodiment four:
[0054] The only difference between the fourth embodiment of the present application and the third embodiment is that:
[0055] The step 4 also includes decoding, inputting the encoded features into the decoder of Pix2Vox to generate a preliminary voxel grid representation, and selecting high-quality reconstruction areas in the preliminary voxel grids of different perspectives through a context-aware fusion module to remove noise and erroneous reconstruction. Specific embodiment five:
[0057] The only difference between the fifth embodiment of the present invention and the fourth embodiment is that:
[0058] The DeepMVS module combines multi-layer feature aggregation and cross-view feature aggregation to extract multi-layer feature maps from a single view, generate a plane scan volume, and then map the plane scan volume of all views into a unified three-dimensional space for feature aggregation.
[0059] Within a single view, contextual information is extracted through layer-by-layer feature fusion; between multiple views, complementary information is integrated through maximum pooling to generate a global depth map candidate set. Finally, through further three-dimensional convolution and loss optimization, the final depth map candidate set is generated. Specific embodiment six:
[0061] The only difference between the sixth embodiment of the present invention and the fifth embodiment is that:
[0062] Depth map completion uses a self-supervised learning method to construct a custom training objective, namely a loss function, to learn depth map completion. The feature extraction and fusion module uses an image neural network to extract multi-pixel features of the depth map, uses a multi-feature pyramid to extract information of different resolutions, and uses a weighted fusion method to fuse the depth map and RGB image; the depth completion module uses deconvolution operations to gradually restore the resolution of the depth map, and uses jump connections to introduce high-resolution features of the feature extraction and fusion module during the completion process, and outputs the completed depth image; the projection and constraint module projects the completed depth map to other perspectives to generate synthetic views and depths, and uses geometric and photometric consistency constraints to generate target views and calculate losses; the self-supervised optimization module uses a loss function to optimize network weights, and gradually improves the completion effect through final training. Specific embodiment seven:
[0064] The only difference between the seventh embodiment of the present invention and the sixth embodiment is that:
[0065] A self-supervised loss function is constructed, whose core mechanism is mainly based on photometric consistency, parallax consistency, and normal consistency. The loss function extracts implicit geometric consistency or physical constraints from the input data itself as a supervision signal for training, helping the model complete depth map completion. Specific embodiment eight:
[0067] The only difference between the eighth embodiment of the present invention and the seventh embodiment is that:
[0068] The present invention provides a three-dimensional reconstruction system for aircraft auxiliary power unit pipelines based on visual measurement, the system comprising:
[0069] A data preprocessing module, which takes multi-view RGB images and corresponding camera poses as input and uses the DeepMVS module to generate a planar scan volume as a preliminary estimate of depth information;
[0070] A feature extraction module generates multi-view depth maps. It uses the DeepMVS plane scanning volume generation module to construct a candidate set of depth maps for each reference view. It uses a convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view.
[0071] The completion and fusion module performs depth map completion and fusion. It uses self-supervised learning technology to complete occluded or missing areas and generates a complete depth map representation by fusing depth maps from multiple perspectives based on a contextual scoring mechanism.
[0072] The codec module converts the multi-view RGB images and depth maps into voxel grids. It uses the Pix2Vox convolutional encoder to fuse the multi-view images with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. The depth completion result is introduced as an additional input feature to enhance the ability to capture local details and global geometry of objects.
[0073] An optimization module refines and optimizes the voxel grid. Using the Pix2Vox refinement network, it locally refines the context-fused voxel grid, corrects errors caused by multi-view or depth completion, and outputs a high-resolution, complete 3D voxel grid that accurately describes the pipelines in the aircraft's auxiliary power unit.
[0074] This paper combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion results, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; introduces self-supervised learning for depth map completion, further improves the quality of depth map completion, and uses multi-level feature fusion to optimize the depth-to-voxel conversion. Specific embodiment nine:
[0076] The only difference between the ninth embodiment of the present invention and the eighth embodiment is that:
[0077] The application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement a visual measurement-based three-dimensional reconstruction method of an auxiliary power unit pipeline of an airplane.
[0078] The method comprises the following steps:
[0079] Step 1: data preprocessing is performed, multi-view RGB images and corresponding camera poses are taken as inputs, a plane scanning volume is generated by using a DeepMVS module, and the plane scanning volume is taken as a preliminary estimation of depth information;
[0080] Step 2: multi-view depth map generation is performed, a depth map candidate set of each reference view is constructed by using a plane scanning volume generation module of DeepMVS, multi-level features of images are extracted by using a convolutional network, and an accurate depth map of the reference view is generated;
[0081] Step 3: depth map completion and fusion are performed, occluded or missing areas are completed by using a self-supervised learning technology, and a complete depth map representation is generated by fusing depth maps of multiple views based on a context score mechanism;
[0082] Step 4: multi-view RGB image and depth map to voxel grid conversion coding is performed, multi-view images and depth maps generated by the DeepMVS module are fused into a context-aware high-dimensional feature representation by using a convolutional encoder of Pix2Vox; and a depth completion result is introduced as an additional input feature to enhance the capturing ability of local details and global geometric shapes of an object;
[0083] Step 5: voxel grid refinement and optimization are performed, the context-fused voxel grid is locally refined by using a refinement network of Pix2Vox, errors generated due to multi-view or depth completion are corrected, and a high-resolution, complete three-dimensional voxel grid is output, which accurately describes complex pipelines in an auxiliary power unit of an airplane. Specific embodiment ten:
[0085] The difference between the embodiment ten and the embodiment nine is only that:
[0086] The application provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements a visual measurement-based three-dimensional reconstruction method of an auxiliary power unit pipeline of an airplane when executing the computer program.
[0087] The method comprises the following steps:
[0088] Step 1: data preprocessing is performed, multi-view RGB images and corresponding camera poses are taken as inputs, a plane scanning volume is generated by using a DeepMVS module, and the plane scanning volume is taken as a preliminary estimation of depth information;
[0089] Step 2: Generate multi-view depth maps. Use the DeepMVS plane scanning volume generation module to build a candidate set of depth maps for each reference view. Use the convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view.
[0090] Step 3: Perform depth map completion and fusion. Use self-supervised learning technology to complete occluded or missing areas. Based on the context scoring mechanism, generate a complete depth map representation by fusing depth maps from multiple perspectives.
[0091] Step 4: Transcode the multi-view RGB image and depth map into a voxel grid. Using the Pix2Vox convolutional encoder, fuse the multi-view image with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. The depth completion result is introduced as an additional input feature to enhance the ability to capture local details and global geometry of the object.
[0092] Step 5: Refine and optimize the voxel grid. Using the Pix2Vox refinement network, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion. This outputs a high-resolution, complete 3D voxel grid that accurately describes the complex piping in the aircraft auxiliary power unit.
[0093] This paper combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion results, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; introduces self-supervised learning for depth map completion, further improves the quality of depth map completion, and uses multi-level feature fusion to optimize the depth-to-voxel conversion.
[0094] The present invention integrates multi-view depth estimation, depth map completion and voxelized reconstruction as a whole, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization. Specific embodiment eleven:
[0096] The only difference between the eleventh embodiment of the present invention and the tenth embodiment is that:
[0097] Aiming at the problem of insufficient restoration of local details in the 3D reconstruction of complex ducts in aircraft auxiliary power units, a 3D reconstruction method for aircraft auxiliary power unit ducts based on visual measurement was proposed.
[0098] This method integrates multi-view depth estimation, depth map completion and voxel reconstruction. The overall framework is as follows Figure 1 As shown in the figure, the DeepMVS module and the depth map completion module are introduced into the traditional Pix2Vox framework. The multi-view image is processed through the DeepMVS module to generate a preliminary depth image, which is then processed through the depth map completion module to generate a completed depth image. The multi-view image, preliminary depth image, and completed depth image are input into the encoder to obtain high-dimensional features and additional features. The decoder then outputs a preliminary voxel grid. The final 3D reconstruction is obtained using Pix2Vox's context-aware fusion module and refiner.
[0099] The following DeepMVS module is constructed, and the algorithm flow chart is as follows Figure 2 This module combines multi-layer feature aggregation and cross-view feature aggregation to extract multi-layer feature maps from a single view, generate a plane scan volume, and then map the plane scan volumes of all views into a unified three-dimensional space for feature aggregation. Within a single view, contextual information is extracted through layer-by-layer feature fusion. Between multiple views, maximum pooling is used to integrate complementary information to generate a global depth map candidate set. Finally, further three-dimensional convolution and loss optimization are performed to generate the final depth map candidate set.
[0100] This module's convolutional network, which combines multi-layer feature aggregation and cross-view feature aggregation, can significantly improve the reconstruction ability of complex scenes, especially when it comes to geometric differences between multiple perspectives.
[0101] The following depth map completion module is built using a self-supervised learning method to build a custom training objective, i.e., a loss function, to learn depth map completion. The algorithm flow is as follows: Figure 3 As shown in the figure, the feature extraction and fusion module uses an image neural network to extract multi-pixel features from the depth map, utilizes a multi-feature pyramid to extract information at different resolutions, and fuses the depth map and RGB image using a weighted fusion approach. The depth completion module uses deconvolution operations to gradually restore the resolution of the depth map. During the completion process, it uses jump connections to introduce high-resolution features from the feature extraction and fusion module and outputs the completed depth image. The projection and constraint module projects the completed depth map to other viewpoints to generate synthetic views and depths. It uses geometric and photometric consistency constraints to generate the target view and calculate the loss. The self-supervised optimization module uses a loss function to optimize the network weights and gradually improves the completion effect through final training.
[0102] Furthermore, a self-supervised loss function is constructed. Its core mechanism is mainly based on photometric consistency, parallax consistency, and normal consistency. The loss function extracts implicit geometric consistency or physical constraints from the input data itself as a supervision signal for training, helping the model complete the depth map. The construction process is as follows:
[0103] Photometric consistency loss function:
[0104]
[0105] Among them I i (p) and I j (π(p,D i )) is the image pixel value at view angle i and j; π(p,D i ) is through the depth D i Project pixel p of view i to the position of view j.
[0106] Disparity consistency loss function:
[0107]
[0108] Among them D i (p) is the depth value of view i, D j (π(p,D i )) is through D i The depth value projected to view angle j.
[0109] Normal consistency loss function:
[0110]
[0111] Where N(p) is the normal vector of pixel p, is the average of the neighborhood normal vectors.
[0112] Smooth loss function:
[0113]
[0114] Occlusion processing loss function:
[0115] L m =L p M(p) (5)
[0116] Where M(p) is the occlusion mask, which can be obtained by multi-view disparity inconsistency detection.
[0117] By combining the above loss functions, the total self-supervised loss function is constructed:
[0118] L=λ1L p +λ2L d +λ3L n +λ4L s +λ5L m (6)
[0119] Among them, λ1, λ2, λ3, λ4, and λ5 are weights.
[0120] End-to-end training is performed using a self-supervised loss function. During training, occlusion masks are dynamically generated, ignoring the contribution of occluded areas to the loss. Finally, the trained model is used to complete the missing areas and generate a completed depth image. By constructing a loss function related to geometric consistency and physical constraints, this module can learn completion rules from unlabeled data, achieving end-to-end training for depth completion.
[0121] This paper combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion results, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; introduces self-supervised learning for depth map completion, further improves the quality of depth map completion, and uses multi-level feature fusion to optimize the depth-to-voxel conversion.
[0122] The present invention integrates multi-view depth estimation, depth map completion and voxelized reconstruction as a whole, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization. Specific embodiment 12:
[0124] The difference between the twelfth embodiment of the present invention and the eleventh embodiment is that:
[0125] There is a pipeline in the auxiliary power unit of an aircraft. Multi-view images have been taken from different angles by a camera and the camera calibration has been completed. Now the pipeline is reconstructed in three dimensions.
[0126] Step 1: Data Preprocessing
[0127] Taking multi-view RGB images and their corresponding camera poses as input, DeepMVS generates planar scan volumes as preliminary depth estimates. These volumes include color projections at different depths for each view, providing the basis for subsequent depth completion. This data is normalized and cropped to ensure that the input multi-view images are compatible with the Pix2Vox network size.
[0128] Step 2: Multi-view depth map generation
[0129] The plane scanning volume generation module of DeepMVS is used to construct a candidate set of depth maps for each reference view. The multi-level features of the image are extracted through the convolutional network, and an accurate depth map of the reference view is generated.
[0130] Multi-layer aggregation and cross-view feature aggregation are introduced, and useful information from different perspectives is integrated through max-pooling to improve the reconstruction ability of occluded areas. The depth image of each perspective is output.
[0131] Step 3: Depth map completion and fusion
[0132] Using self-supervised learning techniques, we can complete occluded or missing areas. Based on a contextual scoring mechanism, we can generate a complete depth map representation by fusing depth maps from multiple perspectives.
[0133] Step 4: Depth Map to Voxel Grid Conversion
[0134] Encoding stage: Use Pix2Vox's convolutional encoder to fuse multi-view images with the generated depth map into a context-aware high-dimensional feature representation; introduce the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric morphology of objects.
[0135] Decoding stage: The encoded features are input into the decoder of Pix2Vox to generate a preliminary voxel grid representation. Through the context-aware fusion module, high-quality reconstruction areas are selected from the preliminary voxel grids at different viewpoints to remove noise and erroneous reconstructions.
[0136] Step 5: Voxel Grid Refinement and Optimization
[0137] Using Pix2Vox's refinement network, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion, outputting a high-resolution, complete 3D voxel grid that can accurately describe the complex piping in the aircraft auxiliary power unit.
[0138] This paper combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion results, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; introduces self-supervised learning for depth map completion, further improves the quality of depth map completion, and uses multi-level feature fusion to optimize the depth-to-voxel conversion.
[0139] The present invention integrates multi-view depth estimation, depth map completion and voxelized reconstruction as a whole, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization.
[0140] The above description is merely a preferred embodiment of a method for 3D reconstruction of aircraft auxiliary power unit piping based on visual measurement. The scope of protection for a method for 3D reconstruction of aircraft auxiliary power unit piping based on visual measurement is not limited to the aforementioned embodiment; all technical solutions based on this concept fall within the scope of protection of the present invention. It should be noted that improvements and variations that do not depart from the principles of the present invention, as readily apparent to those skilled in the art, should also be considered within the scope of protection of the present invention.
Claims
1. A method for 3D reconstruction of aircraft auxiliary power unit piping based on visual measurement, characterized by: The method comprises the following steps: Step 1: Perform data preprocessing, take the multi-view RGB images and the corresponding camera poses as input, and use the DeepMVS module to generate a planar scan volume as a preliminary estimate of the depth information; Step 2: Generate multi-view depth maps. Use the DeepMVS plane scanning volume generation module to build a candidate set of depth maps for each reference view. Use the convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view. Step 3: Perform depth map completion and fusion. Use self-supervised learning technology to complete occluded or missing areas. Based on the context scoring mechanism, generate a complete depth map representation by fusing depth maps from multiple perspectives. Step 4: Transcode the multi-view RGB image and depth map into a voxel grid. Using the Pix2Vox convolutional encoder, fuse the multi-view image with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. The depth completion result is introduced as an additional input feature to enhance the ability to capture local details and global geometry of the object. Step 5: Refine and optimize the voxel grid. Pix2Vox's refinement network locally refines the context-fused voxel grid, correcting errors caused by multi-view or depth completion. This outputs a high-resolution, complete 3D voxel grid that accurately describes the pipes in the aircraft's auxiliary power unit. The DeepMVS module combines multi-layer feature aggregation and cross-view feature aggregation to extract multi-layer feature maps from a single view, generate a plane scan volume, and then map the plane scan volume of all views into a unified three-dimensional space for feature aggregation. Within a single view, contextual information is extracted through layer-by-layer feature fusion; between multiple views, complementary information is integrated through maximum pooling to generate a global depth map candidate set. Finally, through further 3D convolution and loss optimization, the final depth map candidate set is generated. Depth map completion uses a self-supervised learning method to construct a custom training objective, namely a loss function, to learn depth map completion. The feature extraction and fusion module uses an image neural network to extract multi-pixel features of the depth map, uses a multi-feature pyramid to extract information of different resolutions, and uses a weighted fusion method to fuse the depth map and RGB image; the depth completion module uses deconvolution operations to gradually restore the resolution of the depth map, and uses jump connections to introduce high-resolution features of the feature extraction and fusion module during the completion process, and outputs the completed depth image; the projection and constraint module projects the completed depth map to other perspectives to generate synthetic views and depths, and uses geometric and photometric consistency constraints to generate target views and calculate losses; the self-supervised optimization module uses a loss function to optimize network weights, and gradually improves the completion effect through final training.
2. The method according to claim 1, wherein: The volume in step 1 includes color projections at different depths for each view, providing a basis for subsequent depth completion, normalizing and cropping the data, and ensuring that the input multi-view image is compatible with the network size of Pix2Vox.
3. The method according to claim 2, wherein: The step 2 introduces multi-layer aggregation and cross-view feature aggregation, fuses useful information from different viewpoints through the max-pooling method, improves the reconstruction capability of the occluded area, and outputs the depth image of each viewpoint.
4. The method according to claim 3, wherein: The step 4 also includes decoding, inputting the encoded features into the decoder of Pix2Vox to generate a preliminary voxel grid representation, and selecting high-quality reconstruction areas in the preliminary voxel grids of different perspectives through a context-aware fusion module to remove noise and erroneous reconstruction.
5. The method according to claim 4, wherein: A self-supervised loss function is constructed, whose core mechanism is mainly based on photometric consistency, parallax consistency, and normal consistency. The loss function extracts implicit geometric consistency or physical constraints from the input data itself as a supervision signal for training, helping the model complete depth map completion.
6. A three-dimensional reconstruction system for aircraft auxiliary power unit piping based on visual measurement, characterized by: The system comprises: A data preprocessing module, which takes multi-view RGB images and corresponding camera poses as input and uses the DeepMVS module to generate a planar scan volume as a preliminary estimate of depth information; A feature extraction module generates multi-view depth maps. It uses the DeepMVS plane scanning volume generation module to construct a candidate set of depth maps for each reference view. It uses a convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view. The completion and fusion module performs depth map completion and fusion. It uses self-supervised learning technology to complete occluded or missing areas and generates a complete depth map representation by fusing depth maps from multiple perspectives based on a contextual scoring mechanism. The codec module converts the multi-view RGB images and depth maps into voxel grids. It uses the Pix2Vox convolutional encoder to fuse the multi-view images with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. The depth completion result is introduced as an additional input feature to enhance the ability to capture local details and global geometry of objects. An optimization module, which refines and optimizes the voxel grid. Using the Pix2Vox refinement network, it locally refines the context-fused voxel grid, corrects errors caused by multi-view or depth completion, and outputs a high-resolution, complete 3D voxel grid that accurately describes the pipelines in the aircraft auxiliary power unit. The DeepMVS module combines multi-layer feature aggregation and cross-view feature aggregation to extract multi-layer feature maps from a single view, generate a plane scan volume, and then map the plane scan volume of all views into a unified three-dimensional space for feature aggregation. Within a single view, contextual information is extracted through layer-by-layer feature fusion; between multiple views, complementary information is integrated through maximum pooling to generate a global depth map candidate set. Finally, through further 3D convolution and loss optimization, the final depth map candidate set is generated. Depth map completion uses a self-supervised learning method to construct a custom training objective, namely a loss function, to learn depth map completion. The feature extraction and fusion module uses an image neural network to extract multi-pixel features of the depth map, uses a multi-feature pyramid to extract information of different resolutions, and uses a weighted fusion method to fuse the depth map and RGB image; the depth completion module uses deconvolution operations to gradually restore the resolution of the depth map, and uses jump connections to introduce high-resolution features of the feature extraction and fusion module during the completion process, and outputs the completed depth image; the projection and constraint module projects the completed depth map to other perspectives to generate synthetic views and depths, and uses geometric and photometric consistency constraints to generate target views and calculate losses; the self-supervised optimization module uses a loss function to optimize network weights, and gradually improves the completion effect through final training.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 5.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.