Airplane auxiliary power device pipeline three-dimensional reconstruction method based on vision measurement

Abstract

The invention relates to a three-dimensional reconstruction method for an airplane auxiliary power device pipeline based on vision measurement. The invention relates to the technical field of computer vision measurement, and aims to solve the problem that local detail reduction is not fine enough in three-dimensional reconstruction of complex conduit measurement in an aircraft auxiliary power device, data preprocessing is carried out, a multi-view RGB image and a corresponding camera pose are used as input, a DeepMVS module is used for generating a plane scanning volume, and the plane scanning volume is calculated. Performing preliminary estimation on the depth information; generating a multi-view-angle depth map, and generating an accurate depth map of a reference view angle; complementing and fusing the depth maps, and generating a complete depth map representation by fusing the depth maps of multiple visual angles; and performing conversion coding from the multi-view RGB image, the original depth map and the complete depth map to the voxel grid, refining and optimizing the voxel grid, and accurately describing the complex pipeline in the aircraft auxiliary power device.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision measurement, and is a three-dimensional reconstruction method of aircraft auxiliary power unit pipelines based on vision measurement. Background Art

[0002] In the field of aerospace, the aircraft auxiliary power unit is a key component in aircraft operation, and its complex duct system plays an important role in energy distribution, air circulation and cooling. These ducts usually have complex geometric structures, high precision requirements and close distribution. Their measurement and 3D reconstruction are a major technical difficulty in the manufacture and maintenance of aircraft auxiliary power units.

[0003] Traditional 3D reconstruction methods, such as algorithms based on multi-view geometry (MVS), can complete the 3D reconstruction of complex pipelines to a certain extent, but they have problems such as low efficiency, insufficient accuracy, and over-reliance on high-quality annotation data when dealing with local occlusion and complex deformation structures. They cannot fully cope with the challenges brought by data scarcity and local occlusion in the complex pipeline system of aircraft auxiliary power units. In recent years, 3D reconstruction technology based on deep learning has developed rapidly. The Pix2Vox algorithm 3D reconstruction technology generates voxelized 3D models from single-view or multi-view images, which can effectively capture global geometric features and achieve a good balance between speed and accuracy. However, it is not fine enough in restoring local details, and there is still room for improvement in the recovery of occluded areas. Further optimization is needed to improve the accuracy of reconstruction. Summary of the invention

[0004] The present invention aims at the problem that the local details in the three-dimensional reconstruction of the complex conduit measurement in the aircraft auxiliary power unit are not restored finely enough. The present invention provides a three-dimensional reconstruction method of the aircraft auxiliary power unit pipeline based on visual measurement. The present invention provides the following technical solutions:

[0005] A method for three-dimensional reconstruction of aircraft auxiliary power unit pipelines based on visual measurement, the method comprising the following steps:

[0006] Step 1: Perform data preprocessing, take multi-view RGB images and corresponding camera poses as input, and use the DeepMVS module to generate a planar scan volume as a preliminary estimate of depth information;

[0007] Step 2: Generate multi-view depth maps. Use the plane scanning volume generation module of DeepMVS to build a candidate set of depth maps for each reference view. Use the convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view.

[0008] Step 3: Perform depth map completion and fusion. Use self-supervised learning technology to complete the occluded or missing areas. Based on the context scoring mechanism, generate a complete depth map representation by fusing depth maps from multiple perspectives.

[0009] Step 4: Transcode the multi-view RGB image and depth map into a voxel grid. Use the convolutional encoder of Pix2Vox to fuse the multi-view image with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. Introduce the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric morphology of the object.

[0010] Step 5: Refine and optimize the voxel grid. Through the refinement network of Pix2Vox, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion, and a high-resolution, complete three-dimensional voxel grid is output to accurately describe the complex pipelines in the aircraft auxiliary power unit.

[0011] Preferably, the volume in step 1 includes color projections at different depths for each view, providing a basis for subsequent depth completion, and normalizing and cropping the data to ensure that the input multi-view image is compatible with the network size of Pix2Vox.

[0012] Preferably, the step 2 introduces multi-layer aggregation and cross-view feature aggregation, fuses useful information from different viewpoints by means of max-pooling, improves the reconstruction capability of occluded areas, and outputs a depth image of each viewpoint.

[0013] Preferably, step 4 further comprises decoding, inputting the encoded features into a decoder of Pix2Vox to generate a preliminary voxel grid representation, and selecting high-quality reconstruction areas in the preliminary voxel grids at different perspectives through a context-aware fusion module to remove noise and erroneous reconstruction.

[0014] Preferably, the DeepMVS module combines multi-layer feature aggregation and cross-view feature aggregation, extracts multi-layer feature maps from a single view, generates a plane scanning volume, and performs feature aggregation after mapping the plane scanning volumes of all views into a unified three-dimensional space;

[0015] Within a single view, contextual information is extracted through layer-by-layer feature fusion; between multiple views, complementary information is integrated through maximum pooling to generate a global depth map candidate set, and finally, the final depth map candidate set is generated through further three-dimensional convolution and loss optimization.

[0016] Preferably, the depth map completion uses a self-supervised learning method to construct a custom training objective, i.e., a loss function, to learn the depth map completion. The feature extraction and fusion module uses an image neural network to extract multi-pixel features of the depth map, uses a multi-feature pyramid to extract information of different resolutions, and uses a weighted fusion method to fuse the depth map and the RGB image. The depth completion module uses a deconvolution operation to gradually restore the resolution of the depth map, and uses a jump connection to introduce the high-resolution features of the feature extraction and fusion module during the completion process, and outputs the completed depth image. The projection and constraint module projects the completed depth map to other perspectives to generate synthetic views and depths, and uses geometric and photometric consistency constraints to generate target views and calculate losses. The self-supervised optimization module uses a loss function to optimize the network weights, and gradually improves the completion effect through final training.

[0017] Preferably, a self-supervised loss function is constructed, whose core mechanism is mainly based on photometric consistency, parallax consistency and normal consistency. The loss function extracts implicit geometric consistency or physical constraints from the input data itself as a supervision signal for training to help the model complete the depth map completion.

[0018] A three-dimensional reconstruction system for aircraft auxiliary power unit pipelines based on visual measurement, the system comprising:

[0019] A data preprocessing module, which takes the multi-view RGB images and the corresponding camera poses as input and uses the DeepMVS module to generate a planar scan volume as a preliminary estimate of the depth information;

[0020] A feature extraction module, which generates multi-view depth maps, constructs a candidate set of depth maps for each reference view through the plane scanning volume generation module of DeepMVS, extracts multi-level features of the image through a convolutional network, and generates an accurate depth map for the reference view;

[0021] A completion and fusion module, which performs depth map completion and fusion, completes occluded or missing areas through self-supervised learning technology, and generates a complete depth map representation by fusing depth maps of multiple perspectives based on a context scoring mechanism;

[0022] The encoding and decoding module performs conversion encoding of multi-view RGB images and depth maps to voxel grids, uses the convolutional encoder of Pix2Vox to fuse the multi-view images with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation; introduces the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric forms of objects;

[0023] The optimization module refines and optimizes the voxel grid. Through the refinement network of Pix2Vox, the context-fused voxel grid is locally refined, errors caused by multi-view or depth completion are corrected, and a high-resolution, complete three-dimensional voxel grid is output to accurately describe the pipelines in the aircraft auxiliary power unit.

[0024] A computer-readable storage medium stores a computer program, which is executed by a processor to implement a three-dimensional reconstruction method of aircraft auxiliary power unit pipelines based on visual measurement.

[0025] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a three-dimensional reconstruction method of an aircraft auxiliary power unit pipeline based on visual measurement is implemented.

[0026] The present invention has the following beneficial effects:

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] The present invention integrates multi-view depth estimation, depth map completion and voxel reconstruction. The overall framework is as follows Figure 1 As shown in the figure, the DeepMVS module and the depth map completion module are introduced into the traditional Pix2Vox framework. The multi-view image generates a preliminary depth image through the DeepMVS module, and then generates a completed depth image through the depth map completion module. The multi-view image, the preliminary depth image and the completed depth image are input into the encoder to obtain high-dimensional features and additional features, and then the decoder outputs a preliminary voxel grid. The context-aware fusion module and refiner of Pix2Vox are used to obtain the final 3D reconstruction result.

[0029] The present invention combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion result, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; further improves the quality of depth map completion by introducing self-supervised learning for depth map completion, and optimizes the conversion from depth to voxel by using multi-level feature fusion.

[0030] The present invention as a whole integrates multi-view depth estimation, depth map completion and voxelized reconstruction, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 Shown is a schematic diagram of the improved algorithm framework based on Pix2Vox of the present invention;

[0033] Figure 2 Shown is a schematic diagram of the DeepMVS module algorithm flow of the present invention;

[0034] Figure 3 Shown is a schematic diagram of the algorithm flow of the depth map completion module of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with specific embodiments. Specific embodiment one:

[0037] according to Figures 1 to 3 As shown, the specific optimization technical solution adopted by the present invention to solve the above technical problems is: the present invention relates to a three-dimensional reconstruction method of aircraft auxiliary power unit pipelines based on visual measurement.

[0038] A method for three-dimensional reconstruction of aircraft auxiliary power unit pipelines based on visual measurement, the method comprising the following steps:

[0039] Step 1: Perform data preprocessing, take multi-view RGB images and corresponding camera poses as input, and use the DeepMVS module to generate a planar scan volume as a preliminary estimate of depth information;

[0040] Step 2: Generate multi-view depth maps. Use the plane scanning volume generation module of DeepMVS to build a candidate set of depth maps for each reference view. Use the convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view.

[0041] Step 3: Perform depth map completion and fusion. Use self-supervised learning technology to complete the occluded or missing areas. Based on the context scoring mechanism, generate a complete depth map representation by fusing depth maps from multiple perspectives.

[0042] Step 4: Perform conversion encoding from depth map to voxel grid, use Pix2Vox's convolutional encoder to fuse the multi-view image with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation; introduce the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric morphology of the object;

[0043] Step 5: Refine and optimize the voxel grid. Through the refinement network of Pix2Vox, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion, and a high-resolution, complete three-dimensional voxel grid is output to accurately describe the complex pipelines in the aircraft auxiliary power unit.

[0044] The present invention combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion result, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; further improves the quality of depth map completion by introducing self-supervised learning for depth map completion, and optimizes the conversion from depth to voxel by using multi-level feature fusion.

[0045] The present invention as a whole integrates multi-view depth estimation, depth map completion and voxelized reconstruction, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization. Specific embodiment 2:

[0047] The difference between the second embodiment of the present application and the first embodiment is that:

[0048] The step 1 also includes:

[0049] The volume in step 1 includes color projections at different depths for each view, providing a basis for subsequent depth completion, and normalizing and cropping the data to ensure that the input multi-view image is compatible with the network size of Pix2Vox. Specific embodiment three:

[0051] The difference between the third embodiment of the present application and the second embodiment is that:

[0052] The step 2 introduces multi-layer aggregation and cross-view feature aggregation, fuses useful information from different viewpoints through the max-pooling method, improves the reconstruction capability of the occluded area, and outputs a depth image of each viewpoint. Specific embodiment four:

[0054] The difference between the fourth embodiment of the present application and the third embodiment is that:

[0055] The step 4 also includes decoding, inputting the encoded features into the decoder of Pix2Vox to generate a preliminary voxel grid representation, and selecting high-quality reconstruction areas in the preliminary voxel grids of different perspectives through a context-aware fusion module to remove noise and erroneous reconstruction. Specific embodiment five:

[0057] The difference between the fifth embodiment of the present invention and the fourth embodiment is that:

[0058] The DeepMVS module combines multi-layer feature aggregation and cross-view feature aggregation to extract multi-layer feature maps from a single view, generate a plane scanning volume, and perform feature aggregation after mapping the plane scanning volumes of all views into a unified three-dimensional space;

[0059] Within a single view, contextual information is extracted through layer-by-layer feature fusion; between multiple views, complementary information is integrated through maximum pooling to generate a global depth map candidate set, and finally, the final depth map candidate set is generated through further three-dimensional convolution and loss optimization. Specific embodiment six:

[0061] The difference between the sixth embodiment of the present invention and the fifth embodiment is that:

[0062] Depth map completion uses a self-supervised learning method to build a custom training objective, namely a loss function, to learn depth map completion. The feature extraction and fusion module uses an image neural network to extract multi-pixel features of the depth map, uses a multi-feature pyramid to extract information of different resolutions, and uses a weighted fusion method to fuse the depth map and RGB image. The depth completion module uses a deconvolution operation to gradually restore the resolution of the depth map. During the completion process, it uses a jump connection to introduce the high-resolution features of the feature extraction and fusion module and outputs the completed depth image. The projection and constraint module projects the completed depth map to other perspectives to generate synthetic views and depths, and uses geometric and photometric consistency constraints to generate target views and calculate losses. The self-supervised optimization module uses a loss function to optimize the network weights and gradually improves the completion effect through final training. Specific embodiment seven:

[0064] The difference between the seventh embodiment of the present invention and the sixth embodiment is that:

[0065] A self-supervised loss function is constructed, whose core mechanism is mainly based on photometric consistency, parallax consistency and normal consistency. The loss function extracts implicit geometric consistency or physical constraints from the input data itself as a supervision signal for training to help the model complete the depth map completion. Specific embodiment eight:

[0067] The difference between the eighth embodiment of the present invention and the seventh embodiment is that:

[0068] The present invention provides a three-dimensional reconstruction system for aircraft auxiliary power unit pipelines based on visual measurement, the system comprising:

[0069] A data preprocessing module, which takes the multi-view RGB images and the corresponding camera poses as input and uses the DeepMVS module to generate a planar scan volume as a preliminary estimate of the depth information;

[0070] A feature extraction module, which generates multi-view depth maps, constructs a candidate set of depth maps for each reference view through the plane scanning volume generation module of DeepMVS, extracts multi-level features of the image through a convolutional network, and generates an accurate depth map for the reference view;

[0071] A completion and fusion module, which performs depth map completion and fusion, completes occluded or missing areas through self-supervised learning technology, and generates a complete depth map representation by fusing depth maps of multiple perspectives based on a context scoring mechanism;

[0072] The encoding and decoding module performs conversion encoding of multi-view RGB images and depth maps to voxel grids, uses the convolutional encoder of Pix2Vox to fuse the multi-view images with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation; introduces the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric forms of objects;

[0073] The optimization module refines and optimizes the voxel grid. Through the refinement network of Pix2Vox, the context-fused voxel grid is locally refined, errors caused by multi-view or depth completion are corrected, and a high-resolution, complete three-dimensional voxel grid is output to accurately describe the pipelines in the aircraft auxiliary power unit.

[0074] The present invention combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion result, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; further improves the quality of depth map completion by introducing self-supervised learning for depth map completion, and optimizes the conversion from depth to voxel by using multi-level feature fusion. Specific embodiment nine:

[0076] The difference between the ninth embodiment of the present invention and the eighth embodiment is that:

[0077] The present invention provides a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement a three-dimensional reconstruction method of an aircraft auxiliary power unit pipeline based on visual measurement.

[0078] The method comprises the following steps:

[0079] Step 1: Perform data preprocessing, take multi-view RGB images and corresponding camera poses as input, and use the DeepMVS module to generate a planar scan volume as a preliminary estimate of depth information;

[0080] Step 2: Generate multi-view depth maps. Use the plane scanning volume generation module of DeepMVS to build a candidate set of depth maps for each reference view. Use the convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view.

[0081] Step 3: Perform depth map completion and fusion. Use self-supervised learning technology to complete the occluded or missing areas. Based on the context scoring mechanism, generate a complete depth map representation by fusing depth maps from multiple perspectives.

[0082] Step 4: Transcode the multi-view RGB image and depth map into a voxel grid. Use the convolutional encoder of Pix2Vox to fuse the multi-view image with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. Introduce the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric morphology of the object.

[0083] Step 5: Refine and optimize the voxel grid. Through the refinement network of Pix2Vox, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion, and a high-resolution, complete three-dimensional voxel grid is output to accurately describe the complex pipelines in the aircraft auxiliary power unit. Specific embodiment ten:

[0085] The difference between the tenth embodiment of the present invention and the ninth embodiment is that:

[0086] The present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a three-dimensional reconstruction method of an aircraft auxiliary power unit pipeline based on visual measurement is implemented.

[0087] The method comprises the following steps:

[0088] Step 1: Perform data preprocessing, take multi-view RGB images and corresponding camera poses as input, and use the DeepMVS module to generate a planar scan volume as a preliminary estimate of depth information;

[0089] Step 2: Generate multi-view depth maps. Use the plane scanning volume generation module of DeepMVS to build a candidate set of depth maps for each reference view. Use the convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view.

[0090] Step 3: Perform depth map completion and fusion. Use self-supervised learning technology to complete the occluded or missing areas. Based on the context scoring mechanism, generate a complete depth map representation by fusing depth maps from multiple perspectives.

[0091] Step 4: Transcode the multi-view RGB image and depth map into a voxel grid. Use the convolutional encoder of Pix2Vox to fuse the multi-view image with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. Introduce the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric morphology of the object.

[0092] Step 5: Refine and optimize the voxel grid. Through the refinement network of Pix2Vox, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion, and a high-resolution, complete three-dimensional voxel grid is output to accurately describe the complex pipelines in the aircraft auxiliary power unit.

[0093] The present invention combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion result, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; further improves the quality of depth map completion by introducing self-supervised learning for depth map completion, and optimizes the conversion from depth to voxel by using multi-level feature fusion.

[0094] The present invention as a whole integrates multi-view depth estimation, depth map completion and voxelized reconstruction, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization. Specific embodiment eleven:

[0096] The difference between the eleventh embodiment of the present invention and the tenth embodiment is that:

[0097] Aiming at the problem of insufficient restoration of local details in 3D reconstruction of complex ducts in aircraft auxiliary power units, a 3D reconstruction method of aircraft auxiliary power unit pipelines based on visual measurement was proposed.

[0098] This method integrates multi-view depth estimation, depth map completion and voxel reconstruction. The overall framework is as follows Figure 1 As shown in the figure, the DeepMVS module and the depth map completion module are introduced into the traditional Pix2Vox framework. The multi-view image generates a preliminary depth image through the DeepMVS module, and then generates a completed depth image through the depth map completion module. The multi-view image, the preliminary depth image and the completed depth image are input into the encoder to obtain high-dimensional features and additional features, and then the decoder outputs a preliminary voxel grid. The context-aware fusion module and refiner of Pix2Vox are used to obtain the final 3D reconstruction result.

[0099] The following DeepMVS module is constructed, and the algorithm flow chart is as follows Figure 2 This module combines multi-layer feature aggregation and cross-view feature aggregation, extracts multi-layer feature maps from a single view, generates a plane scan volume, and performs feature aggregation after mapping the plane scan volumes of all views to a unified three-dimensional space: within a single view, context information is extracted through layer-by-layer feature fusion; between multiple views, complementary information is integrated through maximum pooling to generate a global depth map candidate set, and finally the final depth map candidate set is generated through further three-dimensional convolution and loss optimization.

[0100] This module's convolutional network that combines multi-layer feature aggregation and cross-view feature aggregation can significantly improve the ability to reconstruct complex scenes, especially when there are geometric differences between multiple perspectives.

[0101] The following depth map completion module is built. Here, the self-supervised learning method is used to build a custom training target, i.e., loss function, to learn depth map completion. The algorithm flow is as follows: Figure 3 As shown in the figure. The feature extraction and fusion module uses an image neural network to extract multi-pixel features of the depth map, uses a multi-feature pyramid to extract information of different resolutions, and fuses the depth map and RGB image in a weighted fusion manner; the depth completion module uses a deconvolution operation to gradually restore the resolution of the depth map, and uses a jump connection to introduce the high-resolution features of the feature extraction and fusion module during the completion process, and outputs the completed depth image; the projection and constraint module projects the completed depth map to other perspectives to generate synthetic views and depths, and uses geometric and photometric consistency constraints to generate target views and calculate losses; the self-supervised optimization module uses a loss function to optimize the network weights, and gradually improves the completion effect through final training.

[0102] Furthermore, a self-supervised loss function is constructed, whose core mechanism is mainly based on photometric consistency, parallax consistency and normal consistency. The loss function extracts implicit geometric consistency or physical constraints from the input data itself as a supervisory signal for training to help the model complete the depth map. The construction process is as follows:

[0103] Photometric consistency loss function:

[0104]

[0105] Among them I i (p) and I j (π(p,D i )) is the image pixel value at view angle i and j; π(p,D i ) is through the depth D i Project pixel p of view i to the position of view j.

[0106] Disparity consistency loss function:

[0107]

[0108] Where D i (p) is the depth value of view i, D j (π(p,D i )) is through D i The depth value projected to viewpoint j.

[0109] Normal consistency loss function:

[0110]

[0111] where N(p) is the normal vector of pixel p, is the average of the neighborhood normal vectors.

[0112] Smooth loss function:

[0113]

[0114] Occlusion processing loss function:

[0115] L m =L p M(p) (5)

[0116] Where M(p) is the occlusion mask, which can be obtained by multi-view disparity inconsistency detection.

[0117] By combining the above loss functions, the total self-supervised loss function is constructed:

[0118] L=λ1L p +λ2L d +λ3L n +λ4L s +λ5L m (6)

[0119] Among them, λ1, λ2, λ3, λ4, λ5 are weights.

[0120] End-to-end training is performed through a self-supervised loss function. During the training process, an occlusion mask is dynamically generated, and the contribution of the occluded area to the loss is ignored. Finally, the trained model is used to complete the missing area and generate a completed depth image. This module can learn the completion rules from unlabeled data by constructing loss functions related to geometric consistency and physical constraints, thus achieving end-to-end training of depth completion.

[0121] The present invention combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion result, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; further improves the quality of depth map completion by introducing self-supervised learning for depth map completion, and optimizes the conversion from depth to voxel by using multi-level feature fusion.

[0122] The present invention as a whole integrates multi-view depth estimation, depth map completion and voxelized reconstruction, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization. Specific embodiment twelve:

[0124] The difference between the twelfth embodiment of the present invention and the eleventh embodiment is that:

[0125] There is a pipeline in an auxiliary power unit of an aircraft. Multi-view images are taken from different angles by a camera and the camera calibration is completed. Now, three-dimensional reconstruction is performed on the pipeline.

[0126] Step 1: Data Preprocessing

[0127] Taking multi-view RGB images and corresponding camera poses as input, DeepMVS method is used to generate planar scan volumes as a preliminary estimate of depth information. These volumes include color projections at different depths for each view, providing a basis for subsequent depth completion. These data are standardized and cropped to ensure that the input multi-view images are compatible with the network size of Pix2Vox.

[0128] Step 2: Multi-view depth map generation

[0129] The plane scanning volume generation module of DeepMVS is used to construct a candidate set of depth maps for each reference view. The multi-level features of the image are extracted through a convolutional network, and an accurate depth map of the reference view is generated.

[0130] Introducing multi-layer aggregation and cross-view feature aggregation, using max-pooling to fuse useful information from different viewpoints, improving the ability to reconstruct occluded areas, and outputting depth images for each viewpoint.

[0131] Step 3: Depth map completion and fusion

[0132] Using self-supervised learning technology, we can complete the occluded or missing areas. Based on the context scoring mechanism, we can generate a complete depth map representation by fusing depth maps from multiple perspectives.

[0133] Step 4: Depth map to voxel grid conversion

[0134] Encoding stage: Use Pix2Vox's convolutional encoder to fuse the multi-view image with the generated depth map into a context-aware high-dimensional feature representation; introduce the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric shapes of objects.

[0135] Decoding stage: The encoded features are input into the decoder of Pix2Vox to generate a preliminary voxel grid representation. Through the context-aware fusion module, high-quality reconstruction areas are selected in the preliminary voxel grids of different views to remove noise and erroneous reconstruction.

[0136] Step 5: Voxel Grid Refinement and Optimization

[0137] Using Pix2Vox’s refinement network, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion, outputting a high-resolution, complete 3D voxel grid that can accurately describe the complex piping in the aircraft auxiliary power unit.

[0138] The present invention combines the context-aware module of Pix2Vox and the cross-view aggregation mechanism of DeepMVS through context-awareness and multi-view fusion, which can select and integrate high-quality reconstruction information from different viewpoints; guides the encoding process of Pix2Vox through the depth completion result, improves the 3D reconstruction capability of occluded areas, and realizes the coordinated optimization of depth completion and voxel generation; further improves the quality of depth map completion by introducing self-supervised learning for depth map completion, and optimizes the conversion from depth to voxel by using multi-level feature fusion.

[0139] The present invention as a whole integrates multi-view depth estimation, depth map completion and voxelized reconstruction, which not only enriches the theoretical framework of 3D reconstruction, but also provides new research ideas for the application of deep learning in occlusion processing, self-supervised learning and multi-view geometric optimization.

[0140] The above is only a preferred implementation of a method for three-dimensional reconstruction of aircraft auxiliary power unit pipelines based on visual measurement. The protection scope of a method for three-dimensional reconstruction of aircraft auxiliary power unit pipelines based on visual measurement is not limited to the above embodiment. All technical solutions under this idea belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, several improvements and changes without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method of aircraft auxiliary power unit pipeline based on visual measurement, characterized by: The method comprises the following steps: Step 1: Perform data preprocessing, take multi-view RGB images and corresponding camera poses as input, and use the DeepMVS module to generate a planar scan volume as a preliminary estimate of depth information; Step 2: Generate multi-view depth maps. Use the plane scanning volume generation module of DeepMVS to build a candidate set of depth maps for each reference view. Use the convolutional network to extract multi-level features of the image and generate an accurate depth map for the reference view. Step 3: Perform depth map completion and fusion. Use self-supervised learning technology to complete the occluded or missing areas. Based on the context scoring mechanism, generate a complete depth map representation by fusing depth maps from multiple perspectives. Step 4: Transcode the multi-view RGB image and depth map into a voxel grid. Use the convolutional encoder of Pix2Vox to fuse the multi-view image with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation. Introduce the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric morphology of the object. Step 5: Refine and optimize the voxel grid. Through the refinement network of Pix2Vox, the context-fused voxel grid is locally refined to correct errors caused by multi-view or depth completion, and a high-resolution, complete three-dimensional voxel grid is output to accurately describe the pipelines in the aircraft auxiliary power unit.

2. The method according to claim 1, characterized in that: The volume in step 1 includes color projections at different depths for each view, providing a basis for subsequent depth completion, and normalizing and cropping the data to ensure that the input multi-view image is compatible with the network size of Pix2Vox.

3. The method according to claim 2, characterized in that: The step 2 introduces multi-layer aggregation and cross-view feature aggregation, fuses useful information from different viewpoints through the max-pooling method, improves the reconstruction capability of the occluded area, and outputs a depth image of each viewpoint.

4. The method according to claim 3, characterized in that: The step 4 also includes decoding, inputting the encoded features into the decoder of Pix2Vox to generate a preliminary voxel grid representation, and selecting high-quality reconstruction areas in the preliminary voxel grids of different perspectives through a context-aware fusion module to remove noise and erroneous reconstruction.

5. The method according to claim 4, characterized in that: The DeepMVS module combines multi-layer feature aggregation and cross-view feature aggregation to extract multi-layer feature maps from a single view, generate a plane scanning volume, and perform feature aggregation after mapping the plane scanning volumes of all views into a unified three-dimensional space; Within a single view, contextual information is extracted through layer-by-layer feature fusion; between multiple views, complementary information is integrated through maximum pooling to generate a global depth map candidate set, and finally, the final depth map candidate set is generated through further three-dimensional convolution and loss optimization.

6. The method according to claim 5, characterized in that: Depth map completion uses a self-supervised learning method to build a custom training target, i.e., a loss function, to learn depth map completion. The feature extraction and fusion module uses an image neural network to extract multi-pixel features of the depth map, uses a multi-feature pyramid to extract information of different resolutions, and fuses the depth map and RGB image using a weighted fusion method. The depth completion module uses a deconvolution operation to gradually restore the resolution of the depth map. In the completion process, a jump connection is used to introduce the high-resolution features of the feature extraction and fusion module, and the completed depth image is output. The projection and constraint module projects the completed depth map to other perspectives to generate synthetic views and depths, uses geometric and photometric consistency constraints to generate target views and calculate losses; The self-supervised optimization module uses the loss function to optimize the network weights and gradually improves the completion effect through final training.

7. The method according to claim 5, characterized in that: A self-supervised loss function is constructed, whose core mechanism is mainly based on photometric consistency, parallax consistency and normal consistency. The loss function extracts implicit geometric consistency or physical constraints from the input data itself as a supervision signal for training to help the model complete the depth map completion.

8. A three-dimensional reconstruction system for aircraft auxiliary power unit pipelines based on visual measurement, characterized by: The system comprises: A data preprocessing module, which takes the multi-view RGB images and the corresponding camera poses as input and uses the DeepMVS module to generate a planar scan volume as a preliminary estimate of the depth information; A feature extraction module, which generates multi-view depth maps, constructs a candidate set of depth maps for each reference view through the plane scanning volume generation module of DeepMVS, extracts multi-level features of the image through a convolutional network, and generates an accurate depth map for the reference view; A completion and fusion module, which performs depth map completion and fusion, completes occluded or missing areas through self-supervised learning technology, and generates a complete depth map representation by fusing depth maps of multiple perspectives based on a context scoring mechanism; The encoding and decoding module performs conversion encoding of multi-view RGB images and depth maps to voxel grids, uses the convolutional encoder of Pix2Vox to fuse the multi-view images with the depth map generated by the DeepMVS module into a context-aware high-dimensional feature representation; introduces the depth completion result as an additional input feature to enhance the ability to capture local details and global geometric forms of objects; The optimization module refines and optimizes the voxel grid. Through the refinement network of Pix2Vox, the context-fused voxel grid is locally refined, errors caused by multi-view or depth completion are corrected, and a high-resolution, complete three-dimensional voxel grid is output to accurately describe the pipelines in the aircraft auxiliary power unit.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to claims 1-7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method of claims 1-7 is implemented.

Images