Improved model reconstruction method for damaged blade of aero-engine based on deep learning

Abstract

The invention discloses an aero-engine damaged blade improved model reconstruction method based on deep learning, and relates to the technical field of damage analysis. The method comprises the following steps: constructing an aero-engine damaged blade data set; according to the technical key points, a UNet network is introduced to reconstruct a multi-scale feature extraction structure of a PatchmatchNet model, and the feature extraction capability of the model is improved; a CBAM attention mechanism is adopted to optimize a feature extraction module, so that the model is focused on a key area, and the feature extraction capability of the model is enhanced; meanwhile, a random initialization depth map module is replaced by a Delaunay triangulation method based on depth gradient sampling to generate an initial depth map, so that the precision of the model prediction depth map is improved; verifying on the public data set and the established aero-engine damaged blade data set; the reconstructed damaged blade point cloud has good integrity and accuracy, and can better meet the requirements of aeroengine blade damage analysis.

Description

Technical Field

[0001] This invention relates to the field of damage analysis technology, specifically to a method for reconstructing an improved model of damaged aero-engine blades based on deep learning. Background Art

[0002] With the rapid development of my country's civil aviation industry, the number of aircraft owned by civil airlines and the frequency of flights have increased dramatically. As a core component of aircraft, the aero engine operates under harsh environments such as high temperature, high pressure, and high speed for extended periods. Its key components are prone to damage such as cracks and ablation. This damage can affect engine performance and even cause serious safety accidents. Therefore, damage analysis of engine components and the development of corresponding maintenance measures are crucial for the safe operation of aircraft. High-precision damage models are the foundation of damage analysis.

[0003] Currently, damage analysis of aero-engine components mainly relies on images and videos of engine parts. This type of data can only provide defect morphology features and cannot express depth and spatial information. However, complete and accurate 3D data of aero-engine damage structures can not only characterize rich damage morphology features, but also provide high-information 3D damage structure data input for advanced damage analysis algorithms, providing important data support for damage maintenance decisions. 3D reconstruction methods for engine components can be divided into traditional reconstruction methods and deep learning-based reconstruction methods. Traditional 3D reconstruction methods for aero-engine blades are divided into active reconstruction and traditional passive reconstruction. Among active reconstruction methods, laser scanning has high accuracy, but relies on sophisticated equipment, which is complex and expensive. Structured light methods, Kinect technology, and TOF (Time of Flight) technology have low equipment requirements, but lower accuracy and are not suitable for large-scale complex scenarios. In traditional passive reconstruction methods, multi-view stereo vision reconstruction systems equipped with cameras are mainly used for reconstruction, which are difficult to handle weak texture areas and cannot well meet the current requirements of high precision, high efficiency, and high automation for aero-engine blade reconstruction.

[0004] In recent years, due to the excellent performance of deep learning in image processing, it has been widely applied in the field of 3D reconstruction. Deep learning-based multi-view Stereo Matching (MVS) 3D reconstruction methods extract features and predict depth from multiple RGB images from consecutive viewpoints, then generate a 3D point cloud model through a fusion network. This reduces the need for training data and also utilizes the disparity generated by multiple views to reduce occluded areas, resulting in higher accuracy reconstruction results. Furthermore, the multi-view... Figure 3 3D reconstruction methods can be divided into implicit reconstruction and explicit reconstruction; such as Figure 1As shown, implicit reconstruction methods express the relationships between points through functions such as the Sign Distance Function (SDF), Neural Radiance Fields (NeRF), and Occupancy, storing only network parameters or functions. They consume little memory but are slow to train and difficult to edit directly. Explicit reconstruction methods directly represent the geometric elements of the reconstruction target, such as points, lines, and surfaces, through point clouds, meshes, and voxels. They consume a lot of memory but are fast in real-time rendering and easy to edit, making them more suitable for the 3D reconstruction of damaged blades in aero-engines.

[0005] Vox-based reconstruction methods first extract 2D features from a single image, then perform 3D feature fusion through joint processing to produce coarse voxels, and finally obtain the final voxel mesh with the help of a fine-grained strategy; such as Pix2Vox, SurfaceNet+, SurfaceNet, etc. However, voxel fusion will occupy a lot of memory and require a lot of computation time, so it cannot meet the real-time requirements of aero-engine damaged blade reconstruction well; mesh-based reconstruction methods usually use an ellipsoidal network as the initial form, and gradually add triangular mesh vertices to achieve deformation and generate a triangular mesh form that matches the scene or object; such as Pixel2Mesh

[10] Pixel2Mesn++, etc. Mesh-based reconstruction supports complex topological structures, but depends on high-quality point cloud input and has a large amount of computation, and is not suitable for the reconstruction of aero-engine damaged blades;

[0006] Point cloud-based reconstruction methods are similar to voxel-based methods. First, convolutional operations are performed on the image to predict the 3D structure from multiple views. Then, a fusion network is used to obtain the final point cloud of the scene or object. Currently, the multi-view MVSNet network, a pioneering work in the multi-view series, has been proposed to achieve end-to-end 3D reconstruction of targets. However, this model processes low-resolution images, and steps such as constructing the matching cost volume require significant memory resources. Therefore, subsequent MVS improvement algorithms such as UCS-Net, CVP-MVSNet, and Fast-MVSNet focus on improving reconstruction performance while also enhancing real-time performance and lightweight design. Other models, such as AA-RMVSNet, MFNet, and SuperMVS, propose a strategy of implementing a feature pyramid, extracting high-level and low-level semantic information from coarse to fine to construct a cascaded cost volume, fusing multi-scale information.

[0007] To address the semantic differences between feature maps of different scales, where fusing direct feature maps reduces the expressive power of multi-scale feature maps, a traditional approach has proposed D2HC-RMVSNet. This approach uses a lightweight DRENet structure to extract features and fuses multi-scale contextual information, enabling the recovery of more scene details. To process high-resolution images on resource-constrained devices, the PatchmatchNet model has been proposed. It abandons the matching cost volume structure and uses an iterative method based on the traditional Patchmatch structure to generate depth maps. Compared to other MVS algorithms, this algorithm boasts faster computation speed, lower memory usage, and maintains a superior reconstruction error. The point cloud-based reconstruction results preserve original geometric details; despite surface discontinuities, point cloud data can be easily converted into mesh data. Its high computation speed and memory efficiency better meet the real-time and high-precision requirements of aero-engine damaged blade reconstruction. Furthermore, it utilizes deep learning-based multi-view... Figure 3 The rapid development of 3D reconstruction methods has provided a rich research foundation for the 3D reconstruction of damaged blades in aero-engines;

[0008] Currently, PatchmatchNet is used as the base model to accurately reconstruct dense point clouds on the blade surface. However, using this method still faces many challenges. First, large areas of weak texture often exist on the surface of aero-engine blades, which increases the failure rate of feature extraction and matching, making it impossible to guarantee the integrity of the blade reconstruction and affecting the reconstruction effect. Second, the 3D reconstruction of damaged aero-engine blades requires a certain level of reconstruction efficiency, and existing multi-view reconstruction methods with relatively fast reconstruction speeds are insufficient. Figure 3 Most 3D reconstruction models use an iterative approach based on random search to predict depth maps. Randomness has a significant impact on the reconstruction effect, leading to a decrease in the accuracy of the reconstruction model. Summary of the Invention

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A deep learning-based improved model reconstruction method for aero-engine damaged blades, the method includes the following steps: S1, Construction of aero-engine damaged blade dataset: collect continuous view images of engine damaged blades, use COLMAP to perform sparse reconstruction of the images to extract image-related data, and construct a damaged blade test set.

[0011] S2. Construction of 3D Reconstruction Model for Damaged Aircraft Engine Blades: A UNet structure integrating the CBAM attention mechanism and a Delaunay triangulation-based depth map initialization structure are introduced to improve the existing PatchmatchNet model and construct the UCD-PatchmatchNet reconstruction model for damaged aircraft engine blades.

[0012] S3. Model Training: Import the training set and validation set into the UCD-PatchmatchNet aero-engine damaged blade reconstruction model, set the training parameters, and train the model to obtain the optimal weight file.

[0013] S4. Performance Testing: Import the test set and the self-built engine damaged blade dataset into the UCD-PatchmatchNet model, use the optimal weight file for prediction, and obtain the model's performance parameters.

[0014] Furthermore, the PatchmatchNet model takes multiple images from consecutive viewpoints as input and obtains a dense point cloud of the reconstructed target after model matching. The PatchmatchNet model includes at least: a feature extraction module, a learnable PatchMatch module, an iterative coarse-to-fine framework, and a spatial refinement module.

[0015] Furthermore, the feature extraction module is a multi-scale feature extraction structure based on the FPN network, consisting of 10 basic convolutional layers and 5 convolutional blocks. It upsamples and fuses multi-scale features through bilinear interpolation and 1×1 convolution, and outputs multi-scale feature maps with resolutions of 1 / 8, 1 / 4, and 1 / 2 of the original image resolution. Features are extracted in layers at several resolutions.

[0016] Furthermore, the steps based on the learnable PatchMatch module are as follows:

[0017] Initialization and local perturbation: Generate random hypotheses. In the first iteration, 48 depth hypothesis samples are randomly initialized. In subsequent iterations, local perturbation is performed by generating a hypothesis for each pixel individually within the normalized inverse depth range.

[0018] Adaptive propagation: The adaptive propagation hypothesis in the neighborhood assumes that the spatial coherence of depth values ​​comes from pixels on the same physical surface, and that adaptive propagation tends to collect data within the same surface.

[0019] Adaptive evaluation: Calculate the matching cost of all assumptions and select the best solution.

[0020] Furthermore, the spatial refinement module: when optimizing the depth map, it upsamples the prediction results from stage 1, and then refines the output depth map by combining the depth residual network and the reference image to obtain the final depth map. In stage k, the size of the output depth map is H / 2. k ×W / 2 k Finally, a dense point cloud is generated through a fusion network; where k = 1, 2, ..., n.

[0021] Furthermore, the improvements of UCD-PatchMatchNet include at least the following: the feature extraction network of the PatchmatchNet model is improved by adopting a UNet-based network, and skip connections and transposed convolutions are used to restore multi-scale feature resolution and perform feature fusion; a lightweight CBAM attention mechanism is adopted, and channel attention and spatial attention are introduced to optimize the feature extraction module to obtain semantic information; and a Delaunay triangulation structure based on depth gradient sampling is used to generate the initial depth map.

[0022] Furthermore, the improvement process of the feature extraction module based on the UNet network is as follows: the multi-scale feature extraction structure based on FPN in PatchmatchNet captures contextual information through different receptive fields; in depth maps or point clouds, the feature extraction structure is improved by using the UNet network; the feature extraction structure based on the UNet network consists of three parts: an encoder, a decoder, and skip connections; the encoder performs downsampling through max pooling convolution, the decoder recovers feature resolution through inverted convolution upsampling, and the encoder-decoder structure is used to expand the receptive field.

[0023] Furthermore, the optimization process of the feature extraction module based on the CBAM attention mechanism is as follows: a lightweight CBAM attention mechanism is introduced, and the feature map passes through the channel attention module and the spatial attention module in sequence; the CBAM attention mechanism consists of two consecutive sub-modules: channel attention and spatial attention; the channel attention first uses max pooling and average pooling operations to summarize the spatial information of the feature map, then feeds it into a shared network composed of a multilayer perceptron (MLP), and finally uses element-wise summation to merge and output the channel feature vector; the spatial attention applies max pooling and average pooling operations on the channel axis to aggregate the channel information of the feature map, and connects them to generate a feature descriptor, and applies convolutional layers to the descriptor to generate a spatial attention map; the feature extraction structure integrating the CBAM attention mechanism consists of 8 convolutional blocks and 4 attention layers, with the CBAM attention mechanism added at each encoding stage, using max pooling 2×2 convolution downsampling; the resolution is gradually restored through decoder transpose upsampling, and it is connected to the encoder feature skip connection.

[0024] Furthermore, the improved initialization process of the Delaunay triangulation depth map based on depth gradient sampling is as follows: An initial depth map is generated using the Delaunay triangulation method based on depth gradient sampling. The depth gradient-based sampling method is an adaptive sampling strategy, which, by analyzing the gradient information in the depth map, prioritizes dense sampling in regions with drastic depth changes and sparse sampling in regions with flat depth changes. Delaunay triangulation is the standard for triangulation, i.e., the process of generating a set of triangles for a given set of planar points. We hope to obtain a set of triangles. ,satisfy:

[0025] Condition 1: The endpoints of all the triangles form a set P;

[0026] Condition 2: The sides of any two triangles do not intersect;

[0027] Condition 3: All faces in the planar graph are triangular faces, and the set of all triangles forms the convex hull of P;

[0028] It also meets the following two criteria set in the pre-built rule engine:

[0029] Criterion 1: Empty Circle Property: Delaunay triangulation is unique, meaning that no four points can be concyclic, and there are no other points inside the circumcircle of any triangle in a Delaunay triangulation; Criterion 2: Minimum Angle Maximization Property: Among all possible triangulations of point set P, the minimum interior angle of the triangle formed by the Delaunay triangulation is the largest.

[0030] Furthermore, it also includes: verifying the performance of the UCD-PatchmatchNet aero-engine damaged blade reconstruction model, selecting evaluation metrics corresponding to the 3D reconstruction, which at least include accuracy (Accc), integrity (Comp), and overall performance; and measuring the overall performance of the model based on the evaluation metrics.

[0031] Ablation experiments were designed by replacing the feature extraction structure of the original network with a UNet feature extraction module that integrates the CBAM attention mechanism; the effectiveness of Delaunay triangulation based on deep gradient sampling was verified; and the reconstructed point cloud of the UCD-PatchmatchNet model was compared with the reconstructed point cloud of the deep learning-based MVSNet model, taking the damaged blade of an aero-engine as the object, and the comparison results were obtained.

[0032] This invention provides a deep learning-based method for reconstructing an improved model of damaged aero-engine blades, which has the following advantages:

[0033] 1) The UNet network is introduced to reconstruct the multi-scale feature extraction structure of the PatchmatchNet model, improving the model's feature extraction capability. The CBAM attention mechanism is used to optimize the feature extraction module, enabling the model to focus on key regions and enhance its feature extraction ability. At the same time, the random initialization depth map module is replaced with the Delaunay triangulation method based on depth gradient sampling to generate the initial depth map, improving the accuracy of the model's predicted depth map. The model is validated on public datasets and the established aero-engine damaged blade dataset. Compared with the PatchmatchNet model, the proposed UCD-PatchmatchNet model reduces the reconstruction accuracy error, integrity error, and overall error. The reconstructed damaged blade point cloud has good integrity and accuracy, which can better meet the needs of aero-engine blade damage analysis.

[0034] 2) The UCD-PatchmatchNet 3D model improves upon the original model by addressing reconstruction integrity and accuracy. The improved model has reconstruction accuracy error, reconstruction integrity error, and overall error of 0.372mm, 0.266mm, and 0.319mm, respectively, achieving improved model reconstruction performance and being applied to the 3D reconstruction of damaged aero-engine blades. The reconstructed blade point cloud has good integrity and visualization effects, and the abnormal structure also shows obvious color and depth differences in the local point cloud, which can provide support for the preliminary judgment of the type and degree of damage of the abnormal structure, proving that this solution is applicable to the reconstruction of damaged aero-engine blades. Attached Figure Description

[0035] Figure 1 A schematic diagram of the classification process for deep learning multi-view reconstruction methods;

[0036] Figure 2 This is a schematic diagram of the PatchmatchNet model structure;

[0037] Figure 3 This is a schematic diagram comparing sampling positions in the learnable PatchMatch module;

[0038] Figure 4 This is a schematic diagram of the UCD-patchmatchNet model structure;

[0039] Figure 5 This is a schematic diagram of a feature extraction module based on the improved UNet.

[0040] Figure 6 This is a schematic diagram of the channel attention submodule;

[0041] Figure 7 This is a schematic diagram of the spatial attention submodule;

[0042] Figure 8 A schematic diagram of a feature extraction module integrating the CBAM attention mechanism;

[0043] Figure 9 This is a schematic diagram of the improved Patchmatch structure;

[0044] Figure 10 A schematic diagram showing the comparison between triangulation with and without corner points;

[0045] Figure 11 Schematic diagram for initializing the Delaunay triangulation depth map;

[0046] Figure 12 This is a schematic diagram of the UCD-PatchmatchNet damaged blade reconstruction method.

[0047] Figure 13 A schematic diagram comparing the reconstruction results of the ablation experiment in the feature extraction module;

[0048] Figure 14 This is a diagram illustrating the comparison of predicted depth maps.

[0049] Figure 15 This is a diagram showing the comparison of reconstruction effects of typical models. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0051] Please see Figures 1 to 15This embodiment provides an improved model reconstruction method for damaged aero-engine blades based on deep learning. Addressing the issues of poor integrity and low accuracy in the reconstruction of 3D point clouds of damaged aero-engine blades, this method is proposed. First, a UNet network is introduced to reconstruct the multi-scale feature extraction structure of the PatchmatchNet model, improving the model's feature extraction capability. Second, the CBAM (Convolutional Block Attention Module) attention mechanism is used to optimize the feature extraction module, enabling the model to focus on key regions and strengthening its feature extraction ability. Simultaneously, the randomly initialized depth map module is replaced with a Delaunay triangulation method based on depth gradient sampling to generate the initial depth map, improving the model's depth map prediction accuracy. Finally, the method is validated on public datasets and a pre-built dataset of damaged aero-engine blades. The results show that, compared with the PatchmatchNet model, the proposed UCD-PatchmatchNet model reduces the reconstruction accuracy error by 12.9%, the integrity error by 4.0%, and the overall error by 9.4%. The reconstructed damaged blade point cloud exhibits good integrity and accuracy, better meeting the needs of aero-engine blade damage analysis.

[0052] The specific steps of this method are as follows:

[0053] S1. Construction of the aircraft engine damaged blade dataset:

[0054] Collect continuous view images of damaged engine blades taken by a camera, and use COLMAP to perform sparse reconstruction of the images to extract image-related data to form a test set of damaged blades; wherein, the image-related data includes at least: camera intrinsic parameters, extrinsic parameters, and depth range of images under different viewpoints.

[0055] S2. Construction of a 3D reconstruction model of damaged aero-engine blades:

[0056] A UNet structure integrating the CBAM attention mechanism and a depth map initialization structure based on Delaunay triangulation are introduced to improve the existing PatchmatchNet model and construct a UCD-PatchmatchNet aero-engine damaged blade reconstruction model.

[0057] The established theory and content of the PatchmatchNet model are explained below:

[0058] The PatchmatchNet model takes multiple images from consecutive viewpoints as input and obtains a dense point cloud of the reconstructed target after model matching. The PatchmatchNet model includes at least: a feature extraction module, a learnable PatchMatch module, an iterative coarse-to-fine framework, and a spatial refinement module.

[0059] The structure of the PatchmatchNet model can be found in [reference]. Figure 2 The results are shown below;

[0060] Feature extraction module: It is a multi-scale feature extraction structure based on FPN (Feature Pyramid Network), consisting of 10 basic convolutional layers and 5 convolutional blocks. It upsamples and fuses multi-scale features through bilinear interpolation and 1×1 convolution, and outputs multi-scale feature maps with resolutions of 1 / 8, 1 / 4, and 1 / 2 of the original image resolution. Features are extracted in layers at multiple resolutions, which helps to predict depth maps from coarse to fine.

[0061] The learnable PatchMatch module comprises the following three steps:

[0062] (1) Initialization and local perturbation: Generate random hypotheses; In the first iteration, 48 depth hypothesis samples are randomly initialized, and in subsequent iterations, local perturbation is performed by generating hypotheses for each pixel individually within the normalized inverse depth range;

[0063] (2) Adaptive Propagation: Adaptive propagation assumption within the neighborhood; the spatial coherence of depth values ​​usually comes from pixels on the same physical surface. Therefore, compared with the sampling method of the classic static propagation assumption, adaptive propagation tends to collect data within the same surface, which helps to provide a more accurate depth map. The sampling location is compared to... Figure 3 As shown, adaptive propagation sampling of pixels located at the image boundary (yellow) is based on the depth assumption of the sampling neighborhood (green).

[0064] (3) Adaptive evaluation: Calculate the matching cost of all assumptions and select the best solution.

[0065] Spatial refinement module: When optimizing the depth map, it directly upsamples the prediction results from stage 1, and then refines the output depth map by combining the depth residual network and the reference image to obtain the final depth map. In stage k, the size of the output depth map is H / 2. k ×W / 2 k Finally, a dense point cloud is generated through a fusion network.

[0066] Where k = 1, 2, ..., n;

[0067] The following is a description of the aero-engine damaged blade reconstruction model based on UCD-PatchmatchNet:

[0068] Compared to the existing PatchmatchNet model, UCD-PatchMatchNet addresses technical limitations in three main ways: ① To address the high failure rate of feature extraction in weak texture regions of leaves, leading to poor reconstruction integrity, a UNet-based network is used to improve the feature extraction network of the PatchmatchNet model. Skip connections and transposed convolutions restore multi-scale feature resolution and perform feature fusion, reducing information loss due to downsampling and improving the model's feature extraction capability; ② A lightweight CBAM attention mechanism is adopted, introducing channel attention and spatial attention to optimize the feature extraction module, thereby obtaining richer semantic information, enhancing the feature representation capability of different channels, and extracting key information from different spatial locations; ③ To address the problem of low reconstruction accuracy caused by randomly generated initial depth maps, a Delaunay triangulation structure based on depth gradient sampling is used to generate initial depth maps, improving the model's reconstruction accuracy. It should be noted that the model name in this embodiment comes from the above three improvements, and the structure is as follows: Figure 4 As shown, U represents the UNet structure, C represents the CBAM attention mechanism, and D represents the Delaunay algorithm;

[0069] S2.1 Improvement of the feature extraction module based on UNet network:

[0070] PatchmatchNet's multi-scale feature extraction structure based on FPN can capture contextual information through different receptive fields. However, the feature differences in weakly textured regions are not obvious, and the local pixel values ​​change little. Convolutional kernels struggle to generate effective descriptors during feature extraction, and feature maps at different scales may maintain high similarity, failing to provide additional effective information. In depth maps or point clouds, weakly textured regions may exhibit holes and noise, leading to reduced reconstruction integrity. To improve the model's reconstruction integrity, the UNet network is used to refine the feature extraction structure. The feature extraction structure based on the UNet network is as follows: Figure 5 As shown, it mainly consists of three parts: encoder, decoder, and skip connections. The encoder performs downsampling through max pooling convolution, and the decoder recovers feature resolution through inverted convolution upsampling. The encoder-decoder structure can expand the receptive field and more effectively capture multi-scale contextual information. Skip connections preserve high-frequency details, prevent low-level features from being overwhelmed by high-level features, and achieve more thorough fusion of deep and shallow features, resulting in higher quality feature output. Compared with the feature extraction structure of PatchmatchNet, the feature extraction structure based on the UNet network can effectively extract and fuse image features, improving the integrity of model reconstruction.

[0071] S2.2 Optimization of the feature extraction module based on the CBAM attention mechanism:

[0072] Feature extraction methods based on the UNet network can improve the model's feature learning ability, but they still lead to the loss of small objects or edge details during downsampling. Furthermore, they treat all channel regions equally, lacking dynamic focus on key features. Therefore, a lightweight CBAM attention mechanism is introduced. The feature map passes sequentially through a channel attention module and a spatial attention module. This not only learns the weights of different channels and highlights important features, but also focuses on key spatial regions, further enhancing the model's feature extraction ability. The CBAM attention mechanism consists of two consecutive sub-modules: channel attention and spatial attention. Channel attention first uses max pooling and average pooling operations to summarize the spatial information of the feature map, then feeds it into a shared network composed of multilayer perceptrons (MLPs), and finally uses element-wise summation to merge the output channel feature vectors. The calculation basis for channel attention is as follows:

[0073] (1); In equation (1), Let F represent the sigmoid function, F represent the input features, c represent channel attention, and M represent the input features. c (F) represents the features passed through channel attention, and AvgPool(F) and MaxPool(F) represent the features passed through average pooling and max pooling, respectively; W0 and W1 are the two weights of the MLP, W0∈R C / r×C W1∈R C×C / r Where R represents the set of real numbers, C represents the number of channels in the input feature map, r represents the dimensionality reduction ratio, and the ReLU (Rectified Linear Unit) activation function is followed by W0;

[0074] Spatial attention applies max pooling and average pooling operations along the channel axis to aggregate channel information of feature maps, concatenates them to generate a feature descriptor, and applies convolutional layers to the descriptor to generate a spatial attention map.

[0075] The calculation of spatial attention is based on the following:

[0076] (2); In equation (2), The sigmoid function is represented by F; the input features are represented by s; spatial attention is represented by M. s (F) represents the feature obtained through spatial attention, and AvgPool(F) and MaxPool(F) represent the feature obtained through average pooling and max pooling, respectively. This indicates a convolution operation with a filter size of 7×7;

[0077] Feature extraction structure integrating CBAM attention mechanism, such as Figure 8 As shown, it mainly consists of 8 convolutional blocks and 4 attention layers. The CBAM attention mechanism is added in each encoding stage. Max pooling 2×2 convolution downsampling is used to reduce the size and number of channels while performing fine calculations on important channels and spatial regions, strengthening geometrically related channels, focusing on object edges and non-planar regions, and reducing information loss caused by downsampling. Resolution is gradually restored by transposing and upsampling the decoder and connecting it with the encoder feature skip connection, which ensures the improvement of model reconstruction performance without significantly increasing the computational cost.

[0078] S2.3 Improved Delaunay triangulation depth map initialization based on depth gradient sampling:

[0079] In PatchmatchNet, the learnable Patchmatch module generates random hypothetical depth samples in the first iteration that may differ significantly from the ground truth depth, requiring more iterations to converge, increasing computational cost and impacting reconstruction accuracy. To improve accuracy, a Delaunay triangulation method based on depth gradient sampling is used to generate an initial depth map that more closely approximates the ground truth depth. This gradient-based sampling method is an adaptive sampling strategy that analyzes gradient information in the depth map, prioritizing dense sampling in regions with dramatic depth changes (such as object edges) and sparse sampling in regions with flat depth changes, minimizing computation while maintaining reconstruction accuracy. Delaunay triangulation is a standard triangulation method that generates a set of triangles from a given set of planar points. We hope to obtain a set of triangles. ,satisfy:

[0080] (1) The endpoints of all the triangles form a set P;

[0081] (2) The sides of any two triangles do not intersect (either coincide or have no intersection).

[0082] (3) All faces in the planar diagram are triangular faces, and the set of all triangles forms the convex hull of P;

[0083] It must also comply with the following two criteria set in the pre-built rules engine:

[0084] (1) Empty circle property: The Delaunay triangulation is unique (no four points can be concyclic), and there are no other points inside the circumcircle of any triangle in the Delaunay triangulation; (2) Minimum angle maximization property: Among all possible triangulations of the point set P, the minimum interior angle of the triangle formed by the Delaunay triangulation is the largest.

[0085] (2) The improved Patchmatch structure based on the Delaunay triangulation depth map initialization is as follows: Figure 9 As shown; since the DTU dataset provides ground truth (GT) images, after determining the number of sampling points, a depth gradient-based sampling method is used to sample 70% of the depth gradient variation regions in the GT image and randomly sample 30% to simulate potential noise. Four corner points are added to the extracted sparse points to ensure the sparse sampled image is a complete image. Then, Delaunay triangulation is performed, as shown... Figure 10 As shown; after forming the Delaunay triangular mesh, the depth values ​​of all points covered by the triangles are interpolated based on the depth values ​​of the three vertices of the triangles to obtain the initial depth map used, as shown. Figure 11 As shown; from Figure 11 As can be seen, the initial depth map formed by Delaunay triangulation has certain similarities to the ground truth map, which can improve the accuracy of the final predicted depth map to a certain extent and improve the accuracy of model reconstruction.

[0086] (3) S3, Model Training:

[0087] (4) Import the training set and validation set into the UCD-PatchmatchNet aero-engine damaged blade reconstruction model, and set the training parameters for training. The training set is used for model training, and the validation set is used to evaluate the training results of each round. Finally, the optimal weight file of the model is obtained.

[0088] (5) S3.1 Current status of multi-view based on deep learning in different fields Figure 3 The validation of the dimensional reconstruction method uses publicly available datasets for model training and self-built datasets for validation testing. In this embodiment, the publicly available dataset DTU is selected for model training, and a self-built aero-engine damaged blade dataset is used for model validation testing, as shown in Table 1.

[0089] (6) Table 1: Dataset used for model performance validation:

[0090]

[0091] Experimental environment: Intel(R) Xeon(R) Gold 6326 CPU @2.90GHz, 192GB RAM, 2 NVIDIA GeForce RTX 4090 graphics cards, 120GB VRAM;

[0092] S3.2 Evaluation Indicators:

[0093] To verify the model's performance and compare it with other deep learning reconstruction methods, commonly used evaluation metrics for 3D reconstruction were selected: Acc (Accuracy), Comp (Completeness), and Overall error. Accuracy measures the distance between the reconstructed point cloud and the ground truth point cloud; completeness measures the distance between the ground truth point cloud and the reconstructed point cloud; and overall error is the average of accuracy and completeness, measuring the model's overall performance. The calculation formula is as follows:

[0094] (3)

[0095] (4)

[0096] (5)

[0097] In equations (3), (4) and (5), P represents the reconstructed point cloud, GT represents the ground truth point cloud (reference point cloud), p∈P means p is a point in the reconstructed point cloud, and q∈GT means q is a point in the ground truth point cloud. denoted by Euclidean distance, pq represents the distance from the reconstructed point cloud to the ground truth point cloud, reflecting the accuracy of the reconstructed point cloud, and qp represents the distance from the ground truth point cloud to the reconstructed point cloud, reflecting the completeness of the reconstructed point cloud.

[0098] S3.3 Ablation test:

[0099] S3.3.1 Validation of the effectiveness of the UNet feature extraction module integrating the CBAM attention mechanism: To verify the effectiveness of the improved feature extraction module, an ablation experiment was designed by replacing the feature extraction structure of the original network with the UNet feature extraction module integrating the CBAM attention mechanism. The reconstruction results are as follows: Figure 13 As shown; from Figure 13 As can be seen, since the number of reconstructed point clouds is in the tens of millions, the overall reconstruction results have good integrity. However, the improved model can significantly improve the reconstruction holes in the leaf tip and leaf back regions of the original model in terms of local details. The feature extraction module proposed in this embodiment can better obtain the features of the reconstruction target, which can improve the reconstruction integrity of the model.

[0100] Table 2: Comparison of quantitative results of ablation experiments from the feature extraction module:

[0101]

[0102] As shown in Table 2 above, replacing the original multi-scale feature extraction module (denoted as the baseline module) with a feature extraction module based on the UNet network that integrates the CABM attention mechanism reduces the accuracy error, integrity error, and overall error by 1.2%, 4.7%, and 2.6%, respectively.

[0103] S3.3.2 Validation of Delaunay triangulation based on depth gradient sampling:

[0104] To verify the effectiveness of Delaunay triangulation based on depth gradient sampling in generating initial depth maps, an ablation experiment was designed using the initial depth maps generated by Delaunay triangulation and the initial depth maps randomly generated by the original model. The model performance is shown in Table 3, and the reconstruction results are as follows: Figure 13 As shown;

[0105] Table 3: Comparison of quantitative results of ablation experiments using Delaunay triangulation:

[0106]

[0107] As shown in Table 3, using Delaunay triangulation based on depth gradient sampling reduced the accuracy error by 6.8%, proving that the depth map generated during the depth map initialization stage is closer to the true value. The integrity error increased by 4.7%, possibly because the initial depth map generated based on depth gradient sampling focuses more on areas where the depth changes and cannot initialize the entire image. In contrast, the method of randomly generating the initial depth map randomly generates samples throughout the entire image, and the sample points are more likely to cover the entire image, but the overall integrity error is reduced by 2.3%.

[0108] To visually demonstrate the effectiveness of the improvements, depth maps of the reconstruction process are compared, such as... Figure 14 As shown, the improved depth map after Dealunay triangulation transitions naturally at the trailing edge of the blade and the trailing edge air slot, with clear boundaries and more accurate predictions; the method proposed in this embodiment is effective in improving the model depth map initialization stage.

[0109] S3.3 Comparative Analysis of Different Models:

[0110] S3.3.1 Comparative Analysis of Reconstruction Effects of Typical Models

[0111] To verify the effectiveness of the improved model, a damaged aero-engine blade was used as the object. The reconstructed point cloud of the UCD-PatchmatchNet model was compared with the reconstructed point cloud of the deep learning-based MVSNet model. The point clouds were compared... Figure 15As shown, overall, the UCD-PatchmatchNet, MVSNet, and PatchmatchNet models all achieved good integrity. However, because MVSNet reconstructs at low resolution, the details of the air film pores are more blurred, making them almost invisible, resulting in the sparsest reconstruction. Compared to the PatchmatchNet model, UCD-PatchmatchNet is more complete and clearer in its detail representation, with better recognition of the air film pores. The UCD-PatchmatchNet model retains the integrity of key information while reconstructing details more precisely, with clearer boundaries at the edges of the air film pores, resulting in the best reconstruction effect.

[0112] S3.3.2 Performance Comparison Analysis of Typical Models:

[0113] To verify the superiority of the model, a comparative experiment was designed using a variety of typical deep learning 3D reconstruction methods. The performance comparison results of the models are shown in Table 4.

[0114] Table 4: Performance Comparison Results of Typical Models

[0115]

[0116] Note: The values ​​in parentheses are the optimal values ​​in their respective columns;

[0117] As shown in Table 4, compared with some deep learning-based reconstruction methods, the reconstruction accuracy error of the method proposed in this embodiment is not optimal. However, compared with the PatchmatchNet model, the accuracy error is reduced by 12.9% through the improvement of the Delaunay triangulation structure based on depth gradient sampling, which is competitive among MVS reconstruction methods. In addition, the integrity error is reduced by 4.0% and the overall error is reduced by 9.4%, which are the best among many reconstruction methods. At the same time, compared with MVSNet, the accuracy error, integrity error and overall error are reduced by 6.1%, 49.5% and 31.0% respectively, which proves the superiority of the method adopted in this embodiment.

[0118] S4. Performance Testing:

[0119] The test set and the self-built engine damaged blade dataset were imported into the UCD-PatchmatchNet model, and the optimal weight file was used for prediction to obtain the model's performance parameters.

[0120] In summary, to achieve multi-view based on deep learning Figure 3 For accurate reconstruction of damaged blades in aero-engines, an improved UCD-PatchmatchNet reconstruction model is proposed, from which the following conclusions can be drawn:

[0121] (1) The UCD-PatchmatchNet 3D model improves the original model by focusing on reconstruction integrity and accuracy. The reconstruction accuracy error, reconstruction integrity error and overall error of the improved model are 0.372mm, 0.266mm and 0.319mm respectively, which realizes the improvement of model reconstruction performance and is applied to the 3D reconstruction of damaged aero-engine blades; (2) The reconstructed blade point cloud has good integrity and visualization effect. The abnormal structure also shows obvious color and depth differences in the local point cloud, which can provide support for the preliminary judgment of the type and degree of damage of the abnormal structure, proving that the method in this paper can be applied to the reconstruction of damaged aero-engine blades.

[0122] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0124] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for reconstructing an improved model of damaged aero-engine blades based on deep learning, characterized in that: The steps of this method are as follows: S1. Construction of the damaged blade dataset for aero-engines: Collect continuous view images of damaged blades of engines, use COLMAP to perform sparse reconstruction of the images to extract image-related data, and construct a test set of damaged blades. S2. Construction of 3D Reconstruction Model for Damaged Aircraft Engine Blades: A UNet structure integrating the CBAM attention mechanism and a Delaunay triangulation-based depth map initialization structure are introduced to improve the existing PatchmatchNet model and construct the UCD-PatchmatchNet reconstruction model for damaged aircraft engine blades. S3. Model Training: Import the training set and validation set into the UCD-PatchmatchNet aero-engine damaged blade reconstruction model, set the training parameters, and train the model to obtain the optimal weight file. S4. Performance Testing: Import the test set and the self-built engine damaged blade dataset into the UCD-PatchmatchNet model, use the optimal weight file for prediction, and obtain the model's performance parameters.

2. The method for reconstructing an improved model of damaged aero-engine blades based on deep learning according to claim 1, characterized in that: The PatchmatchNet model takes multiple images from consecutive viewpoints as input and obtains a dense point cloud of the reconstructed target after model matching. The PatchmatchNet model includes at least: a feature extraction module, a learnable PatchMatch module, an iterative coarse-to-fine framework, and a spatial refinement module.

3. The method for reconstructing an improved model of damaged aero-engine blades based on deep learning according to claim 2, characterized in that: The feature extraction module is a multi-scale feature extraction structure based on the FPN network, consisting of 10 basic convolutional layers and 5 convolutional blocks. It upsamples and fuses multi-scale features through bilinear interpolation and 1×1 convolution, and outputs multi-scale feature maps with resolutions of 1 / 8, 1 / 4, and 1 / 2 of the original image resolution. Features are extracted in layers at several resolutions.

4. The method for reconstructing an improved model of damaged aero-engine blades based on deep learning according to claim 2, characterized in that: The steps based on the learnable PatchMatch module are as follows: Initialization and local perturbation: Generate random hypotheses. In the first iteration, 48 depth hypothesis samples are randomly initialized. In subsequent iterations, local perturbation is performed by generating a hypothesis for each pixel individually within the normalized inverse depth range. Adaptive propagation: The adaptive propagation hypothesis in the neighborhood assumes that the spatial coherence of depth values ​​comes from pixels on the same physical surface, and that adaptive propagation tends to collect data within the same surface. Adaptive evaluation: Calculate the matching cost of all hypotheses and select the best solution.

5. The method for reconstructing an improved model of damaged aero-engine blades based on deep learning according to claim 2, characterized in that: Spatial refinement module: When optimizing the depth map, it upsamples the prediction results from stage 1, and then refines the output depth map by combining the depth residual network and the reference image to obtain the final depth map. In stage k, the size of the output depth map is H / 2. k ×W / 2 k Finally, a dense point cloud is generated through a fusion network; where k = 1, 2, ..., n.

6. The method for reconstructing an improved model of damaged aero-engine blades based on deep learning according to claim 1, characterized in that: The improvements of UCD-PatchMatchNet include at least the following: the feature extraction network of the PatchmatchNet model is improved by using a UNet-based network, skip connections and transposed convolutions are used to restore multi-scale feature resolution and perform feature fusion; a lightweight CBAM attention mechanism is adopted, and channel attention and spatial attention are introduced to optimize the feature extraction module to obtain semantic information; and a Delaunay triangulation structure based on depth gradient sampling is used to generate the initial depth map.

7. The method for reconstructing an improved model of damaged aero-engine blades based on deep learning according to claim 6, characterized in that: The improvement process of the feature extraction module based on the UNet network is as follows: The multi-scale feature extraction structure based on FPN in PatchmatchNet captures contextual information through different receptive fields; in depth maps or point clouds, the feature extraction structure is improved by using the UNet network; the feature extraction structure based on the UNet network consists of three parts: encoder, decoder and skip connections; the encoder performs downsampling through max pooling convolution operation, the decoder recovers feature resolution through inverted convolution upsampling, and the encoder-decoder structure is used to expand the receptive field.

8. The method for reconstructing an improved model of damaged aero-engine blades based on deep learning according to claim 7, characterized in that: The optimization process of the feature extraction module based on the CBAM attention mechanism is as follows: a lightweight CBAM attention mechanism is introduced, and the feature map passes through the channel attention module and the spatial attention module in sequence. The CBAM attention mechanism consists of two consecutive sub-modules: channel attention and spatial attention; Channel attention first uses max pooling and average pooling operations to summarize the spatial information of the feature map, then feeds it into a shared network composed of a multilayer perceptron (MLP), and finally uses element-wise summation to merge the output channel feature vectors. Spatial attention applies max pooling and average pooling operations on the channel axis to aggregate the channel information of the feature map and concatenates them to generate a feature descriptor. Convolutional layers are applied to the descriptor to generate a spatial attention map. The feature extraction structure integrating the CBAM attention mechanism consists of 8 convolutional blocks and 4 attention layers. The CBAM attention mechanism is added at each encoding stage, and max pooling 2×2 convolution downsampling is used. Resolution is gradually restored by transposing and upsampling the decoder, and features are skipped and connected to the encoder features.

9. The method for reconstructing an improved model of damaged aero-engine blades based on deep learning according to claim 7, characterized in that: The improved initialization process of the Delaunay triangulation depth map based on depth gradient sampling is as follows: An initial depth map is generated using the Delaunay triangulation method based on depth gradient sampling. This depth gradient sampling method is an adaptive sampling strategy, which, by analyzing the gradient information in the depth map, prioritizes dense sampling in regions with drastic depth changes and sparse sampling in regions with flat depth changes. The Delaunay triangulation standard is the process of generating a set of triangles for a given set of planar points. We hope to obtain a set of triangles. ,satisfy: Condition 1: The endpoints of all the triangles form a set P; Condition 2: The sides of any two triangles do not intersect; Condition 3: All faces in the planar graph are triangular faces, and the set of all triangles forms the convex hull of P; It also meets the following two criteria set in the pre-built rule engine: Criterion 1: Empty Circle Property: Delaunay triangulation is unique, meaning that no four points can be concyclic, and there are no other points inside the circumcircle of any triangle in a Delaunay triangulation; Criterion 2: Minimum Angle Maximization Property: Among all possible triangulations of point set P, the minimum interior angle of the triangle formed by the Delaunay triangulation is the largest.

10. The method for reconstructing an improved model of damaged aero-engine blades based on deep learning according to claim 1, characterized in that: Also includes: To verify the performance of the UCD-PatchmatchNet aero-engine damaged blade reconstruction model, evaluation metrics corresponding to 3D reconstruction were selected, including at least Accuracy (Acc), Completeness (Comp), and Overall Performance (Overall). The overall performance of the model was measured based on the evaluation metrics. Ablation experiments were designed by replacing the feature extraction structure of the original network with a UNet feature extraction module that integrates the CBAM attention mechanism; the effectiveness of Delaunay triangulation based on deep gradient sampling was verified; and the reconstructed point cloud of the UCD-PatchmatchNet model was compared with the reconstructed point cloud of the deep learning-based MVSNet model, taking the damaged blade of an aero-engine as the object, and the comparison results were obtained.

Images