Complete three-dimensional reconstruction method for damaged blade of aero-engine based on image generation type diffusion model

Abstract

The invention relates to the technical field of aero-engine blade three-dimensional reconstruction, and particularly discloses an aero-engine damaged blade complete three-dimensional reconstruction method based on an image generation type diffusion model, and the method comprises the steps: data collection and data set construction: employing an RGB camera to collect an aero-engine blade damage video frame image; the invention provides a brand-new complete three-dimensional model reconstruction framework of the damaged blade of the aero-engine based on instance segmentation, an image generation type diffusion model and an explicit modeling algorithm, and solves the problems of low modeling efficiency, poor modeling details and low accuracy of the damaged blade. A new thought is provided for three-dimensional modeling in the blade repairing and remanufacturing process; meanwhile, an image generation type diffusion model used for aero-engine damaged blade image restoration is designed, a Stable Diffusion-1. 5 model is adopted as a framework to improve the image restoration efficiency, a dual U-Net structure is introduced to enhance the reference feature extraction capability of the model, and a self-supervised training strategy is adopted to make full use of data to improve the performance of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction of aeroengine blades, and particularly relates to a method for complete three-dimensional reconstruction of damaged aeroengine blades based on an image generation diffusion model. Background Art

[0002] Aeroengine blades, such as turbine and compressor blades, play a core role in the modern aerospace industry. In an environment with very high temperature and pressure and under the influence of foreign objects, aeroengine blades may be affected by various damages, such as wear, corrosion, and fatigue. Therefore, timely and effective detection, repair, and remanufacturing of damaged aeroengine blades are of great significance for ensuring flight safety, extending the service life of the engine, and reducing maintenance costs. In the process of blade repair and remanufacturing, constructing a complete and accurate three-dimensional model of the blade is a core step, which provides stable support for repair and remanufacturing. However, most blades either lack a nominal model or have changed in shape and size during operation, differing from the nominal model, and the nominal model is no longer suitable for the target model after blade repair. Therefore, it is urgent to achieve the complete three-dimensional reconstruction of damaged aeroengine blades.

[0003] The three-dimensional modeling methods of aero-engine blades are generally divided into three categories: software modeling method, instrument modeling method, and image modeling method. The software modeling method requires prior knowledge of the nominal model of the damaged blade corresponding to the model. However, the engine blades work in high-temperature and high-pressure environments for a long time, and the shape and size of the blades have changed. The blade nominal model is no longer suitable as the target model after blade repair. The instrument modeling method is divided into contact type and non-contact type. The contact type method usually uses a contact probe to measure and model in a point-by-point entry and exit manner. The measurement speed is slow, the operation efficiency is low, and the measurement cost is high. It cannot provide the front-line repair personnel with an instant three-dimensional model of the damaged blade. The non-contact type method usually uses a three-dimensional scanner for modeling. Affected by factors such as light and angle, and there are few features on the aero-engine blades, it is very easy to have the problem of losing the scanning target tracking, which will lead to the failure of three-dimensional reconstruction. The image modeling method is divided into explicit modeling and implicit modeling. Explicit modeling refers to the process of directly constructing the three-dimensional geometric representation of a scene or an object. It usually relies on the principles of multi-view geometry and image matching algorithms to generate a three-dimensional model by calculating the depth and normal information of the object surface in the scene. However, explicit modeling usually includes multiple complex steps such as structure from motion (SfM), multi-view stereo vision (MVS), surface reconstruction, and texture reconstruction, and there will be holes due to the lack of perspective images, affecting the effect of the three-dimensional model. Implicit modeling represents the scene by learning a continuous mathematical function. This scene defines whether each point in space belongs to the interior of the object. There are two examples of implicit reconstruction: signed distance function (SDF) and neural radiance field (NeRF). However, implicit modeling omits the explicit three-dimensional surface generation process and directly obtains pictures from new perspectives from the input object photos. This makes it difficult to directly edit the three-dimensional model of the implicit representation and unable to achieve industrial implementation. Moreover, the implicit modeling method is difficult to handle objects with complex structures and geometries and cannot reproduce the high-frequency features of the object, being relatively rough in terms of details. Therefore, no matter which modeling method it is, it either severely depends on the modeling instrument restricted by the operating environment, or cannot achieve industrial implementation, or only reconstructs an incomplete three-dimensional model and cannot build an accurate and complete three-dimensional model of the blade. Currently, there is no method that can directly build a complete three-dimensional model of the blade.

[0004] The image generation diffusion model has the powerful ability to generate real samples. The image restoration method based on the image generation diffusion model can achieve intelligent restoration of damages to construct a complete three-dimensional model of the blade. Most current image restoration methods use text prompts to guide the generation of the content in the editing area and serve generative artificial intelligence (AIGC), and cannot be directly used for the image restoration of damaged blades. Compared with AIGC, the image restoration serving damaged blades needs to overcome the following challenges: the small sample problem of lacking training data, the unconditional image generation problem, and the problem of achieving smooth continuity of the restored area. In addition, it is necessary to improve the restoration speed and quality to meet the requirements of being fast and accurate.

[0005] The present invention proposes a method for complete three-dimensional reconstruction of damaged blades of aero-engines based on the image generation diffusion model. Based on instance segmentation, the image generation diffusion model and the explicit modeling algorithm, a framework for reconstructing the complete three-dimensional model of damaged blades of aero-engines is designed. It solves the problems of low modeling efficiency, poor modeling details, and low accuracy of damaged blades, and provides a new idea for three-dimensional modeling in the process of blade repair and remanufacturing. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for complete three-dimensional reconstruction of damaged blades of aero-engines based on the image generation diffusion model to solve the problems raised in the above-mentioned background technology.

[0007] To achieve the above purpose, the present invention provides the following technical solutions:

[0008] The method for complete three-dimensional reconstruction of damaged blades of aero-engines based on the image generation diffusion model includes:

[0009] S1: Data acquisition and dataset construction. Use an RGB camera to collect video frame images of damaged blades of aero-engines, and use random scaling, random cropping, random rotation, random brightness change, and random projection transformation to simulate stronger deformations.

[0010] S2: Obtain the blade damage mask image. Input the blade video frame images collected by the RGB camera into the damage detector to detect damages, and perform dilation and blurring processing on the detection results.

[0011] S3: Dual U-Net stable diffusion model. By designing a dual U-Net structure, enhance the reference feature extraction ability of the model and ensure the smooth continuity of the restored image.

[0012] S4: Establish a complete three-dimensional model of the blade. Use the explicit modeling algorithm MVSnet based on learning to reconstruct the complete three-dimensional model of the blade.

[0013] S5: Conduct experiments and performance evaluations on the method for complete three-dimensional reconstruction of damaged blades of aero-engines.

[0014] Preferably, the training data of the diffusion model is designed through data selection, data augmentation, and masking strategies, and the experimental implementation and performance evaluation are evaluated through conducting optimization experiments, image restoration experiments, ablation experiments, and application experiments.

[0015] Preferably, in the S1, notches, tears, deformations, and chipped corners are used as the research objects, and a dataset is constructed to conduct experiments.

[0016] Preferably, in the S2, methods of pre-training, blurring processing, and dilation processing are adopted to create accurate and complete repairs for the damaged areas.

[0017] Preferably, in the S3, the StableDiffusion-1.5 architecture based on the latent space diffusion principle and the dual U-Net structure are adopted, and a self-supervised training strategy is designed from data selection, data augmentation, and masking strategies to solve the problem of lack of training data.

[0018] Preferably, in the S3, a dual U-Net structure composed of a single U-Net model and an additional U-Net is used to capture features at different scales and improve the segmentation accuracy by fusing information at different scales.

[0019] Preferably, in the S3, the dual U-Net structure is composed of a source U-Net, a mask U-Net, and a depth model.

[0020] Preferably, in the S4, the 3D image modeling is explicit modeling.

[0021] Preferably, in the S5, by conducting optimization experiments and adopting the effects of different acquisition devices, lighting conditions, shooting angles, and the number of images on the image restoration effect, the best solution is determined.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] The present invention proposes a brand-new framework for reconstructing the complete 3D model of damaged blades of aero-engines based on instance segmentation, image generation diffusion models, and explicit modeling algorithms, which solves the problems of low modeling efficiency, poor modeling details, and low accuracy of damaged blade modeling, and provides new ideas for 3D modeling in the process of blade repair and remanufacturing.

[0024] The present invention designs an image generation diffusion model for image restoration of damaged blades of aero-engines, adopts the StableDiffusion-1.5 model as the architecture to improve the image restoration efficiency, introduces a dual U-Net structure to enhance the reference feature extraction ability of the model, and adopts a self-supervised training strategy to make full use of data to improve the performance of the model.

[0025] The present invention constructs a dataset of damaged blades of aero-engines, and conducts experimental studies on the repair accuracy and modeling feasibility of the proposed method based on this dataset, conducts optimization experiments to study the influence of different factors on the image repair effect, conducts ablation experiments to study the role of the dual U-Net structure and the self-supervised training strategy on the dual U-Net stable diffusion repair model, and conducts application experiments to study the generalization ability of the method proposed by the present invention. Brief Description of the Drawings

[0026] Figure 1 For data acquisition and dataset construction in the embodiments of the present invention;

[0027] Figure 2 For the global network structure and process design in the embodiments of the present invention;

[0028] Figure 3 For the self-supervised training strategy in the embodiments of the present invention;

[0029] Figure 4 For the SSIM algorithm process in the embodiments of the present invention;

[0030] Figure 5 For the process of establishing a complete three-dimensional model of the blade in the embodiments of the present invention;

[0031] Figure 6 For the comparison of the optimization experiment effects of each group in the embodiments of the present invention;

[0032] Figure 7 For the comparison of the image repair experiment effects of each group and the three-dimensional reconstruction effect of the blade in the embodiments of the present invention;

[0033] Figure 8 For the damage condition of the aviation component before the application experiment in the embodiments of the present invention;

[0034] Figure 9 For the repair effect of the aviation component after the application experiment in the embodiments of the present invention;

[0035] Figure 10 For the flow chart of the embodiments of the present invention;

[0036] Figure 11 For the quantitative results of the image repair experiment of the present invention. Detailed Embodiments

[0037] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0038] The embodiment of the present invention proposes a new method for reconstructing a complete three-dimensional model of damaged blades of an aeroengine. Inspired by the powerful ability of the image generation diffusion model to repair damaged images and generate real samples, the present invention designs a new image repair model based on the image generation diffusion model, and introduces a pre-trained instance segmentation model and a pre-trained learning-based explicit modeling algorithm. A dual U-Net structure based on the dual U-Net model and a model training mode based on the self-supervised training strategy are designed, jointly constituting a dual U-Net stable diffusion repair model, enhancing the reference feature extraction ability of the model, ensuring the smooth continuity of the repaired images, solving the problem of lack of training data, making full use of data to improve the performance of the model, and conducting optimization experiments, image repair experiments and ablation experiments to verify that the method proposed by the present invention achieves the best results in the image repair algorithm and has good three-dimensional modeling effects. Application experiments are carried out to test that the method proposed by the present invention has strong generalization ability in both image repair and three-dimensional modeling.

[0039] Embodiment 1

[0040] A method for complete three-dimensional reconstruction of damaged blades of an aeroengine based on an image generation diffusion model includes the following steps:

[0041] S1. Data collection and dataset construction. Use an RGB camera to collect damaged video frame images of aeroengine blades, and use random scaling, random cropping, random rotation, random brightness change, and random projection transformation to simulate stronger deformations.

[0042] S2. Obtain the damaged blade mask image. Input the blade video frame images collected by the RGB camera into a damage detector to detect damage, and perform dilation and blurring processing on the detection results.

[0043] S3. Dual U-Net stable diffusion model. By designing a dual U-Net structure, enhance the reference feature extraction ability of the model and ensure the smooth continuity of the repaired images.

[0044] S4. Establish a complete blade three-dimensional model. Use the learning-based explicit modeling algorithm MVSnet to reconstruct the complete three-dimensional model of the blade.

[0045] S5. Conduct experiments and performance evaluations on the method for complete three-dimensional reconstruction of damaged blades of an aeroengine;

[0046] A method for complete three-dimensional reconstruction of damaged blades of aero-engines based on an image generation diffusion model, as Figure 10 shown, includes the following:

[0047] Further elaborating specifically on the present invention, when an aero-engine is impacted by foreign objects, foreign objects of different shapes, hardnesses, and sizes will cause different types of damage to the engine blades. The number of damage types is as many as more than 20. However, the number of occurrences of most types of damage is extremely small. The present invention selects the four most typical types of damage to the engine blades, namely notch, tear, deformation, and corner breakage, as the research objects to carry out intelligent damage repair research.

[0048] Further elaborating specifically on the present invention, the present invention uses an RGB camera to collect video frame images of damaged aero-engine blades (it should be noted that: considering the subsequent three-dimensional reconstruction of the model, the blades located on the engine are disassembled into individual blades), and constructs a dataset for repairing damaged aero-engine blades, as Figure 1 shown. The dataset consists of 3000 images, and the numbers of corner breakage, tear, deformation, and notch are 1000, 800, 600, and 600 respectively;

[0049] To increase the difference between the source image and the masked image, the present invention uses powerful data augmentation. In addition to random scaling, random cropping, random rotation, and random brightness change, random projective transformation is also used to simulate stronger deformation. These data augmentation methods can reduce the sensitivity of the model to interference information such as the position, size, angle, and environmental light of the blade.

[0050] Further elaborating specifically on the present invention, the damage detector of the present invention is a pre-trained instance segmentation model, using the pre-trained YOLOv8 network, as Figure 2 (a). The detection result is a damage mask image. Specifically, the damaged area of the blade is white and other areas are black. By adjusting the dilation method, the generated mask image is made larger than the actual damaged area without exceeding the boundary;

[0051] By adopting the blur method, the generated part and the main part can be better fitted, and the damaged edge can be better transitioned, so as to obtain a high-quality repair mask. These masks maintain stable geometric consistency between different perspectives, thereby guiding the image generation diffusion model to create accurate and complete repairs for the damaged area.

[0052] The present invention is further described in detail. Since the number of damages in the image is small and the range is small, and there are some redundant parts, the StableDiffusion-1.5 architecture based on the principle of latent space diffusion is adopted in the present invention. It can not only overcome the problem of low efficiency caused by adding and removing noise in traditional diffusion models, but also realize the matching of images and texts through the CLIPTextModel text encoder, solve the problem of unconditional image generation, design a dual U-Net structure, enhance the reference feature extraction ability of the model, ensure the smooth continuity of the repaired image, and design a self-supervised training strategy from three aspects of data selection, data augmentation and masking strategy to solve the problem of lack of training data, and make full use of data to improve the performance of the model.

[0053] The present invention is further described in detail. To improve the effectiveness of extracting fine-grained features from the aero-engine blade damage source image, on the basis of a single U-Net model, the present invention utilizes an additional U-Net to jointly form a dual U-Net structure. The core idea of the dual U-Net structure is to use two U-Net sub-networks to capture features at different scales and improve the segmentation accuracy by fusing information at different scales. Compared with the traditional single U-Net model, the dual U-Net structure introduces an additional branch structure, enabling the network to perform information fusion from multiple levels simultaneously, thereby obtaining a more accurate segmentation of the salient object. Each sub-network can be regarded as a U-Net, which extracts and restores multi-layer features of the image through two stages of downsampling and upsampling, and then these features are merged through a fusion layer to extract more spatial information, as Figure 2 (b) shows.

[0054] The present invention is further described in detail. The dual U-Net structure designed by the present invention mainly consists of a source U-Net, a mask U-Net, and a depth model. When initializing, the mask U-Net model adopts the StableDiffusion-1.5 model, and at the same time, the input is set as a 9-channel tensor. These 9 channels cover the image latent code, the binary mask, as well as the generation area of the image and the background latent code. Specifically, the image latent code represents the low-dimensional compressed form of the input picture in the latent space, and it occupies 4 channels. This kind of image latent code enables the model to process the damaged blade image data in a lower-dimensional space, effectively reducing the computational complexity and improving the processing speed. At the same time, this compressed representation helps the model capture the key features and information in the image, laying a foundation for subsequent image generation tasks. The binary mask occupies 1 channel and is a binary matrix matching the size of the input picture, used to identify the valid area (marked as 1) and the invalid or areas to be ignored (marked as 0). The generation area of the image and the background latent code each occupy 1 channel. The generation area refers to the part of the input picture where the model is expected to generate new content, and the background latent code is the representation of the background part in the latent space. In the StableDiffusion model, by distinguishing the generation area of the image and the background latent code, the model can process the image content more flexibly. At the same time, the introduction of the background latent code also helps the model better understand and process the context information in the image, thereby improving the quality and consistency of the generated image.

[0055] As can be seen from the above, the traditional U-Net takes the CLIP text embedding as the input through the cross-attention mechanism. In the dual U-Net StableDiffusion repair model designed by the present invention, the present invention replaces this mechanism with the CLIP text embedding extracted from the mask image of the damaged area as the input. In addition, the present invention adds a trainable projection layer after the image embedding. During the training process, all the parameters of the mask U-Net and the CLIP projection layer are trainable.

[0056] When initializing, the source U-Net model is based on the StableDiffusion-1.5 model, which converts the input source image into a 4-channel image latent code, realizing the efficient extraction of multi-level features of the source image. By connecting the keys and values of the source features with the mask U-Net, the present invention injects the source features into the mask U-Net in the middle and upsampling stages of the model, as shown in formula (1). This design enables the mask U-Net to utilize the content information of the source image to generate the corresponding content in the masked area of the source image.

[0057]

[0058] The present invention uses a depth arbitrary network to predict the depth map of an unmasked image as a tool for controlling the shape, enabling the dual U-Net stable diffusion restoration model to accurately transfer the texture of the blade surface. In this process, the present invention keeps the parameters of the depth model unchanged (i.e., freezes its parameters) and introduces an additional trainable projection module whose role is to map the predicted 3-channel depth map into a 4-channel depth latent space. To enhance the robustness and generalization ability of the model, during the training phase, the present invention randomly sets the input of the depth model to zero with a probability of 50%, implementing an input dropout strategy.

[0059] The present invention introduces a motion intensity estimation and text reweighting module to reduce the ambiguity of the text-to-motion mapping and strengthen the control of relative motion. In addition, the present invention uses the Structural Similarity Index Measure (SSIM) as an index to measure the similarity between video frames. The flowchart of the SSIM algorithm is shown in Figure 4 wherein the core idea is to ensure that the selected two frames of images contain both semantic correspondence and visual changes by screening out frame pairs with too high or too low similarity. The present invention comprehensively evaluates the similarity of two images from the following three dimensions:

[0060] (1) Luminance, which is characterized by the mean value. Taking the standard image x as an example, (of course, for another image y used for comparison, the formula is the same), as shown in Equation (2).

[0061]

[0062] (2) Contrast, which is characterized by the variance after mean normalization. Taking the standard image x as an example, as shown in Equation (3).

[0063]

[0064] (3) Structure, which is characterized by the correlation coefficient (i.e., r in the statistical sense, the ratio of the covariance to the product of the variances), as shown in Equation (4).

[0065]

[0066] As can be seen from the above, to enhance the difference between the original image and the masked image, the present invention implements a series of powerful data augmentation techniques. In addition to the conventional random scaling, random cropping, random rotation, and random brightness adjustment, the present invention also introduces random projection transformation to simulate more drastic deformation effects.

[0067] Masking strategy. Traditional masking strategies usually divide the masked image into an N*N grid and randomly select grids for masking. However, the present invention has found that this simple method often leads to the emergence of many simple cases. For example, in the image of an aeroengine blade, the background occupies most of the area, and these areas are usually repetitive content; the intact areas of the blade also often have a single and repetitive texture. Therefore, learning these areas does not require the model to seek guidance from the masked image.

[0068] As can be seen from the above, in order to identify more discriminative regions, the present invention applies the SIFT (Scale-Invariant Feature Transform) matching technique between the source image and the masked image to obtain a series of matching points. The SIFT algorithm detects key points in the image and generates a descriptor for each key point. This descriptor has good invariance to the scale change and rotation of the image, and also shows a certain degree of robustness to the illumination change and a certain degree of perspective change. By comparing the similarity of the key point descriptors in the two images, the present invention can achieve the matching of image features, which is sufficient to help the present invention construct better training samples, thereby increasing the possibility of masking the grid with matching feature points.

[0069] It should be noted that: 3D image modeling can be divided into explicit modeling and implicit modeling. Implicit modeling can only obtain photos from new perspectives and cannot directly edit the 3D model represented implicitly. Explicit modeling is divided into traditional methods and learning-based methods. Considering that the learning-based method is superior to the traditional method in terms of speed, reconstruction integrity, and generalization ability, the present invention adopts a learning-based explicit modeling algorithm to reconstruct the complete 3D model of the blade;

[0070] As can be seen from the above, the present invention uses the pre-trained MVSnet model as the 3D modeling method, and its working process is as Figure 5 shown;

[0071] I. Extracting graphic features. MVSnet receives multi-view image inputs, where the images include a reference image and multiple source images. For each input image (N 3-channel images with width W and height H), MVSnet uses an 8-layer 2DCNN to extract deep image features, and these features are encoded into multi-channel feature maps (N groups of 32-channel maps, with each channel having a scale of W / 4, H / 4);

[0072] II. MVSNet transforms the extracted feature maps into the conical stereo space of the reference camera through a series of homography transformations to construct a feature volume. Briefly speaking, for a point x in 3D space, in the present invention, when taking a photo with camera 1, the corresponding two-dimensional pixel point P(x, y) on photo 1 is obtained. When taking a photo with camera 2 at another position, the corresponding two-dimensional pixel point P'(x', y') on photo 2 is obtained. Through a correct homography matrix (including the position transformation parameters R, t of cameras 1 and 2, and the distance d from camera 1 to point x), P' = HP can be achieved. That is, on the premise that the internal and external parameters of the camera have been obtained in advance, only with a depth value variable, the position of point P corresponding to point P' on the source image in the reference image can be found. If the camera parameters (position transformation parameters of cameras 1 and 2) of two poses are known, now set a depth interval [d1, d2] and set the resolution as Δd, thus obtaining D = (d2 - d1) / Δd planes. Then di corresponds to a homography transformation H i (d).

[0073]

[0074] It should be noted that: for each pixel point on an image, using H i (d) for transformation can obtain a transformed image, which means that assuming the true depth of each pixel point is d, the corresponding feature values that each pixel point should have in another pose. And in the present invention, D depths are assumed, that is, D transformed images will be obtained. Each image represents the feature values corresponding to the transformation when the true depth of its pixel points is the current depth. This step is the key to connecting two-dimensional feature extraction and three-dimensional regularization network. It implicitly encodes the camera geometry and ensures the end-to-end of depth map training;

[0075] III. MVSNet aggregates these feature volumes into a unified cost space and measures the similarity of features by calculating the variance. The smaller the variance, the higher the similarity, and the more likely the true depth of this feature point is di, thus obtaining a cost volume. This cost volume reflects the matching degree of feature points under different depth assumptions.

[0076]

[0077] It should be noted that: due to non-Lambertian surfaces, occlusions, etc., the cost volume contains noise. To obtain a probability volume P through regularization, MVSNet performs regularization processing on the cost volume, uses a network structure similar to U-Net to encode and decode the cost volume, and finally compresses the number of channels of each to 1 to obtain a probability space. This probability space can be used to estimate the depth value of each pixel point and give the confidence of depth estimation;

[0078] After obtaining the probability space, MVSnet estimates the initial depth map by calculating the expectation. Using the probability volume from the previous step, it calculates the expectation along the d direction for each point in the (W, H) plane to obtain the initial depth map for the corresponding pixel points. The expected value is the depth estimate for that point. Calculating the expectation for each pixel point transforms the probability volume into a probability map;

[0079]

[0080] As can be seen from the above, the boundary information of the reference image is used to optimize the initial depth map to improve the accuracy of the boundary region;

[0081] 1. Reduce the reference image by 1 / 4 to make its size the same as the initial depth map;

[0082] 2. Normalize the depth map (W, H, 1) to [0, 1];

[0083] 3. Concatenate the two images channel-wise (W, H, 4) and put them into a convolutional network with a 4-layer residual structure for information fusion;

[0084] 4. Restore the single-channel feature map (W, H, 1) output by the residual network to the depth hypothesis interval (inverse to the normalization process), add it element-wise to the initial depth map, and thus obtain the optimized depth map. Then use Loss1 to calculate the losses for the initial depth map and the optimized depth map respectively, and multiply them by the weight coefficient λ, where λ is generally set to 1.0;

[0085]

[0086] Finally, MVSnet generates the final depth map or 3D point cloud through post-processing steps, including depth map filtering and depth map fusion. Depth map filtering proposes two constraints, namely photometric constraint and geometric constraint. The geometric constraint projects the reference point P1 to the source view point Pi through its estimated depth d1, and then projects the point Pi back to the reference view point preproj through its depth estimate di. The depth estimate of the reprojected preproj point is dreproj. If equation (9) is satisfied, it meets the geometric constraint, ensuring that the three views satisfy the consistency of this geometric constraint;

[0087]

[0088] It should be noted that: the photometric constraint needs to satisfy equation (10). Actually, a probability map is calculated while obtaining the initial depth map through the probability volume. Calculate the sum of the probabilities of four neighborhoods along the d direction for each point in the (W, H) plane, and then take the maximum sum of probabilities along the d direction. The more concentrated the depths of each pixel point are around a certain depth, the higher the accurate probability of the depth judgment for that point;

[0089] In the experiment of the present invention, points with a probability less than 0.8 are filtered out;

[0090]

[0091] The depth map fusion of the present invention is to infer depth maps from multiple perspectives and adopt a specific fusion algorithm for fusion. The pixel depth selection of each depth map uses the mean value of the reprojection calculated with geometric constraints as the final depth estimate. Different perspective depth maps are fused by adopting a visualization fusion algorithm that can minimize occlusion and conflict. The reconstruction noise is further suppressed by determining the visible view of each pixel in the depth map filtering step. The final depth selection is completed by using the average value of the reprojection depth as the final depth estimate;

[0092] Through the setting of the above technical solutions, the entire process of the present invention realizes the goal of inferring depth information from multi-perspective images and reconstructing a complete three-dimensional model of the blade.

[0093] Embodiment 2

[0094] During the training of the dual U-Net stable diffusion repair model:

[0095] The present invention uses the Adam optimizer and sets the learning rate to 1e-5. The learning rate adjustment adopts the simulated cosine annealing method. The experiment is carried out on a server with an operating system of Ubuntu 20.04 and a graphics card of NVIDIA GeForce RTX 4090 with a total batch size of 64. The method proposed by the present invention is implemented in PyTorch;

[0096] For the masking strategy of matching images:

[0097] The present invention randomly determines the number of grids N, and the value range of N*N is from 3*3 to 10*10, that is, N can be any integer between 3 and 10. For the grids where the features obtained by SIFT matching are located, the present invention sets a 75% probability of choosing to discard these grids. For the grid areas that do not pass SIFT matching, the present invention sets a 50% probability of choosing to discard. During the training process, the present invention introduces the U-Net network as an unsupervised auxiliary guidance, which means that the output of the U-Net is not directly used for classification, but as auxiliary information to promote model training. At the same time, in order to enhance the robustness of the model, the present invention has a 10% probability of completely ignoring the output of the U-Net during training. In the inference (i.e., prediction) stage of the model, the present invention defaults to using 5 as the guidance scale;

[0098] The present invention collects video frame images of intact blades of the same model as the damaged blades of an aero-engine. After obtaining the mask of the damaged area of the blade, the present invention uses a dual U-Net stable diffusion repair model for repair, and then evaluates the repair effect by comparing the repaired image with the intact image;

[0099] It should be noted that the present invention uses three standard metrics for image repair evaluation:

[0100] Peak Signal-to-Noise Ratio (PSNR);

[0101] Structure Similarity Index Measure;

[0102] Learned Perceptual Image Patch Similarity (LPIPS).

[0103] It should be noted that: PSNR is used to measure the difference between two images, such as a compressed image and the original image, to evaluate the quality of the compressed image, or a restored image and the GT, to evaluate the performance of the restoration algorithm, etc.; the minimum value of PSNR is 0, and the larger the PSNR, the greater the difference between the two images;

[0104] The calculation formula of PSNR is shown in Equation (11),

[0105]

[0106] It should be noted that: where MSE is the mean square error between two images, and MaxValue is the maximum value that image pixels can take.

[0107] The present invention further details that, based on the assumption that the human eye extracts structured information in images, SSIM is more in line with human visual perception than traditional methods. SSIM consists of three parts, namely luminance, contrast, and structure. SSIM ≤ 1, and the larger the SSIM, the more similar the two images. The calculation formula of SSIM is shown in Equation (12);

[0108]

[0109] It should be noted that: where α, β, γ > 0 are used to adjust the weights of the three parts, C1 = (K1L) 2 , C2 = (K2L) 2 , which are used to avoid the case where the denominator is 0. L is equivalent to MaxValue in PSNR. K1, K2 << 1 are very small constants. By default, K1 = 0.01, K2 = 0.03, μ x , σ x , σxy See Formulas (2) to (4);

[0110] It should be noted that LPIPS is a method for measuring image similarity. It evaluates the perceptual differences between two images through a deep learning model. LPIPS believes that even if two images are very close at the pixel level, human observers may still perceive them as different. Therefore, LPIPS uses pre-trained deep networks (such as VGG, AlexNet) to extract image features and then calculates the distance between these features to evaluate the perceptual similarity between images. The lower the LPIPS value, the more similar the two images are; conversely, the greater the difference.

[0111] Example 3

[0112] To evaluate the impact of different acquisition devices on the image restoration effect:

[0113] The present invention compares the impact of images captured by an Olympus borescope (Type I industrial borescope), a Guantai borescope (Type II industrial borescope), and an RGB camera on the restoration effect. Borescopes are highly regarded for their ability to deeply inspect areas that are difficult to directly observe, such as aeroengines. Their non-destructive testing capabilities play an important role in improving inspection efficiency and reducing maintenance costs. RGB cameras are renowned for their color capture ability and detail accuracy, which are particularly crucial for image processing and analysis. Figure 6 (a) shows the shooting and restoration results of different devices. The results show that the images captured by the RGB camera are the clearest and have the best restoration effect; while the images captured by the borescope are relatively blurred and the restoration effect is relatively poor, with sawtooth edges appearing on the blade. This indicates that the high resolution and pixel accuracy of the RGB camera are crucial for capturing image features and improving the restoration effect. Therefore, the RGB camera is used in subsequent experiments to ensure high-quality image restoration.

[0114] To explore the impact of lighting conditions on the image restoration effect:

[0115] The present invention designs four different lighting environments: low light, normal light, medium strong light, and strong light. The classification criteria for these lighting conditions are based on the degree of specular reflection generated when ambient light irradiates the smooth surface of an object during image acquisition by the RGB camera. The specific shooting and restoration results are shown in Figure 6 (b). It can be observed that as the ambient light intensity increases, the specular reflection phenomenon on the smooth part of the blade edge becomes more obvious, which leads to a gradual deterioration of the restoration effect. Therefore, for areas with obvious specular reflection, materials such as contrast enhancers can be sprayed to reduce specular reflection and promote diffuse reflection, thereby suppressing the specular reflection phenomenon on the smooth surface and improving the restoration effect.

[0116] To evaluate the impact of the shooting angle on the image restoration effect;

[0117] The present invention sets three different shooting angles: 0°, 45°, and 90°. The basis for the division of these angles is the angle between the camera and the plane where the blade is located. The specific shooting and repair results are shown in Figure 6 (c). It can be observed that at a shooting angle of 90°, the shooting and repair effects of the blade are the most ideal. This is because when the shooting angle is 90°, the acquisition device is perpendicular to the plane where the blade is located, avoiding the problems of viewing angle differences and occlusion. Therefore, based on this discovery, all subsequent experiments of the present invention are carried out with image acquisition at a shooting angle of 90°.

[0118] To explore the influence of the number of images on the image repair effect;

[0119] The present invention designs seven groups of experiments. The number of images used in each group of experiments is 7, 14, 21, 28, 35, 42, and 49 respectively. The basis for the division of the number of images is to determine the number of frames by dividing the entire video duration by the required number of images. The specific repair effects are shown in Figure 6 (d). The observation results show that as the number of images increases, SSIM and PSNR gradually increase, while LPIPS gradually decreases. Before the number of images reaches 28, the increasing speed of SSIM and PSNR and the decreasing speed of LPIPS are relatively significant. However, when the number of images exceeds 28, the changes of SSIM, PSNR, and LPIPS tend to be stable. At the same time, as the number of images increases, the time required for repair also gradually increases. When the number of images is 49, the time required to repair the image increases to 44.1 seconds, and the index of the repair effect does not improve significantly compared with that of 28 images. Therefore, based on the balance of efficiency and effect, all subsequent experiments of the present invention use 28 images.

[0120] The image repair experiment is carried out to verify the effectiveness of the method proposed by the present invention and compare its performance with existing image repair algorithms:

[0121] The reference models selected in this invention cover six methods in two categories. Among them, traditional image inpainting algorithms (GatedConv, DeepFillv1, PartialConvolution, MAT) mainly rely on background information to repair masked areas, while newer methods (Null-textInversion, Prompt-to-Prompt) use text prompts to guide the generation of content in masked areas. In the experiments of this invention, the model of this invention captures semantic correspondences between images through a self-supervised training strategy. At the same time, this invention also provides additional masks or bounding boxes for other models to indicate the reference areas. The size of all input images is uniformly 600×800 pixels. The experimental results are summarized in Table 1, and the visualization effects of image inpainting are shown in Figure 7 In.

[0122] As Figure 11 shown, Figure 11 The quantitative results of the image inpainting experiment are as follows: The experimental results confirm that the method proposed in this invention is superior to the existing methods, and the inpainted images perform better in terms of difference and similarity. Specifically, after averaging, SSIM reaches 0.90, PSNR is 29.98, and LPIPS drops to 0.08, which are 4%, 5%, and 20% higher than those of the existing optimal method Null-textInversion, respectively. In addition, this invention has successfully reconstructed the complete three-dimensional model of the blade through the MVSnet algorithm, as Figure 7 shown.

[0123] The performance improvement of this invention benefits from the adopted dual U-Net structure and self-supervised training strategy. The dual U-Net structure ingeniously deploys two U-Net sub-networks to capture image features at different scales respectively, and significantly improves the segmentation accuracy by intelligently fusing this multi-scale information. At the same time, this invention has carefully designed a self-supervised training strategy in three aspects: data selection, data augmentation, and masking strategy, effectively solving the problem of insufficient training data and further improving the model performance. In the experimental results, the inpainted images of deformed damages perform slightly worse in terms of difference and similarity, with SSIM and PSNR being 0.85 and 25.17 respectively, and LPIPS being 0.12. This may be because the features of deformed damages are not obvious enough, and it is difficult to accurately define the deformed damage areas even with manual marking.

[0124] In terms of the reference feature extractor, the dual U-Net stable diffusion inpainting model proposed by the present invention adopts a dual U-Net structure to model the features of the source image and the mask image respectively. Previous studies have confirmed that pre-trained diffusion models contain powerful prior knowledge before capturing semantic correspondences. The present invention further explores whether an asymmetric structure can learn semantic correspondence relationships in a self-supervised training framework. For this purpose, the Mask U-Net is replaced with a CLIP / DINOv2 image encoder, and 16×16 patch tokens are injected into the cross-attention layer. By comparing the quantitative results in Table 2, the present invention finds that CLIP and DINOv2 also exhibit competitive performance. However, since the U-Net can provide higher-resolution multi-level feature representations and the feature space is naturally aligned with the active U-Net, it can produce better results when used as a reference feature extractor.

[0125] Table 2 Quantitative results of ablation experiments on the reference feature extractor

[0126] Method SSIM PSNR LPIPS CLIP Encoder 0.84 28.11 0.11 DINOv2 Encoder 0.86 27.56 0.12 U-Net (ours) 0.90 29.98 0.08

[0127] In terms of the training strategy:

[0128] The present invention first confirms the effectiveness of the training method based on video frame images. As shown in Table 3, the experimental results show that the performance drops significantly when only relying on statistical data images, which confirms that the deformation or change of objects in the video is crucial for improving the accuracy of image inpainting. Then, the present invention removes data augmentation methods such as strong color jitter, resizing, and projective transformation, and finds that there is an obvious regression in the experimental results, thus verifying the importance of data augmentation in robust semantic correspondence matching;

[0129] The present invention also explores different image masking strategies and compares the inpainting performance using a single box and different grid ratios. The results show that these pure random masking strategies may lead to a large number of low-quality training samples. Relatively speaking, the method of guiding masking through SIFT matching can obtain better performance.

[0130] The following table shows the quantitative results of ablation experiments on the training strategy:

[0131]

[0132]

[0133] Conduct application experiments by applying the method proposed in the present invention to other common aviation damaged components, including compressors, pipeline components, guides, and turbine blades, etc., and perform model reconstruction experiments after image restoration to evaluate the generalization ability of the present invention. These components play a crucial role in the aviation industry, and the detection and repair of their damages are essential for ensuring flight safety and extending the equipment life. For example, Figure 8 as shown;

[0134] The present invention adopts an optimized image acquisition scheme, combines a dual U-Net stable diffusion restoration model, conducts detailed damage detection and image restoration on the damaged components, and then establishes a complete three-dimensional model. The specific results are shown in Figure 9 ;

[0135] From Figure 8 and Figure 9 it can be seen that even in complex and diverse damage situations, the method proposed in the present invention can effectively restore the original shape of the components and construct high-quality three-dimensional models, which can provide important references for subsequent damage analysis and repair work. The application experiments in this section prove that the present invention is not only effective for damaged blades of aeroengines, but also has good generalization ability and can be applied to image restoration and model reconstruction of various aviation damaged components. These experimental results further confirm the practicability and reliability of the method of the present invention and provide a new technical means for damage detection and repair in the aviation industry.

[0136] Through the setting of the above technical solutions, aiming at the problem of complete three-dimensional model reconstruction of damaged blades of aeroengines, the embodiment of the present invention proposes an innovative method - complete three-dimensional reconstruction of damaged blades of aeroengines based on an image generation diffusion model. The method proposed in the present invention abandons the traditional process of "blade point cloud acquisition - target model fitting - Boolean operation to obtain the blade repair model", and instead adopts a novel strategy: first, intelligently identify the damage mask in the video frame image, then use a dual U-Net model based on a self-supervised training strategy to repair the damaged image, and finally reconstruct the complete three-dimensional model of the blade;

[0137] The present invention proposes an image restoration model - dual U-Net stable diffusion restoration model, which is based on the StableDiffusion-1.5 architecture of the latent space diffusion principle. This architecture not only improves the efficiency of the traditional diffusion model in the noise addition and denoising process, but also realizes the matching of images and texts through the CLIPTextModel text encoder, effectively solving the problem of unconditional image generation. In addition, the introduction of the dual U-Net structure ensures the smooth continuity of the restored images, and the self-supervised training strategy solves the problem of insufficient training data from three aspects: data selection, data augmentation, and masking strategy;

[0138] The present invention has established a dataset of damaged blades of aero-engines. On this basis, extensive experiments have been carried out to verify the effectiveness and advancement of the method proposed by the present invention. The experimental results show that the method proposed by the present invention has reached 0.90, 29.98 and 0.08 in terms of SSIM, PSNR and LPIPS indicators respectively. Compared with the existing optimal method, the performance indicators have been improved by 4%, 5% and 20% respectively. Based on these results, the present invention has not only successfully reconstructed the complete three-dimensional model of the blade, but also extended to the repair of other aero-industry components, testing the strong generalization ability of the method of the present invention;

[0139] The present invention provides a method for complete three-dimensional reconstruction of damaged blades of aero-engines based on an image generation diffusion model, which not only provides a new attempt for the intelligent repair and model reconstruction of damaged blades of aero-engines, but also provides new ideas and methods for the research on the complex damage repair and model reconstruction of other aero-components.

[0140] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for complete three-dimensional reconstruction of damaged blades of aeroengines based on an image generation diffusion model, characterized in that, Including: S1: Data acquisition and dataset construction. Use an RGB camera to collect video frame images of aero-engine blade damage, and use random scaling, random cropping, random rotation, random brightness change, and random projection transformation to simulate stronger deformations. S2: Obtain the blade damage mask image. Input the blade video frame images collected by the RGB camera into a damage detector to detect damage, and perform dilation and blurring processing on the detection results. S3: Dual U-Net stable diffusion model. By designing a dual U-Net structure, enhance the reference feature extraction ability of the model and ensure the smooth continuity of the repaired image. S4: Establish a complete blade three-dimensional model. Use the learning-based explicit modeling algorithm MVSnet to reconstruct the complete three-dimensional model of the blade. S5: Conduct experiments and performance evaluations on the complete three-dimensional reconstruction method for damaged aero-engine blades.

2. The method for complete three-dimensional reconstruction of damaged blades of an aero-engine based on an image generation diffusion model according to claim 1, wherein: The training data of the diffusion model is designed through data selection, data augmentation, and masking strategies. The experiment conduct and performance evaluation are evaluated by conducting optimization experiments, image repair experiments, ablation experiments, and application experiments.

3. The method for complete three-dimensional reconstruction of damaged blades of aeroengines based on an image generation diffusion model according to claim 1, wherein: In the above S1, notches, tears, deformations, and chamfer drops are used as research objects, and a dataset is constructed to conduct experiments.

4. The method for complete three-dimensional reconstruction of damaged blades of aero-engines based on an image generation diffusion model according to claim 1, characterized in that: In the above S2, methods of pre-training, blurring processing, and dilation processing are used to create accurate and complete repairs for the damaged areas.

5. The method for complete three-dimensional reconstruction of damaged blades of aeroengines based on an image generation diffusion model according to claim 1, wherein: In the above S3, the StableDiffusion-1.5 architecture based on the latent space diffusion principle and the dual U-Net structure are adopted, and a self-supervised training strategy is designed from data selection, data augmentation, and masking strategies to solve the problem of lack of training data.

6. The method for complete three-dimensional reconstruction of damaged blades of an aero-engine based on an image generation diffusion model according to claim 1, characterized in that: In the above S3, a dual U-Net structure composed of a single U-Net model and an additional U-Net is used to capture features at different scales and improve the segmentation accuracy by fusing information at different scales.

7. The method for complete three-dimensional reconstruction of damaged blades of an aero-engine based on an image generation diffusion model according to claim 6, characterized in that: In the above S3, the dual U-Net structure is composed of a source U-Net, a mask U-Net, and a depth model.

8. The method for complete three-dimensional reconstruction of damaged blades of an aero-engine based on an image generation diffusion model according to claim 1, wherein: In the above S4, the three-dimensional modeling of the image is explicit modeling.

9. The method for complete three-dimensional reconstruction of damaged blades of an aero-engine based on an image generation diffusion model according to claim 1, characterized in that: In the above S5, by conducting optimization experiments and adopting the effects of different acquisition devices, lighting conditions, shooting angles, and the number of images on the image repair effect, the best solution is determined.

Images