Expansion method and system for fan blade defect infrared image data set

Through improved CycleGAN model and data enhancement technology, high-quality infrared images of fan blade defects are generated, which solves the problem of blurred image details or missing structures in the prior art, and improves image quality and model training effect.

CN119048425APending Publication Date: 2024-11-29PINGGAO GRP CO LTD +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410968040.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing infrared image generation method for fan blade defects is blurred in details or lacks structure, making it difficult to use for the training of defect detection models.

Method used

Using the improved CycleGAN model, the generator G and discriminator are trained, combined with adversarial loss, cyclic consistency loss and perceived image block similarity loss function, to generate higher quality fan blade defect infrared images, and equalize through histogram, Data enhancement methods such as Gaussian filtering and affine transformation further expand the data set.

Benefits of technology

The generated fan blade defect infrared image quality is improved, which can be better used for the training of defect detection models, solves the problem of blurred details or missing structures in the original model generated image, and improves the stability and semantic information consistency of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048425A_ABST
    Figure CN119048425A_ABST
Patent Text Reader

Abstract

The invention relates to an expansion method and system for a fan blade defect infrared image data set, and belongs to the technical field of image processing. Fan blade defect infrared image data are generated according to the original fan blade defect visible light image by using the trained CycleGAN model, and the CycleGAN model comprises two generators G and F and discriminators corresponding to the two generators respectively; a loss function during training of the CycleGAN model is obtained according to a loss function of adversarial loss of generators G and F and corresponding discriminators of the generators G and F; and jointly determining a loss function of circulation consistency loss of the original fan blade defect visible light image and the generated fan blade defect infrared image and a loss function of perception image block similarity loss of the original fan blade defect visible light image and the generated fan blade defect infrared image. According to the method, the loss function of the CycleGAN model is improved, so that the problem that the fan blade defect infrared image is difficult to be used for training a defect detection model due to defect detail blurring or structure missing is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method and a system for expanding a fan blade defect infrared image data set, belonging to the technical field of image processing. Background Art

[0002] Wind energy is a renewable and clean energy source. Wind power generation can effectively solve the current shortage of electricity energy in my country while achieving the sustainable development goals of low carbon and environmental protection. However, due to the long-term operation of wind power generation equipment in a complex and changeable natural environment, wind power generation equipment frequently fails. The wind turbine blades in wind power generation equipment are important components and also the most vulnerable part of the wind power generation system. After the wind turbine blades are damaged, it often costs millions of yuan to repair and maintain them, which not only causes economic losses but also threatens the safety of wind farm workers. Therefore, the defect detection of wind turbine blades directly affects the safe operation and economic benefits of wind farms.

[0003] As a non-destructive testing technology, infrared thermal imaging technology can effectively display the temperature information on the surface of fan blades and the internal structure of fan blade defects, thereby realizing defect detection of fan blades. However, since fan blades work at high altitudes, it is difficult to collect infrared images of fan blade defects, and the quality and scale of the data set of the deep learning model for fan blade defect detection will greatly affect the training effect of the deep learning model for fan blade defect detection. The higher the quality and the larger the data set, the better the generalization ability of the model can be obtained during training. Therefore, it is necessary to expand the infrared image data set of fan blade defects. The Chinese invention patent with application publication number CN117830806A discloses a method and related device for collecting infrared images. The technical solution can use an image training set containing multiple paired visible light images and infrared images suitable for specified scenes to train a CycleGAN model, and then use the trained CycleGAN model to convert the collected visible light images into corresponding infrared images, thereby expanding the infrared image training set. However, due to the large number of defect types of fan blade defects and the simple structure of the basic CycleGAN model, the defect details of the fan blade defect infrared images generated by the basic CycleGAN model are blurred or the structure is missing, making it difficult to use for the training of defect detection models. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for expanding a fan blade defect infrared image data set, so as to solve the problem that the fan blade defect infrared images generated by the existing infrared image generation method have blurred defect details or missing structures, and are difficult to be used for training defect detection models.

[0005] In order to solve the above technical problems, the present invention provides a method for expanding a fan blade defect infrared image data set, comprising: using a trained CycleGAN model to generate fan blade defect infrared image data for expanding a data set according to an original fan blade defect visible light image; the CycleGAN model comprises two generators G and F and discriminators corresponding to the two, the generator G is used to convert the original visible light image into an infrared image, and the generator F is used to convert the infrared image output by the generator G into a visible light image again, and the loss function when training the CycleGAN model is jointly determined according to the loss function of the adversarial loss of the generators G, F and their corresponding discriminators, the loss function of the cycle consistency loss of the original fan blade defect visible light image and the generated fan blade defect infrared image, and the loss function of the perceptual image block similarity loss of the original fan blade defect visible light image and the generated fan blade defect infrared image.

[0006] Furthermore, the method of determining the loss function when the CycleGAN model is trained includes: superimposing the loss function of the cycle consistency loss and the loss function of the perceptual image block similarity loss according to the set weights, and then superimposing the loss functions of the adversarial loss corresponding to the generators G and F respectively, to determine the loss function when the CycleGAN model is trained.

[0007] Furthermore, the adversarial loss functions of the generators G, F and their corresponding discriminators are all least squares loss functions.

[0008] Furthermore, the generators G and F of the CycleGAN model both include an encoder and a decoder; the convolutional layer of the encoder and / or the deconvolutional layer of the decoder include an EMA attention module.

[0009] Furthermore, the method also includes: performing histogram equalization on the generated infrared image data of fan blade defects, and then performing Gaussian filtering on the histogram equalized infrared image data of fan blade defects.

[0010] Furthermore, it also includes: processing the fan blade defect infrared image data after Gaussian filtering by affine transformation and Mosaic data enhancement to further expand the fan blade defect infrared image data set.

[0011] Furthermore, the discriminator uses a PatchGAN structure.

[0012] Furthermore, the affine transformation includes rotation, scaling and cropping.

[0013] Furthermore, the Mosaic data enhancement method includes: splicing two or more random infrared images of fan blade defects after Gaussian filtering by random scaling, random cropping or random arrangement.

[0014] Beneficial effects: As an improved invention, the present invention improves the loss function when training the CycleGAN model. The loss function when training the CycleGAN model is determined jointly according to the adversarial loss loss function of the generator G, F and its corresponding discriminator, the cycle consistency loss loss function of the original fan blade defect visible light image and the generated fan blade defect infrared image, and the loss function of the perceived image block similarity loss of the original fan blade defect visible light image and the generated fan blade defect infrared image. Among them, the adversarial loss function is used to improve the stability of training the model and reduce the possibility of mode collapse and gradient disappearance. The loss function of the cycle consistency loss is used to ensure that the infrared image reconstructed by the cycle has the same geometric shape and spatial relationship as the original visible light image; the loss function of the perceived image block similarity loss is added to ensure that the semantic information of the generated fan blade infrared image is consistent with the original visible image; thus, the present invention can improve the loss function when training the model, so that the model can generate higher quality images in the fan blade defect infrared image generation task, and solves the problem that the fan blade defect infrared image generated by the existing basic CycleGAN model has blurred defect details or missing structure, and is difficult to be used for the training of defect detection models.

[0015] To solve the above technical problems, the present invention also provides a system for expanding a fan blade defect infrared image data set, including a processor storing executable program instructions, wherein the executable program instructions are used to be executed to implement the above-mentioned method for expanding a fan blade defect infrared image data set.

[0016] The fan blade defect infrared image data set expansion system can achieve the same beneficial effects as the fan blade defect infrared image data set expansion method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of expanding a fan blade defect infrared image data set in an embodiment of the fan blade defect infrared image data set expansion method of the present invention;

[0018] Figure 2 1 is a schematic diagram of the basic CycleGAN model structure in an embodiment of the method for expanding the infrared image data set of fan blade defects of the present invention;

[0019] Figure 3It is a schematic diagram of the structural principle of the generator in the improved CycleGAN model structure in the embodiment of the method for expanding the fan blade defect infrared image data set of the present invention. DETAILED DESCRIPTION

[0020] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0021] Embodiment of a method for expanding a fan blade defect infrared image data set

[0022] The present invention proposes a technical solution for a method for expanding a fan blade defect infrared image data set. The method uses a trained CycleGAN model to generate fan blade defect infrared image data from an original fan blade defect visible light image. The CycleGAN model includes two generators G and F and discriminators corresponding to the two. The generator G is used to convert the original visible light image into an infrared image, and the generator F is used to convert the infrared image output by the generator G into a visible light image. The loss function when training the CycleGAN model is determined based on the adversarial loss function of the generators G and F and their corresponding discriminators, the cycle consistency loss function of the original fan blade defect visible light image and the generated fan blade defect infrared image, and the loss function of the perceptual image block similarity loss of the original fan blade defect visible light image and the generated fan blade defect infrared image, so that the model can generate higher quality images in the fan blade defect infrared image generation task. The following is a specific description.

[0023] The flowchart of the expansion of the infrared image data set of fan blade defects in this embodiment is as follows: Figure 1 As shown, the method includes:

[0024] The trained CycleGAN model is used to generate fan blade defect infrared image data for expanding the fan blade defect infrared image dataset based on the original fan blade defect visible light image.

[0025] Moreover, in this embodiment, the generated infrared image data of fan blade defects is also subjected to histogram equalization, thereby improving the overall contrast of the generated infrared image of fan blade defects while retaining the image details, and then Gaussian filtering is performed on the infrared image data of fan blade defects after histogram equalization to remove the noise existing in the infrared image data set of fan blade defects after histogram equalization, and because Gaussian filtering has a smaller degree of image blur, it can better maintain the details of the infrared image of fan blade defects. Then, the infrared image data of fan blade defects after Gaussian filtering is processed by affine transformation and Mosaic data enhancement to expand the data set of infrared images of fan blade defects again.

[0026] Affine transformation includes rotation, scaling and cropping. The method of Mosaic data enhancement includes: splicing two or more infrared images of fan blade defects after Gaussian filtering by random scaling, random cropping or random arrangement. In this embodiment, four infrared images of fan blade defects after Gaussian filtering are used and spliced ​​by random scaling, random cropping or random arrangement.

[0027] The training process of the CycleGAN model is as follows:

[0028] The CycleGAN model is trained by inputting the divided data set (that is, the data set with corresponding labels), calculating the loss function of the model, and iteratively updating the CycleGAN model according to the loss function until the iteration stop condition is met.

[0029] The method for acquiring the divided data set includes: collecting and analyzing the visible light image and infrared image data set of fan blade defects, and dividing the data set into style domains according to the characteristics of the fan blade defect data set. In this embodiment, according to the unique characteristics of the fan blade defect data set, the X and Y style domains are divided based on the visible light image and infrared image of the fan blade defects, and the X style domain is divided into the visible light image of the fan blade defects and the Y style domain is divided into the infrared image of the fan blade defects, which is equivalent to classifying the data set, and randomly dividing the images of the two divided domains in the data set into a training set and a test set according to an equal ratio of 8:2.

[0030] The loss function during training of the CycleGAN model is determined jointly based on the adversarial loss function of the generator G, F and its corresponding discriminator, the cycle consistency loss function of the original fan blade defect visible light image and the generated fan blade defect infrared image, and the perceptual image block similarity loss function of the original fan blade defect visible light image and the generated fan blade defect infrared image, so that the model can generate higher quality images in the fan blade defect infrared image generation task.

[0031] The methods for determining the loss function when training the CycleGAN model include:

[0032] The loss function of the cycle consistency loss and the loss function of the perceptual image block similarity loss are superimposed according to the set weights, and then the loss functions of the adversarial loss corresponding to the generators G and F are superimposed to determine the loss function of the CycleGAN model during training.

[0033] The adversarial loss functions of the generator G, F and their corresponding discriminator are all least squares loss functions (LSGAN), which are used to improve training stability and reduce the possibility of mode collapse and gradient disappearance; the least squares loss function is:

[0034]

[0035] Among them, G and F represent the generation network, D y and D x represents the corresponding discriminant network, E represents the mathematical expectation, x represents the sample in the fan blade defect visible light image dataset X, y represents the sample in the fan blade defect infrared image dataset Y, P data (x) and P data (y) represent the sample distribution of image domains x and y respectively. G(x) represents the enhanced image generated by inputting the visible light image x of the fan blade defect into the generator G, and F(y) represents the enhanced image obtained by inputting the infrared image y of the fan blade defect into the generator F. D x In order to distinguish the real sample x from the generated sample F(y), D y The purpose is to distinguish the real sample y from the generated sample G(x).

[0036] Using the loss function of cycle-consistent loss (L cycle ) measures the similarity between the original image and the reconstructed image in training, ensuring that the cyclically reconstructed infrared image has the same geometric shape and spatial relationship as the original visible light image. The loss function of the cycle consistency loss is:

[0037]

[0038] Since the cycle consistency loss is not enough to recover all the geometric and spatial information, the loss function of the perceptual image block similarity loss (LPIPS) is added to evaluate the difference between the two images by calculating the distance between the image features to ensure that the semantic information of the generated fan blade infrared image is consistent with the original visible image. The loss function of adding the perceptual image block similarity loss is:

[0039]

[0040] x and x0 represent the two points to be compared, H and W represent the pixel positions in the image, H is the height index, and W is the width index. L represents different feature layers. and represents the feature stack extracted from layer L, Indicates the use of vector w l The activation channels are scaled and the L2 distance is calculated, and finally the average is performed in space and the sum is performed on the channels to obtain L lpips Loss. Since the above loss function (LPIPS) actually belongs to the prior art, it will not be described here.

[0041] The loss function of the CycleGAN model during training is:

[0042] L=L lsgan (GD y XY)+L lsgan (FD x XY)+λL cycle (GF)+(1-λ)L lpips (x, x0)

[0043] Among them, the weights of different losses are balanced by setting λ. In this embodiment, λ is set to 0.4, so the weight of cycle consistency loss is 0.4, and the weight of increasing the perceived image block similarity loss is 0.6. The loss function of the CycleGAN model of this embodiment during training is:

[0044] L=L lsgan (GD y XY)+L lsgan (FD x XY)+0.4L cycle (GF)+0.6L lpips (x, x0)

[0045] The CycleGAN model used in this embodiment is actually a model obtained by improving the existing basic CycleGAN model; specifically, the structural diagram of the existing basic CycleGAN model is as follows: Figure 2 As shown in Figure 2, a dual GAN ​​structure is used, which consists of two generators and two discriminators: Generator G(X→Y), F(Y→X), D X (the discriminator corresponding to F) and D Y (The discriminator corresponding to G) forms a bidirectional ring structure to realize the mutual conversion between the visible light image and the infrared image of the fan blade defect. Generator G is used to convert the original visible light image into an infrared image, and generator F is used to convert the infrared image output by generator G into a visible light image.

[0046] The improved CycleGAN model in this embodiment also includes two generators G and F and discriminators corresponding to the two. The structure of the generator in the improved CycleGAN model is as follows: Figure 3As shown, it includes an encoder, a residual network (converter) and a decoder; wherein, compared with the existing basic CycleGAN model, the structure of the generator is improved in the improved CycleGAN model, specifically: the convolution layer of the encoder and the deconvolution layer of the decoder both contain an EMA attention module. In other embodiments, it can also be set to contain an EMA attention module in the convolution layer of the encoder or the deconvolution layer of the decoder; the EMA attention module is added to the encoder to improve the feature extraction capability and retain more detail features while reducing the dimension (downsampling). The EMA attention module is added to the decoder to better restore the detail features of the image while increasing the dimension (upsampling). Therefore, in this embodiment, the EMA attention mechanism is added to the convolution layer of the encoding process of the generator and the deconvolution layer of the decoding process, so that the EMA attention mechanism is used to perform weight processing on the area with poor perception ability of the fan blade defect image, thereby solving the problem of missing defect details in the infrared image of the fan blade. The encoder downsamples the input visible light fan blade surface defect image through a convolution operation to generate a high-dimensional feature vector. The residual network extracts and transforms features while reducing the gradient vanishing problem. The decoder uses deconvolution to upsample the high-dimensional feature vector and decode it into an infrared image of a fan blade defect with the same object content as the input image. Taking the generator G(X→Y) as an example, its goal is to convert a visible image of a fan blade defect into an infrared image of a fan blade defect.

[0047] In addition, in the improved CycleGAN model of this embodiment, the discriminator uses the PatchGAN structure, which can enhance the focus on the overall image and local detail features; the input of the discriminator is the visible light or infrared image of the fan blade defect generated by the generator. Y For example, its purpose is to distinguish the true and false of the real infrared image of the fan blade defect and the generated infrared image of the fan blade defect, which can be regarded as a binary classification task implemented by a multi-layer convolutional neural network.

[0048] Embodiment of a system for expanding a fan blade defect infrared image data set

[0049] This embodiment provides a technical solution for a system for expanding a fan blade defect infrared image data set. The system includes a processor in which executable program instructions are stored. The processor is used to execute the stored executable program instructions to expand the fan blade defect infrared image data set.

[0050] Since the specific implementation process and principle of the system for expanding the infrared image data set of defects of fan blades in this embodiment have been described in detail in the embodiment of the method for expanding the infrared image data set of defects of fan blades, they will not be described in detail here.

Claims

1. A method for expanding the infrared image dataset of fan blade defects, include: The trained CycleGAN model is used to generate infrared image data of fan blade defects for expanding a data set based on the original visible light image of the fan blade defects; the CycleGAN model includes two generators G and F and discriminators corresponding to the two respectively, the generator G is used to convert the original visible light image into an infrared image, and the generator F is used to convert the infrared image output by the generator G into a visible light image again, characterized in that the loss function when training the CycleGAN model is jointly determined according to the loss function of the adversarial loss of the generators G, F and their corresponding discriminators, the loss function of the cycle consistency loss between the original visible light image of the fan blade defects and the generated infrared image of the fan blade defects, and the loss function of the perceptual image block similarity loss between the original visible light image of the fan blade defects and the generated infrared image of the fan blade defects.

2. The method for expanding the infrared image data set of fan blade defects according to claim 1, It is characterized in that Methods for determining the loss function when the CycleGAN model is trained include: The loss function of the cycle consistency loss and the loss function of the perceptual image block similarity loss are superimposed according to the set weights, and then the loss functions of the adversarial loss corresponding to the generators G and F are superimposed respectively to determine the loss function of the CycleGAN model during training.

3. The method for expanding the infrared image data set of fan blade defects according to claim 1, It is characterized in that The adversarial loss functions of the generators G, F and their corresponding discriminators are all least squares loss functions.

4. The method for expanding the infrared image data set of fan blade defects according to claim 3, It is characterized in that The generators G and F of the CycleGAN model both include an encoder and a decoder; the convolutional layer of the encoder and / or the deconvolutional layer of the decoder include an EMA attention module.

5. The method for expanding the infrared image data set of fan blade defects according to claim 1, It is characterized in that Also includes: The generated infrared image data of fan blade defects are subjected to histogram equalization, and then the infrared image data of fan blade defects after histogram equalization are subjected to Gaussian filtering.

6. The method for expanding the infrared image data set of fan blade defects according to claim 5, It is characterized in that Also includes: The infrared image data of fan blade defects after Gaussian filtering are processed by affine transformation and Mosaic data enhancement to further expand the infrared image data set of fan blade defects.

7. The method for expanding the infrared image data set of fan blade defects according to claim 1, It is characterized in that The discriminator uses the PatchGAN structure.

8. The method for expanding the infrared image data set of fan blade defects according to claim 6, It is characterized in that The affine transformation includes rotation, scaling and cropping.

9. The method for expanding the fan blade defect infrared image data set according to claim 8, It is characterized in that The Mosaic data enhancement method includes: splicing two or more random infrared images of fan blade defects after Gaussian filtering by random scaling, random cropping or random arrangement.

10. A system for expanding infrared image datasets of fan blade defects. It is characterized in that It includes a processor, in which executable program instructions are stored, and the executable program instructions are used to be executed to implement the method for expanding the fan blade defect infrared image data set as described in claims 1-9.

Citation Information

Patent Citations

  • Method for converting thermal infrared image into visible-light-like color image based on CycleGAN

    CN114119356A

  • Infrared weak and small target detection method based on infrared small sample amplification and YOLOv5

    CN115861599A

  • Infrared image generation method based on improved generative adversarial network

    CN117541665A