Small sample wind turbine blade dual-light fusion defect detection method based on generative data set

Abstract

The invention relates to the technical field of fan blade detection, in particular to a small-sample wind turbine blade double-light fusion defect detection method based on a generative data set, which comprises the following steps: S1, collecting a small-sample wind turbine blade thermal image and a visible light image, and carrying out image fusion expansion by adopting a self-adaptive Laplacian pyramid fusion algorithm; s2, generating defect thermal image and visible light image samples, and dividing the samples into a training set, a verification set and a test set; s3, training the dual-light fusion defect detection network by using the training set, optimizing the model by using the verification set, and evaluating the detection performance by using the test set; s4, collecting images on line, inputting the images into the model for defect detection, outputting defect types and labeling defect areas; according to the method, small sample data are expanded through the generative AI, a dual-light fusion detection model and a multi-branch detection structure are constructed, and the accuracy, comprehensiveness and robustness of wind turbine blade defect detection are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application relates to the field of fan blade detection technology, and in particular to a small sample wind turbine blade dual-light fusion defect detection method based on a generative dataset. BACKGROUND

[0002] Fan blades are core components of wind turbines, and performing factory detection on the fan blades can effectively avoid operation failures and safety problems caused by manufacturing defects. In recent years, with the development of artificial intelligence technology, target detection technology based on deep learning has been widely applied and is gradually replacing traditional defect detection technologies such as digital image processing and shallow machine learning.

[0003] In actual production, the number of directly obtainable fan blade defect samples is small and the diversity is poor. Deficiency of defect sample data and class imbalance can lead to problems such as poor generalization ability of the deep learning model and low detection accuracy. Therefore, the small sample dataset of fan blade defects collected needs to be expanded. At present, dataset expansion methods mainly fall into two categories: data enhancement methods based on classical image processing techniques, such as rotation, translation, scaling, cropping, color transformation, and adding noise; and dataset expansion methods based on generative AI, that is, by learning the patterns and distribution of sample data to obtain a sample generation model. New samples similar to the original data can be generated through the generation model. SUMMARY

[0004] The application provides a small sample wind turbine blade dual-light fusion defect detection method based on a generative dataset. The deep learning algorithm is used to solve the defect detection problem, which is not only suitable for small sample conditions but also effectively improves the accuracy of fan blade defect detection.

[0005] The small sample wind turbine blade dual-light fusion defect detection method based on a generative dataset comprises the following steps:

[0006] S1. In the offline stage, small sample wind turbine blade thermal image and visible light image acquisition and fusion expansion: use an infrared thermal imager and a camera to collect fan blade defect thermal images and visible light images, use an adaptive Laplacian pyramid image fusion method to fuse and expand the original fan blade defect thermal images and visible light images respectively, and generate a defect image dataset;

[0007] S2. In the offline stage, wind turbine blade defect thermal image and visible light image dataset generation: use the expanded fan blade defect image dataset to train a LENW-DDPM generation model, use the LENW-DDPM generation model to generate a fan blade defect thermal image dataset and a visible light image dataset, and divide them into a training set, a validation set and a test set;

[0008] S3, dual-optical fusion defect detection model training and optimization: in the offline stage, using the training set of wind turbine blade defect thermal images and visible light images to train the dual-optical fusion wind turbine blade defect detection network, obtaining the dual-optical fusion wind turbine blade defect detection model, using the verification set to optimize the dual-optical fusion wind turbine blade defect detection model, using the test set to evaluate the performance of the detection model;

[0009] S4, online defect detection and output result analysis: in the online stage, using infrared thermal imager and camera to collect wind turbine blade thermal images and visible light images, using dual-optical fusion wind turbine blade defect detection model to detect defects of wind turbine blade, outputting defect type, and segmenting defect area in original image.

[0010] Optionally, the adaptive Laplacian pyramid image fusion method in S1 comprises:

[0011] Dynamic mask image generation strategy: in each iteration cycle of the Laplacian pyramid image fusion algorithm, according to the characteristics of two thermal images (or visible light images) I A 、I B , a dynamic mask image M d including feature information and feature direction information is generated by using a dynamic mask generation strategy;

[0012] Mask image Gaussian pyramid construction: the dynamic mask image M d is used as the mask image of the Laplacian pyramid image fusion algorithm, and K layers of mask Gaussian pyramid

[0013] Image Laplacian pyramid construction: I A and I B are used as the to-be-fused images of the Laplacian pyramid image fusion algorithm, and K layers of image Gaussian pyramid and the Laplacian pyramid of each layer are generated , which is expressed as:

[0014]

[0015] Wherein, EXPAND(G k ) is the up-sampling of the kth layer of the Gaussian pyramid according to the resolution of the k-1th layer;

[0016] Multi-level feature map fusion: the Laplacian pyramid is fused by level, and the mask Gaussian pyramid controls the proportion of the two Laplacian pyramids , which is expressed as:

[0017]

[0018] where int is an integer function;

[0019] Image reconstruction and final map generation: G k The low-frequency image obtained after upsampling and FL k-1 The Laplace high-frequency details of the current layer are added to restore the Gaussian image of the current layer, and the iteration is performed until the reconstructed image I is obtained, which is represented as:

[0020] I=G0,G K =FL K ,G k-1 =FL k-1 +EXPAND(G k ),k=1,2...,K.

[0021] Optionally, the dynamic mask generation strategy comprises:

[0022] Generating a feature mask: cutting ResNet50 to remove its fully connected layer and global average pooling layer, using the cut ResNet50 to extract deep feature maps F1 and F2 of two images I A and I B , calculating the difference between F1 and F2 and performing normalization processing to obtain a feature mask map M f containing feature differences, represented as:

[0023] D=||ResNet(I A )-ResNet(I B )||2;

[0024]

[0025] Generating a direction mask with direction information: using a Sobel operator to calculate the horizontal gradient, vertical gradient and diagonal gradient of the feature mask map, and generating a direction mask M o according to the average gradient of each direction, represented as:

[0026] S x =Sobel x (M f ),S y =Sobel y (M f ),S xy =Sobel xy (M f );

[0027]

[0028] wherein, H and W are the height and width of the feature map, respectively;

[0029] Dynamic mask generation: weighted fusion of feature mask map and direction mask to generate dynamic mask M including feature map information and feature direction information d , is expressed as:

[0030] M d =α×M f +(1-α)×M o ,α∈[0,1];

[0031] Wherein, alpha is a weight parameter.

[0032] Optionally, the LENW-DDPM generation model comprises:

[0033] Generation model structure definition: replace the U-net backbone network in DDPM with LENW-Net deep neural network;

[0034] Forward diffusion process modeling: inject noise into the defect image dataset, gradually generate multiple distribution samples, and gradually approach the standard Gaussian distribution to form a forward diffusion path;

[0035] Reverse generation process modeling: starting from random noise x t , the noise added in the forward diffusion process is predicted by training the LENW-Net deep neural network;

[0036] Image reconstruction: denoising the noise map x t , respectively generating wind turbine blade defect thermal image and visible light image samples, and finally constructing defect thermal image dataset and visible light image dataset.

[0037] Optionally, the LENW-Net deep neural network adopts a two-stage W-shaped structure composed of two stages of deep neural networks, the first stage of deep neural network includes a symmetric encoder-decoder architecture composed of three layers of down-sampling and up-sampling, and the second stage of deep neural network includes an encoder-decoder architecture composed of two layers of down-sampling, up-sampling and one layer of nested structure, the processing of the LENW-Net deep neural network on the defect thermal image or visible light image X of the wind turbine blade comprises:

[0038] The first stage of deep neural network adopts two layers of initial convolution to extract shallow features from the input X, and retains spatial details, which is expressed as:

[0039] X=Conv2(σ(Conv1(X)));

[0040] Wherein, Conv is a 3x3 convolution kernel, and sigma is a sigmoid activation function.

[0041] The encoding layer of the first-stage deep neural network down-samples X three times, each time reducing the image resolution by half and doubling the number of channels, gradually compressing the spatial dimension and extracting high-level semantic features. Before down-sampling at the maximum resolution layer, an AEB module is used to enhance the network's learning of local important details, represented as:

[0042] XD k = Downsample(AEB(X)), k = 1;

[0043] XD k = Downsample(ResnetBlock(SA(ResnetBlock(XD k-1 ))), k = 2, 3;

[0044] where Downsample is down-sampling, SA is self-attention mechanism, and k is the number of layers;

[0045] Data XD3 passes through residual blocks, SA attention, and residual blocks in turn, and global feature fusion is performed at the lowest resolution layer, represented as:

[0046] XU3 = ResnetBlock(SA(ResnetBlock(XD3)));

[0047] The decoding layer of the first-stage deep neural network up-samples XU3 three times, gradually restoring the resolution and number of channels, and fusing the features of the corresponding layers of the encoder through skip connection to supplement the detail information. After up-sampling at the maximum resolution layer, an AEB module is used to enhance the network's learning of local important details, represented as:

[0048] XU k = ResnetBlock(SA(ResnetBlock(Upsample(XU k+1 )))) + XD k , k = 1, 2;

[0049] XT = AEB(Upsample(XU1)) + X;

[0050] where Upsample is up-sampling, and k is the number of layers;

[0051] For XT, residual blocks, convolution, and AEB modules are used in turn to enhance local details, and the output of the first-stage deep neural network is obtained, represented as:

[0052] Y1 = AEB(Conv2(σ(Conv1(ResnetBlock(XT))));

[0053] The encoding layer of the second-level deep neural network performs two times of downsampling on Y1, each time of downsampling halves the image resolution and doubles the channel number, gradually compresses the spatial dimension, and extracts high-level semantic features. Before downsampling at the maximum resolution layer, an AEB module is used to strengthen the network to learn local important details, and is expressed as:

[0054] XD11 = Downsample(AEB(Y1));

[0055] XD12 = Downsample(ResnetBlock(SA(ResnetBlock(XD11))));

[0056] XD12 is processed in two parallel ways. The first way is that a first decoder of the second-level deep neural network performs two times of upsampling on XD12, fuses the features of the corresponding layers of the encoder through a skip connection, supplements the detail information, and after upsampling at the maximum resolution layer, an AEB module is used to strengthen the network to learn local important details to obtain XU10. The second way is that global feature fusion is performed on XD12 at the lowest resolution layer to obtain XU22, which is then processed by a second decoder of the second-level deep neural network to perform two times of upsampling, fuse the features of the corresponding layers of the encoder and the first decoder through a skip connection, supplement the detail information, and after upsampling at the maximum resolution layer, an AEB module is used to strengthen the network to learn local important details to obtain XU21, which is then upsampled to obtain XU20, and is expressed as:

[0057] XU11 = ResnetBlock(SA(ResnetBlock(Upsample(XD12)))) + XD11;

[0058] XU10 = AEB(Upsample(XU11)) + Y1;

[0059] XU22 = ResnetBlock(SA(ResnetBlock(XD12)));

[0060] XU21 = ResnetBlock(SA(ResnetBlock(Upsample(XU22)))) + XU11;

[0061] XU20 = AEB(Upsample(XU21)) + XU10;

[0062] For XU20, a residual block and two convolutions are used in sequence to obtain the output of the second-level deep neural network, and are expressed as:

[0063] Y2 = Conv2(sigma(Conv1(ResnetBlock(XU20))).

[0064] The input X of the first level network is element-wise added with the output Y2 of the second level network to obtain the final output of the network, denoted as:

[0065] Y=X+Y2;

[0066] Optionally, the AEB module comprises:

[0067] Two ResNetBlocks are used to extract preliminary features, wherein the ResNetBlock uses stacked convolution and nonlinear activation function to extract features, strengthens the understanding of local details, fuses the time steps of the diffusion process through time embedding, uses the Dropout layer to prevent overfitting, and connects the input and output residual through Conv convolution, denoted as:

[0068]

[0069] X=ResNetBlock(ResNetBlock(X));

[0070] wherein Conv1 and Conv2 are convolution operations, GN1 and GN2 are group normalization operations, sigma is SiLU activation, Dropout is the dropout rate, and Linear(sigma(t)) is the time embedding projection TimeEmbedding.

[0071] An LIA attention mechanism is used to process the features extracted by the ResNetBlock, enhance local detail features, and suppress background noise, denoted as:

[0072] LIA(X)=X·w·G;

[0073] X=LIA(X);

[0074] wherein LIA is the LIA attention mechanism, G is the gate signal, and w is the weight.

[0075] Two ResNetBlocks are used to extract higher-level features to further refine details, denoted as:

[0076] X=ResNetBlock(ResNetBlock(X)).

[0077] Optionally, the dual-light fusion wind turbine blade defect detection network in S3 comprises a visible light branch, an infrared branch and a visible light / infrared fusion branch. According to the defect detection capabilities of the visible light branch, the infrared branch and the visible light / infrared fusion branch, the defect detection results of the visible light branch, the infrared branch and the visible light / infrared fusion branch are selected and output through an output selection link. The visible light branch detects the coating wear defect of the wind turbine blade based on CA-YOLO. The infrared branch detects the omissions and resin-rich defects of the wind turbine blade based on the adaptive threshold heat map prior wind turbine blade infrared defect detection network. The visible light / infrared fusion branch detects the cracks and scratches defects of the wind turbine blade based on the dual-channel YOLOv8 and bidirectional pyramid fusion strategy dual-light fusion wind turbine blade defect detection network DB-YOLOv8.

[0078] Optionally, the CA-YOLO comprises:

[0079] The C2f-CBAM module is used to replace the C2f module of the Neck part of the YOLOv8 network.

[0080] A multi-task learning framework is introduced to simultaneously perform the instance segmentation of defects and the target detection of the blade position where the defects are located under one network structure.

[0081] According to the weight adjustment based on uncertainty, a multi-task optimization loss function is constructed, which is represented as:

[0082]

[0083] Wherein, σ1 and σ2 represent the uncertainty parameters of the detection task and the segmentation task respectively, L seg is the segmentation loss, L det is the detection loss.

[0084]

[0085] L det = α1L cls + α2L iou + α3L dfl .

[0086] Wherein, L cls is the classification loss, L iou is the bounding box regression loss, L dfl is the discretized bounding box regression loss.

[0087] Optionally, the adaptive threshold heat map prior wind turbine blade infrared defect detection network comprises:

[0088] The local mean dynamic adjustment threshold method is used to calculate the dynamic threshold, which is represented as:

[0089] T(x,y) = μ(x,y) + C;

[0090] Wherein, μ(x,y) represents the average temperature value of all pixels in the local window centered on pixel (x,y), and C is an offset;

[0091] According to the dynamic threshold T(x,y), the collected thermal image is binarized to generate a defect mask image, and the mask image is:

[0092]

[0093] The pixel weight factor is calculated and represented as:

[0094] W = 1 + (λ f -1)M(x,y);

[0095] Wherein, λ f ≥1 is the control weighting intensity;

[0096] The pixel weight factor W is used to optimize the YOLOv8 loss function:

[0097]

[0098] Wherein,

[0099] Optionally, the dual-light fusion wind turbine blade defect detection network DB-YOLOv8 comprises:

[0100] Two YOLOv8 networks are respectively used to extract defect features in the visible light image and defect features in the infrared image, and two types of multi-scale features are obtained;

[0101] A bidirectional cross-pyramid fusion module BCPF is used to perform bidirectional fusion on the two types of multi-scale features at different levels, wherein the BCPF is composed of two parallel pyramid feature extraction branches, and the visible light image and the infrared image are processed respectively, each branch outputs multi-scale feature maps of different levels of semantic and spatial information, and after the multi-scale feature maps of the two branches are fused, the fusion feature is sequentially processed through an upsampling module and a C2f convolution, and three scales of fusion features are outputted.

[0102] The fusion feature is added as supplementary information to the two YOLOv8 networks for feature extraction, and three detection results of different scales are generated.

[0103] The beneficial effects of the present application are:

[0104] The present application can generate high-quality and diversified defect image samples by acquiring small sample image data of wind turbine blade defects through an infrared thermal imager and a visible light camera, combining an adaptive Laplacian pyramid image fusion algorithm with an improved LENW-DDPM generation model for data augmentation, effectively alleviating the problems of data scarcity and class imbalance, and thus improving the robustness and generalization ability of a deep learning model in a defect detection task.

[0105] The present application can effectively improve the detection accuracy of complex and fine-grained defects such as cracks and scratches on wind turbine blades by constructing a dual-modal fusion mechanism of visible light and infrared images, designing a dual-channel deep feature extraction and fusion network, and realizing joint perception of multi-source information.

[0106] The present application can significantly improve the comprehensiveness and classification accuracy of wind turbine blade defect detection by setting up visible light branches, infrared branches and dual-light fusion branches, respectively detecting different types of defects such as coating wear, missed filling, rich resin, cracks and scratches, and comprehensively judging through an output selection mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0107] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only a part of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0108] Figure 1 The detection method flowchart of the embodiment of the present application is shown in the figure.

[0109] Figure 2 The adaptive Laplacian pyramid image fusion algorithm principle diagram of the embodiment of the present application is shown in the figure.

[0110] Figure 3 The LENW-Net network structure diagram of the embodiment of the present application is shown in the figure.

[0111] Figure 4 The AEB module structure diagram of the embodiment of the present application is shown in the figure.

[0112] Figure 5 The dual-light fusion defect detection algorithm principle diagram of the embodiment of the present application is shown in the figure.

[0113] Figure 6 The CA-YOLO network structure diagram of the embodiment of the present application is shown in the figure.

[0114] Figure 7 The DB-YOLOv8 network structure diagram of the embodiment of the present application is shown in the figure.

[0115] Figure 8 A bidirectional cross-pyramid fusion BCPF module structure diagram of an embodiment of the present application;

[0116] Figure 9 An infrared image fusion result diagram of a wind turbine blade based on different fusion expansion algorithms of an embodiment of the present application;

[0117] Figure 10 A visible light image fusion result diagram of a wind turbine blade based on different fusion expansion algorithms of an embodiment of the present application;

[0118] Figure 11 An image generation result diagram based on different generation algorithms of an embodiment of the present application. DETAILED DESCRIPTION

[0119] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. Those skilled in the art can also use other alternative ways to implement some known technologies; and the drawings are only used to more specifically describe the embodiments and are not intended to specifically limit the present application.

[0120] As shown in Figures 1-11 , the small sample wind turbine blade dual-light fusion defect detection method based on generative dataset includes infrared thermal image and visible light data acquisition, and uses an MV-GE31GC industrial camera and a TN460 infrared thermal imager to collect original small sample wind turbine blade defect dataset. The thermal image size obtained by the collection is 640x512, and the visible light image size is 752x480.

[0121] The small sample wind turbine blade dual-light fusion defect detection method based on generative dataset includes an offline stage and an online stage, and the specific steps are as shown in Figure 1 . Among them, the offline stage includes fusion expansion of the wind turbine blade small sample dataset, training of the LENW-DDPM network and training of the dual-light fusion wind turbine blade defect detection network.

[0122] In the offline stage of the wind turbine blade dual-light fusion defect detection method, a self-adaptive Laplacian pyramid image fusion data processing method is used to fuse and expand the wind turbine blade small sample dataset, and the algorithm principle is as shown in Figure 2 , which specifically includes the following steps:

[0123] (1) Generating a feature mask. The ResNet50 is cropped to remove its fully connected layer and global average pooling layer, and the cropped ResNet50 is used to extract deep feature maps F1 and F2 of two images I A , I B , calculate the difference between F1 and F2 and perform normalization processing to obtain a feature mask graph M containing feature differencesf The feature mask represents the difference of important features of two images by the feature map obtained by deep learning, reflects the local semantic information of the images in the feature space, and the greater the mask value is, the greater the difference of the two images existing in the corresponding region is, and the greater weight should be given to retain more significant features. f The calculation formula is:

[0124] D = ||ResNet(I A )-ResNet(I B )||2;

[0125]

[0126] (2) The direction mask with direction information is generated. First, the horizontal gradient, the vertical gradient and the diagonal gradient of the feature mask are calculated using the Sobel operator, and then the direction mask M o is generated according to the average gradient of each direction, that is:

[0127] S x = Sobel x (M f ), S y = Sobel y (M f ), S xy = Sobel xy (M f );

[0128]

[0129] In the formula, H and W are the height and width of the feature map respectively.

[0130] (3) The feature mask and the direction mask are weighted and fused to generate a dynamic mask M d containing feature map information and feature direction information, that is:

[0131] M d = α × M f + (1-α) × M o , α ∈ [0, 1];

[0132] In the formula, the weight parameter α is used to control the contribution degree of the feature mask and the direction mask, when the feature region of the two images to be fused is large, α should take a larger value, that is, more feature difference information is retained, otherwise a smaller value is taken, and more structure information is retained. In the present application, α is 0.6.

[0133] (4) The mask image M d is decomposed in multiple scales, and a k-layer mask Gaussian pyramid is generated by Gaussian blur and down-sampling iteration This process can be represented as:

[0134]

[0135] In the formula, Reduce represents Gaussian blur and downsampling operations, i and j are pixel positions, w is an m×n Gaussian kernel, and the size of each layer is halved after downsampling.

[0136] The k-1 layer of the Gaussian pyramid contains both high-frequency and low-frequency information. Since the k-layer is generated by Gaussian blurring and downsampling of the k-1 layer, it primarily retains the low-frequency information from the k-1 layer. Similarly, it can be formed from two images I to be fused. A I B Iterative generation of k-layer Gaussian pyramid images

[0137] (5) For and Generate the Laplace pyramid for each level and (k = 0, 1, 2, ..., K), the generating formula is:

[0138]

[0139] In the formula, EXPAND(G) k To upsample the k-th layer of the Gaussian pyramid to a resolution of k-1 layers, the formula is:

[0140]

[0141] Due to G k It is G k-1 The low-frequency approximation, EXPAND(G) k ) is G k-1 Low-frequency information in, so L k-1 For G k-1 China was not included in G k High-frequency details captured.

[0142] (6) Arrange the results obtained in step (5) by level Perform fusion and, based on the mask pyramid obtained in step S4 The ratio of the two feature maps in each layer is controlled by the following fusion formula:

[0143]

[0144] To ensure that pixel values ​​are integers, the fusion result FL k Perform the floor operation, then set G to the nearest integer. k The low-frequency image obtained after upsampling and FL k-1The Laplace high-frequency details of the current layer are added to restore the Gaussian image of the current layer, and the iteration is continuously carried out until the reconstructed image I is obtained:

[0145] I=G0,G K =FL K ,G k-1 =FL k-1 +EXPAND(G k ),k=1,2...,K;

[0146] In the offline stage of the double-light fusion defect detection method of the wind turbine blade, the LENW-DDPM network is trained by using the small sample data set after fusion expansion, and a LENW-DDPM generation model is obtained. The processing steps of the LENW-DDPM network for input data are as follows:

[0147] (1) LENW-DDPM forward diffusion process. Inject noise in the original data x0, and gradually generate multiple distribution samples x t , which finally tend to the standard Gaussian distribution N(0,1). The probability distribution of the forward process is described as:

[0148]

[0149] In the formula, x t is the noise data of the t-th step, β t is the noise intensity of each step, and N(.) represents the Gaussian distribution.

[0150] (2) LENW-DDPM reverse generation process. Starting from random noise x t , the mean and covariance of the noise predicted by the LENW-Net neural network are used to recover the data x0 step by step through multiple denoising, and finally a sample is generated. The conditional probability distribution of the reverse process is described as:

[0151] p θ (x t-1 |x t )=N(x t-1 ,μ θ (x t ,t),∑ θ (x t ,t));

[0152] In the formula, μ θ (x t ,t) and ∑ θ (x t ,t) are the predicted noise mean and covariance.

[0153] The training objective of the LENW-DDPM network is to minimize the denoising error, and the L2 loss (mean square error) is used to measure the error in each step of the denoising process. The L2 loss function is:

[0154]

[0155] where ∈ is the noise added in the forward process, and ∈ θ is the predicted noise.

[0156] The LENW-DDPM is an improvement of the DDPM, which replaces the backbone network U-Net in the DDPM with a LENW-Net deep neural network. The LENW-Net network structure is shown in Figure 3 , which adopts a two-level W-shaped structure composed of two levels of deep neural networks. The first level of deep neural network includes a symmetric encoder-decoder architecture composed of three layers of downsampling and upsampling, and the second level of deep neural network includes an encoder-decoder architecture composed of two layers of downsampling and upsampling and one layer of nested structure. The processing process of the network for the thermal image or visible light image X of the wind turbine blade is as follows:

[0157] (1) The first level of deep neural network uses two layers of initial convolution to extract shallow features from the input X, preserving spatial details, which is mathematically described as:

[0158]

[0159] where Conv is a 3 × 3 convolution kernel,

[0160] (2) The encoding layer of the first level of deep neural network performs three times of downsampling on X, each time reducing the image resolution by half and doubling the number of channels, gradually compressing the spatial dimension and extracting high-level semantic features. Before downsampling at the maximum resolution layer, an AEB module is used to strengthen the network's learning of local important details:

[0161] XD k = Downsample(AEB(X)), k = 1;

[0162] XD k = Downsample(ResnetBlock(SA(ResnetBlock(XD k-1 )), k = 2, 3;

[0163] where Downsample is downsampling, SA is self-attention mechanism, and k is the number of layers.

[0164] (3) The data XD3 passes through the residual block, SA attention and residual block in turn, and the global feature fusion is performed at the lowest resolution layer:

[0165] XU3 = ResnetBlock(SA(ResnetBlock(XD3)));

[0166] (4) The decoding layer of the first-level deep neural network performs three times of upsampling on XU3, recovers the resolution and the number of channels layer by layer, fuses the features of the corresponding layer of the encoder through the skip connection, and supplements the detailed information. After upsampling at the maximum resolution layer, the AEB module is used to strengthen the learning of local important details by the network:

[0167] XU k = ResnetBlock(SA(ResnetBlock(Upsample(XU k+1 ))00+ XD k , k = 1, 2;

[0168] XT = AEB(Upsample(XU1)) + X;

[0169] where Upsample is upsampling, and k is the number of layers.

[0170] (5) For XT, the residual block, convolution and AEB module are used in turn to strengthen the local details, and the output of the first-level deep neural network is obtained:

[0171] Y1 = AEB(Conv2(σ(Conv1(ResnetBlock(XT))));

[0172] (6) The encoding layer of the second-level deep neural network performs two times of downsampling on Y1, and the resolution of the image is halved and the number of channels is doubled each time, gradually compressing the spatial dimension and extracting high-level semantic features. Before downsampling at the maximum resolution layer, the AEB module is used to strengthen the learning of local important details by the network:

[0173] XD11 = Downsample(AEB(Y1));

[0174] XD12 = Downsample(ResnetBlock(SA(ResnetBlock(XD11))));

[0175] (7) Two parallel processing are performed on XD12: the first processing is that the first decoder of the second deep neural network performs twice upsampling on XD12, fuses the features of the corresponding levels of the encoder through the skip connection, supplements the detail information, and after upsampling at the maximum resolution layer, adopts the AEB module to strengthen the network learning of the local important details, to obtain XU10; the second processing is that global feature fusion is performed on XD12 at the lowest resolution layer to obtain XU22, and then the second decoder of the second deep neural network is used to perform twice upsampling, fuse the features of the corresponding levels of the encoder and the first decoder through the skip connection, supplement the detail information, and after upsampling at the maximum resolution layer, the AEB module is used to strengthen the network learning of the local important details to obtain XU21, and then XU20 is obtained by upsampling. The mathematical description of the above process is as follows:

[0176] XU11 = ResnetBlock(SA(ResnetBlock(Upsample(XD12))) + XD11;

[0177] XU10 = AEB(Upsample(XU11)) + Y1;

[0178] XU22 = ResnetBlock(SA(ResnetBlock(XD12)));

[0179] XU21 = ResnetBlock(SA(ResnetBlock(Upsample(XU22))) + XU11;

[0180] XU20 = AEB(Upsample(XU21)) + XU10;

[0181] (8) For XU20, a residual block and two convolutions are used in sequence to obtain the output of the second deep neural network:

[0182] Y2 = Conv2(σ(Conv1(ResnetBlock(XU20)));

[0183] The input X of the first level network and the output Y2 of the second level network are added element by element to obtain the final output of the network, which is represented as:

[0184] Y = X + Y2;

[0185] The AEB module in LENW-Net has the structure as shown in Figure 4 The processing steps of the input data are as follows:

[0186] (1) Two ResNetBlocks are used to extract preliminary features. Among them, ResNetBlock uses stacked convolution and nonlinear activation function to extract features, strengthens the understanding of local details, provides features for subsequent attention optimization, fuses the time steps of the diffusion process through time embedding, uses the Dropout layer to prevent overfitting, and connects the input and output residual through Conv convolution. The mathematical description of the above process is:

[0187]

[0188] X = ResNetBlock(ResNetBlock(X));

[0189] In the formula, Conv1 and Conv2 are convolution operations, GN1 and GN2 are group normalization operations, σ is SiLU activation, the Dropout dropout rate is 0.1, and Linear(σ(t)) is time embedding TimeEmbedding.

[0190] (2) The LIA attention mechanism is used to process the features extracted by ResNetBlock, enhance local detail features, and suppress background noise:

[0191] LIA(X) = X w G;

[0192] X = LIA(X);

[0193] Among them, LIA is the LIA attention mechanism, G is the gate signal, and w is the weight.

[0194] (3) Two ResNetBlocks are used to extract higher-level features and further refine details:

[0195] X = ResNetBlock(ResNetBlock(X));

[0196] In the offline stage of the double-light fusion defect detection method of the wind turbine blade, the small sample data set after fusion expansion is input into the LENW-DDPM generation model to generate an expanded wind turbine blade defect data set. The double-light fusion wind turbine blade defect detection network is trained using the expanded data set to obtain a double-light fusion wind turbine blade defect detection model. The structure of the double-light fusion wind turbine blade defect detection network is as shown in Figure 5 , which includes the following three branches:

[0197] 1) Visible light branch: a wind turbine blade surface defect detection network CA-YOLO is used to detect the coating wear defects existing on the wind turbine blade.

[0198] 2) Infrared branch: an infrared defect detection network for wind turbine blades based on an adaptive threshold heat map prior is adopted, which is responsible for detecting the defects of wind turbine blades such as missed filling and resin-rich defects;

[0199] 3) Visible light / infrared fusion branch: a dual-light fusion wind turbine blade defect detection network DB-YOLOv8 based on a dual-channel YOLOv8 and a bidirectional pyramid fusion strategy is adopted, which is responsible for detecting the defects of wind turbine blades such as cracks and scratches.

[0200] A wind turbine blade surface defect detection network CA-YOLO in the dual-light fusion wind turbine blade defect detection network, the structure of which is shown in Figure 6 YOLOv8 as the baseline algorithm, the following improvements are made:

[0201] (1) The C2f-CBAM module is used to replace the C2f module in the Neck part of the YOLOv8 network.

[0202] (2) A multi-task learning framework is introduced. The model can simultaneously perform instance segmentation of defects and target detection of the position of defects in a leaf under one network structure. On the one hand, instance segmentation can accurately outline the defect area and obtain more detailed defect contour information to further analyze the morphological features of the defect, such as the length of the crack and the area of the coating wear area. On the other hand, target detection is used to identify the leaf part (blade tip, blade middle, blade root) where the defect is located, so as to combine the defect information with the quality impact of different regions of the leaf and improve the evaluation ability of the overall health status.

[0203] (3) Improve the loss function. According to the characteristics of the multi-task in the CA-YOLO network, the loss function of YOLOv8 is redesigned, and a multi-task optimization loss function is constructed according to the weight adjustment based on uncertainty:

[0204]

[0205] In the formula, σ1 and σ2 represent the uncertainty parameters of the detection task and the segmentation task respectively, which are learnable weight control items used to adaptively balance the relative contributions of each sub-task in joint optimization, L seg is the segmentation loss, L det is the detection loss.

[0206]

[0207] L det = α1L cls + α2L iou + α3L dfl ;

[0208] In the formula, L clsFor the classification loss, L iou For the bounding box regression loss, L dfl For the discretized bounding box regression loss, L

[0209] One of the dual-light fusion wind turbine blade defect detection networks is an infrared defect detection network of a wind turbine blade based on an adaptive threshold heat map prior, which improves the loss function of the YOLOv8 network according to the infrared image prior information, and the specific process is as follows:

[0210] (1) A local mean dynamic threshold adjustment method is used to calculate the dynamic threshold, and the calculation formula is:

[0211] T(x,y) = mu(x,y) + C;

[0212] In the formula, mu(x,y) represents the average temperature value of all pixels in the local window centered on pixel (x,y), and C is an offset for controlling the sensitivity of segmentation.

[0213] (2) According to the dynamic threshold T(x,y), the collected heat map image is binarized to generate a defect mask image, and the mask image is:

[0214]

[0215] (3) Calculate the pixel weight factor:

[0216] W = 1 + (lambda f -1)M(x,y)

[0217] In the formula, lambda f >=1 controls the weighting intensity.

[0218] (4) The pixel weight factor W is used to optimize the YOLOv8 loss function:

[0219]

[0220] In the formula,

[0221] One of the dual-light fusion wind turbine blade defect detection networks is a dual-light fusion wind turbine blade defect detection network DB-YOLOv8, and its structure is shown in Figure 7 The processing steps of the visible light image and the infrared image are as follows:

[0222] (1) Two YOLOv8 networks are used to extract defect features in the visible light image and defect features in the infrared image, respectively, to obtain two types of multi-scale features.

[0223] (2) A bidirectional cross-pyramid fusion module BCPF is used to bidirectionally fuse the multiscale features of the visible light image and the infrared image at different levels. The structure of the BCPF is shown in Figure 8 The BCPF is composed of two parallel pyramid feature extraction branches, which process the visible light image and the infrared image respectively. Each branch outputs multiscale feature maps P1-P5 of different levels of semantic and spatial information. After the fusion of P1-P5 of the two branches, the fusion features are sequentially processed by upsampling modules and C2f convolution to output fusion features of three scales.

[0224] (3) The fused features are added as supplementary information to the two YOLOv8 networks respectively, and further feature extraction is performed to generate detection results of three different scales. Through the bidirectional feature pyramid strategy, different modalities can interact deeply at the same scale, improving the detection accuracy and robustness of crack and scratch defects.

[0225] In the online stage of the dual-light fusion defect detection method for wind turbine blades, the following steps are included:

[0226] (1) Use the MV-GE31GC industrial camera and the TN460 infrared thermal imager to collect wind turbine blade defect data. The collected infrared thermal images with a size of 640x512 and visible light images with a size of 752x480 are transformed to 640x480 for ease of fusion.

[0227] (2) Input the visible light image and the infrared thermal image with the same scale into the dual-light fusion wind turbine blade defect detection model obtained in the offline stage, and use three branches to detect the two types of input data.

[0228] (3) As shown in Figure 5 , in the output selection link, the coating wear defect detection results of the visible light branch (if any), the missed filling and rich resin defect detection results of the infrared branch (if any), and the crack and scratch defect detection results of the visible light / infrared fusion branch (if any) are outputted, and the corresponding defect areas are segmented in the original image according to the segmentation results outputted by different branches.

[0229] Application example:

[0230] In order to verify the effectiveness of the present application, the method of the present application is next experimented on a self-collected data set (referred to as IMUTWD) and a Cifar10 public data set. The CPU used in the experiment is Intel(R) Xeon(R) Platinum 8352V, the GPU is NVIDIA GeForce RTX 4090 24GB, the CUDA version is 11.8, the software development uses the PyTorch 2.4.0 deep learning framework, and the programming language uses Python 3.10. The learning rate of the LENW-DDPM network training is 1e-4, the iteration number is 5000 rounds, the structural similarity measurement (SSIM), the feature mutual information (FMI), the peak signal-to-noise ratio (PSNR) and the correlation coefficient (CC) are used to evaluate the image fusion algorithm, and the FID (Frechet inception distance), the IS (Inception score) and the KID (Kernel Inception Distance) are used to evaluate the generative model.

[0231] 1. Data fusion expansion algorithm experiment:

[0232] The adaptive Laplacian pyramid image fusion (ALPIF) of the present application and the traditional Laplacian pyramid image fusion (LPIF) are respectively used to fuse and expand the IMUTWD data set, the fusion result of the infrared image is as shown in Figure 9 , the fusion result of the visible light image is as shown in Figure 10 , and the performance indicators of the two fusion algorithms are shown in Table 1.

[0233] Table 1 Performance indicators of image fusion algorithm

[0234]

[0235] From Figure 9 , Figure 10 and Table 1, it can be seen that the adaptive Laplacian pyramid algorithm expands the details and structural information of the image more comprehensively, and the fusion effect is better.

[0236] 2. AEB module effectiveness experiment:

[0237] The improved DDPM algorithm with no attention mechanism, increased LIA attention mechanism, CBAM+ResNetBlock, SA+ResNetBlock, LAE+ResNetBlock and LIA+ResNetBlock (AEB) is respectively used to generate images on the Cifar-10 data set, and the evaluation indicators of the generated images are shown in Table 2. From Table 2, it can be seen that the AEB module has the best indicators, and can effectively improve the quality of the generated images.

[0238] Table 2AEB module effectiveness experiment results

[0239]

[0240] 3. LENW-Net effectiveness experiment:

[0241] The mainstream networks U-Net++L 3 , U-Net, StackedU-Net and the LENW-Net of the present application are respectively used as the backbone network of DDPM to perform a generated image experiment on the Cifar-10 dataset, and the evaluation indexes of the generated image are shown in Table 3. As shown in Table 3, the performance indexes of the LENW-Net are all optimal, and the performance of the DDPM is improved the most.

[0242] Table 3 LENW-Net effectiveness experiment results

[0243]

[0244] 4. Comparative experiment of different generative model algorithms:

[0245] The mainstream generation algorithms DCGAN, DDIM and the LENW-DDPM of the present application are respectively used to perform a generated image experiment on the IMUTWD dataset and the Cifar-10 dataset, and the generated image results are shown in Figure 11 , and the evaluation indexes of the generated image are shown in Table 4 and Table 5. As shown in Figure 11 , Table 4 and Figure 5 , the LENW-DDPM has the best detail performance of the generated graph and the highest generation quality.

[0246] Table 4 IMUTWD dataset evaluation index score table

[0247]

[0248] Table 5 CIFAR10 dataset evaluation index score table

[0249]

[0250] 5. Single-light-based target detection experiment:

[0251] The traditional YOLOv8 and the CA-YOLO of the present application are respectively used to perform a target detection experiment on the Nordtank visible light dataset, and the algorithm performance indexes are shown in Table 6. As shown in Table 6, the CA-YOLO significantly improves the performance of YOLOv8 in the wind turbine blade defect detection task.

[0252] Table 6 Target detection experiment results based on visible light images

[0253]

[0254] YOLOv8 and the wind turbine blade infrared defect detection network based on the adaptive threshold heat map prior of the application (heat map prior YOLOv8) are respectively used for defect detection experiments, the IMUTWD dataset is used for experiments, and algorithm performance indicators are shown in Table 7. As shown in Table 7, the heat map prior YOLOv8 significantly improves the performance of YOLOv8 in the wind turbine blade defect detection task.

[0255] Table 7 Comparison of indicators of heat map prior and basic algorithm

[0256]

[0257]

[0258] 6. Wind turbine blade defect detection comparison experiment based on dual light fusion:

[0259] YOLOv8 based on visible light image, YOLOv8 based on infrared image, simple fusion dual-channel YOLOv8 and DB-YOLOv8 proposed in the application are respectively used for defect detection experiments, and scratch and crack defect images in the IMUTWD dataset are selected for defect detection experiments, and performance indicators of different algorithms are shown in Table 8. As shown in Table 8, DB-YOLOv8 performs better in the wind turbine blade scratch and crack defect detection task.

[0260] Table 8 Indexes of dual light fusion comparison algorithm

[0261]

[0262] In comparison with the current mainstream dual-spectrum fusion detection network (such as CFR_3, CFT and GAFF), scratch and crack defect images in the IMUTWD dataset are selected for defect detection experiments, and performance indicators of different algorithms are shown in Table 9. As shown in Table 9, the performance of the DB-YOLOv8 algorithm is the best, and the model size is smaller.

[0263] Table 9 Index comparison of dual-spectrum fusion detection network

[0264]

[0265] The application covers any substitution, modification, equivalent method and scheme made on the essence and scope of the application. In order to make the public have a thorough understanding of the application, specific details are described in the following preferred embodiments of the application, and the application can also be fully understood without the description of these details. In addition, in order to avoid unnecessary confusion to the essence of the application, well-known methods, processes, procedures, elements and circuits are not described in detail.

[0266] The above merely preferred embodiments of the present application, it should be noted that for those of ordinary skill in the art, without departing from the principles of the present application, can also be made several improvements and refinements, these improvements and refinements should also be considered as the scope of protection of the present application.

Claims

1. A small sample wind turbine blade dual-optical fusion defect detection method based on a generative dataset, characterized in that, The method comprises the following steps: S1, collecting and fusing small sample wind turbine blade thermal images and visible light images in the offline stage: using an infrared thermal imager and a camera to collect wind turbine blade defect thermal images and visible light images, using an adaptive Laplacian pyramid image fusion method to fuse and expand the original wind turbine blade defect thermal images and visible light images respectively, and generating a defect image dataset; S2, generating wind turbine blade defect thermal image and visible light image datasets in the offline stage: using the expanded wind turbine blade defect image dataset to train a LENW-DDPM generation model, using the LENW-DDPM generation model to generate wind turbine blade defect thermal image and visible light image datasets, and dividing them into a training set, a validation set and a test set; S3, training and optimizing a dual-light fusion defect detection model: in the offline stage, using the training set of wind turbine blade defect thermal images and visible light images to train a dual-light fusion wind turbine blade defect detection network, obtaining a dual-light fusion wind turbine blade defect detection model, using the validation set to optimize the dual-light fusion wind turbine blade defect detection model, and using the test set to evaluate the performance of the detection model; S4, online defect detection and output result analysis: in the online stage, using an infrared thermal imager and a camera to collect wind turbine blade thermal images and visible light images, using a dual-light fusion wind turbine blade defect detection model to detect defects of the wind turbine blade, outputting the defect type, and segmenting the defect area in the original image.

2. The method of claim 1, wherein the method is a small sample wind turbine blade dual-optical fusion defect detection method based on a generative dataset. The adaptive Laplacian pyramid image fusion method in S1 comprises: Dynamic mask map generation strategy: in each iteration cycle of the Laplacian pyramid image fusion algorithm, according to the characteristics of two to-be-fused heat map images (or visible light images) I A , I B , a dynamic mask map M d including feature information and feature direction information is generated by using a dynamic mask generation strategy. Mask image Gaussian pyramid construction: the dynamic mask image M d As the mask image of the Laplacian pyramid image fusion algorithm, the Gaussian blur and down-sampling are used to generate K layers of mask Gaussian pyramid Image Laplacian pyramid construction: I A and I B As the image to be fused in the Laplacian pyramid image fusion algorithm, K layers of image Gaussian pyramid are generated by Gaussian blur and down-sampling and generate the Laplacian pyramid of each layer is expressed as: where EXPAND(G k ) is an up-sampling of the k-th layer of the Gaussian pyramid at the resolution of the k-1 layer. Multi-level feature map fusion: fusion according to levels on Laplacian pyramid, using mask Gaussian pyramid Control the proportion of two Laplacian pyramids , expressed as: Wherein, int is the integer function; Image reconstruction and final map generation: G k The low frequency image obtained after upsampling and FL k-1 Add Laplacian high frequency details of the current layer, restore the Gaussian image of the current layer, iterate until the reconstructed image I is obtained, represented as: I = G0, G K = FL K , G k-1 = FL k-1 + EXPAND(G k ), k = 1, 2,..., K.

3. The small sample wind turbine blade dual-optical fusion defect detection method based on a generative dataset according to claim 2, characterized in that, The dynamic mask generation strategy comprises: Generate feature mask: cut ResNet50, remove its fully connected layer and global average pooling layer, use the cut ResNet50 to extract the deep feature maps F1, F2 of the two images I A , B , calculate the difference between F1 and F2 and perform normalization processing to obtain a feature mask map M f containing feature differences, represented as: D = ||ResNet(I A ) - ResNet(I B )||2; Generating a direction mask with direction information: using Sobel operators to calculate horizontal, vertical and diagonal gradients of the feature mask map, and generating a direction mask M according to the average gradient of each direction o , is expressed as: S x = Sobel x (M f ), S y = Sobel y (M f ), S xy = Sobel xy (M f ); wherein, H and W are the height and width of the feature map, respectively. Dynamic mask generation: the feature mask map and the direction mask are weighted and fused to generate a dynamic mask M including feature map information and feature direction information d is represented as: M d = a x M f + (1 - a) x M o , a e [0, 1] Wherein, a is a weight parameter.

4. The small sample wind turbine blade dual-optical fusion defect detection method based on a generative dataset according to claim 3, characterized in that, The LENW-DDPM generation model comprises: Generation model structure definition: replace the U-net backbone network in DDPM with a LENW-Net deep neural network; Forward diffusion process modeling: inject noise into the defect image dataset, gradually generate multiple distribution samples, and gradually approach the standard Gaussian distribution to form a forward diffusion path; Reverse generation process modeling: from random noise x t Initially, the noise added in the forward diffusion process is predicted by training a LENW-Net deep neural network; Image reconstruction: on noisy images x t Reverse denoising, respectively generate wind turbine blade defect thermal image and visible light image samples, and finally build defect thermal image dataset and visible light image dataset.

5. The small sample wind turbine blade dual-optical fusion defect detection method based on the generative dataset according to claim 4, characterized in that, The LENW-Net deep neural network adopts a two-stage W-shaped structure and is composed of two stages of deep neural networks. The first stage of deep neural network comprises a symmetric encoder-decoder architecture composed of three layers of down-sampling and up-sampling. The second stage of deep neural network comprises an encoder-decoder architecture of two layers of down-sampling, up-sampling and one layer of nested structure. The processing of the LENW-Net deep neural network on the defect thermal image or visible light image X of the wind turbine blade comprises: The first stage of deep neural network adopts two layers of initial convolution to extract shallow layer features from the input X, and retains spatial details, represented as: X=Conv2(σ(Conv1(X))); Wherein, Conv is a 3*3 convolution kernel, and sigma is a Sigmoid activation function; The encoding layer of the first stage of deep neural network performs three times of down-sampling on X, and the image resolution is halved and the channel number is doubled each time, gradually compressing the spatial dimension and extracting high-level semantic features. Before down-sampling at the maximum resolution layer, an AEB module is used to strengthen the learning of local important details by the network, represented as: XD k = Downsample(AEB(X)), k = 1; XD k = Downsample(ResnetBlock(SA(ResnetBlock(XD k-1 )), k = 2, 3; wherein Downsample is a down-sampling, SA is a self-attention mechanism, and k is a number of layers; The data XD3 is sequentially subjected to a residual block, SA attention, and a residual block, and global feature fusion is performed at the lowest resolution layer, and is expressed as: XU3 = ResnetBlock(SA(ResnetBlock(XD3))); The decoding layer of the first-level deep neural network performs three up-samplings on XU3, restores the resolution and the number of channels layer by layer, fuses the features of the corresponding layers of the encoder through a skip connection, supplements the detail information, and after up-sampling at the maximum resolution layer, uses an AEB module to strengthen the learning of local important details by the network, and is expressed as: XU k = ResnetBlock(SA(ResnetBlock(Upsample(XU k+1 )))) + XD k k = 1, 2; XT = AEB(Upsample(XU1))+X; wherein Upsample is an up-sampling, and k is a number of layers; For XT, a residual block, a convolution, and an AEB module are sequentially used to strengthen the local details, and the output of the first-level deep neural network is obtained, and is expressed as: Y1 = AEB(Conv2(σ(Conv1(ResnetBlock(XT)))); The encoding layer of the second-level deep neural network performs two down-samplings on Y1, and the resolution of the image is halved and the number of channels is doubled each time, gradually compresses the spatial dimension, extracts high-level semantic features, and before down-sampling at the maximum resolution layer, an AEB module is used to strengthen the learning of local important details by the network, and is expressed as: XD11 = Downsample(AEB(Y1)); XD12 = Downsample(ResnetBlock(SA(ResnetBlock(XD11)))); XD12 is subjected to two parallel processes, wherein the first process is that the first decoder of the second-level deep neural network performs two up-samplings on XD12, fuses the features of the corresponding layers of the encoder through a skip connection, supplements the detail information, and after up-sampling at the maximum resolution layer, uses an AEB module to strengthen the learning of local important details by the network, and obtains XU10; the second process is that global feature fusion is performed on XD12 at the lowest resolution layer to obtain XU22, which is subjected to two up-samplings by the second decoder of the second-level deep neural network, fuses the features of the corresponding layers of the encoder and the first decoder through a skip connection, supplements the detail information, and after up-sampling at the maximum resolution layer, uses an AEB module to strengthen the learning of local important details by the network, and obtains XU21, which is up-sampled to obtain XU20, and is expressed as: XU11 = ResnetBlock(SA(ResnetBlock(Upsample(XD12))))+XD11; XU10 = AEB(Upsample(XU11))+Y1; XU22 = ResnetBlock(SA(ResnetBlock(XD12))); XU21 = ResnetBlock(SA(ResnetBlock(Upsample(XU22))))+XU11; XU20 = AEB(Upsample(XU21))+XU10; For XU20, a residual block and two convolutions are sequentially used for processing to obtain the output of the second-level deep neural network, denoted as: Y2=Conv2(σ(Conv1(ResnetBlock(XU20))). The final output of the network is obtained by element-wise addition of the input X of the first-level network and the output Y2 of the second-level network, denoted as: Y = X + Y 2。 6. The small sample wind turbine blade dual-optical fusion defect detection method based on a generative dataset according to claim 5, characterized in that, The AEB module comprises: Two ResNetBlocks are used to extract preliminary features, wherein the ResNetBlock uses stacked convolution and nonlinear activation function to extract features, strengthens the understanding of local details, fuses the time steps of the diffusion process through time embedding, uses the Dropout layer to prevent overfitting, and connects the input and output residual through Conv convolution, denoted as: X=ResNetBlock(ResNetBlock(X)); Wherein, Conv1, Conv2 are convolution operations, GN1, GN2 are group normalization operations, sigma is SiLU activation, Dropout is dropout rate, Linear(sigma(t)) is time embedding projection TimeEmbedding; The LIA attention mechanism is used to process the features extracted by the ResNetBlock, enhance the local detail features, and suppress the background noise, denoted as: LIA(X)=X·w·G; X=LIA(X); Wherein, LIA is the LIA attention mechanism, G is the gate signal, and w is the weight; Two ResNetBlocks are used to extract higher-level features to further refine the details, denoted as: X=ResNetBlock(ResNetBlock(X)).

7. The small sample wind turbine blade dual-optical fusion defect detection method based on a generative dataset according to claim 6, characterized in that, The dual-light fusion wind turbine blade defect detection network in S3 comprises a visible light branch, an infrared branch and a visible light / infrared fusion branch. According to the defect detection capabilities of the visible light branch, the infrared branch and the visible light / infrared fusion branch, the defect detection results of the visible light branch, the infrared branch and the visible light / infrared fusion branch are selected and output through an output selection link. The visible light branch detects the coating wear defect of the wind turbine blade based on CA-YOLO. The infrared branch detects the omissions and rich resin defects of the wind turbine blade based on the wind turbine blade infrared defect detection network with adaptive threshold heat map prior. The visible light / infrared fusion branch detects the cracks and scratches defects of the wind turbine blade based on the dual-light fusion wind turbine blade defect detection network DB-YOLOv8 with dual-channel YOLOv8 and bidirectional pyramid fusion strategy.

8. The small sample wind turbine blade dual-optical fusion defect detection method based on a generative dataset according to claim 7, characterized in that, The CA-YOLO comprises: The C2f-CBAM module is used to replace the C2f module of the Neck part of the YOLOv8 network; A multi-task learning framework is introduced to simultaneously perform the instance segmentation of defects and the target detection of the position of defects in one network structure; According to the weight adjustment based on uncertainty, a multi-task optimization loss function is constructed, denoted as: where σ1and σ2represent the uncertainty parameters of the detection task and the segmentation task, respectively, L seg is the segmentation loss, det is the detection loss; L det = a1L cls + a2L iou + a3L dfl ; where L cls is the classification loss, L iou is the bounding box regression loss, L dfl is the discretized bounding box regression loss.

9. The small sample wind turbine blade dual-optical fusion defect detection method based on a generative dataset according to claim 8, characterized in that, The wind turbine blade infrared defect detection network with adaptive threshold heat map prior comprises: A local mean dynamic threshold adjustment method is used to calculate the dynamic threshold, which is represented as: T(x,y) = mu(x,y) + C; Wherein, mu(x,y) represents the average temperature value of all pixels in the local window centered on pixel(x,y), and C is the offset; According to the dynamic threshold T(x,y), the collected thermal image is binarized to generate a defect mask image, and the mask image is: The pixel weight factor is calculated, which is represented as: W = 1 + (λ f -1)M(x,y); where λ f ≥1 is a control weighting strength; The pixel weight factor W is used to optimize the YOLOv8 loss function: wherein 10. The small sample wind turbine blade dual-optical fusion defect detection method based on a generative dataset according to claim 9, characterized in that, The dual-light fusion wind turbine blade defect detection network DB-YOLOv8 includes: Two YOLOv8 networks are used to extract defect features in the visible light image and defect features in the infrared image respectively, and two types of multi-scale features are obtained; A bidirectional cross pyramid fusion module BCPF is used to bidirectionally fuse the two types of multi-scale features at different levels, wherein the BCPF is composed of two parallel pyramid feature extraction branches, which process the visible light image and the infrared image respectively, each branch outputs multi-scale feature maps of different levels of semantic and spatial information, and after the multi-scale feature maps of the two branches are fused, they are sequentially processed by an upsampling module and a C2f convolution, and three scales of fusion features are outputted; The fusion features are added as supplementary information to the two YOLOv8 networks for feature extraction, and three different scale detection results are generated.

Images