Multi-scale feature extraction system, method and fan blade defect detection method

Abstract

The application relates to a multi-scale feature extraction system which comprises a multi-layer feature extraction module, a multi-layer upper feature fusion module and a multi-layer lower feature fusion module; the multi-layer feature extraction module comprises N first convolutional layers which are connected in sequence; the multi-layer upper feature fusion module comprises N-2 upper sampling layers which are connected in sequence; the multi-layer lower feature fusion module comprises N-2 down sampling layers which are connected in sequence; the Nth first convolutional layer is connected with the 1st upper sampling layer through a second convolutional layer, and the N-2th upper sampling layer is connected with the 1st down sampling layer; a residual module is arranged at the output end of each upper sampling layer and the output end of each down sampling layer. The application realizes the extraction and fusion of multi-scale features, overcomes the technical defects that the existing method based on the image scale of a fan blade cannot realize accurate detection and positioning for small-scale defects, and thus cannot comprehensively recognize the defects of the fan blade.

Description

Technical Field

[0001] This invention relates to the field of wind turbine inspection technology, and in particular to a multi-scale feature extraction system and method, and a wind turbine blade defect detection method and system. Background Technology

[0002] Wind energy is a clean, low-carbon, and renewable energy source. Countries worldwide attach great importance to wind power technology, and the installed capacity of wind turbines is continuously increasing. In the future, wind power will play a vital role in my country's clean energy system. In recent years, wind power generation has developed rapidly. The blades are crucial components for energy conversion, not only transforming wind energy into mechanical energy but also serving as the main load-bearing components of the wind turbine, playing a key role in the safe operation of the entire turbine. However, the harsh environment and complex, variable climate conditions in which wind turbines operate lead to serious defects on the blade surface, such as gel coat peeling, cracks, erosion, and lightning strikes. These defects affect the lifespan of the wind turbine and its power generation, and in severe cases, can cause accidents such as tower collapse.

[0003] Traditional blade inspection is mostly done manually, which has problems such as low efficiency, high cost, strong subjectivity and high risk. Real-time, high efficiency and accuracy of detection have become the basic requirements for wind turbine blade defect identification, and scholars from various countries have conducted in-depth research on the above problems. In the literature - Bo Z, Yanan Z, Changzheng C. Acoustic emission detection of fatigue cracks in wind turbine blades based on blind deconvolution separation. Fatigue & Fracture of Engineering Materials & Structures, 2017, 40(6), a method based on blind deconvolution separation algorithm for acoustic emission signal analysis was proposed to identify surface crack damage of wind turbine blades. In the literature - Li Xiaowei. Research on wind turbine blade detection based on fiber optic grating sensing technology. Nanjing University of Aeronautics and Astronautics, 2013, a static detection experiment of glass fiber composite material was carried out using fiber optic grating sensor, and a damage identification method based on wavelet packet energy spectrum algorithm was proposed. In the literature - Long W, Zijun Z. Automatic Detection of Wind Turbine Blade Surface Cracks Based on UAV-Taken Images. IEEE Transactions on Industrial Electronics, 2017, 64(9), an electromagnetic ultrasonic testing technology is proposed. The electromagnetic ultrasonic transducer is established using the finite element simulation software COMSOL, and various blade models are subjected to electromagnetic ultrasonic testing to obtain stress cloud diagrams and waveform diagrams. Finally, the defective blade waveform is compared with the undamaged blade waveform to analyze the blade defect situation.

[0004] In recent years, target recognition based on deep convolutional neural networks has achieved good results. In the literature, Kang Shuang, Chen Changzheng, Zhou Bo, et al. Research on Infrared Detection of Wind Turbine Blade Defects Based on Temperature Threshold. Acta Energiae Solaris Sinica, 2020, 41(08):337-341, a method based on temperature threshold features for wind turbine blade defect identification is proposed. Experiments using temperature data analysis and processing demonstrate the effectiveness of this method in detecting defects in high-altitude wind turbine blades. In the literature, Yu Y, Cao H, Yan X, et al. Defect identification of wind turbine blades based on defectsemantic features with transfer feature extractor. Neurocomputing, 2020, 376, a wind turbine blade identification method based on defect semantic features is proposed. This method involves training on labeled blade images and selecting the first n layers of a well-trained DCNN as a transfer feature extractor to extract defect semantic features. The literature Zhu Jiawei, Wen Chuanbo. Wind turbine blade defect detection based on improved SSD. Composite Materials Science and Engineering, 2022(03):38-44, proposes to detect defects in wind turbine blades by improving the SSD algorithm. Finally, experiments show that the proposed method has significantly improved detection accuracy and efficiency compared with traditional detection methods. The literature Qu Zhongkan, Li Xuefeng, Li Fengjun, et al. Design of multi-point detection system for wind turbine blade defect images based on Faster R-CNN. Electronic Design Engineering, 2021, 29(04):57-61, proposes the design of a multi-point detection system for gel coat peeling defects based on Faster R-CNN. It optimizes the system using two improvement methods, DCNv2 and Giou, and verifies that the proposed algorithm can significantly improve the detection accuracy of gel coat peeling defects.

[0005] While the above research methods have achieved certain results, they are all based on the image scale of wind turbine blades for detection, which cannot accurately detect and locate small-scale defects, making it difficult to comprehensively identify defects in wind turbine blades. Summary of the Invention

[0006] In view of this, the present invention provides a multi-scale feature extraction system, method and wind turbine blade defect detection method and system to achieve accurate detection and localization of small-scale defects and to achieve comprehensive defect identification of wind turbine blades.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] A multi-scale feature extraction system, comprising: a multi-layer feature extraction module, a multi-layer upper feature fusion module, and a multi-layer lower feature fusion module;

[0009] The multi-layer feature extraction module includes N sequentially connected first convolutional layers;

[0010] The multi-layer up-feature fusion module includes N-2 sequentially connected upsampling layers;

[0011] The multi-layer downsampling module includes N-2 sequentially connected downsampling layers;

[0012] The Nth first convolutional layer is connected to the first upsampling layer through the second convolutional layer, and the (N-2)th upsampling layer is connected to the first downsampling layer;

[0013] Each output terminal of the upsampling layer and each output terminal of the downsampling layer are provided with a residual module;

[0014] The residual module set at the output of the nth upsampling layer is also connected to the output of the Nnth first convolutional layer through a second convolutional layer; n = 1, ..., N-2;

[0015] The residual module located at the output of the m-th downsampling layer is also connected to the input of the N-1-m-th upsampling layer through a third convolutional layer; m = 1, ..., N-2.

[0016] Optionally, the residual module includes: a feature fusion layer, a BN layer, a LeakyReLU activation function layer, and a CA attention mechanism module connected in sequence.

[0017] Optionally, the CA attention mechanism module includes: a coordinate information embedding module and a coordinate attention generation module connected in sequence.

[0018] A multi-scale feature extraction method, the extraction method being based on the aforementioned multi-scale feature extraction system, the extraction method comprising the following steps:

[0019] The input image is subjected to multiple convolutional downsampling operations through multiple first convolutional layers of the multi-layer feature extraction module to obtain multiple first feature maps of different scales;

[0020] Multiple upsampling layers and residual modules of the multi-layer upsampling feature fusion module are used to upsample and fuse multiple first feature maps of different scales to obtain multiple second feature maps of different scales.

[0021] Multiple downsampling layers and residual modules in the multi-layer feature fusion module are used to downsample and fuse multiple second feature maps of different scales to obtain multiple third feature maps of different scales.

[0022] Optionally, the step of upsampling and fusing multiple first feature maps of different scales through multiple upsampling layers and a residual module of the multi-layer upsampling feature fusion module to obtain multiple second feature maps of different scales specifically includes:

[0023] The nth second feature map is upsampled by the nth upsampling layer to obtain the feature map output by the nth upsampling layer; n = 1, ..., N-2; where the first second feature map is the feature map obtained by convolving the first feature map output by the Nth upsampling layer through the second convolutional layer;

[0024] The Nn-th first convolution operation feature map is obtained by fusing the feature map output by the nth upsampling layer with the feature map output by the nth upsampling layer through the residual module set at the output of the nth upsampling layer; the Nn-th first convolution operation feature map is the feature map obtained by convolving the first feature map output by the Nn-th first convolution layer through the second convolution layer.

[0025] Optionally, the step of downsampling and fusing multiple second feature maps of different scales through multiple downsampling layers and a residual module of the multi-layer feature fusion module to obtain multiple third feature maps of different scales specifically includes:

[0026] The m-th third feature map is downsampled by the m-th downsampling layer to obtain the feature map output by the m-th downsampling layer; m = 1, ..., N-2; where the first third feature map is the N-2th second feature map;

[0027] The residual module set at the output of the m-th downsampling layer fuses the Nm-1-th second convolution operation feature map with the feature map output by the m-th downsampling layer to obtain the (m+1)-th third feature map; the Nm-1-th second convolution operation feature map is the feature map obtained after performing a convolution operation on the Nm-1-th second feature map through the third convolution layer.

[0028] A method for detecting defects in wind turbine blades, the method being based on a Faster R-CNN wind turbine blade defect detection model; the Faster R-CNN wind turbine blade defect detection model includes: a region proposal network, a target detection network, and the aforementioned multi-scale feature extraction system, wherein the multi-scale feature extraction system is connected to the region proposal network and the target detection network respectively, and the region proposal network is connected to the target detection network.

[0029] The detection method includes the following steps:

[0030] The image of the wind turbine blades to be detected is input into a multi-scale feature extraction system for feature extraction, resulting in multiple third feature maps at different scales.

[0031] Multiple third feature maps of different scales are input into the region proposal network to extract regions of interest, thus obtaining regions of interest corresponding to third feature maps of different scales;

[0032] By inputting third feature maps of different scales and the corresponding regions of interest of the third feature maps of different scales into the target detection network, the location and type of defects in the wind turbine blades can be determined.

[0033] Optionally, the step of inputting the image of the wind turbine blades to be detected into the aforementioned multi-scale feature extraction system for feature extraction to obtain multiple third feature maps at different scales further includes:

[0034] We used drone aerial photography technology to collect sample images of wind turbine blades in wind farms, and used the labelimg tool to annotate the sample images to construct an initial sample set.

[0035] The initial sample set is subjected to data augmentation and Gaussian denoising to obtain the processed sample set;

[0036] The processed sample set was used to train the Faster R-CNN-based wind turbine blade defect detection model.

[0037] A wind turbine blade defect detection system is provided, the system being based on the Faster R-CNN wind turbine blade defect detection model; the Faster R-CNN wind turbine blade defect detection model includes: a region proposal network, a target detection network, and the aforementioned multi-scale feature extraction system, wherein the multi-scale feature extraction system is connected to the region proposal network and the target detection network respectively, and the region proposal network is connected to the target detection network.

[0038] The detection system includes:

[0039] The feature extraction module is used to input the image of the wind turbine blades to be detected into the multi-scale feature extraction system for feature extraction, and obtain multiple third feature maps at different scales.

[0040] The Region of Interest (ROI) extraction module is used to input multiple third feature maps of different scales into the Region Proposal Network (RPN) to extract the ROI, thus obtaining the ROI corresponding to the third feature maps of different scales.

[0041] The target detection module is used to input third feature maps of different scales and the regions of interest corresponding to the third feature maps of different scales into the target detection network to determine the location and type of defects in the wind turbine blades.

[0042] Optionally, the detection system further includes:

[0043] The initial sample set construction module is used to collect sample images of wind turbine blades in wind farms using drone aerial photography technology, and to annotate the sample images using the labelimg tool to construct the initial sample set;

[0044] The data processing module is used to perform data augmentation and Gaussian denoising on the initial sample set to obtain the processed sample set.

[0045] The training module is used to train the Faster R-CNN-based wind turbine blade defect detection model using the processed sample set.

[0046] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0047] This invention discloses a multi-scale feature extraction system, comprising: a multi-layer feature extraction module, a multi-layer upper feature fusion module, and a multi-layer lower feature fusion module. The multi-layer feature extraction module includes N sequentially connected first convolutional layers. The multi-layer upper feature fusion module includes N-2 sequentially connected upsampling layers. The multi-layer lower feature fusion module includes N-2 sequentially connected downsampling layers. The Nth first convolutional layer is connected to the first upsampling layer via a second convolutional layer, and the N-2th upsampling layer is connected to the first downsampling layer. A residual module is provided at the output of each upsampling layer and the output of each downsampling layer. This invention extracts and fuses multi-scale features based on the multi-layer feature extraction module, the multi-layer upper feature fusion module, and the multi-layer lower feature fusion module, overcoming the technical shortcomings of existing methods that detect wind turbine blade defects based on image scale. These methods cannot accurately detect and locate small-scale defects, thus hindering comprehensive defect identification of wind turbine blades. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the structure of a multi-scale feature extraction system provided in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the CA attention mechanism module provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of the Faster R-CNN wind turbine blade defect detection model provided in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] This invention provides a multi-scale feature extraction system, method, and wind turbine blade defect detection method and system to achieve accurate detection and localization of small-scale defects and to achieve comprehensive defect identification of wind turbine blades.

[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example 1

[0056] Embodiment 1 of the present invention provides a multi-scale feature extraction system, such as... Figure 1 As shown ( Figure 1 The value of N is typically 5. The multi-scale feature extraction system includes: a multi-layer feature extraction module, a multi-layer upper feature fusion module, and a multi-layer lower feature fusion module; the multi-layer feature extraction module includes N sequentially connected first convolutional layers (…). Figure 1 The multi-layer upsampling module includes N-2 sequentially connected upsampling layers (Conv1_x, Conv2_x, Conv3_x, Conv4_x, Conv5_x). Figure 1 Upsample); the multi-layer downsampling module includes N-2 sequentially connected downsampling layers (in the Upsample); Figure 1 Downsample in (the process); the Nth first convolutional layer passes through the second convolutional layer ( Figure 1 The Con2d (1×1) is connected to the first upsampling layer, and the (N-2)th upsampling layer is connected to the first downsampling layer; each output of the upsampling layer and each output of the downsampling layer is provided with a residual module; the residual module at the output of the nth upsampling layer is also connected to the output of the Nnth first convolutional layer through a second convolutional layer; n = 1, ..., N-2; the residual module at the output of the mth downsampling layer is also connected to the input of the (N-1-m)th upsampling layer through a third convolutional layer ( Figure 1 Connect Con2d (3×3); m = 1, ..., N-2.

[0057] like Figure 1The residual module consists of a feature fusion layer, a BN layer, a LeakyReLU activation function layer, and a CA attention mechanism module connected in sequence.

[0058] like Figure 2 As shown, the CA attention mechanism module includes a coordinate information embedding module and a coordinate attention generation module connected in sequence.

[0059] The present invention improves upon the traditional feature extraction network VGG16 as follows to obtain the above-mentioned multi-scale feature extraction system.

[0060] First, while the traditional feature extraction network VGG16 can extract more information with increasing depth, this leads to gradient explosion and vanishing problems. ResNet, building upon CNNs, introduces residual blocks, preventing network degradation with depth. It uses shortcuts to superimpose input and output (features that didn't pass through the parameter blocks but did pass through the shortcuts) with features output through the residual blocks. This simple addition doesn't add extra parameters or computation, but significantly improves training speed and performance.

[0061] While the ReLU activation function converges quickly during backpropagation of network gradients, its output is always 0 when the input is negative, resulting in a "dying ReLU" state. To address this, this invention introduces a LeakyReLU activation function, whose expression is:

[0062]

[0063] In the formula a i The parameters are fixed in the interval (1, +∞). Furthermore, there is no gradient vanishing when x > 0, and the neuron will not "die" when x < 0.

[0064] Secondly, to prevent model degradation and better optimize the model, the order of Batch Normalization (BN) layers, activation layers, and convolutional layers was adjusted based on improvements to the residual module (the original order was convolutional layer, BN layer, activation function layer (ReLU activation function); the improved order is BN layer, activation function layer (LeakyReLU activation function), convolutional layer (further improvements to the convolutional layer are involved, currently referred to as convolutional layer)). The range of the shortcut identity mapping was also expanded. Using BN in pre-activation improves the model's regularization, thereby reducing overfitting. Expanding the identity mapping range simplifies the optimization process, resulting in better training performance for the improved network. This paper replaces the ReLU activation function in the ResNet50 residual structure with the LeakyReLU activation function and proposes an L-ResNet50 network structure based on the ResNet50 network construction method.

[0065] Third, to enable the network to focus on key information about wind turbine blade defects while ignoring irrelevant background information, this invention introduces a Coordinate Attention (CA) module. The CA module consists of a Coordinate Information Gmbedding module and a Coordinate Attention Generation module, arranged sequentially, with the network structure as follows: Figure 2 As shown, the coordinate information embedding module corresponds to Figure 2 The Residual, Horizontal Avg Pool, and Vertical AvgPool functions in the coordinate attention generation module correspond to these. Figure 2 The CA attention mechanism module can be viewed as a computational unit that enhances the network's feature representation capabilities. This unit can take any intermediate tensor as input and output an enhanced, same-size output with improved expressive power.

[0066] For the coordinate information embedding module, global pooling is often used in channel attention to globally encode spatial information as channel descriptors. However, this makes it difficult to preserve positional information. To enable the attention mechanism module to better capture the precise positional information of long-distance interactions in space, the network uses two one-dimensional feature encoding operations instead of global pooling. The input is passed through pooling kernels of sizes (H, 1) and (1, W), and each channel is encoded along the horizontal and vertical directions to obtain the output of the c-th channel with height h and width w. The expressions are as follows:

[0067]

[0068]

[0069] The two transformations described above are aggregated along the horizontal and vertical directions respectively, and a pair of feature maps for perceiving the direction are generated.

[0070] For the coordinate attention generation module, the first step concatenates the two feature maps generated by the previous module; the second step uses a shared 1×1 convolution to perform an F1 transformation and generate an intermediate feature map (f∈R) that can encode spatial information in both the horizontal and vertical directions. C / r×(H+W) ), where r represents the downsampling ratio, and its expression is as follows:

[0071] f=δ(F1([z h ,z w ])) (4)

[0072] In the formula, δ is a nonlinear activation function.

[0073] The third step is to divide f along the spatial dimension into two separate tensors fa. h ∈R C / r×H and f w ∈R C / r×W Using two 1×1 convolutions F h and F w feature map f h and f w The input is transformed to have the same number of channels, and the output is obtained. Its expression is as follows:

[0074] g w =σ(F w (f w (5)

[0075] g h =σ(F h (f h (6)

[0076] In the formula, σ is the Sigmoid activation function.

[0077] Step 4 for g h g w The process is expanded to obtain the final output. Its expression is as follows:

[0078]

[0079] Example 2

[0080] Embodiment 2 of the present invention provides a multi-scale feature extraction method, which is based on the multi-scale feature extraction system of Embodiment 1, and includes the following steps:

[0081] The input image is subjected to multiple convolutional downsampling operations through multiple first convolutional layers of the multi-layer feature extraction module to obtain multiple first feature maps at different scales; that is, the input image is processed by the multi-layer feature extraction module ( Figure 1 Feature extraction is performed on the leftmost linear network structure in the network, and four sets of first feature maps [C2, C3, C4, C5] at different scales are obtained through convolutional downsampling operations;

[0082] Multiple upsampling layers and residual modules in the multi-layer up-feature fusion module are used to upsample and fuse multiple first feature maps of different scales to obtain multiple second feature maps of different scales. This step is the top-down up-feature fusion stage. In this stage, bilinear interpolation is used to perform a 2x upsampling operation to obtain a feature map with the same downsampling scale. The first feature map is then subjected to a 1×1 convolution operation and fused with the upsampling result. The fused result is then subjected to a 3×3 convolution layer to eliminate aliasing. The CA attention mechanism module in the residual module is used to update the channels of the feature map that has eliminated aliasing to obtain the second feature map [P2, P3, P4, P5].

[0083] Multiple downsampling layers and residual modules in the multi-layer feature fusion module are used to downsample and fuse multiple second feature maps of different scales to obtain multiple third feature maps of different scales. This step is the bottom-up feature fusion stage, which yields four third feature maps [F2, F3, F4, F5]. A 3×3 convolution with a stride of 2 is used to fused the F2 and F3 features. i Perform downsampling, and then downsample the shallow features F i With deep features P (i+1) The features F are obtained by addition and fusion. (i+1) Then, the fused result is processed using a 3×3 convolution and a CA module to obtain the feature map F. (i+1) F2 directly copies the value of P2. To prevent gradient explosion during bidirectional fusion, residual modules are added during both top-down and bottom-up fusion processes.

[0084] Example 3

[0085] Embodiment 3 of the present invention provides a method for detecting defects in wind turbine blades, the method being based on the Faster R-CNN wind turbine blade defect detection model; as follows Figure 3 As shown, the Faster R-CNN wind turbine blade defect detection model includes: a region proposal network, a target detection network, and the aforementioned multi-scale feature extraction system. The multi-scale feature extraction system is connected to the region proposal network and the target detection network, respectively. The region proposal network is connected to the target detection network. Figure 3 In this context, Normalize means normalization, Test means testing, Training means training, Proposals means proposals, NMS (Non-Maximum Suppression) means non-maximum suppression, AnchorsGenerator means reference boxes, FeatureMaps means feature maps, ROI Pooling means region of interest pooling, Flatten means flattening, and FC1 and FC2 are both convolutions.

[0086] The detection method includes the following steps:

[0087] The image of the wind turbine blade to be detected is input into the multi-scale feature extraction system mentioned above for feature extraction, and multiple third feature maps of different scales are obtained.

[0088] Multiple third feature maps at different scales are input into a Region Proposal Network (RPN) to extract regions of interest (ROIs), resulting in ROIs corresponding to the third feature maps at different scales. The operations performed in the RPN include: sampling positive and negative samples using a softmax classification branch, obtaining accurate bounding boxes using a bounding box regression branch, removing overlapping bounding boxes using normalized mean squared error (NMS), and selecting the bounding box with the highest output score as the ROI.

[0089] The third feature maps at different scales and their corresponding regions of interest (ROIs) are input into the object detection network to determine the location and type of defects in the wind turbine blades. The operations performed in the object detection network include: mapping the proposed bounding boxes extracted by the region proposal network onto the feature maps at a fixed ratio, then fixing them to a specific size through ROI pooling and sending them to a fully connected layer; finally, obtaining the final output result through Softmax classification and bounding box regression.

[0090] The step of inputting an image of the wind turbine blades to be detected into the aforementioned multi-scale feature extraction system for feature extraction to obtain multiple third feature maps at different scales also includes:

[0091] Aerial photography using drones was employed to collect sample images of wind turbine blades in a wind farm, and these images were then labeled using the labelimg tool to construct an initial sample set. This process typically includes the following steps:

[0092] Step 1: Image Acquisition

[0093] 1-1 Image Acquisition Equipment: Drones can be broadly classified into two categories based on their flight principles and structure: fixed-wing drones and multi-rotor drones. Fixed-wing drones have greater payload capacity and endurance, while multi-rotor drones, due to their ability to hover and fly at low speeds, are more suitable for close-range, low-speed observation tasks or tasks requiring maintaining the same viewpoint for extended periods. Furthermore, multi-rotor drones are easy to operate, convenient to maintain, and cost-effective. The image acquisition component uses a DJI M210 equipped with a DJI Zenmuse Z30 camera gimbal. The M210 offers strong expandability, and its dual battery compartments ensure extended flight time. The DJI Zenmuse Z30 enables high-definition video recording and 12-megapixel still photography, and its three-axis stabilization gimbal keeps the camera stable during intense movements.

[0094] 1-2 Using the drone's navigation system, a preset flight path allows the drone to fly to a designated initial position. Due to the limited positioning accuracy of the drone, it may not necessarily be on the central axis of the wind turbine hub, and its nose may be at an angle to the wind turbine. Hough transform is used to identify the straight lines (blade edges) in the complete image of the wind turbine rotor. The hub center position is obtained by averaging the positions of the edge intersections. After detecting the hub center position, the drone's pose can be corrected. The presence of an off-axis angle is determined by whether the hub center is on the tower axis.

[0095] 1-3 After the UAV locates the wind turbine hub and reaches the designated detection distance, it acquires the first image from the center of the hub. Then, it flies along a blade direction, maintaining an equidistant distance from the blade plane, to acquire subsequent images. When acquiring images, the overlap area between adjacent images should not be too small. The UAV records its horizontal and vertical displacement relative to the previous acquisition while acquiring images. After acquiring images of the front of the blade, it moves to the back of the blade to acquire images of the back side. After completion, it returns to the origin to inspect the next blade. Each blade's front and back images are grouped separately, and after all acquisition is completed, all six groups of images and displacement data are transmitted to the computer.

[0096] Step 2: Dataset Preparation

[0097] This invention exemplifies the use of images captured by drones during wind farm inspections to construct a dataset. Defects in the wind turbine blades are labeled using the labelimg tool and saved in PascalVOC format. Three types of blade defects were selected as experimental subjects: coat off, cracks, and erosion. The training set and test set contain 400 and 150 images, respectively.

[0098] The initial sample set is subjected to data augmentation and Gaussian denoising to obtain the processed sample set. This step includes the following steps:

[0099] Step 3: Data Augmentation

[0100] 3-1 Because deep learning models require a large amount of data for training and convergence, many people use data augmentation techniques in image processing. If the dataset is too small, the deep learning model cannot converge sufficiently. Data augmentation techniques can expand the training set and improve model performance to some extent. Data augmentation involves performing random transformations on the original data, which not only increases the diversity of data samples but also increases sample complexity, achieving the effect of increasing the size of the data sample and thus improving model performance. Common data augmentation methods include translation, scaling, and adding Gaussian noise.

[0101] 3-2 For the enhancement of training samples in this paper, Mosaic data augmentation and static augmentation are used, mainly including rotation, translation and scaling. Through these operations, the number of training samples is increased. After image augmentation, the training set has 800 images, which improves the performance of the model and reduces overfitting.

[0102] 3-3 Gaussian Denoising

[0103] Noise may be generated during the acquisition and transmission of blade defect images. Noise information is invalid information, and its presence blurs the original information of the image, interfering with the accurate extraction of blade edge information in subsequent blade images. Therefore, image smoothing and denoising processing is required.

[0104] The Gaussian filter is a widely used linear smoothing filter. Gaussian filtering achieves noise reduction by applying a weighted average to the entire image. The gray value of each pixel is obtained by weighting its own gray values ​​and those of its neighboring pixels. The weights used ensure that the values ​​follow a normal distribution, making it suitable for eliminating Gaussian noise.

[0105] The basic form of the two-dimensional Gaussian function is as follows:

[0106]

[0107] In the formula, σ is the standard deviation, which controls the dispersion of the Gaussian function. The smaller σ is, the more concentrated the Gaussian function is; the larger σ is, the more diffuse the Gaussian function is. Generally, the coordinates x and y are integers. In order to generate a filter template of size m×n from this function, and to ensure that the sum of the weights of the integerized template is 1, the filter template needs to be normalized to ensure that the uniform gray-level regions of the image are not affected by Gaussian filtering.

[0108] The processed sample set is used to train the Faster R-CNN-based wind turbine blade defect detection model. This step specifically includes the following steps:

[0109] 8-1 Eight hundred collected and labeled images of blade defects were used to train a wind turbine blade defect detection model. The operating system used was Ubuntu 18.04, the deep learning framework was PyTorch, the CPU was an Intel(R) Core(TM) i9-10850K, the memory was Kingston 16GB DDR4, the graphics card was a GeForce RTX 2080Ti, the IOU value was set to 0.5, the batch size was set to 2, the learning rate was set to 0.005, and the epoch was set to 100.

[0110] 8-2 Evaluation Indicators for Wind Turbine Blade Defect Detection Model Training:

[0111] Accuracy is a measure of the accuracy of a model algorithm, and it is calculated using the following formula:

[0112]

[0113] Recall is a measure of a model's ability to comprehensively find positive samples. The formula for calculating recall is:

[0114]

[0115] Average precision (AP) measures the performance of a model algorithm on each class, and is calculated using the following formula:

[0116]

[0117] The average precision (mAP) for all categories is obtained by summing and averaging the average precision (AP), and the calculation formula is as follows:

[0118]

[0119] In the formula is the average precision for each category, and C is the total number of categories.

[0120] The main contributions of this invention are as follows:

[0121] (1) Improve the residual block structure and replace the ReLU function with the LeakyReLU function to obtain the L-ResNet50 feature extraction network, which can retain the effective information of the negative half axis. Adjust the order of the convolutional layer, BN layer and LeakyReLU activation layer to optimize the model training effect.

[0122] (2) A multi-scale feature fusion module is proposed to fuse the output of the feature extraction network at multiple scales to obtain rich image features;

[0123] (3) By embedding an attention mechanism module in the multi-scale feature fusion stage, attention is paid to local key areas, which improves the detection accuracy of small-scale defect targets.

[0124] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0125] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting defects in wind turbine blades, characterized in that, The detection method is based on the Faster R-CNN wind turbine blade defect detection model; the Faster R-CNN wind turbine blade defect detection model includes: a region proposal network, a target detection network, and a multi-scale feature extraction system, wherein the multi-scale feature extraction system is connected to the region proposal network and the target detection network respectively, and the region proposal network is connected to the target detection network. The multi-scale feature extraction system includes: a multi-layer feature extraction module, a multi-layer upper feature fusion module, and a multi-layer lower feature fusion module; The multi-layer feature extraction module includes N sequentially connected first convolutional layers; The multi-layer up-feature fusion module includes N-2 sequentially connected upsampling layers; The multi-layer downsampling module includes N-2 sequentially connected downsampling layers; The Nth first convolutional layer is connected to the first upsampling layer through the second convolutional layer, and the (N-2)th upsampling layer is connected to the first downsampling layer; Each output terminal of the upsampling layer and each output terminal of the downsampling layer are provided with a residual module; The residual module set at the output of the nth upsampling layer is also connected to the output of the Nnth first convolutional layer through a second convolutional layer; n = 1, ..., N-2; The residual module located at the output of the m-th downsampling layer is also connected to the input of the N-1-m-th upsampling layer through a third convolutional layer; m = 1, ..., N-2; The residual module consists of a feature fusion layer, a BN layer, a LeakyReLU activation function layer, and a CA attention mechanism module connected in sequence. The CA attention mechanism module includes: a coordinate information embedding module and a coordinate attention generation module connected in sequence; the coordinate information embedding module includes: residual block, horizontal average pooling and vertical average pooling, and the coordinate attention generation module includes: connection + two-dimensional input convolution, normalization + nonlinearization, splitting, two-dimensional input convolution, S-function and weighting. The detection method includes the following steps: The image of the wind turbine blades to be detected is input into a multi-scale feature extraction system for feature extraction, resulting in multiple third feature maps at different scales. Multiple third feature maps of different scales are input into the region proposal network to extract regions of interest, thus obtaining regions of interest corresponding to third feature maps of different scales; By inputting third feature maps of different scales and the corresponding regions of interest of third feature maps of different scales into the target detection network, the location and type of defects in the wind turbine blades can be determined. The initial sample set for the model used to train the Faster R-CNN wind turbine blade defect detection model was obtained as follows: Using the drone's navigation system, the drone flies to a designated initial position via a preset flight path. The Hough transform is used to identify the straight lines, i.e. the edges of the blades, in the complete image of the wind turbine rotor. The position of the hub center can be obtained by averaging the positions of the intersection points of the edges. After detecting the position of the hub center, the drone's pose is corrected. After locating the wind turbine hub and reaching the designated detection distance, the drone acquires the first image from the center of the hub. Then, it flies along the direction of a blade, maintaining an equal distance from the blade plane, to acquire subsequent images. While acquiring images, the drone records the horizontal and vertical displacements relative to the previous acquisition. After acquiring images of the front of the blade, it moves to the back of the blade to acquire images of the back side. After finishing, it returns to the origin to inspect the next blade. The front and back images of each blade are divided into separate groups. After all acquisition work is completed, the six groups of images and displacement data are transmitted to the computer together.

2. The method for detecting defects in wind turbine blades according to claim 1, characterized in that, The step of inputting an image of the wind turbine blades to be detected into a multi-scale feature extraction system for feature extraction to obtain multiple third feature maps at different scales also includes: We used drone aerial photography technology to collect sample images of wind turbine blades in wind farms, and used the labelimg tool to annotate the sample images to construct an initial sample set. The initial sample set is subjected to data augmentation and Gaussian denoising to obtain the processed sample set; The processed sample set was used to train the Faster R-CNN-based wind turbine blade defect detection model.

3. A wind turbine blade defect detection system, characterized in that, The detection system is applied to the wind turbine blade defect detection method according to any one of claims 1-2, and the detection system includes: The feature extraction module is used to input the image of the wind turbine blades to be detected into the multi-scale feature extraction system for feature extraction, and obtain multiple third feature maps at different scales. The Region of Interest (ROI) extraction module is used to input multiple third feature maps of different scales into the Region Proposal Network (RPN) to extract the ROI, thus obtaining the ROI corresponding to the third feature maps of different scales. The target detection module is used to input third feature maps of different scales and the regions of interest corresponding to the third feature maps of different scales into the target detection network to determine the location and type of defects in the wind turbine blades.

4. The wind turbine blade defect detection system according to claim 3, characterized in that, The detection system also includes: The initial sample set construction module is used to collect sample images of wind turbine blades in wind farms using drone aerial photography technology, and to annotate the sample images using the labelimg tool to construct the initial sample set; The data processing module is used to perform data augmentation and Gaussian denoising on the initial sample set to obtain the processed sample set. The training module is used to train the Faster R-CNN-based wind turbine blade defect detection model using the processed sample set.

Images