An improved UNet fan blade crack detection method fusing multi-directional strip convolution

Abstract

The application discloses an improved UNet fan blade crack detection method fusing multi-directional strip convolution, and relates to the technical field of deep learning.The method comprises the following steps: designing a fan blade crack detection network; inputting a to-be-detected image into the trained fan blade crack detection network to output a detection result; and determining a crack area according to the detection result.The fan blade crack detection network of the application replaces a conventional UNet feature extraction network with a ResNeSt50 network, introduces a coordinate attention module, and proposes a multi-directional crack feature enhancement module, so as to prevent irrelevant information interference, strengthen the attention of the network to important crack information, and improve the crack feature extraction capability of the model.The results show that the application has good effects on the detection of fine and light-colored cracks.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for wind power plants, and in particular to an improved UNet wind turbine blade crack detection method that integrates multi-directional strip convolution. Background Technology

[0002] Cracks in wind turbine blades are one of the most common types of surface damage. If these cracks are not detected promptly, their further propagation will severely compromise the lifespan of the blades and could even cause incalculable losses to the wind power plant. Therefore, it is crucial to monitor the health of wind turbine blades in a timely manner. Previously, wind farms relied primarily on manual inspections for blade damage detection, which were heavily influenced by the subjective opinions of the inspectors and resulted in low efficiency. In recent years, with the development of information technology, the automatic identification of targets using image data has provided a new approach to wind turbine blade crack damage detection.

[0003] Currently, Convolutional Neural Networks (CNNs) are widely used in object detection tasks. However, CNN models have a very limited receptive field, making it difficult to capture long-distance feature dependencies. These dependencies are particularly important for semantic segmentation tasks, especially in wind farms where high-resolution images are used for feature extraction of surface cracks on wind turbine blades. Contextual information is significantly lost in these cases, leading to poor detection results. To address this issue, researchers have proposed the Atrous Spatial Pyramid Pooling (ASPP) module based on dilated convolutions to enhance the receptive field of the network model, solve the problem of lost contextual information, and further improve the network's receptive field. Meanwhile, the convolutional kernels used in convolution operations are mostly square, which reduces the flexibility in capturing light-colored, fine cracks on wind turbine blades. This is because square convolutional kernels not only capture the necessary feature information but also include irrelevant interference information, which fails to fully utilize the network's feature extraction capabilities for cracks with strip-like characteristics, resulting in unsatisfactory detection results for wind turbine blade cracks. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned shortcomings and provide an improved UNet wind turbine blade crack detection method that integrates multi-directional strip convolution to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an improved UNet wind turbine blade crack detection method integrating multi-directional strip convolution, comprising the following steps:

[0006] S1. Collect images of wind turbine blades from wind farms, and label the cracks and defects in the dataset to construct a wind turbine blade crack and defect dataset. Divide the constructed dataset into a training set, a test set, and a validation set.

[0007] S2. Perform preprocessing operations on the dataset images to reduce the impact of noise and other irrelevant factors on the detection results;

[0008] S3. Designed with multi-directional strip convolution that is more suitable for wind turbine blade crack detection;

[0009] S4. Construct a multi-directional crack feature enhancement module to enhance crack features and reduce the loss of edge details in crack images;

[0010] S5. Establish an improved UNet wind turbine blade crack detection network;

[0011] S6. By combining the cross-entropy loss function and the Dice loss function, the network can focus more on the wind turbine blades and the thin, light-colored cracks on the blades.

[0012] S7. Train the network structure to obtain a well-trained wind turbine blade crack defect detection network. Judge the network segmentation performance by selecting evaluation indicators and verify the network's advancement through comparative experiments.

[0013] S8. Input the wind turbine blade image into the trained wind turbine blade crack defect detection network and output the crack detection results.

[0014] Preferably, in step S2, the image is preprocessed, specifically by: firstly, using the averaging method to process the original image into grayscale, preserving complete image information; secondly, using the median filtering method to filter the grayscale image; and thirdly, effectively suppressing noise in the image.

[0015] Preferably, in step S3, a multi-directional strip convolution more suitable for wind turbine blade crack detection is designed. The specific steps are as follows: First, the feature input is fed into four parallel paths after a 1×1 convolution, performing strip convolution operations in four directions: vertical, horizontal, and two diagonal directions. In addition, parallel one-dimensional dilated convolutions are added to each path to extract features from the feature map. Then, the output features from these four parallel paths are fused, and then subjected to another 1×1 convolution to obtain the final improved strip convolution feature output. The specific calculation formula is as follows:

[0016]

[0017] Among them, X in With X out Represents the input and output of the multi-directional strip convolution module; X and X' i This represents the input and output of a convolution operation considering multi-scale features; represents 1×1 convolution, strip convolution in different directions, and dilated convolution, respectively; p×q is the kernel size; i = 1, 2, 3, 4 represent the vertical, horizontal, and two diagonal directions; [] represents the feature fusion operation.

[0018] Preferably, in step S4, the multi-directional crack feature enhancement module comprises the following steps: First, the crack features are input into the multi-directional strip convolution module after a convolution operation. Since the crack lengths are different and the propagation directions are also different, the crack feature information is captured using strip convolutions in different directions and parallel dilated convolutions. Then, the features in each direction are fused, and the fused features are subjected to a convolution operation again. Finally, the output is weighted and processed with the features input from the multi-directional strip convolution module. The specific calculation formula is as follows:

[0019]

[0020] Among them, Y in and Y out These are the feature inputs and outputs of the MCEB module, respectively; Y strip Y represents the output of multi-directional strip convolution; MCEB This represents the feature output before batch normalization. Represents a 1×1 convolution operation; Direction i This represents multi-directional strip convolution, which involves convolution operations in different directions; Norm represents batch normalization of input features; MLP represents a multilayer perceptron classifier. and These represent element-wise addition and element-wise multiplication, respectively.

[0021] Preferably, in step S5, an improved UNet wind turbine blade crack detection network is established. The specific steps are as follows: only the UNet encoder part is improved. First, the UNet feature extraction network is replaced with a ResNeSt50 network to improve the ability to extract deep semantic information. Second, a coordinate attention mechanism is added after the ResNeSt50 network to effectively filter important information in the feature map that is conducive to crack detection. Finally, a multi-directional crack feature enhancement module is added after the coordinate attention mechanism to enhance the crack feature capture capability.

[0022] Preferably, in step S6, the cross-entropy loss function and the Dice loss function are combined. Specifically, the two loss functions are directly weighted and summed to obtain the total loss function. The Dice loss function is used to improve the model's detection of crack defects, while the cross-entropy loss function is used to enhance the model's detection of wind turbine blades. The specific formula is as follows:

[0023] loss Cross =-(y n *log(z n )+(1-y n )*log(1-z n ))

[0024]

[0025] Where, loss Cross Let y represent the cross-entropy loss function. n This represents the label of the nth wind turbine blade data sample, with 0 for positive classes and 1 for negative classes; z n The loss statement represents the probability of predicting that the nth sample is a positive sample. Dice Let N represent the Dice loss function. TP N represents the total number of pixels that were actually detected as cracks; FN N represents the total number of pixels that were actually not cracked but were detected as not cracked. FP This represents the total number of pixels that were actually not cracked but were detected as cracks.

[0026] Preferably, in step S7, the network structure is trained to obtain a trained wind turbine blade crack defect detection network. The specific steps are as follows: using the training set images in the dataset, input the required number of images for each iteration into the designed improved UNet wind turbine blade crack detection network that integrates multi-directional strip convolutions; the network is frozen for 100 training epochs, and the maximum training epoch is 600; after training, the weights with the best performance on the validation set are saved, thus completing the training of the wind turbine blade crack defect detection network.

[0027] Beneficial effects of this invention:

[0028] 1. This invention proposes an improved UNet wind turbine blade crack detection method that integrates multi-directional strip convolution. Compared with the traditional UNet, this invention replaces the original feature extraction network with a ResNeSt50 network, and adds a coordinate attention mechanism and a multi-directional crack feature enhancement module. Comparative experiments show that this invention can detect blade cracks while the wind turbine is in operation, and can accurately and smoothly segment crack defects. All indicators are superior to the traditional UNet network.

[0029] 2. The method of this invention is used in the intelligent inspection of wind turbine blades. This method not only makes up for the limited receptive field of traditional CNN by using dilated convolution, but also combines multi-scale thinking and fully targets the slender characteristics of cracks by using strip convolution operation, which is more suitable for strip-shaped targets, in four directions: horizontal, vertical and two diagonals, to extract crack features, reduce the influence of irrelevant information and improve the network's ability to obtain crack features. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of the method of the present invention;

[0031] Figure 2 This is a diagram of the multi-directional strip convolution structure in the method of the present invention;

[0032] Figure 3 This is a structural diagram of the MCEB module in the method of the present invention;

[0033] Figure 4 The comparison results of the methods of the present invention are visualized. Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0035] Example 1:

[0036] This embodiment provides an improved UNet wind turbine blade crack detection method that integrates multi-directional strip convolution. Based on the traditional UNet, it first replaces the original feature extraction network with a ResNeSt50 network to enhance the extraction capability of deep semantic information. Secondly, a coordinate attention mechanism is added to the network to effectively filter important information in the feature map that is beneficial for crack detection. Finally, a multi-directional crack feature enhancement block (MCEB) is proposed, combining the advantages of strip convolution for crack feature extraction and dilated convolution to increase the receptive field. The specific process is as follows: Figure 1 As shown.

[0037] Step 1: Collect images of wind turbine blades from wind farms, label the cracks and defects in the dataset, construct a wind turbine blade crack and defect dataset, and divide the constructed dataset into training set, test set, and validation set;

[0038] The specific operation is as follows: Images of wind turbine blades are acquired using a gimbal camera and a drone. The images of the wind turbine blades to be detected are then augmented by rotating, flipping, and cropping the acquired images to increase the diversity of the dataset and improve the robustness of the network. Subsequently, the augmented dataset is labeled with cracks using LabelMe, resulting in the final wind turbine blade crack defect dataset. This dataset is divided into training, testing, and validation sets. The preprocessed training set is fed into the constructed wind turbine blade crack detection network to obtain a set of weights through training, which is then validated using the test set. The trained weights are saved. The trained network weights are then used to detect new images of wind turbine blades to be detected, ultimately outputting the wind turbine blade crack detection result image.

[0039] Step 2: Perform preprocessing on the dataset images to reduce the impact of noise and other irrelevant factors on the detection results;

[0040] The specific steps are as follows: First, the original image is processed using the average value method to remove redundant information and highlight the target area; second, the grayscale image is filtered using the median filtering method; and finally, noise in the image is effectively suppressed to ensure the edge characteristics of the image.

[0041] Step 3: Design a multi-directional strip convolution that is more suitable for detecting cracks in wind turbine blades;

[0042] The specific operation is as follows: Most CNN architectures typically use square convolutional kernels. Learning feature maps within square kernels is suitable for most natural objects with volumetric shapes. However, wind turbine blade cracks exhibit fine and strip-like characteristics. Using square kernels not only fails to capture crack features effectively but also inevitably acquires irrelevant information, especially for early-stage cracks on wind turbine blades, which are not only finer but also lighter in color, often requiring more efficient long-range context awareness. Therefore, this invention combines the advantages of dilated convolution and strip convolution, proposing a multi-directional strip convolution that considers multi-scale information. Strip convolution is more consistent with the shape of the crack, utilizing a long convolutional kernel along one spatial direction to capture long-range dependencies in the crack region. Furthermore, it captures local context along another spatial direction, preventing irrelevant regions from interfering with feature learning. The multi-directional strip convolution proposed in this invention considers the importance of multi-scale information for feature extraction, adding parallel dilated convolutions on top of strip convolution operations in different directions to further acquire local contextual relationships. The specific structure is as follows: Figure 2 As shown, the feature input of the multi-directional strip convolution is fed into four parallel paths after a 1×1 convolution, performing convolution operations in four directions: vertical, horizontal, and two diagonal directions. In addition, a set of parallel dilated convolutions is added to each path to extract features from the feature map. Then, the output features from these four parallel paths are fused, and then subjected to another 1×1 convolution to obtain the final improved strip convolution feature output. The specific calculation formula is as follows.

[0043]

[0044] Among them, X in With X out Represents the input and output of the multi-directional strip convolution module; X and X' i This represents the input and output of a convolution operation considering multi-scale features; represents 1×1 convolution, strip convolution in different directions, and dilated convolution, respectively; p×q is the kernel size; i = 1, 2, 3, 4 represent the vertical, horizontal, and two diagonal directions; [] represents the feature fusion operation.

[0045] Step 4: Construct a multi-directional crack feature enhancement block (MCEB) to enhance crack features and reduce the loss of edge details in crack images;

[0046] The specific operation is as follows: The MCEB module utilizes the multi-directional strip convolution proposed in this invention to focus more on the edge details of cracks, and has a good detection effect on early light-colored fine cracks in wind turbine blades. The specific structure is as follows: Figure 3 As shown in the diagram, in the MCEB module, the feature map acquired by the encoder is first input into a multi-directional strip convolution module after a 1×1 convolution. Secondly, since the crack lengths vary and the propagation directions differ, the multi-directional strip convolution module combines multi-scale thinking with depthwise strip convolutions in different directions and parallel dilated convolutions to capture feature information, and then fuses the features from each direction. Finally, the fused features are input into a 1×1 convolution again, and then the output is weighted and processed with the features from the input multi-directional strip convolution. This can more effectively capture strip target information in the image and is more conducive to extracting the main features of wind turbine blade cracks. The specific calculation formula is as follows.

[0047]

[0048] Among them, Y in and Y out These are the feature inputs and outputs of the MCEB module, respectively; Y strip Y represents the output of multi-directional strip convolution; MCEB This represents the feature output before batch normalization. Represents a 1×1 convolution operation; Direction i This represents multi-directional strip convolution, which involves convolution operations in different directions; Norm represents batch normalization of input features; MLP represents a multilayer perceptron classifier. and These represent element-wise addition and element-wise multiplication, respectively.

[0049] Step 5: Establish an improved UNet wind turbine blade crack detection network;

[0050] The specific operation is as follows: Only the UNet encoder part is improved. First, the UNet feature extraction network is replaced with a ResNeSt50 network to enhance the extraction capability of deep semantic information. Second, a coordinate attention mechanism is added after the ResNeSt50 network to effectively filter important information in the feature map that is beneficial for crack detection. Finally, a multi-directional crack feature enhancement block (MCEB) is added after the coordinate attention mechanism to enhance the crack feature capture capability. The decoder part is consistent with the original network. The image of the wind turbine blade to be detected is preprocessed in the encoder part to obtain a grayscale image. First, the grayscale image is input into the ResNeSt50 feature extraction network to obtain a feature map. Second, under the action of the coordinate attention mechanism, the position information is embedded into the channel, making the network focus on important information beneficial for crack detection to reduce feature loss. Then, the feature map obtained in the previous step is enhanced by the MCEB module, and the enhanced feature map is fused with the upsampled feature map. Finally, the crack detection result image of the wind turbine blade is obtained.

[0051] Step 6: By combining the Cross-Entropy Loss function and the Dice Loss function, the network can focus more on the wind turbine blades and the thin, light-colored cracks on the blades.

[0052] The specific operation involves directly weighting and summing the two loss functions to obtain the total loss function. DiceLoss is used to improve the model's detection of crack defects, while Cross-Entropy Loss is used to enhance the model's detection of wind turbine blades. The specific calculation formula is as follows:

[0053] loss Cross =-(y n *log(z n )+(1-y n )*log(1-z n ))

[0054]

[0055] Where, loss Cross Let y represent the cross-entropy loss function. n This represents the label of the nth wind turbine blade data sample, with 0 for positive classes and 1 for negative classes; z n The loss statement represents the probability of predicting that the nth sample is a positive sample. Dice Let N represent the Dice loss function. TP N represents the total number of pixels that were actually detected as cracks; FNN represents the total number of pixels that were actually not cracked but were detected as not cracked. FP This represents the total number of pixels that were actually not cracked but were detected as cracks.

[0056] Step 7: Train the network structure to obtain a trained wind turbine blade crack defect detection network. Judge the network segmentation performance by selecting evaluation indicators and verify the network's advancement through comparative experiments.

[0057] The specific operation is as follows: Using images from the training set in the dataset, the required number of images for each iteration are input into the improved UNet wind turbine blade crack detection network, which incorporates multi-directional strip convolutions. The network is frozen at 100 training epochs, with a maximum of 600 training epochs. Adam is used as the network optimizer during training. After training, the weights with the best performance on the validation set are saved, thus completing the training of the wind turbine blade crack defect detection network. The network performance is evaluated using Mean Pixel Accuracy (MPA) and Mean Intersection over Union (MIoU), calculated using the following formulas:

[0058]

[0059] Where n+1 represents the number of all target categories, p ii This indicates the number of pixels that correctly detected the crack; p ij This represents the number of pixels that belong to class i but are detected as class j; p ji This represents the number of pixels that belong to class j but are predicted to be class i.

[0060] Step 8: Input the wind turbine blade image into the trained wind turbine blade crack defect detection network and output the crack detection results.

[0061] This invention trains UNet and its own model on a self-made dataset training set, selects hyperparameters on a validation set, and finally evaluates the network detection performance using a test set. Table 1 shows the comparison results.

[0062] Table 1: Performance Comparison of Each Model

[0063]

[0064] The MPA value and MIoU value of the method of the present invention reach 89.07% and 84.52% respectively. Compared with the traditional UNet, the MPA value is increased by 3.92% and the MIoU value is increased by 4.68%, which proves that the method of the present invention has certain advantages.

[0065] Figure 4To compare the results, the traditional UNet network performs poorly in segmenting small and complex cracks, lacking precision and missing some small cracks, and the segmentation is not continuous enough. Compared with the traditional UNet network, the method of this invention can more completely identify some small cracks and has smoother crack image edge segmentation, which can fit the label to the greatest extent.

[0066] This invention's wind turbine blade crack detection method replaces the original UNet feature extraction network with a ResNeSt50 network and introduces a coordinate attention module. It proposes a Multi-directional Crack Feature Enhancement Block (MCEB) to prevent interference from irrelevant information, strengthen the network's focus on important crack information, and improve the model's ability to extract crack features. Results show that this invention outperforms the traditional UNet in detecting thin, light-colored cracks.

[0067] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be limited to the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. An improved UNet wind turbine blade crack detection method incorporating multi-directional strip convolution, characterized in that: Includes the following steps: S1. Collect images of wind turbine blades from wind farms, and label the cracks and defects in the dataset to construct a wind turbine blade crack and defect dataset. Divide the constructed dataset into a training set, a test set, and a validation set. S2. Perform preprocessing operations on the dataset images to reduce the impact of noise on the detection results; S3. Designed with multi-directional strip convolution that is more suitable for wind turbine blade crack detection; S4. Construct a multi-directional crack feature enhancement module to enhance crack features and reduce the loss of edge details in crack images; S5. Establish an improved UNet wind turbine blade crack detection network; S6. Combining the cross-entropy loss function and the Dice loss function enables the network to focus more on the wind turbine blades and the thin, light-colored cracks on the blades. S7. Train the network structure to obtain a well-trained wind turbine blade crack defect detection network. Judge the network segmentation performance by selecting evaluation indicators and verify the network's advancement through comparative experiments. S8. Input the wind turbine blade image into the trained wind turbine blade crack defect detection network and output the crack detection results; In step S3, a multi-directional strip convolution more suitable for wind turbine blade crack detection is designed. The specific steps are as follows: First, the feature input is input into four parallel paths after a 1×1 convolution. Strip convolution operations are performed in four directions: vertical, horizontal and two diagonal lines. In addition, parallel one-dimensional dilated convolution is added to each path to extract features from the feature map. The output features of these four parallel paths are then fused, and then subjected to a 1×1 convolution to obtain the final improved strip convolution feature output; the specific calculation formula is as follows: ； ； ； Among them, X in With X out Represents the input and output of the multi-directional strip convolution module; X and X' i This represents the input and output of a convolution operation considering multi-scale features; , , These represent 1×1 convolution, strip convolution in different directions, and dilated convolution, respectively. p × q The kernel size; i =1,2,3,4 represent vertical, horizontal, and two diagonal directions; [ ] represents feature fusion operations; In step S5, an improved UNet wind turbine blade crack detection network is established. The specific steps are as follows: only the UNet encoder part is improved. First, the UNet feature extraction network is replaced by the ResNeSt50 network to improve the ability to extract deep semantic information. Secondly, a coordinate attention mechanism is added after the ResNeSt50 network to effectively filter important information in the feature map that is beneficial for crack detection; finally, a multi-directional crack feature enhancement module is added after the coordinate attention mechanism to enhance the crack feature capture capability.

2. The improved UNet wind turbine blade crack detection method based on multi-directional strip convolution as described in claim 1, characterized in that: In step S2, the image is preprocessed. The specific steps are as follows: First, the original image is processed into grayscale using the averaging method to retain complete image information; second, the grayscale image is filtered using the median filtering method to effectively suppress noise in the image.

3. The improved UNet wind turbine blade crack detection method based on multi-directional strip convolution as described in claim 1, characterized in that: In step S4, a multi-directional crack feature enhancement module is constructed. The specific steps are as follows: First, the crack features are input into a multi-directional strip convolution module after a convolution operation. Since the crack lengths vary and the propagation directions are different, strip convolutions in different directions and parallel dilated convolutions are used to capture crack feature information. Then, the features from each direction are fused, and the fused features are subjected to another convolution operation. Finally, the output is weighted and calculated with the features input from the multi-directional strip convolution module. The specific calculation formula is as follows: ； ； ； Among them, Y in and Y out These are the feature inputs and outputs of the MCEB module, respectively; Y strip Y represents the output of multi-directional strip convolution; MCEB This represents the feature output before batch normalization. This represents a 1×1 convolution operation; This represents multi-directional strip convolution, which involves convolution operations in different directions; Norm represents batch normalization of input features; MLP represents a multilayer perceptron classifier. and These represent element-wise addition and element-wise multiplication, respectively.

4. The improved UNet wind turbine blade crack detection method based on multi-directional strip convolution as described in claim 1, characterized in that: In step S6, the cross-entropy loss function and the Dice loss function are combined. Specifically, the two loss functions are directly weighted and summed to obtain the total loss function. The Dice loss function is used to improve the model's detection of crack defects, while the cross-entropy loss function is used to enhance the model's detection of wind turbine blades. The specific formula is as follows: ； ； in, Let y represent the cross-entropy loss function. n This represents the label of the nth wind turbine blade data sample, with 0 for positive classes and 1 for negative classes; z n This represents the probability of predicting that the nth sample is a positive sample; Represents the Dice loss function. N TP This represents the total number of pixels that were actually detected as cracks. N FN This represents the total number of pixels that were actually non-cracked but were detected as non-cracked. N FP This represents the total number of pixels that were actually not cracked but were detected as cracks.

5. The improved UNet wind turbine blade crack detection method based on multi-directional strip convolution as described in claim 1, characterized in that: In step S7, the network structure is trained to obtain a trained wind turbine blade crack defect detection network. The specific steps are as follows: using the training set images in the dataset, the number of images required for each iteration is input into the designed improved UNet wind turbine blade crack detection network that integrates multi-directional strip convolution. The network is frozen for 100 training rounds, with a maximum of 600 training rounds. After training, the weights with the best performance on the validation set are saved, thus completing the training of the wind turbine blade crack defect detection network.

Images