Fan blade defect detection method and system based on improved SSD model

Abstract

This application proposes a method and system for wind turbine blade defect detection based on an improved SSD model. The method includes: constructing a target detection network including a ResNext backbone network, multiple detection layers, a bidirectional feature pyramid structure, and a detection branch; obtaining a wind turbine blade image dataset and annotating defects on the image; inputting the preprocessed image into the backbone network to obtain a multi-scale feature map, which is then processed by multiple detection layers and then input into a bidirectional feature pyramid structure for feature fusion, and obtaining initial detection results through the detection branch; calculating the loss and updating the network parameters using a target loss function constructed based on FocalLoss, Smooth L1Loss, and IOU Loss until the target detection network is trained; inputting the image to be detected into the trained network and obtaining the defect detection results output by the network. This method reduces the number of parameters and computational complexity required for blade defect detection, improving the detection efficiency of the detection network and the accuracy of the detection results.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a method and system for detecting defects in wind turbine blades based on an improved SSD model. Background Art

[0002] At present, target detection technology has been widely used in many fields such as transportation and medical treatment. Target detection is an image segmentation technology based on the geometric and statistical features of the target. The purpose of target detection is to determine the location and category of the object of interest in the image. For example, for wind blade images, defects such as damage and breakage in the image can be determined by performing target detection. Traditional image target detection methods usually require manual feature design, have poor robustness to complex environmental changes, and the sliding window-based region selection strategy has high time complexity. With the rapid development of deep learning technology and the significant improvement in computer storage and computing power, target detection has evolved from traditional methods to today's deep learning-based detection methods, which has improved the efficiency and accuracy of wind blade defect detection using deep convolutional neural network-based target detection methods.

[0003] In related technologies, convolutional neural network-based object detection methods mainly include single-stage and two-stage detection methods. Single-stage detection methods are represented by models such as YOLO and SSD, while two-stage detection frameworks are represented by the R-CNN series of models. Two-stage detection typically first generates a series of sparse candidate boxes through heuristic methods or CNN networks, and then classifies and regresses these candidate boxes. Single-stage detection typically uniformly and densely samples different locations in the image, using different scales and aspect ratios. Classification and regression are then performed directly after feature extraction using CNN.

[0004] However, the applicant discovered that of the two aforementioned detection methods, two-stage detection involves complex steps and is slow, while single-stage detection requires a large number of parameters and computations, resulting in reduced target detection efficiency, low detection accuracy, and difficulty in model training. Therefore, a solution that can simultaneously ensure both target detection accuracy and efficiency is urgently needed. Summary of the Invention

[0005] The present application aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the first purpose of this application is to propose a wind turbine blade defect detection method based on an improved SSD model. This method lightweights the backbone network of the detection model, reduces the number of parameters and calculations required for blade defect detection, enriches the feature expression, optimizes the loss function, and improves the detection efficiency of the detection network and the accuracy of the detection results.

[0007] The second object of this application is to propose a wind turbine blade defect detection system based on an improved SSD model;

[0008] A third object of the present application is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first embodiment of the present application is to propose a wind turbine blade defect detection method based on an improved SSD model, the method comprising the following steps:

[0010] The classic detection network SSD model is improved to construct an object detection network including a backbone network, multiple detection layers, a bidirectional feature pyramid structure, and a detection branch. The ResNext network is used as the backbone network.

[0011] Obtaining a wind blade image dataset, annotating defects on the images in the dataset to generate corresponding label files, and dividing the annotated wind blade image dataset into a training set, a validation set, and a test set according to a preset ratio;

[0012] Preprocessing the images in the training set, inputting the preprocessed images into the backbone network to obtain a multi-scale feature map, inputting the multi-scale feature map processed by the multiple detection layers into the bidirectional feature pyramid structure for feature fusion, and obtaining an initial detection result through the detection branch;

[0013] Based on the label file and the initial detection result, the loss is calculated and the parameters of the target detection network are updated by a preset target loss function, and the target detection network is iteratively trained until the training of the target detection network is completed, wherein the target loss function is constructed based on the FocalLoss function, the smooth L1 Loss function and the intersection-over-union loss function (IOU Loss);

[0014] The image of the wind turbine blade to be inspected is input into the trained target detection network to obtain the defect detection result output by the trained target detection network.

[0015] Optionally, in one embodiment of the present application, the target detection network is iteratively trained until the training of the target detection network is completed, including: using a stochastic gradient descent SGD optimizer to iteratively train the target detection network for a preset number of times until the prediction accuracy of the result of the target detection network predicting the images in the validation set reaches a preset accuracy threshold; after the training of the target detection network is completed, it also includes: inputting the images in the test set into the trained target detection network to test the accuracy of the trained target detection network in detecting wind turbine blade defects.

[0016] Optionally, in one embodiment of the present application, the images in the training set are preprocessed, including: performing foreground and background segmentation on the original fan blade images in the training set by the maximum inter-class variance method, and extracting the foreground image where the fan blades are located; rotating the foreground image to a vertical direction, and extracting the area where the fan blades are located from the rotated foreground image; and performing sample enhancement on the area where the fan blades are located.

[0017] Optionally, in one embodiment of the present application, the backbone network includes five main convolution modules, wherein the fourth main convolution module and the fifth main convolution module are respectively the first detection layer and the second detection layer in the multiple detection layers, and each detection layer in the multiple detection layers except the first detection layer and the second detection layer includes: a first convolution layer, a first batch normalization layer, a first linear rectification Relu activation function, a second convolution layer, a second batch normalization layer and a second linear rectification Relu activation function, and the convolution kernels in the first convolution layer and the second convolution layer are deformable convolution kernels.

[0018] Optionally, in one embodiment of the present application, a bidirectional feature pyramid structure includes five input nodes, five output nodes, and three intermediate nodes, wherein each input node is connected to a corresponding detection layer, and the bidirectional feature pyramid structure performs feature fusion using the following formula:

[0019]

[0020]

[0021]

[0022]

[0023] in, and They are the input nodes and output nodes of the bidirectional feature pyramid structure, It is the middle node of the bidirectional pyramid structure. and is the learnable weight of each node involved in the operation, Resize() is the size scaling function, Conv is the convolution operation, and ε is a parameter used to ensure the stability of the value, ε = 0.0001.

[0024] Optionally, in one embodiment of the present application, training the object detection network further includes:

[0025] The weight index of the target detection network is smoothed by the exponential moving average (EMA), where the weight exponential smoothing formula is expressed as follows:

[0026] W EMA =λ×W EMA +(1-λ)×W

[0027] Among them, W EMA is the sliding average parameter, W is the currently updated parameter, λ is the decay weight, λ = 0.9998; the label file is smoothed using the following formula to add noise to the true distribution:

[0028] y′=(1-ε)×y+ε / K

[0029] Where y' is the smoothed sample label, ε = 0.1 is the smoothing coefficient, and K is the total number of categories.

[0030] Optionally, in one embodiment of the present application, the method further includes: adopting a Focal Loss function as a classification loss function, adopting a smooth L1 Loss function as a regression loss function, and adding an intersection-over-union loss function (IOU Loss) to the target loss function, and constructing the target loss function by the following formula:

[0031]

[0032] in, is the total loss function, and They represent Focal Loss, smoothL1Loss, and IOU Loss respectively, and θ represents the model parameters.

[0033] To achieve the above objectives, the second embodiment of the present application further proposes a wind turbine blade defect detection system based on an improved SSD model, comprising the following modules:

[0034] A construction module is used to improve the classic detection network SSD model and construct an object detection network including a backbone network, multiple detection layers, a bidirectional feature pyramid structure, and a detection branch. The ResNext network is used as the backbone network.

[0035] a first acquisition module, configured to acquire a wind blade image dataset, annotate defects on the images in the dataset to generate corresponding label files, and divide the annotated wind blade image dataset into a training set, a validation set, and a test set according to a preset ratio;

[0036] A second acquisition module is configured to preprocess the images in the training set, input the preprocessed images into the backbone network to obtain a multi-scale feature map, input the multi-scale feature map processed by the multiple detection layers into the bidirectional feature pyramid structure for feature fusion, and obtain an initial detection result through the detection branch;

[0037] A training module is configured to calculate the loss and update the parameters of the target detection network based on the label file and the initial detection result using a preset target loss function, and iteratively train the target detection network until the target detection network is trained, wherein the target loss function is constructed based on a FocalLoss function, a smooth L1 Loss function, and an intersection-over-union loss function (IOU Loss);

[0038] The detection module is used to input the image of the wind turbine blade to be detected into the trained target detection network and obtain the defect detection result output by the trained target detection network.

[0039] Optionally, in one embodiment of the present application, the training module is further used to: use a stochastic gradient descent SGD optimizer to perform iterative training on the target detection network for a preset number of times until the prediction accuracy of the target detection network for the images in the validation set reaches a preset accuracy threshold; input the images in the test set into the trained target detection network to test the accuracy of the trained target detection network in detecting wind turbine blade defects.

[0040] In order to implement the above-mentioned embodiments, the third aspect embodiment of the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the wind blade defect detection method based on the improved SSD model in the above-mentioned embodiment is implemented.

[0041] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects: the present application first optimizes the original classic detection network SSD model, and replaces VGG with ResNext in the backbone network, making the backbone network of the detection model lightweight while ensuring network performance. In addition, the bidirectional feature pyramid structure is adopted to fully utilize the multi-scale feature feature map while increasing a small amount of computational cost, thereby improving the diversity of feature fusion and fully utilizing multi-scale features to obtain richer feature expressions. In addition, in the loss function part, FocalLoss is used to replace the cross entropy loss function, the smoothL1 Loss function is used as the regression loss function, and the intersection-over-union (IOU) loss function is added for joint optimization. As a result, the present application improves the detection efficiency of the detection network and the accuracy of the detection results while reducing the number of parameters and computational complexity required for blade defect detection.

[0042] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0044] Figure 1 A flow chart of a fan blade defect detection method based on an improved SSD model proposed in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the structure of a target detection network proposed in an embodiment of the present application;

[0046] Figure 3 This is a schematic structural diagram of a detection layer proposed in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of a bidirectional feature pyramid structure proposed in an embodiment of the present application;

[0048] Figure 5 A flowchart of a specific image preprocessing method proposed in an embodiment of the present application;

[0049] Figure 6 A flowchart of a specific method for detecting defects in wind turbine blades based on an improved SSD model proposed in an embodiment of the present application;

[0050] Figure 7 This is a structural diagram of a wind turbine blade defect detection system based on an improved SSD model proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0052] It should be noted that the two-stage detection process in related convolutional neural network-based object detection is relatively complex, often requiring detection algorithms similar to region proposals, resulting in slower detection speeds. The SSD model, on the other hand, leverages the anchor box mechanism of Faster R-CNN, with VGG as the backbone network. It employs multi-scale feature map sampling, where large feature maps are responsible for small object detection, while small feature maps are responsible for large object detection. Combining the advantages of the R-FCN network, the fully connected VGG network is replaced with a fully convolutional network. However, the large number of parameters in the VGG network reduces the overall network inference efficiency, and the related SSD model fails to fully utilize multi-scale features, affecting the network's detection accuracy.

[0053] To this end, this application proposes a fan blade defect detection method based on an improved SSD model, which can improve the accuracy of the target detection network while reducing parameters and calculation amount.

[0054] A method and system for detecting defects in wind turbine blades based on an improved SSD model according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0055] Figure 1 A flow chart of a fan blade defect detection method based on an improved SSD model proposed in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps:

[0056] In step S101, the classic detection network SSD model is improved to construct an object detection network including a backbone network, multiple detection layers, a bidirectional feature pyramid structure, and a detection branch, wherein the ResNext network is used as the backbone network.

[0057] The Single Shot MultiBox Detector (SSD) model is a classic detection network in the field of object detection, belonging to the one-stage detection method. SSD has a direct regression detection algorithm and uses CNN for direct detection. SSD uses prior boxes (i.e., anchors) of different scales and aspect ratios to perform multi-box prediction.

[0058] Specifically, when constructing the target detection network, the improvements made to the SSD model include: replacing the VGG network used in the backbone network of the original SSD model with the ResNext network, and using a bidirectional feature pyramid structure to replace the multi-scale features in the original SSD model. The remaining improvements will be described later. Thus, the present application constructs a target detection network including a backbone network, multiple detection layers, a bidirectional feature pyramid structure, and a detection branch.

[0059] As an example, the application may be constructed as follows Figure 2 The target detection network shown in this example uses ResNext-50 instead of VGG as the backbone network. ResNext is a uniform neural network that reduces the number of network parameters compared to VGG. Compared to networks such as ResNet, the ResNeXt used in this application also reduces the number of hyperparameters, thereby improving the network's reasoning efficiency. Figure 2 In the target detection network shown in the figure, the backbone network includes five main convolution modules (i.e., Conv1 to Conv5 shown in the figure). The feature map output by the previous main convolution module is input into the next main convolution module for feature extraction, thereby generating a new feature map.

[0060] Among them, the fourth main convolution module Conv4 and the fifth main convolution module Conv5 serve as the first detection layer and the second detection layer in the multiple detection layers, and together with the newly added convolution modules: detection layer 3, detection layer 4 and detection layer 5, form the detection layer as a whole. The structure of each detection layer except the first detection layer and the second detection layer in the multiple detection layers, that is, the detection layer 3 to the detection layer 5 is as follows Figure 3 As shown, each detection layer includes: a first convolution layer 10, a first batch normalization layer 20, a first linear rectification ReLU activation function 30, a second convolution layer 40, a second batch normalization layer 50 and a second linear rectification ReLU activation function 60. The convolution kernels in the first convolution layer and the second convolution layer are deformable convolution kernels.

[0061] In this example, the convolution layer of the detection layer is also improved. Specifically, the ordinary convolution kernel in each convolution layer of detection layer 3 to detection layer 5 is replaced with a deformable convolution kernel. Since the deformable convolution adds an additional parameter direction parameter to each element of the convolution kernel, the convolution kernel can be expanded to a large range during training, thereby further improving the performance of the constructed target detection network.

[0062] Furthermore, each detection layer can transmit feature data to a bidirectional feature pyramid structure, and use the bidirectional feature pyramid structure to replace the multi-scale features in the original SSD. In one embodiment of the present application, the constructed bidirectional feature pyramid structure is as follows: Figure 4 As shown in FIG, the bidirectional feature pyramid structure includes five input nodes, five output nodes and three intermediate nodes, wherein each input node is connected to the corresponding detection layer, that is, detection layer 1 to detection layer 5 are the inputs of the bidirectional feature pyramid structure. and Represent the input nodes and output nodes of the bidirectional feature pyramid structure respectively. Represents the middle node of the bidirectional pyramid structure. Among them, the semantic information of high-level features is transmitted from D5 to D1 from top to bottom, and the position information of low-level features is transmitted from D1 to D5 from bottom to top. Also fused input nodes This can enrich the diversity of feature fusion while increasing a small amount of computational cost. Figure 4 From the structure shown, it can be seen that when the bidirectional feature pyramid structure performs feature fusion calculation, feature fusion can be performed using the following formula:

[0063]

[0064]

[0065]

[0066]

[0067] in, and It is the learnable weight of each node involved in the operation. Resize() is a resizing function. For example, it can double or reduce the image by two times. Conv is a convolution operation. ε=0.0001 is to ensure the stability of the value.

[0068] Furthermore, the bidirectional feature pyramid structure can input the fused multi-scale features into the detection branch. In one embodiment of the present application, the constructed detection branch can be composed of two convolutional layers, for example, a 3×3 convolutional layer -> a 1×1 convolutional layer. The predicted value output by each layer is expressed as A×(K+4), where A is the number of anchor boxes in that layer, K is the number of categories, and 4 is the predicted value of the horizontal and vertical coordinates, width, and height of the center of each anchor box.

[0069] Therefore, this application improves the original SSD model and constructs a target detection network. After subsequent training, detection through this network can reduce parameters and calculation amount.

[0070] It should be noted that the above Figure 2 The target detection network shown is only a non-limiting example. In this application, target detection networks with different structures can be constructed according to actual needs. For example, the number of main convolutional modules and detection layers can be adjusted according to the needs of detection accuracy, or a backbone network can be constructed based on the original VGG network combined with networks such as ResNet and InceptionNetworks, combining the repetition strategy of ResNet with the split-convert-merge strategy of InceptionNet to further improve the performance and portability of the detection network.

[0071] Step S102 , obtaining a wind blade image dataset, annotating defects on the images in the dataset to generate corresponding label files, and dividing the annotated wind blade image dataset into a training set, a validation set, and a test set according to a preset ratio.

[0072] Among them, the fan blade image dataset is a training dataset required for training the constructed target detection network. The fan blade image dataset contains a large number of fan blade images, and different fan blade images may contain various types of defects.

[0073] In specific implementations, wind turbine blade image datasets can be obtained in different ways. As an example, a Supervisory Control and Data Acquisition (SCADA) system can be pre-installed in the wind turbine. During the operation of the wind turbine, the SCADA system can be used to monitor the equipment of the wind turbine at the operation site to collect data on the wind turbine blades. The wind turbine blade image dataset is generated by collecting historical image data of each wind turbine blade over different time periods. As another example, wind turbine blade image data can be obtained from network resources and filtered to obtain the wind blade image dataset.

[0074] Furthermore, the defects in the wind blade images in the dataset are labeled according to predefined categories, and corresponding label files are generated. In one embodiment of the present application, the defects in each image in the dataset can be manually labeled using image processing software, accurately marking which category of the preset defect types the blade defects in each image belong to, thereby obtaining a corresponding number of label files. The labeled wind blade image dataset is then processed to remove erroneous data and then divided into a training set, a validation set, and a test set according to a preset ratio, for example, an 8:1:1 principle.

[0075] Step S103: preprocess the images in the training set, input the preprocessed images into the backbone network to obtain multi-scale feature maps, input the multi-scale feature maps processed by multiple detection layers into the bidirectional feature pyramid structure for feature fusion, and obtain the initial detection results through the detection branch.

[0076] Among them, preprocessing is to reduce the interference of wind turbine blade images on defect identification and facilitate the processing operation of extracting blade defects.

[0077] In one embodiment of the present application, in order to more clearly illustrate the process of preprocessing the images in the training set, the present application also proposes a specific image preprocessing method, such as Figure 5 As shown, the method includes the following steps:

[0078] Step S501 , performing foreground and background segmentation on the original wind blade images in the training set by using the maximum inter-class variance method, and extracting the foreground image where the wind blades are located.

[0079] Specifically, in this embodiment, since the complex information in the background of the image may interfere with the network model's extraction of blade defects, the wind blades in the image are first separated from the background by methods such as the maximum inter-class variance method, and the foreground image where the wind blades are located is extracted, which is beneficial to improving the accuracy of detection.

[0080] Step S502 : rotating the foreground image to a vertical direction, and extracting the area where the wind turbine blades are located from the rotated foreground image.

[0081] Among them, rotating the extracted wind turbine blade image to the vertical direction can reduce the interference caused by oblique defects and further improve the accuracy of detection.

[0082] In specific implementations, the minimum enclosing rectangle of the blade is calculated before the blade's angle is calculated based on the rectangle's long and short sides. The difference between the blade's current angle and the vertical angle is then used to determine the rotation angle and direction, thereby rotating the image to a vertical orientation. Furthermore, the defect annotation box generated during the synchronous rotation annotation process, when the blade is in a vertical orientation after rotation, cuts out the area in the foreground image that only contains the blade, and simultaneously translates the defect annotation box.

[0083] Step S503: Perform sample enhancement on the area where the wind turbine blades are located.

[0084] Specifically, the image of the leaf area is randomly rotated by n*90° (n=0, 1, 2, 3), randomly flipped by 90° horizontally and vertically, and the brightness, contrast and saturation of the image are enhanced with random adjustment parameters.

[0085] Therefore, the preprocessing method can reduce the interference of the wind turbine blade image on defect identification, which is more conducive to the subsequent extraction of blade defects in the image.

[0086] Furthermore, after preprocessing the images in the training set, the fan blade images in the training set are input into the constructed target detection model for training. During the training process, the parameters of the target detection model are optimized through the predetermined model training hyperparameters and loss function.

[0087] In a specific implementation, the pre-processed image is first input into the backbone network to obtain a multi-scale feature map. As described in the above embodiment, the feature map output by the previous main convolution module in the multiple convolution modules of the backbone network can be input into the next main convolution module for feature extraction, thereby generating a new feature map. Then, part of the feature map in the generated multi-scale feature map, that is, the multi-scale feature map processed by any detection layer in the entire feature map, is input into the bidirectional feature pyramid structure for feature fusion. For example, Figure 2 The multi-scale feature maps processed by detection layers 1 to 5 shown in the figure are input into the bidirectional feature pyramid. After feature fusion is performed through the feature fusion calculation formula shown in step S101, the initial detection result is obtained through the detection branch. The initial detection result includes the defect type of the blade image in the training set currently judged by the target detection network.

[0088] In step 104, based on the label file and the initial detection results, the loss is calculated and the parameters of the target detection network are updated through a preset target loss function, and the target detection network is iteratively trained until the target detection network is trained. The target loss function is constructed based on the FocalLoss function, the smooth L1 Loss function, and the intersection-over-union loss function (IOU Loss).

[0089] Specifically, the target detection network outputs prediction results for images in the training set and the label files obtained through annotation. The loss function is used to calculate the loss between the prediction results and the true values. The parameters of the target detection network are updated by backpropagation by minimizing the loss. The target detection network is then iteratively trained until the training is complete.

[0090] In one embodiment of the present application, the target detection network is iteratively trained until the target detection network is trained, including using a stochastic gradient descent (SGD) optimizer to iteratively train the target detection network for a preset number of times until the prediction accuracy of the target detection network's prediction results for the images in the validation set reaches a preset accuracy threshold.

[0091] For example, when determining model training hyperparameters and loss functions, optimizing the parameters of the object detection model, and iteratively training until the prediction accuracy of the validation set reaches a preset accuracy threshold, the backbone network can be pre-trained on the ImageNet dataset, using the Xavier initialization strategy for the bidirectional feature pyramid structure and detection branch. Training uses the SGD optimizer for 150,000 iterations, with an initial learning rate of 0.01. The learning rate is reduced by a factor of 10 at iterations 50,000 and 120,000. The L2 regularization coefficient is set to 0.0005, the momentum is set to 0.9, the training scale ranges from 320 to 608, and the batch size is 16. If images in the validation set are input to the object detection network for prediction and the prediction accuracy reaches the preset accuracy threshold, the object detection network training is considered complete and iterations are terminated. The preset accuracy threshold can be set based on the actual detection accuracy requirements.

[0092] Furthermore, in order to ensure that the trained model meets the requirements, in one embodiment of the present application, after the target detection network is trained, the images in the test set can also be input into the trained target detection network to test the accuracy of the trained target detection network in detecting fan blade defects. By comparing whether the accuracy is greater than the preset minimum accuracy threshold, it is verified whether the accuracy of the model in detecting fan defects meets the requirements.

[0093] In one embodiment of the present application, the network weight index and the label file can also be smoothed during the training process. Specifically, the weight index of the target detection network is smoothed by the exponential moving average (EMA), wherein the weight index smoothing formula is expressed as follows:

[0094] W EMS =λ×W EMA +(1-λ)×W

[0095] Among them, W EMA is the sliding average parameter, W is the currently updated parameter, and λ is the decay weight, using λ = 0.9998. In addition, the label file is smoothed using the following formula:

[0096] y′=(1-ε)×y+ε / K

[0097] Here, y' is the smoothed sample label, ε = 0.1 is the smoothing coefficient, and K is the total number of categories. Thus, by using exponential smoothing of network weights, the robustness of the trained object detection network is improved. Label smoothing is also performed. The smoothed label distribution is equivalent to adding noise to the true distribution, thus avoiding model overfitting and improving the generalization performance of the trained object detection network.

[0098] It should be noted that the target loss function preset during model training in this application is constructed based on the FocalLoss function, the smooth L1 Loss function, and the intersection-over-union loss function IOU Loss. Among them, this application is based on the improved SSD model to construct and train the target detection network for single-stage detection. Usually, in the single-stage detection process, the accuracy of defect detection may be lower than that of two-stage detection due to the imbalance of sample categories. The applicant found that for single-stage detection, since the number of negative samples is too large, accounting for the majority of the overall loss and a large part of them are samples that are easy to classify, the optimization direction of the model may not be fully optimized in the expected direction during the model training process.

[0099] To this end, in an embodiment of the present application, the total loss function of the target detection network, i.e., the target loss function, is constructed by the FocalLoss function. The FocalLoss function can reduce the weight of easily classified samples, so that the target detection network is more focused on difficult-to-classify samples during training, thereby improving the training effect of the model. The smooth L1 Loss function is used as the regression loss function, and the intersection of union (IOU) loss function is added to jointly optimize the target detection network to further improve the prediction accuracy of the target detection network.

[0100] In specific implementation, as a possible implementation method, the Focal Loss function can be used as the classification loss function, the smooth L1 Loss function can be used as the regression loss function during the training process, and the intersection-over-union loss function (IOU Loss) can be added to the target loss function. The target loss function is constructed using the following formula:

[0101]

[0102] in, is the total loss function, They represent Focal Loss, smooth L1Loss, and IOU Loss respectively, and θ represents the model parameters.

[0103] Therefore, the present application trains the target detection network by using the target loss function, which can improve the accuracy of the target detection network detection without affecting the original detection speed of the target detection network. The target detection network is trained by the training method shown in step S103 and step S104.

[0104] Step 105 : Input the image of the wind turbine blade to be inspected into the trained target detection network to obtain the defect detection result output by the trained target detection network.

[0105] Specifically, after the target detection model is trained, the wind turbine blade to be inspected is photographed, and the image of the wind turbine blade to be inspected is input into the trained target detection model. The output result of the detection model is the result of defect detection on the photographed wind turbine blade image.

[0106] In order to more fully demonstrate that the wind blade defect detection method based on the improved SSD model in the embodiment of the present application improves the accuracy and speed of wind blade defect detection, the following is an explanation based on the detection result data obtained in actual applications.

[0107] Specifically, using manually annotated images as a reference, we compared the accuracy and efficiency of our method with three object detection networks: Faster R-CNN, YOLO-V3, and SSD. Table 1 shows the comparison results, using Mean Average Precision (MAP) as a metric to measure the accuracy of the object detection network. Table 2 also shows the comparison results, using the number of images processed per second under the same experimental environment (Tesla V100) as a metric to measure the speed of the object detection network.

[0108]

[0109] Table 1

[0110] As can be seen from Table 1, the MAP value of the improved SSD network of this application is the highest, reaching 82.1, which is an increase of 13.2 compared to the MAP value of 68.9 of the SSD network.

[0111]

[0112] Table 2

[0113] As shown in Table 2, while the accuracy is improved by 13.2, the speed of the improved SSD network in this application can still reach 31 frames per second. In terms of comprehensive accuracy and efficiency comparison, the improved SSD network in this application is superior to other methods.

[0114] In summary, the wind turbine blade defect detection method based on the improved SSD model in the embodiment of the present application first optimizes the original classic detection network SSD model, and replaces VGG with ResNext in the backbone network, making the backbone network of the detection model lightweight while ensuring network performance. In addition, the bidirectional feature pyramid structure is adopted to fully utilize the multi-scale feature feature map while increasing a small amount of computational cost, thereby improving the diversity of feature fusion and obtaining richer feature expressions by fully utilizing multi-scale features. In addition, in the loss function part, FocalLoss is used instead of the cross entropy loss function, the smooth L1 Loss function is used as the regression loss function, and the intersection-over-union (IOU) loss function is added for joint optimization. Therefore, this method improves the detection efficiency of the detection network and the accuracy of the detection results while reducing the number of parameters and computational complexity required for blade defect detection.

[0115] In order to more clearly illustrate the implementation process of the wind blade defect detection method based on the improved SSD model in the embodiment of the present application, a specific embodiment of the wind blade defect detection method based on the improved SSD model is described in detail below. Figure 6 This is a flowchart of a specific method for detecting fan blade defects based on an improved SSD model proposed in an embodiment of the present application. Figure 6 As shown, the method includes the following steps:

[0116] Step S601: construct a target detection network, wherein the target detection network is improved based on the classic target detection network SSD.

[0117] In this step, a target detection network is constructed, which includes a backbone network, a detection layer, a bidirectional feature pyramid structure, and a detection branch.

[0118] Step S602: training the target detection network using the wind turbine blade image dataset.

[0119] In this step, the steps of training the target detection network using the wind turbine blade image dataset include:

[0120] A wind blade image dataset was obtained, and the defects of the wind blade images in the dataset were marked according to the defined categories. The corresponding label files were generated and divided into training set, validation set and test set according to the 8:1:1 principle after preprocessing.

[0121] The fan blade images in the training set are input into the target detection model for training, the hyperparameters and loss function of the model training are determined, and the parameters of the target detection model are optimized until the prediction accuracy of the validation set prediction results reaches the preset accuracy threshold.

[0122] The test set is input into the trained target detection model to verify the accuracy of the model in detecting fan defects.

[0123] The object detection model is trained by preprocessing the training set of wind turbine blade images. After preprocessing, the images are fed into the backbone network to generate multi-scale feature maps. Part of these feature maps (the detection layer) are then fed into a bidirectional feature pyramid structure, which is then used by the detection branch to generate the network's detection results. The network parameters are updated based on the loss between the labeled data and the network output. During training, exponential smoothing of the network weights is used to improve network robustness, while label smoothing improves generalization performance.

[0124] In step S603 , the captured wind turbine blade image is input into the trained target detection model, and the output result is the defect detection result of the captured wind turbine blade image.

[0125] In order to implement the above embodiment, the present application also proposes a wind turbine blade defect detection system based on an improved SSD model. Figure 7 This is a structural diagram of a fan blade defect detection system based on an improved SSD model proposed in an embodiment of the present application, as shown in FIG. Figure 7 As shown, the system includes a construction module 100 , a first acquisition module 200 , a second acquisition module 300 , a training module 400 and a detection module 500 .

[0126] Among them, the construction module 100 is used to improve the classic detection network SSD model and construct a target detection network including a backbone network, multiple detection layers, a bidirectional feature pyramid structure and a detection branch, wherein the ResNext network is used as the backbone network.

[0127] The first acquisition module 200 is used to acquire a wind blade image dataset, annotate defects on the images in the dataset to generate corresponding label files, and divide the annotated wind blade image dataset into a training set, a validation set, and a test set according to a preset ratio.

[0128] The second acquisition module 300 is used to preprocess the images in the training set, input the preprocessed images into the backbone network to obtain multi-scale feature maps, input the multi-scale feature maps processed by multiple detection layers into the bidirectional feature pyramid structure for feature fusion, and obtain the initial detection results through the detection branch.

[0129] The training module 400 is used to calculate the loss and update the parameters of the target detection network based on the label file and the initial detection results through a preset target loss function, and iteratively train the target detection network until the target detection network is trained. The target loss function is constructed based on the FocalLoss function, the smooth L1 Loss function and the intersection-over-union loss function IOULoss.

[0130] The detection module 500 is used to input the image of the wind turbine blade to be detected into the trained target detection network, and obtain the defect detection result output by the trained target detection network.

[0131] Optionally, in one embodiment of the present application, the training module 400 is further used to: use a stochastic gradient descent SGD optimizer to perform iterative training on the target detection network for a preset number of times until the prediction accuracy of the target detection network's prediction results for the images in the validation set reaches a preset accuracy threshold; input the images in the test set into the trained target detection network to test the accuracy of the trained target detection network in detecting defects in wind turbine blades.

[0132] Optionally, in one embodiment of the present application, the second acquisition module 300 is also used to: perform foreground and background segmentation on the original fan blade image in the training set by the maximum inter-class variance method, and extract the foreground image where the fan blade is located; rotate the foreground image to a vertical direction, and extract the area where the fan blade is located from the rotated foreground image; and perform sample enhancement on the area where the fan blade is located.

[0133] Optionally, in one embodiment of the present application, the backbone network includes five main convolution modules, wherein the fourth main convolution module and the fifth main convolution module are respectively the first detection layer and the second detection layer in the multiple detection layers, and each detection layer in the multiple detection layers except the first detection layer and the second detection layer includes: a first convolution layer, a first batch normalization layer, a first linear rectification Relu activation function, a second convolution layer, a second batch normalization layer and a second linear rectification Relu activation function, and the convolution kernels in the first convolution layer and the second convolution layer are deformable convolution kernels.

[0134] Optionally, in one embodiment of the present application, a bidirectional feature pyramid structure includes five input nodes, five output nodes, and three intermediate nodes, wherein each input node is connected to a corresponding detection layer, and the bidirectional feature pyramid structure performs feature fusion using the following formula:

[0135]

[0136]

[0137]

[0138]

[0139] in, and They are the input nodes and output nodes of the bidirectional feature pyramid structure, It is the middle node of the bidirectional pyramid structure. and is the learnable weight of each node involved in the operation, Resize() is the size scaling function, Conv is the convolution operation, and ε=0.0001 is to ensure the stability of the value.

[0140] Optionally, in one embodiment of the present application, the training module 400 is further configured to:

[0141] The weight index of the target detection network is smoothed by the exponential moving average (EMA), where the weight exponential smoothing formula is expressed as follows:

[0142] W EMA =λ×W EMA +(1-λ)×W

[0143] Among them, W EMA is the sliding average parameter, W is the currently updated parameter, λ is the decay weight, and λ = 0.9998 is used; the label file is smoothed using the following formula:

[0144] y′=(1-ε)×y+ε / K

[0145] Where y' is the smoothed sample label, ε = 0.1 is the smoothing coefficient, and K is the total number of categories.

[0146] Optionally, in one embodiment of the present application, the training module 400 is further configured to: use the Focal Loss function as the classification loss function, use the Smooth L1 Loss function as the regression loss function, and add the Intersection-over-Union loss function (IOU Loss) to the target loss function, and construct the target loss function using the following formula:

[0147]

[0148] in, is the total loss function, They represent Focal Loss, smooth L1Loss, and IOU Loss respectively, and θ represents the model parameters.

[0149] It should be noted that the above explanation of the embodiment of the wind turbine blade defect detection method based on the improved SSD model is also applicable to the system of this embodiment and will not be repeated here.

[0150] In summary, the wind turbine blade defect detection system based on the improved SSD model in the embodiment of the present application first optimizes the original classic detection network SSD model, and uses ResNext instead of VGG in the backbone network, making the backbone network of the detection model lightweight while ensuring network performance. In addition, the bidirectional feature pyramid structure is adopted to fully utilize the multi-scale feature feature map while increasing a small amount of computational cost, thereby improving the diversity of feature fusion and obtaining richer feature expressions by fully utilizing multi-scale features. In addition, in the loss function part, FocalLoss is used instead of the cross entropy loss function, the smooth L1 Loss function is used as the regression loss function, and the intersection-over-union (IOU) loss function is added for joint optimization. As a result, the system improves the detection efficiency of the detection network and the accuracy of the detection results while reducing the number of parameters and computational complexity required for blade defect detection.

[0151] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the wind blade defect detection method based on the improved SSD model as described in any of the above embodiments.

[0152] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0153] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0154] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0155] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0156] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0157] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0158] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0159] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A fan blade defect detection method based on an improved SSD model, characterized in that: The following steps are involved: The classic detection network SSD model is improved to construct an object detection network including a backbone network, multiple detection layers, a bidirectional feature pyramid structure, and a detection branch. The ResNext network is used as the backbone network, including: replacing the VGG network used in the backbone network of the original SSD model with the ResNext network, and using a bidirectional feature pyramid structure to replace the multi-scale features in the original SSD model; Obtaining a wind blade image dataset, annotating defects on the images in the dataset to generate corresponding label files, and dividing the annotated wind blade image dataset into a training set, a validation set, and a test set according to a preset ratio; Preprocessing the images in the training set, inputting the preprocessed images into the backbone network to obtain a multi-scale feature map, inputting the multi-scale feature map processed by the multiple detection layers into the bidirectional feature pyramid structure for feature fusion, and obtaining an initial detection result through the detection branch; Based on the label file and the initial detection result, the loss is calculated and the parameters of the target detection network are updated by a preset target loss function, and the target detection network is iteratively trained until the training of the target detection network is completed, wherein the target loss function is constructed based on the FocalLoss function, the smooth L1 Loss function and the intersection-over-union loss function (IOU Loss); Inputting the current wind turbine blade image to be inspected into the trained target detection network to obtain the defect detection result output by the trained target detection network; The backbone network includes five main convolution modules, wherein the fourth main convolution module and the fifth main convolution module are respectively the first detection layer and the second detection layer in the multiple detection layers, and each detection layer in the multiple detection layers except the first detection layer and the second detection layer includes: a first convolution layer, a first batch normalization layer, a first linear rectification Relu activation function, a second convolution layer, a second batch normalization layer and a second linear rectification Relu activation function, and the convolution kernels in the first convolution layer and the second convolution layer are deformable convolution kernels.

2. The detection method according to claim 1, wherein The iterative training of the target detection network until the training of the target detection network is completed includes: The object detection network is iterated for a preset number of times using a stochastic gradient descent (SGD) optimizer until the prediction accuracy of the object detection network for the images in the validation set reaches a preset accuracy threshold; After the training of the target detection network is completed, the method further includes: The images in the test set are input into the trained target detection network to test the accuracy of the trained target detection network in detecting defects in wind turbine blades.

3. The detection method according to claim 1 or 2, characterized in that The preprocessing of the images in the training set includes: Performing foreground and background segmentation on the original fan blade images in the training set by using the maximum inter-class variance method to extract the foreground image where the fan blades are located; Rotating the foreground image to a vertical direction, and extracting the area where the wind blades are located from the rotated foreground image; The area where the fan blades are located is sample enhanced.

4. The method according to claim 1, wherein The bidirectional feature pyramid structure includes five input nodes, five output nodes, and three intermediate nodes, wherein each input node is connected to a corresponding detection layer. The bidirectional feature pyramid structure performs feature fusion using the following formula: in, and They are the input nodes and output nodes of the bidirectional feature pyramid structure, It is the middle node of the bidirectional pyramid structure. and is the learnable weight of each node involved in the operation, Resize() is the size scaling function, Conv is the convolution operation, and ε is a parameter used to ensure the stability of the value, ε = 0.0001.

5. The method according to claim 1, wherein The training of the target detection network further includes: The weight index of the target detection network is smoothed by the exponential moving average (EMA), where the weight exponential smoothing formula is expressed as follows: W EMA =λ×W EMA +(1-λ)×W Among them, W EMA is the sliding average parameter, W is the current updated parameter, λ is the decay weight, λ = 0.9998; The label file is smoothed using the following formula to add noise to the true distribution: y′=(1-ε)×y+ε / K Where y' is the smoothed sample label, ε = 0.1 is the smoothing coefficient, and K is the total number of categories.

6. The method according to claim 1, characterized in that Also includes: The Focal Loss function is used as the classification loss function, the Smooth L1 Loss function is used as the regression loss function, and the Intersection-Union Loss function (IOU Loss) is added to the target loss function. The target loss function is constructed using the following formula: in, is the total loss function, and They represent Focal Loss, smoothL1Loss, and IOU Loss respectively, and θ represents the model parameters.

7. A fan blade defect detection system based on an improved SSD model, characterized in that: include: A construction module is used to improve the classic detection network SSD model and construct an object detection network including a backbone network, multiple detection layers, a bidirectional feature pyramid structure, and a detection branch. The ResNext network is used as the backbone network, including: replacing the VGG network used in the backbone network of the original SSD model with the ResNext network, and using a bidirectional feature pyramid structure to replace the multi-scale features in the original SSD model; a first acquisition module, configured to acquire a wind blade image dataset, annotate defects on the images in the dataset to generate corresponding label files, and divide the annotated wind blade image dataset into a training set, a validation set, and a test set according to a preset ratio; A second acquisition module is configured to preprocess the images in the training set, input the preprocessed images into the backbone network to obtain a multi-scale feature map, input the multi-scale feature map processed by the multiple detection layers into the bidirectional feature pyramid structure for feature fusion, and obtain an initial detection result through the detection branch; A training module is configured to calculate the loss and update the parameters of the target detection network based on the label file and the initial detection result using a preset target loss function, and iteratively train the target detection network until the target detection network is trained, wherein the target loss function is constructed based on a FocalLoss function, a smooth L1 Loss function, and an intersection-over-union loss function (IOU Loss); A detection module is used to input the image of the wind turbine blade to be detected into the trained target detection network and obtain the defect detection result output by the trained target detection network; The backbone network includes five main convolution modules, wherein the fourth main convolution module and the fifth main convolution module are respectively the first detection layer and the second detection layer in the multiple detection layers, and each detection layer in the multiple detection layers except the first detection layer and the second detection layer includes: a first convolution layer, a first batch normalization layer, a first linear rectification Relu activation function, a second convolution layer, a second batch normalization layer and a second linear rectification Relu activation function, and the convolution kernels in the first convolution layer and the second convolution layer are deformable convolution kernels.

8. The detection system according to claim 7, characterized in that: The training module is further used to: The object detection network is iterated for a preset number of times using a stochastic gradient descent (SGD) optimizer until the prediction accuracy of the object detection network for the images in the validation set reaches a preset accuracy threshold; The images in the test set are input into the trained target detection network to test the accuracy of the trained target detection network in detecting defects in wind turbine blades.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the wind turbine blade defect detection method based on the improved SSD model as described in any one of claims 1 to 6 is implemented.

Images