Offshore wind turbine blade surface early defect identification method based on improved OfficientNet

Abstract

The invention belongs to the technical field of computer vision, and particularly provides an improved OfficientNet-based offshore wind turbine blade surface early-stage defect identification method, which comprises the steps of constructing a network model, selecting an OfficientNet as a feature extraction network and improving the feature extraction network, realizing feature extraction and processing in combination with an improved bidirectional feature pyramid structure, and identifying the early-stage defect of an offshore wind turbine blade surface based on the improved OfficientNet. In the detection part, Focal-EIOU and Focal Loss loss functions are introduced to calculate position loss and classification loss; constructing an offshore wind turbine blade image data set, marking defects on the image, and constructing a training set, a verification set and a test set based on the data set; training a network model based on the manufactured image data set to obtain an early defect identification model of the surface of the offshore wind turbine blade; and inputting an offshore wind turbine blade image to be identified into the offshore wind turbine blade image identification model to obtain an offshore wind turbine blade defect detection result. According to the method, early-stage defect detection and classification work of the offshore wind turbine blade can be realized, and overhaul and maintenance work of the blade is facilitated.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular, relates to an early defect recognition method for offshore wind turbine blade surfaces based on an improved EfficientNet. Background Art

[0002] Offshore wind power equipment is located in a harsh environment, and the equipment is susceptible to environmental influences such as hot and humid air, ultraviolet rays, high salt fog and sea breeze corrosion. With the continuous increase in the cumulative installed capacity of offshore wind power, the operation and maintenance of offshore wind power equipment has received widespread attention. In order to reduce maintenance time and cost, offshore wind power blade inspection work should focus on early warning of damage before defects spread.

[0003] Traditional manual inspection methods not only require high professional level of inspection workers, but also have very low inspection efficiency. Previously, real-time monitoring technologies for wind turbine blades mainly included acoustic emission monitoring, vibration signal monitoring and strain monitoring; offline inspection technologies mainly included ultrasonic scanning and infrared imaging. With the development of drone technology, the method of using drones to collect images and then analyze blade defects has become mainstream. This method can not only reduce costs, but also provide more intuitive inspection results compared with acoustic emission inspection, vibration inspection and other methods. At present, target detection algorithms are mainly divided into two categories: two-stage and one-stage. The two-stage algorithm first generates a large number of candidate frames and then classifies the candidate frames, such as Fast R-CNN and Faster R-CNN. One-stage target detection regards detection as a regression problem, which can achieve high detection accuracy and speed at the same time. Commonly used one-stage target detection includes YOLO series and SSD. The two-stage algorithm can achieve high detection accuracy, but the detection speed is much lower than that of the one-stage algorithm. Therefore, in practical applications, the one-stage algorithm is more widely used.

[0004] The size of early defects in offshore wind turbine blades is relatively small. The above algorithms have certain difficulties in feature extraction, and the accuracy of defect detection and classification needs to be improved. The current offshore wind turbine blade defect detection algorithm has the problems of high model complexity and low accuracy of small-size defect detection. Summary of the invention The technical problem to be solved by the present invention is to provide a method for identifying early defects on the surface of offshore wind turbine blades based on improved EfficientNet, which can realize early defect detection and classification of offshore wind turbine blades, and is beneficial to the inspection and maintenance of blades.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for identifying early defects on the surface of offshore wind turbine blades based on improved EfficientNet, comprising the following steps: Step 1: Build a network model Improve the design of the EfficientNet backbone feature extraction network: introduce asymmetric convolution to replace the ordinary 3×3 convolution in the EfficientNet backbone feature extraction network; Feature fusion module design: Design a hybrid spatial channel attention module to focus on spatial and channel information, and combine the BiFPN feature fusion module to perform feature fusion on semantic information of different depths; Prediction module design: Calculate the loss and update the parameters of the target detection network through a preset target loss function, and iteratively train the target detection network until the target detection network is trained; Step 2: Construct an offshore wind turbine blade image dataset; Step 3: training a network model based on the prepared image data set to obtain an offshore wind turbine blade surface early defect recognition model; Step 4: input the offshore wind turbine blade image to be identified into the offshore wind turbine blade image recognition model to obtain the offshore wind turbine blade defect detection result.

[0006] In the preferred solution, in step 1, the EfficientNet backbone feature extraction network is improved and designed, including: The backbone network consists of 8 main convolutional modules, of which module 1 is a convolutional layer containing 3×3 convolution, BN layer, and Swish activation function. Modules 2 to 8 are MBConv structures with different parameters, and each MBConv structure contains several repeated MBConv modules. In the MBConv module, the first part is a 1×1 convolution, and MBConvn represents that the number of output channels after the 1×1 convolution is expanded to n times the number of input channels; the second part is a depthwise separable convolution; the third part is an SE attention mechanism module; the fourth part is to use convolution to reduce the number of feature channels; and the fifth part is a Dropout layer.

[0007] In a preferred solution, in step 1, the feature fusion module design includes: The 1st to 3rd layer inputs of the feature fusion module are the outputs of the 6th to 8th MBConv modules of the backbone network respectively; the feature fusion module includes a first layer fusion output, a second layer intermediate output, a second layer fusion output, and a third layer fusion output, the first layer fusion output part receives the first layer input and the second layer intermediate output and fuses the two parts of features to output, the second layer intermediate output part receives the second layer input and the third layer input and fuses the two parts of features to output, the second layer fusion output part receives the second layer input and the second layer intermediate output and fuses the two parts of features to output, and the third layer fusion output part receives the third layer input and the second layer fusion output and fuses the two parts of features to output; The outputs of layers 1 to 3 of the feature fusion module are respectively connected to a hybrid attention mechanism, which adds a double pooling layer in front of the SE module, introduces a spatial attention mechanism SAM on this basis, and links the improved SE module and SAM module in parallel.

[0008] In a preferred solution, in the step 1, the target loss function is based on the Focal-EIOU and Focal Loss loss functions, and the Focal-EIOU loss function and the Focal loss loss function are selected to calculate the position loss and classification loss of the prediction box.

[0009] In a preferred solution, the objective loss function is constructed by the following formula: (1); in, , , They represent IOU loss, distance loss, and width-height loss respectively. IOU is the intersection-over-union ratio, which is used to measure the gap between the predicted border and the actual border. It is the ratio of the intersection area of ​​the predicted box and the actual box to the union area. ; w , h Respectively represent the width and height of the predicted border; b , Represent the center point coordinates of the predicted box and the actual box respectively, represents the Euclidean distance between the two. , Represents the width and height of the minimum enclosing rectangle of the predicted box and the true box; Introducing a parameter that can control the degree of suppression , the final Focal-EIOU formula is: (2); The expression of the Focal loss function is: (3); (4); Among them, y represents the actual category of the target, and its value is 0 or 1. p is the target prediction category, and its value is an interval value in [0, 1]. is used to control the weights of positive and negative samples. is the weight used to control the difficulty and ease of classification samples.

[0010] In the preferred scheme, in the step 2, a drone is used to collect defect images of offshore wind turbine blades, and annotation software is used to annotate the defects in the data set to generate an annotation file containing defect categories and prediction box coordinate information. The image data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0011] In a preferred solution, in step three, training the network model based on the prepared image dataset includes the following steps: 1) Use the stochastic gradient descent SGD optimizer to iteratively train the target detection network for a preset number of iterations until the prediction accuracy of the target detection network for the images in the validation set reaches a preset accuracy threshold, at which point a preliminary model is obtained; 2) The recall rate, accuracy rate and precision rate of the preliminary model are measured based on the test set. If the recall rate, accuracy rate and precision rate all meet the preset standards, the preliminary model is determined to be a recognition model.

[0012] In a preferred solution, the number of iterations of the iterative training is set to 500, the batch size is 4, and the initial learning rate is 0.01.

[0013] In a preferred embodiment, in step four, surface defects of offshore wind turbine blades are detected, and the image of the wind turbine blade to be detected is input into a trained offshore wind turbine blade surface early defect recognition model to obtain defect detection results output by the trained target detection network.

[0014] The present invention provides an offshore wind turbine blade surface early defect recognition method based on improved EfficientNet, which has the following beneficial effects: 1. In view of the problems of high model complexity and low accuracy of small-size defect detection in the current offshore wind turbine blade defect detection algorithm, the present invention proposes an offshore wind turbine blade early defect detection method, which can realize the early defect detection and classification of offshore wind turbine blades, and is beneficial to the inspection and maintenance of blades.

[0015] 2. The offshore wind turbine blade surface early defect detection model established by the present invention takes into account both detection speed and accuracy and has good practicality.

[0016] 3. In step 1, the designed target detection model introduces asymmetric convolution in the EfficientNet feature extraction network to enhance the convolution kernel skeleton information and improve the network's ability to extract defect information; a hybrid spatial channel attention module is proposed to focus on spatial and channel information, combined with the BiFPN feature fusion module to enhance the algorithm's ability to detect small-scale defects; Focal-EIOU and Focal Loss loss functions are introduced to calculate position loss and classification loss, improve positioning accuracy, and solve the problem of imbalance in the proportion of positive and negative image samples during model training.

[0017] 4. In step 2, a self-made image dataset is created through manual annotation methods and divided into training set, validation set and test set in proportion to reduce the risk of model overfitting.

[0018] 5. In step 3, the stochastic gradient descent SGD optimizer is used to iterate the target detection network for a preset number of iterations, which can improve the convergence speed during model training.

[0019] 6. In step 4, when inspecting the image of the wind turbine blade to be inspected, the model not only improves the accuracy but also has a faster inspection speed, which is of great significance to the blade inspection work. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 is a flow chart of the present invention; Figure 2 MBConv structure diagram; Figure 3 This is the principle diagram of asymmetric convolution; Figure 4 It is a weighted bidirectional feature pyramid structure diagram; Figure 5 Schematic diagram of the hybrid attention mechanism; Figure 6 This is the schematic diagram of the prediction module; Figure 7 This is the overall network structure diagram. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] The present invention provides a method for identifying early defects on the surface of offshore wind turbine blades based on improved EfficientNet, such as Figure 1 As shown, the following steps are included: Step 1: Build a network model: Introduce asymmetric convolution to replace ordinary 3×3 convolution in the EfficientNet backbone feature extraction network; design a hybrid spatial channel attention module to focus on spatial and channel information, and combine the BiFPN feature fusion module to perform feature fusion on semantic information of different depths; calculate the loss and update the parameters of the target detection network through a preset target loss function, and iterate the target detection network until the target detection network is trained, wherein the target loss function is based on the Focal-EIOU and Focal Loss loss functions. Specifically, it includes the following steps: Step 1.1, improve the EfficientNet backbone feature extraction network. Considering the situation of the dataset samples, choose the EfficientNet-B2 version as the basic model, remove all layers after the last MBConv6, and only keep the feature extraction layer to focus on feature extraction; The backbone network consists of 8 main convolutional modules, of which module 1 is a convolutional layer containing 3×3 convolution, BN layer, and Swish activation function. Modules 2 to 8 are MBConv structures with different parameters, and each MBConv structure contains several repeated MBConv modules. In the MBConv module, the first part is 1×1 convolution, MBConvn represents that the number of output channels after 1×1 convolution is expanded to n times of the input channels; the second part is depthwise separable convolution; the third part is SE attention mechanism module; the fourth part is to use convolution to reduce the number of feature channels; the fifth part is Dropout layer. The schematic diagram of MBConv module is as follows: Figure 2 shown.

[0023] Asymmetric convolution is introduced to improve EfficientNet-B2. The steps are as follows: The principle diagram of asymmetric convolution is as follows Figure 3 As shown in the figure, an asymmetric kernel is introduced at the corresponding position of the 3×3 convolution kernel to enhance the skeleton information. The improved asymmetric convolution block consists of three convolution kernels of sizes 1×3, 3×1 and 3×3, forming three parallel convolution branches. When the training is completed and the network enters the deployment stage, the three parallel branches extract features from the image respectively, and then perform batch normalization and addition, and the output is equivalent to the feature extraction result extracted by the skeleton enhancement convolution kernel.

[0024] Assume that the size of the convolution kernel is H×W, the number of convolution kernels is D, the number of channels of the input feature map is C, and use represents the convolution kernel, Indicates input, represents the convolution output, batch normalization (BN) is used to reduce overfitting, and a linear change is added after the batch normalization layer. Then for the jth convolution kernel , the corresponding convolution output is shown in formula (1): (1); in, represents a two-dimensional convolution, is the feature map of the kth channel of M, represents the convolution kernel of the kth channel, and are the channel mean and standard deviation of batch normalization, and are the scaling factor and the offset.

[0025] use represents the fused convolution kernel, represents the bias, and Represent the output of 1×3 and 3×1 convolution kernels respectively. The fused results are shown in equations (2) to (4): (2); (3); (4); in, , , Represents the output of the original 3×3, 1×3, and 3×1 branches. , , , Represents the channel mean and standard deviation of 1×3 and 3×1 convolutions for batch normalization, respectively. , , , Represent the scaling factor and offset of 1×3 and 3×1 convolution respectively.

[0026] Step 1.2: Feature fusion module design In the present invention, the bidirectional feature pyramid structure constructed is as follows Figure 4As shown, the bidirectional feature pyramid structure includes: three input nodes, three output nodes and one intermediate node, wherein each 1-3 input nodes is connected to 6-8 MBConv outputs of EfficientNet; in the feature fusion module, the first layer fusion output part receives the first layer input and the second layer intermediate output and fuses the two parts of features to output, the second layer intermediate output part receives the second layer input and the third layer input and fuses the two parts of features to output, the second layer fusion output part receives the second layer input and the second layer intermediate output and fuses the two parts of features to output, and the third layer fusion output part receives the third layer input and the second layer fusion output and fuses the two parts of features to output.

[0027] Figure 4 China-Israel , , Represents the input of layers 1-3, Represents the intermediate output of the second layer, indicating the intermediate node of the bidirectional pyramid structure, , , Represents the fusion output of the 1st to 3rd layers. Taking the second layer as an example, when the bidirectional feature pyramid structure performs feature fusion calculation, feature fusion can be performed using the following formula: (5); (6); , (i=1, 2, 3) is the learnable weight of each node involved in the operation, Resize() is a resizing function that enables the fusion of features of different scales, Conv is a convolution operation, and ε=0.0001 is used to ensure the stability of the value.

[0028] Furthermore, in order to improve the effect of multi-scale feature fusion, a hybrid attention mechanism is added to the output of the bidirectional feature pyramid. Figure 5 As shown in the figure, a double pooling layer is added in front of the SE module, on this basis, the spatial attention mechanism SAM is introduced, and the improved SE module and SAM module are linked in parallel; Step 1.3: Prediction module design The loss is calculated and the parameters of the target detection network are updated through a preset target loss function, and the target detection network is iteratively trained until the target detection network is trained, wherein the target loss function is based on Focal-EIOU and Focal Loss loss functions.

[0029] like Figure 6As shown in the figure, the feature fusion network inputs feature information of different scales into detection modules of different sizes. After convolution operation, it generates information about the category and bounding box regression parameters of each anchor point. H×W represents the size of the feature map, A is the number of anchor points contained in each grid of the feature map, and n is the number of predicted categories. A×n represents the category prediction information of each anchor point in each grid, and A×4 represents the position prediction information of each anchor point.

[0030] Select the Focal-EIOU loss function and the Focal loss loss function to calculate the position loss and classification loss of the prediction box, and construct the target loss function through the following formula: (7); in, , , They represent IOU loss, distance loss, and width-height loss respectively. IOU is the intersection-over-union ratio, which is used to measure the gap between the predicted border and the actual border. It is the ratio of the intersection area of ​​the predicted box and the actual box to the union area. ; w , h Respectively represent the width and height of the predicted border; b , Represent the center point coordinates of the predicted box and the actual box respectively, represents the Euclidean distance between the two. , Indicates the width and height of the minimum bounding rectangle of the predicted box and the real box. Introducing a parameter that can control the degree of suppression , the final Focal-EIOU formula is: (8); The principle of Focal loss is as follows: (9); (10); Where y represents the actual target category, with a value of 0 or 1, and p represents the predicted target category, with a value of [0, 1]. Used to control the weights of positive and negative samples, Used to control the weight of difficult and easy classification samples.

[0031] The overall model structure is as follows Figure 7 shown.

[0032] Step 2: Construct an offshore wind turbine blade image dataset and use annotation software to annotate the defects in the dataset, including coating defects, crack defects, and dirt defects. Generate an annotation file containing defect categories and prediction box coordinate information, and divide the image dataset into training set, validation set, and test set in a ratio of 8:1:1.

[0033] Step 3: Train the network model based on the prepared image dataset to obtain an early defect recognition model for the surface of offshore wind turbine blades.

[0034] During the training phase, after the prediction module parses the prediction box parameter information, it calculates the loss value of the generated prediction box and the annotation box, and finally performs gradient backpropagation and parameter update.

[0035] In the training phase, the number of training iterations of the experiment is set to 500, the batch size is 4, the initial learning rate is 0.01, and the stochastic gradient descent SGD optimizer is used to perform a preset number of iterative training on the target detection network until the prediction accuracy of the target detection network for the images in the validation set reaches a preset accuracy threshold; The recall rate, accuracy rate and precision rate of the preliminary model are measured based on the test set. When the recall rate, accuracy rate and precision rate all meet the preset standards, the preliminary model is determined to be a recognition model.

[0036] In order to verify the effectiveness of each improved strategy, this paper designs an ablation experiment based on EfficientDet-D2, and the data are shown in Table 1.

[0037]

[0038] Step 4: Input the offshore wind turbine blade image to be identified into the trained offshore wind turbine blade surface defect detection model to obtain the offshore wind turbine blade defect detection result.

[0039] To further verify the performance of the algorithm, the method of the present invention is compared with other mainstream target detection algorithms such as Faster R-CNN, SSD and EfficientDet-D2 under the same configuration and data set, and the experimental results are shown in Table 2. As can be seen from Table 2, compared with these networks, although the detection speed of the algorithm proposed in the present invention is not optimal, it has the best effect in detection accuracy, achieves the highest accuracy and mAP value, and also achieves a faster detection speed.

[0040]

[0041] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for early defect recognition on the surface of offshore wind turbine blades based on improved EfficientNet, characterized in that: The following steps are involved: Step 1: Build a network model Improve the design of the EfficientNet backbone feature extraction network: introduce asymmetric convolution to replace the ordinary 3×3 convolution in the EfficientNet backbone feature extraction network; Feature fusion module design: Design a hybrid spatial channel attention module to focus on spatial and channel information, and combine the BiFPN feature fusion module to perform feature fusion on semantic information of different depths; Prediction module design: Calculate the loss and update the parameters of the target detection network through a preset target loss function, and iteratively train the target detection network until the target detection network is trained; Step 2: Construct an offshore wind turbine blade image dataset; Step 3: training a network model based on the prepared image data set to obtain an early defect recognition model for the surface of offshore wind turbine blades; Step 4: input the offshore wind turbine blade image to be identified into the offshore wind turbine blade image recognition model to obtain the offshore wind turbine blade defect detection result.

2. According to claim 1, a method for identifying early defects on the surface of offshore wind turbine blades based on improved EfficientNet is characterized in that: In the step 1, the EfficientNet backbone feature extraction network is improved and designed, including: The backbone network consists of 8 main convolutional modules, of which module 1 is a convolutional layer containing 3×3 convolution, BN layer, and Swish activation function. Modules 2 to 8 are MBConv structures with different parameters, and each MBConv structure contains several repeated MBConv modules. In the MBConv module, the first part is a 1×1 convolution, and MBConvn represents that the number of output channels after the 1×1 convolution is expanded to n times the number of input channels; the second part is a depthwise separable convolution; the third part is an SE attention mechanism module; the fourth part is to use convolution to reduce the number of feature channels; and the fifth part is a Dropout layer.

3. The method for early defect identification on the surface of offshore wind turbine blades based on improved EfficientNet according to claim 1 is characterized in that: In the step 1, the feature fusion module design includes: The 1st to 3rd layer inputs of the feature fusion module are the outputs of the 6th to 8th MBConv modules of the backbone network respectively; the feature fusion module includes a first layer fusion output, a second layer intermediate output, a second layer fusion output, and a third layer fusion output, the first layer fusion output part receives the first layer input and the second layer intermediate output and fuses the two parts of features to output, the second layer intermediate output part receives the second layer input and the third layer input and fuses the two parts of features to output, the second layer fusion output part receives the second layer input and the second layer intermediate output and fuses the two parts of features to output, and the third layer fusion output part receives the third layer input and the second layer fusion output and fuses the two parts of features to output; The outputs of layers 1 to 3 of the feature fusion module are respectively connected to a hybrid attention mechanism, which adds a double pooling layer in front of the SE module, introduces a spatial attention mechanism SAM on this basis, and links the improved SE module and SAM module in parallel.

4. The method for early defect identification on the surface of offshore wind turbine blades based on improved EfficientNet according to claim 1 is characterized in that: In the step 1, the target loss function is based on the Focal-EIOU and Focal Loss loss functions, and the Focal-EIOU loss function and the Focal loss loss function are selected to calculate the position loss and classification loss of the prediction box.

5. The method for early defect identification on the surface of offshore wind turbine blades based on improved EfficientNet according to claim 4 is characterized in that: The objective loss function is constructed by the following formula: (1)； in, , , They represent IOU loss, distance loss, and width-height loss respectively. IOU is the intersection-over-union ratio, which is used to measure the gap between the predicted border and the actual border. It is the ratio of the intersection area of ​​the predicted box and the actual box to the union area. ; w , h Respectively represent the width and height of the predicted border; b , Represent the center point coordinates of the predicted box and the actual box respectively, represents the Euclidean distance between the two. , Represents the width and height of the minimum enclosing rectangle of the predicted box and the true box; Introducing a parameter that can control the degree of suppression , the final Focal-EIOU formula is: (2)； The expression of the Focal loss function is: (3)； (4)； Among them, y represents the actual category of the target, and its value is 0 or 1. p is the target prediction category, and its value is an interval value in [0, 1]. is used to control the weights of positive and negative samples. is the weight used to control the difficulty and ease of classification samples.

6. The method for early defect identification on the surface of offshore wind turbine blades based on improved EfficientNet according to claim 1 is characterized in that: In the step 2, defect images of offshore wind turbine blades are collected by drones, and the defects in the data set are annotated using annotation software to generate an annotation file containing defect categories and prediction box coordinate information. The image data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:

1.

7. The method for early defect identification on the surface of offshore wind turbine blades based on improved EfficientNet according to claim 1, characterized in that: In the step 3, training the network model based on the prepared image dataset includes the following steps: 1) Use the stochastic gradient descent SGD optimizer to iteratively train the target detection network for a preset number of iterations until the prediction accuracy of the target detection network for the images in the validation set reaches a preset accuracy threshold, at which point a preliminary model is obtained; 2) The recall rate, accuracy rate and precision rate of the preliminary model are measured based on the test set. If the recall rate, accuracy rate and precision rate all meet the preset standards, the preliminary model is determined to be a recognition model.

8. The method for early defect identification on the surface of offshore wind turbine blades based on improved EfficientNet according to claim 7, characterized in that: The iteration number of the iterative training is set to 500, the batch size is 4, and the initial learning rate is 0.

01.

9. The method for early defect identification on the surface of offshore wind turbine blades based on improved EfficientNet according to claim 1, characterized in that: In the step 4, the surface defects of the offshore wind turbine blades are detected, and the image of the wind turbine blade to be detected is input into the trained offshore wind turbine blade surface early defect recognition model to obtain the defect detection result output by the trained target detection network.

Images