Offshore wind power blade defect detection method

Abstract

The invention discloses an offshore wind power blade defect detection method, and belongs to the technical field of wind power blade detection. The detection method comprises the following steps: S01, offshore wind power blade defect detection: identifying the position of a blade defect, outputting a positioning mark frame of the defect on an original image, and providing the coordinate of the center position of the mark frame and the corresponding length and width; s02, leaf image splicing: carrying out step-by-step shooting; splicing each detected image by means of an image splicing technology to form a complete fan blade image; and S03, determining the actual physical position of the blade defect in the single image: in the spliced total image, determining the actual physical coordinate of the blade defect in the original single image corresponding to the fan blade. According to the detection method, multiple detection means such as computer vision, feature extraction, image splicing and edge detection technologies are fused, blade surface defect information can be comprehensively obtained, and the detection precision and accuracy are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of wind turbine blade detection, and in particular relates to a method for detecting defects in offshore wind turbine blades. Background Art

[0002] In recent years, my country's wind power industry has developed vigorously. As the scale of wind power continues to expand, the demand for operation and maintenance has also been increasing.

[0003] In wind turbines, blades play a vital role. However, during long-term operation, various defects are prone to appear on the surface of the blades. To prevent these defects from evolving into irreversible damage, the blades must be repaired in a timely manner. Compared with land wind power, offshore wind power faces more severe challenges, such as salt spray erosion, typhoons, lightning strikes and other severe weather conditions, which make blade surface defects more frequent and the maintenance task more arduous. At present, the inspection of offshore wind turbine blades is still mainly based on traditional manual inspection, and professional blade inspection systems are particularly scarce, which makes large-scale aerial operations difficult to carry out, there are many blind areas for maintenance, and accessibility is extremely poor. When using conventional target detection algorithms, there are problems such as low accuracy, high latency, few classifications, and insensitivity to small target defects. Therefore, it has become an urgent problem to achieve efficient real-time detection and safety warning of wind turbine blades in marine environments. Summary of the invention

[0004] An embodiment of the present invention provides an offshore wind turbine blade defect detection method, which aims to solve the problems of low accuracy, high latency, few classifications, and insensitivity to small target defects when using conventional target detection algorithms. The offshore wind turbine blade defect detection method of the present invention adopts a DW feature extraction network structure. Under this architecture, the C2f_CBAM (C2f_CBAM: C2f_Convolutional Block Attention Module: C2f convolutional attention) attention module is integrated, so that the network can not only pay attention to the information interaction between channels, but also deeply explore the distribution of features in the image space, so as to more accurately capture the local details and global information of blade defects. On this basis, the concept of grid segmentation is further introduced to classify and locate defects in the spliced ​​images to improve the accuracy and efficiency of the entire recognition process.

[0005] The technical solution of the present invention is achieved in this way:

[0006] A method for detecting defects in offshore wind turbine blades comprises the following steps:

[0007] S01: Offshore wind turbine blade defect detection: Identify the location of blade defects, output the defect positioning mark frame on the original image, and provide the coordinates of the center position of the mark frame and its corresponding length and width;

[0008] S02: Blade image stitching: By shooting different parts of the blade step by step, the image data of the blade is gradually accumulated; with the help of image stitching technology, each detected image is stitched together to form a complete wind turbine blade image;

[0009] S03: Determine the actual physical position of the blade defect in the single image: In the stitched total image, determine the actual physical coordinates of the wind turbine blade corresponding to the blade defect in the original single image.

[0010] Preferably, in step S01:

[0011] The position of the blade defect is identified by using a DW feature extraction network structure, into which a C2f_CBAM module is introduced.

[0012] The C2f_CBAM module adds the original image features and the enhanced features through residual connections to ensure the continuity of the features. Most offshore wind turbine blades have small target defects. The BEAM module is introduced into the C2f_CBAM module to enhance the DW network's perception of important features, further suppress useless and redundant information in the original image, and further improve the network's accuracy in detecting small blade defects.

[0013] The BEAM module is composed of multiple BottleNeck_CBAMs.

[0014] The method outputs the positioning mark frame of the defect on the original image and provides the coordinates of the center position of the mark frame and its corresponding length and width. Specifically, the input original image is divided into n×n small grid images, each small grid image is input into the DW network structure, and each small grid is specified to predict S anchors with a predetermined aspect ratio (the number of anchor frames used for each grid unit), wherein the center of the bounding box must be within the corresponding network area; through the DW network structure, the length, width and center point coordinates of the predicted bounding box are output, and at the same time, the classifier in the network structure outputs the prediction corresponding to each category, and Scores are set as the confidence score corresponding to each predicted box; the calculation formula of Scores is shown in Formula I:

[0015]

[0016] Among them, P(Class i) represents the probability value of different blade defect categories. PIoU is an improvement on IoU (intersection over union), which is used to evaluate the overlap between the predicted bounding box and the true bounding box, thereby accurately measuring the accuracy of positioning. The higher the PIoU value, the greater the overlap area between the predicted bounding box and the true bounding box, which in turn indicates that the prediction result is more accurate. The calculation formula of the PIoU is shown in Formula II:

[0017]

[0018] Among them, A represents the predicted bounding box, B represents the real label box, and the value of i is 1-4. Represents the distance from the midpoint coordinates of the line connecting the predicted bounding box and the true labeled box to the coordinates of each point in the predicted bounding box. Represents the distance from the midpoint coordinates of the line connecting the predicted bounding box and the true marked box to the coordinates of each point in the true marked box. Indicates taking the maximum value of the distance.

[0019] The use of the improved PIoU loss function improves the convergence efficiency of the model. By combining the corresponding classification probabilities, the parameters in the classifier can be better adjusted to generate a predicted box that is closer to the true marked box, further improving the accuracy of defect location.

[0020] Since each grid is required to generate S bounding boxes, there may be a case where one target has multiple marker boxes, so the marker boxes need to be reasonably screened. Preferably, the screening of the multiple marker boxes specifically includes: 1) discarding the marker boxes whose bounding box confidence scores are lower than a threshold, 2) using matrix non-maximum suppression (Matrix NMS) for each class to remove redundant marker boxes until all redundant marker boxes are processed and the whole process ends.

[0021] So far, the model has shown good results in offshore wind turbine blade defect detection, and can accurately predict the location and corresponding category of blade defects with low false detection rate and missed detection rate.

[0022] When actually performing the offshore wind turbine blade defect identification task, due to the wide area of ​​the blade itself, it is difficult for the drone to cover the entire blade in a single shot. Therefore, a step-by-step shooting strategy is adopted in step S02, by successively capturing different parts of the blade, gradually accumulating the image data of the blade; and then using image stitching technology, the scattered image fragments are merged together, and finally a complete wind turbine blade image is constructed for further defect analysis and identification.

[0023] Preferably, in step S02,

[0024] The image stitching technology is used to stitch each detected image together. The specific steps are as follows:

[0025] S21. Extract features from the original single image and obtain feature point information of each image through supervised learning to facilitate subsequent image stitching operations;

[0026] S22. According to the feature points matched in step S21, the geometric transformation relationship between different images is estimated to complete the stitching.

[0027] Preferably, in step S21,

[0028] The feature extraction of the original single image adopts the QX network structure, uses the FRConv convolution layer to reduce the number of model parameters, introduces the BEAM module, uses the idea of ​​residual connection to ensure the performance of deep learning, pays attention to the information between channels, and attaches importance to the distribution of features in the image space, so as to capture more local and global information about blade defects.

[0029] By utilizing the FRConv convolutional layer, the accuracy of extracting blade defect features is improved while reducing the number of model parameters, and the robustness of the model to the input image is enhanced; the BEAM module is introduced, and the idea of ​​residual connection is used to ensure the performance of deep learning, focus on inter-channel information, and attach importance to the distribution of features in the image space, so as to capture more local and global information about blade defects; through the assembly of modules, a QX network structure is formed, which can deeply learn and extract effective information closely related to blade defects, identify feature points, and give descriptive information to these feature points, forming a series of feature vectors, which provides strong support for the accurate identification and positioning of blade defects.

[0030] In order to more comprehensively extract the feature information on offshore wind turbine blades, the FRConv module is used to integrate context information and regional feature information to help the network better understand the relationship between defects and the surrounding environment, which helps to improve the network's extraction accuracy of blade defect features. The structure of the FRConv module includes a regional feature extractor, a context feature extractor, and an integrated feature extractor, and the regional feature extractor and the context feature extractor are both connected to the integrated feature extractor.

[0031] The FRConv module performs feature extraction, including the following steps: the regional feature extractor is a 3×3 standard convolution layer (Conv), which learns local features from the surrounding 8 adjacent feature vectors; the context feature extractor is a 3×3 dilated convolution (AConv), which has a relatively large receptive field and can effectively learn the surrounding context information; the integrated feature extractor connects the output features of the regional feature extractor and the context feature extractor (Concat), and processes them through a batch normalization BatchNorm2d function and FReLU activation function (BN+FReLU), and finally inputs the extracted feature map. The overall output of the FRConv module is expressed as:

[0032] FRConv(x)=FReLU(BatchNorm2d(Concat(Conv 3×3 (x),AConv 3×3 (x)

[0033] Among them, FRConv is the output of the FRConv module, FReLU is the FReLU activation function, BatchNorm2d is the batch normalization BatchNorm2d function, Concat is the connection, Conv 3×3 AConv is a 3×3 standard convolutional layer. 3×3 is a 3×3 hole convolution, and x is the input feature map.

[0034] The BEAM module adds the original image information to the image information enhanced by the CBAM module through residual connection to ensure the continuity of features and the fluidity of information, and enhance the perception and learning capabilities of blade defect features.

[0035] The CBAM module adopts two module components, specifically a channel attention module (CAM) and a spatial attention module (SAM).

[0036] The operation process of the CBAM module is to input the original image, multiply the output result of the channel attention module (CAM) with the original image to extract features; at the same time, the output result of the channel attention module (CAM) is used as input to enter the two-dimensional convolution of the spatial attention module (SAM), and then the output result is multiplied with the original image, finally enhancing the model's ability to perceive features to better identify small target defects.

[0037] The channel attention module (CAM) is based on the idea of ​​keeping the channel dimension unchanged and compressing the spatial dimension, focusing on the meaningful information in the input image and enhancing the feature expression of each channel. The input feature map (Feature Map) passes through two parallel maximum pooling layers (MaxPool) and global average pooling layers (AvgPool), and the feature map is changed from C*H*W to C*1*1 in size, and then passes through the shared fully connected layer (Share MLP) module. In the shared fully connected layer (Share MLP) module, the number of channels is first compressed to 1 / r times the original number of channels, and then expanded to the original number of channels, and two activated results are obtained through the FReLU activation function; the two output results are added element by element, and then the channel attention weight α is obtained through the Sigmoid activation function, and then this output result is multiplied by each channel of the original feature map to change back to the size of C*H*W.

[0038] The calculation formula in the shared fully connected layer (Share MLP) module is shown in Formula III:

[0039]

[0040] Among them, x is the input feature map, Represents a 1x1 convolution operation, and its output channel number is Where C is the number of channels of the input feature map, and r is a reduction ratio, which is used to reduce parameters and calculations.

[0041] The calculation formula of the channel attention weight α is shown in Formula IV:

[0042] α=Sigmoid(x Sum );

[0043] Formula IV;

[0044] Among them, α is the channel attention weight, Sigmoid is the Sigmoid activation function, and x Sum It is the result of element-by-element addition of two feature maps after activation by the FReLU activation function.

[0045] The spatial attention module (SAM) pays attention to more target position information by compressing the channel dimension according to the idea that the spatial dimension remains unchanged. The spatial attention module (SAM) takes the output result of the channel attention module (CAM) as the input feature (Input Feature), obtains two 1*H*W feature maps through maximum pooling (MaxPool) and average pooling (AvgPool), and then concatenates the two feature maps through the Concat operation to obtain a 2*H*W feature map x Concat, use the convolution layer (Conv layer) to process the feature map, and then pass it through a Sigmoid activation function to obtain the feature map of spatial attention. Finally, multiply the output result by the original image to return to the size of C*H*W, and weight the features of each spatial position.

[0046] The feature map x Concat The formula is shown in Formula V:

[0047] x Concat =Concat(x MaxPool ,x AvgPool );

[0048] Formula V;

[0049] The model structure of the BEAM adds the original image information and the enhanced image information through residual connection to ensure the continuity of features and information fluidity, and enhance the perception and learning ability of blade defect features; a Fosine similarity method is established to calculate the similarity of two images, and feature points are matched according to the calculation results, and the feature point detection network loss function is used to evaluate the prediction authenticity.

[0050] The calculation formula of the Fosine similarity (Fosine_Similarity) is shown in Formula VI:

[0051]

[0052] a and b are the two-dimensional feature vectors of two images respectively.

[0053] The feature point detection network loss function is shown in Formula VII:

[0054] Loss=1-Fosine_Similarity(FRConv(prea),Conv(t));

[0055] Formula VII;

[0056] prea is the feature point of the predicted value, and t is the marked feature point.

[0057] The higher the value of the loss function of the feature point detection network, the more wrong the prediction is, and the lower the value of the loss function, the closer the prediction is to the true value.

[0058] Preferably, in step S22, the geometric transformation relationship between different images is estimated, specifically: transformation parameters are estimated based on minimizing reprojection errors, so as to determine the positional relationship between matching feature points, and translation, rotation, and scaling transformations are fitted through least squares reprojection to provide support for subsequent image stitching.

[0059] The geometric transformation based on minimizing the reprojection error comprises the following steps:

[0060] The feature vector of any matching pair of feature points in the matching image is taken as input. The goal is to find a geometric transformation model so that one feature vector in the matching team is as close as possible to another feature vector after transformation, so as to better judge the transformation relationship. An affine transformation model is used, which includes three transformation parameters: translation, rotation and scaling. The reprojection error is used as the objective function to measure the error between the model prediction value and the actual value.

[0061] The affine transformation model is shown in Formula VIII:

[0062]

[0063] Among them, (x i ,y i ) represents a feature vector in a matching pair, (x i ′,y i ′) represents the eigenvector after geometric transformation, e, f, g, h represent the rotation and scaling parameters respectively, and p and q represent the translation parameters.

[0064] The reprojection error is used as the objective function to measure the error between the model prediction value and the actual value. For a two-dimensional image, assuming there are N matching pairs, the i-th point on the source image is (x i ,y i ), the corresponding point on the target image is (x i ′,y i ′); using the reprojection error, the objective function is minimized by the gradient descent method to solve the optimal parameters of the corresponding geometric transformation, so as to estimate the geometric transformation relationship between the two images; according to the estimated geometric transformation relationship, different images are spliced; for each pixel in the overlapping area, the weighting coefficient is calculated according to the gradient difference of its surrounding pixels; the gradient of the image is calculated using the two-dimensional second-order difference method.

[0065] The calculation formula of the reprojection error is shown in Formula IX:

[0066]

[0067] The operation of stitching different images based on the estimated geometric transformation relationship is specifically: utilizing the geometric transformation relationship between images to perform geometric transformation on the images, and then combining the relevant results of feature matching to identify the overlapping areas between the images, and achieving seamless stitching by pixel fusion of the overlapping areas.

[0068] For each pixel in the overlapping area, the weighted coefficient is calculated according to the gradient difference of the surrounding pixels. Specifically, assuming that the gradients of the two images are G1 and G2 respectively, the weighted fusion formula Blend(x,y) for each pixel in the overlapping area is as described in Formula X:

[0069]

[0070] Among them, I1(x, y) and I2(x, y) are the pixel values ​​of the two images in the overlapping area, and G1(x, y) and G2(x, y) are the gradient values ​​of the two images in the overlapping area.

[0071] The two-dimensional second-order difference method is used to calculate the gradient of the image, and the gradient image is obtained by adding the second-order differences in the x direction and the y direction; the weighted fusion formula and the difference method are applied to each pixel in the overlapping area to effectively eliminate the discontinuity at the image splicing boundary and achieve smooth transition and continuity of the image.

[0072] The two-dimensional second-order difference method formula is shown in Formula XI:

[0073]

[0074] The above operation process is executed for all input images. At this point, a relatively complete and clear blade image is output. The whole operation is closely connected, showing a good image stitching effect, laying a solid foundation for the subsequent identification and positioning of blade defects.

[0075] Preferably, in step S03:

[0076] The method of determining that the blade defect in the original single image corresponds to the actual physical coordinates of the wind turbine blade is specifically as follows: physically locating the defect on the stitched image, and determining the defect coordinates with the lower left corner of the wind turbine blade as the origin; before locating, the invalid area of ​​the image needs to be cropped to remove the non-wind turbine area, and the lower left corner of the removed image is used as the origin of the coordinate system.

[0077] In the process of determining the defect coordinates, a search algorithm based on an 8-domain relationship is used to extract the pixel position of each edge and its extension order on the edge. The algorithm can overcome the problem of pixel accumulation or discontinuity.

[0078] The non-zero points in the binary image F(i,j) represent edges. The ideal state of these non-zero points in the image space is to present continuous, single-pixel distribution. However, some edges will have pixel accumulation or pixel discontinuity.

[0079] The search algorithm based on the 8-domain relationship designs an edge determination operator Q(i, j) within the 8-neighborhood range of a pixel. Its functions are: first, determining whether the current pixel is an edge point according to the Q(i, j) value; second, determining the coordinates of the next edge point. The specific application method of the edge determination operator Q(i, j) is shown in Formula XII:

[0080]

[0081] Among them, the grayscale value of the edge point is 1, the grayscale value of the background point is 0, m and n are the indices used to traverse the 8-neighborhood pixels of the current pixel point (i, j), which can be -1, 0, 1.

[0082] Use the 8-domain relationship search algorithm to find the upper and lower boundaries of the fan blades, that is, find the maximum (or minimum) Y value on the left and the minimum (or maximum) Y value on the right, that is, the two points on the left and right sides, and determine the straight line equation of the upper boundary YS through these two points as YS=KS*X1+BS; similarly, find the two points of the lower boundary through the search algorithm, and determine the straight line equation of the lower boundary YU as YU=KU*X2+BU.

[0083] Find the lowest point of the lower boundary, and modify the original upper boundary and lower boundary equations so that the lowest point of the lower boundary is the origin. The modified upper boundary equation is YS = KS*X1+BS-C1, and the lower boundary equation is YU = KU*X2+BU-C2.

[0084] Where YS is the Y coordinate of any point on the upper boundary, KS is the slope of the upper boundary, X1 is the X coordinate of any point on the upper boundary, BS is the intercept of the upper boundary, and C1 is the constant subtracted from the upper boundary equation.

[0085] Similarly, YU is the Y coordinate of any point on the lower boundary, KU is the slope of the lower boundary, X2 is the X coordinate of any point on the lower boundary, BU is the intercept of the lower boundary, and C2 is the constant subtracted from the lower boundary equation.

[0086] In order to smooth the image features, better process edge detection, and avoid too few data points, the IGP model is used to optimize the boundary equation. Taking the boundary line equation as an example, it is assumed that KS obeys KS~N(μ0,∑0) and BS obeys BS~N(0,σ 2 I), where μ0 is the mean vector of KS, ∑0 is the prior covariance matrix, σ 2 is the variance of the error term and I is the identity matrix.

[0087] Calculate the posterior distribution of KS, the formula is shown in Formula XIII:

[0088] p(KS|X,YS)∝p(YS|X,KS)p(KS);

[0089] Formula XIII;

[0090] Where KS is the slope of the upper boundary, X is the X coordinate of any point on the upper boundary, and YS is the Y coordinate of any point on the upper boundary.

[0091] The mean and covariance of YS are calculated as shown in Formula XIV:

[0092] μ post =∑0M T (M∑0M T +σ 2 I) -1 YS;

[0093] ∑ post =∑0-∑0M T (M∑0M T +σ 2 I) -1 M∑0;

[0094] Formula XIV;

[0095] Among them, μ post is the mean vector of YS, ∑ post is the covariance matrix of YS, ∑0 is the prior covariance matrix; M is a matrix, each row represents the X coordinate of any point on the upper boundary, T is the matrix transpose symbol; σ 2 is the variance of the error term and I is the identity matrix.

[0096] Given a new input x * , predict output y * , the formula is shown in Formula XV:

[0097] p(y * |x * ,X,YS)=∫p(y * |x * ,KS)p(KS|X,YS)dKS;

[0098] Formula XV;

[0099] Where X is the X coordinate of any point on the upper boundary, YS is the Y coordinate of any point on the upper boundary, and KS is the slope of the upper boundary.

[0100] According to the above operation, the final defect positioning mark frame in the original image will be output, including the coordinates of the center position of the mark frame and the corresponding length and width. After the image stitching operation, the position of the segmentation line of different image stitching in the original image can be recorded (the leftmost position is 0), and the order between the images is known. Combining the above two information, the position information of each segmentation line in the entire image can be calculated. According to the position distribution of the segmentation line, a certain digital conversion can be performed to obtain the corresponding position of the leaf defect in the original image before the single stitching.

[0101] The center coordinates of the new image defect positioning mark box (x i ,y i ), length w, width h, it is easy to find the coordinates of the upper right corner, lower right corner, upper left corner and lower left corner corresponding to the marked box, and use the midpoint coordinates of the marked box to represent the coordinates of the defect, that is, X q =(x i +w) / 2,Y q =(y i +h) / 2.

[0102] According to the upper and lower boundaries, first calculate the defect coordinate Y q Perform relative transformation, if Y q At the upper end of the upper limit or the lower end of the lower limit, the defect detection is considered abnormal and will be discarded. q Between the upper and lower boundaries, it is transformed, that is, Y q The Manhattan distance to the origin of the lower boundary is

[0103] For the defect coordinate X q The transformation takes into account the impact of horizontal overlap in image stitching. In this graphical stitching algorithm, the search range of key points can be dynamically adjusted to find the best matching point pair. Use the FRConv module and Fosine_Similarity to find the corresponding key points in the two images, and determine the width of the overlapping part by calculating the distance between the key points. Select the n closest key points and calculate the Manhattan average distance D between them. i , that is, D i =SUM(n i xn j x) / n,n i x and n j x is n i , n j The x coordinates of the adjacent images are obtained by i , the width of each sub-image is w i , then the horizontal position L of the segmentation line of the ith sub-image in the spliced ​​image i It is expressed as:

[0104] Then the transformed X qe =L i -D.

[0105] At this point, the physical positioning step of the blade defect in a single image is completed. The above-mentioned operation network structure has high defect recognition accuracy. At the same time, the calculation process of obtaining the defect coordinate information of a single image from the defect information of the entire image is simple and fast, and the defect information can be well selected with a high accuracy.

[0106] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0107] 1. The detection method of the present invention realizes fully automatic intelligent detection of defects in offshore wind turbine blades, greatly improves the detection efficiency and coverage, and is conducive to timely discovery and treatment of hidden dangers to avoid the expansion of faults.

[0108] 2. The detection method of the present invention integrates multiple detection means such as computer vision, feature extraction, image stitching, edge detection technology, etc., which can comprehensively obtain the surface defect information of the blade, and the detection precision and accuracy are greatly improved.

[0109] 3. The detection method of the present invention is seamlessly integrated with the existing operation and maintenance management system to achieve real-time high-precision detection, significantly improving the operational safety of wind farms and the service life of equipment.

[0110] 4. The detection method of the present invention can be widely used in the intelligent detection and health management of blades in offshore wind farms, which is of great significance for reducing operation and maintenance costs and promoting the sustainable development of the wind power industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Figure 1 This is a flow chart of the offshore wind turbine blade defect detection method in Embodiment 1 of the present invention;

[0112] Figure 2 This is a schematic diagram of the DW network structure in Example 1 of the present invention;

[0113] Figure 3 This is a schematic diagram of the structure of the C2f_CBAM module in Example 1 of the present invention;

[0114] Figure 4 Schematic diagram of the QX network structure in Example 1 of the present invention;

[0115] Figure 5 Schematic diagram of the structure of the FRConv module in Example 1 of the present invention;

[0116] Figure 6 CAM module structure diagram in Embodiment 1 of the present invention;

[0117] Figure 7 This is a structural diagram of the SAM module in Example 1 of the present invention;

[0118] Figure 8 This is a structural diagram of BEAM in Example 1 of the present invention.

[0119] Fig. 9 This is a diagram of the upper and lower boundaries before correction in Example 1 of the present invention.

[0120] Fig.10 4 is a diagram of the corrected upper and lower boundaries in Example 1 of the present invention. DETAILED DESCRIPTION

[0121] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0122] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, top, bottom...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0123] In this application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0124] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it may be directly on the other element or there may be an intermediate element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element at the same time.

[0125] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0126] At present, when using conventional target detection algorithms, there are defects such as low accuracy, high latency, few classifications, and insensitivity to small target defects. In order to solve the above technical problems, the present invention proposes a method for offshore wind turbine blade defect detection. The offshore wind turbine blade defect detection method of the present invention adopts a DW feature extraction network structure. Under this architecture, the C2f_CBAM (C2f_Convolutional Block Attention Module: C2f convolutional attention) attention module is integrated, so that the network can not only pay attention to the information interaction between channels, but also deeply explore the distribution of features in the image space, so as to more accurately capture the local details and global information of blade defects. On this basis, the concept of grid segmentation is further introduced to classify and locate defects in the spliced ​​images to improve the accuracy and efficiency of the entire recognition process.

[0127] Example 1

[0128] like Figure 1 As shown, a method for detecting defects in offshore wind turbine blades comprises the following steps:

[0129] S01: Offshore wind turbine blade defect detection: Identify the location of blade defects, output the defect positioning mark frame on the original image, and provide the coordinates of the center position of the mark frame and its corresponding length and width;

[0130] S02: Blade image stitching: By shooting different parts of the blade step by step, the image data of the blade is gradually accumulated; with the help of image stitching technology, each detected image is stitched together to form a complete wind turbine blade image;

[0131] S03: Determine the actual physical position of the blade defect in the single image: In the stitched total image, determine the actual physical coordinates of the wind turbine blade corresponding to the blade defect in the original single image.

[0132] Preferably, Figure 2 As shown, in step S01,

[0133] In order to effectively solve the problems of low precision, high latency and insufficient sensitivity to small target defects in offshore wind turbine blade defect identification, a DW feature extraction network structure is proposed. The DW feature extraction network structure includes:

[0134] Input layer: receives image data to be processed;

[0135] Feature extraction convolution layer (FRConv): used to preliminarily extract features of the input image;

[0136] C2f_CBAM: used to enhance the network's attention to important features and improve the accuracy of feature extraction by focusing on information interactions between channels and in space;

[0137] Spatial Pyramid Feature Fusion Module (SPFF): used to integrate feature information of different scales and improve the feature representation ability of the network;

[0138] Upsampling operation (Up): used to restore the feature map to its original size or required size;

[0139] Channel fusion layer (CBS): used to fuse feature information from different channels to generate richer feature representation;

[0140] Concat operation: used to concatenate multiple feature maps along the channel dimension to integrate feature information from different sources;

[0141] Feature clarification layer (Clarify): used to further process and optimize the fused features to improve the clarity and interpretability of the features;

[0142] Output layer: Generates the final feature representation or prediction result.

[0143] Through multi-layer convolution operations, attention mechanism and feature fusion technology, this network structure can effectively extract and integrate feature information in leaf images, thereby achieving accurate detection and analysis of leaf defects.

[0144] In this architecture, the C2f_CBAM attention module is integrated, so that the network can not only pay attention to the information interaction between channels, but also deeply explore the distribution of features in the image space, so as to more accurately capture the local details and global information of the blade defects. On this basis, the concept of grid segmentation is further introduced to classify and locate defects in the spliced ​​images to improve the accuracy and efficiency of the entire recognition process.

[0145] The position of the blade defect is identified by using a DW feature extraction network structure, into which a C2f_CBAM module is introduced.

[0146] like Figure 3 As shown in the figure, the C2f_CBAM module adds the original image features and the enhanced features through residual connection to ensure the continuity of the features. Most of the offshore wind turbine blades have small target defects. The BEAM module is introduced into the C2f_CBAM module to enhance the DW network's perception of important features, further suppress useless and redundant information in the original image, and further improve the network's accuracy in detecting small targets of blade defects.

[0147] The BEAM module is composed of multiple BottleNeck_CBAMs.

[0148] The method of outputting the positioning mark box of the defect on the original image and providing the coordinates of the center position of the mark box and its corresponding length and width is as follows: the input image is divided into n×n small grids (n is usually a multiple of 8, such as 8×8, 16×16, or n can take specific values ​​such as 7, 14, 19, etc., which are suitable for YOLO and R-CNN series algorithms), and each small grid image is input into the DW network structure, and each small grid is required to predict S anchors with a predetermined aspect ratio (the number of anchor boxes used for each grid unit), where the center of the bounding box must be in the corresponding network area; through the DW network structure, the length, width and center point coordinates of the predicted bounding box are output, and at the same time, the classifier output in the network structure predicts the probability corresponding to each category. Set Scores as the confidence score corresponding to each predicted box: The calculation formula of Scores is shown in Formula I:

[0149]

[0150] Among them, P(Class i ) represents the probability value of different blade defect categories. PIoU is an improvement on IoU (intersection over union), which is used to evaluate the overlap between the predicted bounding box and the true bounding box, thereby accurately measuring the accuracy of positioning; pred is the predicted value, and truth is the true value. The higher the PIoU value, the larger the overlap area between the predicted bounding box and the true bounding box, which indicates that the prediction result is more accurate. The calculation formula of the PIoU is shown in Formula II:

[0151]

[0152] Among them, A represents the predicted bounding box, B represents the real label box, and the value of i is 1-4. Represents the distance from the midpoint coordinates of the line connecting the predicted bounding box and the true labeled box to the coordinates of each point in the predicted bounding box. Represents the distance from the midpoint coordinates of the line connecting the predicted bounding box and the true marked box to the coordinates of each point in the true marked box. Indicates taking the maximum value of the distance.

[0153] The use of the improved PIoU loss function improves the convergence efficiency of the model. By combining the corresponding classification probabilities, the parameters in the classifier can be better adjusted to generate a predicted box that is closer to the true marked box, further improving the accuracy of defect location.

[0154] Since each grid is required to generate S bounding boxes, there may be a case where one target has multiple marker boxes, so the marker boxes need to be reasonably screened. Preferably, the screening of the multiple marker boxes specifically includes: 1) discarding the marker boxes whose bounding box confidence scores are lower than a threshold, 2) using matrix non-maximum suppression (Matrix NMS) for each class to remove redundant marker boxes until all redundant marker boxes are processed and the whole process ends.

[0155] So far, the model has shown good results in offshore wind turbine blade defect detection, and can accurately predict the location and corresponding category of blade defects with low false detection rate and missed detection rate.

[0156] When actually performing the offshore wind turbine blade defect identification task, due to the wide area of ​​the blade itself, it is difficult for the drone to cover the entire blade in a single shot. Therefore, a step-by-step shooting strategy is adopted in step S02 to gradually accumulate blade image data by capturing different parts of the blade one by one. Then, with the help of image stitching technology, the scattered image fragments are fused together to finally construct a complete wind turbine blade image for further defect analysis and identification.

[0157] Preferably, in step S02,

[0158] By shooting different parts of the blade step by step, the image data of the blade is gradually accumulated; with the help of image stitching technology, each detected image is stitched together to form a complete image of the wind turbine blade;

[0159] The image stitching technology is used to stitch each detected image. The specific steps are as follows:

[0160] S21. Extract features from the original single image and obtain feature point information of each image through supervised learning to facilitate subsequent image stitching operations;

[0161] S22. According to the feature points matched in step S21, the geometric transformation relationship between different images is estimated to complete the stitching.

[0162] Preferably, in step S21,

[0163] The feature extraction of the original single image adopts the QX network structure, uses the FRConv convolution layer to reduce the number of model parameters, introduces the BEAM module, uses the idea of ​​residual connection to ensure the performance of deep learning, pays attention to the information between channels, and attaches importance to the distribution of features in the image space, so as to capture more local and global information about blade defects.

[0164] By utilizing the FRConv convolutional layer, the accuracy of extracting blade defect features is improved while reducing the number of model parameters, and the robustness of the model to the input image is enhanced; the BEAM module is introduced, and the idea of ​​residual connection is used to ensure the performance of deep learning, focus on inter-channel information, and attach importance to the distribution of features in the image space, so as to capture more local and global information about blade defects; through the assembly of modules, a QX network structure is formed, which can deeply learn and extract effective information closely related to blade defects, identify feature points, and give descriptive information to these feature points, forming a series of feature vectors, which provides strong support for the accurate identification and positioning of blade defects.

[0165] The QX network structure is as follows Figure 4 As shown, including:

[0166] Convolutional layer (Conv): It is a neural network layer used to extract image features;

[0167] Average Pooling: It is a neural network layer used to reduce the size of feature maps. It is used to downsample the feature maps output by the Conv layer to reduce the size of the feature maps, thereby reducing the computational complexity of the network.

[0168] The QX network first extracts features from the input image through the Conv layer, and then further extracts blade defect features through the FRConv layer. Next, the feature map is downsampled using the Average Pooling layer to reduce the size of the feature map. After that, the BEAM module is introduced to capture more local and global information about the blade defects. Finally, the Conv layer and Average Pooling layer are used to process the features to accurately identify and locate the blade defects.

[0169] In order to more comprehensively extract the feature information of offshore wind turbine blades, the FRConv module is used to integrate context information and regional feature information to help the network better understand the relationship between defects and the surrounding environment, which helps to improve the network's accuracy in extracting blade defect features. Figure 5As shown, the structure of the FRConv module includes a regional feature extractor, a context feature extractor and an integrated feature extractor, and the regional feature extractor and the context feature extractor are both connected to the integrated feature extractor.

[0170] The FRConv module performs feature extraction, including the following steps: the regional feature extractor is a 3×3 standard convolution layer (Conv), which learns local features from the surrounding 8 adjacent feature vectors; the context feature extractor is a 3×3 dilated convolution (AConv), which has a relatively large receptive field and can effectively learn the surrounding context information; the integrated feature extractor connects the output features of the regional feature extractor and the context feature extractor (Concat), and processes them through a batch normalization BatchNorm2d function and FReLU activation function (BN+FReLU), and finally inputs the extracted feature map. The overall output of the FRConv module is expressed as:

[0171] FRConv(x)=FReLU(BatchNorm2d(Concat(Conv 3×3 (x),AConv 3×3 (x));

[0172] Among them, FRConv is the output of the FRConv module, FReLU is the FReLU activation function, BatchNorm2d is the batch normalization BatchNorm2d function, Concat is the connection, Conv 3×3 AConv is a 3×3 standard convolutional layer. 3×3 is a 3×3 hole convolution, and x is the input feature map.

[0173] The BEAM module adds the original image information to the image information enhanced by the CBAM module through residual connection to ensure the continuity of features and the fluidity of information, and enhance the perception and learning capabilities of blade defect features.

[0174] The CBAM module adopts two module components, specifically a channel attention module (CAM) and a spatial attention module (SAM).

[0175] The operation process of the CBAM module is to input the original image, multiply the output result of the channel attention module (CAM) with the original image to extract features; at the same time, the output result of the channel attention module (CAM) is used as input to enter the two-dimensional convolution of the spatial attention module (SAM), and then the output result is multiplied with the original image, finally enhancing the model's ability to perceive features to better identify small target defects.

[0176] The channel attention module (CAM) focuses on the meaningful information in the input image and enhances the feature expression of each channel based on the idea of ​​keeping the channel dimension unchanged and compressing the spatial dimension. Figure 6 As shown in the figure, the input feature map (Feature Map) passes through two parallel maximum pooling layers (MaxPool) and global average pooling layers (AvgPool), and the feature map is changed from C*H*W to C*1*1 in size, and then passes through the shared fully connected layer (Share MLP) module. In the shared fully connected layer (Share MLP) module, the number of channels is first compressed to 1 / r times the original number, and then expanded to the original number of channels, and two activated results are obtained through the FReLU activation function; the two output results are added element by element, and then the channel attention weight α is obtained through the Sigmoid activation function, and then this output result is multiplied by each channel of the original feature map to change it back to the size of C*H*W.

[0177] The calculation formula in the shared fully connected layer (Share MLP) module is shown in Formula III:

[0178]

[0179] Among them, x is the input feature map, Represents a 1x1 convolution operation, and its output channel number is Where C is the number of channels of the input feature map, and r is a reduction ratio, which is used to reduce parameters and calculations.

[0180] The calculation formula of the channel attention weight α is shown in Formula IV:

[0181] α=Sigmoid(x Sum );

[0182] Formula IV;

[0183] Among them, α is the channel attention weight, Sigmoid is the Sigmoid activation function, and x Sum It is the result of element-by-element addition of two feature maps after activation by the FReLU activation function.

[0184] like Figure 7As shown in the figure, the spatial attention module (SAM) pays attention to more target position information by compressing the channel dimension according to the idea of ​​keeping the spatial dimension unchanged. The spatial attention module (SAM) takes the output result of the channel attention module (CAM) as the input feature (Input Feature), obtains two 1*H*W feature maps through maximum pooling (MaxPool) and average pooling (AvgPool), and then concatenates the two feature maps through the Concat operation to obtain a 2*H*W feature map x Concat , use the convolution layer (Conv layer) to process the feature map, and then pass it through a Sigmoid activation function to obtain the feature map of spatial attention. Finally, multiply the output result by the original image to return to the size of C*H*W, and weight the features of each spatial position.

[0185] The feature map x Concat The formula is shown in Formula V:

[0186] x Concat =Concat(x MaxPool ,x AvgPool );

[0187] Formula V;

[0188] like Figure 8 As shown in the figure, the model structure of the BEAM adds the original image information and the enhanced image information through residual connection to ensure the continuity of features and information fluidity, and enhance the perception and learning ability of blade defect features; a Fosine similarity (Fosine_Similarity) method is established to calculate the similarity of two images, and feature points are matched according to the calculation results, and the feature point detection network loss function is used to evaluate the prediction authenticity.

[0189] The calculation formula of the Fosine similarity (Fosine_Similarity) is shown in Formula VI:

[0190]

[0191] a and b are the two-dimensional feature vectors of two images respectively.

[0192] The feature point detection network loss function is shown in Formula VII:

[0193] Loss=1-Fosine_Similarity(FRConv(prea),Conv(t));

[0194] Formula VII;

[0195] prea is the feature point of the predicted value, and t is the marked feature point.

[0196] The higher the value of the loss function of the feature point detection network, the more wrong the prediction is, and the lower the value of the loss function, the closer the prediction is to the true value.

[0197] Preferably, in step S22, the geometric transformation relationship between different images is estimated, specifically: transformation parameters are estimated based on minimizing reprojection errors, so as to determine the positional relationship between matching feature points, and translation, rotation, and scaling transformations are fitted through least squares reprojection to provide support for subsequent image stitching.

[0198] The geometric transformation based on minimizing the reprojection error comprises the following steps:

[0199] The feature vector of any matching pair of feature points in the matching image is taken as input. The goal is to find a geometric transformation model so that one feature vector in the matching team is as close as possible to another feature vector after transformation, so as to better judge the transformation relationship. An affine transformation model is used, which includes three transformation parameters: translation, rotation and scaling. The reprojection error is used as the objective function to measure the error between the model prediction value and the actual value.

[0200] The affine transformation model is shown in Formula VIII:

[0201]

[0202] Among them, (x i ,y i ) represents a feature vector in a matching pair, (x i ′,y i ′) represents the eigenvector after geometric transformation, e, f, g, h represent the rotation and scaling parameters respectively, and p and q represent the translation parameters.

[0203] The reprojection error is used as the objective function to measure the error between the model prediction value and the actual value. For a two-dimensional image, assuming there are N matching pairs, the i-th point on the source image is (x i ,y i ), the corresponding point on the target image is (x i ′,y i ′); using the reprojection error, the objective function is minimized by the gradient descent method to solve the optimal parameters of the corresponding geometric transformation, so as to estimate the geometric transformation relationship between the two images; according to the estimated geometric transformation relationship, different images are spliced; for each pixel in the overlapping area, the weighting coefficient is calculated according to the gradient difference of its surrounding pixels; the gradient of the image is calculated using the two-dimensional second-order difference method.

[0204] The calculation formula of the reprojection error is shown in Formula IX:

[0205]

[0206] The operation of stitching different images based on the estimated geometric transformation relationship is specifically: utilizing the geometric transformation relationship between images to perform geometric transformation on the images, and then combining the relevant results of feature matching to identify the overlapping areas between the images, and achieving seamless stitching by pixel fusion of the overlapping areas.

[0207] For each pixel in the overlapping area, the weighted coefficient is calculated according to the gradient difference of the surrounding pixels. Specifically, assuming that the gradients of the two images are G1 and G2 respectively, the weighted fusion formula Blend(x,y) for each pixel in the overlapping area is as described in Formula X:

[0208]

[0209] Among them, I1(x, y) and I2(x, y) are the pixel values ​​of the two images in the overlapping area, and G1(x, y) and G2(x, y) are the gradient values ​​of the two images in the overlapping area.

[0210] The two-dimensional second-order difference method is used to calculate the gradient of the image, and the gradient image is obtained by adding the second-order differences in the x direction and the y direction; the weighted fusion formula and the difference method are applied to each pixel in the overlapping area to effectively eliminate the discontinuity at the image splicing boundary and achieve smooth transition and continuity of the image.

[0211] The two-dimensional second-order difference method formula is shown in Formula XI:

[0212]

[0213]

[0214] The above operation process is executed for all input images. At this point, a relatively complete and clear blade image is output. The whole operation is closely connected, showing a good image stitching effect, laying a solid foundation for the subsequent identification and positioning of blade defects.

[0215] Preferably, in step S03:

[0216] The method of determining that the blade defect in the original single image corresponds to the actual physical coordinates of the wind turbine blade is specifically as follows: physically locating the defect on the stitched image, and determining the defect coordinates with the lower left corner of the wind turbine blade as the origin; before locating, the invalid area of ​​the image needs to be cropped to remove the non-wind turbine area, and the lower left corner of the removed image is used as the origin of the coordinate system.

[0217] In the process of determining the defect coordinates, a search algorithm based on an 8-domain relationship is used to extract the pixel position of each edge and its extension order on the edge. The algorithm can overcome the problem of pixel accumulation or discontinuity.

[0218] The non-zero points in the binary image F(i,j) represent edges. The ideal state of these non-zero points in the image space is to present continuous, single-pixel distribution. However, some edges will have pixel accumulation or pixel discontinuity.

[0219] The search algorithm based on the 8-domain relationship designs an edge determination operator Q(i, j) within the 8-neighborhood range of a pixel. Its functions are: first, determining whether the current pixel is an edge point according to the Q(i, j) value; second, determining the coordinates of the next edge point. The specific application method of the edge determination operator Q(i, j) is shown in Formula XII:

[0220]

[0221] Among them, the grayscale value of the edge point is 1, the grayscale value of the background point is 0, m and n are the indices used to traverse the 8-neighborhood pixels of the current pixel point (i, j), which can be -1, 0, 1.

[0222] Use the 8-domain relationship search algorithm to find the upper and lower boundaries of the fan blades, that is, find the maximum (or minimum) Y value on the left and the minimum (or maximum) Y value on the right, that is, the two points on the left and right sides, and determine the straight line equation of the upper boundary YS through these two points as YS=KS*X1+BS; similarly, find the two points of the lower boundary through the search algorithm, and determine the straight line equation of the lower boundary YU as YU=KU*X2+BU.

[0223] Find the lowest point of the lower boundary, and modify the original upper boundary and lower boundary equations so that the lowest point of the lower boundary is the origin. The modified upper boundary equation is YS = KS*X1+BS-C1, and the lower boundary equation is YU = KU*X2+BU-C2. Fig.10 shown.

[0224] Where YS is the Y coordinate of any point on the upper boundary, KS is the slope of the upper boundary, X1 is the X coordinate of any point on the upper boundary, BS is the intercept of the upper boundary, and C1 is the constant subtracted from the upper boundary equation.

[0225] Similarly, YU is the Y coordinate of any point on the lower boundary, KU is the slope of the lower boundary, X2 is the X coordinate of any point on the lower boundary, BU is the intercept of the lower boundary, and C2 is the constant subtracted from the lower boundary equation.

[0226] In order to smooth the image features, better process edge detection, and avoid too few data points, the IGP model is used to optimize the boundary equation. Taking the boundary line equation as an example, it is assumed that KS obeys KS~N(μ0,∑0) and BS obeys BS~N(0,σ 2 I), where μ0 is the mean vector of KS, ∑0 is the prior covariance matrix, σ 2 is the variance of the error term, and I is the identity matrix. The posterior distribution of KS is calculated as shown in Formula XIII:

[0227] p(KS|X,YS)∝p(YS|X,KS)p(KS);

[0228] Formula XIII;

[0229] Where KS is the slope of the upper boundary, X is the X coordinate of any point on the upper boundary, and YS is the Y coordinate of any point on the upper boundary.

[0230] p(KS|X,YS) is the posterior distribution of the slope KS, that is, the probability distribution of the slope KS given the data X and YS; p(YS|X,KS) is the likelihood function of observing YS, which describes the probability of data YS given the data X and KS; p(KS) is the prior distribution of the slope KS, that is, the prior belief about the slope KS in the absence of any data.

[0231] The mean and covariance of YS are calculated as shown in Formula XIV:

[0232] μ post =∑0M T (M∑0M T +σ 2 I) -1 YS;

[0233] ∑ post =∑0-∑0M T (M∑0M T +σ 2 I) -1 M∑0;

[0234] Formula XIV;

[0235] Among them, μ post is the mean vector of YS, ∑ post is the covariance matrix of YS, ∑0 is the prior covariance matrix; M is a matrix, each row represents the X coordinate of any point on the upper boundary, T is the matrix transpose symbol; σ 2 is the variance of the error term and I is the identity matrix.

[0236] Given a new input x * , predict output y *, the formula is shown in Formula XV:

[0237] p(y * |x * ,X,YS)=∫p(y * |x * ,KS)p(KS|X,YS)dKS;

[0238] Formula XV;

[0239] Where X is the X coordinate of any point on the upper boundary, YS is the Y coordinate of any point on the upper boundary, and KS is the slope of the upper boundary.

[0240] p(y * |x * ,X,YS) is the new input x * , input X and output YS, by integrating all possible values ​​of KS, the new output y is predicted * The probability distribution of p(y * |x * ,KS) is given a new input x * and KS, the new output y * The probability distribution of KS; p(KS|X,YS) is the probability distribution of KS given the known input X and output YS.

[0241] According to the above operation, the final defect positioning mark frame in the original image will be output, including the coordinates of the center position of the mark frame and the corresponding length and width. After the image stitching operation, the position of the segmentation line of different image stitching in the original image can be recorded (the leftmost position is 0), and the order between the images is known. Combining the above two information, the position information of each segmentation line in the entire image can be calculated. According to the position distribution of the segmentation line, a certain digital conversion can be performed to obtain the corresponding position of the leaf defect in the original image before the single stitching.

[0242] The center coordinates of the new image defect positioning mark box (x i ,y i ), length w, width h, it is easy to find the coordinates of the upper right corner, lower right corner, upper left corner and lower left corner corresponding to the marked box, and use the midpoint coordinates of the marked box to represent the coordinates of the defect, that is, X q =(x i +w) / 2,Y q =(y i +h) / 2.

[0243] According to the upper and lower boundaries, first calculate the defect coordinate Y q Perform relative transformation, if Y qAt the upper end of the upper limit or the lower end of the lower limit, the defect detection is considered abnormal and will be discarded. q Between the upper and lower boundaries, it is transformed, that is, Y q The Manhattan distance to the origin of the lower boundary is

[0244] For the defect coordinate X q The transformation takes into account the impact of horizontal overlap in image stitching. In this image stitching algorithm, the search range of key points can be dynamically adjusted to find the best matching point pair. Use the FRConv module and Fosine_Similarity to find the corresponding key points in the two images, and determine the width of the overlapping part by calculating the distance between the key points. Select the nearest n key points and calculate the Manhattan average distance D between them. i , that is, D i =SUM(n i xn j x) / n,n i x and n j x is n i , n j The x coordinates of the adjacent images are obtained by i , the width of each sub-image is w i , then the horizontal position L of the segmentation line of the ith sub-image in the spliced ​​image i It is expressed as:

[0245]

[0246] Then the transformed X qe =L i -D.

[0247] At this point, the physical positioning step of the blade defect in a single image is completed. The above-mentioned operation network structure has high defect recognition accuracy. At the same time, the calculation process of obtaining the defect coordinate information of a single image from the defect information of the entire image is simple and fast, and the defect information can be well selected with a high accuracy.

[0248] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0249] 1. The detection method of the present invention realizes fully automatic intelligent detection of defects in offshore wind turbine blades, greatly improves the detection efficiency and coverage, and is conducive to timely discovery and treatment of hidden dangers to avoid the expansion of faults.

[0250] 2. The detection method of the present invention integrates multiple detection means such as computer vision, feature extraction, image stitching, edge detection technology, etc., which can comprehensively obtain the surface defect information of the blade, and the detection precision and accuracy are greatly improved.

[0251] 3. The detection method of the present invention is seamlessly integrated with the existing operation and maintenance management system to achieve real-time high-precision detection, significantly improving the operational safety of wind farms and the service life of equipment.

[0252] 4. The detection method of the present invention can be widely used in the intelligent detection and health management of blades in offshore wind farms, which is of great significance for reducing operation and maintenance costs and promoting the sustainable development of the wind power industry.

[0253] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting defects in offshore wind turbine blades, characterized in that: The following steps are involved: S01: Offshore wind turbine blade defect detection: Identify the location of blade defects, output the defect positioning mark frame on the original image, and provide the coordinates of the center position of the mark frame and its corresponding length and width; S02: Blade image stitching: By shooting different parts of the blade step by step, the image data of the blade is gradually accumulated; with the help of image stitching technology, each detected image is stitched together to form a complete wind turbine blade image; S03: Determine the actual physical position of the blade defect in the single image: In the stitched total image, determine the actual physical coordinates of the wind turbine blade corresponding to the blade defect in the original single image.

2. The offshore wind turbine blade defect detection method according to claim 1, characterized in that: In step S01: the position of the blade defect is identified by using a DW feature extraction network structure, and a C2f_CBAM module is introduced into the DW feature extraction network structure; The C2f_CBAM module adds the original image features and the enhanced features through residual connection to ensure the continuity of the features; The BEAM module is introduced into the C2f_CBAM module to enhance the DW network's ability to perceive important features, further suppress useless and redundant information in the original image, and further improve the network's accuracy in detecting small targets of blade defects; The BEAM module is composed of multiple BottleNeck_CBAMs.

3. The offshore wind turbine blade defect detection method according to claim 1, characterized in that: In step S01: the positioning mark frame of the defect is output on the original image, and the coordinates of the center position of the mark frame and its corresponding length and width are provided. Specifically, the input original image is divided into n×n small grid images, each small grid image is input into the DW network structure, and each small grid is specified to predict S anchors with a predetermined aspect ratio, wherein the center of the bounding box must be within the corresponding network area; through the DW network structure, the length, width and center point coordinates of the predicted bounding box are output, and at the same time, the classifier in the network structure outputs the prediction corresponding to each category, and Scores are set as the confidence score corresponding to each predicted box; the calculation formula of Scores is shown in Formula I: Among them, P(Class i ) represents the probability value of different blade defect categories. PIoU is an improvement on IoU and is used to evaluate the overlap between the predicted bounding box and the true bounding box, thereby accurately measuring the accuracy of positioning. The higher the PIoU value, the larger the overlap area between the predicted bounding box and the true bounding box, which indicates that the prediction result is more accurate. The calculation formula of the PIoU is shown in Formula II: Among them, A represents the predicted bounding box, B represents the real label box, and the value of i is 1-4. Represents the distance from the midpoint coordinates of the line connecting the predicted bounding box and the true labeled box to the coordinates of each point in the predicted bounding box. Represents the distance from the midpoint coordinates of the line connecting the predicted bounding box and the true marked box to the coordinates of each point in the true marked box. Indicates taking the maximum value of the distance.

4. The offshore wind turbine blade defect detection method according to claim 1, characterized in that: In step S01: screening of the plurality of marker boxes specifically includes: 1) discarding marker boxes whose bounding box confidence scores are lower than a threshold, 2) removing redundant marker boxes for each class using matrix non-maximum suppression until all redundant marker boxes are processed and the whole process ends.

5. The offshore wind turbine blade defect detection method according to claim 1, characterized in that: In step S02, each detected image is stitched together using image stitching technology, and the specific steps are as follows: S21. Extract features from the original single image and obtain feature point information of each image through supervised learning to facilitate subsequent image stitching operations; S22. According to the feature points matched in step S21, the geometric transformation relationship between different images is estimated to complete the stitching.

6. The offshore wind turbine blade defect detection method according to claim 5, characterized in that: In step S21, the feature extraction is performed on the original single image, the QX network structure is adopted, the FRConv convolution layer is used to reduce the model parameters, the BEAM module is introduced, the idea of ​​residual connection is used to ensure the performance of deep learning, focus on the information between channels, and attach importance to the distribution of features in the image space, so as to capture more local and global information about blade defects; The structure of the FRConv module includes a regional feature extractor, a context feature extractor and an integrated feature extractor, wherein the regional feature extractor and the context feature extractor are both connected to the integrated feature extractor; The FRConv module performs feature extraction, including the following steps: the regional feature extractor is a 3×3 standard convolution layer, which learns local features from 8 adjacent feature vectors; the context feature extractor is a 3×3 dilated convolution, which has a relatively large receptive field and can effectively learn the surrounding context information; the integrated feature extractor connects the output features of the regional feature extractor and the context feature extractor, and processes them through a batch normalization BatchNorm2d function and an FReLU activation function, and finally inputs the extracted feature map; the overall output of the FRConv module is expressed as: FRConv(x)=FReLU(BatchNorm2d(Concat(Conv 3×3 (x),AConv 3×3 (x)) Among them, FRConv is the output of the FRConv module, FReLU is the FReLU activation function, BatchNorm2d is the batch normalization BatchNorm2d function, Concat is the connection, Conv 3×3 AConv is a 3×3 standard convolutional layer. 3×3 is a 3×3 hole convolution, and x is the input feature map.

7. The offshore wind turbine blade defect detection method according to claim 6, characterized in that: In step S21, the BEAM module adds the original image information and the image information enhanced by the CBAM module through residual connection to ensure the continuity of features and the fluidity of information, and enhance the perception and learning capabilities of blade defect features.

8. The offshore wind turbine blade defect detection method according to claim 5, characterized in that: In step S22, the geometric transformation relationship between different images is estimated, specifically: the transformation parameters are estimated based on minimizing the reprojection error, so as to determine the positional relationship between the matched feature points, and the transformation of translation, rotation and scaling is fitted by least squares reprojection to provide support for subsequent image stitching; The geometric transformation based on minimizing the reprojection error comprises the following steps: The feature vector of any feature point matching pair in the matching image is used as input. The goal is to find a geometric transformation model so that one feature vector in the matching team is as close as possible to another feature vector after transformation, so as to better judge the transformation relationship. The affine transformation model is used, which includes three transformation parameters: translation, rotation and scaling. The reprojection error is used as the objective function to measure the error between the model prediction value and the actual value. The affine transformation model is shown in Formula VIII: Among them, (x i ,y i ) represents a feature vector in a matching pair, (x i ′,y i ′) represents the feature vector after geometric transformation, e, f, g, h represent the rotation and scaling parameters respectively, and p and q represent the translation parameters; The reprojection error is used as the objective function to measure the error between the model prediction value and the actual value. For a two-dimensional image, assuming there are N matching pairs, the i-th point on the source image is (x i ,y i ), the corresponding point on the target image is (x i ′,y i ′); using the reprojection error, the objective function is minimized by the gradient descent method to solve the optimal parameters of the corresponding geometric transformation, so as to estimate the geometric transformation relationship between the two images; according to the estimated geometric transformation relationship, different images are spliced; for each pixel in the overlapping area, the weighting coefficient is calculated according to the gradient difference of the surrounding pixels; the image gradient is calculated using the two-dimensional second-order difference method; The calculation formula of the reprojection error is shown in Formula IX: The operation of stitching different images according to the estimated geometric transformation relationship is specifically: using the geometric transformation relationship between images to perform geometric transformation on the images, and then combining the relevant results of feature matching to identify the overlapping areas between the images, and realizing seamless stitching by pixel fusion of the overlapping areas; For each pixel in the overlapping area, the weighted coefficient is calculated according to the gradient difference of the surrounding pixels. Specifically, assuming that the gradients of the two images are G1 and G2 respectively, the weighted fusion formula Blend(x,y) for each pixel in the overlapping area is as described in Formula X: Among them, I1(x, y) and I2(x, y) are the pixel values ​​of the two images in the overlapping area, and G1(x, y) and G2(x, y) are the gradient values ​​of the two images in the overlapping area; The two-dimensional second-order difference method is used to calculate the gradient of the image, and the gradient image is obtained by adding the second-order differences in the x direction and the y direction; the weighted fusion formula and the difference method are applied to each pixel in the overlapping area to effectively eliminate the discontinuity at the image splicing boundary and achieve smooth transition and continuity of the image; The two-dimensional second-order difference method formula is shown in Formula XI:

9. The offshore wind turbine blade defect detection method according to claim 1, characterized in that: In step S03: The determination of the actual physical coordinates of the fan blade corresponding to the blade defect in the original single image is specifically as follows: physically locating the defect on the spliced ​​image, and determining the defect coordinates with the lower left corner of the fan blade as the origin; before locating, the image needs to be cropped for invalid areas, the non-fan area is removed, and the lower left corner of the removed image is used as the origin of the coordinate system; In the process of determining the defect coordinates, a search algorithm based on an 8-domain relationship is used to extract the pixel position of each edge and its extension order on the edge. This algorithm can overcome the problem of pixel accumulation or discontinuity. The non-zero points in the binary image F(i,j) represent edges. The ideal state of these non-zero points in the image space is to present continuous, single-pixel distribution; however, some edges will have pixel accumulation or pixel discontinuity; The search algorithm based on the 8-domain relationship designs an edge determination operator Q(i, j) within the 8-neighborhood range of a pixel. Its functions are: first, determining whether the current pixel is an edge point according to the Q(i, j) value; second, determining the coordinates of the next edge point. The specific application method of the edge determination operator Q(i, j) is shown in Formula XII: Among them, the gray value of the edge point is 1, the gray value of the background point is 0, m and n are the indices used to traverse the 8-neighborhood pixels of the current pixel (i, j), which can be -1, 0, 1; Use the 8-domain relationship search algorithm to find the upper and lower boundaries of the fan blades, that is, find the maximum or minimum Y value on the left and the minimum or maximum Y value on the right, that is, the two points at the most boundary of the left and right sides, and determine the linear equation of the upper boundary YS through these two points as YS=KS*X1+BS; similarly, find the two points of the lower boundary through the search algorithm, and determine the linear equation of the lower boundary YU as YU=KU*X2+BU; Find the lowest point of the lower boundary, and modify the original upper boundary and lower boundary equations so that the lowest point of the lower boundary is the origin; the modified upper boundary equation is YS=KS*X1+BS-C1, and the lower boundary equation is YU=KU*X2+BU-C2; Where YS is the Y coordinate of any point on the upper boundary, KS is the slope of the upper boundary, X1 is the X coordinate of any point on the upper boundary, BS is the intercept of the upper boundary, and C1 is the constant subtracted from the upper boundary equation; Similarly, YU is the Y coordinate of any point on the lower boundary, KU is the slope of the lower boundary, X2 is the X coordinate of any point on the lower boundary, BU is the intercept of the lower boundary, and C2 is the constant subtracted from the equation of the lower boundary; In order to smooth the image features, better handle edge detection, and avoid too few data points, the IGP model is used to optimize the boundary equation. Taking the boundary equation YS as an example, it is assumed that KS obeys KS~N(μ0,∑0) and BS obeys BS~N(0,σ 2 I), where μ0 is the mean vector of KS, ∑0 is the prior covariance matrix, σ 2 is the variance of the error term, I is the identity matrix; Calculate the posterior distribution of KS, the formula is shown in Formula XIII: p(KS|X,YS)∝p(YS|X,KS)p(KS); Formula XIII; Where KS is the slope of the upper boundary, X is the X coordinate of any point on the upper boundary, and YS is the Y coordinate of any point on the upper boundary; The mean and covariance of YS are calculated as shown in Formula XIV: m post =∑0X T (X∑0X T +s 2 I) -1 Y.S.; ∑ post =∑0-∑0X T (X∑0X T +s 2 I) -1 X∑0; Formula XIV; Among them, μ post is the mean vector of YS, ∑ post is the covariance matrix of YS, ∑0 is the prior covariance matrix; M is a matrix, each row represents the X coordinate of any point on the upper boundary, T is the matrix transpose symbol; σ 2 is the variance of the error term, I is the identity matrix; Given a new input x * , predict output y * , the formula is shown in Formula XV: p(y * |x * ,X,YS)=∫p(y * |x * ,KS)p(KS|X,YS)dKS; Formula XV; Where X is the X coordinate of any point on the upper boundary, YS is the Y coordinate of any point on the upper boundary, and KS is the slope of the upper boundary; According to the above operation, the positioning mark frame of the defect in the final original image will be output, including the coordinates of the center position of the mark frame and the corresponding length and width; after the image stitching operation, the position of the segmentation line of different image stitching in the original image can be recorded, the leftmost position is 0, and the order between the images is known. Combining the above two information, the position information of each segmentation line in the entire image can be calculated; according to the position distribution of the segmentation line, a certain digital conversion is performed to obtain the corresponding position of the leaf defect in the original image before the single stitching.

10. The offshore wind turbine blade defect detection method according to claim 9, characterized in that: In step S03: the center coordinates (x i ,y i ), length w, width h, it is easy to find the coordinates of the upper right corner, lower right corner, upper left corner and lower left corner corresponding to the marked box, and use the midpoint coordinates of the marked box to represent the coordinates of the defect, that is, X q =( x i+w) / 2,Y q =(y i +h) / 2; According to the upper and lower boundaries, first calculate the defect coordinate Y q Perform relative transformation, if Y q At the upper end of the upper limit or the lower end of the lower limit, the defect detection is considered abnormal and will be discarded; q Between the upper and lower boundaries, it is transformed, that is, Y q The Manhattan distance to the origin of the lower boundary is For the defect coordinate X q The transformation takes into account the influence of horizontal overlap in image stitching. In this graphic stitching algorithm, the search range of key points can be dynamically adjusted to find the best matching point pair. The corresponding key points in the two images are found using the FRConv module and Fosine_Similarity. The width of the overlapping part is determined by calculating the distance between the key points. The nearest n key points are selected and the Manhattan average distance D between them is calculated. i , that is, D i =SUM(n i xn j x) / n,n i x and n j x is n i , n j x coordinates; get the D between adjacent images i , the width of each sub-image is w i , then the horizontal position L of the segmentation line of the ith sub-image in the spliced ​​image i It is expressed as: Then the transformed X qe =L i -D.

Images