Wind power defect dual-light fusion automatic detection technology based on unmanned aerial vehicle image acquisition

Through the drone collecting infrared and visible light images and combining parallel channel attention module and self-attention mechanism, efficient detection of wind turbine blades is achieved, solving the problems of high risk, low efficiency and high maintenance costs of traditional detection methods, and providing A brand new testing solution.

CN119027381APending Publication Date: 2024-11-26GUODIAN POWER INNER MONGOLIA NEW ENERGY DEV CO LTD
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411063595.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Due to the height and angle limitations of wind turbine blades, traditional manual inspection has high risks, low efficiency and high maintenance costs, making it difficult to effectively detect the defects of the blades, especially in extreme environments.

Method used

The drone is used to collect infrared and visible light images, extract feature maps through the dual-channel backbone network layer, and combine the parallel channel attention module and self-attention mechanism to integrate feature maps for detection and identification, achieving comprehensive and efficient detection of blades.

Benefits of technology

Significantly reduce the risk of high-altitude operations, improve detection efficiency, reduce downtime, enhance the difference in target characteristics and environmental semantics, solve the problem of different sizes of blade image defects, and provide a brand new detection solution for fan blades.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027381A_ABST
    Figure CN119027381A_ABST
Patent Text Reader

Abstract

The invention discloses a wind power defect dual-light fusion automatic detection technology based on unmanned aerial vehicle image acquisition, and the technology comprises the following steps: firstly, enabling an unmanned aerial vehicle to acquire an infrared image and a visible light image of a wind power generator blade; performing data processing on the infrared image and the visible light image in an input layer; then, the double-input-layer double-path backbone network layer extracts multi-level feature maps of infrared light and visible light; secondly, fusing the multi-level feature maps by the double-channel neck network layer to obtain a feature map; thirdly, inputting the infrared and visible light feature maps into a double-path feature fusion module for feature fusion to obtain a fused feature map; and finally, inputting the fused feature map into a detection layer for detection and identification. According to the wind power defect dual-light fusion automatic detection technology based on unmanned aerial vehicle image acquisition, the labor intensity is reduced through the unmanned aerial vehicle technology; and the global visual scene context is captured, so that the problem of different defect scales of the aerial blade image is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of rewinding and laminating machines, and in particular relates to a dual-light fusion automatic detection technology for wind defects based on unmanned aerial vehicle image acquisition. Background Art

[0002] As an emerging renewable energy, wind energy is gradually playing an important role in my country's electric energy structure due to its remarkable environmental protection characteristics and relatively low cost advantages. As the core equipment for converting wind energy into electrical energy, the importance of its blade components is self-evident.

[0003] Related research shows that the increase in blade surface roughness and the accumulation of defects will directly lead to a significant decrease in power generation efficiency, ranging from 5% to 30%. More seriously, in some extreme cases, these problems may even induce the fracture of the blade structure, posing a serious threat to the safe operation of the wind turbine.

[0004] Failure or defects in wind turbine blades may be caused by a variety of factors. Scratches caused during long-distance transportation, erosion by wind, sand, rain, snow and lightning in harsh environments, and fatigue damage caused by high-frequency alternating loads can all cause irreversible damage to the blades.

[0005] The results of long-term inspection of blades reveal that the main defects of blades include the shedding of surface coating, the formation of sand holes, the expansion of cracks, corrosion of edges, cracking, wrinkling, lightning strike marks, and ice formation. It is worth noting that blade damage is one of the most expensive types of damage to repair, and its repair or replacement cost accounts for about 20% to 30% of the total cost of the wind turbine, which brings significant economic pressure to the long-term operation and maintenance of the wind turbine.

[0006] However, due to the height and angle restrictions of the wind turbine, blade inspection faces great challenges and becomes a major problem in the operation. Traditional inspection methods mainly rely on manual work, but there are obvious limitations in inspection accuracy, speed and safety. Therefore, the inspection process is not only difficult, but also accompanied by high safety risks. A new and more efficient inspection method is urgently needed to meet this challenge. Summary of the invention

[0007] Purpose of the invention: In order to overcome the above shortcomings, the purpose of the present invention is to provide a dual-light fusion automatic detection technology for wind defects based on drone image acquisition. Compared with traditional detection methods, drones can significantly reduce the risk of high-altitude operations, reduce labor intensity, improve detection efficiency and greatly reduce downtime; according to the characteristics of feature maps of different levels and modalities, the complementary content of spatial details and semantic information of infrared and visible light images can be extracted to realize heterogeneous image target recognition; the global visual scene context can be captured, the difference between target features and environmental semantics can be increased, and the problem of different scales of defects in aerial blade images can be solved, providing a new solution for comprehensive and efficient detection of wind turbine blades.

[0008] Technical solution: In order to achieve the above purpose, the present invention provides a wind defect dual-light fusion automatic detection technology based on drone image acquisition, comprising the following steps:

[0009] Step 1: The drone collects infrared images and visible light images of wind turbine blades; Step 2: The infrared images and visible light images are processed in the input layer as the input of the dual-input layer dual-path backbone network layer; Step 3: The dual-input layer dual-path backbone network layer extracts the infrared multi-level feature map and the visible light multi-level feature map; Step 4: The dual-path neck network layer fuses the infrared multi-level feature map and the visible light multi-level feature map respectively to obtain the infrared feature map and the visible light feature map; Step 5: The infrared feature map and the visible light feature map are input into the dual-path feature fusion module for feature fusion to obtain the fused feature map; Step 6: The fused feature map is input into the detection layer for detection and recognition.

[0010] Furthermore, the data processing operation of the infrared image and the visible light image in the input layer in step 2 includes: step 201: copying the information of the infrared image so that the number of channels of the input infrared image increases to 3; step 202: inputting the image into the network layer, at this time the number of channels is 6, of which the visible light image has 3 channels, namely RGB channels; the number of channels of the infrared image is 3. Increasing the number of channels increases the dimension of feature extraction.

[0011] Furthermore, the dual-input layer dual-path backbone network layer in step 2 includes two YOLOv7 backbone networks, which respectively process infrared images and visible light images and extract image features of both. Feature extraction is performed on infrared images and visible light images respectively, enriching the features of the images.

[0012] Furthermore, the dual-path neck network layer in step 4 is a group of neck networks, which respectively process infrared image features and visible light image features; the neck network is composed of a dual-path attention module, a parallel channel attention module and a yolov7 neck network in series. The dual-path attention mechanism, the parallel channel attention module and the yolov7 neck network are used in series to integrate image features.

[0013] Furthermore, the parallel channel attention module retains the channel attention module MC and the convolutional spatial attention module MS with 3D arrangement and multi-layer perceptron, and the calculation process is as follows:

[0014]

[0015] F2=F′+F″

[0016] Among them, F1 is the input feature, F2 is the output feature, and M C () represents the channel attention calculation operation, M S () represents the convolutional spatial attention calculation operation. The parallel channel attention module combines channel attention and convolutional attention to simultaneously focus on global information and important features, capture the global visual scene context, increase the semantic difference between target features and the environment, and to some extent solve the problem of different scales of blade surface defects.

[0017] Furthermore, the dual-path feature fusion module in step 5 includes step 501: dual-path feature information fusion, pooling and convolution; step 502: local feature extraction of the image, using the Sigmoid function in the channel dimension to generate infrared weights and visible light weights, and multiplying the weights with the source image feature map, dynamically adjusting the w value according to the different weight values ​​in the feature map to generate the weight map, and obtaining the local features of the image; at the same time, the product result is added to the source image and convolution smoothing is performed; step 503: context information extraction of the image, performing self-attention mechanism operations through different expansion rates, and extracting the context information of the image in different receptive fields. The dual-path feature fusion module strengthens the information of the fused features and enhances the expression performance of the fused image.

[0018] Furthermore, the dual-path feature information fusion, pooling and convolution calculation process in step 501 is as follows:

[0019] Z r =PW_Conv 2 (AAP(C(I ir +I Vi )))

[0020] Among them, Z r is the processed global feature vector; AAP() is the adaptive average pooling function; PW_Conv2 () represents the point convolution layer; C() represents the concat function; I ir ,I Vi Represents the input infrared and visible light feature maps. This step completes the preliminary generation of the global feature vector.

[0021] Furthermore, the local feature extraction calculation process of the image in step 502 is as follows:

[0022] w=δ(F i (Z f ))

[0023] Where w represents the generated infrared weight; δ() represents the Sigmoid function; F i () represents the result set of batch processing; since infrared images and visible light images have complementary features, the infrared weight is set to w and the visible light weight is 1-w. The product result is added to the source image and convolution smoothed, so that the fused features have both global and local features.

[0024] Furthermore, the context information extraction of the image in step 503 includes self-attention calculation and generation of enhanced fusion feature variables, and the calculation process is as follows:

[0025] X i =Attention(Q, K, V, r i )

[0026] S r =Conv(C(X1, X2, X3, X4))

[0027] Among them, Q, K, V represent query, key and value matrices, which are calculated by weights; r i represents the i-th expansion rate, i∈{1, 2, 3, 4}, Attention() is the self-attention calculation, X i is the fusion result under the i-th expansion rate; Conv() is the convolution operation, C() represents the concat function, S r The self-attention mechanism is operated by different expansion rates to extract the context information of the image in different receptive fields, which improves the performance of the model.

[0028] Furthermore, in step 6, the detection layer in which the fused feature map is input into the detection layer for detection and identification is a classifier of yolov7. The classifier of yolov7 can accurately identify the target object in a complex background, and has high detection accuracy.

[0029] It can be seen from the above technical solution that the present invention has the following beneficial effects:

[0030] The purpose of the present invention is to provide a dual-light fusion automatic detection technology for wind defects based on drone image acquisition. Compared with traditional detection methods, drones can significantly reduce the risk of high-altitude operations, reduce labor intensity, improve detection efficiency and greatly reduce downtime; according to the characteristics of different levels and different modal feature maps, the complementary content of infrared and visible light images in spatial details and semantic information can be extracted to realize heterogeneous image target recognition; the global visual scene context can be captured, the difference between target features and environmental semantics can be increased, and the problem of different scales of defects in aerial blade images can be solved, providing a new solution for comprehensive and efficient detection of wind turbine blades. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a general schematic diagram of a wind defect dual-light fusion automatic detection technology based on drone image acquisition according to the present invention;

[0032] Figure 2 A schematic diagram of a parallel channel attention mechanism of a wind defect dual-light fusion automatic detection technology based on drone image acquisition according to the present invention;

[0033] Figure 3 This is an example diagram of a dual-path feature fusion module of the dual-light fusion automatic detection technology for wind defects based on drone image acquisition described in the present invention. DETAILED DESCRIPTION

[0034] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0035] Example 1

[0036] In this embodiment, Figure 1 The present invention discloses a wind defect dual-light fusion automatic detection technology based on drone image acquisition, comprising the following steps:

[0037] Step 1: The UAV collects infrared images and visible light images of wind turbine blades; Step 2: The infrared images and visible light images are processed in the input layer (input) as the input of the dual-input layer dual-path backbone network layer (Backbone); Step 3: The dual-input layer dual-path backbone network layer (Backbone) extracts infrared multi-level feature maps and visible light multi-level feature maps; Step 4: The dual-path neck network layer fuses the infrared multi-level feature map and the visible light multi-level feature map respectively to obtain the infrared feature map and the visible light feature map; Step 5: The infrared feature map and the visible light feature map are input into the dual-path feature fusion module (Feature Fusion Module) for feature fusion to obtain a fused feature map; Step 6: The fused feature map is input into the detection layer (Detect) for detection and recognition.

[0038] Example 2

[0039] On the basis of Example 1, in this example, Figure 1 The present invention discloses a wind defect dual-light fusion automatic detection technology based on drone image acquisition. In step 2, the infrared image and the visible light image are subjected to data processing operations at the input layer (input), including: step 201: copying the information of the infrared image so that the number of channels of the input infrared image is increased to 3; step 202: inputting the image into the network layer, at this time the number of channels is 6, of which the visible light image has 3 channels, namely RGB channels; the number of channels of the infrared image is 3.

[0040] In this embodiment, Figure 1 In step 2, the dual-input layer dual-path backbone network layer includes two YOLOv7 backbone networks, which respectively process infrared images and visible light images to extract image features of the two; the backbone network part of the YOLOv7 network model is mainly constructed by convolution, E-ELAN module, MPConv module and SPPCSPC module; among them, the E-ELAN (Extended-ELAN) module, based on the original ELAN, changes the calculation block while maintaining the transition layer structure of the original ELAN, and uses the ideas of expand, shuffle, and merge cardinality to enhance the network learning ability without destroying the original gradient path; the SPPCSPC module adds multiple parallel maximum pooling operations to a series of convolutions, avoiding problems such as image distortion caused by image processing operations, and solving the problem of convolutional neural network extracting repeated features of images; in the MPConv module, the maximum pooling operation expands the receptive field of the current feature layer and then merges it with the feature information after normal convolution processing, thereby improving the generalization of the network.

[0041] In this embodiment, Figure 1 and Figure 2 In step 4, the dual-path neck network layer is a group of neck networks, which respectively process infrared image features and visible light image features; the neck network is composed of a dual-routing attention module (Bi-Level Routing Attention, BRA), a parallel channel attention module (Parallel Channel Attention, PCA) and a yolov7 neck network in series.

[0042] In this embodiment, Figure 1 and Figure 2 The parallel channel attention module retains the channel attention module MC and the convolutional spatial attention module MS with 3D arrangement and multilayer perceptron (MLP), wherein the channel attention module retains feature information in three dimensions through 3D arrangement, and then uses the multilayer perceptron to amplify the dependency of the cross-dimensional channel space, thereby enhancing the model's ability to deeply understand the relationship between channels; the spatial attention module uses two 7×7 convolution kernels for spatial information fusion, and the model can focus on and extract key information in space more accurately; in addition, in order to avoid the interference of the serial structure channel attention module on the spatial attention module, the parallel channel attention module adopts a parallel connection structure, and the calculation process is as follows:

[0043]

[0044]

[0045] F2=F′+F″

[0046] Among them, F1 is the input feature, F2 is the output feature, and M C () represents the channel attention calculation operation, M S () represents the convolutional spatial attention calculation operation.

[0047] In this embodiment, Figure 1 and Figure 3 The dual-path feature fusion module in step 5 includes step 501: dual-path feature information fusion, pooling and convolution; step 502: local feature extraction of the image, using the Sigmoid function in the channel dimension to generate infrared weights and visible light weights, and multiplying the weights with the source image feature map, dynamically adjusting the w value according to the different weight values ​​in the feature map to generate a weight map to obtain the local features of the image; at the same time, the product result is added to the source image and convolution smoothing is performed; step 503: context information extraction of the image, performing self-attention mechanism operations through different expansion rates, and extracting the context information of the image on different receptive fields.

[0048] In this embodiment, Figure 1 and Figure 3 The process of dual-path feature information fusion, pooling and convolution calculation in step 501 is as follows:

[0049] Z r =PW_Conv 2 (AAP(C(I ir +I vi )))

[0050] Among them, Z r is the processed global feature vector; AAP() is the adaptive average pooling function; PW__Conv 2 () represents the point convolution layer; C() represents the concat function; I ir , I Vi Represents the input infrared and visible light feature maps.

[0051] In this embodiment, Figure 1 and Figure 3 The local feature extraction calculation process of the image in step 502 is as follows:

[0052] w=δ(F i (Z r ))

[0053] Where w represents the generated infrared weight; δ() represents the Sigmoid function; F i () represents the result set of batch processing; since infrared images and visible light images have complementary features, the infrared weight is set to w and the visible light weight is set to 1-w.

[0054] In this embodiment, Figure 1 and Figure 3 , the context information extraction of the image in step 503 includes self-attention calculation and generation of enhanced fusion feature variables, and the calculation process is as follows:

[0055] X i =Attention(Q, K, V, r i )

[0056] S r =Conv(C(X1, X2, X3, X4))

[0057] Among them, Q, K, V represent query, key and value matrices, which are calculated by weights; r i represents the i-th expansion rate, i∈{1, 2, 3, 4}, Attention() is the self-attention calculation, X i is the fusion result under the i-th expansion rate; Conv() is the convolution operation, C() represents the concat function, S ris the fusion feature variable.

[0058] In this embodiment, Figure 1 In step 6, the fused feature map is input into the detection layer for detection and recognition, and the detection layer is the classifier of yolov7.

[0059] The working principle of the above embodiment is:

[0060] The present invention discloses a wind defect dual-light fusion automatic detection technology based on drone image acquisition, which uses drone technology to collect infrared images and visible light images; then constructs a dual-path neck network layer to fuse its own multi-layer feature map. In particular, a parallel channel attention mechanism is set in the dual-path neck network layer, which combines channel attention and convolution attention to pay attention to global information and important features at the same time; then, the image feature information is enhanced through a dual-light feature fusion module; finally, the output fusion feature map is input into the detection layer for detection.

[0061] The algorithm in this paper is trained on the wind turbine blade dataset, and the experimental results on this dataset are compared with those of current mainstream target detection algorithms such as YOLOv8, YOLOv5, and Faster RCNN, as shown in Table 1.

[0062] Table 1

[0063]

[0064] As shown in Table 1, the proposed algorithm achieves 85.32% in mAP, which is a significant improvement compared to other algorithms. Due to the use of infrared and visible light feature modules, it can focus on important blade defect features while paying attention to global information; in addition to reducing large-area surface corrosion defects, the proposed algorithm has a significant overall improvement in the detection accuracy of small target defects on wind turbine blades.

[0065] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be regarded as within the protection scope of the present invention.

Claims

1. A wind defect dual-light fusion automatic detection technology based on drone image acquisition, Features: The steps include: Step 1: The drone collects infrared images and visible light images of the wind turbine blades; Step 2: Process the infrared image and the visible light image in the input layer, copy the information of the infrared image, and use the processed infrared image and visible light image as the input of the dual-input layer dual-path backbone network layer; Step 3, the dual-input layer dual-path backbone network layer extracts infrared multi-level feature maps and visible light multi-level feature maps; Step 4: The dual-path neck network layer fuses the infrared multi-level feature map and the visible light multi-level feature map respectively to obtain an infrared feature map and a visible light feature map; Step 5, inputting the infrared feature map and the visible light feature map into a dual-path feature fusion module for feature fusion to obtain a fused feature map; Step 6: Input the fused feature map into the detection layer for detection and recognition.

2. According to the wind defect dual-light fusion automatic detection technology based on drone image acquisition according to claim 1, Features: The step 2 includes performing data processing operations on the infrared image and the visible light image at the input layer: Step 201: copying the information of the infrared image so that the number of channels of the input infrared image increases to 3; Step 202: Input the image into the network layer. At this time, the number of channels is 6, of which the visible light image has 3 channels, namely RGB channels; the number of channels of the infrared image is 3.

3. According to the wind defect dual-light fusion automatic detection technology based on drone image acquisition according to claim 1, Features: The dual-input layer dual-path backbone network layer in step 2 includes two YOLOv7 backbone networks, which respectively process infrared images and visible light images and extract image features of the two.

4. According to the wind defect dual-light fusion automatic detection technology based on drone image acquisition according to claim 1, Features: The dual-path neck network layer in step 4 is a group of neck networks, which respectively process infrared image features and visible light image features; the neck network is composed of a dual-path attention module, a parallel channel attention module and a yolov7 neck network in series.

5. According to the wind defect dual-light fusion automatic detection technology based on drone image acquisition according to claim 4, Features: The parallel channel attention module retains the channel attention module MC and the convolutional spatial attention module MS with 3D arrangement and multi-layer perceptron, and the calculation process is as follows: F 2 =F′+F″ Among them, F 1 is the input feature, F 2 is the output feature, M C () represents the channel attention calculation operation, M S () represents the convolutional spatial attention calculation operation.

6. According to claim 1, the wind defect dual-light fusion automatic detection technology based on drone image acquisition, Features: The dual-path feature fusion module in step 5 includes: Step 501: dual-path feature information fusion, pooling and convolution; Step 502: extract local features of the image, generate infrared weights and visible light weights in the channel dimension using the Sigmoid function, and multiply the weights with the source image feature map. According to the different weight values ​​in the feature map, dynamically adjust the w value to generate the weight map to obtain the local features of the image; at the same time, add the product result to the source image and perform convolution smoothing processing; Step 503: extracting context information of the image by performing self-attention mechanism operation at different dilation rates to extract context information of the image at different receptive fields.

7. According to claim 6, the wind defect dual-light fusion automatic detection technology based on drone image acquisition, Features: The process of dual-path feature information fusion, pooling and convolution calculation in step 501 is as follows: Z r =PW_Conv 2 (AAP(C(I ir +I Vi , Among them, Z r is the processed global feature vector; AAP() is the adaptive average pooling function; PW_Conv 2 () represents the point convolution layer; C() represents the concat function; I ir ,I Vi Represents the input infrared and visible light feature maps.

8. According to claim 7, the wind defect dual-light fusion automatic detection technology based on drone image acquisition, Features: The calculation process of extracting local features of the image in step 502 is as follows: w=δ(F i (Z r )) Where w represents the generated infrared weight; δ() represents the Sigmoid function; F i () represents the result set of batch processing; since infrared images and visible light images have complementary features, the infrared weight is set to w and the visible light weight is set to 1-w.

9. The wind defect dual-light fusion automatic detection technology based on drone image acquisition according to claim 8, Features: The context information extraction of the image in step 503 includes self-attention calculation and generation of enhanced fusion feature variables, and the calculation process is as follows: X i =Attention(Q,K,V,r i ) S r =Conv(C(X 2 ,X 2 ,X 3 X 4 )) Among them, Q, K, V represent query, key and value matrices, which are calculated by weights; r i represents the i-th expansion rate, i∈{1,2,3,4}, Attention() is the self-attention calculation, X i is the fusion result under the i-th expansion rate; Conv() is the convolution operation, C() represents the concat function, S r is the fusion feature variable.

10. The wind defect dual-light fusion automatic detection technology based on drone image acquisition according to claim 1, Features: The step 6 inputs the fused feature map into the detection layer for detection and recognition, wherein the detection layer is a classifier of yolov7.

Citation Information

Cited By

  • Fan blade icing monitoring method and device based on bimodal image fusion

    CN119741215A

  • Wind turbine blade icing monitoring method and device based on dual-modal image fusion

    CN119741215B

  • Multi-modal data fused strain clamp defect unmanned aerial vehicle detection method

    CN119888547A

  • Intelligent wind power station inspection management method, device and equipment and storage medium

    CN121304139A

  • Bimodal unmanned aerial vehicle blade inspection method and system based on lightweight target detection algorithm

    CN122049753A