Wind power plant unmanned aerial vehicle intelligent inspection method based on coupling of high-definition image capture and YOLO image recognition

Abstract

The invention relates to a wind power plant unmanned aerial vehicle intelligent inspection method based on coupling of high-definition image capture and YOLO image recognition, and aims to solve the problems that fan components and defects thereof are difficult to accurately recognize in a traditional target detection technology, and a large amount of manual participation is still needed in the inspection process. According to the wind power plant unmanned aerial vehicle intelligent inspection method based on coupling of high-definition image capture and YOLO image recognition, a perfect fan damage image database is established by using Mosaic image enhancement and high-resolution reconstruction technologies, and the diversity and completeness of data required by model training are effectively enhanced. Besides, the method combines the advantages of the pruning model, can greatly reduce model parameters and calculation amount, improve the real-time performance and deployment flexibility of the inspection system and realize the accuracy, automation and intelligentization of the inspection process on the premise of keeping the identification performance of the model basically unchanged, and has obvious technical advantages.

Description

Technical Field

[0001] The present invention relates to the field of wind farm drone intelligent inspection and image recognition, and in particular to a wind farm drone intelligent inspection method based on the coupling of high-definition image capture and YOLO image recognition, which is used to realize automatic recognition and damage detection of wind turbine blades, towers and other outer surface parts, thereby improving inspection efficiency and accuracy. Background Art

[0002] With the rapid development of the wind power industry, the number of wind turbines has increased rapidly, and the demand for regular inspections of wind turbines has also increased. Traditional wind turbine inspections mainly rely on manual climbing inspections or ground telescope observations, which have high safety risks and low efficiency. In recent years, although there has been a practice of using drones to take images of wind turbines to assist in inspections, there is a lack of efficient target detection algorithms to automatically identify wind turbine components and their defects, resulting in a large amount of manual participation in the inspection process, which cannot give full play to the advantages of drones. Summary of the invention

[0003] In view of the above situation, in order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a wind farm drone intelligent inspection method based on the coupling of high-definition image capture and YOLO image recognition, so as to realize automatic recognition and damage detection of wind turbine blades, towers and other external surface parts.

[0004] The technical solution provided by the present invention is:

[0005] 1. A wind farm drone intelligent inspection method based on high-definition image capture and YOLO image recognition coupling, characterized in that it includes the following steps:

[0006] S1. Image acquisition: The drone equipped with a high-definition camera flies to the top of the designated wind turbine according to the preset route, quickly locates and finds a suitable hovering position, and controls the gimbal to shoot the target. The data of key parts of the wind turbine are collected one by one according to the preset route. The data of key parts of the wind turbine include the wind turbine blades, tower and other external surface parts. After completing the data collection task of a single wind turbine, the drone automatically flies to the next target wind turbine according to the preset route, and repeatedly performs positioning, hovering and shooting operations until the information of all wind turbines in the wind farm is collected.

[0007] S2. Image preprocessing and expansion: Perform multiple rounds of image enhancement processing on the images collected by the drone, including conventional image enhancement, histogram equalization, and mosaic image enhancement; this method uses conventional image enhancement methods such as flipping, brightness, blurring, and scaling to simulate the changes in the brightness of the shooting light, blurring to simulate the changes in the camera's defocus, and scaling to simulate the changes in the distance of the camera. The histogram equalization formula is used to indirectly improve the image contrast:

[0008]

[0009] Among them, I out (x, y) is the output image, cdf(I(x, y)) is the cumulative distribution function of the input image, M×N is the total number of image pixels, and cdf min is the minimum cumulative frequency.

[0010] The Mosaic image enhancement method is used to increase the diversity of sample data, enrich the scenes of the data set, and simulate the situation of the target in different environmental combinations. For the four images I1(x, y), I2(x, y), I3(x, y) and I4(x, y) in the original sample set, where (x, y) represents the pixel coordinates of the image, the newly generated I mosaic The size of (x, y) is usually twice the size of the original single image (2ω×2h), and the specific coordinate expression is:

[0011] When 0≤x<ω and 0≤y<h, I mosaic (x,y)=I1(x,y);

[0012] When ω≤x<2ω and 0≤y<h, I mosaic (x,y)=I2(x-ω,y);

[0013] When 0≤x<ω and h≤y<2h, I mosaic (x,y)=I3(x,yh);

[0014] When ω≤x<2ω and h≤y<2h, I mosaic (x,y)=I4(x-ω,yh);

[0015] Where ω and h are the width and height of the original image respectively.

[0016] When the comprehensive image quality index IQI exceeds 0.85 and the change of IQI does not exceed 5% after three consecutive image enhancement operations, the enhancement is stopped and the enhanced image is output. The specific expression of IQI is as follows:

[0017] IQI=S cont +S hist +S diver

[0018] In the formula, S cont is the contrast suitability of the image after enhancement, S hist is the histogram uniformity, S diver Added strength to Mosaic diversity.

[0019] S3, Image super-resolution reconstruction: This reconstruction is mainly based on the image degradation model to convert the original image into a high-resolution image to reduce the impact of noise. Its mathematical formula can be summarized as:

[0020] Y=(Xn) / H

[0021] Among them, Y represents the high-resolution image, X represents the low-resolution image, H represents the degradation function, and n represents the additional noise;

[0022] S4. Image defect annotation: Annotate the processed sample set and divide the images into six categories: normal, crack, epidermal exfoliation, wear, lightning strike and other damage; (1) Normal category: The fan components in the image are intact, without any visible signs of damage, with clear and continuous surface texture and uniform and normal color; (2) Crack category: Obvious linear cracks appear on the surface of the fan components, such as fine hairline or wider cracks; (3) Epidermal exfoliation: The protective coating and composite materials on the fan surface fall off; (4) Wear: The surface texture of the component is uneven and the roughness increases; (5) Lightning strike: The surface shows typical lightning damage characteristics, such as ablation marks, carbonization, local material melting or explosive damage. (6) Other damage: Any other visible damage that cannot be classified into the above five types of damage is classified into other damage categories;

[0023] S5, model training: divide the sample data set marked in step S4 into training set, validation set, and test set according to 6:3:1; input the training set into the YOLOv5 model for model training, and the overall loss function (L sum ) is the sum of classification loss, positioning loss and confidence loss. The specific expression is:

[0024] L sum =λ cls ·L cls +λ loc ·L loc +λ conf ·L conf

[0025] Among them, L cls , L loc , L conf They are classification loss, positioning loss and confidence loss, respectively, cls , loc , conf They are the weight factors corresponding to classification loss, positioning loss and confidence loss respectively;

[0026] Classification loss L cls The specific expression is:

[0027]

[0028] Where N is the total number of samples, y i is the true label of the i-th sample, p i is the predicted probability of the i-th sample;

[0029] Positioning loss L loc The specific expression is:

[0030] L loc =1-r iou +L r +L center

[0031] Among them, r iou is the intersection-over-union ratio between the predicted bounding box and the true bounding box, L r is the aspect ratio loss, which is used to measure the difference between the aspect ratio of the predicted bounding box and the true bounding box; L center is the center point loss, which is used to measure the distance between the center point of the predicted bounding box and the true bounding box;

[0032] Confidence loss L conf The specific expression is:

[0033]

[0034] Where M is the total number of predicted bounding boxes, c i is the true confidence of the i-th predicted bounding box, is the prediction confidence of the i-th predicted bounding box;

[0035] S6, model optimization: input the validation set into the YOLOv5 model. When the output loss function is greater than 1.5, adjust the YOLOv5 model parameters, including the number of C3 network structure component modules in the backbone part, the width_multiple scaling factor, and the number of convolutional layer channels, and go to step S5 to continue model training; if the output loss function is less than 1.5, stop adjusting the model parameters and complete the model training;

[0036] S7, Deployment Inference: Use model pruning methods to remove redundant weights to reduce model size while maintaining model accuracy;

[0037] Using the Network Slimming pruning algorithm, the scaling factor γ in the BN (Batch Normalization) layer is used to evaluate the importance of the feature map and prune based on this. The formula is:

[0038] y(k)=γ(k)x(k)+β(k)

[0039] Among them, y(k) is the output of the BN layer, x(k) is the input, and γ(k) and β(k) are training parameters. By introducing L1 regularization to control γ, the importance of each feature is extracted, and then pruning is performed.

[0040] Model evaluation: Input the test set of step S5 into the optimized YOLOv5 model, output the defect recognition and classification results, and use the three indicators of accuracy, recall rate, and mAP to evaluate the model effect.

[0041] Accuracy = TP / (TP+FP)

[0042] Recall = TP / (TP+FN)

[0043]

[0044] Among them, TP is a true positive example, FP is a false positive example, FN is a false negative example, C is the number of categories, and AP(c) is the area under the precision-recall curve corresponding to the category.

[0045] S8. Inspection implementation: The wind turbine drone inspection takes high-definition images according to steps S1-S3 and enhances the images. The enhanced images are input into the improved YOLOv5 model in step S7, and the image recognition results are output. The results will clearly identify the possible fault points and fault categories on the wind turbine.

[0046] In the face of the problem that traditional target detection technology is difficult to accurately identify wind turbine components and their defects, resulting in the inspection process still requiring a lot of manual participation, the wind farm drone intelligent inspection method based on high-definition image capture and YOLO image recognition coupling uses Mosaic image enhancement and high-resolution reconstruction technology to establish a complete wind turbine damage image database, effectively enhancing the diversity and completeness of the data required for model training. In addition, the method of the present invention combines the advantages of the pruning model, and can significantly reduce the model parameters and calculation amount while keeping the model recognition performance basically unchanged, improve the real-time performance and deployment flexibility of the inspection system, and realize the accuracy, automation, and intelligence of the inspection process, which has obvious technical advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic flow chart of a method for intelligent inspection of wind farms by drones based on coupling high-definition image capture and YOLO image recognition provided in an embodiment of the present invention.

[0048] Figure 2 Comparison of image resolution before and after reconstruction provided by an embodiment of the present invention.

[0049] Figure 3 The output result of the YOLOv5 model provided by the embodiment of the present invention.

[0050] Figure 4 This is the wind turbine blade damage detection effect provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0051] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0052] See also Figure 1 , a wind farm drone intelligent inspection method based on high-definition image capture and YOLO image recognition coupling in this embodiment includes the following steps:

[0053] S1. Image acquisition: The drone equipped with a high-definition camera flies to the top of the designated wind turbine according to the preset route, quickly locates the hovering position, and controls the gimbal to shoot the target, and collects data on key parts of the wind turbine one by one according to the preset route. After completing the data collection task of a single wind turbine, the drone automatically flies to the next target wind turbine according to the preset route, and repeatedly performs positioning, hovering, and shooting operations until all wind turbine information in the wind farm is collected. This embodiment shoots 800 data sets.

[0054] S2. Image preprocessing and expansion: Multiple rounds of image enhancement processing are performed on the images collected by the drone, mainly including conventional image enhancement, histogram equalization and Mosaic image enhancement. After Mosaic image enhancement and data screening, the data set size increases to 2200 images.

[0055] S3, Image super-resolution reconstruction: This reconstruction is mainly based on the image degradation model to convert the original image into a high-resolution image to reduce the impact of noise. Its mathematical formula can be summarized as:

[0056] Y=(Xn) / H

[0057] Among them, Y represents the high-resolution image, X represents the low-resolution image, H represents the degradation function, and n represents the additional noise.

[0058] S4. Image defect annotation: The processed sample set is processed by manual annotation, and the images are divided into six categories: normal, crack, epidermal exfoliation, wear, lightning strike and other damage. (1) Normal category: The fan components in the image are intact, without any visible signs of damage, with clear and continuous surface texture and uniform and normal color; (2) Crack category: Obvious linear cracks appear on the surface of the fan components, such as fine hairline or wider cracks; (3) Epidermal exfoliation: The protective coating and composite materials on the fan surface fall off; (4) Wear: The surface texture of the component is uneven and the roughness increases; (5) Lightning strike: The surface shows typical lightning damage characteristics, such as ablation marks, carbonization, local material melting or explosive damage. (6) Other damage: Any other visible damage that cannot be classified into the above five types of damage is classified into the other damage category.

[0059] S5, model training: divide the sample data set marked in step S4 into training set, validation set, and test set according to 6:3:1; input the training set into the YOLOv5 model for model training, and the overall loss function (L sum ) is the sum of classification loss, positioning loss and confidence loss. The specific expression is:

[0060] L sum =λ cls ·L cls +λ loc ·L loc +λ conf ·L conf

[0061] Among them, L cls , L loc , L conf They are classification loss, positioning loss and confidence loss, respectively, cls , loc , conf They are the weight factors corresponding to classification loss, positioning loss and confidence loss respectively.

[0062] Classification loss L cls The specific expression is:

[0063]

[0064] Where N is the total number of samples, y i is the true label of the i-th sample, p i is the predicted probability of the ith sample.

[0065] Positioning loss L loc The specific expression is:

[0066] L loc =1-riou +L r +L center

[0067] Among them, r iou is the intersection-over-union ratio between the predicted bounding box and the true bounding box, L r is the aspect ratio loss, which is used to measure the difference between the aspect ratio of the predicted bounding box and the true bounding box; L center is the center point loss, which is used to measure the distance between the center point of the predicted bounding box and the true bounding box.

[0068] Confidence loss L conf The specific expression is:

[0069]

[0070] Where M is the total number of predicted bounding boxes, c i is the true confidence of the i-th predicted bounding box, is the prediction confidence of the i-th predicted bounding box.

[0071] S6, model optimization: input the validation set into the YOLOv5 model. When the output loss function is greater than 1.5, adjust the YOLOv5 model parameters and go to step S5 to continue model training. If the output loss function is less than 1.5, stop adjusting the model parameters and complete the model training. The parameters of this embodiment are adjusted as follows: in the backbone part of YOLOv5, increase the number of C3 network structure component modules from 9 to 12, set the width_multiple scaling factor to 1.2, and increase the number of channels of the convolutional layer by 20%. Adjust the activation function, replace the default SiLU activation function with the Hardswish activation function, alleviate the gradient vanishing problem while maintaining the sparsity of the network. Use the SGD optimizer to update the parameters of the model.

[0072] S7, Deployment Inference: Use model pruning methods to remove redundant weights to reduce model size while maintaining model accuracy;

[0073] Using the Network Slimming pruning algorithm, the scaling factor γ in the BN (Batch Normalization) layer is used to evaluate the importance of the feature map and prune based on this. The formula is:

[0074] y(k)=γ(k)x(k)+β(k)

[0075] Among them, y(k) is the output of the BN layer, x(k) is the input, and γ(k) and β(k) are training parameters. By introducing L1 regularization to control γ, the importance of each feature is extracted, and then pruning is performed.

[0076] Model evaluation: The test set of step S5 is input into the optimized YOLOv5 model. The loss function gradually converges to 0.03 with the increase of iteration number. The accuracy, recall rate and average accuracy of the model on the test set increase with the increase of iteration number. After about 200 iterations, it gradually shows a stable trend, with the accuracy reaching 87%, the recall rate reaching 92%, and the mAP reaching 94%. Figure 3 shown.

[0077] S8. Inspection implementation: The wind turbine drone inspection takes high-definition images according to steps S1-S3 and enhances the images. The enhanced images are input into the improved YOLOv5 model in step S7, and the image recognition results are output. The results will clearly identify the possible fault points and fault categories on the wind turbine, such as Figure 4 shown.

[0078] Faced with the problem that traditional target detection technology is difficult to accurately identify wind turbine components and their defects, resulting in a large amount of manual participation in the inspection process, the wind farm drone intelligent inspection method based on high-definition image capture and YOLO image recognition coupling uses Mosaic image enhancement and high-resolution reconstruction technology to establish a complete wind turbine damage image database, effectively enhancing the diversity and completeness of the data required for model training. On this basis, this method deeply optimizes and improves the YOLOV5 model, specifically adjusting the number of C3 network structure component modules in the backbone part, the width_multiple scaling factor, and the number of convolutional layer channels, which can flexibly adjust the model depth and balance the complexity and performance of the model. In addition, this method combines the advantages of the pruning model, and can significantly reduce the model parameters and calculation amount while keeping the model recognition performance basically unchanged, improve the real-time performance and deployment flexibility of the inspection system, and realize the accuracy, automation, and intelligence of the inspection process, with obvious technical advantages.

Claims

1. A wind farm drone intelligent inspection method based on high-definition image capture and YOLO image recognition coupling, characterized in that: The following steps are involved: S1. Image acquisition: The drone equipped with a high-definition camera flies to the top of the designated wind turbine according to the preset route, locates and finds the hovering position, and controls the gimbal to shoot the target, and collects data on the wind turbine blades, tower and other external surface parts one by one according to the preset route; after completing the data collection task of a single wind turbine, the drone automatically flies to the next target wind turbine according to the preset route, and performs positioning, hovering and shooting operations repeatedly until the information of all wind turbines in the wind farm is collected; S2. Image preprocessing and expansion: Perform multiple rounds of image enhancement processing on the images collected by the drone. When the comprehensive image quality index IQI exceeds 0.85 and the change of IQI does not exceed 5% after three consecutive image enhancement operations, stop the enhancement and output the enhanced image. The specific expression of IQI is as follows: IQI=S cont +S hist +S diver In the formula, S cont is the contrast suitability of the image after enhancement, S hist is the histogram uniformity, S diver Adding strength to Mosaic diversity; S3, Image super-resolution reconstruction: This reconstruction is mainly based on the image degradation model to convert the original image into a high-resolution image to reduce the impact of noise. Its mathematical formula can be summarized as: Y=(Xn) / H Where Y represents the high-resolution image, X represents the low-resolution image, H represents the degradation function, and n represents the additional noise; S4, image defect annotation: annotate the processed sample set and divide the images into six categories: normal, crack, epidermal peeling, wear, lightning strike and other damage; S5, model training: divide the sample data set marked in step S4 into training set, validation set, and test set according to 6:3:1; input the training set into the YOLOv5 model for model training, and the overall loss function (L sum ) is the sum of classification loss, positioning loss and confidence loss. The specific expression is: L sum =λ cls ·L cls +λ loc ·L loc +λ conf ·L conf Among them, L cls , L loc , L conf They are classification loss, positioning loss and confidence loss, respectively, cls , loc , conf They are the weight factors corresponding to classification loss, positioning loss and confidence loss respectively; S6, model optimization: input the validation set into the YOLOv5 model. When the output loss function is greater than 1.5, adjust the YOLOv5 model parameters, including the number of C3 network structure component modules in the backbone part, the width_multiple scaling factor, and the number of convolutional layer channels, and go to step S5 to continue model training; if the output loss function is less than 1.5, stop adjusting the model parameters and complete the model training; S7, Deployment Inference: Use model pruning methods to remove redundant weights to reduce model size while maintaining model accuracy; S8. Inspection implementation: The wind turbine drone inspection takes high-definition images according to steps S1-S3 and enhances the images. The enhanced images are input into the improved YOLOv5 model in step S7, and the image recognition results are output. The results will clearly identify the possible fault points and fault categories on the wind turbine.

2. The wind farm drone intelligent inspection method based on coupling high-definition image capture and YOLO image recognition according to claim 1 is characterized in that: The specific method of image preprocessing and expansion in step S2 is: Use conventional image enhancement methods such as flipping, brightness, blurring, and scaling to simulate changes in the brightness of the shooting light, blurring to simulate changes in camera focus, and scaling to simulate changes in the distance of the camera; or use the histogram equalization formula to indirectly improve the image contrast: Among them, I out (x, y) is the output image, cdf(I(x, y)) is the cumulative distribution function of the input image, M×N is the total number of image pixels, and cdf min is the minimum cumulative frequency; The Mosaic image enhancement method is used to increase the diversity of sample data, enrich the scenes of the data set, and simulate the situation of the target in different environmental combinations. For the four images I1(x, y), I2(x, y), I3(x, y) and I4(x, y) in the original sample set, where (x, y) represents the pixel coordinates of the image, the newly generated I mosaic The size of (x, y) is usually twice the size of the original single image (2ω×2h), and the specific coordinate expression is: When 0≤x<ω and 0≤y<h, I mosaic (x,y)=I1(x,y); When ω≤x<2ω and 0≤y<h, I mosaic (x,y)=I2(x-ω,y); When 0≤x<ω and h≤y<2h, I mosaic (x,y)=I3(x,yh); When ω≤x<2ω and h≤y<2h, I mosaic (x,y)=I4(x-ω,yh); Where ω and h are the width and height of the original image respectively.

3. The wind farm drone intelligent inspection method based on coupling of high-definition image capture and YOLO image recognition according to claim 1 is characterized in that: The model classification loss L in step S5 cls The specific expression is: Where N is the total number of samples, y i is the true label of the i-th sample, p i is the predicted probability of the i-th sample; Positioning loss L loc The specific expression is: L loc =1-r iou +L r +L center Among them, r iou is the intersection-over-union ratio between the predicted bounding box and the true bounding box, L r is the aspect ratio loss, which is used to measure the difference between the aspect ratio of the predicted bounding box and the true bounding box; L center The center point loss is used to measure the distance between the center point of the predicted bounding box and the true bounding box; Confidence loss L conf The specific expression is: Where M is the total number of predicted bounding boxes, c i is the true confidence of the i-th predicted bounding box, is the prediction confidence of the i-th predicted bounding box.

4. The wind farm drone intelligent inspection method based on coupling high-definition image capture and YOLO image recognition according to claim 1 is characterized in that: The specific steps in step S7 are: Using the Network Slimming pruning algorithm, the scaling factor γ in the BN (Batch Normalization) layer is used to evaluate the importance of the feature map and prune based on this. The formula is: y(k)=γ(k)x(k)+β(k) Among them, y(k) is the output of the BN layer, x(k) is the input, and γ(k) and β(k) are training parameters. By introducing L1 regularization to control γ, the importance of each feature is extracted, and then pruning is performed.

Images