A wind turbine blade fault estimation system and method

Abstract

This invention provides a system and method for estimating wind turbine blade faults. The system acquires each visible light and infrared image, marking the wind turbine blade fault type and fault prediction bounding box in each image. An improved YOLOV5 model network is constructed, and an optimized YOLOV5 network is obtained through architectural improvements. A GoogLeNet network is constructed and optimized through Adam algorithm training to obtain an optimized GoogLeNet network. For each selected visible light image of the wind turbine blade captured by the processing terminal, the location and type of the corresponding fault are predicted using the optimized YOLOV5 network. Similarly, the location and type of the corresponding fault are predicted using the GoogLeNet network for each selected infrared image of the wind turbine blade. Finally, the location and type of the final wind turbine blade fault are determined through dual-spectral comparison and fusion identification. The advantages of this invention are that it reduces the working time of inspection workers in harsh environments; and by improving the traditional YOLOV5 network and GoogLeNet network, it improves image processing speed and accuracy to meet inspection requirements.

Description

Technical Field

[0001] This invention belongs to the field of image data processing technology, and in particular relates to a system and method for estimating the faults of wind turbine blades. Background Technology

[0002] As a critical component in wind turbine power generation, blade failures account for 25% of all failures. The aerodynamic efficiency of the blades directly impacts the power generation efficiency of the wind turbine. Common blade failures include surface material corrosion, cracking, and pinholes. Severe blade damage includes front and rear end cracking, skin peeling, and damage or breakage caused by extreme weather events such as hail and lightning strikes. Skin damage caused by alternating loads should be given high priority, especially early detection, as timely intervention can effectively prevent further deterioration and significantly reduce maintenance costs. Severe blade damage or even breakage can paralyze the entire wind turbine system. Therefore, blade system failure not only incurs high costs but also results in prolonged downtime.

[0003] Currently, the main methods of inspection include visual inspection, manual inspection, and drone inspection. Visual inspection involves directly observing damage to the wind turbine blades using high-powered telescopes. This method can quickly determine the extent of damage within the field of view, but it is highly dependent on the observer's experience and skill level, and is unsuitable for poor lighting conditions or when internal blade faults exist. Manual inspection is currently the most common practice, but it is inefficient, dangerous, and unsuitable for inspecting large-area wind farms. Drone-based visible light inspection uses drones to photograph the wind turbines to identify blade faults. This method is efficient, but it cannot identify internal blade faults.

[0004] Infrared radiation is a type of invisible light with a wavelength range of 0.76–1,000 μm, located outside the red spectrum. Objects with a Kelvin temperature above absolute zero can emit infrared radiation. When a wind turbine blade malfunctions internally or through a point of failure, the blade's originally uniform and orderly structure is disrupted, resulting in different temperature fields under solar radiation or excitation heat sources. Infrared cameras capture thermal image video signals, and by comparing the temperature field of the target blade with corresponding visible light images, this information serves as a basis for blade fault diagnosis. Summary of the Invention

[0005] This invention provides a method for detecting faults in wind turbine blades. By improving the YOLOV5 network architecture and introducing the GoogLeNet model supervision signal, the recognition speed and detection accuracy are significantly improved. The method uses visible light and infrared light fusion technology from UAVs to intelligently inspect the blades. This method can reduce the workload and risk of inspection personnel, and can simultaneously detect surface and internal faults on the blades, quickly locate the fault position, and achieve rapid inspection.

[0006] To address the aforementioned technical problems, this invention provides a system and method for estimating the faults of wind turbine blades.

[0007] The technical solution of the present invention is a wind turbine blade fault estimation system, including: a processing terminal, a color visible light camera, an infrared thermal imaging camera, and a drone;

[0008] The processing terminal, color visible light camera, and infrared thermal imaging camera are all mounted on the drone.

[0009] The processing terminal is connected in sequence to the color visible light camera and the infrared thermal imaging camera, respectively.

[0010] The processing terminal acquires visible light images of the wind turbine blades at multiple moments using the color visible light camera and infrared light images of the wind turbine blades at multiple moments using the infrared thermal imaging camera.

[0011] The processing terminal obtains the actual wind turbine blade fault type, fault prediction box, fault marking box, and actual wind turbine blade fault type for each visible light image and each infrared light image by labeling. An improved YOLOV5 model network is constructed and optimized using the Adam algorithm. A GoogLeNet network is also constructed and optimized using the Adam algorithm. The processing terminal acquires multiple filtered visible light and infrared light images of the wind turbine blades. The optimized YOLOV5 model network predicts the location and type of faults in each filtered visible light image, and the optimized GoogLeNet network predicts the type and location of faults in each filtered infrared light image. Finally, the location and type of faults in both the visible light and infrared light images are fused together using a dual-spectral comparison to identify the final fault location and type of the wind turbine blade.

[0012] The technical solution of this invention is a method for detecting faults in wind turbine blades, specifically including the following steps:

[0013] Step 1: Input multiple raw visible light images into the processing terminal, and obtain each raw visible light image by image removal preprocessing method. Mark the actual wind turbine blade fault type and fault prediction box of each visible light image. Input multiple raw infrared light images, and obtain each raw infrared light image by image quality evaluation preprocessing method. Mark the fault marking box and the actual wind turbine blade fault type of each infrared light image.

[0014] Step 2: Construct an improved YOLOV5 model network. Input each visible light image into the improved YOLOV5 model network for prediction to obtain the predicted wind turbine blade fault type and the fault prediction box for each visible light image. Combine the fault marking box and the actual wind turbine blade fault type of each visible light image to construct the YOLOV5 model network loss function model. Optimize and train the network using the Adam algorithm to obtain the optimized improved YOLOV5 model network.

[0015] Step 3: Construct the GoogLeNet network. Input each infrared image into the GoogLeNet network sequentially to predict the predicted wind turbine blade fault type and the fault prediction box for each infrared image. Combine the fault marking boxes and the actual wind turbine blade fault types for each infrared image to construct the GoogLeNet network loss function model. Optimize and train the network using the Adam algorithm to obtain the optimized GoogLeNet network.

[0016] Step 4: The processing terminal acquires visible light images of the wind turbine blades at multiple times using the color visible light camera and infrared light images of the wind turbine blades at multiple times using the infrared thermal imaging camera; the processing terminal obtains multiple filtered visible light images of the wind turbine blades by performing image removal preprocessing on the visible light images of the wind turbine blades at multiple times, and obtains multiple filtered infrared light images of the wind turbine blades by performing image removal preprocessing on the infrared light images of the wind turbine blades at multiple times.

[0017] Step 5: The processing terminal uses an optimized and improved YOLOV5 model network to predict the location and type of faults in each of the selected visible light images of the wind turbine blades. It also uses an optimized GoogLeNet network to predict the type and location of faults in each of the selected infrared images of the wind turbine blades. Finally, the location and type of faults in the visible light images and the infrared images of the wind turbine blades are identified by dual-spectrum comparison and fusion to obtain the final location and type of faults in the wind turbine blades.

[0018] Preferably, the improved YOLOV5 model network described in step 2 includes:

[0019] The image input module, image data enhancement module, image quality evaluation module, CSPDarknet53 backbone network, improved feature pyramid network, and prediction module are cascaded together.

[0020] The image input module is used to input each visible light image and output it to the image data enhancement module;

[0021] The image data enhancement module is used to process each visible light image using the Mosaic data enhancement method to obtain multiple enhanced images corresponding to each visible light image, and output them to the image quality evaluation module.

[0022] The image quality evaluation module calculates the image sharpness of each enhanced image corresponding to each visible light image using the Brenner gradient function, filters out the enhanced images corresponding to each visible light image whose image sharpness is greater than the sharpness threshold, constructs multiple filtered images corresponding to each visible light image, and outputs the multiple filtered images corresponding to each visible light image to the CSPDarknet53 backbone network.

[0023] The CSPDarknet53 backbone network extracts multiple filtered images corresponding to each visible light image through multiple convolutional features in sequence to obtain image features of multiple filtered images corresponding to each visible light image, and outputs the image features of multiple filtered images corresponding to each visible light image to the improved feature pyramid network.

[0024] The improved feature pyramid network aggregates the image features of multiple filtered images corresponding to each visible light image to obtain the image aggregated features corresponding to each visible light image, and outputs the image aggregated features corresponding to each visible light image to the prediction module.

[0025] The improved feature pyramid network includes: an adaptive attention mechanism module, a first downsampling feature enhancement module, a second downsampling feature enhancement module, ..., a Kth downsampling feature enhancement module, and a path aggregation network;

[0026] The adaptive attention mechanism module, the first downsampling feature enhancement module, the second downsampling feature enhancement module, ..., the Kth downsampling feature enhancement module are cascaded in sequence; the first downsampling feature enhancement module, the second downsampling feature enhancement module, ..., the Kth downsampling feature enhancement module are respectively connected to the path aggregation network;

[0027] The k-th downsampling feature enhancement module is composed of a cascaded k-th downsampling module and a k-th feature enhancement module, where k∈[1,K].

[0028] The adaptive attention mechanism module is used to aggregate the image features of multiple filtered images corresponding to each visible light image to obtain the 0th downsampled enhanced image feature, and output the 0th downsampled enhanced image feature to the 1st downsampled feature enhancement module.

[0029] The first downsampling feature enhancement module obtains the first downsampling enhanced image feature by processing the 0th downsampling enhanced image feature through image downsampling and image feature enhancement, and outputs the first downsampling enhanced image feature to the second downsampling feature enhancement module and the path aggregation network, respectively.

[0030] The second downsampling feature enhancement module obtains the second downsampling enhanced image features from the first downsampling enhanced image features through image downsampling processing and image feature enhancement processing, and outputs the second downsampling enhanced image features to the third downsampling feature enhancement module and the path aggregation network, respectively.

[0031] The k-th downsampling feature enhancement module obtains the k-th downsampling enhanced image feature by processing the (k-1)-th downsampling enhanced image feature through image downsampling and image feature enhancement. The k-th downsampling enhanced image feature is then output to the (k+1)-th downsampling feature enhancement module and the path aggregation network, where k∈[1,K-1].

[0032] The path aggregation network processes the first downsampled enhanced image features, the second downsampled enhanced image features, ..., the Kth downsampled enhanced image features through path aggregation to obtain the image aggregation features corresponding to each visible light image;

[0033] The prediction module predicts the image aggregation features corresponding to each visible light image to obtain the predicted wind turbine blade fault type and the fault prediction box for each visible light image.

[0034] The YOLOv5 model network loss function described in step 2 is as follows:

[0035]

[0036]

[0037] L CIOU =1-IoU+R CIoU

[0038]

[0039] Among them, R CIoU b represents the penalty term in the network loss function of the YOLOv5 model. i b represents the center point of the fault prediction box in the i-th visible light image. i gt ρ(b) represents the center point of the fault marker box in the i-th visible light image. i b i gt ) represents the Euclidean distance between the center points of the fault prediction box and the fault marker box in the i-th visible light image, c represents the diagonal length of the smallest enclosed box between the predicted and ground truth boxes in the visible light image, αv represents the influence factor, v represents the parameter measuring aspect ratio consistency, α represents the weighting function, and w i w represents the width of the fault prediction box in the i-th visible light image. i gt h represents the width of the fault marker box in the i-th visible light image. i h represents the length of the fault prediction bounding box for each visible light image. i gt L represents the length of the fault marker box in each visible light image. CIoU denoted as the network loss function of the YOLOV5 model, IoU represents the degree of overlap between the fault prediction box and the fault marker box of each visible light image, and n represents the number of visible light images;

[0040] Preferably, the GoogLeNet network loss function model constructed in step 3 is defined as follows:

[0041] L = L s +βL c

[0042] Among them, L c For the Center loss function, L s Here, β represents the softmax loss function;

[0043] The Center loss function is specifically defined as follows:

[0044]

[0045] Where m represents the number of infrared images, x k Let cen represent the feature center of the k-th infrared image. k This represents the center of the feature values ​​of the k-th infrared image. The square of the Euclidean norm represents the square of the Euclidean linear distance between two vector matrices in space.

[0046] The softmax loss function is specifically defined as follows:

[0047]

[0048] Where num represents the number of columns in the fully connected layer parameter matrix, i.e., the total number of fault types in the input image, and w l w represents the weight vector in the l-th column of the fully connected layer parameter matrix, which corresponds to the l-th fault category. j Let z be the weight vector of the j-th column of the fully connected layer parameter matrix, which represents the j-th fault category. l a is the feature center of type l fault in the infrared image. l Let a be the bias value of the l-th column of the fully connected layer parameter matrix. j This represents the bias value of the j-th column of the fully connected layer parameter matrix.

[0049] The advantages of this invention are that,

[0050] Compared to traditional manual inspection, its complexity, danger, and labor costs will be significantly reduced, solving the problem of high workload in traditional inspection methods while improving inspection efficiency;

[0051] Optimization of deep learning models YOLOV5 and GoogLeNet improves the efficiency and accuracy of image information processing for wind turbine blade faults.

[0052] The image collection device used in this invention can adjust the optimal shooting distance and angle of the camera according to the current status of the inspection drone and the status of the detected blades, ensuring the clarity of the obtained visible light and infrared images.

[0053] This invention employs a fault diagnosis method based on visible light and infrared image fusion recognition using optimized YOLOV5 and GoogLeNet models via deep learning algorithms. This method can not only detect surface faults in wind turbine blades but also promptly identify the extent of internal faults. Attached Figure Description

[0054] Figure 1: Flowchart of the method according to an embodiment of the present invention;

[0055] Figure 2 The following is a schematic diagram of image fault box marking and training provided in an embodiment of the present invention;

[0056] Figure 3 The following is a flowchart of data augmentation provided in this embodiment of the invention;

[0057] Figure 4 The optimized feature pyramid network structure diagram provided in this embodiment of the invention;

[0058] Figure 5 : A schematic diagram of image feature path aggregation processing provided in an embodiment of the present invention;

[0059] Figure 6 : A schematic diagram of dual-spectral comparison and fusion identification provided in this embodiment of the invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.

[0062] The technical solution of the system in this embodiment of the invention is a wind turbine blade fault estimation system, comprising:

[0063] Processing terminals, color visible light cameras, infrared thermal imaging cameras, and drones;

[0064] The processing terminal, color visible light camera, and infrared thermal imaging camera are all mounted on the drone.

[0065] The processing terminal is connected in sequence to the color visible light camera and the infrared thermal imaging camera, respectively.

[0066] The processing terminal acquires visible light images of the wind turbine blades at multiple moments using the color visible light camera and infrared light images of the wind turbine blades at multiple moments using the infrared thermal imaging camera.

[0067] The processing terminal is an ARM9 chip processing terminal;

[0068] The color visible light camera is model ML-NC66;

[0069] The infrared thermal imaging camera is model SAT-CK350;

[0070] The drone model is DJI Mavic 3;

[0071] The following is combined Figure 1-6 This invention introduces a method for detecting blade faults in wind turbine generators, as detailed below:

[0072] Figure 1 This is a flowchart of the method of the present invention.

[0073] Step 1: The processing terminal inputs multiple raw visible light images. Each raw visible light image is processed using an image removal preprocessing method to obtain a separate visible light image. The actual wind turbine blade fault type and fault prediction bounding box are then labeled for each visible light image. Next, multiple raw infrared light images are input. These are processed using an image quality evaluation preprocessing method to obtain a separate infrared light image. The fault marking box and the actual wind turbine blade fault type are then labeled for each infrared light image. Figure 2 As shown

[0074] Step 2: Construct an improved YOLOV5 model network. Input each visible light image into the improved YOLOV5 model network for prediction to obtain the predicted wind turbine blade fault type and the fault prediction box for each visible light image. Combine the fault marking box and the actual wind turbine blade fault type of each visible light image to construct the YOLOV5 model network loss function model. Optimize and train the network using the Adam algorithm to obtain the optimized improved YOLOV5 model network.

[0075] The improved YOLOV5 model network described in step 2 includes:

[0076] The image input module, image data enhancement module, image quality evaluation module, CSPDarknet53 backbone network, improved feature pyramid network, and prediction module are cascaded together.

[0077] The image input module is used to input each visible light image and output it to the image data enhancement module;

[0078] The image data enhancement module, such as Figure 3It is used to process each visible light image using the Mosaic data augmentation method to obtain multiple augmented images corresponding to each visible light image, and output them to the image quality evaluation module;

[0079] The image quality evaluation module calculates the image sharpness of each enhanced image corresponding to each visible light image using the Brenner gradient function, filters out the enhanced images corresponding to each visible light image whose image sharpness is greater than the sharpness threshold, constructs multiple filtered images corresponding to each visible light image, and outputs the multiple filtered images corresponding to each visible light image to the CSPDarknet53 backbone network.

[0080] The CSPDarknet53 backbone network extracts multiple filtered images corresponding to each visible light image through multiple convolutional features in sequence to obtain image features of multiple filtered images corresponding to each visible light image, and outputs the image features of multiple filtered images corresponding to each visible light image to the improved feature pyramid network.

[0081] The improved feature pyramid network aggregates the image features of multiple filtered images corresponding to each visible light image to obtain the image aggregated features corresponding to each visible light image, and outputs the image aggregated features corresponding to each visible light image to the prediction module.

[0082] The improved feature pyramid network is as follows: Figure 4 The network includes: an adaptive attention mechanism module, a first downsampling feature enhancement module, a second downsampling feature enhancement module, ..., a Kth downsampling feature enhancement module, and a path aggregation network. According to the YOLOv5 model network, K = 3.

[0083] Figure 4 The adaptive attention mechanism module described above is abbreviated as AMM, the feature enhancement module is abbreviated as FEM, and PANet is a bidirectional fusion backbone network used to shorten the path between layers;

[0084] The adaptive attention mechanism module, the first downsampling feature enhancement module, the second downsampling feature enhancement module, ..., the Kth downsampling feature enhancement module are cascaded in sequence; the first downsampling feature enhancement module, the second downsampling feature enhancement module, ..., the Kth downsampling feature enhancement module are respectively connected to the path aggregation network;

[0085] The k-th downsampling feature enhancement module is composed of a cascaded k-th downsampling module and a k-th feature enhancement module, where k∈[1,K].

[0086] The adaptive attention mechanism module is used to aggregate the image features of multiple filtered images corresponding to each visible light image to obtain the 0th downsampled enhanced image feature, and output the 0th downsampled enhanced image feature to the 1st downsampled feature enhancement module.

[0087] The first downsampling feature enhancement module obtains the first downsampling enhanced image feature by processing the 0th downsampling enhanced image feature through image downsampling and image feature enhancement, and outputs the first downsampling enhanced image feature to the second downsampling feature enhancement module and the path aggregation network, respectively.

[0088] The second downsampling feature enhancement module obtains the second downsampling enhanced image features from the first downsampling enhanced image features through image downsampling processing and image feature enhancement processing, and outputs the second downsampling enhanced image features to the third downsampling feature enhancement module and the path aggregation network, respectively.

[0089] The k-th downsampling feature enhancement module obtains the k-th downsampling enhanced image feature by processing the (k-1)-th downsampling enhanced image feature through image downsampling and image feature enhancement. The k-th downsampling enhanced image feature is then output to the (k+1)-th downsampling feature enhancement module and the path aggregation network, where k∈[1,K-1].

[0090] The path aggregation network processes the first downsampled enhanced image features, the second downsampled enhanced image features, ..., the Kth downsampled enhanced image features through path aggregation to obtain the image aggregation features corresponding to each visible light image, such as... Figure 5 As shown;

[0091] The prediction module predicts the image aggregation features corresponding to each visible light image to obtain the predicted wind turbine blade fault type and the fault prediction box for each visible light image.

[0092] The YOLOv5 model network loss function described in step 2 is as follows:

[0093]

[0094]

[0095] L CIOU =1-IoU+R CIoU

[0096]

[0097] Among them, R CIoU b represents the penalty term in the network loss function of the YOLOv5 model. ib represents the center point of the fault prediction box in the i-th visible light image. i gt ρ(b) represents the center point of the fault marker box in the i-th visible light image. i， b i gt ) represents the Euclidean distance between the center points of the fault prediction box and the fault marker box in the i-th visible light image, c represents the diagonal length of the smallest enclosed box between the predicted and ground truth boxes in the visible light image, αv represents the influence factor, v represents the parameter measuring aspect ratio consistency, α represents the weighting function, and w i w represents the width of the fault prediction box in the i-th visible light image. i gt h represents the width of the fault marker box in the i-th visible light image. i h represents the length of the fault prediction bounding box for each visible light image. i gt L represents the length of the fault marker box in each visible light image. CIoU denoted as the network loss function of the YOLOV5 model, IoU represents the degree of overlap between the fault prediction box and the fault marker box of each visible light image, and n=52 represents the number of visible light images. This value is determined empirically, and obviously, it is only one of the more than 50 possible values ​​of n.

[0098] Step 3: Construct the GoogLeNet network. Input each infrared image into the GoogLeNet network sequentially to predict the predicted wind turbine blade fault type and the fault prediction box for each infrared image. Combine the fault marking boxes and the actual wind turbine blade fault types for each infrared image to construct the GoogLeNet network loss function model. Optimize and train the network using the Adam algorithm to obtain the optimized GoogLeNet network.

[0099] Step 3 describes the construction of the GoogLeNet network loss function model, defined as follows:

[0100] L = L s +βL c

[0101] Among them, L c For the Center loss function, L s Here, β represents the softmax loss function;

[0102] The Center loss function is specifically defined as follows:

[0103]

[0104] Where m = 52 represents the number of infrared images, x k Let cen represent the feature center of the k-th infrared image. k This represents the center of the feature values ​​of the k-th infrared image. The square of the Euclidean norm represents the square of the Euclidean linear distance between two vector matrices in space. This value is determined empirically, and it is clearly only one of the more than 50 possible values ​​for m.

[0105] The softmax loss function is specifically defined as follows:

[0106]

[0107] Where num represents the number of columns in the fully connected layer parameter matrix, i.e., the total number of fault types in the input image, and w l w represents the weight vector in the l-th column of the fully connected layer parameter matrix, which corresponds to the l-th fault category. j Let z be the weight vector of the j-th column of the fully connected layer parameter matrix, which represents the j-th fault category. l a is the feature center of type l fault in the infrared image. l Let a be the bias value of the l-th column of the fully connected layer parameter matrix. j This represents the bias value of the j-th column of the fully connected layer parameter matrix;

[0108] Step 4: The processing terminal acquires visible light images of the wind turbine blades at multiple times using the color visible light camera and infrared light images of the wind turbine blades at multiple times using the infrared thermal imaging camera; the processing terminal obtains multiple filtered visible light images of the wind turbine blades by performing image removal preprocessing on the visible light images of the wind turbine blades at multiple times, and obtains multiple filtered infrared light images of the wind turbine blades by performing image removal preprocessing on the infrared light images of the wind turbine blades at multiple times.

[0109] Step 5: The processing terminal uses an optimized and improved YOLOV5 model network to predict the location and type of faults in each selected visible light image of the wind turbine blade. It then uses an optimized GoogLeNet network to predict the type and location of faults in each selected infrared image of the wind turbine blade. Finally, the location and type of faults in both the visible light and infrared images of the selected wind turbine blades are identified through dual-spectrum comparison and fusion. Figure 6 The location and type of the final wind turbine blade fault are obtained.

[0110] Specifically, fault diagnosis is performed on abnormal images using a dual-spectral fault analysis mode based on visible and infrared light; when analyzing abnormal image groups output by the YOLO network, such as... Figure 6 When a fault box is identified in the visible light image, a fault is considered to exist if the confidence level is higher than 0.72. If the infrared image does not show an abnormal temperature area or the abnormal temperature area only exists at the boundary of the identified fault box, the fault is considered to be a surface fault of the blade. If the abnormal temperature area shown in the infrared image exceeds the boundary of the fault box, the fault is considered to have caused damage to the internal structure of the blade, which is a through-type fault. When a fault box is identified in the visible light image, a fault is considered to be possible if the confidence level is lower than 0.72. If the fault box shown in the infrared image contains an abnormal temperature area, the fault is considered to exist. If the infrared image shows a normal temperature range, the suspected fault image is uploaded to manual inspection for secondary verification. When no fault box is identified in the visible light image, but the infrared image shows an abnormal temperature range in this image group, the internal structure of the blade is considered to have been damaged, causing the fault.

[0111] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0112] Although this document uses terms such as processing terminal, color visible light camera, infrared thermal imaging camera, and drone frequently, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing the essence of the invention, and interpreting them as any additional limitation would contradict the spirit of the invention.

[0113] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A wind turbine blade fault estimation system, characterized in that, include: Processing terminals, color visible light cameras, infrared thermal imaging cameras, and drones; The processing terminal, color visible light camera, and infrared thermal imaging camera are all mounted on the drone. The processing terminal is connected in sequence to the color visible light camera and the infrared thermal imaging camera, respectively. The processing terminal acquires visible light images of the wind turbine blades at multiple moments using the color visible light camera and infrared light images of the wind turbine blades at multiple moments using the infrared thermal imaging camera. The processing terminal obtains the actual wind turbine blade fault type, fault prediction box, fault marking box, and actual wind turbine blade fault type for each visible light image and each infrared light image by labeling. An improved YOLOV5 model network is constructed and optimized using the Adam algorithm. A GoogLeNet network is also constructed and optimized using the Adam algorithm. The processing terminal acquires multiple filtered visible light and infrared light images of the wind turbine blades. The optimized YOLOV5 model network predicts the location and type of faults in each filtered visible light image, and the optimized GoogLeNet network predicts the type and location of faults in each filtered infrared light image. Finally, the location and type of faults in both the visible light and infrared light images are fused together using a dual-spectral comparison to identify the final fault location and type of the wind turbine blade.

2. A method for estimating blade faults in a wind turbine generator set using the wind turbine generator set blade fault estimation system described in claim 1, characterized in that, Includes the following steps: Step 1: Input multiple raw visible light images into the processing terminal, and obtain each raw visible light image by image removal preprocessing method. Mark the actual wind turbine blade fault type and fault prediction box of each visible light image. Input multiple raw infrared light images, and obtain each raw infrared light image by image quality evaluation preprocessing method. Mark the fault marking box and actual wind turbine blade fault type of each infrared light image. Step 2: Construct an improved YOLOV5 model network. Input each visible light image into the improved YOLOV5 model network for prediction to obtain the predicted wind turbine blade fault type and fault prediction box for each visible light image. Combine the fault marking box of each visible light image and the actual wind turbine blade fault type to construct the YOLOV5 model network loss function model. Optimize and train the network using the Adam algorithm to obtain the optimized improved YOLOV5 model network. Step 3: Construct the GoogLeNet network. Input each infrared image into the GoogLeNet network for prediction to obtain the predicted wind turbine blade fault type and fault prediction box for each infrared image. Combine the fault marking boxes of each infrared image and the actual wind turbine blade fault type to construct the GoogLeNet network loss function model. Optimize the training using the Adam algorithm to obtain the optimized GoogLeNet network. Step 4: The processing terminal acquires visible light images of the wind turbine blades at multiple times using the color visible light camera, and infrared light images of the wind turbine blades at multiple times using the infrared thermal imaging camera. The visible light images of the wind turbine blades at multiple times are processed by an image removal preprocessing method to obtain multiple filtered visible light images of the wind turbine blades. The infrared light images of the wind turbine blades at multiple times are processed by an image removal preprocessing method to obtain multiple filtered infrared light images of the wind turbine blades. Step 5: The processing terminal uses an optimized and improved YOLOV5 model network to predict the location and type of faults in each of the selected wind turbine blade visible light images. It also uses an optimized GoogLeNet network to predict the location and type of faults in each of the selected wind turbine blade infrared light images. Finally, the location and type of faults in the visible light images and infrared light images of each selected wind turbine blade are identified by dual-spectrum comparison and fusion to obtain the final location and type of faults in the wind turbine blade.

3. The method for estimating blade faults in wind turbine generators according to claim 2, characterized in that: The improved YOLOV5 model network described in step 2 is composed of an image input module, an image data augmentation module, an image quality evaluation module, a CSPDarknet53 backbone network, an improved feature pyramid network, and a prediction module, which are cascaded in sequence. The image input module is used to input each visible light image and output it to the image data enhancement module; The image data enhancement module is used to process each visible light image using the Mosaic data enhancement method to obtain multiple enhanced images corresponding to each visible light image, and output them to the image quality evaluation module. The image quality evaluation module calculates the image sharpness of each enhanced image corresponding to each visible light image using the Brenner gradient function, filters out the enhanced images corresponding to each visible light image whose image sharpness is greater than the sharpness threshold, constructs multiple filtered images corresponding to each visible light image, and outputs the multiple filtered images corresponding to each visible light image to the CSPDarknet53 backbone network. The CSPDarknet53 backbone network extracts multiple filtered images corresponding to each visible light image through multiple convolutional features in sequence to obtain image features of multiple filtered images corresponding to each visible light image, and outputs the image features of multiple filtered images corresponding to each visible light image to the improved feature pyramid network. The improved feature pyramid network aggregates the image features of multiple filtered images corresponding to each visible light image to obtain the image aggregated features corresponding to each visible light image, and outputs the image aggregated features corresponding to each visible light image to the prediction module. The prediction module predicts the image aggregation features corresponding to each visible light image to obtain the predicted wind turbine blade fault type and the fault prediction box for each visible light image.

4. The method for estimating blade faults in wind turbine generators according to claim 3, characterized in that: The improved feature pyramid network includes: an adaptive attention mechanism module, a first downsampling feature enhancement module, a second downsampling feature enhancement module, ..., a Kth downsampling feature enhancement module, and a path aggregation network; The adaptive attention mechanism module, the first downsampling feature enhancement module, the second downsampling feature enhancement module, ..., the Kth downsampling feature enhancement module are cascaded in sequence; the first downsampling feature enhancement module, the second downsampling feature enhancement module, ..., the Kth downsampling feature enhancement module are respectively connected to the path aggregation network.

5. The method for estimating blade faults in wind turbine generators according to claim 4, characterized in that: The k-th downsampling feature enhancement module is composed of a cascaded k-th downsampling module and a k-th feature enhancement module, where k∈[1,K]. The adaptive attention mechanism module is used to aggregate the image features of multiple filtered images corresponding to each visible light image to obtain the 0th downsampled enhanced image feature, and output the 0th downsampled enhanced image feature to the 1st downsampled feature enhancement module. The first downsampling feature enhancement module obtains the first downsampling enhanced image feature by processing the 0th downsampling enhanced image feature through image downsampling and image feature enhancement, and outputs the first downsampling enhanced image feature to the second downsampling feature enhancement module and the path aggregation network, respectively. The second downsampling feature enhancement module obtains the second downsampling enhanced image features from the first downsampling enhanced image features through image downsampling processing and image feature enhancement processing, and outputs the second downsampling enhanced image features to the third downsampling feature enhancement module and the path aggregation network, respectively. The k-th downsampling feature enhancement module obtains the k-th downsampling enhanced image feature by processing the (k-1)-th downsampling enhanced image feature through image downsampling and image feature enhancement. The k-th downsampling enhanced image feature is then output to the (k+1)-th downsampling feature enhancement module and the path aggregation network, where k∈[1,K-1]. The path aggregation network processes the first downsampled enhanced image features, the second downsampled enhanced image features, ..., the Kth downsampled enhanced image features through path aggregation to obtain the image aggregation features corresponding to each visible light image.

6. The method for estimating blade faults in wind turbine generators according to claim 5, characterized in that: The YOLOv5 model network loss function described in step 2 is as follows: L CIOU =1-IoU+R CIoU Among them, R CIoU b represents the penalty term in the loss function of the YOLOv5 model network. i b represents the center point of the fault prediction box in the i-th visible light image. i gt ρ(b) represents the center point of the fault marker box in the i-th visible light image. i b i gt ) represents the Euclidean distance between the center points of the fault prediction box and the fault marker box in the i-th visible light image, c represents the diagonal length of the smallest enclosed box between the predicted and ground truth boxes in the visible light image, αv represents the influence factor, v represents the parameter measuring aspect ratio consistency, α represents the weighting function, and w i w represents the width of the fault prediction box in the i-th visible light image. i gt h represents the width of the fault marker box in the i-th visible light image. i h represents the length of the fault prediction bounding box for each visible light image. i gt L represents the length of the fault marker box in each visible light image. CIoU denoted as the network loss function of the YOLOV5 model, IoU represents the degree of overlap between the fault prediction box and the fault marker box of each visible light image, and n represents the number of visible light images.

7. The method for estimating blade faults in wind turbine generators according to claim 2, characterized in that: Step 3 describes the construction of the GoogLeNet network loss function model, defined as follows: L=L s +βL c Among them, L c For the Centerloss function, L s Here, β represents the softmax loss function; The Centerloss function is specifically defined as follows: Where m represents the number of infrared images, x k Let cen represent the feature center of the k-th infrared image. k This represents the center of the feature values ​​of the k-th infrared image. The square of the Euclidean norm represents the square of the Euclidean linear distance between two vector matrices in space. The softmax loss function is specifically defined as follows: Where num represents the number of columns in the fully connected layer parameter matrix, i.e., the total number of fault types in the input image, and w l w represents the weight vector in the l-th column of the fully connected layer parameter matrix, which corresponds to the l-th fault category. j Let z be the weight vector of the j-th column of the fully connected layer parameter matrix, which represents the j-th fault category. l a is the feature center of type l fault in the infrared image. l Let a be the bias value of the l-th column of the fully connected layer parameter matrix. j This represents the bias value of the j-th column of the fully connected layer parameter matrix.

Images