Multi-mode defect detection system for key equipment of wind power plant based on unmanned aerial vehicle

Abstract

The invention provides a multi-mode defect detection system for key equipment of a wind power plant based on an unmanned aerial vehicle. The multi-modal defect detection system for the key equipment of the wind power plant comprises an unmanned aerial vehicle, a control end for controlling the flight of the unmanned aerial vehicle, a model module for performing data learning on the structural characteristics of the equipment of the wind power plant to construct a defect identification model, and a route planning module for planning the cruise route of the unmanned aerial vehicle. According to the method, by fusing multi-modal feature detection, intelligent path planning and defect evaluation of the deep learning model, the defect detection efficiency, precision and comprehensiveness of the key equipment of the wind power plant are remarkably improved, and efficient support is provided for operation and maintenance of a wind turbine.

Description

Technical Field

[0001] The present invention relates to the technical field of wind farm monitoring, and in particular to a multi-modal defect detection system for key equipment in a wind farm based on an unmanned aerial vehicle. Background Art

[0002] With the development of drone technology, drone inspections have been widely used in power transmission, substations and other fields, forming a human-machine collaborative inspection model with drone inspections and technical defense as the main focus, and manual inspections as the auxiliary. However, new energy stations face problems such as a shortage of pilots, low automation, unstable inspection image quality, slow image transmission, and low data utilization in operation and maintenance, which makes it difficult to ensure the efficiency and quality of operation and inspection. At the same time, the data assets of new energy companies have not been fully transformed into productivity, and the application of artificial intelligence technology is relatively lagging behind. To this end, the application of new technologies such as centralized monitoring, autonomous drone inspections, and intelligent image processing is becoming more and more urgent to improve the operation and maintenance efficiency and quality of new energy stations.

[0003] This experimental team has been browsing and researching a large number of related records and materials on the related technologies of drone technology for a long time. At the same time, it relies on relevant resources and conducts a large number of related experiments. After a lot of searches, it is found that there are existing technologies such as CN118363932B, CN117891270B, CN115272037B, and CN114782805B disclosed in the prior art. For example, a smart city regional public security management early warning method and system based on the Internet of Things disclosed in the prior art includes: obtaining user query instructions for each area based on the user platform, and sending the query instructions to the public security management platform via the service platform; responding to Query instructions, based on the sensor network sub-platform, obtain a monitoring image of at least one target area from at least one monitoring device in at least one target area; based on the sensor network sub-platform, send the corresponding monitoring image of the target area to the management sub-platform; based on the management sub-platform, process the monitoring image to determine the risk index of at least one target area; in response to the risk index of the target area being greater than a first threshold, generate early warning information, and send the early warning information to the user platform via the general database of the public security management platform and the service platform; and obtain management instructions based on the user platform, and control the drone to go to the target area to monitor the target area according to the management instructions.

[0004] The present invention is made to solve the common problems in the field such as low inspection automation, unstable image quality, low transmission efficiency, low data utilization, etc. Summary of the invention

[0005] The purpose of the present invention is to propose a multimodal defect detection system for key equipment in a wind farm based on an unmanned aerial vehicle (UAV) in response to the shortcomings in the current field.

[0006] In order to overcome the shortcomings of the prior art, the present invention adopts the following technical solutions: A multimodal defect detection system for key wind farm equipment based on an unmanned aerial vehicle (UAV), the multimodal defect detection system for key wind farm equipment comprising an unmanned aerial vehicle (UAV), a control terminal for controlling the flight of the unmanned aerial vehicle (UAV), a model module for performing data learning on the structural characteristics of the wind farm equipment to construct a defect recognition model, and a route planning module for planning the cruising route of the unmanned aerial vehicle.

[0007] Furthermore, the drone is an intelligent drone capable of image capture, and the camera of the drone is a dual-spectrum camera that can generate infrared images and visible light images simultaneously.

[0008] Furthermore, the control end includes a terminal device and a control application installed in the terminal device, and the position and status of the identified key equipment of the wind farm are displayed in real time on the display screen of the terminal device, so that the operator can monitor the progress of the inspection in real time. The control application can automatically adjust the flight route of the drone to ensure that the inspection covers all key equipment areas of the wind farm, improving the comprehensiveness and efficiency of the inspection. The control application can automatically adjust the drone's attitude and camera parameters, including focal length, exposure and gimbal angle, to ensure clear images and avoid overexposure and out-of-focus issues that affect recognition accuracy.

[0009] Furthermore, the route planning module realizes the generation of a path for the UAV to cruise in the wind farm according to the existing path planning algorithm, real-time obstacle avoidance technology and / or efficient coverage path algorithm.

[0010] Furthermore, the model module includes a feature extraction unit for analyzing and processing the features of the images or videos taken by the drone, a defect assessment model obtained based on historical data training for assessing different defect conditions of different parts of the drone, and an identification unit for inputting the features extracted by the feature extraction unit into the defect assessment module to further obtain the defect type of the target component of the wind turbine, wherein the features include color, texture, shape and thermal features. The target components of the wind turbine include at least the entire tower, tower head, tower body, tower base, ancillary facilities, cross arm side hanging points, conductor side hanging points, insulators and substation portal frame.

[0011] Further, the drone transmits the collected images to the terminal device at the control end through a wireless module for the operator to view in real time. The control application program generates flight and shooting adjustment instructions in real time and conveys them to the drone flight control module. The image data captured by the drone after execution is fed back to the control end. The control end uploads the picture data captured by the drone to the model module through a wireless network. After obtaining the picture data from the control end, the model module generates defect information of the wind turbine and transmits it back to the control application program for subsequent tasks of the control end and the drone. Among them, the model module is implemented by being established in the cloud, and the drone communicates with the model module through an APN card. To ensure data security, the data transmission between the drone, the control application program, and the model module all uses asymmetric encoding and decoding encryption technology.

[0012] Further, the recognition unit implements the following steps: S301: Input X = [CT i , TE i , ST i , Hem i , i] into the defect assessment model, and receive the probabilities P of six defect categories output by the defect assessment model: P = MOD(X), P = {P1, P2, P3, P4, P5, P6}, where i is the number of the target component type, i = {1, 2, 3... N}, and CT i is the color feature of the visible light image of the i-th component, TE i is the texture feature parameter of the visible light image of the i-th component, ST i is the shape feature parameter of the visible light image of the i-th component, Hem i is the thermal feature value of the infrared image of the i-th component, The defect types of deformation, crack, defect, breakage, corrosion, and overheating are represented by serial numbers 1, 2, 3, 4, 5, and 6 respectively, and the probabilities corresponding to deformation, crack, defect, breakage, corrosion, and overheating are represented as P1, P2, P3, P4, P5, and P6 respectively. S302: Use Pu1, Pu2, Pu3, Pu4, Pu5, and Pu6 to represent the upper limit probabilities corresponding to deformation, crack, defect, breakage, corrosion, and overheating respectively, and compare the probabilities of the six defect categories output by the improved model. S303: Compare the output probability Pc and the upper limit probability Puc of each defect category in turn, and c represents the serial number of each defect type respectively, c ∈ {1, 2, 3, 4, 5, 6}. For each serial number category, if it satisfies: When there is Pc ≥ Puc, it is judged that the key part corresponding to X has the defect type corresponding to the serial number c. When all Pc < Puc, it is judged that the target components of the wind turbine are all intact.

[0013] The beneficial effects achieved by the present invention are: 1. The present invention provides a new technical solution based on centralized monitoring, autonomous drone inspection and intelligent image processing, which can improve the operation and maintenance efficiency and quality of new energy stations, realize the effective use of data assets, and promote the in-depth application of artificial intelligence in the field of new energy.

[0014] 2. The present invention realizes the synchronous acquisition of infrared and visible light images through a dual-spectrum camera, combined with optimized route planning and intelligent posture adjustment, to ensure that the inspection covers the entire field equipment and the shooting is clear and reliable. The defect assessment model trained based on historical data can accurately identify defect types such as cracks, damage, corrosion, etc., and provide real-time feedback to the control end for subsequent tasks. The cloud model module and asymmetric encryption technology further enhance data processing capabilities and security. The overall design reduces labor costs, improves inspection efficiency, and provides an intelligent solution for wind farm equipment operation and maintenance.

[0015] 3. The present invention comprehensively reflects the status of wind power equipment through multimodal feature extraction and analysis, and significantly improves the accuracy of defect detection by combining the fusion calculation of color, texture, shape and thermal features. With the help of image processing technology and target detection algorithm, key components can be accurately located and features can be efficiently extracted. At the same time, historical data is used to drive the optimization of model parameters to ensure the reliability and applicability of the detection results, providing strong technical support for the diagnosis of wind power equipment defects.

[0016] 4. The present invention realizes efficient and accurate defect detection of key components of wind turbines by integrating multimodal features, adopting deep learning modeling and a probability-based defect judgment mechanism. It has the advantages of clear logic, strong adaptability and good scalability, and can provide strong technical support for equipment operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention can be further understood from the following description in conjunction with the accompanying drawings. The components in the figures are not necessarily drawn to scale, but the emphasis is placed on illustrating the principles of the embodiments. In different views, the same reference numerals designate corresponding parts.

[0018] Figure 1 It is a modular schematic diagram of the multimodal defect detection system for key equipment in a wind farm based on an unmanned aerial vehicle of the present invention.

[0019] Figure 2 Schematic diagram of experimental distribution of defect output probability and upper limit probability of one of the target components of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is further described in detail in combination with its embodiments below; it should be pointed out that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present case. For those skilled in the art, after reviewing the following detailed description, other systems, methods and / or features of this embodiment will become apparent. And the terms describing the positional relationship in the accompanying drawings are only used for illustrative purposes and cannot be understood as limitations on this patent. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0021] Embodiment 1: Combining with Figure 1 and attached Figure 2 This embodiment constructs a multimodal defect detection system for key wind farm equipment based on drones. The multimodal defect detection system for key wind farm equipment includes a drone, a control terminal for controlling the flight of the drone, a model module responsible for data learning of structural characteristics of wind farm equipment to build a defect recognition model, and a route planning module for planning the cruising route of the drone.

[0022] The drone is an intelligent drone with image shooting function in the prior art, and the camera of the drone is a dual-spectrum camera, which can generate infrared images and visible light images at the same time.

[0023] The control end includes a terminal device and a control application installed in the terminal device, and displays the position and status of the identified key equipment of the wind farm in real time on the display screen of the terminal device, so that the operator can monitor the progress of the inspection in real time.

[0024] The flight route of the drone can be automatically adjusted through the control application to ensure that the inspection covers all key equipment areas of the wind farm, improving the comprehensiveness and efficiency of the inspection.

[0025] The control application can automatically adjust the drone's attitude and camera parameters, including focal length, exposure and gimbal angle, to ensure clear images and avoid overexposure and out-of-focus issues that affect recognition accuracy.

[0026] The route planning module generates a path for the UAV to cruise in the wind farm based on the existing path planning algorithm, real-time obstacle avoidance technology and / or efficient coverage path algorithm.

[0027] Among them, the model module includes a feature extraction unit for analyzing and processing the features of images or videos taken by the drone, a defect assessment model obtained based on historical data training for evaluating different defect conditions in different parts of the drone, and an identification unit for inputting the features extracted by the feature extraction unit into the defect assessment module to further obtain the defect type of the target component of the wind turbine, wherein the features include color, texture, shape and thermal characteristics.

[0028] The UAV transmits the collected images to the terminal device at the control end through the wireless module for the operator to view in real time. The control application generates flight and shooting adjustment instructions in real time and transmits them to the UAV flight control module. The image data taken by the UAV after execution is fed back to the control end. The control end uploads the picture data taken by the UAV to the model module through the wireless network. After obtaining the picture data from the control end, the model module generates the defect information of the wind turbine and transmits it back to the control application for use by the control end and the UAV in subsequent tasks. Among them, the model module is implemented by establishing it in the cloud, and the UAV and the model module communicate through the APN card. To ensure data security, the data transmission between the UAV, the control application and the model module adopts asymmetric encoding and decoding encryption technology.

[0029] The target components of the wind turbine include at least the entire tower, tower head, tower body, tower base, ancillary facilities, cross arm side hanging points, conductor side hanging points, insulators and substation portal frame.

[0030] The present invention realizes the synchronous acquisition of infrared and visible light images through a dual-spectrum camera, combined with optimized route planning and intelligent posture adjustment, to ensure that the inspection covers all equipment in the field and the shooting is clear and reliable. The defect assessment model trained based on historical data can accurately identify defect types such as cracks, damage, corrosion, etc., and provide real-time feedback to the control end for subsequent tasks. The cloud model module and asymmetric encryption technology further enhance data processing capabilities and security. The overall design reduces labor costs, improves inspection efficiency, and provides an intelligent solution for the operation and maintenance of wind farm equipment.

[0031] Embodiment 2: Combining with Figure 1 and attached Figure 2 In addition to the contents of the above embodiments, the feature extraction unit implements the following operation steps: S101: Receive video or image data taken by a drone, use OpenCV to segment the video into static images frame by frame, detect blurred images in the captured images and static images by Laplace transform, and use inter-frame difference method to remove redundant frames, retain clear and effective static images, and retain the remaining static images and captured images as target images; S102: using the static image and the captured image as the target image, applying a Gaussian smoothing filter to remove high-frequency noise in the target image, reducing background interference with wind turbine monitoring, performing a median filter to remove noise from the target image, while retaining the outline of the wind turbine in the target image, using histogram equalization to enhance the contrast of each frame of the target image, highlighting the details of the wind turbine in the target image, and using a Laplace filter to sharpen each frame of the target image to highlight the edge of the wind turbine in the target image; S103: Obtaining N frames of processed target images, where the target images include visible light images and infrared images; S104: using YOLO and / or FasterR-CNN target detection algorithms to automatically locate the positions of the components of the entire tower, tower head, tower body and tower base of the wind turbine in the target image, and generate identification areas of various components, wherein the pixel segmentation algorithm is used to accurately segment the auxiliary facilities, cross arm side hanging points, conductor side hanging points and insulators to extract the component areas; S105: extract and number each component graphic detected in the target image, and represent the target components of the whole tower, tower head, tower body, tower base, auxiliary facilities, cross arm side hanging points, conductor side hanging points, insulators and substation portal frame in sequence as 1, 2, 3...9 according to the numbers, and represent the component graphics of the visible light in the target image as P1, P2, P3...P9 according to the component types of the whole tower, tower head, tower body, tower base, auxiliary facilities, cross arm side hanging points, conductor side hanging points, insulators and substation portal frame, and represent the component graphics of the infrared light as Pr1, Pr2, Pr3...Pr9 according to the component types of the whole tower, tower head, tower body, tower base, auxiliary facilities, cross arm side hanging points, conductor side hanging points, insulators and substation portal frame, and represent Pi as the visible light image of the i-th component, Pri as the infrared image of the i-th component, i={1,2,3...N}; S106: Calculate the color feature CT of each component graphic i : , CT i is the color feature of the visible light image of the ith component, Ni is the total number of pixels in the visible light image of the ith component, ni is the nth pixel in the visible light image of the ith component, H ni is the hue value of the nth pixel of the visible light image of the ith component, S ni is the saturation value of the nth pixel of the visible light image of the ith component, V niis the brightness value of the nth pixel of the visible light image of the ith component, hi is the mean hue value of all pixels of the visible light image of the ith component, si is the mean saturation value of all pixels of the visible light image of the ith component, and vi is the mean brightness value of all pixels of the visible light image of the ith component; S107: Calculate the texture feature parameters TE of each component graphic i : , Among them, TE i is the texture feature parameter of the visible light image of the i-th component, Con i 、Ener i 、Ho i and Ent i are the contrast, energy, uniformity and entropy of the visible light image of the i-th component obtained by the GLCM statistical method, respectively. Among them, α is the energy suppression coefficient, which is used to control the suppression strength of energy Eneri on texture complexity, and 0.1≤α≤10, β is the energy suppression growth index, β is used to control the nonlinear suppression effect of the energy term, and 0.5≤β≤3, γ is the uniformity suppression coefficient, γ is used to control the uniformity Ho i The suppression strength of texture complexity, and 0.1≤γ≤10, λ is the uniformity suppression growth index, and λ is used to control the uniformity Ho i The nonlinear suppression effect of , and 0.5≤λ≤3; The purpose of introducing parameters α, β, γ and λ is to flexibly control the influence of different texture features on image complexity, so that the formula can more accurately reflect the true characteristics of texture. These parameters can not only balance the contribution of contrast, energy, uniformity and entropy, but also enhance the adaptability of the formula to smooth or complex texture images through nonlinear suppression, thereby improving the accuracy and applicability of texture complexity measurement; S108: Calculate the shape characteristic parameters ST of each component graphic i : , ST i is the shape feature parameter of the visible light image of the i-th component, Per i is the total length of the visible light image boundary of the ith component, SAr i is the total number of pixels of the visible light image of the i-th component, Hu k is the kth Hu invariant moment, there are 7 Hu invariant moments in total, w k is the weight coefficient used to control the kth Hu invariant moment term in the shape feature calculation, and ; S109: , Hemi is the thermal characteristic value of the infrared image of the i-th component, ji is the total number of pixels in the horizontal direction of the infrared image of the i-th component, ki is the total number of pixels in the vertical direction of the infrared image of the i-th component, and Tmax is i is the maximum temperature value corresponding to the pixel of the infrared image of the i-th component, Tavg i is the average temperature value of the infrared image of the i-th component, T i (x, y) is the temperature value corresponding to the pixel with coordinate position (x, y) in the infrared image of the i-th component, T i (x+1, y+1) is the temperature value corresponding to the pixel with coordinate position (x+1, y+1) in the infrared image of the i-th component, μ is the temperature adjustment coefficient used to adjust the influence of the temperature distribution gradient on the temperature characteristic value, and c1 is the priority parameter of μ; The initial values ​​of the temperature control coefficient μ and the priority parameter c1 are set according to historical experience and infrared monitoring data of a large number of wind power equipment. They are used to reflect the thermal distribution of components and the influence of temperature gradient on thermal characteristic values. By further analyzing the correlation between thermal characteristic values ​​and actual defect categories and the deviation between predicted results and actual detection results, the values ​​of μ and c1 are dynamically adjusted so that the thermal characteristic values ​​can more accurately reflect the thermal distribution characteristics of components. Finally, the optimal μ and c1 values ​​are determined through verification experiments to ensure the reliability and accuracy of thermal characteristic values ​​for defect detection.

[0032] The running code of the feature extraction unit is as follows: import cv2 import numpy as np from skimage.feature import greycomatrix, greycoprops # Parameters ALPHA, BETA, GAMMA, LAMBDA = 0.5, 2, 0.5, 2 HU_WEIGHTS = [0.1, 0.2, 0.15, 0.15, 0.1, 0.15, 0.15] MU, C1 = 1.0, 1.0 def preprocess_video(video_path): cap = cv2.VideoCapture(video_path) frames = [] while cap.isOpened(): ret, frame = cap.read() if not ret: break gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) laplacian_var = cv2.Laplacian(gray, cv2.CV_64F).var() if laplacian_var>50: frames.append(frame) cap.release() return frames def enhance_image(image): smoothed = cv2.GaussianBlur(image, (5, 5), 0) denoised = cv2.medianBlur(smoothed, 5) yuv = cv2.cvtColor(denoised, cv2.COLOR_BGR2YUV) yuv[:, :, 0] = cv2.equalizeHist(yuv[:, :, 0]) enhanced = cv2.cvtColor(yuv, cv2.COLOR_YUV2BGR) kernel = np.array([[0, -1, 0], [-1, 5, -1], [0, -1, 0]]) sharpened = cv2.filter2D(enhanced, -1, kernel) return sharpened def compute_color_features(image): hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV) h, s, v = cv2.split(hsv) h_mean, s_mean, v_mean = np.mean(h), np.mean(s), np.mean(v) ni = h.size ct = np.sum(np.sqrt((h - h_mean)**2 + (s - s_mean)**2 + (v - v_mean)**2)) return ct def compute_texture_features(gray_image): glcm = greycomatrix(gray_image, [1], [0], 256, symmetric=True, normed=True) con = greycoprops(glcm, 'contrast')[0, 0] ener = greycoprops(glcm, 'energy')[0, 0] hom = greycoprops(glcm, 'homogeneity')[0, 0] ent = -np.sum(glcm * np.log2(glcm + 1e-10)) te = np.log((1 + con) * (1 + ent)) - ALPHA / ((1 + ener) ** BETA) -GAMMA / ((1 + hom) ** LAMBDA) return te def compute_shape_features(contours): perimeter = cv2.arcLength(contours, True) area = cv2.contourArea(contours) hu_moments = cv2.HuMoments(cv2.moments(contours)).flatten() st = perimeter / area + np.dot(HU_WEIGHTS, hu_moments) return st def compute_heat_features(tmax, tavg, gradient_matrix): ji, ki = gradient_matrix.shape heat = tavg + np.sqrt((tmax - tavg) / (ji * ki) * np.sum(np.abs(np.gradient(gradient_matrix)))) * MU ** C1 return heat def main(video_path): frames = preprocess_video(video_path) results = [] for frame in frames: enhanced = enhance_image(frame) gray = cv2.cvtColor(enhanced, cv2.COLOR_BGR2GRAY) ct = compute_color_features(enhanced) te = compute_texture_features(gray) contours, _ = cv2.findContours(gray, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) st = compute_shape_features(contours[0]) if contours else 0 tmax, tavg = 80, 40 gradient_matrix = np.random.rand(100, 100) hem = compute_heat_features(tmax, tavg, gradient_matrix) results.append({"ColorFeature": ct, "TextureFeature": te, "ShapeFeature": st, "HeatFeature": hem}) return results video_path = "sample_video.mp4" features = main(video_path) print(features).

[0033] The present invention comprehensively reflects the status of wind power equipment through multimodal feature extraction and analysis, and significantly improves the accuracy of defect detection by combining the fusion calculation of color, texture, shape and thermal features. With the help of image processing technology and target detection algorithm, key components can be accurately located and features can be efficiently extracted. At the same time, historical data is used to drive the optimization of model parameters to ensure the reliability and applicability of the detection results, providing strong technical support for the defect diagnosis of wind power equipment.

[0034] Embodiment 3: Combining with Figure 1 and attached Figure 2 In addition to the contents of the above embodiments, the modeling steps of the defect assessment model are as follows: S201: After extracting color, texture, shape and thermal features, combine the component classification identifier i to generate a fused feature vector X: X=[CT i ,TE i ,ST i ,Hem i ,i], the fused feature vector X contains multiple features and classification identifiers of each component, which is used for subsequent classification modeling; S202: Deformation, crack, defect, breakage, corrosion, and overheating are used as defect categories of target components of wind turbines; S203: Using a deep neural network as a training model, taking the fused feature vector X as an input of the training model and the defect category label corresponding to X for training, the deep neural network includes an input layer for receiving the fused feature vector X, a hidden layer composed of more than two fully connected layers, and an output layer that uses a Softmax activation function to convert the output of the network into a probability distribution corresponding to six defect categories; The ReLU activation function is used in each hidden layer. The ReLU activation function introduces nonlinearity to help the network learn the complex relationship between different features and defect categories, so as to better capture the association pattern between the fusion features and defect categories. The Softmax function represents the output of each detection result category as the corresponding probability value of the six defect categories, which helps the model estimate the possibility that the input sample belongs to each defect category. S204: During the training process of the deep neural network, the cross-entropy loss function is used as the optimization objective to measure the difference between the probability distribution of the defect categories predicted by the model and the actual category labels. The gradient of the loss with respect to the network parameters is calculated through the backpropagation algorithm, and the Adam or SGD optimization algorithm is used to iteratively update the network weights and bias parameters to make the model more accurately predict the defect categories of the input samples, so as to obtain the defect evaluation model MOD.

[0035] The recognition unit implements the following steps: S301: Input X = [CT i , TE i , ST i , Hem i , i] into the defect evaluation model, and receive the probabilities P of the six defect categories output by the defect evaluation model: P = MOD(X), P = {P1, P2, P3, P4, P5, P6}, where i is the number of the target component type, i = {1, 2, 3... N}, CT i is the color feature of the visible light image of the ith component, TE i is the texture feature parameter of the visible light image of the ith component, ST i is the shape feature parameter of the visible light image of the ith component, Hem i is the thermal feature value of the infrared image of the ith component, The defect types of deformation, crack, defect, breakage, corrosion, and overheating are represented by the serial numbers 1, 2, 3, 4, 5, and 6 respectively, and the probabilities corresponding to deformation, crack, defect, breakage, corrosion, and overheating are represented as P1, P2, P3, P4, P5, and P6 respectively; S302: Use Pu1, Pu2, Pu3, Pu4, Pu5, and Pu6 to represent the upper limit probabilities corresponding to deformation, crack, defect, breakage, corrosion, and overheating respectively, and compare the probabilities of the six defect categories output by the improved model; S303: Compare the output probability Pc and the upper limit probability Puc of each defect category in turn, and c represents the serial number of each defect type respectively, c ∈ {1, 2, 3, 4, 5, 6}. For each serial number category, if it satisfies: When there is Pc ≥ Puc, it is judged that the key part corresponding to X has the defect type corresponding to the serial number c, When all Pc < Puc, it is judged that the target components of the wind turbine are all intact.

[0036] The upper limit probability Puc is determined by technical personnel in this field based on statistics and analysis of defect detection data of a large number of wind power equipment, by summarizing the actual occurrence probabilities of different defect categories, extracting the probability distribution characteristics of each category, and combining the structural characteristics of wind power equipment and the occurrence laws of common defects. A reasonable upper limit value of probability is preliminarily determined. For different target parts and possible types of defects, the differences in actual working conditions and defect occurrence probabilities are comprehensively considered, and corresponding upper limit values ​​are assigned to each defect category of different components. The actual characteristics of defect occurrence in each component are fully reflected, the rationality of the probability upper limit is ensured, and a more accurate reference basis is provided for subsequent defect determination.

[0037] The present invention realizes efficient and accurate defect detection of key components of wind turbines by integrating multimodal features, adopting deep learning modeling and a probability-based defect judgment mechanism. It has the advantages of clear logic, strong adaptability and good scalability, and can provide strong technical support for equipment operation and maintenance.

[0038] Although the present invention has been described above with reference to various embodiments, it should be understood that many changes and modifications may be made without departing from the scope of the present invention. That is, the methods, systems and devices discussed above are examples. Various configurations may appropriately omit, replace or add various processes or components. For example, in alternative configurations, the method may be performed in an order different from that described, and / or various components may be added, omitted and / or combined. Moreover, the features described with respect to certain configurations may be combined in various other configurations, such as different aspects and elements of the configuration may be combined in a similar manner. In addition, as technology develops, the elements therein may be updated, i.e., many elements are examples and do not limit the scope of the present disclosure or claims. And it should be understood that after reading the contents of the record of the present invention, the technician may make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A multi-modal defect detection system for key wind farm equipment based on drones, characterized in that: The multimodal defect detection system for key equipment in a wind farm includes a drone, a control terminal for controlling the flight of the drone, a model module responsible for data learning of the structural characteristics of the wind farm equipment to build a defect recognition model, and a route planning module for planning the cruising route of the drone.

2. The multi-modal defect detection system for key equipment in a wind farm according to claim 1, characterized in that: The drone is an intelligent drone capable of image shooting, and the camera of the drone is a dual-spectrum camera that can generate infrared images and visible light images simultaneously.

3. The multi-modal defect detection system for key equipment in a wind farm according to claim 2, characterized in that: The control end includes a terminal device and a control application installed in the terminal device, which displays the position and status of the identified key equipment of the wind farm in real time on the display screen of the terminal device, so that the operator can monitor the progress of the inspection in real time. The control application can automatically adjust the flight route of the drone to ensure that the inspection covers all key equipment areas of the wind farm, improving the comprehensiveness and efficiency of the inspection. The control application can automatically adjust the drone's attitude and camera parameters, including focal length, exposure and gimbal angle, to ensure clear images and avoid overexposure and out-of-focus issues that affect recognition accuracy.

4. The multi-modal defect detection system for key equipment in a wind farm according to claim 3, characterized in that: The route planning module generates a path for the UAV to cruise in the wind farm based on the existing path planning algorithm, real-time obstacle avoidance technology and / or efficient coverage path algorithm.

5. The multi-modal defect detection system for key equipment in a wind farm according to claim 4, characterized in that: The model module includes a feature extraction unit for analyzing and processing the features of the image or video taken by the drone, a defect assessment model obtained based on historical data training for assessing different defect conditions of different parts of the drone, and an identification unit for inputting the features extracted by the feature extraction unit into the defect assessment module to further obtain the defect type of the target component of the wind turbine, wherein the features include color, texture, shape and thermal features. The target components of the wind turbine include at least the entire tower, tower head, tower body, tower base, ancillary facilities, cross arm side hanging points, conductor side hanging points, insulators and substation portal frame.

6. The multi-modal defect detection system for key equipment in a wind farm according to claim 5, characterized in that: The UAV transmits the collected images to the terminal device at the control end through the wireless module for the operator to view in real time. The control application generates flight and shooting adjustment instructions in real time and transmits them to the UAV flight control module. The image data taken by the UAV after execution is fed back to the control end. The control end uploads the picture data taken by the UAV to the model module through the wireless network. After obtaining the picture data from the control end, the model module generates the defect information of the wind turbine and transmits it back to the control application for use by the control end and the UAV in subsequent tasks. Among them, the model module is implemented by establishing it in the cloud, and the UAV and the model module communicate through the APN card. To ensure data security, the data transmission between the UAV, the control application and the model module adopts asymmetric encoding and decoding encryption technology.

7. The multi-modal defect detection system for key equipment in a wind farm according to claim 5, characterized in that: The identification unit implements the following steps: S301: X=[CT i ,TE i ,ST i ,Hem i ,i] is input into the defect assessment model, and the probability P of the six defect categories output by the defect assessment model is received: P = MOD(X), P = {P1, P2, P3, P4, P5, P6}, Where i is the number of the target component type, i = {1, 2, 3…N}, CT i is the color feature of the visible light image of the i-th component, TE i is the texture feature parameter of the visible light image of the i-th component, ST i is the shape characteristic parameter of the visible light image of the i-th component, Hem i is the thermal characteristic value of the infrared image of the i-th component, Deformation, crack, defect, damage, corrosion, and overheating are represented by serial numbers 1, 2, 3, 4, 5, and 6, respectively. The corresponding probabilities of deformation, crack, defect, damage, corrosion, and overheating are represented by P1, P2, P3, P4, P5, and P6, respectively. S302: Use Pu1, Pu2, Pu3, Pu4, Pu5, and Pu6 to represent the upper limit probabilities corresponding to deformation, crack, defect, breakage, corrosion, and overheating respectively. Compare the probabilities of the six defect categories output by the improved model. S303: Compare the output probability Pc and the upper limit probability Puc of each defect category in turn, where c represents the serial number of each defect type, c ∈ {1, 2, 3, 4, 5, 6}. For each serial number category, if the following condition is met: When Pc ≥ Puc exists, it is determined that the key part corresponding to X has the defect type corresponding to the serial number c. When all Pc < Puc, it is determined that the target components of the wind turbine are undamaged.

Images