Unmanned aerial vehicle insulator inspection system based on multi-light fusion and method thereof

Abstract

The invention belongs to the technical field of unmanned aerial vehicle inspection, and particularly relates to an unmanned aerial vehicle insulator inspection system and method based on multi-light fusion. Comprising an unmanned aerial vehicle platform and a ground control station. The unmanned aerial vehicle platform is used for carrying inspection equipment and inspecting the insulator according to a preset path; comprising an unmanned aerial vehicle body, a multi-light fusion sensor module, a data processing module, a control module and a communication module, the multi-light fusion sensor module integrates a visible light sensor, an infrared sensor and an ultraviolet sensor and is used for acquiring visible light image information, infrared thermal image information and ultraviolet image information of the insulator, potential faults can be found in the early stage through the three-light fusion technology, and important data support is provided for preventive maintenance; through integration of data of different wavelengths, insulator fault diagnosis under the three-light fusion technology reduces the possibility of misjudgment of insulator faults, and high detection reliability and high efficiency can be maintained under various environmental conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle inspection, and specifically relates to a unmanned aerial vehicle insulator inspection system and method based on multi-light fusion. Background Art

[0002] As a key electrical equipment in power transmission lines, the performance of insulators is crucial to the safe and stable operation of the power grid system. With the continuous expansion of the scale of power grids and the increase in operating time, insulators face a variety of failure risks, such as surface contamination, damage, cracks, and internal insulation degradation.

[0003] The traditional manual inspection method is not only inefficient and labor-intensive, but also greatly affected by environmental and human factors, making it difficult to ensure the accuracy and comprehensiveness of detection; although the drone inspection method based on a single sensor has improved efficiency to a certain extent, due to the limitations of different sensor detection, it is impossible to fully obtain the status information of the insulator.

[0004] For example, visible light cameras are limited by lighting conditions and have difficulty detecting internal defects; infrared thermal imagers are insensitive to weak discharges or early defects and have limited detection distances; and ultraviolet imagers are only suitable for detecting specific defect types.

[0005] Therefore, it is of great practical significance to develop a UAV insulator inspection system based on multi-light fusion. Summary of the invention

[0006] In order to make up for the shortcomings of the prior art, the technical solution adopted by the present invention to solve its technical problems is: a drone insulator inspection system based on multi-light fusion, including a drone platform and a ground control station; the drone platform is used to carry inspection equipment and inspect insulators according to a preset path; it includes a drone body and an integrated multi-light fusion sensor module, a data processing module, a control module and a communication module;

[0007] The ground control station is used to remotely control and monitor the UAV, receive and process the data transmitted by the UAV, display the inspection results and provide decision support, including the automatic cruise module and the manual control module;

[0008] The multi-light fusion sensor module integrates a visible light sensor, an infrared sensor and an ultraviolet sensor, and is used to obtain the visible light image, infrared thermal image and ultraviolet image information of the insulator, and is connected to the data processing module and the control module respectively;

[0009] The control module is connected to the ground control station through the communication module, receives the signal from the ground control station to control the UAV platform, and is connected to the data processing module at the same time, based on the data information fed back by the data processing module;

[0010] The data processing module is used to process and analyze the data collected by the multi-light fusion sensor module, including operations of image fusion, feature extraction and defect recognition;

[0011] Communication module, used to realize data transmission between the UAV and the ground control station;

[0012] The automatic cruise module is used to realize the automatic cruise inspection of the UAV platform after inputting the cruise command, including the automatic obstacle avoidance system and real-time positioning system;

[0013] The manual control module is used to manually control the UAV platform for inspection after personnel input manual control commands.

[0014] Further, the data processing module includes an image registration unit, an image processing unit and an image fusion unit;

[0015] The image registration unit is used to find a spatial transformation model so that corresponding points in the images transmitted by the visible light sensor, the infrared sensor and the ultraviolet sensor are aligned with each other, thereby improving the image fusion effect transmitted by the visible light sensor, the infrared sensor and the ultraviolet sensor;

[0016] The image processing unit is used to pre-process the image registered by the image registration unit, including image enhancement and image filtering;

[0017] The image fusion unit is used to perform a fusion operation on the image processed by the image processing unit, so that the images transmitted by the visible light sensor, the infrared sensor and the ultraviolet sensor are fused to form a fused image.

[0018] Furthermore, the image registration unit includes the following registration steps:

[0019] Step 1: Extract the feature space information of the two images and calculate the mutual information between the ultraviolet image and the visible light image;

[0020] Step 2: Update and iterate the spatial transformation matrix through the optimization search algorithm to find the optimal transformation matrix T that maximizes the mutual information of the similarity measure between the two images;

[0021] Step 3: According to the transformation matrix, the ultraviolet image is spatially transformed so that it is spatially aligned with the visible light image to achieve the image fusion requirement;

[0022] Step 4: Use the fused image and the infrared image to perform image registration again using mutual information as a similarity measure, and finally obtain the registration of the three images.

[0023] Furthermore, the image processing unit enhances the image brightness by linear transformation, as shown in the following formula: O(x, y) = a*I(x, y) + b0≤x<h, 0≤y<w,

[0024] Where: I is the information of the input image, w and h are the width and height of the image respectively, O is the information of the output image, a is a coefficient. When 0 < a < 1, the contrast of the image decreases; when a > 1, the contrast of the image increases; b is a constant term used to adjust the brightness of the image. When b < 0, the image brightness decreases; when b > 0, the image brightness increases.

[0025] Furthermore, the image processing unit performs noise reduction processing in different ways according to different noises, including one or more of median filtering and Gaussian filtering;

[0026] The median filtering method adjusts the window value accordingly according to the change amplitude of the noise to obtain a better filtering effect; as shown in the following formula:

[0027] g(x, y) = median{f(x - i, y - j)}(i, j) ∈ W,

[0028] Where: the sliding window is represented by W, and the median is represented by median{};

[0029] Gaussian filtering is a linear filtering method, as shown in the following formula:

[0030]

[0031] Where: the smoothing and width of the filter are determined by the parameter σ. The larger the value of σ, the smoother the image.

[0032] Furthermore, the image fusion unit performs image fusion based on SD - MSR, including the following steps:

[0033] Step 1: The input images are respectively represented as I A and I B . First, perform multi - scale Gaussian low - pass filtering on the original images, and then use two complementary intensity transfer functions to process the filtered output at each scale, and then obtain two Retinex output quantities with complementary characteristics and (or and ), where the complementary intensity transfer function takes into account the details of the shadow and highlight parts in the image, and the function expression is as follows:

[0034]

[0035] Where, I(x, y) represents any point taken from the set of image pixels, D represents the dynamic range of the image. For an eight - bit grayscale image, D is 256;

[0036] Step 2: All output quantities and By decomposing the output into SD-MSR components and Then, according to the sub-band decomposition homomorphic filtering component fusion method described in the previous section, the fusion process is performed to obtain the

[0037] Step 3: After fusion Component and the subband gain G in the corresponding scale space n The initial fused image is obtained by multiplying and adding the operation results of all components. Since the pixel values ​​of the initial fused image may not fall completely within the dynamic range of the image, a simple gain and offset operation is required to finally obtain a high-contrast fused image.

[0038] Furthermore, the multi-light fusion sensor module is mounted on the UAV platform, and the position and angle of the multi-light fusion sensor module need to be accurately adjusted. The formula is as follows:

[0039] R(x,y,z,θ)=(x arm ,y arm , z arm ,θ arm )

[0040] (x arm ,y arm , z arm ) is the current position of the multi-light fusion sensor module, θ arm is the angle of the multi-light fusion sensor module. Adjust the position and angle of the multi-light fusion sensor module to align with the target.

[0041] Furthermore, the multi-light fusion sensor module is mounted on the UAV platform, and PID control is used to accurately adjust the position and angle of the multi-light fusion sensor module. The formula is as follows:

[0042] e arm (t) = (x target -x arm (t), y target -y arm (t), z target -z arm (t))

[0043] (x arm ,y arm , z arm ) is the current position of the multi-light fusion sensor module, θ am is the angle of the multi-light fusion sensor module. Adjust the position and angle of the multi-light fusion sensor module to align with the target.

[0044] Furthermore, the data processing module also includes a damage detection unit for damage detection and identification of defects;

[0045] The damage detection unit uses YOLOv8n in YOLOv8 as the basic network, and divides the YOLOv8 network structure into five parts: Input layer, Backbone layer, Neck layer, Head layer, and loss function;

[0046] The input layer is the starting part of the entire structure. It processes the input image and then passes it into the model. It uses four methods: Mosaic enhancement, Mixup enhancement, spatial perturbation (random perspective) and color perturbation (HSV augment).

[0047] The Backbone layer uses the CSPNet structure in YOLOv5 and the C2f module. The C2f module is modified from the C3 module in YOLOv5 and designed with the idea of ​​the ELAN module used in YOLOv7. A shortcut is introduced in the C2f module, which consists of n bottlenecks as the main gradient flow branches and 3 CBS (Conv+BN+SiLU) modules.

[0048] The Neck layer uses the FPN+PAN module, where PAN adds a bottom-up pyramid structure after FPN.

[0049] The Head layer is used as the detection head to detect the input image features, replacing the original coupled detection head of YOLOv5 with a decoupled detection head, and changing from Anchor Based to Anchor Free.

[0050] A method for inspecting insulators using a drone based on multi-light fusion, comprising the following steps:

[0051] Step 1: System initialization: Set the inspection path, flight altitude, speed, and shooting angle parameters of the drone at the ground control station, start the drone and multi-light fusion sensor module, and perform system self-check and initialization;

[0052] Step 2: UAV inspection: The UAV flies along the preset inspection route, and the multi-light fusion sensor module collects visible light images, infrared thermal images, and ultraviolet images of the insulator in real time, and transmits the data to the data processing module;

[0053] Step 3: Data fusion and processing: the data processing module pre-processes the received image data, including image registration, image enhancement and denoising, and image fusion operations to obtain a fused image;

[0054] Step 4: Defect identification: Input the fused image into the pre-trained deep learning model for defect identification. Combined with the features extracted by traditional image processing methods, it comprehensively determines whether there are defects in the insulator and the type, location and severity of the defects.

[0055] Step 5: Result transmission and display. The data processing module transmits the defect identification results to the ground control station in real time through the communication module, and the ground control station displays and stores the results.

[0056] The beneficial effects of the present invention are as follows:

[0057] The UAV insulator inspection system and method based on multi-light fusion described in the present invention can comprehensively cover different types of defect detection by combining ultraviolet light, visible light and infrared light, thereby providing more detailed and accurate detection results. In a single inspection, the three-light fusion technology not only improves the efficiency of detection, but also enhances the granularity and grading capabilities of detection. Each light source can reveal defects at different levels, making the detection results more detailed, so that tiny cracks or damage can be discovered in time, and the degree of degradation of the insulator can be more accurately assessed, so that more targeted maintenance measures can be taken.

[0058] More importantly, the three-light fusion technology can detect potential faults at an early stage, providing important data support for preventive maintenance. By integrating data at different wavelengths, insulator fault diagnosis under the three-light fusion technology reduces the possibility of misjudgment of insulator faults, and can maintain high detection reliability and efficiency under various environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention will be further described below in conjunction with the accompanying drawings.

[0060] Figure 1 It is a flow chart of the inspection system of the present invention;

[0061] Figure 2 It is a flow chart of the inspection method of the present invention;

[0062] Figure 3 is the image registration flow chart of the present invention;

[0063] Figure 4 The image enhancement comparison diagram of the present invention is shown in the figure before and after enhancement, with the left side showing the image before enhancement and the right side showing the image after enhancement;

[0064] Figure 5 This is a comparison of image preprocessing. The left side is the image before preprocessing, and the right side is the image after preprocessing.

[0065] Figure 6 This is the image fusion framework diagram based on SD-MSR;

[0066] Figure 7 It is the image contrast after three-light fusion;

[0067] Figure 8 It is the UV image of abnormal corona discharge of insulator;

[0068] Fig. 9 It is the LBP characteristic diagram of the self-exploding insulator string;

[0069] Fig.10 It is a diagram of the fuzzy comprehensive evaluation index system of the present invention;

[0070] Fig.11 It is a diagram of the combined membership function of triangle and semi-trapezoid of the present invention;

[0071] Fig.12 It is the YOLOv8 network structure diagram of the present invention;

[0072] Fig.13 From left to right in the figure are the ELAN module structure diagram, C3 module structure diagram, and C2f module structure diagram;

[0073] Fig.14 This is a schematic diagram of the FPN+PAN structure of the present invention;

[0074] Fig.15 It is a schematic diagram of the coupling detection head of the present invention;

[0075] Fig.16 This is the CIoU structure diagram of the present invention. DETAILED DESCRIPTION

[0076] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0077] like Figure 1-Figure 16 As shown, the present invention proposes a UAV insulator inspection system and method based on multi-light fusion. First, the system is initialized, the inspection path, flight altitude, speed, and shooting angle parameters of the UAV are set at the ground control station, the UAV and the multi-light fusion sensor module are started, and the system self-check and initialization are performed;

[0078] The UAV flies along the preset inspection route through the automatic cruise module, while the automatic obstacle avoidance system and navigation system work. The multi-light fusion sensor module collects visible light images, infrared thermal images and ultraviolet images of insulators in real time, and transmits the data to the data processing module; the image registration unit registers the image and then pre-processes the collected image through the image processing unit, including image enhancement to adjust the contrast and brightness, and Gaussian filtering and median filtering to remove noise to improve image quality; the image fusion unit then effectively fuses the infrared, visible light and ultraviolet image information to achieve higher recognition accuracy in complex environments and display more detailed information.

[0079] In terms of fault detection, the present invention performs fault detection through a damage detection unit, including using local binary pattern (LBP) texture features and longitudinal grayscale analysis to identify insulator self-explosion faults, using an improved algorithm based on grayscale gradient and lateral contrast to detect stain faults, and identifying zero-value insulators, that is, abnormal temperature faults, through infrared image analysis.

[0080] In order to comprehensively evaluate the status of insulators, the present invention adopts a fuzzy comprehensive evaluation model based on the entropy weight method. The model uses the entropy weight method to determine the weights of evaluation indicators, reduce the influence of subjectivity, and transforms qualitative evaluation into quantitative evaluation through the established indicator system and membership function, thereby improving the objectivity and accuracy of the evaluation results.

[0081] Image Registration:

[0082] like Figure 3 The image registration using mutual information as similarity measure refers to finding a spatial transformation model so that the corresponding points in two images are aligned with each other for comparison or fusion;

[0083] The image registration process based on mutual information is as follows: extracting the feature space information of the two images, calculating the mutual information between the ultraviolet image and the visible light image, and then updating and iterating the spatial transformation matrix through the optimization search algorithm to find the optimal transformation matrix T that maximizes the mutual information of the similarity measure of the two images. Finally, the spatial transformation of the ultraviolet image is realized according to the transformation matrix so that it is spatially registered with the visible light image to meet the requirements of image fusion. Then, the fused image and the infrared image are used for another image registration using the mutual information as the similarity measure, and finally the registration of the three images is obtained.

[0084] Preprocessing:

[0085] (1) Image enhancement:

[0086] During the process of collecting images of transmission lines by drone inspection, various factors will have an impact. When the shooting background is too bright or on a sunny day facing the sun, the brightness of the picture will be extremely high; when the shooting background is too dark or on a cloudy day, the picture brightness will be too low; when the background is relatively close in color to the insulator, the features of the insulator in the image will not be very obvious;

[0087] All of the above situations will affect the effect of the detection algorithm. To address this problem, the present invention first adopts the technical means of image enhancement to adjust the contrast of the picture pixels and increase its brightness;

[0088] Adopt the method of linear transformation, as shown in the following formula:

[0089] O(x, y) = a * I(x, y) + b 0 ≤ x < h, 0 ≤ y < w,

[0090] In the formula: I is the information of the input image, w and h are the width and height respectively, o is the information of the output image; a is the coefficient. When 0 < a < 1, the contrast of the image decreases, and when a > 1, the contrast of the image increases; b is the constant term, and its function is to adjust the brightness of the image. When b < 0, the image brightness decreases; when b > 0, the image brightness increases; The result after the image undergoes linear transformation is as Figure 4 shown.

[0091] (2) Image filtering:

[0092] When collecting images, the processes of generating, transmitting, and storing image data are all prone to being affected by noise, mainly including impulse, Gaussian noise, etc. In the actual process of dealing with noise, there are often many types of noise. Therefore, it is necessary to first analyze the filtering methods and types, and then combine different noises with corresponding filtering methods for processing;

[0093] According to the characteristics of this image dataset, the present invention adopts the methods of Gaussian filtering and median filtering to eliminate the noise in the picture;

[0094] Median filtering first sorts the pixels in the neighborhood by gray value, and then calculates the gray value of the central pixel according to the result. Because all the extreme values in the distribution sequence do not affect the median, this method relies on this characteristic to achieve excellent results in noise reduction. Although this method may cause the image edges to become less clear, in the image data collected by the present invention, most insulators are in the center of the picture and the target is relatively large, so the impact can be ignored. The median filtering method adjusts the window value accordingly according to the change amplitude of the noise to obtain a better filtering effect, as shown in the following formula:

[0095] g(x, y) = median{f(x - i, y - j)}(i, j) ∈ W,

[0096] Where: the sliding window is represented by W, and the median is represented by median{};

[0097] Gaussian noise has the characteristic that the probability density function is Gaussian distributed. The output value of Gaussian filtering is the weighted average of the pixels in the neighborhood. Because the Gaussian function is a single-valued function, the closer the distance to the center of the image, the higher the pixel weight. In the data of the image collected by the present invention, most insulators are in the center of the picture and the targets are relatively large, so the weight is relatively high. The two-dimensional Gaussian function has the characteristics of rotational symmetry, and the degree of smoothness in all directions is equal;

[0098]

[0099] Where: The smoothness and width of the filter are determined by the parameter σ. The larger the value of σ, the smoother the image.

[0100] Gaussian functions are separable and can effectively implement Gaussian filtering of larger sizes. Convolution is divided into two steps. First, the image is convolved with a one-dimensional Gaussian function, and then the result is convolved with the same one-dimensional Gaussian function vertically. Therefore, as the template width increases, the amount of calculation for two-dimensional Gaussian filtering also increases, and there is a linear relationship between the two.

[0101] In this processing, in order to improve the calculation efficiency of the model, a pre-processing method is adopted, that is, the template is calculated first and saved in advance;

[0102] After filtering, the effect is as follows Figure 5 As shown in the figure; through the comparative analysis of the effect pictures before and after processing, it is found that after filtering, the characteristics of the insulator in the image are more obvious, which improves the detection effect of the subsequent experimental training model.

[0103] Image Fusion:

[0104] The present invention obtains the temperature information of the equipment through an infrared thermal imager, obtains the surrounding environment information through a visible light sensor, and detects the discharge information generated by corona discharge through an ultraviolet imager. Since the infrared, visible light imaging and ultraviolet imaging principles are different, the image information obtained has different focuses and is complementary. The infrared, visible light and ultraviolet images are fused to improve the recognition ability of the dual video system in a complex electromagnetic environment. The fused image can also show more information details, such as targets blocked by smoke and water vapor, thermal defects inside the target, text on the signboard, needle position, the discharge amount detected by the ultraviolet imager, and the evaluation of the pollution level of the insulator. The useful information obtained by the three sensors can be reflected in one image.

[0105] Here’s how:

[0106] (1) SD-MSR theory

[0107] The SD-MSR method has three notable features:

[0108] 1) An improved logarithmic function is used as the intensity transfer function to enhance the details of the shadow and highlight areas of the image;

[0109] 2) The Retinex output is decomposed into multiple sub-band components with almost no overlap between the sub-bands, so the adaptive gain can be effectively applied to each sub-band component separately;

[0110] The specific process of sub-band decomposition is expressed as follows:

[0111]

[0112] R' n =mlog(I)-mlog(F n *I),

[0113] In the above formula: R n 'and Respectively represent the Retinex output component and the Retinex output component after sub-band decomposition (SD-MSR output component), F n Represents the standard deviation σ from the Gaussian function n The associated wraparound function, σ n Satisfy the conditions: n+1 >σ n ,n=1,2,...,N-1, the symbol * represents the convolution operation, and N represents the number of scales;

[0114] 3) In order to effectively enhance the image contrast, the subband gains in different scale spaces are determined by the SD-MSR output components, which are given by the following formula:

[0115]

[0116] In the above formula, NR n (x, y) is the normalized SD-MSR output component, where σ n is the normalized standard deviation of the Gaussian function, which is defined as follows:

[0117]

[0118] (2) SD-MSR output component fusion method

[0119] Assume that the SD-MSR output components of image A at scale n are expressed as and The SD-MSR output components of the image at each scale are expressed as and. The four components at each scale need to be fused separately, and the weighted coefficient during fusion directly determines the final fusion effect;

[0120] Since the SD-MSR output components The standard deviation of the Gaussian function is n The SD-MSR output component can be calculated by the minimum value σ1 Represents the detail texture of the image, and the maximum value σ n The resulting SD-MSR output components It represents the rough outline of the image, so it also shows that σ n The smaller the value, the higher the frequency of the information. The weighting coefficient can be calculated accordingly, as shown in the following formula:

[0121]

[0122]

[0123] In the above formula, i∈{A+,B+,A-,B-}, represents the normalized result of the SD-MSR output component, σ n represents the normalized standard deviation of the Gaussian function, σ n The expression is:

[0124]

[0125] From the above formula, we can know that: when σ n The smaller it is, the lower the scale of the SD-MSR component is, and the weighting coefficient and The greater the difference between them; and when σ n The larger the SD-MSR component is, the higher the weighting coefficient is. and tend to be equal;

[0126] Therefore, in the lower scale, the image details are enhanced, and in the higher scale, the rough outline of the image is simply averaged and mixed, according to the formula It can be seen that when the constant c w and ε w After the value of is determined, the weighting coefficient can be output by SD-MSR and σ n Calculated;

[0127] By combining the weighted coefficients and the SD-MSR output components, the fused sub-band decomposition in the scale n space can be finally obtained, as shown below:

[0128]

[0129] (3) Image fusion steps

[0130] like Figure 6 As shown, the input images are represented as I A and I B First, the original image is subjected to multi-scale Gaussian low-pass filtering, and then the filtering output at each scale is processed using two complementary intensity transfer functions to obtain two Retinex outputs with complementary characteristics. and (or and ), where the complementary intensity transfer function takes into account the details of the shadows and highlights in the image, and the function expression is as follows:

[0131]

[0132] Where I(x,y) represents any point in the image pixel set, D represents the dynamic range of the image, and for an eight-bit grayscale image, D is 256;

[0133] By using log(·) and pslog(·) instead of mlog(·) in SD-MSR, since mlog(·) has certain shortcomings in taking into account both shadow and highlight parts at the same time, two complementary logarithmic functions are used as intensity transfer functions to further enhance the contrast of the fused image;

[0134] Furthermore, all output and By decomposing the output into SD-MSR components and Then, according to the sub-band decomposition homomorphic filtering component fusion method described in the previous section, the fusion process is performed to obtain the

[0135] After fusion Component and the subband gain G in the corresponding scale space n The initial fused image is obtained by multiplying and adding the operation results of all components. Since the pixel values ​​of the initial fused image may not fall completely within the dynamic range of the image, a simple gain and offset operation is required to finally obtain a high-contrast fused image. The fused image is as follows: Figure 7 shown.

[0136] The damage detection unit is used for insulator fault detection. Insulators are exposed to the field for a long time and are affected by weather, mechanical external forces, etc., which may cause various faults.

[0137] Insulator faults are mainly divided into the following categories: insulator self-explosion fault, insulator crack fault, insulator surface stain fault, insulator temperature abnormality fault (zero value insulator) and insulator abnormal corona discharge; due to the limitations of visible light images, the complex background of infrared images and the fact that ultraviolet images can only detect specific faults, existing methods usually only perform targeted detection on one or two of the faults; the present invention uses visible light, infrared and ultraviolet images taken at the same time and position to achieve comprehensive detection of all the above insulator faults.

[0138] (1) Insulator string self-explosion fault detection

[0139] The present invention evenly divides the insulator string into individual insulator disks, and then extracts the LBP texture features of the individual disk insulator images. Since the self-exploding insulator appears as the absence of the insulator in the image, there will be obvious texture differences from the normal insulator, and the faulty insulators can be identified based on the difference in texture features. The LBP feature is an operator used to describe the local texture features of an image, and is widely used in image detection and classification in computer vision;

[0140] The present invention simulates the algorithm. In order to make the result clearer and more explicit, an artificially simulated self-explosion fault insulator image is used in the simulation. The insulator string image is divided into a single insulator disk image and the LBP feature of each insulator disk is extracted. The LBP feature of each image is made into a histogram form. The difference between each image is compared and analyzed to obtain the result as follows Fig. 9 The LBP characteristic diagram of the self-exploding insulator string is shown.

[0141] Considering that if the insulator string is divided according to each insulator disk, the simulated self-explosion fault may cause the loss of a single insulator and thus completely lose the texture feature, and lose the meaning of comparing the LBP feature, it is chosen here to group the insulators in pairs for feature recognition; as can be seen from the above figure, the self-explosion fault simulated by the insulator is very obvious, but when using the LBP feature for analysis, although there is a slight difference in the LBP histogram between the faulty insulator and the normal insulator, the distinction is very small, and it is difficult to distinguish by direct observation or simple analysis. When a less obvious self-explosion fault occurs, it will be even more difficult to judge; it is not necessary to introduce a learning model here to accurately classify the texture features, so the present invention adopts the insulator self-explosion fault detection based on the longitudinal grayscale according to the characteristics of the insulator image;

[0142] The self-explosion failure of insulators is manifested as a gap in the regularly arranged continuous insulator string. There is only the insulator center axis rod at the self-explosion location, but no insulator. According to the arrangement rule of the insulator horizontal axis, the insulator horizontal axis will be missing at the self-explosion location. By calculating the spacing between adjacent horizontal axes, the spacing of normal insulators increases or decreases regularly and steadily from top to bottom. However, due to the missing horizontal axis at the self-explosion location, the spacing will increase irregularly at the self-explosion location. By analyzing the spacing of the insulator center axis, the self-explosion failure of insulators can be judged.

[0143] (2) Insulator contamination fault detection

[0144] In areas where power transmission lines pass, smoke from factories, salt mist from sea breezes, dust floating in the air, and dust from strong winds accumulate and adhere to the surface of insulators to form a dirt layer, which has a certain degree of moisture retention and conductivity. When there is a small amount of precipitation, melting snow, fog and condensation, and other humid weather, the humidity is high. In this way, the insulation level of the insulator will be greatly reduced, thereby increasing the leakage current on the surface of the insulator. Insulation flashover has not occurred under the action of the working voltage, and even the wooden poles will burn, causing more serious disasters such as fires.

[0145] A normal insulator string has regular image features, and the grayscale changes evenly along the main axis. If the insulator is covered with stains such as bird droppings, dust, ice and snow, the grayscale value of the image at the covered area will change significantly. The present invention is based on the insulator string stain detection based on grayscale gradient and lateral contrast within the insulation string, and determines and detects faults by setting the image longitudinal gradient threshold. Good results can be achieved in dealing with the problem that the overall grayscale change of the insulator is very small and the fault texture is more obvious. At the same time, it can solve the problem of misidentification when there is texture on the insulator as a whole or the insulator is uniformly affected by light to produce a universal grayscale ladder, and when the insulator string is fault-free, the fault area will also be detected.

[0146] (3) Abnormal temperature detection of insulators

[0147] On high-voltage transmission lines, the conductors are insulated by insulator strings. As time goes by, the outdoor wind and sun, and long-term voltage effects may cause individual ceramic bottles to lose their insulation function, which means they can no longer serve as insulators; because the insulation of the insulator is damaged, the insulation resistance is close to 0, and the voltage value borne on the insulator string is also close to 0, so the insulator is called a "zero-value insulator". Although the circuit has a certain margin in design, if the damaged insulator is not replaced in time, the insulation function of the insulator string will be affected, which will cause great safety hazards. High voltage may break down all the remaining good insulators, resulting in single-phase grounding and affecting normal power supply. In order to prevent the above situation from happening, it is necessary to regularly measure and find out the damaged insulators;

[0148] Since visible light images are not sensitive to temperature and cannot detect the temperature of insulators, the present invention uses infrared images to detect zero-value insulators. Zero-value insulators appear darker in color than normal insulators in infrared images. Normal insulators are usually red, and the temperature of the entire insulator string does not fluctuate much. However, since there is no voltage at both ends of zero-value insulators and no heat is generated, the temperature at the zero-value insulator will decrease and appear bluish purple. According to the above-mentioned geometric characteristics, the infrared image is converted into a grayscale value. If the grayscale gradually decreases in the central axis direction and gets closer and closer to the background grayscale value, when the maximum temperature difference on the insulator string (the maximum grayscale difference in the image) exceeds a critical value of the ambient temperature and the insulator temperature difference, it is determined that a zero-value insulator appears at a lower grayscale. If necessary, use grayscale stretching to increase the contrast and refine the grayscale features.

[0149] (4) Abnormal discharge failure of insulator

[0150] Corona discharge occurs when the local electric field strength on the surface of the insulator exceeds the ionization strength of the gas. The current intensity of the corona discharge depends on the voltage between the loading electrodes, the shape of the electrodes, the radius of curvature of the insulator, the distance between the electrodes, the nature and density of the gas, the weather humidity, and the flow rate of the air. The electrical pressure on the insulating materials used in power equipment is increasing day by day. With the increase in the frequency of use and the extension of the use time, the equipment will have different degrees of broken strands, pollution, cracks, and insulation medium damage, thereby generating corona or arcs. During the discharge process, electrons in the air continuously gain and release energy, and when electrons release energy, they emit ultraviolet rays. Using this principle, the obtained ultraviolet image is extracted and segmented for the discharge area, so as to achieve the purpose of determining the location and intensity of the corona, so that further abnormality detection can be performed;

[0151] Combining the Thomson discharge theory and streamer discharge theory, the principle of insulator corona discharge, the principle of insulator pollution flashover discharge, and the working and imaging principles of solar-blind ultraviolet imaging equipment, the noise degradation and restoration model of the insulator ultraviolet detection image is adopted. NSE, MSE, SNR, and PSNR are used as objective evaluation criteria for denoising effect, and the denoising effect is achieved through an adaptive median filtering algorithm; the insulator ultraviolet image is enhanced through Gamma correction, and the ultraviolet detection image is enhanced by selecting a suitable correction coefficient. While ensuring the image background recognition, the ultraviolet spot area is made more obvious and prominent, which is conducive to the effective identification and discrimination of the discharge area; the image is threshold segmented by the Otsu method, and then the binary opening operation of mathematical morphology is performed, and the edge detection and contour extraction of the ultraviolet spot are realized through the Canny edge detection operator, and the image parameters such as the spot area, perimeter, and equivalent diameter are simultaneously calculated as the characteristic parameters and quantitative standards of the insulator discharge intensity.

[0152] The UV image of abnormal corona discharge on insulators is also used to evaluate the insulator status as follows:

[0153] The present invention adopts a fuzzy comprehensive evaluation model based on the entropy weight method to evaluate the pollution flashover state of the insulator, and uses the entropy weight method to determine the index weight, so as to solve the problem of strong subjectivity in weight determination during the fuzzy evaluation process.

[0154] Collect the basic data of transmission lines and the relevant data of insulator flashover test, and establish the fuzzy comprehensive evaluation index system of insulator flashover state based on the inherent factors of insulators and operating environment factors. Fig.10 As shown; Fig.10 The flashover status of insulators in the middle target layer can be decomposed into two evaluation items in the project layer, namely A = {B1, B2}, and then an indicator system is established under each project layer, for example, B1 = {C11, C12, C13}.

[0155] In the fuzzy comprehensive evaluation model of the insulator flashover state of the present invention, the comment factors include both qualitative factors and quantitative factors. According to the data characteristics of the factor set, the distribution function form of the triangle and semi-trapezoidal combination is selected, such as Fig.11 shown.

[0156] The specific method for determining the membership function is as follows: According to the relevant design data, experimental results and monitoring data, Fig.11 The distribution functions of the triangle and semi-trapezoid shown in the figure give the fuzzy boundary intervals of the five state levels, and finally establish the membership function of each state level, as follows:

[0157] Use the j-th evaluation index C of the i-th project layer i jEvaluate the flashover status of insulators, and the status of the comments is l i The membership degrees of j, l2j, l3j, l4j, and l5j are v respectively. j 1, v j2 , v j3 , v j4 , v j5 , then the membership set can be used;

[0158] In order to ensure the objectivity and authenticity of the fuzzy comprehensive evaluation results, not only the influence of the main evaluation indicators on the insulator flashover state should be considered, but also all the information of a single evaluation indicator should be retained. Therefore, the weighted average fuzzy comprehensive evaluation model is selected in the present invention, namely:

[0159]

[0160] The final fuzzy comprehensive evaluation result is: C = [c1 c2 … c N ];

[0161] This operation is a limit sum operation, which takes into account the influence of both main factors and non-main factors, and is suitable for the evaluation of system comprehensive indicators. In the formula, vector C is the comprehensive evaluation result. According to the maximum membership principle, the maximum c i That is the corresponding l i of the evaluation results.

[0162] The present invention applies the entropy weight method and fuzzy theory to the evaluation of the pollution flashover state of insulators. In view of the fuzziness of pollution flashover influencing factors and evaluation knowledge, the membership theory is used to transform the qualitative evaluation into a quantitative evaluation. In order to reduce the influence of subjective factors on the evaluation results, the entropy weight method is introduced, and the weights of the evaluation indicators are corrected by the entropy weight, thereby enhancing the objectivity of the evaluation results.

[0163] When an unclear fault point is found during routine inspection or the fault point needs to be re-verified, or the point where the fault is expected to occur is marked through previous assessment, and the ground control station assigns the UAV platform to the specific point for review or re-testing. It can also be applied to daily fixed-point inspections. The position and angle formula of the multi-light fusion sensor module need to be accurately adjusted as follows:

[0164] R(x,y,z,θ)=(x arm ,y arm , z arm ,θ arm )

[0165] (x arm ,y arm , z arm ) is the current position of the multi-light fusion sensor module, θ arm is the angle of the multi-light fusion sensor module; adjust the position and angle of the multi-light fusion sensor module to align with the target.

[0166] At the same time, the position adjustment of the multi-light fusion sensor module can also be controlled by PID:

[0167]

[0168] The damage detection strategy is as follows:

[0169] YOLOv8 includes five types: YOLOv8n, YOLOv8s, YOLOv8m, YOLOv8l and YOLOv8x. Considering the parameter quantities of the five types and the running speed under model deployment, it is better to use YOLOv8n as the basic network.

[0170] like Fig.12 The YOLOv8 network structure diagram shown can be divided into five parts: input layer, Backbone layer, Neck layer, Head layer, and loss function:

[0171] a). The input layer is the starting part of the entire structure. It processes the input image and then passes it into the model. In order to improve the efficiency of model training, four methods are used: Mosaic enhancement, Mixup enhancement, spatial perturbation (random perspective) and color perturbation (HSV augment);

[0172] b). The Backbone layer uses the CSPNet structure in YOLOv5, and mainly adopts the C2f module; the C2f module is modified from the C3 module of YOLOv5, and is designed with the idea of ​​the ELAN module used in YOLOv7, such as Fig.13 The ELAN, C3 and C2f module structure diagrams shown in the figure, because a large number of shortcuts are introduced in the C2f module, the gradient diversion problem caused by the deep model is alleviated, and the reuse of shallow feature information is improved; the module is composed of n bottlenecks as the main gradient flow branches, and 3 CBS (Conv+BN+SiLU) modules. First, the input features are divided into two branches with different gradients through a convolutional layer with a step size of 1×1. One branch directly reaches Concat, and the other branch passes through n serial Bottleneck modules. The two branches are connected in parallel and finally superimposed to obtain the extracted features. C2f keeps the input and output feature map sizes unchanged through the CBS module, and obtains more hidden information through two layers of gradients of different lengths. This method solves the convergence deterioration caused by the deepening of the model, which makes the overall network performance weaker than the shallow network. In addition, the network model in C3 conforms to the "3 / 6 / 9 / 3" rule, while C2f is "3 / 6 / 6 / 3", the model scale is smaller, and the memory consumption during training is less;

[0173] c). The Neck layer uses the FPN+PAN module. FPN is a top-down feature pyramid structure that aims to optimize the network structure by transmitting high-level semantic features. However, due to the long transmission path, FPN often cannot effectively transmit positioning information, resulting in poor transmission effect. In order to solve this problem, PAN adds a bottom-up pyramid structure after FPN to supplement the shortcomings of FPN. In this way, the positioning features of the lower layer can be effectively transmitted to the higher layer, thus forming a pyramid structure that contains both semantic information and positioning information, further improving the performance of target detection. Its structure is as follows: Fig.14 As shown;

[0174] d). The Head layer is used as the detection head to detect the input image features. The original coupled detection head of YOLOv5 is replaced with a decoupled detection head, and the Anchor Based is changed to Anchor Free.

[0175] YOLOv8 adopts the Anchor Free design, which no longer requires pre-defined anchor boxes to accurately predict the position of the bounding box. This design can directly predict the position and confidence level of the bounding box on the feature map. Compared with the Anchor Based design in YOLOv5, Anchor Free is more versatile and can adapt to objects of various sizes. In the AnchorBased design, since a large number of different anchor boxes need to be generated, many of them are marked as negative samples because they do not reach a certain IoU threshold with the true candidate box, which leads to the problem of uneven distribution of positive and negative samples. In addition, the Anchor Based design relies too much on the setting of hyperparameters and is difficult to optimize. Anchor Free avoids the use of anchor box offsets and solves many problems generated by anchor boxes;

[0176] like Fig.15 The structure of the decoupled detection head is shown in the figure. The design of the decoupled head is to separate the classification and detection heads. Two parallel branches extract category features and position features respectively, and then use a layer of 1×1 convolution to complete the classification and positioning tasks. This improves the accuracy of target detection.

[0177] e). The loss function of YOLOv8 is a multi-task mechanism loss function, which is composed of the regression loss function CIou Loss+DFL (Distribution Focal Loss) and the classification loss function VFL (Vorifocal Loss);

[0178] The CIou Loss function is mainly used to measure the classification accuracy of the model for the target category. In YOLOv8, the corresponding classification loss is mainly calculated through the cross entropy (CIoU). The accuracy of the model is measured by comparing the difference between the category probability distribution at the model output and the true label. The larger the cross entropy, the higher the overlap between the predicted box and the true box. Compared with the original IoU, CIoU has a higher accuracy for small target detection.

[0179] like Fig.16 The CIoU structure diagram shown in the figure, s = ρ 2 (b,b gt ) represents the minimum distance between the predicted box and the real box, l represents the diagonal distance of the minimum rectangle combined between the predicted box and the real box;

[0180] The calculation formula of CIoU is as follows:

[0181]

[0182] In the above formula, w and h represent the length and width of the prediction box. and Represents the length and width of the real box, z represents the difference in the aspect ratio between the predicted box and the real box, and α represents the weight ratio of z, which is a constant;

[0183] The VFL function is an improved version based on the Focal loss function. This loss function proposes a nonlinear weighted method to address the problem of uneven samples. The formula of the VFL function is as follows.

[0184]

[0185] In the above formula, q is the label, which represents the IoU between the real box and the predicted box; when q>0, it is a positive sample, and when q=0, it is a negative sample; when it is a positive sample, it is equivalent to the BCE function, and when it is a negative sample, it is equivalent to the Focal function; this loss function takes into account the proportion of positive and negative samples, and highlights the positive samples as the main body.

[0186] When detecting input images, there may be multiple targets overlapping and occlusion, which may result in the inability to correctly reflect the image semantics. The DFL function is used, which allows the network to quickly focus on the distribution of locations close to the target location.

[0187] The formula for the DFL function is as follows:

[0188] DFL(S i ,S i+1 )=-((y i+1 -y)log(S i )+(yy i )log(S i+1 ));

[0189] In the above formula, S represents the probability output value. i+1 The closer to y and S i The larger it is, the smaller DFL will become, and it will move closer to the vicinity of the target area. That is to say, the distribution theory obtained through training is near the real floating-point coordinate point, and the weights of the integer coordinates on the left and right of the distance are obtained by linear interpolation. Therefore, DFL can improve the efficiency of the network for image extraction and accelerate convergence.

[0190] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A drone insulator inspection system based on multi-light fusion, characterized in that: It includes a UAV platform and a ground control station; the UAV platform is used to carry inspection equipment and inspect insulators according to a preset path; it includes a UAV body and an integrated multi-light fusion sensor module, a data processing module, a control module and a communication module; The ground control station is used to remotely control and monitor the UAV, receive and process the data transmitted by the UAV, display the inspection results and provide decision support, including the automatic cruise module and the manual control module; The multi-light fusion sensor module integrates a visible light sensor, an infrared sensor and an ultraviolet sensor, and is used to obtain the visible light image, infrared thermal image and ultraviolet image information of the insulator, and is connected to the data processing module and the control module respectively; The control module is connected to the ground control station through the communication module, receives the signal from the ground control station to control the UAV platform, and is connected to the data processing module at the same time, based on the data information fed back by the data processing module; The data processing module is used to process and analyze the data collected by the multi-light fusion sensor module, including operations of image fusion, feature extraction and defect recognition; Communication module, used to realize data transmission between the UAV and the ground control station; The automatic cruise module is used to realize the automatic cruise inspection of the UAV platform after inputting the cruise command, including the automatic obstacle avoidance system and real-time positioning system; The manual control module is used to manually control the UAV platform for inspection after personnel input manual control commands.

2. The UAV insulator inspection system based on multi-light fusion according to claim 1 is characterized by: The data processing module includes an image registration unit, an image processing unit and an image fusion unit; The image registration unit is used to find a spatial transformation model so that corresponding points in the images transmitted by the visible light sensor, the infrared sensor and the ultraviolet sensor are aligned with each other, thereby improving the image fusion effect transmitted by the visible light sensor, the infrared sensor and the ultraviolet sensor; The image processing unit is used to pre-process the image registered by the image registration unit, including image enhancement and image filtering; The image fusion unit is used to perform a fusion operation on the image processed by the image processing unit, so that the images transmitted by the visible light sensor, the infrared sensor and the ultraviolet sensor are fused to form a fused image.

3. The UAV insulator inspection system based on multi-light fusion according to claim 2 is characterized by: The image registration unit comprises the following registration steps: Step 1: Extract the feature space information of the two images and calculate the mutual information between the ultraviolet image and the visible light image; Step 2: Update and iterate the spatial transformation matrix through the optimization search algorithm to find the optimal transformation matrix T that maximizes the mutual information of the similarity measure between the two images; Step 3: According to the transformation matrix, the ultraviolet image is spatially transformed so that it is spatially aligned with the visible light image to achieve the image fusion requirement; Step 4: Use the fused image and the infrared image to perform image registration again using mutual information as a similarity measure, and finally obtain the registration of the three images.

4. The UAV insulator inspection system based on multi-light fusion according to claim 2 is characterized by: The image processing unit enhances the image brightness by linear transformation, as shown in the following formula: O(x, y) = a*I(x, y) + b0≤x <h,0≤y<w, Where: I is the information of the input image, w and h are the width and height of the image respectively, 0 is the information of the output image, a is a coefficient. When 0 < a < 1, the contrast of the image decreases; when a > 1, the contrast of the image increases; b is a constant term used to adjust the brightness of the image. When b < 0, the image brightness decreases; when b > 0, the image brightness increases.

5. The UAV insulator inspection system based on multi-light fusion according to claim 4 is characterized by: The image processing unit performs noise reduction processing in different ways according to different noises, including one or more of median filtering and Gaussian filtering; The median filtering method adjusts the window value accordingly according to the change amplitude of the noise to obtain a better filtering effect; as follows: g(x, y) = median{f(x - i, y - j)}(i, j) ∈ W, Where: the sliding window is represented by W, and the median is represented by median{}; Gaussian filtering is a linear filtering method, as shown in the following formula: Where: the smoothing and width of the filter are determined by the parameter σ. The larger the value of σ, the smoother the image.

6. The UAV insulator inspection system based on multi-light fusion according to claim 5 is characterized by: The image fusion unit performs image fusion based on SD-MSR, including the following steps: Step 1: The input images are represented as I A and I B First, the original image is subjected to multi-scale Gaussian low-pass filtering, and then the filtering output at each scale is processed using two complementary intensity transfer functions to obtain two Retinex outputs with complementary characteristics. and (or and ), where the complementary intensity transfer function takes into account the details of the shadows and highlights in the image, and the function expression is as follows: In the formula, I(x, y ) represents any point in the image pixel set, D represents the dynamic range of the image, for an eight-bit grayscale image, D is 256; Step 2: All outputs and By decomposing the output into SD-MSR components and Then, according to the sub-band decomposition homomorphic filtering component fusion method described in the previous section, the fusion process is performed to obtain the Step 3: After fusion Component and the subband gain G in the corresponding scale space n The initial fused image is obtained by multiplying and adding the operation results of all components. Since the pixel values ​​of the initial fused image may not fall completely within the dynamic range of the image, a simple gain and offset operation is required to finally obtain a high-contrast fused image.

7. The UAV insulator inspection system based on multi-light fusion according to claim 1 is characterized by: The multi-light fusion sensor module is mounted on the UAV platform, and the position and angle of the multi-light fusion sensor module need to be accurately adjusted. The formula is as follows: R(x,y,z,θ)=(x arm y arm ,z arm ,θ arm ) (x arm ,y arm , z arm ) is the current position of the multi-light fusion sensor module, θ arm is the angle of the multi-light fusion sensor module. Adjust the position and angle of the multi-light fusion sensor module to align with the target.

8. The UAV insulator inspection system based on multi-light fusion according to claim 1 is characterized by: The multi-light fusion sensor module is mounted on the UAV platform, and the position and angle of the multi-light fusion sensor module are accurately adjusted using PID control. The formula is as follows: e arm (t)=(x target -x arm (t),y target -y arm (t),z target -z arm (t)) (x arm ,y arm , z arm ) is the current position of the multi-light fusion sensor module, θ arm is the angle of the multi-light fusion sensor module. Adjust the position and angle of the multi-light fusion sensor module to align with the target.

9. The UAV insulator inspection system based on multi-light fusion according to claim 2 is characterized by: The data processing module further includes a damage detection unit for damage detection, identification of insulator defects, and insulator status evaluation; The damage detection unit uses YOLOv8n in YOLOv8 as the basic network, and divides the YOLOv8 network structure into five parts: Input layer, Backbone layer, Neck layer, Head layer, and loss function; The Input layer, as the starting part of the whole structure, processes the input image and then transmits it into the model, using four methods: Mosaic augmentation, Mixup augmentation, spatial perturbation (random perspective), and color perturbation (HSV augment); The Backbone layer follows the CSPNet structure in YOLOv5 and uses the C2f module; the C2f module is modified from the C3 module in YOLOv5 and is designed with the idea of the ELAN module adopted in YOLOv7. A shortcut is introduced in the C2f module, which consists of n bottlenecks as the main gradient flow branches and 3 CBS (Conv + BN + SiLU) modules; The Neck layer uses the FPN + PAN module, and PAN adds a bottom-up pyramid structure after FPN; The Head layer, as the detection head part, detects the features of the input picture, replaces the original coupled detection head in YOLOv5 with a decoupled detection head, and changes from Anchor Based to Anchor Free.

10. A drone insulator inspection method based on multi-light fusion, characterized in that: Applicable to the inspection system described in any one of the above claims 1-9, the method includes the following steps: Step 1: System initialization: Set the inspection path, flight altitude, speed, and shooting angle parameters of the drone at the ground control station, start the drone and multi-light fusion sensor module, and perform system self-check and initialization; Step 2: UAV inspection: The UAV flies along the preset inspection route, and the multi-light fusion sensor module collects visible light images, infrared thermal images, and ultraviolet images of the insulator in real time, and transmits the data to the data processing module; Step 3: Data fusion and processing: the data processing module pre-processes the received image data, including image registration, image enhancement and denoising, and image fusion operations to obtain a fused image; Step 4: Defect identification: Input the fused image into the pre-trained deep learning model for defect identification. Combined with the features extracted by traditional image processing methods, it comprehensively determines whether there are defects in the insulator and the type, location and severity of the defects. Step 5: Result transmission and display. The data processing module transmits the defect identification results to the ground control station in real time through the communication module, and the ground control station displays and stores the results.

Images