Unmanned aerial vehicle power grid inspection device and method based on multi-modal vision

Abstract

The invention provides an unmanned aerial vehicle electric power grid inspection device and method based on multi-modal vision, belongs to the technical field of electric power equipment inspection, and aims to solve the technical problems of poor image quality, low defect detection accuracy and low manual marking efficiency in traditional unmanned aerial vehicle inspection. The device comprises an unmanned aerial vehicle, a multi-modal image acquisition device and a computer. The method comprises the following steps of: performing contrast enhancement, deblurring and image feature enhancement on an image by using a PGI-Enhance framework; carrying out feature extraction and feature fusion on the preprocessed image by adopting a DTCFuse model; and carrying out equipment defect detection on the fused thermal imaging data by using a thermal imaging defect detector based on deep learning, and when a defect is found, automatically carrying out real-time circling and labeling on an image so as to realize detection on the power equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power system inspection, in particular to a kind of unmanned aerial vehicle power grid inspection device and method based on multi-modal vision. BACKGROUND

[0002] Power grid inspection is the core link of guaranteeing the safe and stable operation of power system, and has irreplaceable protection function: first, by early detection of power equipment hidden danger and external threat, sudden power failure, equipment damage and other faults can be eliminated in the potential stage, from the source to guarantee the continuity of power supply;Second, based on the actual operation state data of equipment obtained by inspection, accurate decision basis can be provided for equipment maintenance, replacing the traditional uniform periodic maintenance mode, which not only avoids the waste of resources caused by excessive maintenance, but also solves the risk accumulation caused by insufficient maintenance, and improves the utilization efficiency of power system operation and maintenance resources;Third, the inspection data can further provide real data support for power grid planning and load scheduling, which is the key basis for maintaining the reliability, economy and safety of power grid, and is also an important prerequisite for guaranteeing the stable supply of social energy.

[0003] However, the current mainstream power grid inspection method still has significant technical shortcomings, as follows:

[0004] 1) Manual inspection: relying on personnel on-site operation, which has the problems of low efficiency, high cost and easy to be affected by bad weather;And the working environment (high-voltage line, substation) has high safety risk (electric shock, high-altitude fall), and the detection accuracy of hidden defects (internal overheating) and subtle defects (insulator crack) is insufficient, which is easy to miss;

[0005] 2) Helicopter inspection: although it improves the inspection range and efficiency, it is greatly limited by terrain (mountainous area, complex building group), weather (thick fog, lightning) and airspace control;The purchase and operation cost is high, and it is only suitable for large-scale backbone power grid, and cannot cover distribution lines, underground cables and other scenes;And the identification accuracy of subtle defects (wire breakage) is limited, which needs manual review, and it is difficult to realize full automation.

[0006] In order to solve the above problems, in recent years, the industry has proposed unmanned aerial vehicle inspection technology, among which the scheme based on single-mode camera image acquisition and combined with improved ResNet network to construct image classification model has become a research hotspot, but this kind of scheme still has the following key technical shortcomings, which limits its large-scale engineering application:

[0007] 1) High data dependence: model training needs large-scale labeled data set, while power inspection defect samples are scarce and have large feature difference, and the difficulty and cost of data set collection and labeling are high, which limits the model adaptability;

[0008] 2) Poor environmental adaptability: existing models are not optimized for complex environments (strong light, shadow, haze) of power inspection, have weak noise and uneven light suppression ability, poor generalization ability, and detection stability is difficult to guarantee in complex environments;

[0009] 3) Lack of data processing: traditional unmanned aerial vehicle inspection directly detects raw images without targeted preprocessing (contrast enhancement, deblurring), background interference and detail blur result in low detection accuracy, which cannot meet the high precision requirement.

[0010] In summary, the existing power grid inspection methods all have technical shortcomings, and an inspection technology scheme that takes into account efficiency, cost, accuracy and environmental adaptability is urgently needed to solve the shortcomings of existing technologies. SUMMARY

[0011] In view of the above technical deficiencies, the purpose of the present application is to provide a kind of unmanned aerial vehicle power grid inspection device and method based on multi-modal vision, adaptive real-time power grid equipment detection and intelligent marking in different environments such as strong light, haze and night, with high detection accuracy, environmental robustness and inspection timeliness.

[0012] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0013] A kind of unmanned aerial vehicle power grid inspection device based on multi-modal vision, unmanned aerial vehicle, built-in flight control system and loaded with multi-modal image acquisition device, wireless communication module;Multi-modal image acquisition device is erected in the front of unmanned aerial vehicle, for collecting infrared image and visible light image of power facility;Wireless communication module is integrated in the fuselage of unmanned aerial vehicle, is connected multi-modal image acquisition device and flight control system of unmanned aerial vehicle through data interface;

[0014] Computer, through wireless communication link and wireless communication module interaction to control unmanned aerial vehicle flight, while receiving image data and executing in turn: PGI-Enhance framework preprocessing, DTCFuse model feature extraction and fusion, deep learning based thermal imaging defect detector detection.

[0015] Preferably, the multi-modal image acquisition device is a high-definition camera, and the wireless communication module is an industrial-grade 4G / 5G dual-mode communication module.

[0016] A kind of unmanned aerial vehicle power grid inspection method based on multi-modal vision, comprising the following steps:

[0017] 1) Image acquisition: the multi-modal image acquisition device below the unmanned aerial vehicle synchronously collects infrared and visible light images of power facility, and transmits to computer through wireless communication link;

[0018] 2) Image preprocessing: The computer performs contrast enhancement, motion deblurring, and feature enhancement on infrared and visible light images respectively using the PGI-Enhance framework;

[0019] 3) Feature extraction and fusion: The computer uses the DTCFuse model to extract and fuse the thermal features of the preprocessed infrared image and the detailed features of the visible light image, providing support for defect detection;

[0020] 4) Defect detection: The computer uses a deep learning thermal imaging defect detector to detect defects in power equipment by fusing features; if a defect is detected, the defect area is circled and marked in real time.

[0021] Preferably, the specific steps of image preprocessing in step 2) include:

[0022] 2.1) Differential histogram equalization is used to enhance image contrast:

[0023] The image to be processed is input into the adaptive histogram equalization module. Different parameters are used in the contrast-limiting adaptive histogram equalization formula to address the differences in characteristics between infrared and visible light images. Its mathematical expression is:

[0024] (1)

[0025] in, To enhance the image in coordinates Pixel value at that location, The original input image in coordinates Pixel value at that location, To limit the contrast threshold, The tile size for image segmentation; when processing infrared images, , =16; When processing visible light images , =8;

[0026] 2.2) An intelligent denoising strategy is used to perform deblurring on the image:

[0027] Input the noisy image into the intelligent denoising module, and automatically select the appropriate denoising method according to the image type;

[0028] For infrared images, the standard deviation is used. The standard deviation was determined by Gaussian filtering for noise reduction, and the value of the standard deviation was optimized based on the noise statistical characteristics of 20 sets of infrared images.

[0029] For visible light images, spatial domain standard deviation is used. Standard deviation of grayscale range bilateral filtering to remove noise; wherein and The value of is determined based on the structural features and gray scale distribution of 20 groups of visible light images.

[0030] 2.3) Adopting power grid inspection double threshold The edge enhancement algorithm performs enhancement processing on image features:

[0031] The power grid inspection double threshold The edge enhancement algorithm refers to a double threshold optimized for target edge features in power grid images Operator; the image after deblurring in step 2.2) is input into the edge enhancement module, and its mathematical expression is:

[0032] (2)

[0033] wherein, is the output edge enhanced image, is the edge detection operator, is the input image, is the low threshold, is the high threshold.

[0034] Preferably, the specific steps of image feature extraction and fusion in step 3) are as follows:

[0035] 3.1) Input the preprocessed image into a three-level feature pyramid to extract multi-scale features:

[0036] The three-level feature pyramid is used to extract device detail features and global features at different scales, wherein:

[0037] First level: original scale, using 3x3 convolution kernel, step 1, padding 1, for feature extraction;

[0038] Second level: 1 / 2 scale, after 3x3 convolution + step 2 downsampling, followed by 2 3x3 convolutions, step 1, for feature extraction;

[0039] Third level: 1 / 4 scale, after 3x3 convolution + step 2 downsampling, followed by 3 3x3 convolutions, step 1, for feature extraction;

[0040] Finally, the original scale feature , 1 / 2 scale feature , and 1 / 4 scale feature are obtained.

[0041] 3.2) Project the infrared image features and visible light image features using 1x1 convolution respectively:

[0042] Projecting infrared image features to generate query matrix Projecting visible light image features to generate key matrix and value matrix Calculate the spatial correlation of infrared and visible light features through query matrix , key matrix and value matrix , combined with dynamic attention mask to generate attention weight matrix, whose mathematical expression is:

[0043] (3)

[0044] wherein, is the dimension of the key vector, is the dynamic attention mask based on the region of power equipment, which is generated by performing 1x1 convolution operation on the original scale feature and then activating (i.e. ), is the function; 3.3) Perform integration and adaptive fusion on multi-scale features:

[0045] Downsampled feature maps

[0046] , obtained in step 3.1) are upsampled to the original size by bilinear interpolation, and , upsampled and are integrated by feature concatenation, and adaptive feature fusion is realized by 1x1 convolution, whose mathematical expression is:

[0047] (4) wherein,

[0048] is the output of the fused feature, is the activation function, is the bilinear interpolation upsample operator, is the feature concatenation operator, and are the weight matrices of the corresponding features, respectively;

[0049] 3.4) Perform weighted fusion on dual-modal features based on attention weights:

[0050] The attention weight matrix obtained in step 3.2) is used to perform weighted fusion on the value matrix of the visible light feature and the value matrix of the infrared feature, and its mathematical expression is:

[0051] (5)

[0052] wherein, is the bimodal fusion feature;

[0053] 3.5) the loss function of the fusion model is:

[0054] (6)

[0055] wherein, is the intensity loss term, which is calculated by the norm constraint fusion feature and the consistency of the maximum intensity feature of the source image, to ensure that the infrared image thermal radiation feature and the visible light image brightness feature are not lost, and its mathematical expression is:

[0056] (7)

[0057] wherein, is the visible light image feature, is the infrared image feature, is the fused target feature, is the norm;

[0058] is the gradient loss term, which is calculated by the norm constraint fusion feature gradient and the consistency of the maximum gradient feature of the source image, to strengthen the retention of the device edge and texture feature, and its mathematical expression is:

[0059] (8)

[0060] wherein, is the gradient of the visible light image feature, is the gradient of the infrared image feature, is the gradient of the fused target feature.

[0061] Preferably, the specific steps of defect detection in step 4) are as follows:

[0062] 4.1) a deep learning-based thermal imaging defect detector is used to perform processing on the fused thermal imaging data (the processing process retains the original temperature value and inherits the infrared thermal radiation feature and the visible light structure feature), and the thermal anomaly probability of each pixel position is calculated by mathematical expression (9):

[0063] (9)

[0064] wherein, is the thermal anomaly probability value at the image coordinate , and the value range is , is an activation function, is a learnable weight vector, is a feature extraction function, is a coordinate of the fused infrared feature tensor;

[0065] 4.2) First, 4-layer progressive down-sampling is used to extract multi-scale features, and then 4 times super-resolution reconstruction is realized through two times of transposed convolution to restore the details of the defect area and improve the positioning accuracy. For the detected potential defect area, mathematical expression (10) is used for judgment to filter invalid small area interference:

[0066] (10)

[0067] wherein, is the final judged defect area binary graph, is a morphological opening operation, and 0.5 is a probability threshold, is the area of the connected region, is the minimum defect area threshold, and the default value is 10 pixels, which can be flexibly adjusted according to the power inspection scene;

[0068] 4.3) The method of combining dynamic temperature threshold with CNN probability graph is used to perform spatial consistency verification on the defect area, and the specific implementation process is as follows:

[0069] 4.3.1) The temperature abnormality degree is evaluated by calculating the difference between the highest temperature in the detection area and the average temperature of the background area, and the mathematical expression of the temperature abnormality index is shown in equation (11):

[0070] (11)

[0071] wherein, is the temperature abnormality index value, is the highest temperature value in the detection area, is the average temperature of the background area;

[0072] 4.3.2) Based on the principle, the dynamic temperature threshold is calculated, and the mathematical expression of the dynamic threshold is shown in equation (12):

[0073] (12)

[0074] wherein, is the dynamic temperature threshold, is the mean of the temperature distribution of the detection area, is the standard deviation of the temperature distribution of the detection area;

[0075] 4.3.3) Perform a logical AND operation between the CNN output probability map and the temperature anomaly detection result at the pixel level to ensure that the defect area simultaneously satisfies "temperature anomaly index exceeds dynamic threshold" and "CNN prediction probability is greater than 0.5". The mathematical expression for the judgment is shown in Equation (13):

[0076] (13)

[0077] in, This is the binary image of the final defect detection result. This is a heat map output by a CNN network, and its pixel values ​​range from... ;

[0078] 4.4) In the defect detection process, a multi-task joint optimization strategy is adopted (the optimization goal is to improve the defect detection accuracy and positioning accuracy). The optimization is achieved by weighted combination of two key loss functions, and the mathematical expression is shown in equation (14):

[0079] (14)

[0080] in, To predict and match the actual pixel values Norm, constraining the strength and consistency of test results; For structural similarity loss, the mean, variance, and covariance within a local window are calculated to capture image structural features. The core calculation relationship is as follows:

[0081] (15)

[0082] In the formula, These are the mean values ​​of the predicted image and the real image, respectively. These represent the variances of the predicted image and the real image, respectively. To predict the covariance between the image and the real image, The value and weight 0.3 were determined with reference to existing technologies.

[0083] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0084] 1) This invention integrates drone inspection with deep learning, which not only retains the advantages of drones in terms of high efficiency and low risk compared to manual inspection, and low cost and strong adaptability compared to helicopter inspection, but also solves the shortcomings of traditional drones that rely only on visible light and cannot detect abnormal equipment temperature.

[0085] 2) The application detects through the DTCFuse multi-modal fusion model, uses infrared image thermal radiation features (captures temperature anomalies) and visible light image detail features (recognizes structural defects), solves the detection failure problem of traditional single modal in harsh environments (night, haze, strong light), and improves detection accuracy;

[0086] 3) The application does not need large-scale labeled data sets, realizes defect determination through multi-modal fusion, simplifies the process, improves efficiency, and the model does not depend on specific scene labeled data, has strong generalization ability in different regions and equipment type inspection;

[0087] 4) The application realizes adaptive defect detection, the DTCFuse model can automatically complete feature learning and defect labeling, does not need manual frame-by-frame image checking, solves the time-consuming and easy-to-miss problem of traditional unmanned plane manual review, and greatly improves the inspection efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0088] Figure 1 The method flowchart of the application;

[0089] Figure 2 The structural schematic diagram of the device of the application.

[0090] Among them:

[0091] 1, multi-modal image acquisition device; 2, unmanned plane; 3, wireless communication module; 4, computer; 5, power facility. DETAILED DESCRIPTION

[0092] The application will be further described below in conjunction with the drawings.

[0093] As shown in the drawings, an unmanned plane power grid inspection device based on multi-modal vision comprises: Figure 2

[0094] The multi-modal image acquisition device 1 is erected directly below the unmanned plane 2 body, can synchronously acquire infrared images and visible light images of the power facility 5, and transmit the acquired images to the computer 4 through the wireless communication link of the unmanned plane 2;

[0095] The wireless communication module 3 is integrated in the unmanned plane 2 body, one end is connected with the multi-modal image acquisition device 1 and the flight control system of the unmanned plane 2 through a data interface, and the other end establishes a wireless communication link with the computer 4 through 4G / 5G dual-mode communication technology, and is used for transmitting multi-modal image data and flight control instructions;

[0096] ​Computer 4 establishes data interaction with wireless communication module 3 through wireless communication link to realize flight control of UAV 2. Computer 4 is also used to receive image data transmitted by multimodal image acquisition device 1 and perform the following operations on the image data in sequence: preprocessing using PGI-Enhance framework, feature extraction and feature fusion using DTCFuse model, and power equipment defect detection using deep learning-based thermal imaging defect detector.

[0097] As a preferred embodiment of the present invention, the multimodal image acquisition device 1 is a high-definition camera with simultaneous infrared and visible light shooting capabilities, and the resolution of the high-definition camera is not less than 1920×1080 pixels. The wireless communication module 3 of the present invention adopts an industrial-grade 4G / 5G dual-mode communication module, which meets the requirements of a maximum transmission rate of up to 300Mbps, latency ≤20ms, and anti-electromagnetic interference capability. Of course, other cameras and wireless communication modules can also be selected as needed.

[0098] like Figure 1 As shown, a method for UAV power grid inspection based on multimodal vision is described, with the following specific steps:

[0099] 1) Image acquisition: The multimodal image acquisition device 1 directly below the UAV 2 simultaneously acquires infrared and visible light images of the power facility 5 and transmits them to the computer 4 via a wireless communication link;

[0100] 2) Image preprocessing: Computer 4 performs contrast enhancement, motion deblurring, and feature enhancement on infrared and visible light images respectively using the PGI-Enhance framework. The specific steps are as follows:

[0101] 2.1) Differential histogram equalization is used to enhance image contrast:

[0102] The image to be processed is input into the adaptive histogram equalization module. Taking into account the differences in characteristics between infrared images (emphasizing temperature distribution features) and visible light images (emphasizing detail and texture features), different parameter-limited adaptive histogram equalization formulas are used for each. Its mathematical expression is:

[0103] (1)

[0104] in, To enhance the image in coordinates Pixel value at that location, The original input image in coordinates Pixel value at that location, To limit the contrast threshold, The tile size for image segmentation. When processing infrared images, , = 16; when processing visible light images, , = 8;

[0105] 2.2) Perform deblurring processing on the image using an intelligent denoising strategy:

[0106] Input the noisy image into the intelligent denoising module, and automatically select the appropriate denoising method according to the image type (infrared image or visible light image);

[0107] For infrared images, use standard deviation Gaussian filter for denoising processing, and the standard deviation is determined based on the noise statistical characteristics of 20 groups of infrared images (covering typical scenes such as night strong light and night low temperature);

[0108] For visible light images, use spatial domain standard deviation , gray domain standard deviation Bilateral filter for denoising processing; wherein, and The values of and are optimized based on the structural characteristics and gray distribution of 20 groups of visible light images (covering complex lighting scenes such as sunny strong light and cloudy shadow);

[0109] 2.3) Perform enhancement processing on the image features using the power grid inspection double-threshold edge enhancement algorithm:

[0110] The power grid inspection double-threshold edge enhancement algorithm refers to a double-threshold operator optimized for target edge features such as conductors and insulators in power grid images; input the deblurred image from step 2.2) into the edge enhancement module, and its mathematical expression is:

[0111] (2)

[0112] Where, is the output edge enhanced image, edge detection operator, is the input image, is the low threshold, is the high threshold.

[0113] 3) Feature extraction and fusion: The computer 4 uses the DTCFuse model to extract and fuse the thermal features of the preprocessed infrared image and the detail features of the visible light image, providing support for defect detection, and the specific steps are as follows:

[0114] 3.1) The preprocessed image is input into a three-level feature pyramid to extract multi-scale features:

[0115] The three-level feature pyramid is used to extract device detail features and global features at different scales, where:

[0116] The first level (original scale) uses a 3x3 convolution kernel (stride 1, padding 1) for feature extraction;

[0117] The second level (1 / 2 scale) uses a 3x3 convolution + stride 2 downsampling, followed by two 3x3 convolutions (stride 1) for feature extraction;

[0118] The third level (1 / 4 scale) uses a 3x3 convolution + stride 2 downsampling, followed by three 3x3 convolutions (stride 1) for feature extraction;

[0119] Finally, the original scale feature , 1 / 2 scale feature , and 1 / 4 scale feature are obtained.

[0120] 3.2) Project the infrared image features and visible light image features using a 1x1 convolution:

[0121] Project the infrared image features to generate a query matrix , project the visible light image features to generate a key matrix , and a value matrix ; Calculate the spatial correlation of infrared and visible light features through the query matrix , key matrix , and value matrix , and generate an attention weight matrix combined with dynamic attention mask, the mathematical expression is:

[0122] (3)

[0123] where is the dimension of the key vector, is the dynamic attention mask based on the power equipment region, which is generated by performing a 1x1 convolution operation on the original scale feature and then activation (i.e. ), is the function;

[0124] 3.3) Perform integration and adaptive fusion on multi-scale features:

[0125] Downsampled feature maps , Up-sampling to the original size by bilinear interpolation, integrating the features in a feature concatenation manner , up-sampled and , and realizing adaptive feature fusion through 1x1 convolution, the mathematical expression of which is:

[0126] (4)

[0127] wherein, is the output of the fused features, is an activation function, is a bilinear interpolation up-sampling operator, is a feature concatenation operator, are weight matrices of the corresponding features, respectively; 3.4) Perform weighted fusion on the dual-modal features based on the attention weights:

[0128] Use the attention weight matrix obtained in step 3.2)

[0129] to perform weighted fusion on the value matrix of the visible light feature and the value matrix of the infrared feature , the mathematical expression of which is:

[0130] (5) wherein,

[0131] is the dual-modal fused feature; 3.5) The loss function of the fusion model is:

[0132]

[0133] (6) wherein,

[0134] is the intensity loss term, which constrains the consistency of the fused features and the maximum intensity features of the source images through norm, ensuring that the thermal radiation features of the infrared images and the brightness features of the visible light images are not lost, the mathematical expression of which is:

[0135] (7) wherein,

[0136] is the feature of the visible light image, is the feature of the infrared image, is the fused target feature, is norm;

[0137] is the gradient loss term, which constrains the consistency of the gradients of the fused features and the source images through ​Norm constraints ensure consistency between the fusion feature gradient and the maximum gradient feature of the source image, enhancing the preservation of device edge and texture features (such as insulator contours and conductor edges). Its mathematical expression is:

[0138] (8)

[0139] In the formula, The gradient of visible light image features. The gradient of infrared image features. The gradient of the fused target features ( All were extracted using the Sobel operator.

[0140] The weight coefficient of 10 strengthens the proportion of the gradient loss term, making the model training more focused on the fusion quality of device edge and texture features, which is in line with the requirements of power inspection for the clarity of device structure.

[0141] 4) Defect Detection: Computer 4 uses a deep learning thermal imaging defect detector to detect defects in power equipment based on fused features; if a defect is detected, the defect area is selected and marked in real time. The specific steps are as follows:

[0142] 4.1) A deep learning-based thermal imaging defect detector is used to process the fused thermal imaging data (the processing retains the original temperature value and inherits the infrared thermal radiation features and visible light structural features), and the position of each pixel is calculated by mathematical expression (9). Thermal anomaly probability:

[0143] (9)

[0144] in, Image coordinates The probability value of thermal anomaly at the location is within the range of . , for Activation function For learnable weight vectors, For feature extraction function, coordinates The fused infrared feature tensor at the location;

[0145] 4.2) First, a 4-layer progressive downsampling method is used to extract multi-scale features, and then two transposed convolutions are used to achieve 4x super-resolution reconstruction to restore the details of the defect area and improve the positioning accuracy; for the detected potential defect area, the mathematical expression (10) is used for judgment to filter out invalid small area interference:

[0146] (10)

[0147] in, binary image of defect region for final decision, morphological opening operation, 0.5 is the probability threshold (used to convert the thermal anomaly probability Figure Two to binary image), area of connected region (calculated by Shoelace formula), minimum defect area threshold, the default value is 10 pixels, which can be flexibly adjusted according to the power inspection scene (such as different defect types such as insulator crack, conductor joint overheating, etc.);

[0148] 4.3) The method of combining dynamic temperature threshold with CNN probability map is adopted to perform spatial consistency verification on the defect region, and the specific implementation process is as follows:

[0149] 4.3.1) The temperature anomaly degree is evaluated by calculating the difference between the highest temperature in the detection region and the average temperature in the background region, and the mathematical expression of the temperature anomaly index is shown in equation (11):

[0150] (11)

[0151] wherein, is the temperature anomaly index value, is the highest temperature value in the detection region, is the average temperature of the background region (selecting the region of 10% of the image edge to calculate the average temperature);

[0152] 4.3.2) Based on the principle, the dynamic temperature anomaly threshold is calculated, and the mathematical expression of the dynamic threshold is shown in equation (12):

[0153] (12)

[0154] wherein, is the dynamic temperature threshold, is the mean of the temperature distribution of the detection region, is the standard deviation of the temperature distribution of the detection region;

[0155] 4.3.3) The logical AND operation of the CNN output probability map and the temperature anomaly detection result is performed for the pixel level, and it is ensured that the defect region meets the conditions of "temperature anomaly index exceeding dynamic threshold" and "CNN prediction probability greater than 0.5" at the same time, and the mathematical expression of the decision is shown in equation (13):

[0156] (13)

[0157] wherein, is the final defect detection result binary image (the value of 1 represents the defect region, and 0 represents the normal region), The heat anomaly probability map output by the CNN network has pixel value in the range of .

[0158] 4.4) In the defect detection process, a multi-task joint optimization strategy (optimization target is to improve the defect detection accuracy and positioning accuracy) is adopted, and optimization is realized by weighted combination of two key loss functions, and the mathematical expression is shown as formula (14):

[0159] (14)

[0160] wherein, is the L2 norm of the predicted and real pixel values, and the intensity consistency of the detection result is constrained; is a structural similarity loss, which captures the structural features of the image by calculating the mean, variance and covariance in the local window (11*11), and the core calculation relationship is:

[0161] (15)

[0162] In the formula, are the mean of the predicted image and the real image, respectively, are the variance of the predicted image and the real image, respectively, is the covariance of the predicted image and the real image, The value and the weight 0.3 are determined with reference to the prior art, and the present application combines and with the weight, so that the model can ensure the intensity matching of the defect area and accurately retain the device structure details in the back propagation process.​

Claims

1. A multi-modal vision based unmanned aerial vehicle power grid inspection device, characterized in that, The utility model relates to a kind of unmanned aerial vehicle and its image processing method, including: Unmanned aerial vehicle, built-in flight control system and carry multi-modal image acquisition device, wireless communication module; Multi-modal image acquisition device is erected in the front of unmanned aerial vehicle, for collecting infrared image and visible light image of electric power facility; Wireless communication module is integrated in the fuselage of unmanned aerial vehicle, and multi-modal image acquisition device and the flight control system of unmanned aerial vehicle are connected by data interface; Computer is interacted with wireless communication module by wireless communication link to control unmanned aerial vehicle flight, while receiving image data and sequentially executing: PGI-Enhance framework preprocessing, DTCFuse model feature extraction and fusion, deep learning-based thermal imaging defect detector detection.

2. The unmanned aerial vehicle power grid inspection device based on multi-modal vision of claim 1, wherein, The multi-modal image acquisition device is a high-definition camera, and the wireless communication module is an industrial-grade 4G / 5G dual-mode communication module.

3. A multi-modal vision based unmanned aerial vehicle power grid inspection method, characterized in that, Including the following steps: 1) image acquisition: the multi-modal image acquisition device in the front of unmanned aerial vehicle synchronously collects infrared and visible light images of electric power facility, and transmits to computer via wireless communication link; 2) image preprocessing: computer executes contrast enhancement, motion deblurring and feature enhancement on infrared and visible light images respectively by PGI-Enhance framework; 3) feature extraction and fusion: computer extracts thermal features of preprocessed infrared image and detail features of visible light image and fuses them using DTCFuse model to provide support for defect detection; 4) defect detection: computer detects defects of electric power equipment by deep learning thermal imaging defect detector on fused features;If defect is detected, real-time circle selection and defect area labeling are performed.

4. The unmanned aerial vehicle power grid inspection method based on multi-modal vision of claim 3, wherein, The specific steps of image preprocessing in step 2) include: 2.1) differential histogram equalization technology is used to enhance the contrast of the image: The image to be processed is input into an adaptive histogram equalization module, and different parameters of a limited contrast adaptive histogram equalization formula are respectively used according to the characteristic differences of the infrared image and the visible light image The mathematical expression is: （1） wherein, is the pixel value of the enhanced image at coordinates , is the pixel value of the original input image at coordinates , is a contrast limit threshold, is a tile size for image segmentation; when processing infrared images, , = 16; when processing visible light images, , = 8; 2.2) intelligent denoising strategy is used to deblur the image: The noisy image is input into the intelligent denoising module, and the appropriate denoising method is automatically selected according to the image type; For infrared images, the standard deviation is used. The standard deviation was determined by Gaussian filtering for noise reduction, and the value of the standard deviation was optimized based on the noise statistical characteristics of 20 sets of infrared images. For the visible light image, a bilateral filter with spatial domain standard deviation and gray domain standard deviation is used for denoising; wherein, and are determined based on the structure characteristics and gray distribution of 20 groups of visible light images. 2.3) Adopting double threshold of power grid inspection The edge enhancement algorithm performs enhancement processing on image features: The power grid inspection double threshold The edge enhancement algorithm refers to a double threshold value optimized for target edge features in a power grid image Operator; the image deblurred in step 2.2) is input into an edge enhancement module, and the mathematical expression is: （2） wherein, is an output edge enhanced image, is an edge detection operator, is an input image, is a low threshold value, is a high threshold value.

5. The unmanned aerial vehicle power grid inspection method based on multi-modal vision of claim 3, wherein, The specific steps of image feature extraction and fusion in step 3) are as follows: 3.1) the preprocessed image is input into a three-level feature pyramid to extract multi-scale features: The three-level feature pyramid is used to extract device detail features and global features at different scales, wherein: The first level: the original scale, 3x3 convolution kernel, step 1, padding 1, feature extraction; The second level: 1 / 2 scale, after 3x3 convolution + step 2 downsampling, 2 3x3 convolutions are connected, step 1, feature extraction; The third level: 1 / 4 scale, after 3x3 convolution + step 2 downsampling, 3 3x3 convolutions are connected, step 1, feature extraction; final scale features , 1 / 2 scale features , and 1 / 4 scale features ; 3.2) 1x1 convolution is used to project infrared image features and visible light image features respectively: Projecting infrared image features to generate a query matrix Projecting visible light image features to generate a key matrix and a value matrix Calculate the spatial correlation of infrared and visible light features through the query matrix , the key matrix and the value matrix Combine the dynamic attention mask to generate an attention weight matrix, and its mathematical expression is: （3） wherein, is the dimension of the key vector, is a dynamic attention mask based on the power equipment area, specifically generated by performing a 1x1 convolution operation on the original scale feature after activation (i.e. ), is a function; 3.3) multi-scale features are integrated and adaptively fused: upsample to the original size by bilinear interpolation, integrate , by feature concatenation , the upsampled and , and realize adaptive feature fusion by 1x1 convolution, and the mathematical expression is: （4） wherein, is a fused feature output, is an activation function, is a bilinear interpolation up-sampling operator, is a feature concatenation operator, are weight matrices for the corresponding features, respectively. 3.4) weighted fusion is performed on dual-modal features based on attention weights: The attention weight matrix obtained in step 3.2) The value matrix of the visible light feature The value matrix of the infrared feature The weighted fusion is performed, and the mathematical expression is: （5） wherein, is a bimodal fusion feature; 3.5) the loss function of the fusion model is: （6） wherein, is the intensity loss term, which is calculated by The norm constraint fuses the consistency of the feature and the maximum intensity feature of the source image, ensuring that the thermal radiation feature of the infrared image and the brightness feature of the visible light image are not lost, and its mathematical expression is: （7） wherein, is a visible light image feature, is an infrared image feature, is a fused target feature, is norm; For the gradient loss term, by The norm constraint fuses the consistency of the gradient features and the maximum gradient features of the source image, and strengthens the preservation of the device edge and texture features. The mathematical expression is as follows: （8） wherein is a gradient of a feature of the visible light image, is a gradient of a feature of the infrared image, is a gradient of a feature of the fused target.

6. The unmanned aerial vehicle power grid inspection method based on multi-modal vision of claim 3, wherein, The specific steps of defect detection in step 4) are as follows: 4.1) Adopting a deep learning based thermal imaging defect detector to perform processing on the fused thermal imaging data (the processing procedure preserves the original temperature values and inherits both the infrared thermal radiation features and the visible light structural features), and calculate the thermal anomaly probability of each pixel location by mathematical expression (9): P = 1 - exp(- (T - T0) / T0) （9） wherein, is a thermal anomaly probability value at image coordinates , , is a activation function, is a learnable weight vector, is a feature extraction function, is a fused infrared feature tensor at coordinates ; 4.2) First, use 4-layer progressive down-sampling to extract multi-scale features, and then use two transposed convolutions to realize 4 times super-resolution reconstruction, recover the details of the defect area to improve the positioning accuracy; for the detected potential defect area, use mathematical expression (10) for judgment, filter invalid small area interference: （10） wherein, is the final determined defect region binary image, is a morphological opening operation, and 0.5 is a probability threshold value, is the area of a connected region, is a minimum defect area threshold value, and a default value is 10 pixels, which can be flexibly adjusted according to the power inspection scene. 4.3) Use the method of combining dynamic temperature threshold with CNN probability map to perform spatial consistency verification on the defect area, the specific implementation process is as follows: 4.3.1) Calculate the difference between the highest temperature in the detection area and the average temperature in the background area to evaluate the degree of temperature anomaly, the mathematical expression of the temperature anomaly index is shown in equation (11): （11） wherein, is the temperature anomaly index value, is the highest temperature value in the detection area, is the average temperature of the background area; 4.3.2) Based on the principle of dynamically calculating the temperature anomaly threshold, the mathematical expression of the dynamic threshold is shown as equation (12): temperature anomaly threshold, the mathematical expression of the dynamic threshold is shown as equation (12): （12） wherein is a dynamic temperature threshold, is a mean value of the temperature distribution of the detection area, is a standard deviation of the temperature distribution of the detection area; 4.3.3) Perform logical AND operation between pixel-level CNN output probability map and temperature anomaly detection result, ensure that the defect area meets the conditions of "temperature anomaly index exceeding dynamic threshold" and "CNN prediction probability greater than 0.5" at the same time, the mathematical expression of the judgment is shown in equation (13): （13） wherein, is the final defect detection result binary image, is the hot anomaly probability map output by the CNN network, and the pixel value range of the hot anomaly probability map is ; 4.4) In the defect detection process, use multi-task joint optimization strategy (the optimization goal is to improve the accuracy of defect detection and positioning accuracy), realize optimization by weighted combination of two key loss functions, the mathematical expression is shown in equation (14): （14） wherein, is the L2 norm, constraining the intensity consistency of the detection results; is the L2 norm, constraining the intensity consistency of the detection results; is the structural similarity loss, capturing the structural features of the image by computing the mean, variance and covariance within a local window, and the core computational relationship is: （15） wherein respectively the mean of the predicted image, the real image, respectively the variance of the predicted image, the real image, is the covariance of the predicted image and the real image, values and weights 0.3 determined with reference to the prior art.

Images