Power system line patrol detection method based on multi-source image, program product and unmanned aerial vehicle

Abstract

The invention belongs to the field of electric power safety, and particularly relates to an electric power system line patrol detection method based on a multi-source image, a program product and an unmanned aerial vehicle. The method comprises the following steps: 1, acquiring a multi-source image containing RGB, infrared and SAR images synchronously acquired by a binocular camera and a millimeter wave radar in a line patrol stage of the power unmanned aerial vehicle; and 2, synchronously inputting the multi-source image into a pre-trained target detection network, and generating a segmentation result by the target detection network. Wherein the target detection network comprises a feature extraction module, a feature enhancement module, a feature fusion module and a decoder. The feature extraction module comprises a local decoder and a global decoder; the local decoder comprises two branches which are respectively used for extracting feature information in the RGB image and the infrared image; the decoding layer is in jump connection with a corresponding layer of the local decoder, and the decoder is used for outputting a corresponding segmented image. The problems that the target segmentation difficulty in the unmanned aerial vehicle power line patrol image is large and the robustness of an existing scheme is poor are solved.

Description

Technical Field

[0001] The present invention belongs to the field of electric power safety, and in particular relates to a power system line inspection detection method based on multi-source images, a program product and a drone. Background Art

[0002] With economic development and the rapid increase in electricity demand, the number of power grids is increasing, and their coverage is expanding. Large-scale, decentralized transmission networks bear the heavy responsibility of power distribution. Because transmission line equipment is exposed to the elements for extended periods, it is subject to constant mechanical stress, lightning flashovers, material aging, icing, and human factors, leading to tower collapses, broken strands, wear, corrosion, and wobbling. If not addressed promptly, these problems can seriously impact grid operation and power supply. Furthermore, human activities can cause fishing line, kites, and other debris to come into contact with power lines, and small animals and birds can land on lines or equipment, all of which can pose a threat to the smooth operation of the power grid.

[0003] To eliminate these hidden dangers, operations and maintenance personnel typically conduct inspections of power line equipment and related electrical equipment. However, with the continuous expansion of power grids, traditional manual line inspection methods are not only labor-intensive, complex, rigorous, time-consuming, and risky, especially for inspecting transmission lines in mountainous areas, across large rivers, and across complex terrain. This is particularly true for tasks such as inspecting lines in complex terrain, such as during ice storms, floods, earthquakes, landslides, and at night. In recent years, with the continued advancement of drone technology, this new equipment is gradually replacing manual inspections of power lines under complex conditions. In different operation and maintenance areas, power line inspection drones can patrol lines along pre-defined routes or under manual control. Compared to traditional manual inspection methods, drone-based power line inspections offer the advantages of being faster, more convenient, and safer. Furthermore, power line inspection drones can perform more specialized inspection tasks, such as using visible light equipment to inspect lines and their ancillary facilities; inspecting channel conditions, towers, foundations, insulators, and more; and using radar and infrared instruments for specialized inspections.

[0004] During power line safety inspections, image data captured by drone patrols typically needs to be transmitted to a backend for review by technicians or assisted by machine vision-based artificial intelligence algorithms for fault detection and identification. For example, detecting foreign objects on power lines can now be accomplished with intelligent detection solutions based on image segmentation and object recognition, significantly reducing the workload for technicians. However, due to the high altitudes of patrol drones, foreign objects on power lines appear small in the images, making them difficult to identify. Furthermore, the complex backgrounds in images captured by drone patrols in the field complicate the resolution of targets. Furthermore, image quality issues caused by complex weather conditions can complicate image segmentation. These issues ultimately result in insufficient segmentation accuracy, reliability, and robustness in traditional image segmentation algorithms used in drone-based power line inspections. Summary of the Invention

[0005] In order to solve the problems of difficulty in target segmentation in UAV power line inspection images, insufficient accuracy and reliability, and poor robustness of existing segmentation schemes, the present invention provides a power system line inspection detection method based on multi-source images, a program product, and a UAV.

[0006] The present invention is achieved by adopting the following technical solutions:

[0007] A multi-source image-based power system line inspection method is used to identify targets on the line based on input multi-source image data and generate corresponding segmented images. The method includes:

[0008] 1. Acquire multi-source image data collected synchronously by the binocular camera and millimeter-wave radar during the power line inspection phase of the UAV, and preprocess it to obtain RGB images, infrared images, and SAR images of a specified depth.

[0009] Second, the RGB image, infrared image, and SAR image are simultaneously input into a pre-trained target detection network, which generates segmentation results. The target detection network includes a feature extraction module, a feature enhancement module, a feature fusion module, and a decoder.

[0010] Among them, the feature extraction module includes a local decoder and a global decoder; the global encoder uses a three-layer Res-Block to extract the features of the input SAR image, and then obtains the global feature Gf of the corresponding layer. i(i=1, 2, 3). The local decoder contains two branches for extracting feature information from RGB images and infrared images respectively. Each branch contains four Tansfomer coding layers with multi-head attention. The outputs of the two branches in each Tansfomer coding layer are processed by a fusion unit including a deep convolution and a linear layer, and then residually connected with the output of the corresponding branch of the RGB image to serve as the local feature Lf of the current layer. i (i=1, 2, 3, 4). The feature enhancement module is used to input the corresponding level Lf1, Gf i and Lf i+1 Generate pre-segmentation map Ps i (i=1, 2, 3); Feature fusion module for Ps i and Gf i Perform feature fusion to obtain fusion feature Ff i (i=1, 2, 3). The decoder consists of three decoding layers and a bilinear interpolation layer; the decoding layer is connected to the corresponding layer of the local decoder by jump; the decoder will i+1 After upsampling, it is combined with Ff i Perform feature splicing to generate decoding features D (i=1, 2, 3); after upsampling the decoding features D1 through a bilinear interpolation layer, a segmented image P of the target contained in the line is obtained.

[0011] As a further improvement of the present invention, the data processing process of the feature enhancement module is as follows:

[0012] (1) The local features Lf1 of the first layer are processed by the channel attention module to obtain the edge feature map Fc. (2) The local features Lf1 of the subsequent layers are processed by the convolution attention module. i+1 (i=1, 2, 3) are processed to obtain the first attention map Lp i (3) By parallelizing patch-aware attention to the full local feature Gf i Process and get the second attention map Pf i (4) Gf i After binarization and Reverse, i Multiply pixel by pixel with Fc, and the result is the same as Fc i The output of the convolution module is processed by the multi-layer perceptron. The features before and after MLP processing are multiplied pixel by pixel and then added to Pf i Perform pixel-by-pixel addition to obtain the required pre-segmentation map Ps i .

[0013] As a further improvement of the present invention, the data processing process of the feature fusion module is as follows:

[0014] Gfi and Ps i The three outputs after processing by the BAM module and the dilated convolution module are input to the EMA module for processing. The output of the EMA module is combined with the output of the previous layer of the feature fusion module Ff. i+1 Perform pixel-by-pixel subtraction; the result is convolved and then added pixel-by-pixel with the output of the EMA module; and then the fusion feature Ff of the corresponding level is obtained i .

[0015] As a further improvement of the present invention, the data processing process of the Tansfomer coding layer includes: the input features are first processed by a multi-head attention module, and the input and output of the multi-head attention module are residually connected in the first Add&Layer Norm layer; the output of the first Add&Layer Norm layer is processed by a linear layer; the input and output of the linear layer are residually connected in the second Add&Layer Norm layer and then used as the output of the Tansfomer coding layer.

[0016] As a further improvement to the present invention, the multi-source image data includes RGB and infrared images collected by a binocular camera, as well as SAR images at a specified depth collected by a millimeter-wave radar. The area at the specified altitude needs to exclude the area between the drone's flight altitude and the height of the power facilities.

[0017] As a further improvement of the present invention, preprocessing of multi-source image data includes adjusting the images to the same size and performing range motion correction on the SAR images.

[0018] As a further improvement of the present invention, a method for performing range motion correction on a SAR image includes:

[0019] First, the received echo signal undergoes pulse compression in the range direction. Then, it undergoes a Fourier transform in the azimuth direction, transforming the signal into the range-Doppler domain. Next, a chirp accelerometer (CAL) factor is designed to focus signals at different Doppler frequencies onto the same range gate. This factor is multiplied by the signal in the frequency domain to achieve Chirp accelerometer correction. The signal undergoes quadratic phase compensation in the range direction, followed by an inverse Fourier transform in the range direction, transforming the signal into the time domain. The data undergoes matched filtering in the azimuth direction, followed by an inverse Fourier transform in the azimuth direction, to obtain the desired SAR image at the specified depth.

[0020] As a further improvement of the present invention, the training process of the target detection network is as follows:

[0021] S1: Synchronously collect a large amount of multi-source image data consisting of RGB images, infrared images, and SAR images, pre-process the multi-source image data, and then manually add label information representing the segmentation results.

[0022] S2: Obtain a large amount of multi-source image data containing label information as a sample dataset, expand the sample dataset through data augmentation methods, and then divide it into a training set, a validation set, and a test set.

[0023] S3: Perform multiple rounds of iterative training and verification on the target detection network using the training set and the validation set, and retain the parameters of several network models with better training effects.

[0024] Among them, the training stage adopts the joint loss including cross entropy loss and focal loss as the loss function Loss;

[0025]

[0026] In the above formula, G represents the true mask in the label information; L CE represents the cross entropy loss function, L focal represents the focal loss function; α and β represent the weights used to balance the two loss functions.

[0027] S4: Use the test set to test the retained network model, select the best network model based on the scores of each model on the four evaluation indicators of Recall, Specificity, Precision, and Accuracy, and save its corresponding model parameters.

[0028] As a further improvement of the present invention, in step S2, the data enhancement method of the sample data set includes performing linear grayscale transformation, histogram equalization, image sharpening, and image smoothing on the RGB image.

[0029] The present invention also includes a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the power system line inspection detection method based on multi-source images as described above, and further realizes the identification of targets contained in the line based on the input multi-source image data and generates corresponding segmented images.

[0030] The present invention also includes a line patrol drone comprising a body, a binocular camera, a millimeter-wave radar, and a detection module. The binocular camera and millimeter-wave radar are both mounted on the body. The binocular camera includes a natural light imaging module and an infrared imaging module, which are used to acquire RGB and infrared images of the area below the drone. The millimeter-wave radar module is used to scan the area below the drone and acquire SAR images at a specified depth below the drone.

[0031] The detection module includes a memory, a processor, and a computer program stored in the memory and running on the processor. The detection module receives synchronously acquired RGB images, infrared images, and SAR images as input and preprocesses the relevant image data. When the processor in the detection module executes the computer program, it implements the aforementioned multi-source image-based power system line inspection method, thereby identifying targets on the line and generating corresponding segmented images based on the synchronously acquired RGB images, infrared images, and SAR images input by the binocular camera and millimeter-wave radar.

[0032] The technical solution provided by the present invention has the following beneficial effects:

[0033] This method uses multi-source imagery collected by drones, including RGB images, infrared images, and radar-derived SAR images, to identify and segment power facilities within the images. A corresponding target detection network is designed. Compared to existing solutions, this method can extract richer fusion information from multi-source images to achieve more accurate segmentation results, avoiding the problem that single-source image segmentation is susceptible to interference from image quality and environmental factors. Compared to existing solutions, the power facility image segmentation solution provided by this method, based on drone line inspection videos, has better generalization and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of the steps of the power system line inspection detection method based on multi-source images provided in Example 1 of the present invention.

[0035] Figure 2 This is a model architecture diagram of the target detection network constructed in Example 1 of the present invention.

[0036] Figure 3 Schematic diagram of the global encoder used in the target detection network.

[0037] Figure 4 Schematic diagram of the dual-branch local encoder used in the target detection network.

[0038] Figure 5 Schematic diagram of the Tansfomer coding layer used in the local encoder.

[0039] Figure 6 Schematic diagram of the feature enhancement module used in the target detection network.

[0040] Figure 7 Schematic diagram of the feature fusion module used in the target detection network.

[0041] Figure 8 This is a flowchart of the steps of the target detection network training method provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] Example 1

[0044] The drone-based intelligent power grid inspection mission consists of two phases: the first phase uses drones to collect on-site images, and the second phase uses artificial intelligence algorithms to process these images and then detect whether power facilities have stored faults based on the information contained in the images. Currently, in the first phase, drones can completely replace manual on-site image collection. However, in the second phase, the accuracy of automatic identification of hidden dangers in power facilities based on artificial intelligence algorithms still needs to be improved. The defect detection rate is not meeting expectations, and it can only be used as an auxiliary means to manual analysis.

[0045] To improve the automation level of hidden danger and defect identification, we must first improve the standards and quality of inspection data and results, and build a massive defect sample. Secondly, we need to develop more accurate and efficient hidden danger and defect identification algorithms, vigorously promoting hidden danger and defect judgment based on image recognition and intelligent defect diagnosis based on algorithms such as artificial neural networks. Finally, leveraging the rapid development of AI technology and the increase in big data computing power, we can achieve full automation and intelligentization of power drone inspections.

[0046] The AI algorithm used to identify defects and hidden dangers in power facilities consists of two main components. The first is to quickly identify target power facilities for inspection, such as high-voltage lines, towers, transformers, power cabinets, and power boxes, from the large number of images captured by drones, and then segment them. The second component combines these segmented images of power facilities for fault detection, identifying various defects such as: bird nests, foreign objects hanging on towers, rust, missing or loose parts; insulator flashover, breakage, spontaneous explosion, lightning strike, flashover, and zero-value porcelain insulators; broken or loose ground wire strands, foreign objects hanging on them, rust, crossings, and insufficient spacing; broken or corroded grounding devices, damaged grounding resistance, and insufficient grounding area; heated tension jumper clamps, flat bolt caps, missing nuts, misaligned spacers, worn hardware, missing screws, missing screws, and rusted hardware; skewed towers, rusted or missing tower signs, faded signage, and missing or inadequate bird protection devices.

[0047] On this basis, this embodiment provides a power system line inspection detection method based on multi-source images, which is used to identify targets contained on the line based on the input multi-source image data and generate corresponding segmented images. This solution can accurately identify and segment the power facilities contained in the captured video stream data based on the multi-source image data containing visible light imaging, infrared imaging and radar imaging collected synchronously by the drone. In the solution provided in this embodiment, a new target detection network is constructed in combination with the new data type. The network model can use the depth-direction obstacle contour information obtained by the millimeter-wave radar to guide the model to identify obstacles more efficiently, and extract and fuse the feature information contained in the three types of images at multiple levels, thereby improving the accuracy of the final target recognition and segmentation.

[0048] Specifically, if Figure 1 As shown, the power system line inspection detection method based on multi-source images provided in this embodiment includes the following steps:

[0049] 1. Acquire multi-source image data collected synchronously by the binocular camera and millimeter-wave radar during the power line inspection phase of the UAV, and preprocess it to obtain RGB images, infrared images, and SAR images of a specified depth.

[0050] In this embodiment, to address the difficulty in detecting and segmenting targets in existing RGB image-based power line inspection videos, this embodiment further introduces infrared images collected by infrared imaging equipment and radar images collected by millimeter-wave radar, thereby achieving more accurate target recognition and segmentation through the integration of multimodal data. In practical applications, preprocessing of multi-source image data includes adjusting the images to the same size.

[0051] Unlike RGB images obtained using natural light imaging technology, the infrared images used in this embodiment can display the temperature distribution of an object's surface, which is very useful for detecting the boundary between the edge of the device and the surrounding background. Infrared images can also help detect defects that are not visible in RGB images. Infrared images are also very effective in identifying defects and aging faults in power line equipment.

[0052] The SAR image in this embodiment is generated based on data obtained by a drone scanning downward from above the patrol area. Specifically, in practical applications, this embodiment uses image information within the plane where the power equipment resides to better capture the outline of the power facility, guiding the network model for more accurate target recognition. Therefore, in this embodiment, the millimeter-wave radar is primarily used to collect SAR images at a specified depth. This specified altitude must exclude the area between the drone's flight altitude and the height of the power facility.

[0053] In this embodiment, in order to improve the quality of the SAR image, it is also necessary to perform range motion correction on the SAR image acquired at a specified depth. The method of performing range motion correction on the SAR image in this embodiment includes:

[0054] First, the received echo signal undergoes pulse compression in the range direction. Then, it undergoes a Fourier transform in the azimuth direction, transforming the signal into the range-Doppler domain. Next, a chirp accelerometer (CAL) factor is designed to focus signals at different Doppler frequencies onto the same range gate. This factor is multiplied by the signal in the frequency domain to achieve Chirp accelerometer correction. The signal undergoes quadratic phase compensation in the range direction, followed by an inverse Fourier transform in the range direction, transforming the signal into the time domain. The data undergoes matched filtering in the azimuth direction, followed by an inverse Fourier transform in the azimuth direction, to obtain the desired SAR image at the specified depth.

[0055] Second, the RGB image, infrared image and SAR image are simultaneously input into a pre-trained target detection network, and the target detection network generates the segmentation result.

[0056] Specifically, if Figure 2 As shown in FIG, the target detection network constructed in this embodiment includes a feature extraction module, a feature enhancement module, a feature fusion module and a decoder. The feature extraction module in the target detection network of this embodiment includes a global decoder and a local decoder. The global encoder is used to extract the features of the input SAR image and obtain the global feature Gf of the corresponding layer. i(i=1, 2, 3). The local decoder contains two branches for extracting feature information from RGB images and infrared images respectively. The two branches extract feature information from RGB images and infrared images respectively, and perform preliminary fusion of the two features to obtain the local feature Lf i (i=1, 2, 3, 4). The feature enhancement module is used to extract the Lf1 and Gf of the corresponding level according to the input. i and Lf i+1 Generate pre-segmentation map Ps i (i=1, 2, 3); Feature fusion module for Ps i and Gf i Perform feature fusion to obtain fusion feature Ff i (i=1, 2, 3); The decoder and the local encoder form a U-shaped network, and the decoder is used to combine the fusion feature Ff i (i=1, 2, 3) Perform multi-level upsampling on the local features Lf4 inputted by the last layer of the local encoder to obtain the segmented image P of the target contained in the line.

[0057] To make the functions and principles of the target detection network provided by this embodiment clearer, each module in the network model is described in detail below:

[0058] 2.1 Global Encoder

[0059] In this embodiment, if Figure 3 As shown in the figure, the global encoder uses a 3-layer Res-Block to downsample the input SAR image. The global encoder extracts features from the preprocessed SAR image to obtain the global features Gfi (i=1, 2, 3) in the corresponding layer containing the target contour information. Each Res-Block in the local encoder consists of two convolutional layers and a skip connection. In this embodiment, each convolutional layer contains two 3×3 convolutional layers, each followed by a ReLU activation function, where the convolutional layer is responsible for feature extraction; the skip connection directly connects the input to the output of the convolutional layer and performs element-level addition operations, thereby enabling the network model to learn the identity mapping and alleviate the possible vanishing gradient problem.

[0060] 2.2 Local Encoder

[0061] like Figure 4As shown, the local decoder provided in this embodiment includes two branches for extracting feature information from RGB images and infrared images respectively, and each branch includes 4 Tansfomer coding layers with multi-head attention. One of the branches is the RGB branch, which is used to perform multi-level downsampling of the input RGB image through 4 Tansfomer coding layers, thereby obtaining comprehensive features containing rich color and texture information; the other branch is the RGB branch, which is used to perform multi-level downsampling of the input RGB image through 4 Tansfomer coding layers, thereby obtaining information containing shape features and other deep features. Among them, in the local decoder of this embodiment, the outputs of the two branches in each Tansfomer coding layer are processed by a fusion unit (Fusion) including a deep convolution and a linear layer, and then residually connected with the output of the RGB branch to serve as the local feature Lf of the current layer. i (i=1, 2, 3, 4).

[0062] The Transformer encoding layer used in the local encoder of this embodiment can gradually extract the semantic information of the input features through multi-layer stacked encoding blocks, and can simultaneously extract shallow features and deep semantic relationships; thereby enabling the network model to capture rich contextual information and provide powerful semantic features for subsequent segmentation tasks. The architecture of the Tansfomer encoding layer used in this embodiment is as follows: Figure 5 As shown in the figure, it includes 1 multi-head attention layer (Multi-HeadAttention), 1 linear layer (Linear), and 2 residual connections and layer normalization (Add&LayerNorm). In detail, the data processing process of the Tansfomer encoding layer is as follows:

[0063] The input features are first processed by the multi-head attention module, and the input and output of the multi-head attention module are residually connected in the first Add&Layer Norm layer; the output of the first Add&Layer Norm layer is processed by the linear layer; the input and output of the linear layer are residually connected in the second Add&Layer Norm layer and then used as the output of the Tansfomer encoding layer.

[0064] 2.3 Feature Enhancement Module

[0065] In this embodiment, the channel attention module (CAM Block), convolutional attention module (CBAM), parallelized patch perception attention (PPA) and multi-layer perceptron (MLP) can be used to extract global features from the preprocessed SAR image of a specified depth and local features extracted by the local encoder to perform pre-segmentation at different levels, and obtain the corresponding pre-segmentation map Psi (i=1, 2, 3).

[0066] Specifically, if Figure 6 As shown, the feature enhancement module used in this embodiment consists of a channel attention module, a convolutional attention module, a convolution module, a parallel plate perception attention module, and a multi-layer perception machine; its data processing process is as follows:

[0067] (1) The local features Lf1 of the first layer are processed by the channel attention module (CAM) to obtain the edge feature map Fc. (2) The local features Lf1 of the subsequent layers are processed by the convolutional attention module (CBAM). i+1 (i=1, 2, 3) are processed to obtain the first attention map Lp i (3) Parallelizing Patch-Perceptual Attention (PPA) on the full local feature Gf i Process and get the second attention map Pf i (4) Gf i After binarization and Reverse, i Multiply pixel by pixel with Fc, and the result is the same as Fc i The output of the convolution module is processed by a multi-layer perceptron (MLP). The features before and after MLP processing are multiplied pixel by pixel and then added to Pf i Perform pixel-by-pixel addition to obtain the required pre-segmentation map Ps i .

[0068] 2.4 Feature Fusion Module

[0069] This embodiment uses two encoders to extract a large amount of feature information at different levels from different types of input images. Among them, the low-level features have higher resolution and contain more location and detail information of the target, but this part of the information has lower semantics and more noise. Features at higher levels have stronger semantic information, but lower resolution and poorer perception of details. This embodiment uses a feature fusion module to efficiently fuse different types of features to improve the segmentation performance of the network model. Specifically, the feature fusion module of this embodiment performs a high-efficiency fusion on Ps i and Gf i Perform feature fusion to obtain fusion feature Ff i (i=1, 2, 3).

[0070] In detail, such as Figure 7As shown, the feature fusion module used in this embodiment consists of a multimodal fusion module CMF (Cross-Modal Fusion block), a BAM attention module, two dilated convolutions with different dilation rates, a cross-space efficient multi-scale attention module EMA, and a 3×3 convolution. The data processing process of the feature fusion module is as follows:

[0071] Gf i and Ps i The three outputs after processing by the BAM module and the dilated convolution module are input to the EMA module for processing. The output of the EMA module is combined with the output Ff of the previous layer of the feature fusion module. i+1 Perform pixel-by-pixel subtraction; the result is convolved and then added pixel-by-pixel with the output of the EMA module; and then the fusion feature Ff of the corresponding level is obtained i .

[0072] 2.5 Decoder

[0073] The decoder consists of three decoding layers and a bilinear interpolation layer (BIU); the decoding layer is connected to the corresponding layer of the local decoder by jump; the decoder will i+1 After upsampling, it is then compared with Ff i Feature concatenation is performed to generate decoded features D (i=1, 2, 3); decoded features D1 are upsampled by a bilinear interpolation layer to obtain a segmented image P of the target contained in the line. In this embodiment, the decoding layer can adopt the decoding layer of the Transformer, and each layer of the decoder includes a feedforward network and a self-attention layer.

[0074] The target detection network provided in this embodiment needs to be trained before actual application. Among them, some modules in the local encoder and the global encoder can directly adopt the pre-trained models. Therefore, it is only necessary to use a high-quality sample input data set to fine-tune or transfer learning the overall target detection network.

[0075] Specifically, if Figure 8 As shown, the training process of the target detection network provided in this embodiment includes the following steps:

[0076] S1: Synchronously collect a large amount of multi-source image data consisting of RGB images, infrared images, and SAR images, pre-process the multi-source image data, and then manually add label information representing the segmentation results.

[0077] S2: Obtain a large amount of multi-source image data containing label information as a sample dataset, expand the sample dataset through data augmentation methods, and then divide it into a training set, a validation set, and a test set.

[0078] In this embodiment, to improve the generalization and robustness of the model and reduce overfitting, data augmentation is performed on the obtained sample dataset. Specifically, the sample data in this embodiment includes interrelated RGB images, infrared images, and SAR images. The data augmentation operation in this embodiment is primarily targeted at RGB images. For example, after performing data augmentation on an RGB image in the same sample dataset and then combining it with the original RGB image, new sample data can be obtained, where the new sample data still retains the label information of the original sample data.

[0079] In practical applications, the data enhancement methods for the sample datasets in this implementation include linear grayscale transformation, histogram equalization, image sharpening, and image smoothing on RGB images. Linear grayscale transformation enhances high-frequency regions of an image by scaling its grayscale values or increasing its contrast, making it appear clearer. Histogram equalization begins with the image's grayscale graph, creating a grayscale histogram. The histogram then counts the number of times each grayscale value appears in the histogram, and the grayscale values of corresponding points in the grayscale graph are recorded in the histogram. The histogram is then equalized to make the grayscale values of the pixels more evenly distributed, thereby enhancing the overall contrast and clarity of the image. Image sharpening can compensate for image contours, enhance edges, and areas with grayscale transitions, making the image clearer. It can also highlight the edges, contours, or features of certain linear features in the image. Image smoothing is based on averaging or filtering pixel values within the image. Averaging the pixels in the image can reduce the effects of noise and create a smoother image. In this embodiment, the image may be balanced by using a mean filter, a median filter, or a Gaussian filter.

[0080] S3: Perform multiple rounds of iterative training and verification on the target detection network using the training set and the validation set, and retain the parameters of several network models with better training effects.

[0081] Among them, the training stage adopts the joint loss including cross entropy loss and focal loss as the loss function Loss;

[0082]

[0083] In the above formula, G represents the true mask in the label information; L CE represents the cross entropy loss function, L focalrepresents the focal loss function; α and β represent the weights used to balance the two loss functions.

[0084] S4: Use the test set to test the retained network model, select the best network model based on the scores of each model on the four evaluation indicators of Recall, Specificity, Precision, and Accuracy, and save its corresponding model parameters.

[0085] Example 2

[0086] Example 1 provides a technical solution for identifying and segmenting power facilities or equipment in image data collected during drone inspections based on multi-source image data. In practical applications, this method is primarily implemented by executing corresponding computer program code. Therefore, based on the solution in Example 1, this embodiment further provides a computer program product comprising a computer program. When executed by a processor, the computer program implements the aforementioned multi-source image-based power system line inspection detection method, thereby identifying targets contained on the line based on the input multi-source image data and generating corresponding segmented images.

[0087] Based on the solution in Example 1, this embodiment further provides a storage medium, which includes a computer program. When the computer program is executed by a processor, it implements the power system line inspection detection method based on multi-source images as in Example 1, and further realizes the identification of targets contained in the line based on the input multi-source image data and generates corresponding segmented images.

[0088] Example 3

[0089] Based on the solution in Example 1, this embodiment further provides a multi-source image-based power system line inspection device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it employs the multi-source image-based power system line inspection method described in Example 1 to identify targets on the line based on input multi-source image data and generate corresponding segmented images.

[0090] The multi-source image-based power system line inspection and detection device provided in this embodiment is essentially a computer device. In practical applications, this computer device can be implemented as an embedded module and deployed on a power line inspection drone. Alternatively, it can be implemented as a standalone computer device and deployed in a power grid control center. In practical applications, this computer device can be implemented as a medium-to-large computer device such as a laptop, tablet computer, desktop computer, or a rack-mounted server, blade server, tower server, or cabinet server (including standalone servers or server clusters consisting of multiple servers) capable of executing computer programs.

[0091] In the two different deployment methods described above, in the former, the drone can simultaneously identify the power equipment targets contained in the image while capturing it, complete image segmentation, and transmit it to the data center at the rear. In the latter, the drone needs to capture images from the front and transmit them to the data center at the rear. The server in the data center then uses the solution in Example 1 to identify the target disease and complete image segmentation. Various fault detection is then performed based on the segmented equipment images.

[0092] Regardless of the deployment scheme, in this embodiment, the computer devices used include at least, but are not limited to, memory and processors that can be interconnected via a system bus. In this embodiment, memory (i.e., readable storage media) includes flash memory, hard disks, multimedia cards, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disks, optical disks, and the like. In some embodiments, the memory can be an internal storage unit of the computer device, such as the computer device's hard disk or internal memory. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like. Of course, the memory can also include both the internal storage unit of the computer device and its external storage devices. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that has been output or is about to be output.

[0093] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of a computer device.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0095] Example 4

[0096] Based on the solution in Example 3, this embodiment also includes a line patrol drone, which includes a body, a binocular camera, a millimeter-wave radar, and a detection module. The binocular camera and millimeter-wave radar are both mounted on the body. The binocular camera includes a natural light imaging module and an infrared imaging module, which are used to obtain RGB and infrared images of the area below the drone. The millimeter-wave radar module is used to scan the area below the drone and obtain SAR images at a specified depth below the drone.

[0097] The detection module includes a memory, a processor, and a computer program stored in the memory and running on the processor. The detection module receives synchronously acquired RGB images, infrared images, and SAR images as input and preprocesses the relevant image data. When the processor in the detection module executes the computer program, it implements the multi-source image-based power system line inspection detection method described in Example 1, thereby identifying targets on the line and generating corresponding segmented images based on the synchronously acquired RGB images, infrared images, and SAR images input by the binocular camera and millimeter-wave radar.

[0098] Performance Testing

[0099] To verify the effectiveness of the multi-source image-based power system line inspection method provided by this invention, researchers conducted an experiment using the invention as an experimental group and a traditional U-Net-based image segmentation algorithm as a control group, comparing their performance.

[0100] This experiment used a self-made dataset of 1,360 power line inspection video images as a sample dataset for model training. The control group's network model was trained solely on RGB images from the sample dataset, while the experimental group used a multi-source imagery scheme, including RGB, infrared, and SAR images. The sample dataset was divided into training, validation, and test sets with an 8:1:1 data size ratio. After each training round, 10% of the validation data was immediately used for testing. After 400 rounds of training, the best model was saved.

[0101] During training, the experimental group first loaded the pretrained model. Then, during training, parameters such as batch size and learning rate were adjusted until the model converged. During training, the model's final output and the output of the feature enhancement module were monitored, and the corresponding losses were calculated.

[0102] This experiment uses four evaluation indicators: Recall, Specificity, Precision, and Accuracy to compare the model performance of the experimental group and the control group. The experimental results are as follows:

[0103] Table 1: Performance test results of different schemes in the control experiment

[0104]

[0105] By analyzing the above experimental data, it can be found that the proposed solution is superior to the traditional U-Net solution in all indicators, and the segmentation performance is very outstanding.

[0106] Furthermore, in order to evaluate the contribution of the newly designed dual-branch local feature extraction module, feature enhancement module, and feature fusion module used in the target detection network of the present invention to network performance, technicians developed the following ablation experiment. The ablation experiment tested the model performance without the above modules. The experimental results are shown in the following table:

[0107] Table 2: Performance test results of different schemes in ablation experiments

[0108]

[0109] Analysis of the above experimental data shows that the model performance of the target detection network without the three modules is significantly inferior to that of the complete solution, which verifies the performance contribution brought by the new design provided by the present invention.

Claims

1. A power system line inspection detection method based on multi-source images, characterized in that: It is used to identify the targets contained in the line according to the input multi-source image data and generate the corresponding segmented image; It includes: Acquire multi-source image data collected synchronously by binocular cameras and millimeter-wave radar during the power line inspection phase of the UAV, and preprocess it to obtain RGB images, infrared images, and SAR images at a specified depth; The RGB image, infrared image and SAR image are synchronously input into a pre-trained target detection network, and the target detection network generates the segmentation result; the target detection network includes a feature extraction module, a feature enhancement module, a feature fusion module and a decoder; the feature extraction module includes a local decoder and a global decoder; the global encoder uses a 3-layer Res-Block to extract the features of the input SAR image, and then obtains the global feature Gf of the corresponding layer i (i=1, 2, 3); The local decoder contains two branches for extracting feature information from RGB images and infrared images respectively. Each branch contains four Tansfomer coding layers with multi-head attention. The outputs of the two branches in each Tansfomer coding layer are processed by a fusion unit including a deep convolution and a linear layer, and then residually connected with the output of the corresponding branch of the RGB image as the local feature Lf of the current layer. i (i=1, 2, 3, 4); The feature enhancement module is used to input the corresponding level Lf1, Gf i and Lf i+1 Generate pre-segmentation map Ps i (i=1, 2, 3); Feature fusion module for Ps i and Gf i Perform feature fusion to obtain fusion feature Ff i (i=1, 2, 3); The decoder consists of three decoding layers and a bilinear interpolation layer; The decoding layer is jump-connected to the corresponding layer of the local decoder; The decoder will i+1 After upsampling, it is then compared with Ff i Feature splicing is performed to finally generate decoding features D (i=1, 2, 3); the decoding features D1 are upsampled by a bilinear interpolation layer to obtain a segmented image P of the target contained in the line.

2. The power system line inspection detection method based on multi-source images according to claim 1, characterized in that: The data processing process of the feature enhancement module is as follows: The local features Lf1 of the first layer are processed by the channel attention module to obtain the edge feature map Fc; the local features Lf of subsequent layers are processed by the convolution attention module i+1 (i=1, 2, 3) are processed to obtain the first attention map Lp i ; Parallelized patch-aware attention (PPA) for the full local feature Gf i Process and get the second attention map Pf i ; Gf i After binarization and Reverse, i Multiply pixel by pixel with Fc, and the result is the same as Fc i The output of the convolution module is processed by the multi-layer perceptron. The features before and after MLP processing are multiplied pixel by pixel and then added to Pf i Perform pixel-by-pixel addition to obtain the required pre-segmentation map Ps i .

3. The power system line inspection detection method based on multi-source images according to claim 2, characterized in that: The data processing process of the feature fusion module is as follows: Gf i and Ps i The data are inputted into the multimodal fusion module for feature information fusion; The result is processed by 3*3 convolution and divided into three paths. The first path is processed by BAM module, the second path is processed by the first dilated convolution module, and the third path is processed by the second dilated convolution module after being concatenated with the output of the first dilated convolution module according to the channel. The dilation rates of the two dilated convolution modules are different. The three-path output is input to the EMA module for processing. The output of the EMA module is combined with the output Ff of the previous layer of the feature fusion module. i+1 Perform pixel-by-pixel subtraction; the result is convolved and then added pixel-by-pixel with the output of the EMA module; and then the fusion feature Ff of the corresponding level is obtained i .

4. The power system line inspection detection method based on multi-source images according to claim 3, characterized in that: The data processing process of the Tansfomer coding layer includes: the input features are first processed by the multi-head attention module, and the input and output of the multi-head attention module are residually connected in the first Add&Layer Norm layer; the output of the first Add&Layer Norm layer is processed by the linear layer; the input and output of the linear layer are residually connected in the second Add&Layer Norm layer and then used as the output of the Tansfomer coding layer.

5. The power system line inspection detection method based on multi-source images according to claim 4, characterized in that: The multi-source image data includes RGB images and infrared images collected by a binocular camera, and SAR images of a specified depth collected by a millimeter-wave radar; the area corresponding to the specified height needs to exclude the area between the flight altitude of the drone and the height of the power facilities; and / or The preprocessing of the multi-source image data includes adjusting the images to the same size and performing range motion correction on the SAR images.

6. The power system line inspection detection method based on multi-source images according to claim 5, characterized in that: The method for performing range motion correction on the SAR image comprises: First, the received echo signal is pulse compressed in the range direction; then, it is Fourier transformed in the azimuth direction to transform the signal into the range Doppler domain; a Chirp Ccaling factor is designed to focus signals at different Doppler frequencies onto the same range gate, and the Chirp Ccaling factor is multiplied by the signal in the frequency domain to achieve Chirp Ccaling correction; the signal is subjected to quadratic phase compensation in the range direction and then an inverse Fourier transform in the range direction to transform the signal into the time domain; the data is matched filtered in the azimuth direction and then an inverse Fourier transform in the azimuth direction to obtain the required SAR image at the specified depth.

7. The power system line inspection detection method based on multi-source images according to claim 1, characterized in that: The training process of the target detection network is as follows: S1: synchronously collect a large amount of multi-source image data consisting of RGB images, infrared images and SAR images, pre-process the multi-source image data and then manually add label information representing the segmentation results; S2: Obtain a large amount of multi-source image data containing label information as a sample dataset, expand the sample dataset through data augmentation, and then divide it into training set, validation set, and test set; S3: Perform multiple rounds of iterative training and verification on the target detection network using the training set and the validation set, and retain the parameters of several network models with better training results; Among them, the training stage adopts the joint loss including cross entropy loss and focal loss as the loss function Loss; In the above formula, G represents the true mask in the label information; L CE represents the cross entropy loss function, L focal represents the focal loss function; α and β represent the weights used to balance the two loss functions; S4: Use the test set to test the retained network model, select the best network model based on the scores of each model on the four evaluation indicators of Recall, Specificity, Precision, and Accuracy, and save its corresponding model parameters.

8. The power system line inspection detection method based on multi-source images according to claim 7, characterized in that: In step S2, the data enhancement method of the sample data set includes performing linear grayscale transformation, histogram equalization, image sharpening, and image smoothing on the RGB image.

9. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, it implements the power system line inspection detection method based on multi-source images as described in any one of claims 1 to 8, and further realizes the identification of targets contained in the line according to the input multi-source image data and the generation of corresponding segmented images.

10. A line patrol drone, comprising a body, characterized in that: It also includes: A binocular camera, which is mounted on the drone and used to obtain RGB images and infrared images of the area below the drone; A millimeter-wave radar module, which is mounted on the aircraft and is used to acquire SAR images at a specified depth below the drone; and A detection module, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the method for detecting power system line patrol based on multi-source images according to any one of claims 1 to 8 is implemented, thereby identifying targets contained on the line and generating corresponding segmented images based on RGB images, infrared images, and SAR images synchronously collected by a binocular camera and a millimeter-wave radar input.

Images