A power equipment target detection method in a complex environment

Abstract

The application provides a power equipment target detection method in a complex environment, which comprises the following steps: 1) acquiring a visible light image and an infrared image of electrical equipment; 2) obtaining edge information of the infrared image and the visible light image by using an optimization threshold O-Canny edge detection algorithm combined with gradient values, and detecting the curvature of the edge information by using a curvature scale space technology to obtain feature points of the infrared image and the visible light image; 3) completing feature point matching of the infrared image and the visible light image by using a GSIFT algorithm; 4) screening the feature point matching result by using a random sample consensus algorithm; 5) fusing and registering the infrared image and the visible light image by using a two-layer ResNet deep learning network; 6) marking electrical equipment in the infrared image, the visible light image and a fused image respectively by using labelImg software; 7) training YOLOv5 by using the data set with labels, and then respectively performing visible light image target detection, infrared image target detection and fused image target detection to obtain target detection accuracies of the three groups of experiments respectively.

Description

Technical Field

[0001] This invention relates to the field of power equipment target detection technology, and more specifically to a method for power equipment target detection in complex environments. Background Technology

[0002] The operational status of power equipment often determines the safety and stability of the entire power system. With the development of artificial intelligence, a large number of intelligent detection devices have been deployed in the safe operation and maintenance of power equipment. Multimodal sensor patrol robots and drones capture and upload images of operating power equipment for status monitoring and fault diagnosis. In complex environments such as low light and rain / fog, the specific details of the visible light images of power equipment transmitted by robots during inspection cannot be displayed, affecting image detection. In complex environments, infrared images are a good alternative to visible light images. Combining infrared and visible light images can effectively utilize the target specification characteristics of infrared images and the rich scene details of visible light images, thereby enhancing image quality and improving the ability to detect and recognize targets in complex environments. When registering infrared and visible light images, there are many mismatched points due to the influence of objective factors such as nonlinear illumination and camera saturation. Key point feature vectors with excessively large gradient values ​​can affect the registration results.

[0003] Clear edges are crucial for subsequent image processing and the identification and analysis of power equipment. In complex substation environments, traditional power equipment image detection and recognition are significantly hampered, affecting the selection of feature points for subsequent registration and fusion. Traditional edge detection operators lack adaptability in threshold selection and fail to consider the influence of gradient values, leading to the loss of edge map details in complex environments and subsequent loss of feature information during registration and fusion.

[0004] Image fusion is the process of processing and combining two images from different observation environments into a new image. Many scholars have researched fusion algorithms for visible light and infrared images, which can be mainly categorized into two types: one is based on traditional algorithms for fusion of infrared and visible light images. This type of method first separates the images into base layers and detail layers, then formulates fusion strategies for each layer based on their respective characteristics, and finally reconstructs the fused base and detail layers to obtain the fused image. The other type is based on deep learning image fusion algorithms. This type of method uses deep learning networks to obtain features from infrared and visible light images, and then uses a feature fusion layer for fusion. These methods have two main problems: first, the performance of the fused image using traditional fusion methods largely depends on the fusion strategy, and the computation time is relatively long; second, image fusion based on deep learning networks mostly does not consider the deep information of the images, which to some extent increases the noise of the fused image and reduces the structural similarity of the images.

[0005] In recent years, with the popularization of intelligent power stations, the requirements for power equipment transmitted by intelligent robots during inspection in complex environments such as low light are higher. It is necessary to fully combine the feature information of visible light images and infrared images when using image fusion technology, and at the same time use deep learning networks to detect target objects. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing a method for detecting power equipment targets in complex environments.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] Step 1: Use the camera equipment mounted on the inspection robot to obtain visible light images of electrical equipment in complex low-light environments, and at the same time use an infrared thermal imager to obtain corresponding infrared images.

[0009] Step 2: Use the optimized threshold O-Canny edge detection algorithm combined with gradient values ​​to obtain edge information of infrared and visible light images, and use curvature scale space technology to detect the curvature of edge information to obtain feature points of infrared and visible light images;

[0010] Step 3: Use the GSIFT algorithm to extract feature descriptors of feature points in infrared and visible light images, and complete feature point matching between infrared and visible light images;

[0011] Step 4: Use the random sampling consensus algorithm to filter the feature point matching results to obtain accurate matching feature points;

[0012] Step 5: Use two ResNet deep learning networks to fuse the accurately matched infrared and visible light images, and output the fused image;

[0013] Step 6: Use labelImg software to label the power equipment in the infrared image, visible light image and fused image respectively, generate XML files, and create a labeled dataset;

[0014] Step 7: Set the training parameters of the YOLOv5 deep learning network, train the YOLOv5 deep learning network using a labeled dataset, and conduct three sets of experiments using the trained YOLOv5 deep learning network: visible light image target detection, infrared image target detection, and fused image target detection. Measure the target detection accuracy of each of the three sets of experiments to verify the necessity of fusing infrared and visible light images.

[0015] To optimize the above technical solution, the specific measures also include:

[0016] Further, in step 2, the step of obtaining edge information from the infrared and visible light images using the optimized threshold O-Canny edge detection algorithm incorporating gradient values ​​specifically involves:

[0017] Step 2.1: Calculate the weights based on the sensitivity of the human eye to the three RGB components, and then perform a weighted average according to the weight of each component to obtain the preprocessed images of the infrared and visible light images of the power equipment.

[0018] Step 2.2: Remove noise from the preprocessed image using a two-way filtering method that combines similarity and proximity between pixels.

[0019] Step 2.3: Calculate the magnitude and direction of the gradient at each pixel.

[0020] Step 2.4: Compare the gradient value of each pixel with the gradient values ​​of the two adjacent pixels in the gradient direction of that pixel to determine the candidate edge points;

[0021] Step 2.5: Use the maximum inter-class variance method to find adaptive high threshold T1 and low threshold T2 that match the image's own characteristics;

[0022] Step 2.6: Traverse the entire image and obtain the gradient magnitude of all pixels in the image. Based on the gradient magnitude of all pixels, find the gradient values ​​G0 and G1 respectively, so that the gradient magnitude of 98% of the pixels in the entire image is less than G0. Among them, G1 is the maximum value of the gradient magnitude of the pixels in the entire image.

[0023] Step 2.7: Optimize the adaptive high threshold T1 and low threshold T2 using the following formula to obtain high and low thresholds that are more suitable for image features:

[0024]

[0025]

[0026] In the formula, TN1 is the optimized high threshold and TN2 is the optimized low threshold. By using the optimized thresholds TN1 and TN2 that combine the gradient magnitude, edge detection is performed again under the two thresholds to select more accurate edge points.

[0027] Furthermore, step 3 specifically involves:

[0028] Step 3.1: Use the GSIFT algorithm to extract feature descriptors of feature points in infrared and visible light images, and use the feature descriptors to convert feature points into feature vectors;

[0029] Step 3.2: Perform standard normalization on the feature vectors;

[0030] Step 3.3: For key feature vectors with excessively large gradient values, first set a threshold value l0 based on the distribution of gradient magnitudes at feature points, so that the gradient magnitude is within the threshold value l0. o The probability of the following is 0.95;

[0031] Step 3.4: Find all key feature points whose gradient magnitude is greater than l0, and set the gradient value of the key feature point to l0, while leaving the gradient values ​​of key feature points whose gradient magnitude is not greater than l0 unchanged.

[0032] Step 3.5: Then, standardize the feature vectors and feature point gradient values;

[0033] Step 3.6: Feature point matching is completed by calculating the similarity and proximity between each feature vector after standardization.

[0034] Step 3.7: Remove significant matching feature points.

[0035] Furthermore, step 4 specifically involves:

[0036] The random sampling consensus algorithm is used to first randomly sample the entire feature point dataset, then linearly fit these sampled points to obtain effective fitted data. The random sampling and linear fitting process is repeated until accurate matching feature points are obtained.

[0037] Furthermore, step 5 specifically includes:

[0038] Step 5.1: Remove the average pooling layer, fully connected (FC) layer, and softmax layer from the tail of the ResNet deep learning network;

[0039] Step 5.2: Input the registered infrared image and visible light image into the ResNet deep learning network, and use a weighted fusion strategy to fuse the max pooling layer output feature maps of the two images to obtain fused feature map A;

[0040] Step 5.3: The ResNet deep learning network includes 5 stages. The visible light output from the max pooling layer of stage 3 is fused with the deep feature map of the infrared image using a weighted fusion strategy to obtain fused feature map B; the visible light output from the max pooling layer of stage 4 is fused with the deep feature map of the infrared image using a weighted fusion strategy to obtain fused feature map C.

[0041] Step 5.4: Combine fused feature map A with fused feature map B, and fused feature map A with fused feature map C into two pairs of images. Input these two pairs of images back into the ResNet network, and use the same weighted fusion strategy to fuse the max pooling layer feature maps of the two pairs of images, as shown in the following formula:

[0042] F = w1fstage +w2f fusion

[0043] In the formula, F represents the fusion result, w1 and w2 are the fusion weights, and f stage For f to fuse feature map B or fuse feature map C, fusion To fuse feature map A;

[0044] Step 5.5: Select the one with more significant features from the two fusion results as the final output.

[0045] Furthermore, in step 1, the resolution of the visible light image is 600*800, and the resolution of the infrared image is 576*768.

[0046] Furthermore, in step 5.4, the values ​​of the fusion weights w1 and w2 are both taken as 0.5.

[0047] Furthermore, in step 6, the resolution of the infrared image, visible light image, and fused image are all standardized to 640*480.

[0048] The beneficial effects of this invention are:

[0049] (1) The O-Canny edge detection algorithm based on the combined gradient value of the present invention detects more real edges, improves the accuracy of edge detection, and retains more feature information for subsequent registration and fusion to a certain extent.

[0050] (2) The visible light image and infrared image registration based on the improved GSIFT algorithm of the present invention largely solves the problem of excessive gradient values ​​of key point vectors caused by objective factors;

[0051] (3) The dual ResNet deep learning fusion network of the present invention can retain more of the original information of infrared and visible light images, and improve the quality of the fused image and the structural similarity between images;

[0052] (4) The method of the present invention has application value for the study of target detection of power equipment in complex environments.

[0053] (5) To address the lack of adaptive threshold selection in traditional edge detection operators, the O-Canny edge detection algorithm, which combines gradient values, is used to obtain image edges and feature points. The maximum inter-class variance algorithm is used to adaptively generate high and low thresholds. The obtained gradient magnitude is substituted into the optimized threshold formula to calculate the threshold again, which can preserve local details to the maximum extent while avoiding too much redundant information.

[0054] (6) To address the problem of numerous mismatched points caused by nonlinear illumination and other factors during the registration of infrared and visible light images, for key point feature vectors with excessively large gradient values, a truncation threshold l0 is determined based on the distribution of gradient magnitude, so that the probability of the gradient magnitude being below the threshold l0 is 0.95. This is then standardized again to reduce the impact of such changes and further eliminate errors.

[0055] (7) To address the issue that the fusion quality and noise of the fused image are affected when performing image fusion using deep learning, an image fusion algorithm based on a double ResNet network is designed to extract more image depth information. Attached Figure Description

[0056] Figure 1 A flowchart for a target detection method for power equipment in complex environments;

[0057] Figure 2 Flowchart for edge detection to optimize threshold selection;

[0058] Figure 3 A comparison of the effects of feature point registration using SIFT and GSIFT;

[0059] Figure 4 This is a flowchart of a deep learning fusion algorithm using two ResNet networks. Detailed Implementation

[0060] The invention will now be described in further detail with reference to the accompanying drawings.

[0061] This invention proposes a method for detecting power equipment targets in complex environments, the overall flowchart of which is shown below. Figure 1 As shown, it includes the following steps:

[0062] Step 1: Use the camera mounted on the inspection robot to acquire visible light images of electrical equipment in complex low-light environments, and at the same time use an infrared thermal imager to acquire corresponding infrared images; the resolution of the visible light images is 600*800, and the resolution of the infrared images is 576*768.

[0063] Step 2: Use the optimized threshold O-Canny edge detection algorithm combined with gradient values ​​to obtain edge information of infrared and visible light images. Use curvature scale space (CSS) technology to detect the curvature of edge information to obtain feature points of infrared and visible light images; preserve global features and important structural properties of the image.

[0064] Step 3: Use the GSIFT algorithm to extract feature descriptors of feature points in infrared and visible light images, and complete feature point matching between infrared and visible light images;

[0065] Step 4: Use the Random Sample Consensus (RANSAC) algorithm to filter the feature point matching results to obtain accurate matching feature points;

[0066] Step 5: Use two ResNet deep learning networks to fuse the accurately matched infrared and visible light images, and output the fused image;

[0067] Step 6: Use labelImg software to label the electrical equipment in the infrared image, visible light image, and fused image respectively, generate XML files, and create a labeled dataset. When manually labeling the target electrical equipment in the image using labelImg software, all clear target areas in the image should be labeled. The electrical equipment specifically refers to insulators. The resolution of the infrared image, visible light image, and fused image is uniformly set to 640*480.

[0068] Step 7: Set the training parameters of the YOLOv5 deep learning network. Train the YOLOv5 deep learning network using a labeled dataset. Perform three sets of experiments using the trained YOLOv5 deep learning network: visible light image target detection, infrared image target detection, and fused image target detection. Measure the target detection accuracy of each of the three sets of experiments. The IOU value is 0.5, which verifies the necessity of fusing infrared and visible light images.

[0069] In step 2, the step of obtaining edge information from infrared and visible light images using the optimized threshold O-Canny edge detection algorithm that combines gradient values ​​specifically involves:

[0070] Step 2.1: Calculate the weights based on the sensitivity of the human eye to the three RGB components, and then perform a weighted average according to the weight of each component to obtain the preprocessed images of the infrared and visible light images of the power equipment.

[0071] Step 2.2: Remove noise from the preprocessed image using a two-way filtering method that combines similarity and proximity between pixels.

[0072] Step 2.3: Calculate the magnitude and direction of the gradient at each pixel.

[0073] Step 2.4: Compare the gradient value of each pixel with the gradient values ​​of the two adjacent pixels in the gradient direction of that pixel to determine the candidate edge points;

[0074] Step 2.5: Use the maximum inter-class variance method to find adaptive high threshold T1 and low threshold T2 that match the image's own characteristics;

[0075] Step 2.6: Traverse the entire image and obtain the gradient magnitude of all pixels in the image. Based on the gradient magnitude of all pixels, find the gradient values ​​G0 and G1 respectively, so that the gradient magnitude of 98% of the pixels in the entire image is less than G0. Among them, G1 is the maximum value of the gradient magnitude of the pixels in the entire image.

[0076] Step 2.7: Optimize the adaptive high threshold T1 and low threshold T2 using the following formula to obtain high and low thresholds that are more suitable for image features:

[0077]

[0078]

[0079] In the formula, TN1 is the optimized high threshold and TN2 is the optimized low threshold. By using the optimized thresholds TN1 and TN2 that combine the gradient magnitude, edge detection is performed again under the two thresholds to select more accurate edge points.

[0080] Step 3 specifically involves:

[0081] Step 3.1: Use the GSIFT algorithm to extract feature descriptors of feature points in infrared and visible light images, and use the feature descriptors to convert feature points into feature vectors; feature descriptors can uniquely describe feature points and are suitable for registration tasks.

[0082] Step 3.2: To remove the influence of changes such as illumination, the feature vectors are normalized.

[0083] Step 3.3: For key feature vectors with excessively large gradient values, first set a threshold value l0 based on the distribution of gradient magnitudes of feature points, so that the probability of the gradient magnitude being below the threshold value l0 is 0.95.

[0084] Step 3.4: Find all key feature points whose gradient magnitude is greater than l0, and set the gradient value of the key feature point to l0, while leaving the gradient values ​​of key feature points whose gradient magnitude is not greater than l0 unchanged.

[0085] Step 3.5: Standardize the feature vectors and feature point gradient values ​​to further eliminate the influence of errors.

[0086] Step 3.6: Feature point matching is completed by calculating the similarity and proximity between each feature vector after standardization.

[0087] Step 3.7: Remove significant matching feature points.

[0088] Step 4 is as follows:

[0089] The random sampling consensus algorithm is used to first randomly sample the entire feature point dataset, then linearly fit these sampled points to obtain effective fitted data. The random sampling and linear fitting process is repeated until accurate matching feature points are obtained.

[0090] Step 5 specifically involves:

[0091] Step 5.1: Remove the average pooling layer, fully connected (FC) layer, and softmax layer from the tail of the ResNet deep learning network;

[0092] Step 5.2: Input the registered infrared image and visible light image into the ResNet deep learning network, and use a weighted fusion strategy to fuse the max pooling layer output feature maps of the two images to obtain fused feature map A;

[0093] Step 5.3: The ResNet deep learning network includes 5 stages. The visible light output from the max pooling layer of stage 3 is fused with the deep feature map of the infrared image using a weighted fusion strategy to obtain fused feature map B; the visible light output from the max pooling layer of stage 4 is fused with the deep feature map of the infrared image using a weighted fusion strategy to obtain fused feature map C.

[0094] Step 5.4: Combine fused feature map A with fused feature map B, and fused feature map A with fused feature map C into two pairs of images. Input these two pairs of images back into the ResNet network, and use the same weighted fusion strategy to fuse the max pooling layer feature maps of the two pairs of images, as shown in the following formula:

[0095] F = w1f stage +w2f fusuon

[0096] In the formula, F represents the fusion result, w1 and w2 are the fusion weights, and f stage For f to fuse feature map B or fuse feature map C, fusion The feature map is fused; the values ​​of the fusion weights w1 and w2 are both 0.5.

[0097] Step 5.5: Select the one with more significant features from the two fusion results as the final output.

[0098] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for detecting power equipment targets in complex environments, characterized in that, Includes the following steps: Step 1: Use the camera equipment mounted on the inspection robot to obtain visible light images of electrical equipment in complex low-light environments, and at the same time use an infrared thermal imager to obtain corresponding infrared images. Step 2: Obtain edge information from infrared and visible light images using the O-Canny edge detection algorithm with optimized thresholding combined with gradient values. Utilize curvature scale space technology to detect the curvature of the edge information to obtain feature points from the infrared and visible light images. Specifically, in Step 2, obtaining edge information from infrared and visible light images using the O-Canny edge detection algorithm with optimized thresholding combined with gradient values ​​involves: Step 2.1: Calculate the weights based on the sensitivity of the human eye to the three RGB components, and then perform a weighted average according to the weight of each component to obtain the preprocessed images of the infrared and visible light images of the power equipment. Step 2.2: Remove noise from the preprocessed image using a two-way filtering method that combines similarity and proximity between pixels. Step 2.3: Calculate the magnitude and direction of the gradient at each pixel. Step 2.4: Compare the gradient value of each pixel with the gradient values ​​of the two adjacent pixels in the gradient direction of that pixel to determine the candidate edge points; Step 2.5: Use the Otsu's method to find an adaptive high threshold that matches the image's own features. T 1 and low threshold T 2; Step 2.6: Traverse the entire image and obtain the gradient magnitude of all pixels in the image. Find the gradient value based on the gradient magnitude of each pixel. G 0 and G 1, so that the gradient magnitude of 98% of the pixels in the entire image is less than 1. G 0, where, G 1 represents the maximum gradient magnitude of a pixel in the entire image; step 2.

7. Use the following formula to apply the adaptive high threshold. T 1 and low threshold T 2. Optimize to obtain high and low thresholds that are more suitable for image features: In the formula, To optimize the high threshold, To optimize the low threshold, an optimized threshold combining gradient magnitude is used. and Edge detection is performed again under both thresholds to select more accurate edge points; Step 3: Use the GSIFT algorithm to extract feature descriptors of feature points in infrared and visible light images, and complete feature point matching between infrared and visible light images; Step 4: Use the random sampling consensus algorithm to filter the feature point matching results to obtain accurate matching feature points; Step 5: Use two ResNet deep learning networks to fuse the accurately matched infrared and visible light images, and output the fused image; Step 6: Use labelImg software to label the power equipment in the infrared image, visible light image and fused image respectively, generate XML files, and create a labeled dataset; Step 7: Set the training parameters of the YOLOv5 deep learning network, train the YOLOv5 deep learning network using a labeled dataset, and conduct three sets of experiments using the trained YOLOv5 deep learning network: visible light image target detection, infrared image target detection, and fused image target detection. Measure the target detection accuracy of each of the three sets of experiments to verify the necessity of fusing infrared and visible light images.

2. The method for detecting power equipment targets in complex environments as described in claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Use the GSIFT algorithm to extract feature descriptors of feature points in infrared and visible light images, and use the feature descriptors to convert feature points into feature vectors; Step 3.2: Perform standard normalization on the feature vectors; Step 3.3: For key feature vectors with excessively large gradient values, first set a threshold value based on the distribution of gradient magnitudes at feature points. This makes the gradient magnitude within the threshold value. The probability of the following is 0.95; Step 3.4: Find all gradient values ​​whose magnitudes are greater than 1. Let the key feature point be denoted by and let the gradient value of the key feature point be denoted by . The gradient magnitude is no greater than The gradient values ​​of the key feature points remain unchanged; Step 3.5: Then, standardize the feature vectors and feature point gradient values; Step 3.6: Feature point matching is completed by calculating the similarity and proximity between each feature vector after standardization. Step 3.7: Remove significant matching feature points.

3. The method for detecting power equipment targets in complex environments as described in claim 1, characterized in that, Step 4 is as follows: The random sampling consensus algorithm is used to first randomly sample the entire feature point dataset, then linearly fit these sampled points to obtain effective fitted data. The random sampling and linear fitting process is repeated until accurate matching feature points are obtained.

4. The method for detecting power equipment targets in complex environments as described in claim 1, characterized in that, Step 5 specifically involves: Step 5.1: Remove the average pooling layer, fully connected (FC) layer, and softmax layer from the tail of the ResNet deep learning network; Step 5.2: Input the registered infrared image and visible light image into the ResNet deep learning network, and use a weighted fusion strategy to fuse the max pooling layer output feature maps of the two images to obtain fused feature map A; Step 5.3: The ResNet deep learning network includes 5 stages. The visible light output from the max pooling layer of stage 3 is fused with the deep feature map of the infrared image using a weighted fusion strategy to obtain fused feature map B; the visible light output from the max pooling layer of stage 4 is fused with the deep feature map of the infrared image using a weighted fusion strategy to obtain fused feature map C. Step 5.4: Combine fused feature map A with fused feature map B, and fused feature map A with fused feature map C into two pairs of images. Input these two pairs of images back into the ResNet network, and use the same weighted fusion strategy to fuse the max pooling layer feature maps of the two pairs of images, as shown in the following formula: In the formula, F For the fusion result, and To integrate weights, To fuse feature map B or fuse feature map C, To fuse feature map A; Step 5.5: Select the one with more significant features from the two fusion results as the final output.

5. The method for detecting power equipment targets in complex environments as described in claim 1, characterized in that, In step 1, the resolution of the visible light image is 600. 800, the resolution of the infrared image is 576.

768.

6. The method for detecting power equipment targets in complex environments as described in claim 4, characterized in that, In step 5.4, the fusion weights and The values ​​are all taken as 0.

5.

7. The method for detecting power equipment targets in complex environments as described in claim 1, characterized in that, In step 6, the resolution of the infrared image, visible light image, and fused image are all standardized to 640. 480.

Images