Power transmission line inspection image target detection method and system and computer readable medium

Abstract

The invention discloses a power transmission line inspection image target detection method and system and a computer readable medium, and relates to the technical field of image processing, and the method comprises the following steps: employing a random probability resampling strategy to generate a resampling data set for a power transmission line inspection infrared image data set C in a fine rain weather, and obtaining a resampling data set RC; performing image augmentation on the resampling data set RC by adopting a multi-scale adaptive hybrid strategy to obtain a to-be-trained data set TC; training a target detection algorithm by using the to-be-trained data set TC; and using the trained target detection algorithm to carry out target detection on the power transmission line inspection infrared image in the fine rain weather. According to the invention, accurate detection of the transmission line inspection infrared image target in the fine rain weather can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for detecting targets in transmission line inspection images. Background Art

[0002] In recent years, my country's ultra-high voltage and ultra-high voltage transmission systems have been vigorously developed. The safe and stable operation of the power grid is of vital importance. The transmission lines may increase their resistance, heat up, and break the wires due to the effects of electromechanical loads, complex weather and natural environment for many years, which may cause serious hidden dangers to the safe and stable operation of the power grid. In the actual maintenance of the power grid, infrared temperature measurement is a method of using infrared imaging equipment to monitor the internal temperature of an object. However, traditional thermometers require workers to climb to the corresponding position during operation, and cannot perform real-time inspections, so they usually have great limitations during measurement. With the maturity of artificial intelligence technology and infrared thermal imaging technology, data-driven transmission line fault detection has become the mainstream. However, in the actual drone infrared inspection process, due to the long inspection cycle and the high uncertainty of temperature changes in drizzling weather, the collected infrared image temperature value samples are small, there is a problem of data imbalance, and it is impossible to meet the training requirements of the deep detection model, resulting in low accuracy of transmission line inspection image target detection. Summary of the invention

[0003] In view of this, the present invention aims at the deficiencies in the prior art and provides a method for detecting targets in transmission line inspection images.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for detecting targets in a transmission line inspection image, comprising the following steps:

[0005] The random probability resampling strategy is used to generate the resampled dataset C of infrared image data set C of power transmission line inspection under drizzle weather, and the resampled dataset RC is obtained;

[0006] A multi-scale adaptive hybrid strategy is used to augment the resampled dataset RC to obtain the training dataset TC.

[0007] Use the training data set TC to train the target detection algorithm;

[0008] The trained target detection algorithm is used to perform target detection on infrared images of power line inspections under drizzling rain.

[0009] Furthermore, the random probability resampling strategy is:

[0010] The resampled data OC is determined according to the extraction probability of the instance image in the infrared image dataset C of power line inspection under drizzle weather. Based on the distribution of the proportion of targets of different scales in the infrared image, the extraction probability P of the infrared image dataset of power line inspection under drizzle weather is defined as a factor to measure the performance of small target detection, where the extraction probability P is

[0011]

[0012] In the formula, S Small , S All is an array, which stores the total number of small targets and the total number of targets of each scale contained in each instance image in the infrared image dataset of power transmission line inspection under drizzle weather; the resampled data OC is determined according to the extraction probability, that is,

[0013]

[0014] In the formula, is the rounding symbol, when the extraction probability p of image c(c∈C) c When it is greater than the random number Rand (Rand∈(0,1)), the image is resampled;

[0015] The resampled data OC is spliced ​​with the infrared image dataset C of power transmission line inspection under drizzle weather to obtain the resampled dataset RC, that is,

[0016]

[0017] Furthermore, the multi-scale adaptive hybrid strategy is:

[0018] For each image in RC, extract the mask of large and medium-scale targets from the annotation box, and obtain its extraction probability p from P rc , calculate the maximum number of augmented targets n, that is

[0019] n=A rc ×{(1-p rc )-E(C)}

[0020] In the formula, A rc is the total number of objects in the pasted image, p rc is the extraction probability of the pasted image, 1-p rc is the actual proportion of large and medium objects in the pasted image, and E(C) is the expected proportion of large and medium objects in the dataset;

[0021] The augmented image is

[0022]

[0023] Where Mask∈{0,1} W*HIt is a binary mask with a width of W and a height of H. Its width and height are consistent with the augmented image. The value of the augmented target label box is 0, and the rest are 1. i is the randomly selected original image containing a small target, rc is the augmented image, It is a pixel-level multiplication operation. The number of copy and paste targets is determined by the hyperparameter k and the maximum number of augmented targets n.

[0024] The augmented images of each image in RC constitute the training data set TC.

[0025] Furthermore, the target detection algorithm is a YOLOX target detection algorithm.

[0026] The present invention also provides a transmission line inspection image target detection system, comprising:

[0027] The resampled data set construction module is configured to generate a resampled data set using a random probability resampling strategy for the transmission line inspection infrared image data set C under drizzle weather, and obtain a resampled data set RC;

[0028] The image augmentation module is configured to adopt a multi-scale adaptive hybrid strategy to perform image augmentation on the resampled data set RC to obtain a data set to be trained TC;

[0029] The target detection algorithm module is configured to train the target detection algorithm using the training data set TC;

[0030] The target detection module is configured to use the trained target detection algorithm to perform target detection on the infrared image of the power transmission line inspection under drizzling weather.

[0031] Furthermore, the random probability resampling strategy is:

[0032] The resampled data OC is determined according to the extraction probability of the instance image in the infrared image dataset C of power line inspection under drizzle weather. Based on the distribution of the proportion of targets of different scales in the infrared image, the extraction probability P of the infrared image dataset of power line inspection under drizzle weather is defined as a factor to measure the performance of small target detection, where the extraction probability P is

[0033]

[0034] In the formula, S Small , S All is an array, which stores the total number of small targets and the total number of targets of each scale contained in each instance image in the infrared image dataset of power transmission line inspection under drizzle weather; the resampled data OC is determined according to the extraction probability, that is,

[0035]

[0036] In the formula, is the rounding symbol, when the extraction probability p of image c(c∈C) c When it is greater than the random number Rand (Rand∈(0,1)), the image is resampled;

[0037] The resampled data OC is spliced ​​with the infrared image dataset C of power transmission line inspection under drizzle weather to obtain the resampled dataset RC, that is,

[0038]

[0039] Furthermore, the multi-scale adaptive hybrid strategy is:

[0040] For each image in RC, extract the mask of large and medium-scale targets from the annotation box, and obtain its extraction probability p from P rc , calculate the maximum number of augmented targets n, that is

[0041] n=A rc ×{(1-p rc )-E(C)}

[0042] In the formula, A rc is the total number of objects in the pasted image, p rc is the extraction probability of the pasted image, 1-p rc is the actual proportion of large and medium objects in the pasted image, and E(C) is the expected proportion of large and medium objects in the dataset;

[0043] The augmented image is

[0044]

[0045] Where Mask∈{0,1} W*H It is a binary mask with a width of W and a height of H. Its width and height are consistent with the augmented image. The value of the augmented target label box is 0, and the rest are 1. i is the randomly selected original image containing a small target, rc is the augmented image, It is a pixel-level multiplication operation. The number of copy and paste targets is determined by the hyperparameter k and the maximum number of augmented targets n.

[0046] The augmented images of each image in RC constitute the training data set TC.

[0047] Furthermore, the target detection algorithm is a YOLOX target detection algorithm.

[0048] The present invention also provides a computer-readable medium storing a computer program, and when the computer program is executed, the above-mentioned method for detecting targets in transmission line inspection images is implemented.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The target detection method of the transmission line inspection image of the present invention first uses a random probability resampling strategy to generate a resampled data set for the transmission line inspection infrared image under drizzling weather, and strengthens the target detection algorithm to learn small target features at the image level; then, a scale adaptive hybrid strategy is used to generate the final training data set, and the probability of large and medium-scale targets being over-enhanced is reduced at the target level, alleviating the imbalance between the training samples of targets of each scale caused by the improved random probability resampling strategy. A multi-scale adaptive hybrid strategy is adopted to obtain the training data set; finally, the target detection algorithm is trained to achieve accurate detection of targets in the transmission line inspection infrared image under drizzling weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a method flow chart of an embodiment of the present invention;

[0052] Figure 2 is a method structure diagram of an embodiment of the present invention;

[0053] Figure 3 is a structural diagram of a random probability resampling strategy in an embodiment of the present invention;

[0054] Figure 4 is an algorithm flow chart of an embodiment of the present invention;

[0055] Figure 5 It is a schematic diagram of infrared image target detection in power transmission line inspection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0057] Example 1

[0058] Reference Figure 1-5, an embodiment of the present invention proposes a method for target detection in transmission line inspection images, which integrates a random probability resampling strategy and a multi-scale adaptive hybrid strategy. First, a random resampling strategy is used to improve the extraction scheme of training images in resampling at the image level, and the target detection algorithm is strengthened to learn small target features at the image level; then a multi-scale adaptive hybrid strategy is used to reduce the probability of large and medium-scale targets being over-enhanced at the target level, and alleviate the imbalance between training samples of targets of different scales caused by the improved resampling strategy. The specific steps are as follows:

[0059] S1. A random probability resampling strategy is used to generate a resampled dataset C of infrared image data set C of power transmission line inspection under drizzle weather, and a resampled dataset RC is obtained.

[0060] Random probability resampling strategy: In the form of random probability, the infrared image data containing small targets of power transmission lines in drizzling weather are repeatedly sampled at the image level. The specific steps are as follows:

[0061] S11. Determine the resampled data OC according to the extraction probability of the instance image in the infrared image dataset C of the power transmission line inspection under drizzle weather; Based on the distribution of the proportion of targets of each scale in the infrared image, define the extraction probability P of the infrared image dataset of the power transmission line inspection under drizzle weather as a factor to measure the impact of small target detection performance, where the extraction probability P is

[0062]

[0063] In the formula, S Small , S All is an array, which stores the total number of small targets and the total number of targets of each scale contained in each instance image in the infrared image dataset of power transmission line inspection under drizzle weather; the resampled data OC is determined according to the extraction probability, that is,

[0064]

[0065] In the formula, is the rounding symbol, when the extraction probability p of image c(c∈C) c When it is greater than the random number Rand (Rand∈(0,1)), the image is resampled;

[0066] S12, the resampled data OC is spliced ​​with the transmission line inspection infrared image dataset C under drizzle weather to obtain the resampled dataset RC, that is,

[0067]

[0068] This strategy allows training images containing more small targets, that is, training images that have a greater impact on small target detection performance, to fully participate in the training of the model, prompting the model to learn more small target features of transmission line inspection under drizzle weather.

[0069] The comparison of data augmentation results of the algorithm using random probability resampling strategy and equal probability resampling strategy is shown in Table 1. Small target detection accuracy AP after using equal probability resampling strategy s The accuracy has not been significantly improved, but has dropped by 0.2%, and the average accuracy of all categories has also dropped by 0.3% compared with the Baseline algorithm. The experimental results show that equal probability resampling of the data set will lead to data overfitting, which will affect the detection accuracy. After adopting the random probability resampling strategy proposed in this chapter, the small target detection accuracy AP s The average precision mAP of all categories increased by 1%, and the average precision mAP of all categories increased by 1.1%. This experimental result shows that the random probability resampling strategy based on the extraction probability can increase the number of small target training images and sample more training images that have a greater impact on small target detection performance.

[0070] Table 1 Augmentation effect under different resampling strategies%

[0071]

[0072] S2. A multi-scale adaptive hybrid strategy is used to augment the resampled data set RC to obtain the training data set TC.

[0073] Infrared images taken by drones from multiple perspectives will cause a certain degree of occlusion of the transmission line inspection targets in drizzling weather during the imaging process due to different backgrounds and temperatures. The earliest method to solve the occlusion problem was inspired by Dropout, which randomly cuts a continuous area on the image and uses a fixed-size rectangle to occlude the target in the image.

[0074] Since most single images in the infrared image dataset of power line inspection contain only 1 to 2 targets, if a fixed-size rectangle is used for coverage and all values ​​inside the rectangular coverage are set to 0 or other pure color values, the target information in the image will be missing. The present invention simulates occlusion in a form similar to copy and paste, and adds multi-view targets to the infrared image dataset to simulate the occlusion phenomenon in infrared images. And because the number of training samples of targets of various scales in the infrared image dataset of power line inspection in drizzling weather is sparse and uneven, the detection capabilities of large and medium-scale targets will be over-enhanced. To address this problem, this paper proposes a multi-scale adaptive hybrid strategy (Adaptive Scale matchingCopyPaste, AdaSMCP) to copy the area where some scale targets are located in the image and randomly paste it into the background or foreground area of ​​the infrared image to alleviate the imbalance of training samples of targets of various scales.

[0075] The multi-scale adaptive hybrid strategy is:

[0076] For each image in RC, extract the mask of large and medium-scale targets from the annotation box (GT bbox), and obtain its extraction probability p from P rc , calculate the maximum number of augmented targets n, that is

[0077] n=A rc ×{(1-p rc )-E(C)}

[0078] In the formula, A rc is the total number of objects in the pasted image, p rc is the extraction probability of the pasted image, 1-p rc is the actual proportion of large and medium objects in the pasted image, and E(C) is the expected proportion of large and medium objects in the dataset;

[0079] The augmented image is

[0080]

[0081] Where Mask∈{0,1} W*H It is a binary mask with a width of W and a height of H. Its width and height are consistent with the augmented image. The value of the augmented target box is 0, and the rest are 1. i is the randomly selected original image containing a small target, rc is the augmented image, It is a pixel-level multiplication operation. The number of copy and paste targets is determined by the hyperparameter k and the maximum number of augmented targets n.

[0082] The augmented images tc of each image in RC constitute the training data set TC.

[0083] The multi-scale adaptive hybrid strategy (Adaptive Scale matching CopyPaste, AdaSMCP) inherits the idea of ​​hybrid augmentation strategies such as Mosaic, Mixup and CopyPaste. In order to verify the effectiveness of the multi-scale adaptive hybrid strategy, the Baseline algorithm without any augmentation strategy and the algorithm integrating the Mosaic and Mixup hybrid augmentation strategies are selected for comparison. The experimental results are shown in Table 2. Under the action of the separated image hybrid augmentation algorithm, the mAP detection accuracy of all categories of transmission line inspection in drizzle weather is improved by 1.1%, and the small target detection accuracy is AP s Under the AdaSMCP strategy proposed in this invention, the mAP of all-category target detection is improved by 4.4% compared with the Baseline, and the small target accuracy is also improved by 1.8%, which effectively overcomes the difficulty of detecting occluded samples.

[0084] Table 2 Experimental results under different hybrid augmentation strategies

[0085]

[0086] In order to verify the effectiveness of the random probability resampling strategy and the multi-scale adaptive hybrid strategy proposed in this chapter, an ablation experiment was conducted. The experimental results are shown in Table 3. When the above two strategies work together, that is, the sampling hybrid fusion augmentation strategy is used, the mAP accuracy of all-category target detection is improved by 4.1% compared with the Baseline, and the small target detection accuracy is AP s Improved by 2.8% and the detection accuracy of medium-sized objects AP m Improved by 4%, large-scale target detection accuracy AP l Improved by 2.5%, average accuracy AP 50 The results show that the infrared image dataset generated by the sampling hybrid augmentation algorithm proposed in the present invention has a significant improvement on the subsequent target detection algorithm in mining small targets and improving the anti-occlusion ability of the model.

[0087] Table 3 Ablation experiment results of sampling hybrid fusion augmentation strategy

[0088]

[0089] S3. Use the training data set TC to train the YOLOX target detection algorithm.

[0090] S4. Use the trained target detection algorithm to perform target detection on the infrared images of power transmission line inspections under drizzling weather.

[0091] In the embodiment of the present invention, the infrared image of the transmission line inspection in drizzling weather takes the composite insulator target as an example, and the detection results are as follows: Figure 5 shown.

[0092] Example 2

[0093] The embodiment of the present invention is a transmission line inspection image target detection system, comprising:

[0094] The resampled data set construction module is configured to generate a resampled data set using a random probability resampling strategy for the transmission line inspection infrared image data set C under drizzle weather, and obtain a resampled data set RC;

[0095] The image augmentation module is configured to adopt a multi-scale adaptive hybrid strategy to perform image augmentation on the resampled data set RC to obtain a data set to be trained TC;

[0096] The target detection algorithm module is configured to train the target detection algorithm using the training data set TC;

[0097] The target detection module is configured to use the trained target detection algorithm to perform target detection on the infrared image of the power transmission line inspection under drizzling weather.

[0098] In the embodiment of the present invention, the random probability resampling strategy, the multi-scale adaptive hybrid strategy and the target detection algorithm are the same as those in Embodiment 1 and will not be described in detail.

[0099] Example 3

[0100] An embodiment of the present invention provides a computer-readable medium storing a computer program. When the computer program is executed, the method for detecting targets in a transmission line inspection image as described in Embodiment 1 is implemented.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art can still modify or make equivalent substitutions to the specific implementations of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention are within the scope of protection of the claims of the present invention to be approved.

Claims

1. A method for detecting targets in transmission line inspection images, characterized in that: The following steps are involved: The random probability resampling strategy is used to generate the resampled dataset C of infrared image data set C of power transmission line inspection under drizzle weather, and the resampled dataset RC is obtained; A multi-scale adaptive hybrid strategy is used to augment the resampled dataset RC to obtain the training dataset TC. Use the training data set TC to train the target detection algorithm; The trained target detection algorithm is used to perform target detection on infrared images of power line inspections under drizzling rain.

2. The method for detecting targets in power transmission line inspection images according to claim 1, characterized in that: The random probability resampling strategy is: The resampled data OC is determined according to the extraction probability of the instance image in the infrared image dataset C of power line inspection under drizzle weather. Based on the distribution of the proportion of targets of different scales in the infrared image, the extraction probability P of the infrared image dataset of power line inspection under drizzle weather is defined as a factor to measure the performance of small target detection, where the extraction probability P is In the formula, S Small , S All is an array, which stores the total number of small targets and the total number of targets of each scale contained in each instance image in the infrared image dataset of power transmission line inspection under drizzle weather; the resampled data OC is determined according to the extraction probability, that is, In the formula, is the rounding symbol, when the extraction probability p of image c(c∈C) c When it is greater than the random number Rand (Rand∈(0,1)), the image is resampled; The resampled data OC is spliced ​​with the infrared image dataset C of power transmission line inspection under drizzle weather to obtain the resampled dataset RC, that is, 3. The method for detecting targets in power transmission line inspection images according to claim 2, characterized in that: The multi-scale adaptive hybrid strategy is: For each image in RC, extract the mask of large and medium-scale targets from the annotation box, and obtain its extraction probability p from P rc , calculate the maximum number of augmented targets n, that is n=A rc ×{(1-p rc )-E(C)} In the formula, A rc is the total number of objects in the pasted image, p rc is the extraction probability of the pasted image, 1-p rc is the actual proportion of large and medium objects in the pasted image, and E(C) is the expected proportion of large and medium objects in the dataset; The augmented image is Where Mask∈{0,1} W*H It is a binary mask with a width of W and a height of H. Its width and height are consistent with the augmented image. The value of the augmented target label box is 0, and the rest are 1. i is the randomly selected original image containing a small target, rc is the augmented image, It is a pixel-level multiplication operation. The number of copy and paste targets is determined by the hyperparameter k and the maximum number of augmented targets n. The augmented images of each image in RC constitute the training data set TC.

4. The method for detecting targets in power transmission line inspection images according to claim 1, characterized in that: The target detection algorithm is the YOLOX target detection algorithm.

5. A transmission line inspection image target detection system, characterized in that: include: The resampled data set construction module is configured to generate a resampled data set using a random probability resampling strategy for the transmission line inspection infrared image data set C under drizzle weather, and obtain a resampled data set RC; The image augmentation module is configured to adopt a multi-scale adaptive hybrid strategy to perform image augmentation on the resampled data set RC to obtain a data set to be trained TC; The target detection algorithm module is configured to train the target detection algorithm using the training data set TC; The target detection module is configured to use the trained target detection algorithm to perform target detection on the infrared image of the power transmission line inspection under drizzling weather.

6. The transmission line inspection image target detection system according to claim 5, characterized in that: The random probability resampling strategy is: The resampled data OC is determined according to the extraction probability of the instance image in the infrared image dataset C of power line inspection under drizzle weather. Based on the distribution of the proportion of targets of different scales in the infrared image, the extraction probability P of the infrared image dataset of power line inspection under drizzle weather is defined as a factor to measure the performance of small target detection, where the extraction probability P is In the formula, S Small , S All is an array, which stores the total number of small targets and the total number of targets of each scale contained in each instance image in the infrared image dataset of power transmission line inspection under drizzle weather; the resampled data OC is determined according to the extraction probability, that is, In the formula, is the rounding symbol, when the extraction probability p of image c(c∈C) c When it is greater than the random number Rand (Rand∈(0,1)), the image is resampled; The resampled data OC is spliced ​​with the infrared image dataset C of power transmission line inspection under drizzle weather to obtain the resampled dataset RC, that is, 7. The transmission line inspection image target detection system according to claim 6, characterized in that: The multi-scale adaptive hybrid strategy is: For each image in RC, extract the mask of large and medium-scale targets from the annotation box, and obtain its extraction probability p from P rc , calculate the maximum number of augmented targets n, that is n=A rc ×{(1-p rc )-E(C)} In the formula, A rc is the total number of objects in the pasted image, p rc is the extraction probability of the pasted image, 1-p rc is the actual proportion of large and medium objects in the pasted image, and E(C) is the expected proportion of large and medium objects in the dataset; The augmented image is Where Mask∈{0,1} W*H It is a binary mask with a width of W and a height of H. Its width and height are consistent with the augmented image. The value of the augmented target label box is 0, and the rest are 1. i is the randomly selected original image containing a small target, rc is the augmented image, It is a pixel-level multiplication operation. The number of copy and paste targets is determined by the hyperparameter k and the maximum number of augmented targets n. The augmented images of each image in RC constitute the training data set TC.

8. The power transmission line inspection image target detection system according to claim 5, characterized in that: The target detection algorithm is the YOLOX target detection algorithm.

9. A computer readable medium storing a computer program, characterized in that: When the computer program is executed, the method for detecting objects in power transmission line inspection images as claimed in any one of claims 1 to 4 is implemented.

Images