Method for detecting and quantifying hardware corrosion in electric power inspection image

Abstract

The invention provides a hardware corrosion detection and quantification method in an electric power inspection image, and relates to the technical field of electric power inspection image processing, and the method comprises the following steps: collecting an unmanned aerial vehicle inspection power transmission line aerial image; positioning hardware equipment in the image by using the improved YOLOv5s; separating the hardware fitting from the background by using a GrabCut segmentation algorithm and a morphological optimization algorithm; extracting and separating color images of three channels in a YCrCb color space, and obtaining a grayscale image of a Cr channel; according to the threshold segmentation of the Cr channel, obtaining a hardware corrosion binary image, and according to the binary image, judging whether the hardware is corroded or not; and counting the number of pixel points in the corrosion area and the number of pixel points in the hardware fitting area, and judging the corrosion grade of the hardware fitting by calculating the area ratio of the two areas. According to the method, deep learning and traditional image processing are combined, hardware identification, corrosion area detection and corrosion degree quantification in the inspection image are realized, and the defects of low working efficiency and high subjectivity caused by manual visual inspection of corrosion detection of power transmission line hardware equipment are overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power inspection image processing, and particularly relates to a method for detecting and quantifying fittings rust in a power inspection image. BACKGROUND

[0002] A power transmission line is an important part of a power system, and regular inspection of a power transmission line channel is an important measure to ensure stable operation of the power system. The inspection content mainly includes appearance defect detection of power transmission line equipment and dangerous ground object prediction around the power line. The power transmission line is composed of insulators, power towers, power lines, fittings and other power equipment with different functions. Due to the long span of the power transmission line and the complex environment of the power equipment, damage and rust are inevitable due to exposure to wind, sun and rain for a long time, which brings great security risks to the stable operation of the power grid.

[0003] With the continuous expansion of the power grid and the increasing use of power equipment, the difficulty of power transmission and transformation operation and maintenance is increasing. Compared with traditional manual inspection, unmanned aerial vehicle inspection not only saves funds and labor costs, but also reduces the risk of manual inspection operation. Compared with manual inspection, it does not need to consider the obstruction of traffic, reduces the difficulty of overcoming complex natural environment for the inspection personnel, and can more effectively find the location of the power equipment fault point. Therefore, power companies have adopted unmanned aerial vehicles equipped with camera devices to inspect power transmission and transformation lines. However, unmanned aerial vehicle inspection will inevitably collect a large amount of images, and manual searching for power equipment and labeling of defect positions are prone to visual fatigue, which may lead to missed or false detection of defects. At the same time, the power transmission line contains a large number of fittings, and serious rust of the fittings will affect the safe and stable operation of the power system. Since the rust defect of the power equipment is an important defect type in the power transmission link, how to quickly and accurately find the rust defect of the power equipment and repair it is a problem to be solved in power transmission and operation and maintenance. SUMMARY

[0004] (I) Technical problems solved In view of the defects in the above-mentioned detection of fittings rust in the power transmission line inspection image, the present application provides a method for detecting and quantifying fittings rust in a power inspection image, which adopts a combination of deep learning and traditional image processing to identify fittings and detect rust areas in the inspection image, thereby solving the defects of low work efficiency and large subjectivity in manual visual inspection of fittings rust in the power transmission line.

[0005] (II) Technical solutions To achieve the above-mentioned purpose, the present application is implemented by the following technical solutions: A method for detecting and quantifying fittings rust in a power inspection image, comprising the following steps: S1, collecting unmanned aerial vehicle inspection power transmission line aerial image; S2. Input the aerial image from step S1 into a deep learning network model, use the deep learning network model to locate the hardware in the image, and then cut out the hardware. S3. First, use a semi-interactive segmentation algorithm to perform image segmentation processing on the image cropped in step S2, and then use a morphological optimization algorithm to separate the hardware from the background. S4. Convert the segmented hardware equipment image from RGB color space to YCrCb color space, extract and separate the three-channel color image of YCrCb color space, and obtain the grayscale image of Cr channel. S5. Perform threshold segmentation on the grayscale image of step S4 according to the Cr channel threshold to obtain a binary image of hardware corrosion. Determine whether the hardware is corroded based on the binary image. S6. Based on the binary image after threshold segmentation in step S5, perform hardware corrosion quantification, count the number of pixels in the corrosion area and the number of pixels in the hardware area, and determine the hardware corrosion level by calculating the area ratio of the two areas.

[0006] Furthermore, the resolution of the aerial image acquired in step S1 is adjusted to 608×608.

[0007] According to an embodiment of the present invention, the deep learning network model used in step S2 includes a feature extraction network, a feature fusion network, and a prediction network; The feature extraction network includes a Focus module, a first CBL module, a first DCSP module, a second CBL module, a second DCSP module, a third CBL module, a third DCSP module, a fourth CBL module, and an SPP module. The Focus module, the first CBL module, the first DCSP module, the second CBL module, the second DCSP module, the third CBL module, the third DCSP module, the fourth CBL module, and the SPP module are sequentially connected to extract image features. The input of the feature fusion network is connected to the four effective feature layers (19×19, 38×38, 76×76, 152×152) of the feature extraction network. The feature fusion network is used for deep fusion of shallow and deep image features. The output of the feature fusion network is connected to the three inputs of the prediction network.

[0008] According to one embodiment of the present invention, the Focus module is connected to the input image 608×608×3, and the output feature of the Focus module is 304×304×32; The first CBL module consists of a convolutional layer, a batch normalization function, and an activation function. The convolutional layer is a 3×3 convolution with a stride of 2. The output feature of the first CBL convolutional module is 152×152×64. The first DCSP module is used to extract image features of 152×152, and the output features of the first DCSP module are 152×152×64. The second CBL module consists of a convolutional layer, a batch normalization function, and an activation function. The convolutional layer is a 3×3 convolution with a stride of 2. The output feature of the second CBL convolutional module is 76×76×128. The second DCSP module is used to extract image features of 76×76, and the output features of the second DCSP module are 76×76×128. The third CBL module consists of a convolutional layer, a batch normalization, and an activation function. The convolutional layer is a 3×3 convolution with a stride of 2. The output feature of the third CBL convolutional module is 38×38×256. The third DCSP module is used to extract image features of 38×38, and the output features of the third DCSP module are 38×38×256. The fourth CBL module consists of a convolutional layer, a batch normalization function, and an activation function. The convolutional layer is a 3×3 convolution with a stride of 2. The output feature of the fourth CBL convolutional module is 19×19×512.

[0009] According to an embodiment of the present invention, the first DCSP module includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. The first and second convolutional layers are 1×1×64 convolutions, respectively connected to the input feature 152×152×64. The third convolutional layer is a 1×1×32 convolution, and the fourth convolutional layer is a 3×3×32 convolution. The third and fourth convolutional layers constitute a first Dense module. The output of the first convolutional layer is concatenated with the output of the fourth convolutional layer to serve as the output of the first Dense module. The output of the first Dense module is concatenated with the output of the second convolutional layer and then input to the fifth convolutional layer. The fifth convolutional layer is a 1×1×64 convolution, and the output feature of the fifth convolutional layer is 152×152×64. The second DCSP module includes a sixth, seventh, eighth, ninth, and tenth convolutional layer. The sixth and seventh convolutional layers are 1×1×128 convolutions, each connected to an input feature of 76×76×128. The eighth convolutional layer is a 1×1×32 convolution, and the ninth convolutional layer is a 3×3×32 convolution. The eighth and ninth convolutional layers constitute the second Dense module. The output of the sixth convolutional layer is concatenated with the output of the ninth convolutional layer to serve as the output of the second Dense module. The output of the second Dense module is then concatenated with the output of the seventh convolutional layer and input to the tenth convolutional layer. The tenth convolutional layer is a 1×1×128 convolution, and its output feature is 76×76×128. The third DCSP module includes an eleventh, twelfth, thirteenth, fourteenth, and fifteenth convolutional layer. The eleventh and twelfth convolutional layers are 1×1×256 convolutions, each connected to an input feature of 38×38×256. The thirteenth convolutional layer is a 1×1×64 convolution, and the fourteenth convolutional layer is a 3×3×64 convolution. The thirteenth and fourteenth convolutional layers constitute the third Dense module. The output of the eleventh convolutional layer is concatenated with the output of the fourteenth convolutional layer to serve as the output of the third Dense module. The output of the third Dense module is concatenated with the output of the twelfth convolutional layer and then input to the fifteenth convolutional layer. The fifteenth convolutional layer is a 1×1×256 convolution, and its output feature is 38×38×256.

[0010] According to an embodiment of the present invention, the feature fusion network includes a first 1×1 convolution, a second 1×1 convolution, a third 1×1 convolution, a fourth 1×1 convolution, a fifth 1×1 convolution, a sixth 1×1 convolution, a seventh 1×1 convolution, a first ACSP module, a second ACSP module, a third ACSP module, a fourth ACSP module, a first upsampling module, a second upsampling module, a third upsampling module, a fourth upsampling module, a fifth upsampling module, a first downsampling module, a second downsampling module, and a third downsampling module; The inputs of the first ACSP module, the second 1×1 convolution, the fourth 1×1 convolution, and the seventh 1×1 convolution are connected to the effective feature layers (19×19, 38×38, 76×76, 152×152) of the feature extraction network, respectively. The output of the first ACSP module is connected to the first 1×1 convolution. The first 1×1 convolution, after being upsampled by the first convolution and concatenated with the output of the second 1×1 convolution, is input to the third 1×1 convolution. The first ACSP module, after being upsampled by the fourth convolution, and the third 1×1 convolution, after being upsampled by the second convolution and concatenated with the output of the fourth 1×1 convolution, are input to the fifth 1×1 convolution. The second 1×1 convolution, after being upsampled by the fifth convolution, is input to the fifth 1×1 convolution. The outputs of the upsampled fifth 1×1 convolution are concatenated with the outputs of the third upsampled and seventh 1×1 convolutions, and then concatenated with the fifth 1×1 convolutions via the first downsampled method before being input to the second ACSP module. The output of the second ACSP module is concatenated with the output of the third 1×1 convolution via the second downsampled method before being input to the third ACSP module. The output of the third ACSP module is concatenated with the output of the first 1×1 convolution via the third downsampled method before being input to the fourth ACSP module. The outputs of the second, third, and fourth ACSP modules are connected to the first, second, and third prediction networks, respectively.

[0011] According to an embodiment of the present invention, the first ACSP module, the second ACSP module, the third ACSP module, and the fourth ACSP module include a bypass 1×1 convolution, a residual unit, and an attention mechanism module. The bypass 1×1 convolution and the residual unit are connected in parallel, and their outputs are concatenated and input to the attention mechanism module. The attention mechanism module concatenates the image features after performing global max pooling and average pooling operations, then performs convolution and sigmoid function operations, and finally multiplies and accumulates the output features with the input image features to obtain the output features.

[0012] Furthermore, the semi-interactive segmentation algorithm in step S3 adopts the GrabCut algorithm, the steps of which are as follows: S3.1 Input image, mark foreground and background; S3.2 Assign a label to each pixel of the foreground and background; S3.3. Use the K-means algorithm to cluster the foreground and background regions into K types of pixels respectively; S3.4. Use GMM to model the foreground and background; S3.5 Substitute each pixel in the foreground region into the GMM model and calculate the probability that the pixel belongs to the foreground and background regions respectively; S3.6 Calculate the Euclidean distance between all pairs of adjacent pixels in the foreground region to obtain the boundary term; S3.7. Use the maximum flow minimum cut algorithm to obtain the minimum energy value; S3.8 Repeat steps S3.4 to S3.7 until convergence, and output the image.

[0013] Furthermore, in step S5, the threshold value of the Cr channel is 135-160.

[0014] Furthermore, the determination of the corrosion level of the hardware in step S6 is as follows: (1) If the area ratio of the rusted area to the area of ​​the hardware is less than 0.1%, the rust level is determined to be no rust; (2) If the area ratio of the corroded area to the area of ​​the hardware is greater than 0.1% and less than 1%, the corrosion level is determined to be primary corrosion; (3) If the area ratio of the rusted area to the hardware area is greater than 1% and less than 33%, the rust level is judged as medium rust. (4) If the area ratio of the rusted area to the hardware area is greater than 33%, the rust level is judged as high rust.

[0015] (III) Beneficial Effects The beneficial effects of this invention are as follows: A method for detecting and quantifying corrosion of hardware in power line inspection images is proposed. This method combines deep learning with traditional image processing to identify hardware and detect corrosion areas in inspection images. By improving the YOLOv5s feature extraction network and feature fusion network, the detection performance of small target hardware is effectively improved, and the false detection and missed detection of hardware corrosion are reduced. The feature extraction network introduces a Dense network on the basis of the original CSP residual module, and the feature fusion network aggregates the contextual information of different features in a multi-scale fusion manner. At the same time, a channel attention mechanism is introduced to enhance and extract features by assigning different attention weights to different feature channels. This invention uses the GrabCut algorithm and Cr channel threshold segmentation to detect corrosion areas. The degree of hardware corrosion is quantified by the ratio of the area of ​​the corrosion area to the area of ​​the hardware, which solves the problems of low efficiency and high subjectivity of manual visual inspection for corrosion detection of power transmission line hardware. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the metal fitting corrosion detection and quantification method of the present invention; Figure 2 This is a diagram of the deep learning network model structure of the present invention; Figure 3 This is a structural diagram of the first DCSP module of the present invention; Figure 4 This is a structural diagram of the second DCSP module of the present invention; Figure 5 This is a structural diagram of the third DCSP module of the present invention; Figure 6 This is a structural diagram of the ACSP module of the present invention; Figure 7 This is a structural diagram of the attention mechanism module of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Combination Figure 1 A method for detecting and quantifying corrosion of hardware in power line inspection images, comprising the following steps: S1. Collect aerial images of power transmission lines inspected by drones; adjust the resolution of the collected aerial images to 608×608.

[0020] S2. Input the aerial image from step S1 into a deep learning network model, use the deep learning network model to locate the hardware in the image, and then cut out the hardware.

[0021] S3. First, use a semi-interactive segmentation algorithm to perform image segmentation processing on the image cropped in step S2, and then use a morphological optimization algorithm to separate the hardware from the background.

[0022] Preferably, the semi-interactive segmentation algorithm in step S3 adopts the GrabCut algorithm, and the GrabCut algorithm steps are as follows: S3.1 Input image, mark foreground and background; S3.2 Assign a label to each pixel of the foreground and background; S3.3. Use the K-means algorithm to cluster the foreground and background regions into K types of pixels respectively; S3.4. Use GMM to model the foreground and background; S3.5 Substitute each pixel in the foreground region into the GMM model and calculate the probability that the pixel belongs to the foreground and background regions respectively; S3.6 Calculate the Euclidean distance between all pairs of adjacent pixels in the foreground region to obtain the boundary term; S3.7. Use the maximum flow minimum cut algorithm to obtain the minimum energy value; S3.8 Repeat steps S3.4 to S3.7 until convergence, and output the image.

[0023] S4. Convert the segmented hardware equipment image from RGB color space to YCrCb color space, extract the three-channel color image of YCrCb color space, and obtain the grayscale image of the Cr channel.

[0024] S5. Threshold segmentation is performed on the grayscale image of step S4 according to the Cr channel threshold to obtain a binary image of hardware corrosion. The hardware is then judged to be corroded based on the binary image.

[0025] Preferably, the threshold value of the Cr channel in step S5 is 135-160.

[0026] S6. Based on the binary image after threshold segmentation in step S5, perform hardware corrosion quantification, count the number of pixels in the corrosion area and the number of pixels in the hardware area, and determine the hardware corrosion level by calculating the area ratio of the two areas.

[0027] YOLOv5s boasts superior detection performance, significantly outperforming two-stage detection algorithms like Faster R-CNN, and has been widely applied across numerous fields. To further improve the accuracy of hardware identification in transmission line inspection images, this study modifies the feature extraction and fusion networks, training an improved YOLOv5s model as the underlying deep learning network for hardware recognition in aerial images. Figure 2 Step S2 employs a deep learning network model comprising a feature extraction network, a feature fusion network, and a prediction network. The feature extraction network comprises a Focus module, a first CBL module, a first DCSP module, a second CBL module, a second DCSP module, a third CBL module, a third DCSP module, a fourth CBL module, and an SPP module. These modules are sequentially connected to extract image features. The input of the feature fusion network is connected to the four effective feature layers (19×19, 38×38, 76×76, 152×152) of the feature extraction network. The feature fusion network is used for deep fusion of shallow and deep image features. The output of the feature fusion network is connected to the three inputs of the prediction network.

[0028] The Focus module performs self-replication and then slicing to reduce the computational load of the network and speed up the extraction of features from candidate regions. The Focus module is connected to the input image of 608×608×3, and the output features of the Focus module are 304×304×32.

[0029] The CBL module consists of a convolutional layer (Conv), a batch normalization function (Batch Normalization), and a Leaky ReLU activation function.

[0030] The first CBL module consists of convolutional layers, batch normalization, and activation functions. The convolutional layers are 3×3 convolutions with a stride of 2 and 64 filters. The output feature of the first CBL convolutional module is 152×152×64.

[0031] The DCSP module consists of a Cross-Stage Partial (CSP) network and a DenseNet network, enhancing feature reuse and propagation, and effectively avoiding gradient vanishing and gradient exploding. Figure 3 The first DCSP module includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. The first and second convolutional layers are 1×1×64 convolutions, each connected to the input feature of 152×152×64. The third convolutional layer is a 1×1×32 convolution, and the fourth convolutional layer is a 3×3×32 convolution. The third and fourth convolutional layers constitute the first Dense module. The output of the first convolutional layer is concatenated with the output of the fourth convolutional layer to obtain the output of the first Dense module. The output of the first Dense module is then concatenated with the output of the second convolutional layer and input to the fifth convolutional layer. The fifth convolutional layer is a 1×1×64 convolution, and its output feature is 152×152×64. The first DCSP module is used to extract the image feature of 152×152, and its output feature is 152×152×64.

[0032] The second CBL module consists of convolutional layers, batch normalization, and activation functions. The convolutional layers are 3×3 convolutions with a stride of 2 and 128 filters. The output features of the second CBL convolutional module are 76×76×128.

[0033] Combination Figure 4The second DCSP module includes a sixth, seventh, eighth, ninth, and tenth convolutional layer. The sixth and seventh convolutional layers are 1×1×128 convolutions, each connected to the input feature of 76×76×128. The eighth convolutional layer is a 1×1×32 convolution, and the ninth convolutional layer is a 3×3×32 convolution. The eighth and ninth convolutional layers constitute the second Dense module. The output of the sixth convolutional layer is concatenated with the output of the ninth convolutional layer to obtain the output of the second Dense module. The output of the second Dense module is then concatenated with the output of the seventh convolutional layer and input to the tenth convolutional layer. The tenth convolutional layer is a 1×1×128 convolution, and its output feature is 76×76×128. The second DCSP module is used to extract the image feature of 76×76, and its output feature is 76×76×128.

[0034] The third CBL module consists of convolutional layers, batch normalization, and activation functions. The convolutional layers are 3×3 convolutions with a stride of 2 and 256 filters. The output features of the third CBL convolutional module are 38×38×256.

[0035] Combination Figure 5 The third DCSP module includes an eleventh, twelfth, thirteenth, fourteenth, and fifteenth convolutional layer. The eleventh and twelfth convolutional layers are 1×1×256 convolutions, each connected to an input feature of 38×38×256. The thirteenth convolutional layer is a 1×1×64 convolution, and the fourteenth convolutional layer is a 3×3×64 convolution. The thirteenth and fourteenth convolutional layers constitute the third Dense module. The output of the eleventh and fourteenth convolutional layers is concatenated to obtain the output of the third Dense module. The output of the third Dense module is then concatenated with the output of the twelfth convolutional layer and input to the fifteenth convolutional layer. The fifteenth convolutional layer is a 1×1×256 convolution, and its output feature is 38×38×256. The third DCSP module is used to extract image features of 38×38, and its output feature is 38×38×256.

[0036] The fourth CBL module consists of convolutional layers, batch normalization, and activation functions. The convolutional layers are 3×3 convolutions with a stride of 2 and 512 filters. The output features of the fourth CBL convolutional module are 19×19×512.

[0037] The SPP (Small Pyramid Pooling) module includes 5×5 average pooling, 9×9 max pooling, and 13×13 max pooling. The feature map 19×19×512 first undergoes 5×5 max pooling, 9×9 max pooling, and 13×13 max pooling operations, and then a Concat concatenation operation is used to obtain the effective 19×19 image features. By employing max pooling kernels of different sizes, the SPP module significantly separates important contextual features, achieving multi-scale feature fusion.

[0038] like Figure 2 The feature fusion network shown includes a first 1×1 convolution, a second 1×1 convolution, a third 1×1 convolution, a fourth 1×1 convolution, a fifth 1×1 convolution, a sixth 1×1 convolution, a seventh 1×1 convolution, a first ACSP module, a second ACSP module, a third ACSP module, a fourth ACSP module, a first upsampling (2x upsampling), a second upsampling (2x upsampling), a third upsampling (2x upsampling), a fourth upsampling (4x upsampling), a fifth upsampling (4x upsampling), a first downsampling (2x downsampling), a second downsampling (2x downsampling), and a third downsampling (2x downsampling).

[0039] The ACSP module introduces an attention mechanism into the Cross-Stage Partial (CSP) network. By assigning different attention weights to different feature channels, it enhances and extracts features. Figure 6 The ACSP module includes a bypass 1×1 convolution, residual units, and an attention mechanism module. The bypass 1×1 convolution and residual units are connected in parallel, and their outputs are concatenated and input to the attention mechanism module. Figure 7 The attention mechanism module concatenates the image features after performing global max pooling and average pooling operations, then performs convolution and sigmoid function operations, and finally multiplies and accumulates them with the input image features to obtain the output features.

[0040] The inputs of the first ACSP module, the second 1×1 convolution, the fourth 1×1 convolution, and the seventh 1×1 convolution are connected to the effective feature layers (19×19, 38×38, 76×76, 152×152), respectively. The output of the first ACSP module is connected to the first 1×1 convolution. The first 1×1 convolution, after being upsampled and concatenated with the output of the second 1×1 convolution, is input to the third 1×1 convolution. The first ACSP module, after being upsampled by the fourth convolution, and the third 1×1 convolution, after being upsampled by the second convolution and concatenated with the output of the fourth 1×1 convolution, are input to the fifth 1×1 convolution. The second 1×1 convolution, after being upsampled by the fifth convolution, ... The output of the fifth 1×1 convolution is concatenated with the output of the seventh 1×1 convolution after third upsampling, and then concatenated with the fifth 1×1 convolution after first downsampling before being input to the second ACSP module. The output of the second ACSP module is concatenated with the output of the third 1×1 convolution after second downsampling before being input to the third ACSP module. The output of the third ACSP module is concatenated with the output of the first 1×1 convolution after third downsampling before being input to the fourth ACSP module. The outputs of the second, third, and fourth ACSP modules are connected to the first, second, and third prediction networks, respectively. The feature fusion network aggregates contextual information of different features in a multi-scale feature fusion manner, improving the ability to acquire global information containing different scales.

[0041] To further quantify the degree of corrosion of the hardware, the corrosion level determination in step S6 is as follows: (1) If the area ratio of the rusted area to the area of ​​the hardware is less than 0.1%, the rust level is determined to be no rust; (2) If the area ratio of the corroded area to the area of ​​the hardware is greater than 0.1% and less than 1%, the corrosion level is determined to be primary corrosion; (3) If the area ratio of the rusted area to the hardware area is greater than 1% and less than 33%, the rust level is judged as medium rust. (4) If the area ratio of the rusted area to the hardware area is greater than 33%, the rust level is judged as high rust.

[0042] In summary, this invention provides a method for detecting and quantifying corrosion of hardware in power line inspection images. It employs a combination of deep learning and traditional image processing to identify hardware and detect corrosion areas in inspection images. By improving the YOLOv5s feature extraction and feature fusion networks, the detection performance of small-target hardware is effectively enhanced, reducing false positives and false negatives in hardware corrosion detection. The feature extraction network introduces a Dense network based on the original CSP residual module, and the feature fusion network aggregates contextual information of different features through multi-scale fusion. Simultaneously, a channel attention mechanism is introduced, assigning different attention weights to different feature channels to enhance and extract features. This invention utilizes the GrabCut algorithm and Cr channel threshold segmentation for corrosion area detection, quantifying the degree of hardware corrosion by comparing the area ratio of the corrosion area to the hardware area. This solves the problems of low efficiency and high subjectivity associated with manual visual inspection of power transmission line hardware corrosion detection.

[0043] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting and quantifying corrosion of hardware in power line inspection images, characterized in that, Includes the following steps: S1. Collect aerial images of power transmission lines inspected by drones; S2. Input the aerial image from step S1 into a deep learning network model, use the deep learning network model to locate the hardware in the image, and then cut out the hardware. S3. First, use a semi-interactive segmentation algorithm to perform image segmentation processing on the image cropped in step S2, and then use a morphological optimization algorithm to separate the hardware from the background. S4. Convert the segmented hardware equipment image from RGB color space to YCrCb color space, extract and separate the three-channel color image of YCrCb color space, and obtain the grayscale image of Cr channel. S5. Perform threshold segmentation on the grayscale image of step S4 according to the Cr channel threshold to obtain a binary image of hardware corrosion. Determine whether the hardware is corroded based on the binary image. S6. Based on the binary image after threshold segmentation in step S5, perform hardware corrosion quantification, count the number of pixels in the corrosion area and the number of pixels in the hardware area, and determine the hardware corrosion level by calculating the area ratio of the two areas.

2. The method for detecting and quantifying metal corrosion in power inspection images as described in claim 1, characterized in that: The resolution of the aerial image acquired in step S1 is adjusted to 608×608.

3. The method for detecting and quantifying metal corrosion in power inspection images as described in claim 2, characterized in that, The deep learning network model used in step S2 includes a feature extraction network, a feature fusion network, and a prediction network; The feature extraction network includes a Focus module, a first CBL module, a first DCSP module, a second CBL module, a second DCSP module, a third CBL module, a third DCSP module, a fourth CBL module, and an SPP module. The Focus module, the first CBL module, the first DCSP module, the second CBL module, the second DCSP module, the third CBL module, the third DCSP module, the fourth CBL module, and the SPP module are sequentially connected to extract image features. The input of the feature fusion network is connected to the four effective feature layers (19×19, 38×38, 76×76, 152×152) of the feature extraction network. The feature fusion network is used for deep fusion of shallow and deep image features. The output of the feature fusion network is connected to the three inputs of the prediction network.

4. The method for detecting and quantifying metal corrosion in power line inspection images as described in claim 3, characterized in that, The Focus module is connected to the input image 608×608×3, and the output feature of the Focus module is 304×304×32. The first CBL module consists of a convolutional layer, a batch normalization function, and an activation function. The convolutional layer is a 3×3 convolution with a stride of 2. The output feature of the first CBL convolutional module is 152×152×64. The first DCSP module is used to extract image features of 152×152, and the output features of the first DCSP module are 152×152×64. The second CBL module consists of a convolutional layer, a batch normalization function, and an activation function. The convolutional layer is a 3×3 convolution with a stride of 2. The output feature of the second CBL convolutional module is 76×76×128. The second DCSP module is used to extract image features of 76×76, and the output features of the second DCSP module are 76×76×128. The third CBL module consists of a convolutional layer, a batch normalization, and an activation function. The convolutional layer is a 3×3 convolution with a stride of 2. The output feature of the third CBL convolutional module is 38×38×256. The third DCSP module is used to extract image features of 38×38, and the output features of the third DCSP module are 38×38×256. The fourth CBL module consists of a convolutional layer, a batch normalization function, and an activation function. The convolutional layer is a 3×3 convolution with a stride of 2. The output feature of the fourth CBL convolutional module is 19×19×512.

5. A method for detecting and quantifying metal corrosion in power line inspection images as described in claim 3 or 4, characterized in that, The first DCSP module includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. The first and second convolutional layers are 1×1×64 convolutions, which are connected to the input feature of 152×152×64 respectively. The third convolutional layer is a 1×1×32 convolution, and the fourth convolutional layer is a 3×3×32 convolution. The third and fourth convolutional layers constitute the first Dense module. The output of the first convolutional layer is concatenated with the output of the fourth convolutional layer to serve as the output of the first Dense module. The output of the first Dense module is concatenated with the output of the second convolutional layer and then input to the fifth convolutional layer. The fifth convolutional layer is a 1×1×64 convolution, and the output feature of the fifth convolutional layer is 152×152×64. The second DCSP module includes a sixth, seventh, eighth, ninth, and tenth convolutional layer. The sixth and seventh convolutional layers are 1×1×128 convolutions, each connected to an input feature of 76×76×128. The eighth convolutional layer is a 1×1×32 convolution, and the ninth convolutional layer is a 3×3×32 convolution. The eighth and ninth convolutional layers constitute the second Dense module. The output of the sixth convolutional layer is concatenated with the output of the ninth convolutional layer to serve as the output of the second Dense module. The output of the second Dense module is then concatenated with the output of the seventh convolutional layer and input to the tenth convolutional layer. The tenth convolutional layer is a 1×1×128 convolution, and its output feature is 76×76×128. The third DCSP module includes an eleventh, twelfth, thirteenth, fourteenth, and fifteenth convolutional layer. The eleventh and twelfth convolutional layers are 1×1×256 convolutions, each connected to an input feature of 38×38×256. The thirteenth convolutional layer is a 1×1×64 convolution, and the fourteenth convolutional layer is a 3×3×64 convolution. The thirteenth and fourteenth convolutional layers constitute the third Dense module. The output of the eleventh convolutional layer is concatenated with the output of the fourteenth convolutional layer to serve as the output of the third Dense module. The output of the third Dense module is concatenated with the output of the twelfth convolutional layer and then input to the fifteenth convolutional layer. The fifteenth convolutional layer is a 1×1×256 convolution, and its output feature is 38×38×256.

6. A method for detecting and quantifying metal corrosion in power line inspection images as described in any one of claims 3-5, characterized in that, The feature fusion network includes a first 1×1 convolution, a second 1×1 convolution, a third 1×1 convolution, a fourth 1×1 convolution, a fifth 1×1 convolution, a sixth 1×1 convolution, a seventh 1×1 convolution, a first ACSP module, a second ACSP module, a third ACSP module, a fourth ACSP module, a first upsampling module, a second upsampling module, a third upsampling module, a fourth upsampling module, a fifth upsampling module, a first downsampling module, a second downsampling module, and a third downsampling module; The inputs of the first ACSP module, the second 1×1 convolution, the fourth 1×1 convolution, and the seventh 1×1 convolution are connected to the effective feature layers (19×19, 38×38, 76×76, 152×152) of the feature extraction network, respectively. The output of the first ACSP module is connected to the first 1×1 convolution. The first 1×1 convolution is concatenated with the output of the second 1×1 convolution after first upsampling and then input to the third 1×1 convolution. The first ACSP module is concatenated with the output of the third 1×1 convolution after fourth upsampling and second upsampling and then input to the fifth 1×1 convolution. The second 1×1 convolution is concatenated with the output of the seventh 1×1 convolution after fifth upsampling and third upsampling, and then concatenated with the fifth 1×1 convolution after first downsampling and then input to the second ACSP module. The output of the second ACSP module is concatenated with the output of the third 1×1 convolution after the second downsampling and then input to the third ACSP module; The output of the third ACSP module is concatenated with the output of the first 1×1 convolution after the third downsampling and then input to the fourth ACSP module. The outputs of the second, third, and fourth ACSP modules are connected to the first, second, and third prediction networks, respectively.

7. The method for detecting and quantifying metal corrosion in power inspection images as described in claim 6, characterized in that, The first, second, third, and fourth ACSP modules include a bypass 1×1 convolution, a residual unit, and an attention mechanism module. The bypass 1×1 convolution and the residual unit are connected in parallel, and their outputs are concatenated and input to the attention mechanism module. The attention mechanism module concatenates the image features after performing global max pooling and average pooling operations, then performs convolution and sigmoid function operations, and finally multiplies and accumulates the output features with the input image features to obtain the output features.

8. The method for detecting and quantifying metal corrosion in power line inspection images as described in claim 1, characterized in that, The semi-interactive segmentation algorithm in step S3 uses the GrabCut algorithm, and the steps of the GrabCut algorithm are as follows: S3.1 Input image, mark foreground and background; S3.2 Assign a label to each pixel of the foreground and background; S3.

3. Use the K-means algorithm to cluster the foreground and background regions into K types of pixels respectively; S3.

4. Use GMM to model the foreground and background; S3.5 Substitute each pixel in the foreground region into the GMM model and calculate the probability that the pixel belongs to the foreground and background regions respectively; S3.6 Calculate the Euclidean distance between all pairs of adjacent pixels in the foreground region to obtain the boundary term; S3.

7. Use the maximum flow minimum cut algorithm to obtain the minimum energy value; S3.8 Repeat steps S3.4 to S3.7 until convergence, and output the image.

9. The method for detecting and quantifying metal corrosion in power inspection images as described in claim 1, characterized in that, In step S5, the threshold value for the Cr channel is 135-160.

10. The method for detecting and quantifying metal corrosion in power line inspection images as described in claim 1, characterized in that, The determination of the corrosion level of the hardware in step S6 is as follows: (1) If the area ratio of the rusted area to the area of ​​the hardware is less than 0.1%, the rust level is determined to be no rust; (2) If the area ratio of the corroded area to the area of ​​the hardware is greater than 0.1% and less than 1%, the corrosion level is determined to be primary corrosion; (3) If the area ratio of the rusted area to the hardware area is greater than 1% and less than 33%, the rust level is judged as medium rust. (4) If the area ratio of the rusted area to the hardware area is greater than 33%, the rust level is judged as high rust.

Images