Rail-mounted robot inspection image processing method

Abstract

The invention provides an on-rail robot inspection image processing method, and relates to the field of image enhancement. According to the electric power system defect and fault image recognition and segmentation process, defect image data of a wire, an insulator and a pole under different environment conditions are collected, a filtering enhancement module is constructed, noise in an input image is eliminated through pooling and up-sampling, and beneficial features are enhanced; constructing a multi-order cross-scale module, and extracting high-level global semantic information and low-level detail texture information through adaptive average pooling; the method comprises the following steps of: constructing a decoder, fusing and restoring feature information of different levels, and finally using a Sigmoid activation function to output a segmented image, so that an electric power system defect image shot by an inspection robot is accurately processed, and the target recognition capability of a model when the model faces recognition targets of different scales and complex scenes are changed is improved.

Description

Technical Field

[0001] The invention belongs to the field of image enhancement, and in particular relates to a rail-hanging robot inspection image processing method. Background Art

[0002] Traditional inspections of distribution rooms mainly rely on manual on-site inspections, which is not only time-consuming and labor-intensive, but also poses safety hazards in severe weather conditions. By using inspection robots equipped with high-precision cameras and combining them with deep learning image processing enhancement technology, automated inspections of transmission lines and ancillary facilities can be achieved, which not only improves the accuracy and efficiency of inspections, but also greatly reduces the risks faced by staff.

[0003] As power facilities age, their maintenance costs and frequency are also increasing. Through inspection robots and image recognition and segmentation technology, the status changes of power facilities can be monitored in real time, and early warnings can be given for abnormal situations. Deep learning algorithms can be used to identify and analyze images, determine cable fault status, and predict equipment failure trends in advance. This can not only reduce unexpected downtime, but also reduce maintenance costs and extend the service life of equipment.

[0004] By analyzing and processing the high-definition images taken by the inspection robot, the fault location can be discovered in a timely manner and the cause of the fault can be analyzed. This can significantly improve the efficiency and safety of power facility maintenance and provide strong support for the long-term stable operation of the power system. This is of great significance for reducing unexpected power outages and ensuring the reliability and continuity of power supply. Summary of the invention

[0005] The present invention provides a method for processing inspection images of a rail-hanging robot, aiming to propose a filtering and strengthening module, which is helpful for filtering interference information in an input image, and strengthening features in the image that are beneficial to recognition, which is helpful for subsequent feature processing and improves the accuracy of model recognition; proposes a multi-order cross-scale module, uses a multi-scale pyramid structure to extract features of input vectors at different scales, makes full use of high-level semantic information and low-level detail texture information in the vector, and further improves the accuracy of segmentation; proposes a decoder module, fuses and extracts feature information of different scales, and improves the generalization ability of the model and the ability to recognize targets during detection.

[0006] The present invention aims to propose an image segmentation model for a power system inspection robot and provide a method for processing images of a power system inspection robot, comprising the following steps:

[0007] S1. Production of power system fault image dataset: sort out the fault images taken by the inspection robot during inspection in the power system production environment to obtain power system defect fault images, and annotate the images of several scenes with the highest frequency, including images of six major scenes: insulator damage, insulator cracks, conductor breakage, conductor strand breakage, conductor corrosion, and tower rust. Annotate the corresponding defects and faults in the images. After the annotation is completed, the power system defect and fault dataset is obtained;

[0008] S2, build a filter enhancement module, including 3×3 convolution, upsampling, downsampling and skip connection;

[0009] S3, build a multi-stage cross-scale module, including adaptive average pooling, filter enhancement module and upsampling;

[0010] S4. Build an encoder, including filter enhancement module, multi-order cross-scale module, skip connection and 3×3 convolution;

[0011] S5. Build a decoder, including feature splitting, skip connection and 3×3 convolution;

[0012] S6. Construct a segmentation network model for fault defect recognition in a power system, which consists of input, encoder, decoder, full connection, activation function and output in sequence;

[0013] S7, training a power system fault defect recognition and segmentation network model, using a power system fault image dataset to train a power system fault defect recognition and segmentation network model;

[0014] S8. Power system robot inspection image processing, obtaining the power system robot inspection image that needs to be identified, inputting it into the power system fault defect identification and segmentation network model, and obtaining the power system robot inspection image after identification and segmentation.

[0015] Preferably, in step S1, a power system inspection robot is used to capture fault images in the production environment, including images of insulator damage, insulator cracks, conductor breakage, conductor strand breakage, conductor corrosion, and tower rust caused by moisture, wind, and exposure under natural conditions; the image clarity is high, the features of each defect in the image are obvious, and the number of samples of each defect type in the collected images is sufficient and uniform, thereby improving the accuracy and robustness of model recognition; at the same time, for each defect type, images are collected under various lighting conditions such as sunny days, cloudy days, and nights including infrared and night vision shooting, so that the model can adapt to complex natural lighting environments and further improve the recognition performance of the model; in addition, for the collected defect image data, the image annotation program Labelme is used to mark the defect area in the image, and the defect area boundary box is drawn, so that the model can identify the location of the defect in the image.

[0016] Preferably, in step S2, for the filter enhancement module, the original image I of the power system robot inspection is input EL , I EL ∈R H×W×3 , where H is the height of the image, W is the width of the image, and the image includes three dimensions: red, green, and blue. First, the input image feature matrix is ​​convolved to obtain Where w is the size of the input feature matrix, p is the number of zero-padding layers, the convolution kernel size is k, and the step size is S; then the feature vector is average pooled twice F = avgpool(w′); at this time, the size of the feature vector is reduced, and the module can filter the noise therein while strengthening the features that are beneficial to feature recognition; then, the feature vector is upsampled twice F′ = Upsample(conn(F1+F2)), where conn() represents a jump connection, F1 and F2 represent the current feature and the feature of the corresponding size before the module performs average pooling, respectively; after upsampling, the feature vector is restored to the same size as the feature vector corresponding to the input image and output.

[0017] Preferably, the filtering and strengthening module proposed in step S2 is used to filter and strengthen the input features. The power system is in a complex environment, and the image content in its background part is often random. This part of the noise has a certain interference effect on the identification of power system faults, which is not conducive to model recognition. The filtering and strengthening module continuously extracts high-order features in the vector through pooling operations, gradually reducing the interference of noise on the image content while still retaining the target information in the image; in addition, the extracted high-order features are gradually restored and strengthened through continuous upsampling, thereby enhancing the distinguishability of the features, and improving the efficiency of model training by reducing unnecessary information, thereby reducing the computing resources required for the model to process features.

[0018] Preferably, in step S3, for the multi-order cross-scale module, the output feature I obtained by the input filtering and strengthening module is FRU , I FRU ∈R H×W×3 , followed by 4 adaptive average pooling of different scales to obtain feature vectors z of 4 sizes k , for z k For each value (i, j) in Where n is the pooling window size; then z k The features are further extracted through the filtering and strengthening modules to obtain z′ k :z′ k =FRU(z k ), where FRU() represents the filter enhancement unit.

[0019] Preferably, in step S3, for the obtained feature vectors z′ of different sizes k , use the upsampling method to restore its size; for the eigenvalue I(x, y) at the point (x, y),

[0020] , where (x1, y1), (x1, y2), (x2, y1), (x2, y2) are the four coordinate points adjacent to the point (x, y), x1≤x≤x2, y1≤y≤y1.

[0021] Preferably, in step S4, for the encoder module, the input power system robot inspection image is sequentially subjected to feature extraction by a filtering enhancement module and a multi-order cross-scale module, and is equipped with a jump connection, and the feature vector of the original input image is merged with the feature vector output by the multi-order cross-scale module in the channel dimension, and the rich detail features in the original input image are fused with the feature vector output by the multi-order cross-scale module, and then a 3×3 convolution is performed as the output of the encoder module.

[0022] Preferably, the multi-order cross-scale module proposed in step S3 is used to process the input features. The fault defects of the power system are diverse, and the fault targets in the image are often of uncertain sizes. The multi-order cross-scale module is used to extract feature maps of different scales from the input features, so that the model can maintain a high segmentation accuracy when facing a large target scale span. In addition, feature mappings at different levels can capture information at different abstract levels from low-level edge texture information to high-level global semantic information, thereby improving the adaptability and generalization ability of the model in complex and changing scenarios.

[0023] Preferably, in step S5, for the decoder module, the encoded feature F output by the encoder including high-order semantic features and low-order detail features is input. Encoder , calculate the decoded decoding feature F Decoder Specifically, first perform feature F Encoder Split to get F1, F2 = Split (F Decoder ), then perform feature splitting on F2 to obtain F 21 , F 22 = split(F2); finally, F is calculated Decoder =Conv(Conn(F Encoder ,merge(merge(F1,F 21 ), F 22 ))), where merge() is the feature vector fusion.

[0024] Preferably, for the decoder structure in step S5, the output features of the encoder are divided into multiple branches and recombined at different stages, which effectively fuses feature information at different levels and gradually refines the feature representation so that the processing results contain rich high-order semantic information and low-order detail information at the same time, thereby improving the recognition ability of the model. In addition, through the fusion of multi-scale feature information, the model can better cope with recognition targets of different scales and complex scene changes, improve robustness, and help retain the detail information in the input features, which plays an important role in restoring fine-grained details.

[0025] Preferably, in step S6, the input image I passes through the encoder module and the decoder module in sequence to obtain a feature vector I Decoder ; Secondly, for the output of the decoder, the Sigmoid function is used to activate it after full connection, and the value of the feature vector is mapped to between 0 and 1 as the model's prediction probability for the target. The specific calculation method is Thus, the probability distribution of each pixel in the image belonging to the segmentation target is obtained. The closer the value of the pixel is to 1, the higher the confidence that the pixel is considered to be the segmentation target. The final segmentation result can be obtained by threshold filtering.

[0026] Preferably, in step S7, the acquired power system robot inspection images are divided into a training set and a test set in a ratio of 3:1; in the model evaluation stage, accuracy, precision, recall rate and F1 score are used as evaluation indicators of model segmentation performance.

[0027] In summary, due to the adoption of the technical scheme, compared with the prior art, the beneficial effects of the present invention are as follows: the present invention proposes a defect image recognition and segmentation processing method for images taken by an electric power system inspection robot, including a filtering enhancement module, a multi-order cross-scale module and a decoder module. The filtering enhancement module filters the noise factors in the input image and enhances the beneficial features. The multi-order cross-scale module extracts high-level global semantic features and low-level detail texture features in the vector. The decoder module efficiently fuses and retains high-level and low-level features, thereby improving the model's target recognition capability when facing recognition targets of different scales and complex scene changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a step diagram of an image processing method for inspection by a rail-hanging robot.

[0029] Figure 2 This is the FRU structure diagram of the filter enhancement module.

[0030] Figure 3 This is a diagram of the multi-stage scale module structure.

[0031] Figure 4 This is the encoder structure diagram.

[0032] Figure 5 This is the decoder structure diagram. DETAILED DESCRIPTION

[0033] The present invention discloses an image processing method applied to an electric power system inspection robot. The core of the method is to introduce a filtering enhancement module, which can effectively eliminate irrelevant interference in the input image and highlight those features that are crucial to recognition, thereby optimizing the subsequent feature processing process and enhancing the accuracy of model recognition; a multi-order cross-scale module is designed, and a multi-scale pyramid architecture is used to capture the features of input data at different levels, ensuring that both high-level semantic information and low-level detail textures are fully utilized, thereby improving the accuracy of the segmentation task; in addition, a decoder module is proposed, which aims to integrate and refine and restore feature information from different scales, thereby enhancing the adaptability of the model and the ability to identify the target object during the detection process, and finally generating a probability distribution of each pixel through an activation function, thereby realizing the accurate positioning of the defect position in the image of the electric power system inspection robot. The technical solution in the embodiment of the present invention will be clearly and completely described below, which specifically includes the following steps, such as Figure 1 shown.

[0034] The fault images taken by the inspection robot during its inspection in the power system production environment were sorted out to obtain a power system defect fault image dataset. The size of a single image is 512×512×3, and the entire collection process takes one month. The six scene images with the highest fault frequency are annotated with the corresponding fault locations in the images using the Lableme software. The six major scenes include insulator damage, insulator cracks, conductor breakage, conductor strand breakage, conductor corrosion, and tower rust; each class contains 320 images, and the training set and test set are divided in a 4:1 ratio.

[0035] A filter enhancement module FRU is constructed, which includes two 3×3 convolution operations, two average pooling operations, two upsampling operations and four skip connections.

[0036] Further, a filtering enhancement module is constructed, and its structure is as follows Figure 2 As shown, the input power system robot inspection original image I EL , I EL ∈R H×W×3 , where H = W = 512, first convolution operation is performed on the input image feature matrix to obtain Where w is the size of the input feature matrix, the number of zero-filling layers is p=2, the convolution kernel size is k=3×3, and the step size is s=2. Then, the feature vector is averagely pooled twice F=avgpool(W). At this time, the feature vector size is reduced, and the first pooling results in Where H1=W1=256, and the feature vector is obtained after the second pooling Where H2 = W2 = 128, then the feature vector is upsampled twice U =Upsample(conn(F1+F2)), where F1 and F2 represent the current feature and the feature of the corresponding size before the module performs average pooling, respectively. After upsampling, the feature vector is restored to the same size as the feature vector corresponding to the input image and output.

[0037] Construct a multi-stage cross-scale module, which includes an adaptive average pooling operation, four filtering enhancement modules and an upsampling operation.

[0038] Furthermore, a multi-order cross-scale module is constructed, whose structure is as follows: Figure 3 As shown, the output feature I obtained by the input filtering and strengthening module FRU , I FRU ∈R 512×512×3 , followed by 4 adaptive average pooling of different scales to obtain feature vectors z of 4 sizes k , for z k For each value (i, j) in 1≤k≤4, where n is the pooling window size. In the four adaptive average pooling, the pooling window sizes are n1=2, n2=4, n3=8, and n4=16, respectively, and the pooled feature vector z1∈R 256×256×16 , z2∈R 128×128×16 , z3∈R 64×64×16 , z4∈R 32×32×16 , and then k The features are further extracted through the filtering and strengthening modules to obtain z k :z′ k =FRU(z k ), where FRU() represents the filter enhancement module. k (1≤k≤4), use upsampling to convert the feature vectors Z′ of four different scales into k Restore to the same size, the size after upsampling is denoted as W out =W in ×factor, H out =H in ×dactor, in the 4 upsampling, the factors are 2, 4, 8, and 16 respectively, and the 4 F Up ∈R 512×512×16 Merge in the channel dimension to get F Merge ∈R 512×512×64 is the output of the multi-stage cross-scale module.

[0039] Construct an encoder consisting of a filter enhancement module, a multi-stage cross-scale module, a skip connection, and a 3×3 convolution operation.

[0040] Furthermore, the encoder module is constructed, and its structure is as follows Figure 4 As shown, inside the encoder, the original image vector I EL ∈R 512 ×512×3 After the input is first processed by the filtering and strengthening module, the feature vector F is obtained by filtering the noise and enhancing the useful features. FRU ∈R 512×512×3， Then, the multi-order cross-scale module is input to extract the features of different scales in the vector, and the feature vector F is obtained which combines high-level semantic features and low-level detail features. M ∈R 512×512×64 , followed by the original input I EL Same as F M Merge in the channel dimension through skip connection to get F′ M ∈R 512×512×67 Finally, the features are fused using a convolution with a kernel size of k = 3 × 3 to obtain the encoder output F Encoder ∈R 512×512×128 .

[0041] Construct a decoder, which includes two feature splits, a skip connection and a 3×3 convolution operation.

[0042] Further, a decoder module is constructed, whose structure is as follows Figure 5 As shown, the feature vector F output by the encoder is input into the decoder Encoder ∈R 512×512×128 First, perform feature splitting to obtain F1, F2 = Split (F Decoder ), F1, F2∈R 512×512×64 , then perform feature splitting on F2 to obtain F 21 , F 22 = split(F2), F 21 , F 22 ∈R 512×512×32 , then, F1, F 21 Add and merge pixel by pixel to get F′∈R 512×512×64 , F′ and F 22 Then add and merge pixel by pixel to get F″∈R 512×512×64 , finally, F″ and F Encoder After merging in the channel dimension, k = 3 × 3 convolution is performed for feature extraction to obtain FD ecoder∈R 512×512×128.

[0043] A segmentation network model for fault defect identification in power system is constructed, which consists of input, an encoder, a decoder, a full connection, a Sigmod activation function and output in sequence.

[0044] Furthermore, the decoder input feature F Decoder ∈R 512×512×128 Perform a full connection operation to map the features of all channels to a single channel and obtain F conn ∈R 512×512×1 , and then the segmented image is output after activation by the activation function Sigmoid.

[0045] Train the power system fault defect recognition and segmentation network model, and use the power system fault image dataset to train the power system fault defect recognition and segmentation network model.

[0046] Furthermore, the platform system used for training the model is Ubuntu system, the language is Python3.9.11, and the PyTorch deep learning framework is used for training. The hardware platform is NVIDIA RTX 3090 (24G video memory). The data set contains 1920 images of six major scenes: insulator damage, insulator cracks, conductor breakage, conductor strand breakage, conductor corrosion, and tower rust. The training set and test set are divided into 4:1 ratio; during the training process, Adam is used as the optimizer, and the initial learning rate is 0.001. After every 60 rounds of training, the learning rate decays to the original 0.1, and a total of 300 rounds of training are conducted.

[0047] The power system robot inspection image is processed to obtain a single power system robot inspection image that needs to be identified, and is input into the power system fault defect recognition and segmentation network model to obtain the power system robot inspection image after recognition and segmentation.

Claims

1. A rail-hanging robot inspection image processing method, characterized in that: The following steps are involved: S1. Production of power system fault image dataset: sort out the fault images taken by the inspection robot during inspection in the power system production environment, obtain power system defect fault images, and annotate the images of several scenes with the highest frequency, including images of six major scenes: insulator damage, insulator cracks, conductor breakage, conductor strand breakage, conductor corrosion, and tower rust. Annotate the corresponding defects and faults in the images; after the annotation is completed, obtain the power system defect and fault dataset; S2, build a filter enhancement module, including 3×3 convolution, upsampling, downsampling and skip connection; S3, build a multi-stage cross-scale module, including adaptive average pooling, filter enhancement module and upsampling; S4. Build an encoder, including filter enhancement module, multi-order cross-scale module, skip connection and 3×3 convolution; S5. Build a decoder, including feature splitting, skip connection and 3×3 convolution; S6. Construct a segmentation network model for fault defect recognition in a power system, which consists of input, encoder, decoder, full connection, activation function and output in sequence; S7, training a power system fault defect recognition and segmentation network model, using a power system fault image dataset to train a power system fault defect recognition and segmentation network model; S8. Power system robot inspection image processing, obtaining the power system robot inspection image that needs to be identified, inputting it into the power system fault defect identification and segmentation network model, and obtaining the power system robot inspection image after identification and segmentation.

2. The inspection image processing method of a rail-hanging robot according to claim 1, characterized in that: In step S1, a power system inspection robot is used to capture fault images in the production environment, including images of insulator damage, insulator cracks, conductor breakage, conductor strand breakage, conductor corrosion, and tower rust caused by moisture, wind, and exposure under natural conditions; the image clarity is high, the features of each defect in the image are obvious, and the number of samples of each defect type in the collected images is sufficient and uniform, thereby improving the accuracy and robustness of model recognition; at the same time, for each defect type, images are collected under various lighting conditions such as sunny days, cloudy days, and nights including infrared and night vision shooting, so that the model can adapt to complex natural lighting environments and further improve the recognition performance of the model; in addition, for the collected defect image data, the image annotation program Labelme is used to mark the defect area in the image and draw the defect area boundary box, so that the model can recognize the location of the defect in the image.

3. A rail-hanging robot inspection image processing method according to claim 2, characterized in that: In step S2, for the filtering enhancement module, the original image I of the power system robot inspection is input EL , I EL ∈R H×W×3， Where H is the height of the image, W is the width of the image, and the image includes three dimensions: red, green, and blue. First, the input image feature matrix is ​​convolved to obtain Where w is the size of the input feature matrix, p is the number of zero-padding layers, the convolution kernel size is k, and the step size is S; then the feature vector is average pooled twice F = avgpool (w); at this time, the feature vector size is reduced, the module can filter the noise therein, and at the same time strengthen the features that are beneficial to feature recognition; then, the feature vector is upsampled twice F′ = Upsample (conn (F1 + F2)), where conn () represents a skip connection, and F1 and F2 represent the current feature and the feature of the corresponding size before the module performs average pooling, respectively; After upsampling, the feature vector is restored to the same size as the corresponding feature vector of the input image and output.

4. A rail-hanging robot inspection image processing method according to claim 3, characterized in that: In step S3, for the multi-order cross-scale module, the output feature I obtained by the input filtering and strengthening module FRU , I FRU ∈R H×W×3 , followed by 4 adaptive average pooling of different scales to obtain feature vectors z of 4 sizes k , for z k For each value (i, j) in Where n is the pooling window size; then z k The features are further extracted through the filtering and strengthening modules to obtain z′ k :z′ k =FRU(z k ), where FRU() represents the filter enhancement unit.

5. The method for processing images of a rail-hanging robot inspection according to claim 4, characterized in that: In step S3, for the feature vectors z′ of different sizes obtained k , use the upsampling method to restore its size; for the eigenvalue I(x, y) at the point (x, y), , Among them, (x1, y1), (x1, y2), (x2, y1), and (x2, y2) are the four coordinate points adjacent to the point (x, y), x1≤x≤x2, y1≤y≤y1.

6. A rail-hanging robot inspection image processing method according to claim 5, characterized in that: In step S4, for the encoder module, the input power system robot inspection image is sequentially extracted through the filtering enhancement module and the multi-order cross-scale module, and is equipped with a jump connection. The feature vector of the original input image is merged with the feature vector output by the multi-order cross-scale module in the channel dimension, and the rich detail features in the original input image are fused with the feature vector output by the multi-order cross-scale module, and then a 3×3 convolution is performed as the output of the encoder module.

7. The method for processing inspection images of a rail-hanging robot according to claim 6, characterized in that: In step S5, for the decoder module, the encoded feature F containing high-order semantic features and low-order detail features output by the encoder is input Encoder , calculate the decoded decoding feature F Decoder Specifically, first perform feature F Encoder Split to get F1, F2 = Split (F Decoder ), then perform feature splitting on F2 to obtain F 21 , F 22 = split(F2); finally, F is calculated Decoder =Conv(Conn(F Encoder ,merge(merge(F1,F 21 ), F 22 ))), where merge() is the feature vector fusion.

8. The method for processing inspection images of a rail-hanging robot according to claim 7, characterized in that: In step S6, the input image I passes through the encoder module and the decoder module in sequence to obtain the feature vector I Decoder ; Secondly, for the output of the decoder, the Sigmoid function is used to activate it after full connection, and the value of the feature vector is mapped to between 0 and 1 as the model's prediction probability for the target. The specific calculation method is Thus, the probability distribution of each pixel in the image belonging to the segmentation target is obtained. The closer the value of the pixel is to 1, the higher the confidence that the pixel is considered to be the segmentation target. The final segmentation result can be obtained by threshold filtering.

9. The method for processing images of a rail-hanging robot inspection according to claim 1, characterized in that: In step S7, the acquired power system robot inspection images are divided into training set and test set in a ratio of 3:1; in the model evaluation stage, accuracy, precision, recall rate and F1 score are used as evaluation indicators of model segmentation performance.

Images