Insulator defect identification method

Abstract

The invention discloses an insulator defect identification method, which is characterized in that information such as the size, attitude and key point of an identification target is predicted from the center point of the target, and the feature extraction process is realized through a full convolutional network; a thermodynamic diagram about the center of the target is generated after feature extraction, and then the center of the target is positioned by finding a local maximum value in the thermodynamic diagram; features are used at peak positions to infer the size and pose of a target. The multi-resolution feature map is used for target category prediction, so that the algorithm provided by the invention can meet the requirements of deformation and constant scale change of the image acquired by the unmanned aerial vehicle, and image and adaptive scale feature extraction and other preprocessing are integrated before target recognition, so that the target recognition accuracy is improved. Therefore, the identification accuracy of the model on insulator defects in different environments is improved.

Description

Technical Field

[0001] The invention belongs to the field of insulator defect identification, and in particular, relates to an insulator defect identification method. Background Art

[0002] Insulators, as a special and critical insulation control component, play an important role in the transmission and distribution network. Accurate identification and location of insulator defects are of great significance in the field of power inspection. With the development of drone technology, the use of drones for transmission line inspection has been widely used.

[0003] In recent years, more and more domestic and foreign researchers have devoted themselves to applying deep learning models to UAV power inspection based on computer vision, and have achieved many results. At present, insulator defect recognition algorithms based on deep learning models in UAV inspection images are mainly divided into two categories: detection frame (Anchor) recognition method and detection frame-free (Anchor-free) recognition method.

[0004] The detection box method first finds a pre-selected box representing the entire target, and then classifies and regresses the pre-selected box to further classify, identify and bounding box the target.

[0005] The above object recognition algorithms based on detection boxes need to enumerate all possible object positions and scales, and classify each position to identify the object, resulting in a waste of computing resources. In addition, this type of method is difficult to perceive objects with special shapes, resulting in low recognition rates for objects of arbitrary shapes and scales, and objects with severe shape changes.

[0006] Most detection box-free recognition methods use key point estimation-based methods to inspire the target detection and recognition process, which recognizes a single area of ​​the target instead of considering the target as a whole. Subsequently, the various areas belonging to the same target will be combined together through trainable post-processing methods.

[0007] Insulator defect images collected by drones are mostly small objects with a size of less than 30×30 pixels, and the problem of small object recognition has always been a difficult research problem that needs to be solved. The following factors have posed major challenges to the identification of insulator defects during drone inspections:

[0008] 1) When inspecting, drones approach targets from far to near, which is reflected in the process of targets from large to small in the image. Smaller targets (width and height less than 30 pixels) are unevenly distributed in the entire image, resulting in a waste of computing resources and making it impossible for the algorithm to reason quickly;

[0009] 2) In addition, during the inspection process, the drone will inevitably rotate, shake and other movements, which will cause the collected image targets to be easily scaled, rotated and deformed, resulting in high false positive and missed detection rates for the model's recognition of small deformed targets;

[0010] 3) In the images collected by drones, the changes in target positions are mostly nonlinear and random. If they are predicted based on linear changes, it is very easy to cause inaccurate target positioning.

[0011] In view of this, the present invention is proposed. Summary of the invention

[0012] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide an insulator defect identification method to solve the problems raised in the above background technology.

[0013] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0014] An insulator defect identification method comprises the following steps:

[0015] S1, collecting images of insulators on power lines through drones, standardizing the images, and performing data enhancement operations;

[0016] S2, converting the image processed in step S1 into a set of 5-dimensional data points including position coordinates and color information, performing hierarchical clustering and feature assignment on the pixels in the image using point reduction and context clustering methods to form context clustering centers, and superimposing the clustering results onto the original image;

[0017] S3, using the context clustering result formed in step S2, using ResNet101 as the backbone network, extracting features of the image through a multi-resolution feature pyramid network, and after completing the feature extraction, detecting and performing deformation convolution operations on small targets and deformable targets through a deformable convolution operator and a path aggregation network, and finally generating a multi-resolution feature map based on the features after the deformation convolution operation;

[0018] S4, based on the multi-resolution feature map generated in step S3, extract the upper left corner, lower right corner and center key points of the target, and locate the target position by fusing regression and heat map method to generate a bounding box of the target;

[0019] S5, based on the key point positions and bounding boxes determined in step S4, the prediction loss of the triplet key points and the target center point offset loss are calculated using the Bhattacharyya distance, and the size and position of the target detection box are optimized by combining the Focal Loss and the GIoU loss;

[0020] S6, using the optimization result in step S5, training a multi-resolution and context clustering CenterNet++ model, and in actual application, based on the trained model, extracting triplet key points and cluster center information in the image to achieve real-time recognition and positioning of insulator defects.

[0021] Optionally, after inputting the sequence of images, each pixel of each frame is regarded as a (r, g, b, x, y) 5-dimensional data point containing color information and position information. Convert to a collection of pixels in, The target is visually represented based on the point cloud analysis method, and then each frame of the image is processed as follows:

[0022] S2.1, the image is processed in four stages, and each stage of processing must go through two steps: point reduction and context clustering. Point reduction is to select several cluster center points in the image space according to the hierarchical clustering method, and splice and merge the pixels in the 8-neighborhood of these center points through linear projection to form cluster regions, while context clustering is to perform feature aggregation and assignment on the pixels to be aggregated and feature assigned in each cluster region;

[0023] S2.2, when interactively processing features in each clustering area, the Euclidean distance between corresponding point pairs between frames is used to determine whether the feature points are the same and dynamically aggregate the same feature points. The aggregation process is expressed as formula (1) and formula (2), and the whole process is to project the points in the aggregated feature cluster into a new point value space;

[0024]

[0025] Among them, g represents the aggregated features, v c is the center point of the cluster area, represents the i-th cluster region, there are M cluster regions in total, v i is the point value space P v ,in, d' is the number of points contained in the cluster area, sig(·) is the sigmoid function, α, β are learnable scaling and offset parameters, C is the normalization factor, and dynamic feature assignment is performed on the aggregated feature points based on cosine similarity. For each point in the cluster, the full connection operation in formula (3) is used to map from the value space back to the original point space to match the feature dimension;

[0026] p' i =p i +FC(sig(αs i+β)×g)(3)

[0027] The meaning of FC full connection in formula (3) is to match the feature dimension from the value space d' to the point space d;

[0028] S2.3, after forming the cluster centers, select the first 10 cluster centers with larger areas, superimpose these clustering results on the original input image, and finally input the image superimposed with the clustering results into the CenterNet++ network for inference and recognition.

[0029] Optionally, in step S2, context clustering processing has been performed on the input image to form context information and cluster center results of each cluster area. Based on these clustering results, the following steps are performed:

[0030] S3.1, using ResNet101 as the backbone network, input the context clustering result generated in step S2 into the network as the starting point for feature extraction;

[0031] S3.2, using the preprocessed image, the ResNet101 network is used as the backbone network to start feature extraction. ResNet101 extracts deep features from the image layer by layer through multiple residual modules;

[0032] S3.3, after ResNet101 extracts preliminary features, these features will be passed to the multi-resolution feature pyramid network, which generates feature maps of different resolutions by upsampling and horizontally connecting feature maps;

[0033] S3.4, after the multi-resolution feature map is generated, a deformable convolution operator and a path aggregation network are introduced. The deformable convolution operator and the path aggregation network increase the offset of the convolution kernel so that the convolution kernel can be rotated, scaled, and other deformed during the detection process;

[0034] S3.5, after being processed by the multi-resolution feature pyramid network and the deformable convolutional network, a multi-resolution feature map is finally generated.

[0035] Optionally, in step S3, multi-resolution feature maps have been generated. Based on these feature maps, the following steps are performed to extract the key points of the target and generate the bounding box of the target:

[0036] S4.1: Extract the top left and bottom right keypoint positions of the target from the feature map;

[0037] S4.2: A method for determining the target center area by key points, and optimizing the center area by context clustering;

[0038] S4.3: Combine the information of key points and central area to generate the final bounding box of the target, and optimize the position and size of the bounding box by comparing the intersection-over-union ratio of the predicted result with the true value.

[0039] Optionally, in step S4 and step S5, the key point positions and bounding boxes of the target have been determined. Based on this information, the following detailed steps are performed to optimize the size and position of the target detection box:

[0040] S5.1: Obtain the coordinates of the upper left corner of the target detection box (tL x , tL y ) and the lower right corner coordinates (br x ,br y ), calculate the key point position of the target center area, and obtain the upper left corner coordinates (tL x , tL y ) and the lower right corner coordinates (br x ,br y ) Calculate the coordinates of the upper left corner and lower right corner of the center area (ctl x , ctl y ) and (cbr x ,cbr y ), whose expression is: Where n is an odd number, which is used to adjust and determine the size of the target center area;

[0041] S5.2: During the object detection training process, a multi-resolution CenterNet++ architecture is used, and the loss function of the ContextCluster CenterNet++ model is calculated, which is expressed as: Among them, L CC L is the clustering loss, calculated using Focal Loss. reg is the regression loss, calculated by GIoU loss. is the keypoint prediction loss for the context clustering region and the bias loss for the center keypoint. is the other relevant loss of the context clustering area, which is used to further optimize the center position of the target. are weight parameters that control the contribution of each loss term to the total loss so that they can be adjusted according to the specific needs of the model.

[0042] The values ​​are 2, 0.25 and 1 respectively.

[0043] S5.3: Predicting loss at key points The Bhattacharyya distance is introduced in the calculation of Get the Bhattacharyya coefficient cBha And use the obtained Bhattacharyya coefficient c Bha Calculate Bhattacharyya distance d Bha , and finally use the Bhattacharyya distance d Bha To calculate the prediction loss of the key points, the expression is: Among them, p(x) and q(x) represent the probability distribution of predicted key points and true key points respectively;

[0044] S5.4: After calculating the Bhattacharyya distance, the total loss function of the model is LCCL CC-CN By combining the classification loss L CC , regression box loss L reg , key point loss and and offset-related losses and to optimize the model.

[0045] Optionally, the optimization result in step S5 is used to train a multi-resolution and context clustering CenterNet++ model, and in practical applications, based on the trained model, the steps of extracting triplet key points and cluster center information in the image to achieve real-time recognition and positioning of insulator defects are as follows:

[0046] S6.1: Use the multi-resolution feature map and bounding box information obtained by optimization in step S5 as input data for training the CenterNet++ model for multi-resolution and context clustering;

[0047] S6.2: Using the optimization results in step S5, start training the CenterNet++ model with multi-resolution and context clustering. During the training process, the model learns different features of insulator defects through multi-resolution feature maps, and uses the context clustering results to enhance the model's ability to distinguish between background and foreground.

[0048] S6.3: While the model is being trained, use the validation set to periodically validate the model to check its performance on unseen data.

[0049] S6.4: After training is completed, the trained multi-resolution and context clustering CenterNet++ model is deployed to actual applications.

[0050] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0051] The present invention predicts the size, posture, key points and other information of the identified target from the center point of the target. The feature extraction process is implemented through a fully convolutional network. After feature extraction, a heat map about the center of the target is generated, and then the center of the target is located by finding the local maximum in the heat map. The size and posture of the target are inferred by using features at the peak position. Since the present invention uses a multi-resolution feature map for target category prediction, the algorithm proposed in the present invention can meet the requirements of deformation and scale changes of images collected by drones. Before target recognition, preprocessing such as image and adaptive scale feature extraction is integrated to improve the model's recognition accuracy of insulator defects in different environments.

[0052] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described below are only some embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0054] Figure 1 A schematic diagram for determining the central area based on key points;

[0055] Figure 2 This is a block diagram of the Context Cluster CenterNet++ recognition method;

[0056] Figure 3 It is a schematic diagram of the context clustering process;

[0057] Figure 4 It is a schematic diagram of feature detection module;

[0058] Figure 5 Fig.9 is a schematic diagram showing the comparison of the average precision (mAP) and loss results of CenterNet, CenterNet++ and CC-CenterNet++;

[0059] Figure 6 Schematic diagram of comparison of mAP with the CenterNet&CenterNet++;

[0060] Figure 7 Several defect recognition result diagrams are shown;

[0061] Figure 8 Comparison of small target recognition results for insulator defect recognition;

[0062] Fig. 9 This is a comparison chart of medium target identification results for insulator defect identification;

[0063] Fig.10 This is a comparison chart of large target recognition results for insulator defect recognition.

[0064] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0065] The present invention will now be described in further detail with reference to the accompanying drawings.

[0066] See also Figure 1-10 As shown, in this embodiment, a method for identifying insulator defects is provided, comprising the following steps:

[0067] S1, collecting images of insulators on power lines through drones, standardizing the images, and performing data enhancement operations;

[0068] S2, converting the image processed in step S1 into a set of 5-dimensional data points including position coordinates and color information, performing hierarchical clustering and feature assignment on the pixels in the image using point reduction and context clustering methods to form context clustering centers, and superimposing the clustering results onto the original image;

[0069] S3, using the context clustering result formed in step S2, adopting ResNet101 as the backbone network, extracting features of the image through a multi-resolution feature pyramid network, and after completing the feature extraction, detecting and deforming small targets and deforming targets through a deformable convolution operator and a path aggregation network, and finally generating a multi-resolution feature map based on the features after the deformable convolution operation.

[0070] S4, based on the multi-resolution feature map generated in step S3, extract the upper left corner, lower right corner and center key points of the target, and locate the target position by fusing regression and heat map method to generate a bounding box of the target;

[0071] S5, based on the key point positions and bounding boxes determined in step S4, the prediction loss of the triplet key points and the target center point offset loss are calculated using the Bhattacharyya distance, and the size and position of the target detection box are optimized by combining the Focal Loss and the GIoU loss;

[0072] S6, using the optimization result in step S5, training a multi-resolution and context clustering CenterNet++ model, and in actual application, based on the trained model, extracting triplet key points and cluster center information in the image to achieve real-time recognition and positioning of insulator defects.

[0073] It should be noted that: the present invention proposes a method based on context clustering (Context Cluster, CC), which aims to solve the problem of low recognition accuracy caused by uneven distribution of targets in the image. Traditional target detection methods are often difficult to maintain efficient and high-precision recognition performance when facing unevenly distributed targets. To this end, the present invention clusters the target areas in the image by means of context clustering, and performs target recognition and position prediction on this basis. This method can not only better capture the correlation information between targets, but also effectively reduce false detections caused by complex backgrounds. At the same time, the present invention introduces an improved loss function to feedback and correct the cluster centers, so that the model can significantly improve the computational efficiency while improving the recognition accuracy. The loss function can dynamically adjust the weights of the cluster centers, so that the target position prediction is more accurate and robust.

[0074] In the detection module, the present invention further processes the image based on the deformable convolution operator (DCNv2) combined with the path aggregation network (PAN). The deformable convolution operator enables the model to flexibly respond to the deformation and scale changes of the target in the image by adaptively adjusting the shape of the convolution kernel. In particular, after combining with the path aggregation network, the model can not only predict the regression box and keypoint triplet (Keypoint Triplet, KP) more accurately, but also improve the positioning ability of the target position of any shape and any scale. This design effectively reduces the model's sensitivity to target scale changes and deformations, thereby significantly improving the recognition accuracy of the entire model.

[0075] See also Figure 1 As shown, this method predicts the size, posture, key points and other information of the identified target from the center point of the target. The feature extraction process is implemented through a fully convolutional network. After feature extraction, a heat map about the center of the target is generated, and then the center of the target is located by finding the local maximum in the heat map. The size and position of the target are inferred by using features at the peak position. Since the present invention uses a multi-resolution feature map for target category prediction, the algorithm proposed in the present invention can meet the needs of deformation and scale changes of images collected by drones. Before target recognition, preprocessing such as image and adaptive scale feature extraction is integrated to improve the model's recognition accuracy of insulator defects in different environments.

[0076] See also Figure 2 As shown, in this embodiment, after the sequence of images is input, each pixel point of each frame of the image is regarded as a (r, g, b, x, y) 5-dimensional data point containing color information and position information. Convert to a collection of pixels Among them, (N = H × W), and the target is visually represented according to the point cloud analysis method, and then each frame of the image is processed as follows:

[0077] S2.1, the image is processed in four stages, and each stage of processing must go through two steps: point reduction and context clustering. Point reduction is to select several cluster center points in the image space according to the hierarchical clustering method, and splice and merge the pixels in the 8-neighborhood of these center points through linear projection to form cluster regions, while context clustering is to perform feature aggregation and assignment on the pixels to be aggregated and feature assigned in each cluster region;

[0078] S2.2, when interactively processing features in each clustering area, the Euclidean distance between corresponding point pairs between frames is used to determine whether the feature points are the same and dynamically aggregate the same feature points. The aggregation process is expressed as formula (1) and formula (2), and the whole process is to project the points in the aggregated feature cluster into a new point value space;

[0079]

[0080]

[0081] Among them, g represents the aggregated features, v c is the center point of the cluster area, represents the i-th cluster region, there are M cluster regions in total, v i is the point value space P v ,in, d' is the number of points contained in the cluster area, sig(·) is the sigmoid function, α, β are learnable scaling and offset parameters, C is the normalization factor, and dynamic feature assignment is performed on the aggregated feature points based on cosine similarity. For each point in the cluster, the full connection operation in formula (3) is used to map from the value space back to the original point space to match the feature dimension;

[0082] p' i =p i +FC(sig(αs i +β)×g)(3)

[0083] The meaning of FC full connection in formula (3) is to match the feature dimension from the value space d' to the point space d;

[0084] S2.3, after forming the cluster centers, select the first 10 cluster centers with larger areas, superimpose these clustering results on the original input image, and finally input the image superimposed with the clustering results into the CenterNet++ network for inference and recognition.

[0085] like Figure 2 As shown in the figure, specifically, the PS follows the hierarchical clustering method to select some cluster center points in the image space, and splices and fuses the points in their 8-neighborhood in the cluster area through linear projection (LP), so as to reduce the data points and improve the calculation efficiency; the CC step is to perform feature aggregation and feature assignment on each cluster area, which is similar to the local attention mechanism. After dividing the image into different cluster areas, the features are interacted between the areas, first aggregated and then assigned.

[0086] See also Figure 3-4 As shown, in this embodiment, in step S2, the input image has been subjected to context clustering processing to form context information and cluster center results of each clustering area. Based on these clustering results, the following steps are performed:

[0087] S3.1, using ResNet101 as the backbone network, input the context clustering result generated in step S2 into the network as the starting point for feature extraction;

[0088] S3.2, using the preprocessed image, the ResNet101 network is used as the backbone network to start feature extraction. ResNet101 extracts deep features from the image layer by layer through multiple residual modules; the residual block can effectively extract low-level, medium-level and high-level features of the image while retaining the spatial structure and semantic information of the image. Since the input image has been processed by context clustering, ResNet101 can more accurately extract deep features containing contextual information. In this process, ResNet101 generates feature maps of multiple scales through layer-by-layer convolution and downsampling operations. These feature maps provide the basis for the subsequent path aggregation network (PAN).

[0089] It should be noted that: the present invention uses ResNet101 as the backbone network to generate regression graph branches and heat map branches. Among them, the generated heat map branch is multi-resolution, which makes it easier to identify small targets. In the regression graph branch, the present invention uses the deformable convolution v2 (DCNv2) method to replace the fixed template convolution method in the original model. Because its convolution kernel adds an offset during the training process, and various transformations such as rotation and scaling are added during the convolution process to overcome various small targets and deformation problems, while suppressing background information, it effectively improves the extraction accuracy of small target features, and improves computational efficiency while slightly increasing the complexity of the model. Its convolution process is as follows Figure 3 shown.

[0090] S3.3, after ResNet101 extracts preliminary features, these features will be passed to the multi-resolution feature pyramid network, which generates feature maps of different resolutions by upsampling and horizontally connecting feature maps;

[0091] S3.4, after the multi-resolution feature map is generated, a deformable convolution operator and a path aggregation network are introduced. The deformable convolution operator and the path aggregation network increase the offset of the convolution kernel so that the convolution kernel can be rotated, scaled, and other deformed during the detection process;

[0092] It should be noted that the present invention uses DCNv2 to improve the regression graph branch in the feature detection module, so that the sampling point is offset to the target position, reducing the interference of the background on the detection. Figure 4 shown.

[0093] S3.5, after being processed by the multi-resolution feature pyramid network and the deformable convolutional network, a multi-resolution feature map is finally generated.

[0094] In this embodiment, in step S3, a multi-resolution feature map has been generated. Based on these feature maps, the following steps are performed to extract the key points of the target and generate a bounding box of the target:

[0095] S4.1: Extract the top left and bottom right keypoint positions of the target from the feature map;

[0096] S4.2: A method for determining the target center area by key points, and optimizing the center area by context clustering;

[0097] S4.3: Combine the information of key points and central area to generate the final bounding box of the target, and optimize the position and size of the bounding box by comparing the intersection-over-union ratio of the predicted result with the true value.

[0098] In this embodiment, in step S4, the key point positions and bounding boxes of the target have been determined. Based on this information, the following detailed steps are performed to implement step S5 to optimize the size and position of the target detection box:

[0099] S5.1: Obtain the coordinates of the upper left corner of the target detection box (tL x , tL y ) and the lower right corner coordinates (br x ,br y ), calculate the key point position of the target center area, and obtain the upper left corner coordinates (tL x , tL y ) and the lower right corner coordinates (br x ,br y ) Calculate the coordinates of the upper left corner and lower right corner of the center area (ctl x , ctl y ) and (cbr x ,cbr y ), whose expression is: Where n is an odd number, which is used to adjust and determine the size of the target center area;

[0100] S5.2: During the object detection training process, a multi-resolution CenterNet++ architecture is used, and the loss function of the ContextCluster CenterNet++ model is calculated, which is expressed as: Among them, L CC L is the clustering loss, calculated using Focal Loss. reg is the regression loss, calculated by GIoU loss. is the keypoint prediction loss for the context clustering region and the bias loss for the center keypoint. is the other relevant loss of the context clustering area, which is used to further optimize the center position of the target. are weight parameters that control the contribution of each loss term to the total loss so that they can be adjusted according to the specific needs of the model. The values ​​are 2, 0.25 and 1 respectively. The regression branch and the heat map branch are calculated in parallel. First, a deformable convolution is performed on the context-based clustering result, and the offset of the cluster center is constrained to produce a more accurate position and size prediction; and compared with the true value (Ground Truth, GT) to calculate the clustering loss and convolution loss. In the heat map branch, the present invention uses a path aggregation network (PAN) to generate multi-resolution heat maps. After feature detection calculation, two types of feature maps can be obtained. One is a triplet of key points including the upper left, lower right and cluster center, so as to determine the center of the target and the width and height of the detection box; the other is a multi-resolution heat map, and according to the formula The calculated loss value determines the predicted target position. Then the loss function L designed according to the present invention is used. CC To finally determine the position of the target, the above expression consists of four parts: (1) classification loss, which is calculated by the Focal Loss method; (2) regression box size loss, which is obtained by calculating the GIoU loss; (3) prediction loss of the keypoints triple (KP), which is calculated by the FocalLoss method; (4) target center point bias loss, which is calculated by the L1 loss. α, β, γ are the corresponding loss weights, which are set to 2, 0.25 and 1 respectively.

[0101] S5.3: Predicting loss at key points The Bhattacharyya distance is introduced in the calculation of Get the Bhattacharyya coefficient c Bha And use the obtained Bhattacharyya coefficient c Bha Calculate Bhattacharyya distance d Bha , and finally use the Bhattacharyya distance d Bha To calculate the prediction loss of the key points, the expression is: Among them, p(x) and q(x) represent the probability distribution of predicted key points and true key points respectively;

[0102] S5.4: After calculating the Bhattacharyya distance, the total loss function of the model is LCCL CC-CN By combining the classification loss L CC , regression box loss L reg , key point loss and and offset-related losses and to optimize the model.

[0103] In this embodiment, the optimization result in step S5 is used to train a multi-resolution and context clustering CenterNet++ model, and in practical applications, based on the trained model, the steps of extracting triple key points and cluster center information in the image to achieve real-time recognition and positioning of insulator defects are as follows:

[0104] S6.1: Use the multi-resolution feature map and bounding box information obtained by optimization in step S5 as input data for training the CenterNet++ model for multi-resolution and context clustering;

[0105] S6.2: Using the optimization results in step S5, start training the CenterNet++ model with multi-resolution and context clustering. During the training process, the model learns different features of insulator defects through multi-resolution feature maps, and uses the context clustering results to enhance the model's ability to distinguish between background and foreground.

[0106] The training process includes forward propagation, loss calculation (such as using Bhattacharyya distance, Focal Loss, GIoU loss, etc.), and optimization of model parameters through back propagation. Through continuous iteration, the model can gradually converge to a state that can accurately identify insulator defects.

[0107] S6.3: While the model is being trained, use the validation set to periodically validate the model to check its performance on unseen data.

[0108] S6.4: After training is completed, the trained multi-resolution and context clustering CenterNet++ model is deployed to actual applications.

[0109] Experimental example: the dataset, data preprocessing and data enhancement process, evaluation indicators and implementation details used in the experiment of the present invention; then the ablation experiment carried out is introduced to verify the effectiveness of the backbone module, detection module and improved damage function of the proposed innovative part; finally, from the quantitative and qualitative perspectives, the model proposed in the present invention is compared with the most advanced (State of the Art, SOTA) model to prove the superiority of the proposed model.

[0110] The data set used in the experiment of this invention is the insulator images collected during the standardized UAV inspection of the State Grid Corporation of China. There are a total of 2400 frames of images with a resolution of 640×640. The recognition algorithm is mainly used to study the five defects of insulator self-explosion, flashover, glass contamination, breakage and foreign matter. The training set consists of 1600 frames of images, with a total of 7568 labels, including: 2636 self-explosion, 1188 flashover, 1284 contamination, 1140 breakage, and 1320 foreign matter; the test set consists of 800 frames of images, with a total of 503 labels, including: 147 self-explosion, 64 flashover, 89 contamination, 123 breakage, and 80 foreign matter.

[0111] Insulator Data Collection Enhancements

[0112] Since the insulator datasets collected by drones are rarely combined, the present invention performs data enhancement processing on the images in the training set based on the existing insulator dataset: rotating, denoising, cropping, color conversion and other data enhancement processing are performed on the images, and finally the original dataset is expanded into 7,600 insulator defect datasets containing 5 types.

[0113] Table 1 Various types of image data (frames) in the data set

[0114]

[0115]

[0116] Evaluation indicators

[0117] The present invention uses two types of evaluation indicators, namely average precision (AP) and model reasoning speed (FPS), to measure the performance of the proposed model.

[0118] Average Precision (AP)

[0119] AP is the area under the precision (Precision, P) and recall (Recall, R) curves. The larger the AP, the better the performance of the model. AP is calculated as follows: in, TP (True Positives), FP (False Positives), FP (False Positives), FN (False Negatives). The present invention uses AP, AP 50 ,AP 75 As an evaluation indicator, AP represents the average precision of all categories. 50 and AP 75 It represents the average precision when the IoU threshold is 0.5 and 0.75 in all categories. In addition, the present invention also uses AP S ,AP M ,AP L Evaluation indicators are used to measure the recognition performance of the model for small, medium and large scale targets.

[0120] Experimental Results

[0121] The present invention has conducted extensive experiments on the MS-COCO dataset and the State Grid standardized drone inspection insulator image dataset, respectively, to verify the effectiveness of the proposed model and the superiority of the most advanced method (State-of-the-Art, SOTA). Among them, the MS-COCO dataset contains 80 categories and more than 1.5 million object instances. A large number of small targets make it a widely used dataset. The present invention uses train2017 as the training dataset, test2017 as the test set, and val2017 for ablation experiments and visualization experiments.

[0122] Ablation experiment

[0123] In order to verify the effectiveness of the three innovative methods proposed in this invention, this invention conducts an ablation experiment on the data set introduced in Section 3.1. When the innovative method is introduced and the model converges, the ablation experiment comparison results are shown in Table 2. Among them, "√" and "×" in the table indicate whether this method is used for innovation. CC, DCN-KP and Bha in the table represent the three improved methods proposed by this invention.

[0124] Table 2. Comparison of ablation experiments

[0125]

[0126] 1. The influence of CC. This paper proposes to use context clustering (CC) to solve the problem of non-uniform distribution and long tail. From the second row of Table 2, it can be seen that after the regression branch image and the heat map branch image are processed by clustering, the model improves the IoU (mAP) and median (AP) with a threshold of 1. 50 ), large target (AP 75 ) (by 2.2%, 3.5% and 2.0% respectively), and the reasoning speed is also significantly improved (11.5%). In addition, the present invention uses an improved loss function to perform feedback correction on the cluster center, which further improves the recognition accuracy (Table 2 - rows 5, 6, and 8).

[0127] 2. Impact of DCN-KP. This paper uses a deformable convolution operator (DCNv2) combined with a path aggregation network (PAN) to perform a deformable convolution operation on the image to more accurately locate the regression box and the key point triple (KP), thereby improving the positioning of targets of any shape and scale, reducing the model's sensitivity to target scale changes and deformations, and improving the model's recognition accuracy. From Table 2-3rd, 5th, 6th, and 8th rows, it can be observed that in addition to improving the overall recognition accuracy of the model, the improved module also improves the recognition accuracy of small targets (AP S ) were particularly effective, improving the accuracy by 0.5%, 1.5%, 1.1%, and 1.7%, respectively.

[0128] 3. The influence of Bha. The present invention uses Bha distance to calculate the key point triple prediction loss and target center point offset loss. The advantage of using a nonlinear distance calculation method instead of a linear distance calculation method is that it is not limited by the change of the target scale within the frame and the target position between frames, and can significantly improve the positioning accuracy of the same target between different frames. From Table 2-rows 4, 6, 7, and 8, it can be seen that the improved loss function can not only improve the AP of the model 50 and AP 75 In addition to recognition accuracy, the recognition accuracy of targets of any scale (AP S ,AP M ,APL ) all increased (0.4%, 0.5%, 1.3%, 1.2%, 1.5%).

[0129] In summary, through the ablation experiment comparison results, it can be seen that when the three improvement schemes of CC, DCN-KP and Bha are introduced in sequence, mAP, AP 50 ,AP 75 and AP S ,AP M ,AP L It can be seen that when the algorithm gradually introduces the three improved solutions, the accuracy and efficiency of model recognition can be further improved, further verifying the feasibility of the solution of the present invention.

[0130] Model performance comparison experiment

[0131] The algorithm of the present invention is compared with mainstream target recognition models with detection boxes such as Faster R-CNN with pyramid operation (FPN), Mask R-CNN, RetinaNet, YOLOv3, DETR, FOS-DCN, and recognition models without detection boxes such as CenterNet, CenterNet++, and YOLC on the MC-COCO dataset and the power grid dataset of the present invention. The comparison is performed when each model reaches convergence. The experimental results are shown in Table 3.

[0132] Table 3 Comparison of the performance of different algorithms

[0133]

[0134]

[0135] FPN is a pyramid operation. R-101 represents ResNet101, R-50 represents ResNet50, X-101 represents ResNeXt-101, HG-104 represents Hourglass-104, and n represents the number of candidate target areas to be identified.

[0136] Table 3 gives a quantitative comparison with the most advanced methods on the MS-COCO dataset. The results show that the model proposed in the present invention performs better than SOTA on the dataset. In general, the methods without detection boxes are better than the methods with boxes in terms of recognition accuracy and reasoning speed. The method proposed in the present invention adopts strategies such as context clustering, key point triples, and nonlinear loss functions, fully considering the impact of adverse factors such as target scale change and deformation on recognition and detection accuracy, and uses point sets instead of regions to solve the target position. Therefore, under the premise of ensuring recognition accuracy, the recognition efficiency is greatly improved, the amount of calculation is greatly reduced, and the basic model is deployed on downstream tasks such as drone systems. Great possibilities are brought.

[0137] Table 4 shows the comparison of recognition results on the insulator defect data set. From the recognition results of the five types of defects, it can be seen that the algorithm of the present invention can accurately identify various types of insulator defects.

[0138] Table 4 Comparison of different algorithms for identifying insulator defects

[0139]

[0140]

[0141] Model parameters

[0142] (1) The environment configuration and basic hyperparameter settings are shown in Table 5.

[0143] Table 5 Experimental environment configuration

[0144]

[0145] Visualization and analysis of results

[0146] The improved algorithm of the present invention is analyzed and verified by the curve of the model accuracy changing with the number of iterations and the recognition results of the test pictures.

[0147] Comparison of average accuracy (mAP) and loss of algorithms

[0148] Figure 5This is a comparison chart of the average precision mA and loss value of the model (CC-CenterNet++) proposed in the present invention and the CenterNet and CenterNet++ models. A total of 8 types of insulators are compared in the figure, among which two glass (two glass insulators), insulator (insulator) and polymer (polymer) are all normal insulators without defects; the latter 5 types: pollution-flashover (flashover), broken disc (breakage), polymerdirty (foreign matter), glassdirty (dirty) and glassloss (self-explosion) are the insulator defects to be identified by the present invention. It can be seen from the figure that the mAP value of the model proposed in the present invention is 2.97% and 1.92% higher than the mAP values ​​of CenterNet and CenterNet++, respectively. From the loss comparison chart, the loss values ​​of CenterNet and CenterNet++ models are higher than 5 and higher than 4.5, respectively, while the loss value of the model proposed in the present invention is below 4.5.

[0149] Insulator defect identification results visualization and performance analysis

[0150] In order to verify the effectiveness of the improved algorithm for insulator defect detection, the present invention visualizes the identification pictures of self-explosion, flashover, dirt, breakage and foreign body defects, as well as the identification pictures of small targets, medium targets and large targets to verify the effectiveness of the improved algorithm of the present invention. Figure 6 These are the visualization results of several defect identifications.

[0151] Figure 7 is the visualization result of defect recognition at different target scales. Figure 7 It can be seen that the model proposed in the present invention can detect insulator defects of any scale and any deformation without missing or misdetecting them, showing high recognition accuracy and efficiency.

[0152] Comparative experimental visualization results and performance analysis

[0153] Figure 6This is a comparison of the recognition results of the recognition model of the present invention and the CenterNet and CenterNet++ models for small insulator defects. It can be found from the recognition results that the CenterNet and CenterNet++ models did not recognize small defects, while the Context Cluster CenterNet++ (CC-CenterNet++) model proposed in the present invention can accurately identify small defects (glassloss: 0.38). Through experimental comparison, it can be seen that compared with the CenterNet and CenterNet++ models, the algorithm of the present invention has a lower missed detection rate, better recognition effect on small targets, and more accurate positioning.

[0154] Fig. 9 This is a comparison of the recognition results of the recognition model of the present invention and the CenterNet and CenterNet++ models for medium insulator defects. It can be found from the recognition results that the model proposed by the present invention can accurately identify any defect with higher accuracy and lower missed detection rate.

[0155] Fig.10 This is a comparison of the recognition results of the recognition model of the present invention and the CenterNet and CenterNet++ models for large insulator defects. It can be found from the recognition results that the model proposed by the present invention can accurately identify any defect with higher accuracy and stronger generalization.

[0156] Through the insulator defect recognition results at the above scales, it can be seen that the algorithm in this paper can effectively avoid the mispickup and missed detection of insulator defects when facing various scales and deformations, and improve the algorithm's recognition ability for insulator defects in various scales, deformations and complex environments, verifying the effectiveness and feasibility of the improved algorithm in this paper.

[0157] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.

Claims

1. A method for identifying insulator defects, characterized in that: The following steps are involved: S1, collecting images of insulators on power lines through drones, standardizing the images, and performing data enhancement operations; S2, converting the image processed in step S1 into a set of 5-dimensional data points including position coordinates and color information, performing hierarchical clustering and feature assignment on the pixels in the image using point reduction and context clustering methods to form context clustering centers, and superimposing the clustering results onto the original image; S3, using the context clustering result formed in step S2, using ResNet101 as the backbone network, extracting features of the image through a multi-resolution feature pyramid network, and after completing the feature extraction, detecting and performing deformation convolution operations on small targets and deformable targets through a deformable convolution operator and a path aggregation network, and finally generating a multi-resolution feature map based on the features after the deformation convolution operation; S4, based on the multi-resolution feature map generated in step S3, extract the upper left corner, lower right corner and center key points of the target, and locate the target position by fusing regression and heat map method to generate a bounding box of the target; S5, based on the key point positions and bounding boxes determined in step S4, the prediction loss of the triplet key points and the target center point offset loss are calculated using the Bhattacharyya distance, and the size and position of the target detection box are optimized by combining the Focal Loss and the GIoU loss; S6, using the optimization result in step S5, training a multi-resolution and context clustering CenterNet++ model, and in actual application, based on the trained model, extracting triplet key points and cluster center information in the image to achieve real-time recognition and positioning of insulator defects.

2. The insulator defect identification method according to claim 1, characterized in that: After inputting the sequence of images, each pixel of each frame is regarded as a (r, g, b, x, y) 5-dimensional data point containing color information and position information. Convert to a collection of pixels Among them, (N = H × W), and the target is visually represented according to the point cloud analysis method, and then each frame of the image is processed as follows: S2.1, the image is processed in four stages, and each stage of processing must go through two steps: point reduction and context clustering. Point reduction is to select several cluster center points in the image space according to the hierarchical clustering method, and splice and merge the pixels in the 8-neighborhood of these center points through linear projection to form cluster regions, while context clustering is to perform feature aggregation and assignment on the pixels to be aggregated and feature assigned in each cluster region; S2.2, when interactively processing features in each clustering area, the Euclidean distance between corresponding point pairs between frames is used to determine whether the feature points are the same and dynamically aggregate the same feature points. The aggregation process is expressed as formula (1) and formula (2), and the whole process is to project the points in the aggregated feature cluster into a new point value space; Among them, g represents the aggregated features, v c is the center point of the cluster area, represents the i-th cluster region, there are M cluster regions in total, v i is the point value space P v ,in, d' is the number of points contained in the cluster area, sig(·) is the sigmoid function, α, β are learnable scaling and offset parameters, C is the normalization factor, and among the aggregated feature points, dynamic feature assignment is performed based on cosine similarity. For each point in the cluster, the full connection operation in formula (3) is used to map from the value space back to the original point space to match the feature dimension; p' i (p i +FC(sig(αs). i +β)×g)(3) The meaning of FC full connection in formula (3) is to match the feature dimension from the value space d' to the point space d; S2.3, after forming the cluster centers, select the first 10 large-area cluster centers, superimpose these clustering results on the original input image, and finally input the image superimposed with the clustering results into the CenterNet++ network for inference and recognition.

3. The insulator defect identification method according to claim 1, characterized in that: In step S2, the input image has been subjected to context clustering processing to form context information and cluster center results of each cluster area. Based on these clustering results, the following steps are performed: S3.1, using ResNet101 as the backbone network, input the context clustering result generated in step S2 into the network as the starting point for feature extraction; S3.2, using the preprocessed image, the ResNet101 network is used as the backbone network to start feature extraction. ResNet101 extracts deep features from the image layer by layer through multiple residual modules; S3.3, after ResNet101 extracts preliminary features, these features will be passed to the multi-resolution feature pyramid network, which generates feature maps of different resolutions by upsampling and horizontally connecting feature maps; S3.4, after the multi-resolution feature map is generated, a deformable convolution operator and a path aggregation network are introduced. The deformable convolution operator and the path aggregation network increase the offset of the convolution kernel so that the convolution kernel can be rotated, scaled, and other deformed during the detection process; S3.5, after being processed by the multi-resolution feature pyramid network and the deformable convolutional network, a multi-resolution feature map is finally generated.

4. The insulator defect identification method according to claim 1, characterized in that: In step S3, multi-resolution feature maps have been generated. Based on these feature maps, the following steps are performed to extract the key points of the target and generate the bounding box of the target: S4.1: Extract the top left and bottom right keypoint positions of the target from the feature map; S4.2: A method for determining the target center area by key points, and optimizing the center area by context clustering; S4.3: Combine the information of key points and central area to generate the final bounding box of the target, and optimize the position and size of the bounding box by comparing the intersection-over-union ratio of the predicted result with the true value.

5. The insulator defect identification method according to claim 1, characterized in that: In step S4 and step S5, the key point positions and bounding boxes of the target have been determined. Based on this information, the following steps are performed to optimize the size and position of the target detection box: S5.1: Obtain the coordinates of the upper left corner of the target detection box (tL x , tL y ) and the lower right corner coordinates (br x ,br y ), calculate the key point position of the target center area, and obtain the upper left corner coordinates (tL x , tL y ) and the lower right corner coordinates (br x ,br y ) Calculate the coordinates of the upper left corner and lower right corner of the center area (ctl x , ctl y ) and (cbr x ,cbr y ), whose expression is: Where n is an odd number, which is used to adjust and determine the size of the target center area; S5.2: During the object detection training process, a multi-resolution CenterNet++ architecture is used, and the loss function of the ContextCluster CenterNet++ model is calculated, which is expressed as: Among them, L CC is the clustering loss, calculated using Focal Loss, L reg is the regression loss, calculated by GIoU loss, is the key point prediction loss of the context clustering region and the bias loss of the central key point, is the other relevant loss of the context clustering area, which is used to further optimize the center position of the target. are weight parameters that control the contribution of each loss term to the total loss so that they can be adjusted according to the specific needs of the model. The values ​​of are 2, 0.25, and 1 respectively; S5.3: Predicting loss at key points The Bhattacharyya distance is introduced in the calculation of Get the Bhattacharyya coefficient c Bha And use the obtained Bhattacharyya coefficient c Bha Calculate Bhattacharyya distance d Bha , and finally use the Bhattacharyya distance d Bha To calculate the prediction loss of the key points, the expression is: Among them, p(x) and q(x) represent the probability distribution of predicted key points and true key points respectively; S5.4: After calculating the Bhattacharyya distance, the total loss function of the model is By combining the classification loss L CC , regression box loss L reg , key point loss and and offset-related losses and to optimize the model.

6. The insulator defect identification method according to claim 1, characterized in that: Using the optimization results in step S5, a multi-resolution and context clustering CenterNet++ model is trained. In practical applications, based on the trained model, the steps of extracting triple key points and cluster center information in the image to achieve real-time recognition and positioning of insulator defects are as follows: S6.1: Use the multi-resolution feature map and bounding box information obtained by optimization in step S5 as input data for training the CenterNet++ model for multi-resolution and context clustering; S6.2: Using the optimization results in step S5, start training the CenterNet++ model with multi-resolution and context clustering. During the training process, the model learns different features of insulator defects through multi-resolution feature maps, and uses the context clustering results to enhance the model's ability to distinguish between background and foreground. S6.3: While the model is being trained, use the validation set to periodically validate the model to check its performance on unseen data. S6.4: After training is completed, the trained multi-resolution and context clustering CenterNet++ model is deployed to actual applications.

Images