A power grid equipment defect detection method based on multi-level multi-scale feature fusion

Abstract

The application discloses a power grid equipment defect detection method based on multi-level multi-scale feature fusion. The method comprises the following steps: collecting inspection image data of a power grid equipment to be detected; performing all-class defect detection on the inspection image data through an all-class defect detection model corresponding to the power grid equipment to determine a first defect detection result; performing single-class defect detection on the inspection image data through a plurality of single-class defect detection models corresponding to the power grid equipment to determine a second defect detection result; and performing overlapping result elimination on the first defect detection result and the second defect detection result through a non-maximum value elimination method to determine a defect detection result of the power grid equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent inspection of electric power equipment, and more specifically, to a method for detecting defects in power grid equipment based on multi-level and multi-scale feature fusion. Background Art

[0002] With the nation's economic development and rising living standards, the scale of the power grid is also expanding. Maintaining safe and reliable transmission lines and ensuring the grid's operation is a crucial component of the power system. Due to long-term exposure to the elements, transmission line components such as towers, fittings, insulators, and ground wires are prone to corrosion, damage, and broken strands. Furthermore, improper component installation poses a significant risk to the safe operation of transmission lines. Leveraging drone inspection data and deep learning technology to efficiently and accurately capture transmission line defects is crucial for line defect inspection. However, the complex and ever-changing environments in which transmission lines operate present significant challenges for defect identification.

[0003] Currently, drones are used to collect data from key locations on transmission towers. The image data is then transmitted back to a server, where it is processed by an object detection algorithm (such as FASTER-RCNN or YOLOv3) to detect equipment defects. Power equipment defects are numerous and challenging to detect. The results obtained by these algorithms often contain a high number of false positives or missed detections, requiring review by frontline power grid personnel.

[0004] Furthermore, the use of drones for transmission line inspections relies on massive amounts of image data captured by a large number of onboard visible light image acquisition devices. During inspections, cameras are constantly capturing data and uploading this massive amount of data to servers for centralized processing and analysis, or storing the captured images on onboard storage devices. Because network bandwidth is a time-varying and limited resource, uploading this massive amount of data to servers for centralized processing and analysis consumes significant bandwidth, causing network congestion and excessive communication latency, making real-time fault identification impossible and potentially overlooking equipment defects.

[0005] Conventional target detection algorithms, such as the two-stage detection algorithm, have high accuracy but slow speed, and the one-stage algorithm is fast but has low accuracy. The review requires a lot of manpower and material resources, making it difficult to achieve real-time monitoring. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention provides a power grid equipment defect detection method based on multi-level and multi-scale feature fusion.

[0007] According to one aspect of the present invention, a method for detecting defects in power grid equipment based on multi-level and multi-scale feature fusion is provided, comprising:

[0008] Collect inspection image data of power grid equipment to be detected for defects;

[0009] Performing full-category defect detection on the inspection image data using a full-category defect detection model corresponding to the power grid equipment to determine a first defect detection result;

[0010] Performing single-class defect detection on the inspection image data using multiple single-class defect detection models corresponding to the power grid equipment to determine a second defect detection result;

[0011] The overlapping results of the first defect detection result and the second defect detection result are eliminated by using a non-maximum elimination method to determine the defect detection result of the power grid equipment.

[0012] Optionally, the full-category defect detection model is used to detect all defect types covered in the current scenario of power grid equipment inspection. The full-category defect detection model includes: a full-category equipment defect detection model for transmission lines, a full-category equipment defect detection model for distribution lines, a full-category equipment defect detection model for substations, and a full-category equipment defect detection model for converter stations.

[0013] Optionally, performing all-category defect detection on the inspection image data using a full-category defect detection model corresponding to the power grid equipment to determine a first defect detection result includes:

[0014] Extract features from inspection image data through the feature extraction layer of the full-class detection model;

[0015] All the extracted features are concatenated and fused after being fused into the fusion self-attention module of the full-category detection model;

[0016] Classify and locate the spliced ​​and fused data features to determine the first defect detection result.

[0017] Optionally, the single-class defect detection model adopts the NanoDet-Plus-m method.

[0018] Optionally, using a non-maximum elimination method to eliminate overlapping results from the first defect detection result and the second defect detection result to determine the defect detection result of the power grid equipment includes:

[0019] Constructing a plurality of candidate frames to be processed of the defect type according to the first defect detection result and the second defect detection result, and constructing a set M for storing the optimal frames;

[0020] Calculate the confidence of each candidate box in the candidate box set respectively, sort the candidate boxes in the candidate box set according to the confidence, and select the candidate box with the highest confidence and move it into the set M;

[0021] Traverse the candidate boxes in the candidate box set, calculate the dynamic intersection-over-union ratio with the set M respectively, and remove the candidate box from the candidate box set if the intersection-over-union ratio is higher than a preset first threshold, and move the candidate box to the set M if the intersection-over-union ratio is not higher than the first threshold;

[0022] When the candidate frame set is empty, multiple sets M corresponding to candidate frame sets to be processed of multiple defect types are obtained in sequence, and the defect detection results are determined according to the multiple sets M.

[0023] Optionally, traverse the candidate boxes in the candidate box set and calculate the dynamic intersection-union ratio with the set M respectively, including:

[0024] According to the dynamic intersection-and-union ratio calculation formula corresponding to the defect type of each candidate frame, the intersection-and-union ratio with the set M is calculated respectively.

[0025] Optionally, a dynamic intersection-over-union (IoU) calculation formula corresponding to the defect type of the candidate frame includes:

[0026] When the defect type of the candidate frame is a preset small-size defect or a small target defect, the dynamic intersection-over-union ratio calculation formula is:

[0027]

[0028] When the defect type of the candidate frame is a non-small size defect or a non-small target defect, the dynamic intersection-over-union ratio is calculated as follows:

[0029]

[0030] Among them, MIOU is the dynamic intersection-over-union ratio, d is the distance between the center point of the candidate box and the box in the set M, c is the diagonal of the circumscribed rectangle containing the candidate box and the box in the set M, and w s is the width of the candidate box, h s is the height of the candidate box.

[0031] Optionally, the operation of determining the defect detection result according to the multiple sets M includes:

[0032] Detect whether the candidate boxes in the multiple sets M have the same attribute and whether the loss function is greater than a preset second threshold;

[0033] When the candidate boxes in the multiple sets M have the same attribute and the loss function is greater than a preset second threshold, the candidate box is removed from the multiple sets M to determine the defect detection result.

[0034] According to another aspect of the present invention, a power grid equipment defect detection device based on multi-level and multi-scale feature fusion is provided, comprising:

[0035] An acquisition module, used to collect inspection image data of power grid equipment to be detected for defects;

[0036] A first determination module is configured to perform all-category defect detection on the inspection image data using a full-category defect detection model corresponding to the power grid equipment, and determine a first defect detection result;

[0037] A second determination module is configured to perform single-class defect detection on the inspection image data using a plurality of single-class defect detection models corresponding to the power grid equipment to determine a second defect detection result;

[0038] The third determination module is used to eliminate overlapping results of the first defect detection result and the second defect detection result by using a non-maximum elimination method to determine the defect detection result of the power grid equipment.

[0039] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method according to any one of the above aspects of the present invention.

[0040] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the method described in any one of the above aspects of the present invention.

[0041] Therefore, the present application performs full-category defect detection on the inspection image data through a full-category defect detection model corresponding to the power grid equipment to determine the first defect detection result; performs single-category defect detection on the inspection image data through multiple single-category defect detection models corresponding to the power grid equipment to determine the second defect detection result; and uses the non-maximum elimination method to eliminate overlapping results of the first defect detection result and the second defect detection result to determine the defect detection result of the power grid equipment. Based on cutting-edge artificial intelligence and computer vision technology, and with deep convolutional neural networks as the foundation, a model based on multi-level and multi-scale feature fusion is used to detect multiple defects in power equipment. Compared with existing power equipment defect detection methods, the effect of identifying multiple defects in power equipment is better, with high accuracy and relatively low error detection, and faster operation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0043] Figure 1 1 is a flow chart of a method for detecting defects in power grid equipment based on multi-level and multi-scale feature fusion according to an exemplary embodiment of the present invention;

[0044] Figure 2 This is an overview of the overall process of defect detection provided by an exemplary embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of MIOU value calculation parameters provided by an exemplary embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of the ACDDet structure provided by an exemplary embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of depth-wise separable convolution provided by an exemplary embodiment of the present invention;

[0048] Figure 6 Schematic diagrams of two MHRes-Unit structures provided by an exemplary embodiment of the present invention;

[0049] Figure 7 is a schematic diagram of an FSAM module provided by an exemplary embodiment of the present invention;

[0050] Figure 8 1 is a schematic structural diagram of a power grid equipment defect detection device based on multi-level and multi-scale feature fusion provided by an exemplary embodiment of the present invention;

[0051] Figure 9 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0052] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0053] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0054] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0055] It should also be understood that, in the embodiments of the present invention, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0056] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0057] In addition, the term "and / or" in this invention merely describes an association relationship between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " in this invention generally indicates that the related objects are in an "or" relationship.

[0058] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0059] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0060] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0061] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0062] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0063] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or specialized computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above.

[0064] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0065] Exemplary Methods

[0066] Figure 1 This is a flow chart of a method for detecting defects in power grid equipment based on multi-level and multi-scale feature fusion provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic equipment, such as Figure 1 As shown, the power grid equipment defect detection method 100 based on multi-level and multi-scale feature fusion includes the following steps:

[0067] Step 101: Collect inspection image data of power grid equipment to be inspected for defects.

[0068] Step 102 : Perform all-category defect detection on the inspection image data using a full-category defect detection model corresponding to the power grid equipment to determine a first defect detection result.

[0069] Optionally, the full-category defect detection model is used to detect all defect types covered in the current scenario of power grid equipment inspection. The full-category defect detection model includes: a full-category equipment defect detection model for transmission lines, a full-category equipment defect detection model for distribution lines, a full-category equipment defect detection model for substations, and a full-category equipment defect detection model for converter stations.

[0070] Optionally, performing all-category defect detection on the inspection image data using a full-category defect detection model corresponding to the power grid equipment to determine a first defect detection result includes:

[0071] Extract features from inspection image data through the feature extraction layer of the full-class detection model;

[0072] All the extracted features are concatenated and fused after being fused into the fusion self-attention module of the full-category detection model;

[0073] Classify and locate the spliced ​​and fused data features to determine the first defect detection result.

[0074] Step 103 : Perform single-type defect detection on the inspection image data using multiple single-type defect detection models corresponding to the power grid equipment to determine a second defect detection result.

[0075] Optionally, the single-class defect detection model adopts the NanoDet-Plus-m method.

[0076] Step 104 : Using a non-maximum elimination method, overlapping results of the first defect detection result and the second defect detection result are eliminated to determine a defect detection result of the power grid equipment.

[0077] Optionally, using a non-maximum elimination method to eliminate overlapping results from the first defect detection result and the second defect detection result to determine the defect detection result of the power grid equipment includes:

[0078] Constructing a plurality of candidate frames to be processed of the defect type according to the first defect detection result and the second defect detection result, and constructing a set M for storing the optimal frames;

[0079] Calculate the confidence of each candidate box in the candidate box set respectively, sort the candidate boxes in the candidate box set according to the confidence, and select the candidate box with the highest confidence and move it into the set M;

[0080] Traverse the candidate boxes in the candidate box set, calculate the dynamic intersection-over-union ratio with the set M respectively, and remove the candidate box from the candidate box set if the intersection-over-union ratio is higher than a preset first threshold, and move the candidate box to the set M if the intersection-over-union ratio is not higher than the first threshold;

[0081] When the candidate frame set is empty, multiple sets M corresponding to candidate frame sets to be processed of multiple defect types are obtained in sequence, and the defect detection results are determined according to the multiple sets M.

[0082] Optionally, traverse the candidate boxes in the candidate box set and calculate the dynamic intersection-union ratio with the set M respectively, including:

[0083] According to the dynamic intersection-and-union ratio calculation formula corresponding to the defect type of each candidate frame, the intersection-and-union ratio with the set M is calculated respectively.

[0084] Optionally, a dynamic intersection-over-union (IoU) calculation formula corresponding to the defect type of the candidate frame includes:

[0085] When the defect type of the candidate frame is a preset small-size defect or a small target defect, the dynamic intersection-over-union ratio calculation formula is:

[0086]

[0087] When the defect type of the candidate frame is a non-small size defect or a non-small target defect, the dynamic intersection-over-union ratio is calculated as follows:

[0088]

[0089] Among them, MIOU is the dynamic intersection-over-union ratio, d is the distance between the center point of the candidate box and the box in the set M, c is the diagonal of the circumscribed rectangle containing the candidate box and the box in the set M, and w s is the width of the candidate box, h s is the height of the candidate box.

[0090] Optionally, the operation of determining the defect detection result according to the multiple sets M includes:

[0091] Detect whether the candidate boxes in the multiple sets M have the same attribute and whether the loss function is greater than a preset second threshold;

[0092] When the candidate boxes in the multiple sets M have the same attribute and the loss function is greater than a preset second threshold, the candidate box is removed from the multiple sets M to determine the defect detection result.

[0093] Specifically, refer to Figure 2 As shown, the present invention is based on cutting-edge artificial intelligence and computer vision technologies, adopts deep convolutional neural networks as the basis, and designs a target detection algorithm based on multi-level and multi-scale feature fusion to detect various defects such as poles, hardware, insulators and ground wires in the power grid. The backbone network part of the detection model adopts deep separable convolution to improve the computing speed of the algorithm model.

[0094] The multi-level and multi-feature fusion defect recognition algorithm mainly includes two detection models: 1) all-category defect detection model; 2) single-category defect detection model.

[0095] Comprehensive defect detection models are categorized into four types based on the power transmission, transformation, and distribution equipment scenarios: transmission line comprehensive equipment defect detection model, distribution line comprehensive equipment defect detection model, substation comprehensive equipment defect detection model, and converter station comprehensive equipment defect detection model. Comprehensive defect detection models can detect all defect types that may be covered in the current power grid equipment inspection scenario. For example, the comprehensive defect detection model for transmission line comprehensive equipment can identify defects in all equipment under the transmission line inspection scenario, including all types of transmission line equipment defects, including towers, hardware, insulators, and ground wires.

[0096] The single-class defect detection model is responsible for detecting a certain subclass of defects. The single-class defect detection model can be a model for detecting defects of a certain equipment component (such as pin-type defects) or a model for detecting a specific type of defect (such as a cotter pin).

[0097] Power equipment defects vary widely. The method in this patent can detect multiple major categories of defects at once. We refer to the algorithm model in the patent as a full-category defect detection model. For example, a full-category defect detection model for transmission lines can detect a variety of defects, including towers, insulators, large- and small-size hardware, and ground wires. Furthermore, the detectable defect categories can be expanded based on actual application needs. Because a full-category defect detection model detects multiple defects, the characteristics of major defect categories vary significantly, while the differences between subcategories within a major category are relatively small. Therefore, the detection model must possess strong fitting capabilities and the ability to perform fine-grained classification.

[0098] In response to the difficulties and requirements of all-class defect detection in power equipment, the present invention specifically designs a model called ACDDet (All-class Defects Detector). The feature extraction layer of ACDDet consists of four sub-backbone networks that can perform multi-scale feature fusion. Each sub-backbone contains an MHRes-Unit (Multi-hierarchies Residual Unit) structure to reduce the vanishing gradient during training and accelerate model convergence. Each set of features is then spliced ​​and fused through a FSAM module (Fusion self-attention Module), and finally the defects are classified and located.

[0099] A single-class defect detection model is only responsible for detecting a single class of defects, and there are multiple single-class defect detection models. Therefore, the capacity of a single-class defect detection model cannot be too large. In this method, the single-class defect detection model uses NanoDet-Plus-m.

[0100] When identifying power grid equipment inspection images, both full-category defect detection models and single-category defect detection models are run simultaneously to detect defects in batches of images. Up to nine single-category defect detection models are required, with the specific number of models determined by the number of grid equipment component types in the inspection scene. If the on-site intelligent processing hardware module has sufficient video memory, all models can be run simultaneously. If not, the full-category defect detection model can be run first, followed by the single-category defect detection models.

[0101] Perform non-maximum elimination (I-NMD) on the output results of multiple models to eliminate overlapping output results. The specific process of non-maximum elimination is as follows:

[0102] (1) Input: The candidate set H1 to be processed is composed of the output results of the full-class model and the single-class model for detecting a certain type of defect in an image. If there are 9 types of defects, there are 9 single-class models, and there may be 9 types of defects on the detected image. Then the candidate set H to be processed is only the set of output frames for detecting one type of defect target in the 9 categories.

[0103] (2) The output frame with the highest confidence is selected as the starting target frame S1, which is regarded as the defect result of the detection. The output set R1 that has pixel intersection with S1, the defect type of each output frame in R1 is consistent with S1. Calculate the MIOU of each output frame in the set R1 with S1, and traverse the R1 set according to the order of confidence. The output frames greater than the MIOU threshold are removed from the R1 set and divided into the redundant set D; the output frames Rn less than or equal to the MIOU threshold remain in the output set R1. This continues until all frames in the set R1 are traversed.

[0104] (3) Select the output box with the second largest confidence from the set H as the target box S2 for the second traversal, and repeat the traversal process of 2. Repeat in sequence until all the remaining boxes in the set H are selected as the target box S.

[0105] (4) Output: Display the set S on the image as the defect detection result.

[0106] (5) The MIOU value involved in the above process 2-4 is not a fixed value. The MIOU value is calculated based on the identified defect type, and the calculation method is:

[0107] A) If the type of the starting target frame S1 is a small size defect or a small target defect (the ratio of the short side of the width and height to the short side of the width and height of the image is less than 2%), The MIOU threshold is 0.75 (other thresholds can be set according to specific needs and are not the only limit).

[0108] Among them, d is the distance between the center points of the two rectangular frames, and c is the diagonal of the rectangle circumscribing the two rectangular frames, such as Figure 3 As shown. s is the width of the starting box s, h s is the height of the starting frame s.

[0109] B) If the type of the starting target frame S1 is not a small-sized target, but another type of target. The MIOU threshold is set to 0.25 (other thresholds can be set here based on specific needs, and this is not a single limit). To determine whether to stay in the output set R1, it is necessary to require that the distance between the center point of the rectangular box in the set H and the center point of the target box S is greater than 1 / 2 of the diagonal of the target box (other thresholds can be set here based on specific needs, and this is not a single limit).

[0110] (6) For different types of defects, process them in sequence according to steps (1) to (5) to obtain the output results of different defect types, namely the R set.

[0111] (7) For all the results in the R set, check whether there are boxes with the same attribute and DIOU greater than 0.9. If so, remove the boxes with low confidence from the R set to form the final output result.

[0112] In addition, in order to detect various defects of power equipment with faster operation speed, fewer false alarms and higher detection rate, the patent of this invention adopts a target detection algorithm based on multi-level and multi-scale feature fusion, which is called the all-category defect detector ACDDet.

[0113] (1)ACDDet overall architecture design

[0114] ACDDet can simultaneously detect defects in four types of power equipment: towers, hardware, insulators, and ground wires. Its specific network structure is as follows: Figure 4 Due to the large size variations among the four major defect categories, the feature extraction layer of ACDDet consists of four sub-backbones containing the MHRes-Unit structure. The inputs of the four sub-backbones are scaled to (3500, 2100), (3000, 1800), (2500, 1500), and (2000, 1200), corresponding to the extraction of features for hardware, ground wires, insulators, and pole towers. The reason for this design is that as the network deepens, the feature maps are continuously downsampled. While semantic information is continuously enhanced, position and texture information is continuously weakened. In particular, information about small objects may disappear directly. Therefore, in order to extract features of small objects, the network input size needs to be increased. Finally, the feature maps from the four sub-backbones are each concatenated into a set of higher-level feature maps through an FSAM module to improve the network model's ability to fit and extract features of the four major defect categories with large scale variations. This is one of the manifestations of multi-scale feature fusion.

[0115] Furthermore, small scale differences exist between subcategories of defects within each major defect category, so multi-scale features are extracted within each sub-backbone. As shown in Figure 3, four convolution kernel sizes (1×1, 3×3, 5×5, and 7×7) are used within each sub-backbone to extract feature information with smaller scale differences. The feature maps generated by these four convolution kernels are then concatenated to form a new feature map, demonstrating the second embodiment of multi-scale feature fusion. This multi-scale feature fusion, both internally and externally, maximizes ACDDet's ability to fit and extract defect features of multiple scales, thereby improving the detection accuracy of various power equipment defects.

[0116] In order to reduce the computational complexity of ACDDet and improve the operation speed, the present invention replaces the conventional convolution of the MHRes-Unit in the sub-backbone of the first two large-scale inputs of ACDDet ((3500, 2100) and (3000, 1800)) with DSC (Depth Separable Convolution, a convolution with very low computational complexity). Figure 5 As shown in the figure, a standard DSC consists of depthwise convolution (channel dimension convolution) and pointwise convolution (1×1 convolution). That is, the MHRes-Units in the first two sub-backbones are composed of DSC, denoted as D-MHRes-Unit; the MHRes-Units in the last two sub-backbones are composed of regular convolution, denoted as C-MHRes-Unit.

[0117] (2) MHRes-Unit structure

[0118] The overall structure and components of MHRes-Unit (Multi-hierarchies Residual Unit) are as follows Figure 6 As shown in Figure 2, there are two skip-layer connections in an MHRes-Unit. Compared with the single-layer (single skip-layer connection) residual unit, the MHRes-Unit can make the gradient backpropagation smoother and make the model easier to train and converge.

[0119] in addition, Figure 6 There are two types of MHRes-Unit: the convolutions in D-MHRes-Unit are all depth-separable convolutions DSC. This residual structure has small computational complexity and can speed up the model's operation; the convolutions in C-MHRes-Unit are all conventional convolutions Conv. Figure 6 The 'BN' in the figure is the most commonly used BatchNorm layer in convolutional neural networks, and Leaky-RELU is the activation function.

[0120] (3) FSAM module

[0121] Generally speaking, feature fusion may cause information redundancy. The purpose of using FSAM (Fusion Self-Attention Module) is to reduce the dimension and filter the information features so that the model can focus more on the effective and most important features.

[0122] The specific FSAM module structure is as follows Figure 7As shown, the specific calculation process is:

[0123] ① A set of feature maps X c (c is the number of channels of the feature map) is respectively subjected to three functions f1, f2 and f3 (in order to simplify the calculation, the present invention uses the simplest three groups of 1×1 convolution kernels) to obtain three new groups of feature maps with the same number of channels (set as d) and That is (* in the following formula represents convolution)

[0124]

[0125]

[0126]

[0127] ②Then, and The two sets of feature maps are multiplied by the corresponding channel and spatial position elements to perform feature fusion, that is, the Hadamard product is performed to obtain the feature map Z d (In the following formula represents the Hadamard product)

[0128]

[0129] ③ Then Z d Perform softmax calculation on the channel dimension to obtain a self-attention space map F 1 (The number of channels becomes 1)

[0130] F 1 =softmax(Z d )

[0131] ④F 1 Broadcast in the channel dimension to expand to d channels F d ,and Do the Hadamard product to get the final set of self-attention feature maps A d :

[0132] F d =Broadcasting(F 1 )

[0133]

[0134] (2) Single-class defect detection model

[0135] There are four single-type defect detection models, which detect four types of defects: towers, hardware, insulators, and ground wires. All four models use the NanoDet-Plus-m detection model.

[0136] NanoDet-Plus-m is an ultra-fast and lightweight object detection model that is easy to deploy on the edge. The model file is only 2.3MB when stored at half precision (FP16) and can achieve a speed of 50.6 frames per second on ARM with an input resolution of 416*416. NanoDet is a single-stage anchor-free object detection model based on FCOS (Fully Convolutional One-Stage Object Detection). It uses ATSS (Adaptive Training Sample Selection) for object sampling and the Generalized FocalLoss loss function for classification and box regression.

[0137] (3) Fusion of multi-model detection results

[0138] The model fusion used in this invention is different from the model ensemble commonly referred to in academia. It does not obtain the detection results by voting or averaging the results of multiple models, because the targets and task levels that each model in this invention is responsible for detecting are different.

[0139] Therefore, in the present invention, an image of the power equipment to be tested is simultaneously input into the full-category defect detection model and four single-category defect detection models (the models can be run sequentially if the video memory is insufficient), and then the output results of multiple models are mixed together to perform non-maximum suppression (NMS), thereby obtaining the final power defect detection result and the overall detection process. This application is mainly based on cutting-edge artificial intelligence and computer vision technology, with deep convolutional neural networks as the basis, and adopts a model based on multi-level and multi-scale feature fusion to detect various defects in power equipment. Compared with the existing power equipment defect detection method, the present invention has a better effect in identifying various defects in power equipment, has a high accuracy rate, and has a relatively low error detection rate, and runs at a faster speed.

[0140] This application can be used in intelligent inspection scenarios of power grid equipment. After being deployed on ground hardware with high-performance computing, it can detect and determine defects in power grid equipment in real time, reduce the data backhaul pressure of on-site inspection equipment such as drones and robots, improve the timeliness of equipment operation and maintenance, and ensure the safe and stable operation of power grid equipment.

[0141] Exemplary devices

[0142] Figure 8 FIG is a schematic diagram of a structure of a power grid equipment defect detection device based on multi-level and multi-scale feature fusion provided by an exemplary embodiment of the present invention. Figure 8 As shown, the apparatus 800 includes:

[0143] The acquisition module 810 is used to acquire inspection image data of the power grid equipment to be detected for defects;

[0144] A first determination module 820 is configured to perform a full-category defect detection on the inspection image data using a full-category defect detection model corresponding to the power grid equipment to determine a first defect detection result;

[0145] A second determination module 830 is configured to perform single-class defect detection on the inspection image data using multiple single-class defect detection models corresponding to the power grid equipment to determine a second defect detection result;

[0146] The third determination module 840 is configured to eliminate overlapping results of the first defect detection result and the second defect detection result by using a non-maximum elimination method to determine a defect detection result of the power grid equipment.

[0147] Optionally, the full-category defect detection model is used to detect all defect types covered in the current scenario of power grid equipment inspection. The full-category defect detection model includes: a full-category equipment defect detection model for transmission lines, a full-category equipment defect detection model for distribution lines, a full-category equipment defect detection model for substations, and a full-category equipment defect detection model for converter stations.

[0148] Optionally, the first determining module 820 includes:

[0149] The feature extraction submodule is used to extract features from the inspection image data through the feature extraction layer of the full-class detection model;

[0150] The splicing and fusion submodule is used to splice and fuse all the extracted features after they pass through the fusion self-attention module of the full-category detection model;

[0151] The first determination submodule is used to classify and locate the data features after splicing and fusion, and determine the first defect detection result.

[0152] Optionally, the single-class defect detection model adopts the NanoDet-Plus-m method.

[0153] Optionally, the third determining module 840 includes:

[0154] A construction submodule, configured to construct a plurality of candidate frames to be processed of the defect type according to the first defect detection result and the second defect detection result, and to construct a set M storing the optimal frames;

[0155] The move-in submodule is used to calculate the confidence of each candidate box in the candidate box set, sort the candidate boxes in the candidate box set according to the confidence, and select the candidate box with the highest confidence to move into the set M;

[0156] A removal submodule is used to traverse the candidate boxes in the candidate box set, calculate the dynamic intersection-over-union ratio with the set M respectively, and remove the candidate box from the candidate box set if the intersection-over-union ratio is higher than a preset first threshold, and move the candidate box to the set M if the intersection-over-union ratio is not higher than the first threshold;

[0157] The second determination submodule is configured to sequentially obtain multiple sets M corresponding to candidate frame sets to be processed of multiple defect types when the candidate frame set is empty, and determine the defect detection result according to the multiple sets M.

[0158] Optionally, remove the candidate boxes in the traversed candidate box set in the submodule, and calculate the dynamic intersection and union ratio with the set M respectively, including:

[0159] The calculation unit is used to calculate the intersection-and-union ratio of each candidate frame with the set M according to the dynamic intersection-and-union ratio calculation formula corresponding to the defect type of each candidate frame.

[0160] Optionally, a dynamic intersection-over-union (IoU) calculation formula corresponding to the defect type of the candidate frame includes:

[0161] When the defect type of the candidate frame is a preset small-size defect or a small target defect, the dynamic intersection-over-union ratio calculation formula is:

[0162]

[0163] When the defect type of the candidate frame is a non-small size defect or a non-small target defect, the dynamic intersection-over-union ratio is calculated as follows:

[0164]

[0165] Among them, MIOU is the dynamic intersection-over-union ratio, d is the distance between the center point of the candidate box and the box in the set M, c is the diagonal of the circumscribed rectangle containing the candidate box and the box in the set M, and w s is the width of the candidate box, h s is the height of the candidate box.

[0166] Optionally, the second determining submodule includes:

[0167] A detection unit, configured to detect whether the candidate boxes in the multiple sets M have the same attribute and whether the loss function is greater than a preset second threshold;

[0168] The determination unit is configured to remove the candidate box from the multiple sets M and determine the defect detection result when the candidate boxes in the multiple sets M have the same attribute and the loss function is greater than a preset second threshold.

[0169] Exemplary electronic devices

[0170] Figure 9This is the structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 9 As shown, the electronic device 90 includes one or more processors 91 and a memory 92 .

[0171] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0172] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may further include: an input device 93 and an output device 94, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0173] In addition, the input device 93 may also include, for example, a keyboard, a mouse, and the like.

[0174] The output device 94 can output various information to the outside. The output device 94 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0175] Of course, to simplify, Figure 9 Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0176] Exemplary computer program products and computer-readable storage media

[0177] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the method according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0178] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0179] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method for information mining of historical change records according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0180] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0181] The basic principles of the present invention have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in the present invention are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present invention. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. These details do not necessarily limit the present invention to being implemented using these specific details.

[0182] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For system embodiments, since they are essentially identical to the method embodiments, their description is relatively simple. For relevant parts, refer to the descriptions of the method embodiments.

[0183] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0184] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above sequence of steps for the method is for illustration only, and the steps of the method of the present invention are not limited to the sequence specifically described above, unless otherwise specified. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers recording media that store programs for executing the method according to the present invention.

[0185] It should also be noted that, in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in this field to make or use the present invention. Various modifications to these aspects will be very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but according to the widest scope consistent with the principles disclosed here and novel features.

[0186] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for detecting defects in power grid equipment based on multi-level and multi-scale feature fusion, characterized in that: include: Collect inspection image data of power grid equipment to be detected for defects; Performing full-category defect detection on the inspection image data using a full-category defect detection model corresponding to the power grid equipment to determine a first defect detection result; performing single-class defect detection on the inspection image data using a plurality of single-class defect detection models corresponding to the power grid equipment to determine a second defect detection result; Using a non-maximum elimination method to eliminate overlapping results from the first defect detection result and the second defect detection result, and determining a defect detection result of the power grid equipment, wherein conventional convolutions of multi-level residual units in the sub-backbone of the first two large-size inputs of the full-class defect detector adopted by the defect detection method are replaced with DSCs; Eliminating overlapping results of the first defect detection result and the second defect detection result by using a non-maximum elimination method to determine the defect detection result of the power grid device includes: Constructing a plurality of candidate frames to be processed of the defect type according to the first defect detection result and the second defect detection result, and constructing a set M for storing the optimal frames; Calculate the confidence of each candidate box in the candidate box set respectively, sort the candidate boxes in the candidate box set according to the confidence, and select the candidate box with the highest confidence and move it into the set M; traversing the candidate boxes in the candidate box set, and calculating a dynamic intersection-over-union (IoU) with the set M respectively; if the IoU is higher than a preset first threshold, removing the candidate box from the candidate box set; and if the IoU is not higher than the first threshold, moving the candidate box into the set M; When the candidate frame set is empty, sequentially obtaining multiple sets M corresponding to candidate frame sets to be processed of multiple defect types, and determining the defect detection result according to the multiple sets M; Traversing the candidate boxes in the candidate box set and calculating the dynamic intersection-union ratio with the set M respectively, including: Calculate the intersection-and-union ratio of each candidate frame with the set M according to the dynamic intersection-and-union ratio calculation formula corresponding to the defect type of each candidate frame; The dynamic intersection-over-union ratio calculation formula corresponding to the defect type of the candidate frame includes: When the defect type of the candidate frame is a preset small-size defect or a small target defect, the dynamic intersection-over-union ratio calculation formula is: When the defect type of the candidate frame is a non-small size defect or a non-small target defect, the calculation formula of the dynamic intersection-over-union ratio is: in, MIOU is the dynamic intersection-union ratio, d is the distance between the center point of the candidate box and the box in the set M, c is the diagonal of the rectangle circumscribing the candidate box and the box in set M, w s is the width of the candidate box, h s is the height of the candidate box.

2. The method according to claim 1, characterized in that The full-category defect detection model is used to detect all defect types covered in the current scenario of the power grid equipment inspection. The full-category defect detection model includes: a full-category equipment defect detection model for transmission lines, a full-category equipment defect detection model for distribution lines, a full-category equipment defect detection model for substations, and a full-category equipment defect detection model for converter stations.

3. The method according to claim 1, characterized in that Performing full-category defect detection on the inspection image data using a full-category defect detection model corresponding to the power grid equipment to determine a first defect detection result includes: Performing feature extraction on the inspection image data through the feature extraction layer of the full-category defect detection model; All the extracted features are concatenated and fused after passing through the fusion self-attention module of the full-category defect detection model; Classify and locate the spliced ​​and fused data features to determine the first defect detection result.

4. The method according to claim 1, wherein The single-class defect detection model adopts the NanoDet-Plus-m method.

5. The method according to claim 1, wherein The operation of determining the defect detection result according to the multiple sets M includes: Detecting whether the candidate boxes in the multiple sets M have the same attribute and whether the loss function is greater than a preset second threshold; When the candidate boxes in the multiple sets M have the same attribute and the loss function is greater than a preset second threshold, the candidate box is removed from the multiple sets M to determine the defect detection result.

6. A power grid equipment defect detection device based on multi-level and multi-scale feature fusion, characterized in that: include: An acquisition module, used to collect inspection image data of power grid equipment to be detected for defects; A first determination module is configured to perform a full-category defect detection on the inspection image data using a full-category defect detection model corresponding to the power grid equipment to determine a first defect detection result; a second determination module, configured to perform single-class defect detection on the inspection image data using a plurality of single-class defect detection models corresponding to the power grid equipment, and determine a second defect detection result; a third determination module, configured to eliminate overlapping results from the first defect detection result and the second defect detection result using a non-maximum elimination method to determine a defect detection result of the power grid device, wherein conventional convolutions of multi-level residual units in the sub-backbone of the first two large-size inputs of the full-class defect detector adopted in the defect detection method are replaced with DSCs; The third determination module includes: Constructing a plurality of candidate frames to be processed of the defect type according to the first defect detection result and the second defect detection result, and constructing a set M for storing the optimal frames; Calculate the confidence of each candidate box in the candidate box set respectively, sort the candidate boxes in the candidate box set according to the confidence, and select the candidate box with the highest confidence and move it into the set M; traversing the candidate boxes in the candidate box set, and calculating a dynamic intersection-over-union (IoU) with the set M respectively; if the IoU is higher than a preset first threshold, removing the candidate box from the candidate box set; and if the IoU is not higher than the first threshold, moving the candidate box into the set M; When the candidate frame set is empty, sequentially obtaining multiple sets M corresponding to candidate frame sets to be processed of multiple defect types, and determining the defect detection result according to the multiple sets M; Traversing the candidate boxes in the candidate box set and calculating the dynamic intersection-union ratio with the set M respectively, including: Calculate the intersection-and-union ratio of each candidate frame with the set M according to the dynamic intersection-and-union ratio calculation formula corresponding to the defect type of each candidate frame; The dynamic intersection-over-union ratio calculation formula corresponding to the defect type of the candidate frame includes: When the defect type of the candidate frame is a preset small-size defect or a small target defect, the dynamic intersection-over-union ratio calculation formula is: When the defect type of the candidate frame is a non-small size defect or a non-small target defect, the calculation formula of the dynamic intersection-over-union ratio is: in, MIOU is the dynamic intersection-union ratio, d is the distance between the center point of the candidate box and the box in the set M, c is the diagonal of the rectangle circumscribing the candidate box and the box in set M, w s is the width of the candidate box, h s is the height of the candidate box.

7. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 5.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 5.

Images