Insulator defect detection method based on improved neural network model and storage medium

Abstract

The application discloses an insulator defect detection method based on an improved neural network model and a storage medium, and the method comprises the following steps: acquiring an image of an insulator to be detected; performing defect detection on the image of the insulator to be detected by using a trained improved neural network model to obtain a defect detection result of the image of the insulator to be detected, wherein the model comprises a feature extraction module used for extracting features of an image, a region proposal module used for generating a candidate box, an interest domain pooling layer and an output module used for classification and regression. The application realizes multi-scale fusion of deep features and shallow features, enhances the weight of important features, improves the capture probability of a target and the detection accuracy of targets of different scales, simultaneously improves the target frame regression mechanism of the model and the scoring mechanism when the overlap degree of a predicted frame is high, reduces the probability that a predicted frame is mistakenly deleted, and can effectively improve the recognition accuracy and precision of insulator defects. The application is applied to the technical field of insulator defect detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of insulator defect detection, and particularly relates to an insulator defect detection method based on an improved neural network model and a storage medium. BACKGROUND

[0002] An insulator is a special insulating control and can play an important role in an overhead transmission line. Since the insulator is exposed to the natural environment for a long time and is negatively affected by severe weather, the insulator is prone to failure, thereby affecting the stable operation of a power system. Therefore, regular inspection of the quality and structure of the insulator can effectively ensure the safe operation of the power system. Currently, regular inspection of the insulator is usually achieved by using a UAV inspection method, which refers to detection by collecting massive insulator aerial images, and can greatly improve the safety and efficiency of the inspection.

[0003] With the development of deep learning, neural networks have been introduced into the field of insulator defect detection, that is, the neural network is used to identify and classify the insulator target of the insulator aerial image collected by the UAV inspection, thereby realizing insulator defect detection. Most of the deep learning methods commonly used for insulator aerial images are based on convolutional neural networks, such as Faster R-CNN, YOLOv3, SSD, etc. These neural networks have high requirements for the quality of the insulator aerial image when detecting the target, and most of them rely on convolutional neural networks to achieve feature extraction, which cannot realize the fusion of the shallow and deep features of the image. This leads to the fact that when they face the task of target detection and classification of the insulator aerial image, they are easily affected by complex environmental factors such as background and noise. Not only can they not extract more effective features from the image, but also the feature extraction effect and target detection effect are poor, and they cannot quickly and accurately detect the insulator defect in the presence of background interference. The algorithm has poor generalization ability, and the real-time detection accuracy of the algorithm needs to be further improved. SUMMARY

[0004] The purpose of the present application is to provide an insulator defect detection method based on an improved neural network model and a storage medium to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0005] The solution to the technical problem of the present application is: in a first aspect, the present application provides an insulator defect detection method based on an improved neural network model, comprising the following steps:

[0006] obtaining an insulator image to be tested;

[0007] The improved neural network model is trained by using labeled sample insulator images and corresponding label results.

[0008] Further, in some embodiments of the present application, the improved neural network model comprises:

[0009] A feature extraction module is configured to extract multi-scale features from the to-be-tested insulator image to generate a plurality of to-be-tested feature maps.

[0010] A region proposal module is configured to generate a corresponding candidate box according to each to-be-tested feature map.

[0011] An interest domain pooling layer is configured to generate a plurality of candidate feature maps by using the plurality of to-be-tested feature maps and the corresponding candidate boxes.

[0012] An output module is configured to calculate the type of insulator defect and the location of the insulator defect in the to-be-tested insulator image, i.e., the defect detection result, according to the plurality of candidate feature maps.

[0013] Further, in some embodiments of the present application, the step of constructing the feature extraction module comprises the following steps: first, a GAM module is introduced after each Swin-Transformer encoder in the original Swin-Transformer network to obtain an improved Swin-Transformer network; then, a PAFPN feature pyramid network is built on the basis of the improved Swin-Transformer network to construct the feature extraction module.

[0014] Further, in some embodiments of the present application, the step of generating a corresponding candidate box according to each to-be-tested feature map comprises:

[0015] For each to-be-tested feature map, the region proposal module is used to classify and perform bounding box regression on the to-be-tested feature map to generate a corresponding initial candidate box, and the redundant initial candidate boxes are filtered out by Soft-NMS to obtain the candidate box corresponding to the to-be-tested feature map.

[0016] Further, in some embodiments of the present application, the region proposal module is RPN.

[0017] Further, in some embodiments of the present application, the region proposal module uses MPDIOU as the loss function of bounding box regression.

[0018] Further, in some embodiments of the present application, the interest domain pooling layer is further configured to unify the size of all candidate feature maps.

[0019] Further, in some embodiments of the present application, the output module comprises a full connection layer and a regression classification structure, which is composed of a regression unit and a classification unit, wherein:

[0020] The classification unit is configured to calculate each candidate feature map by a Softmax function to obtain the maximum probability of the type to which the candidate box of each candidate feature map belongs, and determine the type of insulator defect existing in the to-be-tested insulator image according to the maximum probability.

[0021] The regression unit is configured to perform bounding box regression on the candidate box of each candidate feature map and use Soft-NMS to screen out redundant candidate boxes to obtain the position offset of the candidate box of each candidate feature map, and determine the position of the insulator defect in the to-be-tested insulator image according to the position offset of each candidate box.

[0022] Further, in some embodiments of the present application, the regression unit uses MPDIOU as the loss function of the bounding box regression.

[0023] In a second aspect, the present application provides a storage medium, wherein the instructions executable by the processor are configured to execute the insulator defect detection method based on the improved neural network model when executed by the processor.

[0024] The application has the beneficial effects that the insulator defect detection method and storage medium based on the improved neural network model are provided, the detection of the insulator defect is realized through the improved neural network model, and the improved neural network model comprises a feature extraction module, a region proposal module, an interest domain pooling layer and an output module.

[0025] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be achieved and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The flowchart of the insulator defect detection method provided by the embodiment of the present application is provided;

[0027] Figure 2 The structural diagram of the improved neural network model provided by the embodiment of the present application is provided;

[0028] Figure 3 The structural diagram of the feature extraction module provided by the embodiment of the present application is provided;

[0029] Figure 4 The structural diagram of the improved Swin-Transformer network provided by the embodiment of the present application is provided;

[0030] Figure 5A performance chart of a traditional Faster R-CNN algorithm provided by an embodiment of the present application on the same insulator defect detection task is shown in FIG. 1.

[0031] Figure 6 A performance chart of an improved neural network model provided by an embodiment of the present application on the same insulator defect detection task is shown in FIG. 2. DETAILED DESCRIPTION

[0032] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0033] The present application is further described below in combination with the drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0034] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] An insulator is a special insulating control. With the increase of line voltage level, the insulating voltage of the insulator should be increased, and the structure type of the insulator should be adapted to the line structure type and the operating environment, so that it can play an important role in overhead transmission lines and provide electrical insulation and mechanical support for high-voltage transmission lines. Since the insulator is exposed to the natural environment for a long time and is negatively affected by severe weather, it is prone to failure, which in turn affects the stable operation of the power system. Therefore, regular inspection of the quality and structure of the insulator can effectively ensure the safe operation of the power system.

[0037] Traditional insulator inspection method adopts manual inspection to complete, in order to make up for the deficiency of traditional manual inspection, improve the safety of inspection, adopt the way of unmanned aerial vehicle inspection becomes popular. Unmanned aerial vehicle inspection refers to collecting massive insulator aerial images for detection, which can greatly improve the safety and efficiency of inspection. However, due to the large amount of data collected by unmanned aerial vehicle inspection, the complexity of background, noise and shooting angle leads to the fact that the quality of aerial image cannot meet the detection requirements, and the use of unmanned aerial vehicle inspection to complete insulator defect detection method faces the problems of difficulty in feature extraction, poor detection real-time performance, low recognition accuracy, and the requirement of parameter adjustment and data set cannot meet the requirement of insulator defect detection. Therefore, it is urgent to combine the image processing technology with good robust performance and the defect positioning intelligent algorithm with high execution efficiency to form a general insulator defect positioning algorithm and realize the defect detection of insulator.

[0038] Target detection is a very hot research field in computer vision, which has been widely used in face detection, automatic driving car, power equipment fault detection and other fields. The traditional target detection method mainly extracts specific features of the image according to prior knowledge, and then uses classification algorithm to realize target detection. However, compared with traditional target detection algorithm, deep learning has more advantages in extracting high-level features of image. Therefore, neural network is introduced into the field of insulator defect detection, that is, through neural network, insulator aerial image collected by unmanned aerial vehicle inspection is used for insulator target recognition and classification, and then insulator defect detection is realized. The current deep learning method commonly used in insulator aerial image can be divided into two types, one is the two-stage target detection algorithm based on candidate region, the two-stage detection algorithm is to generate a series of sample candidate frames by a specific algorithm, then classify the samples by convolutional neural network, and finally refine the position of the boundary frame, such as Faster R-CNN; The second is a single-stage target detection algorithm based on regression problem, the difference of single-stage target detection algorithm is that it does not generate candidate frame, but directly regresses, classifies the predicted target, such as YOLO series, SSD, etc. The former is more effective in detection accuracy and positioning accuracy, while the latter has faster detection speed.

[0039] The related deep learning algorithm has the following defects when facing the target detection and classification task of the insulator aerial image: the neural network has a high requirement on the quality of the insulator aerial image when detecting the target, and most of them rely on the convolutional neural network to realize feature extraction, and cannot realize the fusion of the shallow features and the deep features of the image, which leads to that they are easily affected by the complex environmental factors such as background and noise when facing the target detection and classification task of the insulator aerial image, not only cannot extract more effective features from the image, but also cannot detect the insulator defects quickly and accurately in the presence of background interference, the algorithm generalization ability is poor, and the real-time detection accuracy of the algorithm needs to be further improved.

[0040] In view of the problems and defects of the related art, the embodiment of the present application proposes an insulator defect detection method based on an improved neural network model and a storage medium, which is realized based on the Faster R-CNN, and can effectively improve the recognition accuracy and precision of the insulator defects, and has better detection effect on insulator targets of different scales.

[0041] A kind of insulator defect detection method based on improved neural network model proposed by the present application will be described in detail below with reference to the drawings.

[0042] The method in the embodiment of the present application can be applied in a terminal, can also be applied in a server, and can also be software running in a terminal or a server, etc.The terminal can be a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto.The server can be a stand-alone physical server, can also be a server cluster or distributed system composed of multiple physical servers, and can also be a cloud server providing cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and basic cloud computing services such as big data and artificial intelligence platform.

[0043] Referring to Figure 1 The method proposed by the embodiment of the present application mainly includes the following steps:

[0044] S101, obtaining an insulator image to be tested.

[0045] It should be noted that the insulator image to be tested includes at least one insulator defect, and the insulator defect is the target object in the target detection task.

[0046] S102, using the trained improved neural network model to detect the insulator image to be tested, to obtain the defect detection result of the insulator image to be tested.

[0047] It should be noted that the trained improved neural network model is obtained by training using the labeled insulator image samples and the corresponding labeled results.

[0048] Optionally, the step of training the improved neural network model comprises the following steps: firstly, obtaining insulator image samples taken from multiple backgrounds and multiple angles, labeling through Labelimg software, and constructing an insulator image dataset in the format of VOC2007. Then, processing the dataset through data enhancement, performing operations such as flipping, translation, rotation, and adding data noise on the collected images, and expanding the dataset to 6000. Finally, training and testing the improved neural network model using the expanded dataset, and then obtaining the final trained improved neural network model.

[0049] Further, referring to Figure 2 The improved neural network model proposed in the embodiments of the present application comprises the following parts:

[0050] The feature extraction module 201, i.e., the backbone network, functions to extract multi-scale features of the to-be-tested insulator image and generate a plurality of to-be-tested feature maps.

[0051] The region proposal module 202 functions to generate corresponding candidate boxes according to each to-be-tested feature map.

[0052] The interest domain pooling layer 203 functions to generate a plurality of candidate feature maps using the plurality of to-be-tested feature maps and the corresponding candidate boxes.

[0053] The output module 204 functions to calculate the type of insulator defect and the position of the insulator defect in the to-be-tested insulator image, i.e., the defect detection result, according to the plurality of candidate feature maps.

[0054] One embodiment of the present application will be further described below with reference to the drawings to describe the structure and functions of the feature extraction module, the region proposal module, the interest domain pooling layer, and the output module of the improved neural network model proposed in the present application.

[0055] 1. The structure and functions of the feature extraction module 201 are as follows:

[0056] Referring to Figure 3The feature extraction module 201 of the embodiment of the application is composed of an improved Swin-Transformer network and a PAFPN. The steps of constructing the feature extraction module 201 mainly include: first, a GAM module is introduced behind each Swin-Transformer encoder in the original Swin-Transformer network to obtain an improved Swin-Transformer network. Then, on the basis of the improved Swin-Transformer network, a PAFPN feature pyramid network is built, and then the feature extraction module is constructed.

[0057] As a further implementation manner, with reference to Figure 4 The improved Swin-Transformer network is sequentially provided with a block input layer and four feature extraction units connected in sequence. In the four feature extraction units, the output of the previous feature extraction unit will be used as the input of the next feature extraction unit. The first feature extraction unit is composed of a linear embedding layer and a Swin-Transformer encoder, and the last three feature extraction units are each composed of a convolution-free downsampling layer and a Swin-Transformer encoder. Each Swin-Transformer encoder is connected with a GAM module behind it.

[0058] Specifically, the functions of each component in the improved Swin-Transformer network are as follows:

[0059] The function of the block input layer is to perform pixel block division on the image of the insulator to be tested to obtain a plurality of pixel blocks and input them into the first feature extraction unit.

[0060] For the first feature extraction unit, the function of the linear embedding layer is to perform linear transformation on the plurality of pixel blocks, and the plurality of pixel blocks after linear transformation are used as the input of the Swin-Transformer encoder. The function of the Swin-Transformer encoder is to use the multi-head self-attention mechanism of the shiftable window to extract features from the input data, and the output of the Swin-Transformer encoder is used as the input of the GAM module. The function of the GAM module is to obtain a global context vector by calculating the feature representation of the input sequence, use the global context vector to calculate the weight of each position to determine the attention allocation of the model at different positions, and perform weighted summation on the input sequence according to the weights to obtain the weighted output representation, i.e. the output of the first feature extraction unit.

[0061] For the last three feature extraction units, the main role of the non-convolution downsampling layer is to downsample the output input to the previous feature extraction unit to reduce the resolution, adjust the number of channels, and thus form a hierarchical feature map, while saving some computational load. The output of the non-convolution downsampling layer will be input to the Swin-Transformer encoder, and the output of the Swin-Transformer encoder will be input to the GAM module. The functions of the Swin-Transformer encoder and the GAM module of the last three feature extraction units are the same as those of the Swin-Transformer encoder and the GAM module of the first feature extraction unit, and will not be repeated here.

[0062] In the embodiment of the application, the Swin-Transformer network draws on the idea of the CNN network, which embodies the hierarchical and localized characteristics in the network architecture. Among them, the Swin-Transformer network adopts hierarchical design and contains four stages in total, each of which corresponds to a feature extraction unit. Except for the first stage, each stage will first reduce the resolution of the input and perform downsampling operation through the non-convolution downsampling layer, and gradually expand the receptive field to obtain global information. It can be seen that the hierarchical nature of the Swin-Transformer network lies in the use of 4-fold, 8-fold and 16-fold downsampling in turn, so that its receptive field gradually increases, achieving multi-scale feature extraction effect. For the Swin-Transformer network, the window multi-head self-attention mechanism (W-MSA) and the shift window multi-head self-attention mechanism (SW-MSA) are the implementation forms of its Swin-Transformer encoder. The Swin-Transformer network uses the shift window multi-head self-attention mechanism to model the dependency between different spatial position features, effectively capture global context information, and has better feature extraction capability. It can be seen that the superiority of the Swin-Transformer network lies in that the calculation of the attention of the Swin-Transformer network is carried out in the window, so its computational complexity is significantly reduced, and the information interaction between multiple windows is realized by relying on the shift window self-attention mechanism.

[0063] Further, based on the Swin-Transformer network, the application introduces a GAM (Global Attention Mechanism) module behind each stage of the Swin-Transformer network. The GAM module is implemented based on the GAM attention mechanism. The GAM module obtains a global context vector by calculating the feature representation of the input sequence, uses the global context vector to calculate the weight of each position to determine the attention distribution of the model at different positions, and performs weighted summation on the input sequence according to the weights to obtain the weighted representation. The embodiment of the application introduces the GAM attention mechanism into each stage of the Swin-Transformer network, so that the feature extraction module 201 can better utilize the global context information and improve the attention ability to different positions, thereby improving the performance of various sequence tasks.

[0064] As a further implementation manner, referring to Figure 3 The original PAFPN includes four stages, which can be one-to-one corresponding to the four stages of the Swin-Transformer network. Based on this, the embodiment of the application proposes to combine the four stages of the improved Swin-Transformer network with the four stages of the original PAFPN on the basis of improving the Swin-Transformer network, so as to build a PAFPN feature pyramid network and further realize the fusion of multi-scale features. The PAFPN combines the FPN and the bottom-up path enhancement, shortens the information path between the bottom layer and the top layer, can be used for detecting objects of different scales, can be used for constructing advanced semantic feature maps of all scales, can combine low-resolution, strong semantic features with high-resolution, weak semantic features; and the pyramid channel and various transverse connections such as cross-scale and intra-scale are interwoven to realize the exchange of information of all levels, so as to build a powerful network with fine resolution and high resolution ability across spatial dimensions.

[0065] More specifically, the implementation process of the feature extraction module 201 can be divided into four implementation stages, each implementation stage including a feature extraction stage and a feature fusion stage, and the feature extraction stage is mainly executed by the improved Swin-Transformer network. The data flow of each implementation stage is as follows:

[0066] The to-be-tested insulator image is input to the feature extraction module 201 and also input to the first implementation stage.

[0067] For the first implementation stage, in the first feature extraction stage, the input data is output to the second feature extraction unit after being extracted by the first feature extraction unit, and is also output to the first feature fusion stage.

[0068] In the first feature fusion stage, first, the input data of the first feature extraction stage is subjected to convolution processing by a convolution layer with a convolution kernel of 1x1 to adjust the channels of different input data and ensure that each input data has the same size for subsequent feature fusion. The input data after convolution processing is fused with the data transferred from the second feature fusion stage to the first feature fusion stage by upsampling, and then a fused feature map of the first feature fusion stage is obtained. The fused feature map is sequentially subjected to processing by convolution layers with convolution kernels of 3x3, 1x1 and 3x3, and is output as the first to-be-tested feature map. In addition, the fused feature map processed by the convolution layers with convolution kernels of 3x3 and 1x1 is input into the concatenation layer of the second feature fusion stage by downsampling.

[0069] For the second and third implementation stages, the output of the previous feature extraction unit is input into the current feature extraction unit and subjected to feature extraction in the current feature extraction stage. The output of the current feature extraction unit is used as the input of the next implementation stage and the input of the current feature fusion stage.

[0070] In the second feature fusion stage, first, the input data of the second feature extraction stage is subjected to convolution processing by a convolution layer with a convolution kernel of 1x1. The input data after convolution processing is fused with the data transferred from the third feature fusion stage to the second feature fusion stage by upsampling, and then a fused feature map of the second feature fusion stage is obtained. The fused feature map is then subjected to processing by a convolution layer with a convolution kernel of 3x3, and is concatenated with the data transferred from the first feature fusion stage to the current feature fusion stage by downsampling to obtain a concatenated feature map of the second feature fusion stage. Finally, the concatenated feature map of the second feature fusion stage is subjected to processing by convolution layers with convolution kernels of 1x1 and 3x3, and is output as the second to-be-tested feature map. In addition, in the second feature fusion stage, the fused feature map is transferred to the first feature fusion stage by upsampling, and the concatenated feature map processed by the convolution layer with a convolution kernel of 1x1 is input into the third feature fusion stage by downsampling.

[0071] In the third feature fusion stage, firstly, the input data of the third feature extraction stage is processed by a convolution layer with a convolution kernel of 1*1, and the processed input data is fused with the data transmitted from the fourth fusion stage to the third feature fusion stage by upsampling, to obtain the fusion feature map of the third feature fusion stage. Then, the fusion feature map of the third feature fusion stage is processed by a convolution layer with a convolution kernel of 3*3, and the processed fusion feature map is spliced with the data transmitted from the second feature fusion stage to the third feature fusion stage by downsampling, to obtain the spliced feature map of the third feature fusion stage. After that, the spliced feature map is processed by a convolution layer with a convolution kernel of 1*1 and 3*3, and the processed spliced feature map is output as the third to-be-tested feature map. In addition, in the third feature fusion stage, the fusion feature map is transmitted to the second feature fusion stage by upsampling, and the spliced feature map processed by a convolution layer with a convolution kernel of 1*1 is transmitted to the fourth feature fusion stage by downsampling.

[0072] For the fourth implementation stage, in the fourth feature extraction stage, the output of the feature extraction unit of the third implementation stage is the input of the feature extraction unit of the fourth implementation stage. After the input data is processed by the feature extraction unit of the fourth implementation stage, the processed data is output to the fourth feature fusion stage.

[0073] In the fourth feature fusion stage, firstly, the input data of the fourth feature extraction stage is processed by a convolution layer with a convolution kernel of 1*1 and 3*3 in sequence, and the processed input data is spliced with the data transmitted from the third feature fusion stage to the fourth feature fusion stage by downsampling, to obtain the spliced feature map of the fourth feature fusion stage. After that, the spliced feature map is processed by a convolution layer with a convolution kernel of 1*1 and 3*3, and the processed spliced feature map is output as the fourth to-be-tested feature map. In addition, the input data processed by a convolution layer with a convolution kernel of 1*1 is transmitted to the third feature fusion stage by upsampling.

[0074] The four feature fusion stages of the embodiment of the present application are combined to form a pyramid structure, each stage only processes single scale information, the process from bottom to top is performed by upsampling, and the transverse connection is to fuse the results of upsampling and the data of the same size generated from top to bottom. After fusion, a convolution layer with a 3*3 convolution kernel is used to convolve each fusion result to eliminate the aliasing effect caused by upsampling. After generating feature maps of different scales, a convolution layer with a 1*1 convolution kernel is used to obtain spliced feature maps of different results by downsampling and tensor connection operation, and finally a 3*3 convolution kernel is used to convolve each spliced result to generate to-be-tested feature maps with different scale information.

[0075] The application replaces the traditional backbone network of Faster R-CNN with a brand-new backbone network, and introduces a PAFPN feature pyramid in the latter half of the backbone network, which not only enhances the weight of important features, makes the positioning of the detection frame more accurate, but also multi-scale fuses deep features and shallow features, improves the capture probability of the target and the detection accuracy of different scale targets, realizes the extraction of strong semantic information and fine resolution features, and can achieve better feature extraction effect, which is beneficial to the subsequent identification work of the model. At the same time, the GAM attention mechanism is introduced into the Swin-Transformer network, so that the model can better utilize the global context information and improve the attention ability to different positions, thereby improving the performance of various sequence tasks.

[0076] 2. The structure and function of the region proposal module 202 are as follows:

[0077] The region proposal module 202 of the embodiment of the application is an RPN (Region Proposal Network), which generates corresponding candidate boxes according to a plurality of to-be-tested feature maps output by the feature extraction module 201. The specific implementation steps are as follows: for each to-be-tested feature map, the region proposal module 202 performs classification and bounding box regression on the to-be-tested feature map to generate corresponding initial candidate boxes, and then uses Soft-NMS to screen the initial candidate boxes. Specifically, redundant initial candidate boxes are screened out, and the initial candidate boxes left after screening are the candidate boxes corresponding to the to-be-tested feature map.

[0078] More specifically, first, on each to-be-tested feature map, nine anchors are preset for each pixel point. The purpose of setting the anchors is to cover targets of different sizes at different positions on the image, and these anchors can be used for subsequent classification and regression operations in the RPN. Second, a Softmax classifier is used to determine whether there is an insulating sub defect in the preset anchors, that is, to determine whether the anchors belong to the foreground or the background, and to extract the anchors belonging to the foreground. Then, the displacement of the anchors is corrected by using the Bounding Box Regression (B-Box Regression) method to obtain more accurate candidate boxes. In the process of bounding box regression, Soft-NMS is used to screen out redundant candidate boxes generated in the above bounding box regression process, thereby obtaining more accurate candidate boxes.

[0079] Further, the region proposal module 202 uses MPDIOU as the loss function of the bounding box regression.

[0080] 3. The structure and function of the interest domain pooling layer 203 are as follows:

[0081] Reference Figure 2The interest domain pooling layer 203 is a pooling layer based on a maximum pooling manner. The interest domain pooling layer 203 has two main functions. One is to generate a plurality of candidate feature maps according to a plurality of to-be-tested feature maps and corresponding candidate boxes. The other is to unify the sizes of all candidate feature maps. This is because the sizes of the plurality of candidate boxes generated by the RPN are different, and the sizes of the input features of the subsequent output module 204 must be the same. Therefore, the operation of unifying the sizes needs to be performed by the interest domain pooling layer 203.

[0082] 4. The configuration and functions of the output module 204 are as follows:

[0083] Referring to Figure 2 The output module 204 includes a fully connected layer and a regression classification structure. The regression classification structure is composed of a classification unit and a regression unit. Each candidate feature map is input to the classification unit and the regression unit through the fully connected layer. The classification unit is used as a classification branch, and the regression unit is used as a regression branch. Specifically:

[0084] The function of the classification unit is to calculate each candidate feature map by using a Softmax function to obtain the maximum probability of the type to which the candidate box of each candidate feature map belongs. According to the maximum probability, the type of the insulator defect existing in the to-be-tested insulator image is determined, so as to realize classification. It should be noted that the value range of the maximum probability is [0, 1].

[0085] The function of the regression unit is to perform bounding box regression on the candidate box of each candidate feature map and use Soft-NMS to filter out redundant candidate boxes to obtain the position offset of the candidate box of each candidate feature map. According to the position offset of each candidate box, the position of the insulator defect in the to-be-tested insulator image is determined, so as to realize positioning.

[0086] Further, the regression unit uses MPDIOU as the loss function of the bounding box regression.

[0087] In the bounding box regression processing of the RPN and the recognition part of the improved neural network model of the embodiment of the present application, MPDIOU is used as the target box regression loss calculation method, the Soft NMS algorithm is used to further optimize the defect detection of the insulator, the model target box regression mechanism and the scoring mechanism when the prediction box overlap degree is high are improved, the probability of the prediction box being mistakenly deleted is reduced, and the position accuracy of the target detection box is improved.

[0088] The overall implementation process of the technical solution proposed in the embodiment of the present application is as follows:

[0089] Firstly, insulator image samples are collected in multiple backgrounds and multiple angles, and are labeled using Labelimg software to construct an insulator dataset in VOC2007 format. The dataset is processed through data enhancement, and the collected images are flipped, translated, rotated, and increased in data noise, etc. to expand the dataset to 6000. The improved neural network model is trained and tested using the insulator dataset, and then the final trained improved neural network model is generated. Then, the insulator images to be tested are collected, and the insulator images to be tested are input into the improved neural network model. The feature extraction module 201 of the model extracts features of the images to be tested to obtain a plurality of feature maps (Feature Maps) to be tested, the plurality of feature maps to be tested are input into the region proposal module 202, i.e. RPN, to obtain a plurality of candidate boxes (Region Proposals), and the size of the plurality of candidate boxes is unified through the region of interest pooling layer 203 (ROIPooling) to generate a plurality of candidate feature maps (Proposal Feature Maps). The plurality of candidate feature maps are input into the subsequent input module, and classification and positioning are performed through the classification branch and the regression branch respectively to obtain the insulator detection result.

[0090] In addition, the disclosed insulator defect image dataset is used to verify the effectiveness, efficiency and generalization of the improved neural network model. Referring to Figure 5 and Figure 6 , Figure 5 is a performance diagram of the traditional Faster R-CNN algorithm on the same insulator defect detection task, Figure 6 is a performance diagram of the improved neural network model proposed in the embodiment of the present application on the same insulator defect detection task. It can be seen that the mAP value of the improved neural network model proposed in the embodiment of the present application reaches 96.74%, which is 7.35% higher than that of the traditional Faster R-CNN algorithm model with Resnet50 as the backbone network. The embodiment of the present application effectively improves the recognition accuracy of insulator defects and has better detection effect on targets of different scales.

[0091] In addition, the embodiment of the present application also provides a storage medium having processor-executable instructions stored therein, and the processor-executable instructions are used to execute the insulator defect detection method based on the improved neural network model when executed by the processor.

[0092] Similarly, the contents in the above method embodiments are applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved by the present storage medium embodiment are also the same as those of the above method embodiments.

[0093] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of accomplishing functionalities that are either the same or similar to that of other embodiments of the application. Moreover, the terms "comprises", "comprising", "includes", "including" and the like, are specifically intended to be open-ended. That is, references to individual steps and / or structures are intended to suggest that it is possible to "comprise", "comprising", "includes", "including" two or more steps or structures, without requiring such steps or structures to be present. The use of the terms "a" and "an" and "the" and "said" and "at least one" are intended to bring into consideration the fact that more than one can exist.

[0094] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0095] In several embodiments provided in the application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0096] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0097] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0098] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes various media that can store program codes, such as a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

[0099] For the step numbers in the foregoing method embodiments, they are set only for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

Claims

1. A method for detecting insulator defects based on an improved neural network model, characterized in that, Includes the following steps: Acquire an image of the insulator to be tested; The defect detection results of the insulator image under test are obtained by using a trained improved neural network model. The trained improved neural network model is obtained by training on labeled sample insulator images and their corresponding labeling results. The improved neural network model includes: The feature extraction module is used to extract multi-scale features from the image of the insulator to be tested, and generate multiple feature maps to be tested; The region suggestion module is used to generate corresponding candidate boxes based on each of the feature maps to be tested; Interest region pooling layer is used to generate multiple candidate feature maps using multiple feature maps to be tested and the corresponding candidate boxes; The output module is used to calculate the type of insulator defect present in the image of the insulator to be tested and the location of the insulator defect in the image of the insulator to be tested based on multiple candidate feature maps, i.e., the defect detection result; Based on the improved Swin-Transformer network, a PAFPN feature pyramid network is built, and then a feature extraction module is constructed. The improved Swin-Transformer network includes four sequentially connected feature extraction units. The first feature extraction unit includes a linear embedding layer and a Swin-Transformer encoder, while the next three feature extraction units include a non-convolutional downsampling layer and a Swin-Transformer encoder. Each Swin-Transformer encoder is connected to a GAM module. In the first feature fusion stage, the output of the first feature extraction unit is processed by a convolutional layer with a 1×1 kernel and then fused with the data from the second feature fusion stage, which is upsampled and passed to the first feature fusion stage, to obtain the fused feature map of the first feature fusion stage. The fused feature map of the first feature fusion stage is then processed by convolutional layers with 3×3, 1×1, and 3×3 kernels in sequence to output the test feature map of the first feature fusion stage. Among them, the fused feature map of the first feature fusion stage is processed by convolutional layers with 3×3 and 1×1 kernels and then downsampled and passed to the second feature fusion stage. For the second and third feature fusion stages, the output of the feature extraction unit corresponding to the current feature fusion stage is processed by a 1×1 convolutional layer and then fused with the data passed to the current feature fusion stage through upsampling from the next feature fusion stage to obtain the fused feature map of the current feature fusion stage. The fused feature map of the current feature fusion stage is then processed by a 3×3 convolutional layer and then concatenated with the data passed to the current feature fusion stage through downsampling from the previous feature fusion stage to obtain the concatenated feature map of the current feature fusion stage. The concatenated feature map of the current feature fusion stage is then processed by convolutional layers with 1×1 and 3×3 kernels to output the test feature map of the current feature fusion stage. Among these, the fused feature map of the current feature fusion stage is passed to the previous feature fusion stage through upsampling, and the concatenated feature map of the current feature fusion stage is passed to the next feature fusion stage through downsampling after being processed by a 1×1 convolutional layer. In the fourth feature fusion stage, the output of the fourth feature extraction unit is processed by convolutional layers with kernels of 1×1 and 3×3, and then concatenated with the data from the third feature fusion stage that has been passed to the fourth feature fusion stage through downsampling. This concatenated feature map of the fourth feature fusion stage is then processed by convolutional layers with kernels of 1×1 and 3×3 to output the test feature map of the fourth feature fusion stage. The output of the fourth feature extraction unit is processed by a convolutional layer with kernels of 1×1 and then passed to the third feature fusion stage through upsampling.

2. The insulator defect detection method based on an improved neural network model according to claim 1, characterized in that, The step of generating a corresponding candidate box based on each of the feature maps to be tested includes: For each feature map to be tested, the region proposal module classifies and regresses bounding boxes to generate corresponding initial candidate boxes. Redundant initial candidate boxes are then filtered out using Soft-NMS to obtain the candidate boxes corresponding to the feature map to be tested.

3. The insulator defect detection method based on an improved neural network model according to claim 1, characterized in that, The region suggestion module is RPN.

4. The insulator defect detection method based on an improved neural network model according to claim 1, characterized in that, The region suggestion module uses MPDIOU as the loss function for bounding box regression.

5. The insulator defect detection method based on an improved neural network model according to claim 1, characterized in that, The interest region pooling layer is also used to unify the size of all candidate feature maps.

6. The insulator defect detection method based on an improved neural network model according to claim 1, characterized in that, The output module includes a fully connected layer and a regression classification structure. The regression classification structure is composed of regression units and classification units connected in parallel, wherein: The classification unit is used to calculate the maximum probability of the candidate box type of each candidate feature map by using the Softmax function, and to determine the type of insulator defect in the insulator image to be tested based on the maximum probability. The regression unit is used to perform bounding box regression on the candidate boxes of each candidate feature map and use Soft-NMS to filter out redundant candidate boxes to obtain the position offset of the candidate boxes of each candidate feature map. Based on the position offset of each candidate box, the position of the insulator defect in the image of the insulator to be tested is determined.

7. The insulator defect detection method based on an improved neural network model according to claim 6, characterized in that, The regression unit uses MPDIOU as the loss function for bounding box regression.

8. A storage medium storing processor-executable instructions, characterized in that, The processor-executable instructions, when executed by the processor, are used to perform the insulator defect detection method based on the improved neural network model as described in any one of claims 1-7.

Images