A pin defect detection method based on a dynamic fine-grained neural network
The pin defect detection method based on dynamic fine-grained neural networks, utilizing image acquisition equipment and an improved RetinaNet model, combined with adaptive training sample selection and feature fusion techniques, solves the problems of high cost of traditional manual inspection and low recognition accuracy of deep learning, and achieves efficient and automated pin defect detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-30
- Publication Date
- 2026-03-20
AI Technical Summary
Traditional manual inspection of pin defects is costly and inaccurate. Existing deep learning networks are unable to effectively identify pin defects in similar targets, resulting in low efficiency in power transmission line inspection.
A pin defect detection method based on dynamic fine-grained neural network is adopted. The dataset is acquired through image acquisition device, and the RetinaNet-ResNet50 model is used for training. ACmix convolution and fine-grained dynamic routing space structure are introduced, and an adaptive training sample selection method is combined to optimize the feature extraction and detection accuracy of the model.
It achieves highly efficient and automated pin defect detection, improves the detection accuracy of similar targets, reaches an AP0.5 detection accuracy rate of 94.0%, and reduces manpower and material costs.
Smart Images
Figure CN115908345B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image recognition, and in particular to a pin defect detection method based on a dynamic fine-grained neural network. BACKGROUND
[0002] In recent years, with the continuous development of the power industry, the demand for safety detection of power transmission lines is also expanding. As an indispensable part of the power transmission line, the pin plays a role in fixing the nut to prevent the soft connection part from moving. The loss, loosening and other abnormal states of the pin may affect the normal operation of the remaining components on the power transmission line, causing large-area power transmission line failure, and seriously threatening the safety and stability of the power grid. Therefore, it is necessary to detect the defects of the pins on the power transmission line to ensure the safety of the power transmission line.
[0003] Traditional pin defect detection is mainly through on-site inspection by technical personnel to check whether the pin has defects. However, with the expansion of the power grid, this manual inspection method not only greatly increases the labor cost but also cannot guarantee the accuracy. In recent years, with the rapid development of deep learning, power grid enterprises have gradually used target detection networks based on deep learning to detect the defect state of the pin. However, due to the following difficulties in pin defect detection:
[0004] 1. The loosened pin is very similar to the normal state pin;
[0005] 2. The pin defect state is diverse;
[0006] 3. The aerial picture generally has a large background, small target and multiple targets in one picture;
[0007] 4. The commonly used single-stage network (RetinaNet, Yolo series) and two-stage network (RCNN series) are difficult to effectively identify similar targets, so the current pin defect detection is still a difficult problem. SUMMARY
[0008] The present application provides a pin defect detection method based on a dynamic fine-grained neural network to solve the technical problem of high labor cost and low efficiency in the prior art that pin defect detection is generally determined by technical personnel on site.
[0009] To achieve the above purpose, the technical solution of the present application is as follows:
[0010] The present application provides a pin defect detection method based on a dynamic fine-grained neural network, comprising the following steps:
[0011] Step S1, using an image acquisition device to acquire multiple pin images, and using the multiple pin images to construct a pin image dataset;
[0012] Step S2, the pin image in the pin image data set is labeled, and the labeled pin image is data enhanced to obtain a data enhanced pin image data set; the enhanced pin image data set is divided into a training set and a verification set;
[0013] Step S3, a RetinaNet-ResNet50 model is established, and an improved RetinaNet model is constructed based on the RetinaNet-ResNet50 model;
[0014] Step S4, the improved RetinaNet model is trained by using the training set to obtain a trained RetinaNet model;
[0015] Step S5, the verification set is input into the trained RetinaNet model to obtain a pin defect recognition result.
[0016] Preferably, the step S2 specifically comprises the following steps:
[0017] Step S21, the pin image data set is labeled by using a data labeling software to obtain a labeled pin image data set;
[0018] Step S22, the labeled pin image data set is image enhanced to obtain an image enhanced pin image data set;
[0019] Step S23, the enhanced pin image data set is divided into a training set and a verification set according to a specified proportion.
[0020] Preferably, the image enhancement in the step S22 includes scaling, horizontal flipping, inversion and clipping.
[0021] Preferably, the improved RetinaNet model in the step S3 includes a backbone network, an improved feature pyramid network and a detection head connected in sequence.
[0022] The backbone network is composed of an improved ResNet-50, and the backbone network is used for extracting features in the pin image in the pin image data set;
[0023] The improved feature pyramid network is used for fusing the features extracted by the backbone network and outputting a plurality of scale different fine-grained features;
[0024] The detection head includes a classification head and a regression head, and the classification head and the regression head are connected with the improved feature pyramid network, and the classification head and the regression head are used for decoupling the plurality of scale different fine-grained features and respectively predicting the category and the position of the target.
[0025] Preferably, the improved ResNet-50 comprises five convolution modules connected in sequence, and the last convolution module comprises two Bottleneck and one improved Bottleneck, and the two Bottleneck and the improved Bottleneck are connected in sequence.
[0026] Preferably, the improved feature pyramid network comprises a FPN structure and a dynamic fine-grained feature fusion spatial structure connected in sequence.
[0027] The FPN structure is used for feature fusion of features extracted by the backbone network.
[0028] The dynamic fine-grained feature fusion spatial structure is mainly composed of a fine-grained channel selector unit, and the fine-grained channel selector unit is used for converting the fused features into a plurality of fine-grained features with different scales.
[0029] Preferably, the step S4 specifically comprises the following steps:
[0030] Step S41, inputting the training set into the improved RetinaNet model;
[0031] Step S42, extracting features in the pin image by the backbone network;
[0032] Step S43, inputting the extracted features into the FPN structure for feature fusion, and converting the fused features into a plurality of fine-grained features with different scales by using the dynamic fine-grained feature fusion spatial structure;
[0033] Step S44, decoupling the plurality of fine-grained features with different scales by using the classification head and the regression head, and respectively predicting the category and the position of the target;
[0034] Step S45, performing positive and negative sample assignment on the predicted category and position by using the adaptive training sample selection method;
[0035] Step S46, establishing a Focal loss loss function, repeating steps S41 to S46, minimizing the loss function until the loss function converges.
[0036] Preferably, the step S45 specifically comprises the following steps:
[0037] Step S451, for the category and position of the target, there are a plurality of real boxes for each pin image, and candidate positive samples are found according to a specified rule;
[0038] Step S452, calculating the intersection-over-union between the candidate positive sample points and the real boxes, and calculating the mean and standard deviation thereof, denoted as m g and v g , wherein the intersection-over-union refers to b1 and b2 represent candidate positive samples and real boxes respectively;
[0039] Step S453, the intersection over union threshold of the real box is calculated as t g g =m g +v g ;
[0040] Step S454, the candidate sample with the intersection over union greater than or equal to the threshold t g is selected as the final positive sample.
[0041] Preferably, the Focal loss loss function in the step S46 is specifically:
[0042]
[0043] Wherein: p represents the predicted target probability, and alpha is a factor for controlling the weight of positive and negative samples, and gamma is a weight factor for controlling difficult samples.
[0044] Preferably, the step S4 further comprises the following steps:
[0045] Step S47, the improved RetinaNet model is fine-tuned to obtain the trained RetinaNet model.
[0046] The beneficial effects of the present application are:
[0047] 1. Image acquisition is performed by an image acquisition device such as a drone, which greatly saves manpower and resources;
[0048] 2. The introduction of ACmix convolution improves the ability of the backbone network to extract features, so that the model can focus on different regions in the nail picture and capture more features;
[0049] 3. The Fine-Grained Dynamic Routing Space structure (dynamic fine-grained feature fusion space structure) dynamically selects pixel-level combinations of different scales of FPN (feature pyramid network) structure for each target to be detected, strengthens the multi-scale representation ability of the FPN structure, and enables the model to extract fine-grained features of the nail, thereby improving the detection accuracy of similar targets; the adaptive training sample selection method alleviates the problem of imbalance between positive and negative samples of the model;
[0050] 4. The improved RetinaNet model can effectively detect similar targets in the nail data set, and the AP 0.5 of the nail defect detection can reach 94.0%, which can meet the needs of nail defect detection. BRIEF DESCRIPTION OF DRAWINGS
[0051] Figure 1 is the implementation flowchart of the present application;
[0052] Figure 2 is the overall framework diagram of the improved RetinaNet model constructed by the present application;
[0053] Figure 3 is the structure diagram of the Fine-Grained Dynamic Routing Space (dynamic fine-grained feature fusion space) in the present application;
[0054] Figure 4 is the transmission line pin defect state detection effect diagram described in the present application. DETAILED DESCRIPTION
[0055] The present application will be further described in detail below in combination with the drawings and specific embodiments.
[0056] In the present application, "first", "second", etc. are used for description purposes only, and cannot be understood as indicating or implying the relative importance of the technical features indicated or the number or order of the technical features indicated. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically limited.
[0057] With reference to Figure 1 , the embodiment of the present application provides a pin defect detection method based on a dynamic fine-grained neural network, comprising the following steps:
[0058] Step S1, using an image acquisition device to acquire a plurality of pin images, and using a plurality of pin images to construct a pin image dataset; the image acquisition device can be a drone, or other devices with camera and photography functions;
[0059] Step S2, labeling the pin images in the pin image dataset, and data enhancing the labeled pin images to obtain a data enhanced pin image dataset; dividing the enhanced pin image dataset into a training set and a validation set; through the distribution diversity of the data enhanced pin image dataset, the generalization ability of the model is improved.
[0060] Step S3, establishing a RetinaNet-ResNet50 model, and constructing an improved RetinaNet model based on the RetinaNet-ResNet50 model;
[0061] Step S4, training the improved RetinaNet model using the training set to obtain a trained RetinaNet model;
[0062] Step S5, input the verification set into the trained RetinaNet model to obtain a pin defect recognition result, Figure 4 For the transmission line pin defect state detection effect diagram, for reference;
[0063] In the embodiment, the step S2 specifically comprises the following steps:
[0064] Step S21, labeling the pin images in the pin image data set by using a data labeling software to obtain a labeled pin image data set; the data labeling software is preferably Labelme (a data labeling software);
[0065] Step S22, image enhancement is performed on the labeled pin image data set to obtain an image-enhanced pin image data set;
[0066] Step S23, dividing the enhanced pin image data set into a training set and a verification set according to a specified ratio; the specified ratio can be 9:1 or other ratios.
[0067] In the embodiment, the image enhancement in the step S22 includes scaling, horizontal flipping, inversion and clipping. By enhancing the distribution diversity of the pin image data set, the generalization ability of the model is improved.
[0068] Referring to Figure 2 In the embodiment, the improved RetinaNet model in the step S3 comprises a backbone network, an improved feature pyramid network and a detection head connected in sequence.
[0069] The backbone network is composed of an improved ResNet-50, and the backbone network is used for extracting features in the pin images in the pin image data set;
[0070] The improved feature pyramid network is used for fusing the features extracted by the backbone network and outputting a plurality of fine-grained features with different scales;
[0071] The detection head comprises a classification head and a regression head, both of which are connected with the feature pyramid network, and the classification head and the regression head are used for decoupling the plurality of fine-grained features with different scales and respectively predicting the category and the position of the target.
[0072] In the embodiment, the improved ResNet-50 comprises five convolution modules connected in sequence, which are a first convolution module, a second convolution module, a third convolution module, a fourth convolution module and a fifth convolution module;
[0073] The fifth convolutional module includes two bottleneck layers and one improved bottleneck layer, and the two bottleneck layers and the improved bottleneck layer are sequentially connected.
[0074] The backbone network of the traditional RetinaNet model is composed of ResNet-50, ResNet-50 includes five convolutional modules, and the fifth convolutional module includes three bottleneck layers, and the last bottleneck layer in the fifth convolutional module is converted into an ACmix convolution in the application, the self-attention mechanism can capture the internal correlation of the features, and the ordinary convolution shares the weight value in feature extraction, which introduces an inductive bias to the model, the ACmix convolution has the advantages of both, and the introduction of the ACmix convolution enables the model to focus on different regions in the nail picture and capture more features.
[0075] The ACmix convolution is a hybrid convolution integrating Self-Attention and convolution, specifically: first, an ordinary convolution is used to map the input features to obtain intermediate features, and then the intermediate features are reused and aggregated in the manner of ordinary convolution and Self-Attention, the ACmix convolution has the advantages of both modules and can enhance the feature extraction capability of ResNet-50. The improved backbone network structure is shown in Table 1:
[0076] Table 1: Backbone network structure
[0077]
[0078] In this embodiment, the improved feature pyramid network includes a FPN structure and a dynamic fine-grained feature fusion spatial structure connected in sequence;
[0079] The FPN structure is used for feature fusion of the features extracted by the backbone network; the FPN structure fuses the spatial information of the low-level features and the semantic information of the high-level features, increases the receptive field of the low-level features, and increases the resolution of feature mapping for small targets, so as to obtain more useful information about small targets and improve the detection accuracy of small targets in nail defect detection;
[0080] Reference Figure 3The dynamic fine-grained feature fusion spatial structure is a spatial structure with a depth of 8 and a neighboring scale of 3, and the spatial structure is composed of a unit named a fine-grained channel selector, which dynamically selects a subsequent path for each pixel according to the characteristics of the feature map, and can fully utilize the different characteristics of different scale features. The introduction of the dynamic fine-grained feature fusion spatial structure dynamically selects the pixel-level combination of the FPN structure of different scales for each target to be detected, enhances the multi-scale representation capability of the FPN structure, and enables the model to extract the fine-grained features of the pin, thereby improving the detection accuracy of similar targets.
[0081] In the embodiment, the step S4 specifically comprises the following steps:
[0082] Step S41, inputting the training set into the improved RetinaNet model;
[0083] Step S42, extracting features in the pin image by the backbone network;
[0084] Step S43, inputting the extracted features into the FPN structure for feature fusion, and converting the fused features into a plurality of fine-grained features with different scales by using the dynamic fine-grained feature fusion spatial structure;
[0085] Step S44, decoupling the plurality of fine-grained features with different scales by the classification head and the regression head, and respectively predicting the category and the position of the target;
[0086] Step S45, performing positive and negative sample assignment on the category and the position of the predicted target by using the adaptive training sample selection method;
[0087] Since the pin dataset has the problem of unbalanced distribution of positive and negative samples, the traditional positive and negative sample assignment method based on the IoU threshold is replaced by an adaptive training sample selection method (ATSS) in the present application, which adaptively selects positive and negative samples according to the statistical characteristics of the target during training, thereby alleviating the problem of unbalanced positive and negative samples of the model.
[0088] Step S46, establishing a Focal loss loss function, repeating steps S41 to S46, minimizing the loss function, and stopping until the loss function converges.
[0089] In the present embodiment, in the process of constructing the improved RetinaNet model, an adaptive training sample selection method (ATSS) is adopted in the process of positive and negative sample assignment. The method adaptively assigns positive and negative samples to each prediction box according to the distribution of positive and negative samples in the input image. The process of the method, i.e., the step S45, is specifically as follows:
[0090] In step S451, for the category and position of the target, there are several real boxes for each pin image. Candidate positive samples are found according to a specified rule. The target is the object in the real box. The real box refers to the label. The specified rule is that, on each pyramid layer, K anchor boxes closest to the center of the real box are selected according to the L2 distance. The L2 distance refers to the actual distance, i.e., the distance between the center of the anchor box and the center of the real box. There are 5 feature layers in the pyramid layer. Each real value box will have Kx5 candidate positive samples. Here, K is 9.
[0091] In step S452, the intersection over union (IoU) between the candidate positive sample point and the real box is calculated. The mean and standard deviation are calculated and marked as m g and v g , respectively. The intersection over union (IoU) is b1 and b2 represent the candidate positive sample and the real box, respectively.
[0092] For each real box on the pin image, candidate positive samples are first found according to a specified rule. The specified rule is that, on each pyramid layer, K anchor boxes closest to the real box are selected according to the L2 distance. There are 5 feature layers in total. Each real value box will have Kx5 candidate positive samples. Here, K is 9.
[0093] In step S453, the intersection over union threshold of the real box is calculated as t g . t g =m g +v g .
[0094] In step S454, candidate samples with an intersection over union greater than or equal to the threshold t g are selected as the final positive samples, and the others are selected as negative samples.
[0095] In the present embodiment, the Focal loss loss function in the step S46 is specifically as follows:
[0096]
[0097] Wherein, p represents the predicted target probability, a is a factor for controlling the weight of positive and negative samples, and g is a weight factor for controlling the difficulty of samples.
[0098] In the embodiment, the step S4 further comprises the following steps:
[0099] The step S47 comprises fine tuning the improved RetinaNet model to obtain a trained RetinaNet model.
[0100] In the embodiment, the improved RetinaNet model is initialized by a pre-trained model on a CoCo dataset before model training, the CoCo dataset belongs to a public dataset, and then the model is fine tuned. Compared with training from scratch, fine tuning can save a large amount of computing resources and computing time, and improve the computing efficiency and accuracy.
[0101] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Moreover, the technical solutions of each embodiment of the present application can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the protection scope required by the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A method for detecting pin defects based on a dynamic fine-grained neural network, characterized in that: It includes the following steps: Step S1: Use an image acquisition device to acquire multiple pin images, and use the multiple pin images to construct a pin image dataset; Step S2: Label the pin images in the pin image dataset, and perform data augmentation on the labeled pin image dataset to obtain the augmented pin image dataset; divide the augmented pin image dataset into a training set and a validation set. Step S3: Build a RetinaNet-ResNet50 model. Based on the RetinaNet-ResNet50 model, construct an improved RetinaNet model. Step S4: Train the improved RetinaNet model using the training set to obtain the trained RetinaNet model; Step S5: Input the validation set into the trained RetinaNet model to obtain the pin defect identification results; The improved RetinaNet model in step S3 includes a backbone network, an improved feature pyramid network, and a detection head connected in sequence. The backbone network consists of an improved ResNet-50, which is used to extract features from pin images in the pin image dataset. An improved feature pyramid network is used to fuse features extracted by the backbone network and output features at multiple scales. The detection head includes a classification head and a regression head, both of which are connected to an improved feature pyramid network. The classification and regression heads are used to decouple fine-grained features at multiple scales and predict the target's category and location, respectively. The improved ResNet50 includes five convolutional modules connected in sequence. The last convolutional module contains two Bottlenecks and one improved Bottleneck, which are connected in sequence. The improved feature pyramid network includes a sequentially connected FPN structure and a dynamic fine-grained feature fusion spatial structure; The FPN structure is used to perform feature fusion on features extracted from the backbone network; The dynamic fine-grained feature fusion spatial structure is mainly composed of fine-grained channel selector units, which are used to convert the fused features into multiple fine-grained features of different scales. The backbone of the traditional RetinaNet model consists of ResNet-50, which contains five convolutional modules. The fifth convolutional module contains three Bottlenecks. The backbone of the improved RetinaNet model converts the ordinary 3x3 convolution in the last Bottleneck of the fifth convolutional module into an ACmix convolution.
2. The pin defect detection method according to claim 1, characterized in that, Step S2 specifically includes the following steps: Step S21: Use data annotation software to annotate the pin image dataset to obtain the annotated pin image dataset; Step S22: Perform image enhancement on the labeled pin image dataset to obtain the image-enhanced pin image dataset; Step S23: Divide the enhanced pin image dataset into a training set and a validation set according to the specified ratio.
3. The pin defect detection method according to claim 2, characterized in that, The image enhancement in step S22 includes scaling, horizontal flipping, inversion, and cropping.
4. The pin defect detection method according to claim 1, characterized in that, Step S4 specifically includes the following steps: Step S41: Input the training set into the improved RetinaNet model; Step S42: The backbone network extracts features from the pin image; Step S43: Input the extracted features into the FPN structure for feature fusion, and use the dynamic fine-grained feature fusion spatial structure to transform the fused features into multiple fine-grained features of different scales. Step S44: Decouple fine-grained features of multiple scales using the classification head and regression head, and predict the category and location of the target respectively; Step S45: Use an adaptive training sample selection method to assign positive and negative samples to the predicted category and location; Step S46: Establish the Focalloss loss function. Repeat steps S41 to S46 to minimize the loss function until it converges.
5. The pin defect detection method according to claim 4, characterized in that, Step S45 specifically includes the following steps: Step S451: For the category and location of the target, each pin image has several ground truth bounding boxes. Find candidate positive samples according to the specified rules. Step S452: Calculate the intersection-union ratio (IUR) between candidate positive sample points and the ground truth bounding boxes; and calculate its mean and standard deviation, denoted as m. g and v g The intersection-union ratio refers to b1 and b2 represent candidate positive samples and ground truth boxes, respectively; Step S453: Calculate the intersection-union ratio (CUI) threshold t for the ground truth bounding boxes. g , t g =m g +v g ; Step S454: Select intersection-union ratios greater than or equal to the threshold t. g The candidate samples are used as the final positive samples.
6. The pin defect detection method according to claim 4, characterized in that, The Focalloss loss function in step S46 is specifically as follows: Where: p represents the predicted target probability, α is a factor used to control the weights of positive and negative samples, and γ is a weighting factor used to control the weights of easy and difficult samples.