Power transmission line hardware defect detection method and system based on feature fusion

Abstract

The invention discloses a power transmission line fitting defect detection method based on feature fusion. The method comprises the following steps: acquiring defect image data information of a power transmission line fitting and preprocessing the defect image data information to construct a training data set; based on the convolutional layer, the pooling layer and the connection layer, constructing a power transmission line hardware defect detection primary model, and training to obtain a power transmission line hardware defect detection model; and carrying out feature fusion-based power transmission line hardware defect detection on the target power transmission line hardware by adopting the obtained power transmission line hardware defect detection model. The invention also discloses a system for realizing the feature fusion-based power transmission line hardware defect detection method. According to the invention, information fusion is carried out on the acquired power transmission line hardware fitting image, and the constructed defect detection model is adopted to detect the hardware fitting image data; therefore, the method not only can realize the defect detection of the power transmission line hardware fitting based on feature fusion, but also is higher in reliability, better in accuracy and higher in efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of electrical automation, and in particular relates to a method and system for detecting defects in power transmission line fittings based on feature fusion. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, electric energy has become an indispensable secondary energy source in people's production and life, bringing endless convenience to people's production and life. Therefore, ensuring the stable and reliable supply of electric energy has become one of the most important tasks of the power system. Transmission lines are an important part of the power system, and transmission line hardware plays an important role in the transmission lines. Therefore, the detection of transmission line hardware is of great significance to the power system.

[0003] At present, the commonly used hardware inspection scheme for power systems is still the manual inspection scheme, that is, the power system maintenance personnel go to the site to inspect the hardware of the transmission line. However, this manual inspection scheme is not only time-consuming and labor-intensive, but also has low safety, low efficiency, and low accuracy and reliability. At present, although there are also schemes for using drones for hardware inspection, the drone inspection process also uses drones to shoot the hardware images, and then the hardware inspection is performed manually. This inspection method still has the problems of poor reliability and poor accuracy. Summary of the invention

[0004] One of the purposes of the present invention is to provide a method for detecting defects in power transmission line fittings based on feature fusion with high reliability, good accuracy and high efficiency.

[0005] A second objective of the present invention is to provide a system for implementing the power transmission line fittings defect detection method based on feature fusion.

[0006] The method for detecting defects in power transmission line fittings based on feature fusion provided by the present invention comprises the following steps:

[0007] S1. Obtaining defect image data information of transmission line fittings;

[0008] S2. Preprocess the data information obtained in step S1 to construct a training data set;

[0009] S3. Based on the convolution layer, pooling layer and connection layer, a primary model for transmission line hardware defect detection is constructed;

[0010] S4. Using the training data set obtained in step S2, the primary model for power transmission line fitting defect detection constructed in step S3 is trained to obtain a power transmission line fitting defect detection model;

[0011] S5. Using the power transmission line fitting defect detection model obtained in step S4, perform feature fusion-based power transmission line fitting defect detection on the target power transmission line fittings.

[0012] The primary model for detecting defects in power transmission line fittings is constructed based on the convolution layer, the pooling layer and the connection layer as described in step S3, and specifically includes the following steps:

[0013] The input image data is processed by the first convolution layer and the second convolution layer in sequence to extract defect features;

[0014] The defect feature is then processed in sequence by the first defect feature construction module, the third convolution layer, the second defect feature construction module, the fourth convolution layer, the third defect feature construction module, the fifth convolution layer and the sixth convolution layer to complete the construction of the defect feature and obtain the constructed defect feature;

[0015] The defect features are constructed and processed by the global feature extraction module to extract the global features;

[0016] After upsampling, the global feature is spliced ​​with the output of the third defect feature construction module on the channel to obtain the first spliced ​​feature, which is used as the input of the first defect feature enhancement module; after the output of the third defect feature construction module is processed by the first maximum pooling layer, it is fused with the output of the first defect feature enhancement module through feature weighting and input to the first detection head for detection;

[0017] After passing through the seventh convolution layer and upsampling, the first concatenated feature is concatenated with the output of the third convolution layer on the channel to obtain the second concatenated feature, which is used as the input of the second defect feature enhancement module;

[0018] After upsampling, the second concatenated feature is concatenated with the output of the second convolutional layer on the channel to obtain the third concatenated feature, which is used as the input of the third defect feature enhancement module;

[0019] After the third splicing feature is processed by the second maximum pooling layer, it is fused with the output of the second defect feature enhancement module through feature weighting and input to the second detection head for detection;

[0020] The output of the third defect feature enhancement module is directly input to the third detection head for detection;

[0021] The first detection head, the second detection head and the third detection head are used to perform classification and frame regression processing to obtain the final hardware defect detection results.

[0022] The first convolution layer is a 1*1 convolution with a step size of 1; the second convolution layer is a 3*3 convolution with a step size of 2; the third convolution layer is a 3*3 convolution with a step size of 2; the fourth convolution layer is a 3*3 convolution with a step size of 2; the fifth convolution layer is a 1*1 convolution with a step size of 1; the sixth convolution layer is a 3*3 convolution with a step size of 2; the seventh convolution layer is a 3*3 convolution with a step size of 1; the first maximum pooling layer is a maximum pooling layer with a step size of 2 and a pooling kernel size of 2; the second maximum pooling layer is a maximum pooling layer with a step size of 2 and a pooling kernel size of 2; upsampling adopts bilinear interpolation upsampling;

[0023] The feature weighted fusion specifically includes the following steps:

[0024] The following formula is used for feature weighted fusion:

[0025]

[0026] Where F out is the output feature after weighted fusion of features; w i is the weight parameter; F i is the i-th feature of the input; ε is a set constant used to prevent the denominator from being 0.

[0027] The processing of the first defect feature construction module, the second defect feature construction module and the third defect feature construction module specifically includes the following steps:

[0028] The structures of the first defect feature building module, the second defect feature building module and the third defect feature building module are the same;

[0029] The defect feature construction module includes the eighth convolution layer, the ninth convolution layer, the Sigmoid activation function layer, the tenth convolution layer, the first batch normalization layer, the eleventh convolution layer, the twelfth convolution layer, the thirteenth convolution layer, the second batch normalization layer and the SiLU activation function layer;

[0030] After the input features are processed by the eighth convolutional layer, they are divided into three features: the first feature is processed by the ninth convolutional layer and the Sigmoid activation function layer in turn to obtain the first branch feature;

[0031] After the second branch feature and the first branch feature are concatenated in terms of the number of channels, they are processed in sequence through the tenth convolutional layer, the first batch normalization layer, and the eleventh convolutional layer to obtain the second branch feature;

[0032] After the third branch feature is processed by the twelfth convolution layer, it is concatenated with the second branch feature in terms of the number of channels, and then processed by the thirteenth convolution layer, the second batch normalization layer and the SiLU activation function layer in sequence to obtain the output features of the defect feature construction module.

[0033] The eighth convolutional layer is a 3*3 convolutional layer with a step size of 1; the ninth convolutional layer is a 1*1 depth-separable convolutional layer with a step size of 1; the tenth convolutional layer is a 3*3 convolution with a step size of 1; the eleventh convolutional layer is a 1*1 convolution with a step size of 1; the twelfth convolutional layer is a 3*3 convolution with a step size of 1; the thirteenth convolutional layer is a 1*1 convolution with a length of 1;

[0034] The Sigmoid activation function layer uses the following formula as the nonlinear activation function:

[0035]

[0036] Where σ(x) is the processing result of the Sigmoid activation function layer; x is the element in the feature processed by the Sigmoid activation function layer;

[0037] The SiLU activation function layer uses the following formula as the nonlinear activation function:

[0038]

[0039] Where SiLU(x) is the processing result of the SiLU activation function layer; x is the element in the feature processed by the SiLU activation function layer.

[0040] The processing process of the global feature extraction module specifically includes the following steps:

[0041] The global feature extraction module includes a pooling layer, a fourteenth convolutional layer, an upsampling layer, a first dilated convolutional layer, a fifteenth convolutional layer, a second dilated convolutional layer, a sixteenth convolutional layer, a normalization layer, a GELU activation function layer, and a seventeenth convolutional layer;

[0042] The input features are divided into four paths:

[0043] The first input feature is processed by the pooling layer, the fourteenth convolutional layer and the upsampling layer in sequence to obtain the first branch feature;

[0044] The second input feature is processed by the first hole convolution layer to obtain the second branch feature;

[0045] The third input feature is processed by the fifteenth convolutional layer and the second atrous convolutional layer to obtain the third branch feature;

[0046] After the fourth input feature is processed by the sixteenth convolutional layer, the fourth branch feature is obtained;

[0047] After the first branch features, the second branch features, the third branch features and the fourth branch features are concatenated on the channel, they are processed in turn through the normalization layer, the GELU activation function layer and the seventeenth convolutional layer to obtain the features output by the global feature extraction module.

[0048] The pooling layer is an adaptive mean pooling layer with an output size of 2; the fourteenth convolution layer is a 1*1 convolution layer with a step size of 1; the upsampling layer adopts bilinear interpolation upsampling; the first hole convolution layer is a 3*3 hole convolution layer with an expansion rate of 6; the fifteenth convolution layer is a 3*3 depth-separable convolution layer with a step size of 1; the second hole convolution layer is a 3*3 hole convolution layer with an expansion rate of 12; the sixteenth convolution layer is a 1*1 convolution layer with a step size of 1; the seventeenth convolution layer is a 1*1 convolution layer with a step size of 1;

[0049] The GELU activation function layer uses the following formula as the nonlinear activation function:

[0050]

[0051] Where GELU(x) is the processing result of the GELU activation function layer; x is the element in the feature processed by the GELU activation function layer.

[0052] The processing of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module specifically includes the following steps:

[0053] The structures of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module are all the same;

[0054] The defect feature enhancement module includes an eighteenth convolution layer, a first feature fusion layer, a maximum pooling layer, a nineteenth convolution layer, a first upsampling layer, a second feature fusion layer, a second upsampling layer, a twentieth convolution layer, a GULU activation function layer, a twenty-first convolution layer, a Sigmoid activation function layer, a summation layer, a normalization layer, and a SiLU activation function layer;

[0055] The input features are divided into two paths:

[0056] The first input feature is processed by the eighteenth convolutional layer and the first feature fusion layer;

[0057] The second input feature passes through the maximum pooling layer, the nineteenth convolutional layer, the first upsampling layer, and the second feature fusion layer;

[0058] The output of the maximum pooling layer is processed by the second upsampling layer, and then concatenated with the output of the eighteenth convolutional layer on the channel. It is then processed in sequence by the twentieth convolutional layer, the GULU activation function layer, the twenty-first convolutional layer, and the Sigmoid activation function layer to obtain the relative attention weight.

[0059] The relative attention weight is multiplied by the set first constant to obtain a first relative attention weight. The first feature fusion layer is used to multiply the obtained first relative attention weight by the output of the eighteenth convolutional layer at the pixel level, and upload the result to the summation layer;

[0060] The relative attention weight is multiplied by the set second constant to obtain a second relative attention weight. The second feature fusion layer is used to multiply the obtained second relative attention weight by the output of the first upsampling layer at the pixel level, and upload the result to the summation layer;

[0061] After the summation layer adds the received features, the result is processed by the normalization layer and the SiLU activation function layer in turn to obtain the output of the defect feature enhancement module.

[0062] The eighteenth convolution layer is a 3*3 convolution layer with a stride of 1; the maximum pooling layer is a maximum pooling layer with a stride of 2; the nineteenth convolution layer is a 3*3 convolution layer with a stride of 1; the first upsampling layer uses bilinear interpolation upsampling; the second upsampling layer uses bilinear interpolation upsampling; the twentieth convolution layer is a 1*1 convolution layer with a stride of 1; the twenty-first convolution layer is a 3*3 convolution layer with a stride of 1;

[0063] The Sigmoid activation function layer uses the following formula as the nonlinear activation function:

[0064]

[0065] Where σ(x) is the processing result of the Sigmoid activation function layer; x is the element in the feature processed by the Sigmoid activation function layer;

[0066] The SiLU activation function layer uses the following formula as the nonlinear activation function:

[0067]

[0068] Where SiLU(x) is the processing result of the SiLU activation function layer; x is the element in the feature processed by the SiLU activation function layer;

[0069] The GULU activation function layer uses the following formula as the nonlinear activation function:

[0070]

[0071] Where GELU(x) is the processing result of the GELU activation function layer; x is the element in the feature processed by the GELU activation function layer.

[0072] The processing process of the first detection head, the second detection head and the third detection head specifically includes the following steps:

[0073] The first detection head, the second detection head and the third detection head have the same structure;

[0074] The detection head includes a twenty-second convolutional layer, a twenty-third convolutional layer, a twenty-fourth convolutional layer, a first fully connected layer, a twenty-fifth convolutional layer, a twenty-sixth convolutional layer, a twenty-seventh convolutional layer, a twenty-eighth convolutional layer, a twenty-ninth convolutional layer, and a second fully connected layer;

[0075] The input features are divided into two paths:

[0076] The first input feature is processed in turn by the 22nd convolutional layer, the 23rd convolutional layer, the 24th convolutional layer and the first fully connected layer to obtain the center point coordinates, prediction box width, prediction box height and confidence value of the prediction box;

[0077] The second input features are processed in sequence by the 25th convolutional layer, the 26th convolutional layer, the 27th convolutional layer, the 28th convolutional layer, the 29th convolutional layer and the second fully connected layer to obtain the defect category prediction results of the hardware.

[0078] The twenty-second convolutional layer is a 3*3 convolutional layer with a stride of 1; the twenty-third convolutional layer is a 3*3 convolutional layer with a stride of 1; the twenty-fourth convolutional layer is a 1*1 convolutional layer with a stride of 1; the twenty-fifth convolutional layer is a 3*3 depthwise separable convolutional layer with a stride of 1; the twenty-sixth convolutional layer is a 1*1 convolutional layer with a stride of 1; the twenty-seventh convolutional layer is a 3*3 depthwise separable convolutional layer with a stride of 1; the twenty-eighth convolutional layer is a 1*1 convolutional layer with a stride of 1; the twenty-ninth convolutional layer is a 1*1 convolutional layer with a stride of 1.

[0079] The training described in step S4 specifically includes the following steps:

[0080] The first loss function:

[0081]

[0082] Where Loss1 is the first loss function value; b is the prediction box; b gt is the real box; IoU(b,b gt ) is the intersection-over-union ratio of the predicted box and the true box; λ is a parameter that controls the degree of influence of shape difference; α is a parameter that controls the degree of weighting of geometric shape difference; ΔShape is the shape difference metric of the box, and ΔShape = |x pred -x gt |+|y pred -y gt |, x pred is the horizontal coordinate of the center of the prediction box, y pred is the ordinate of the center of the prediction box, x gt is the horizontal coordinate of the center of the real frame, y gtis the ordinate of the center of the real frame;

[0083] The second loss function:

[0084] Loss2=-((yy j+1 -yy)logS j +(yy-yy j )logS j+1 )

[0085] Where Loss2 is the second loss function value; yy j+1 is the label value rounded up; yy is the actual value of the label; yy j is the label value after rounding down; S j is the predicted value of the output; S j+1 is the output immediate prediction value;

[0086] The third loss function:

[0087]

[0088] Where N is the total number of hardware defect categories; y i is the true label of the i-th defect category; p i is the predicted probability of the i-th defect category;

[0089] The total loss function Loss is calculated using the following formula:

[0090] Loss=λ1Loss1+λ2Loss2+λ3Loss3

[0091] Wherein λ1 is the first weight parameter; λ2 is the second weight parameter; λ3 is the third weight parameter;

[0092] During training, the total loss function Loss is used to train the model.

[0093] The present invention also provides a system for realizing the power transmission line fitting defect detection method based on feature fusion, comprising a data acquisition module, a data processing module, a model construction module, a model training module and a fitting detection module; the data acquisition module, the data processing module, the model construction module, the model training module and the fitting detection module are connected in series in sequence; the data acquisition module is used to acquire defect image data information of the power transmission line fittings, and upload the data information to the data processing module; the data processing module is used to pre-process the acquired data information according to the received data information to construct a training data set, and upload the data information to the model construction module; the model The construction module is used to construct a primary model for transmission line fittings defect detection based on the convolution layer, the pooling layer and the connection layer according to the received data information, and upload the data information to the model training module; the model training module is used to train the constructed primary model for transmission line fittings defect detection based on the received data information and the obtained training data set, obtain the transmission line fittings defect detection model, and upload the data information to the fittings detection module; the fittings detection module is used to perform transmission line fittings defect detection on the target transmission line fittings based on feature fusion using the obtained transmission line fittings defect detection model according to the received data information.

[0094] The method and system for detecting defects in transmission line fittings based on feature fusion provided by the present invention fuse information of acquired images of transmission line fittings, and use a constructed defect detection model to perform detection through the image data of the fittings; therefore, the present invention can not only realize defect detection of transmission line fittings based on feature fusion, but also has higher reliability, better accuracy and higher efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 The figure is a schematic diagram of the method flow of the present invention.

[0096] Figure 2 Schematic diagram of the functional modules of the system of the present invention. DETAILED DESCRIPTION

[0097] like Figure 1 The method flow chart of the method of the present invention is shown as follows: The method for detecting defects of power transmission line fittings based on feature fusion disclosed in the present invention comprises the following steps:

[0098] S1. Obtaining defect image data information of transmission line fittings;

[0099] S2. Preprocess the data information obtained in step S1 to construct a training data set;

[0100] In specific implementation, the preprocessing of data information includes sorting, classification, cleaning, labeling and image enhancement to obtain a data set that meets the set requirements;

[0101] S3. Based on the convolution layer, pooling layer and connection layer, a primary model for transmission line hardware defect detection is constructed;

[0102] The specific steps include:

[0103] The input image data is processed by the first convolution layer and the second convolution layer in sequence to extract defect features;

[0104] The defect feature is then processed in sequence by the first defect feature construction module, the third convolution layer, the second defect feature construction module, the fourth convolution layer, the third defect feature construction module, the fifth convolution layer and the sixth convolution layer to complete the construction of the defect feature and obtain the constructed defect feature;

[0105] The defect features are constructed and processed by the global feature extraction module to extract the global features;

[0106] After upsampling, the global feature is spliced ​​with the output of the third defect feature construction module on the channel to obtain the first spliced ​​feature, which is used as the input of the first defect feature enhancement module; after the output of the third defect feature construction module is processed by the first maximum pooling layer, it is fused with the output of the first defect feature enhancement module through feature weighting and input to the first detection head for detection;

[0107] After passing through the seventh convolution layer and upsampling, the first concatenated feature is concatenated with the output of the third convolution layer on the channel to obtain the second concatenated feature, which is used as the input of the second defect feature enhancement module;

[0108] After upsampling, the second concatenated feature is concatenated with the output of the second convolutional layer on the channel to obtain the third concatenated feature, which is used as the input of the third defect feature enhancement module;

[0109] After the third splicing feature is processed by the second maximum pooling layer, it is fused with the output of the second defect feature enhancement module through feature weighting and input to the second detection head for detection;

[0110] The output of the third defect feature enhancement module is directly input to the third detection head for detection;

[0111] The first detection head, the second detection head and the third detection head are used to perform classification and frame regression processing to obtain the final hardware defect detection results;

[0112] In the overall description of the primary model for transmission line hardware defect detection, the first convolution layer is a 1*1 convolution with a step size of 1; the second convolution layer is a 3*3 convolution with a step size of 2; the third convolution layer is a 3*3 convolution with a step size of 2; the fourth convolution layer is a 3*3 convolution with a step size of 2; the fifth convolution layer is a 1*1 convolution with a step size of 1; the sixth convolution layer is a 3*3 convolution with a step size of 2; the seventh convolution layer is a 3*3 convolution with a step size of 1; the first maximum pooling layer is a maximum pooling layer with a step size of 2 and a pooling kernel size of 2; the second maximum pooling layer is a maximum pooling layer with a step size of 2 and a pooling kernel size of 2; upsampling uses bilinear interpolation upsampling;

[0113] The feature weighted fusion specifically includes the following steps:

[0114] The following formula is used for feature weighted fusion:

[0115]

[0116] Where F out is the output feature after weighted fusion of features; w i is the weight parameter; F i is the i-th feature of the input; ε is a set constant used to prevent the denominator from being 0;

[0117] When implementing:

[0118] The processing of the first defect feature construction module, the second defect feature construction module and the third defect feature construction module specifically includes the following steps:

[0119] The structures of the first defect feature building module, the second defect feature building module and the third defect feature building module are the same;

[0120] The defect feature construction module includes the eighth convolution layer, the ninth convolution layer, the Sigmoid activation function layer, the tenth convolution layer, the first batch normalization layer, the eleventh convolution layer, the twelfth convolution layer, the thirteenth convolution layer, the second batch normalization layer and the SiLU activation function layer;

[0121] After the input features are processed by the eighth convolutional layer, they are divided into three features: the first feature is processed by the ninth convolutional layer and the Sigmoid activation function layer in turn to obtain the first branch feature;

[0122] After the second branch feature and the first branch feature are concatenated in terms of the number of channels, they are processed in sequence through the tenth convolutional layer, the first batch normalization layer, and the eleventh convolutional layer to obtain the second branch feature;

[0123] After the third-path feature is processed by the twelfth convolution layer, it is concatenated with the second-path feature in terms of the number of channels, and then processed by the thirteenth convolution layer, the second batch normalization layer, and the SiLU activation function layer in sequence to obtain the output feature of the defect feature construction module; among them, the branch through which the third-path feature passes retains the original input feature, which can alleviate the problems of gradient vanishing and gradient explosion in the deep network training process, and also helps to enhance the robustness of the network;

[0124] Moreover, the splicing operation involved in the defect feature construction module also enables the defects in different receptive fields to be fully utilized, effectively constructing the defect features of the hardware;

[0125] In the above description of the processing of the first defect feature construction module, the second defect feature construction module and the third defect feature construction module: the eighth convolution layer is a 3*3 convolution layer with a step size of 1; the ninth convolution layer is a 1*1 depth-separable convolution layer with a step size of 1; the tenth convolution layer is a 3*3 convolution with a step size of 1; the eleventh convolution layer is a 1*1 convolution with a step size of 1; the twelfth convolution layer is a 3*3 convolution with a step size of 1; the thirteenth convolution layer is a 1*1 convolution with a length of 1;

[0126] The Sigmoid activation function layer uses the following formula as the nonlinear activation function:

[0127]

[0128] Where σ(x) is the processing result of the Sigmoid activation function layer; x is the element in the feature processed by the Sigmoid activation function layer;

[0129] The SiLU activation function layer uses the following formula as the nonlinear activation function:

[0130]

[0131] Where SiLU(x) is the processing result of the SiLU activation function layer; x is the element in the feature processed by the SiLU activation function layer;

[0132] When implementing:

[0133] The processing process of the global feature extraction module specifically includes the following steps:

[0134] The global feature extraction module includes a pooling layer, a fourteenth convolutional layer, an upsampling layer, a first dilated convolutional layer, a fifteenth convolutional layer, a second dilated convolutional layer, a sixteenth convolutional layer, a normalization layer, a GELU activation function layer, and a seventeenth convolutional layer;

[0135] The input features are divided into four paths:

[0136] The first input feature is processed by the pooling layer, the fourteenth convolutional layer and the upsampling layer in sequence to obtain the first branch feature;

[0137] The second input feature is processed by the first hole convolution layer to obtain the second branch feature;

[0138] The third input feature is processed by the fifteenth convolutional layer and the second atrous convolutional layer to obtain the third branch feature;

[0139] After the fourth input feature is processed by the sixteenth convolutional layer, the fourth branch feature is obtained;

[0140] After the first branch feature, the second branch feature, the third branch feature and the fourth branch feature are concatenated on the channel, they are processed in sequence through the normalization layer, the GELU activation function layer and the seventeenth convolution layer to obtain the features output by the global feature extraction module;

[0141] The global feature extraction module generates feature maps with different voiding rates, so that information of different receptive fields can be processed in parallel on the same feature map, thereby capturing contextual information of different scales. At the same time, the size is changed to 1*1 through adaptive mean pooling, and the global features are spliced ​​on the channel through upsampling to extract features of different sizes, effectively extracting the global features of hardware defects;

[0142] In the above description of the processing process of the global feature extraction module: the pooling layer is an adaptive mean pooling layer with an output size of 2; the fourteenth convolution layer is a 1*1 convolution layer with a step size of 1; the upsampling layer adopts bilinear interpolation upsampling; the first hole convolution layer is a 3*3 hole convolution layer with an expansion rate of 6; the fifteenth convolution layer is a 3*3 depth-separable convolution layer with a step size of 1; the second hole convolution layer is a 3*3 hole convolution layer with an expansion rate of 12; the sixteenth convolution layer is a 1*1 convolution layer with a step size of 1; the seventeenth convolution layer is a 1*1 convolution layer with a step size of 1;

[0143] The GELU activation function layer uses the following formula as the nonlinear activation function:

[0144]

[0145] Where GELU(x) is the processing result of the GELU activation function layer; x is the element in the feature processed by the GELU activation function layer;

[0146] When implementing:

[0147] The processing of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module specifically includes the following steps:

[0148] The structures of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module are all the same;

[0149] The defect feature enhancement module includes an eighteenth convolution layer, a first feature fusion layer, a maximum pooling layer, a nineteenth convolution layer, a first upsampling layer, a second feature fusion layer, a second upsampling layer, a twentieth convolution layer, a GULU activation function layer, a twenty-first convolution layer, a Sigmoid activation function layer, a summation layer, a normalization layer, and a SiLU activation function layer;

[0150] The input features are divided into two paths:

[0151] The first input feature is processed by the eighteenth convolutional layer and the first feature fusion layer;

[0152] The second input feature passes through the maximum pooling layer, the nineteenth convolutional layer, the first upsampling layer, and the second feature fusion layer;

[0153] The output of the maximum pooling layer is processed by the second upsampling layer, and then concatenated with the output of the eighteenth convolutional layer on the channel. It is then processed in sequence by the twentieth convolutional layer, the GULU activation function layer, the twenty-first convolutional layer, and the Sigmoid activation function layer to obtain the relative attention weight.

[0154] The relative attention weight is multiplied by the set first constant to obtain a first relative attention weight. The first feature fusion layer is used to multiply the obtained first relative attention weight by the output of the eighteenth convolutional layer at the pixel level, and upload the result to the summation layer;

[0155] The relative attention weight is multiplied by the set second constant to obtain a second relative attention weight. The second feature fusion layer is used to multiply the obtained second relative attention weight by the output of the first upsampling layer at the pixel level, and upload the result to the summation layer;

[0156] After the summation layer adds the received features, the result is processed by the normalization layer and the SiLU activation function layer in turn to obtain the output of the defect feature enhancement module;

[0157] The defect feature enhancement module uses the maximum pooling Maxpool to fuse features with other branches, enhancing the defect features of hardware, so that the network focuses on the prominent features of the defects.

[0158] In the above description of the processing of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module: the eighteenth convolution layer is a 3*3 convolution layer with a step size of 1; the maximum pooling layer is a maximum pooling layer with a step size of 2; the nineteenth convolution layer is a 3*3 convolution layer with a step size of 1; the first upsampling layer adopts bilinear interpolation upsampling; the second upsampling layer adopts bilinear interpolation upsampling; the twentieth convolution layer is a 1*1 convolution layer with a step size of 1; the twenty-first convolution layer is a 3*3 convolution layer with a step size of 1;

[0159] The Sigmoid activation function layer uses the following formula as the nonlinear activation function:

[0160]

[0161] Where σ(x) is the processing result of the Sigmoid activation function layer; x is the element in the feature processed by the Sigmoid activation function layer;

[0162] The SiLU activation function layer uses the following formula as the nonlinear activation function:

[0163]

[0164] Where SiLU(x) is the processing result of the SiLU activation function layer; x is the element in the feature processed by the SiLU activation function layer;

[0165] The GULU activation function layer uses the following formula as the nonlinear activation function:

[0166]

[0167] Where GELU(x) is the processing result of the GELU activation function layer; x is the element in the feature processed by the GELU activation function layer;

[0168] When implementing:

[0169] The processing process of the first detection head, the second detection head and the third detection head specifically includes the following steps:

[0170] The first detection head, the second detection head and the third detection head have the same structure;

[0171] The detection head includes a twenty-second convolutional layer, a twenty-third convolutional layer, a twenty-fourth convolutional layer, a first fully connected layer, a twenty-fifth convolutional layer, a twenty-sixth convolutional layer, a twenty-seventh convolutional layer, a twenty-eighth convolutional layer, a twenty-ninth convolutional layer, and a second fully connected layer;

[0172] The input features are divided into two paths:

[0173] The first input feature is processed in turn by the 22nd convolutional layer, the 23rd convolutional layer, the 24th convolutional layer and the first fully connected layer to obtain the center point coordinates, prediction box width, prediction box height and confidence value of the prediction box;

[0174] The second input feature is processed by the 25th convolutional layer, the 26th convolutional layer, the 27th convolutional layer, the 28th convolutional layer, the 29th convolutional layer and the second fully connected layer in turn to obtain the defect category prediction result of the hardware;

[0175] In the description of the processing of the first detection head, the second detection head and the third detection head above: the twenty-second convolution layer is a 3*3 convolution layer with a step size of 1; the twenty-third convolution layer is a 3*3 convolution layer with a step size of 1; the twenty-fourth convolution layer is a 1*1 convolution layer with a step size of 1; the twenty-fifth convolution layer is a 3*3 depth-separable convolution layer with a step size of 1; the twenty-sixth convolution layer is a 1*1 convolution layer with a step size of 1; the twenty-seventh convolution layer is a 3*3 depth-separable convolution layer with a step size of 1; the twenty-eighth convolution layer is a 1*1 convolution layer with a step size of 1; the twenty-ninth convolution layer is a 1*1 convolution layer with a step size of 1;

[0176] Among them, the branch through which the second input feature passes uses two depth-separable convolutions with 3*3 convolution kernels, which reduces the complexity of the model and improves the performance of hardware defect detection;

[0177] S4. Using the training data set obtained in step S2, the primary model for power transmission line fitting defect detection constructed in step S3 is trained to obtain a power transmission line fitting defect detection model;

[0178] The training specifically includes the following steps:

[0179] During training, the following function is used as the loss function:

[0180] The first loss function:

[0181]

[0182] Where Loss1 is the first loss function value; b is the prediction box; b gt is the real box; IoU(b,b gt ) is the intersection-over-union ratio of the predicted box and the true box; λ is a parameter that controls the degree of influence of shape difference; α is a parameter that controls the degree of weighting of geometric shape difference; ΔShape is the shape difference metric of the box, and ΔShape = |x pred -x gt |+|y pred -y gt |, x pred is the horizontal coordinate of the center of the prediction box, y predis the ordinate of the center of the prediction box, x gt is the horizontal coordinate of the center of the real frame, y gt is the ordinate of the center of the real frame;

[0183] The second loss function:

[0184] Loss2=-((yy j+1 -yy)logS j +(yy-yy j )logS j+1 )

[0185] Where Loss2 is the second loss function value; yy j+1 is the label value rounded up; yy is the actual value of the label; yy j is the label value after rounding down; S j is the predicted value of the output; S j+1 is the output immediate prediction value;

[0186] The third loss function:

[0187]

[0188] Where N is the total number of hardware defect categories; y i is the true label of the i-th defect category; p i is the predicted probability of the i-th defect category;

[0189] The total loss function Loss is calculated using the following formula:

[0190] Loss=λ1Loss1+λ2Loss2+λ3Loss3

[0191] Wherein λ1 is the first weight parameter; λ2 is the second weight parameter; λ3 is the third weight parameter;

[0192] During training, the total loss function Loss is used to train the model;

[0193] S5. Using the power transmission line fitting defect detection model obtained in step S4, perform feature fusion-based power transmission line fitting defect detection on the target power transmission line fittings.

[0194] like Figure 2The functional module diagram of the system of the present invention is shown as follows: the system disclosed in the present invention for realizing the power transmission line fitting defect detection method based on feature fusion comprises a data acquisition module, a data processing module, a model construction module, a model training module and a fitting detection module; the data acquisition module, the data processing module, the model construction module, the model training module and the fitting detection module are connected in series in sequence; the data acquisition module is used to acquire defect image data information of the power transmission line fittings, and upload the data information to the data processing module; the data processing module is used to pre-process the acquired data information according to the received data information to construct a training data set, and upload the data information to the model The model construction module is used to construct a primary model for transmission line fittings defect detection based on the convolution layer, the pooling layer and the connection layer according to the received data information, and upload the data information to the model training module; the model training module is used to train the constructed primary model for transmission line fittings defect detection based on the received data information and the obtained training data set, so as to obtain the transmission line fittings defect detection model, and upload the data information to the fittings detection module; the fittings detection module is used to perform transmission line fittings defect detection on the target transmission line fittings based on feature fusion using the obtained transmission line fittings defect detection model according to the received data information.

Claims

1. A method for detecting defects in power transmission line fittings based on feature fusion, comprising the following steps: S1. Obtaining defect image data information of transmission line fittings; S2. Preprocess the data information obtained in step S1 to construct a training data set; S3. Based on the convolution layer, pooling layer and connection layer, a primary model for transmission line hardware defect detection is constructed; S4. Using the training data set obtained in step S2, the primary model for power transmission line fitting defect detection constructed in step S3 is trained to obtain a power transmission line fitting defect detection model; S5. Using the power transmission line fitting defect detection model obtained in step S4, perform feature fusion-based power transmission line fitting defect detection on the target power transmission line fittings.

2. The method for detecting defects in power transmission line fittings based on feature fusion according to claim 1 is characterized in that The primary model for detecting defects in power transmission line fittings is constructed based on the convolution layer, the pooling layer and the connection layer as described in step S3, and specifically includes the following steps: The input image data is processed by the first convolution layer and the second convolution layer in sequence to extract defect features; The defect feature is then processed in sequence by the first defect feature construction module, the third convolution layer, the second defect feature construction module, the fourth convolution layer, the third defect feature construction module, the fifth convolution layer and the sixth convolution layer to complete the construction of the defect feature and obtain the constructed defect feature; The defect features are constructed and processed by the global feature extraction module to extract the global features; After upsampling, the global feature is concatenated with the output of the third defect feature construction module on the channel to obtain the first concatenated feature, which is used as the input of the first defect feature enhancement module; The output of the third defect feature construction module is processed by the first maximum pooling layer, and then fused with the output of the first defect feature enhancement module through feature weighting, and then input to the first detection head for detection; After passing through the seventh convolution layer and upsampling, the first concatenated feature is concatenated with the output of the third convolution layer on the channel to obtain the second concatenated feature, which is used as the input of the second defect feature enhancement module; After upsampling, the second concatenated feature is concatenated with the output of the second convolutional layer on the channel to obtain the third concatenated feature, which is used as the input of the third defect feature enhancement module; After the third splicing feature is processed by the second maximum pooling layer, it is fused with the output of the second defect feature enhancement module through feature weighting and input to the second detection head for detection; The output of the third defect feature enhancement module is directly input to the third detection head for detection; The first detection head, the second detection head and the third detection head are used to perform classification and frame regression processing to obtain the final hardware defect detection results.

3. The method for detecting defects in power transmission line fittings based on feature fusion according to claim 2 is characterized in that The processing of the first defect feature construction module, the second defect feature construction module and the third defect feature construction module specifically includes the following steps: The structures of the first defect feature building module, the second defect feature building module and the third defect feature building module are the same; The defect feature construction module includes the eighth convolution layer, the ninth convolution layer, the Sigmoid activation function layer, the tenth convolution layer, the first batch normalization layer, the eleventh convolution layer, the twelfth convolution layer, the thirteenth convolution layer, the second batch normalization layer and the SiLU activation function layer; After the input features are processed by the eighth convolutional layer, they are divided into three features: the first feature is processed by the ninth convolutional layer and the Sigmoid activation function layer in turn to obtain the first branch feature; After the second branch feature and the first branch feature are concatenated in terms of the number of channels, they are processed in sequence through the tenth convolutional layer, the first batch normalization layer, and the eleventh convolutional layer to obtain the second branch feature; After the third branch feature is processed by the twelfth convolution layer, it is concatenated with the second branch feature in terms of the number of channels, and then processed by the thirteenth convolution layer, the second batch normalization layer and the SiLU activation function layer in sequence to obtain the output features of the defect feature construction module.

4. The method for detecting defects in power transmission line fittings based on feature fusion according to claim 3 is characterized in that The processing process of the global feature extraction module specifically includes the following steps: The global feature extraction module includes a pooling layer, a fourteenth convolutional layer, an upsampling layer, a first dilated convolutional layer, a fifteenth convolutional layer, a second dilated convolutional layer, a sixteenth convolutional layer, a normalization layer, a GELU activation function layer, and a seventeenth convolutional layer; The input features are divided into four paths: The first input feature is processed by the pooling layer, the fourteenth convolutional layer and the upsampling layer in sequence to obtain the first branch feature; The second input feature is processed by the first hole convolution layer to obtain the second branch feature; The third input feature is processed by the fifteenth convolutional layer and the second atrous convolutional layer to obtain the third branch feature; After the fourth input feature is processed by the sixteenth convolutional layer, the fourth branch feature is obtained; After the first branch features, the second branch features, the third branch features and the fourth branch features are concatenated on the channel, they are processed in turn through the normalization layer, the GELU activation function layer and the seventeenth convolutional layer to obtain the features output by the global feature extraction module.

5. The method for detecting defects in power transmission line fittings based on feature fusion according to claim 4 is characterized in that The processing of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module specifically includes the following steps: The structures of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module are all the same; The defect feature enhancement module includes an eighteenth convolution layer, a first feature fusion layer, a maximum pooling layer, a nineteenth convolution layer, a first upsampling layer, a second feature fusion layer, a second upsampling layer, a twentieth convolution layer, a GULU activation function layer, a twenty-first convolution layer, a Sigmoid activation function layer, a summation layer, a normalization layer, and a SiLU activation function layer; The input features are divided into two paths: The first input feature is processed by the eighteenth convolutional layer and the first feature fusion layer; The second input feature passes through the maximum pooling layer, the nineteenth convolutional layer, the first upsampling layer, and the second feature fusion layer; The output of the maximum pooling layer is processed by the second upsampling layer, and then concatenated with the output of the eighteenth convolutional layer on the channel. It is then processed in sequence by the twentieth convolutional layer, the GULU activation function layer, the twenty-first convolutional layer, and the Sigmoid activation function layer to obtain the relative attention weight. The relative attention weight is multiplied by the set first constant to obtain a first relative attention weight. The first feature fusion layer is used to multiply the obtained first relative attention weight by the output of the eighteenth convolutional layer at the pixel level, and upload the result to the summation layer; The relative attention weight is multiplied by the set second constant to obtain a second relative attention weight. The second feature fusion layer is used to multiply the obtained second relative attention weight by the output of the first upsampling layer at the pixel level, and upload the result to the summation layer; After the summation layer adds the received features, the result is processed by the normalization layer and the SiLU activation function layer in turn to obtain the output of the defect feature enhancement module.

6. The method for detecting defects in power transmission line fittings based on feature fusion according to claim 5 is characterized in that The processing process of the first detection head, the second detection head and the third detection head specifically includes the following steps: The first detection head, the second detection head and the third detection head have the same structure; The detection head includes a twenty-second convolutional layer, a twenty-third convolutional layer, a twenty-fourth convolutional layer, a first fully connected layer, a twenty-fifth convolutional layer, a twenty-sixth convolutional layer, a twenty-seventh convolutional layer, a twenty-eighth convolutional layer, a twenty-ninth convolutional layer, and a second fully connected layer; The input features are divided into two paths: The first input feature is processed in turn by the 22nd convolutional layer, the 23rd convolutional layer, the 24th convolutional layer and the first fully connected layer to obtain the center point coordinates, prediction box width, prediction box height and confidence value of the prediction box; The second input features are processed in sequence by the 25th convolutional layer, the 26th convolutional layer, the 27th convolutional layer, the 28th convolutional layer, the 29th convolutional layer and the second fully connected layer to obtain the defect category prediction results of the hardware. The twenty-second convolutional layer is a 3*3 convolutional layer with a stride of 1; the twenty-third convolutional layer is a 3*3 convolutional layer with a stride of 1; the twenty-fourth convolutional layer is a 1*1 convolutional layer with a stride of 1; the twenty-fifth convolutional layer is a 3*3 depthwise separable convolutional layer with a stride of 1; the twenty-sixth convolutional layer is a 1*1 convolutional layer with a stride of 1; the twenty-seventh convolutional layer is a 3*3 depthwise separable convolutional layer with a stride of 1; the twenty-eighth convolutional layer is a 1*1 convolutional layer with a stride of 1; the twenty-ninth convolutional layer is a 1*1 convolutional layer with a stride of 1.

7. The method for detecting defects in power transmission line fittings based on feature fusion according to claim 6 is characterized in that In the primary model of power transmission line hardware defect detection: the first convolution layer is a 1*1 convolution with a step size of 1; the second convolution layer is a 3*3 convolution with a step size of 2; the third convolution layer is a 3*3 convolution with a step size of 2; the fourth convolution layer is a 3*3 convolution with a step size of 2; the fifth convolution layer is a 1*1 convolution with a step size of 1; the sixth convolution layer is a 3*3 convolution with a step size of 2; the seventh convolution layer is a 3*3 convolution with a step size of 1; the first maximum pooling layer is a maximum pooling layer with a step size of 2 and a pooling kernel size of 2; the second maximum pooling layer is a maximum pooling layer with a step size of 2 and a pooling kernel size of 2; upsampling uses bilinear interpolation upsampling; The feature weighted fusion specifically includes the following steps: The following formula is used for feature weighted fusion: Where F out It is the output feature after weighted fusion of features; w i is the weight parameter; F i is the i-th feature of the input; ε is a set constant used to prevent the denominator from being 0; In the defect feature construction module: the eighth convolution layer is a 3*3 convolution layer with a step size of 1; the ninth convolution layer is a 1*1 depth-separable convolution layer with a step size of 1; the tenth convolution layer is a 3*3 convolution layer with a step size of 1; the eleventh convolution layer is a 1*1 convolution layer with a step size of 1; the twelfth convolution layer is a 3*3 convolution layer with a step size of 1; the thirteenth convolution layer is a 1*1 convolution layer with a length of 1; The Sigmoid activation function layer uses the following formula as the nonlinear activation function: Where σ(x) is the processing result of the Sigmoid activation function layer; x is the element in the feature processed by the Sigmoid activation function layer; The SiLU activation function layer uses the following formula as the nonlinear activation function: Where SiLU(x) is the processing result of the SiLU activation function layer; x is the element in the feature processed by the SiLU activation function layer; In the global feature extraction module: the pooling layer is an adaptive mean pooling layer with an output size of 2; the fourteenth convolution layer is a 1*1 convolution layer with a stride of 1; the upsampling layer uses bilinear interpolation upsampling; the first dilated convolution layer is a 3*3 dilated convolution layer with a dilation rate of 6; the fifteenth convolution layer is a 3*3 depth-separable convolution layer with a stride of 1; the second dilated convolution layer is a 3*3 dilated convolution layer with a dilation rate of 12; the sixteenth convolution layer is a 1*1 convolution layer with a stride of 1; the seventeenth convolution layer is a 1*1 convolution layer with a stride of 1; The GELU activation function layer uses the following formula as the nonlinear activation function: Where GELU(x) is the processing result of the GELU activation function layer; x is the element in the feature processed by the GELU activation function layer; In the defect feature enhancement module: the eighteenth convolution layer is a 3*3 convolution layer with a step size of 1; the maximum pooling layer is a maximum pooling layer with a step size of 2; the nineteenth convolution layer is a 3*3 convolution layer with a step size of 1; the first upsampling layer uses bilinear interpolation upsampling; the second upsampling layer uses bilinear interpolation upsampling; the twentieth convolution layer is a 1*1 convolution layer with a step size of 1; the twenty-first convolution layer is a 3*3 convolution layer with a step size of 1; The Sigmoid activation function layer uses the following formula as the nonlinear activation function: Where σ(x) is the processing result of the Sigmoid activation function layer; x is the element in the feature processed by the Sigmoid activation function layer; The SiLU activation function layer uses the following formula as the nonlinear activation function: Where SiLU(x) is the processing result of the SiLU activation function layer; x is the element in the feature processed by the SiLU activation function layer; The GULU activation function layer uses the following formula as the nonlinear activation function: Where GELU(x) is the processing result of the GELU activation function layer; x is the element in the feature processed by the GELU activation function layer.

8. The method for detecting defects in power transmission line fittings based on feature fusion according to claim 7 is characterized in that The training described in step S4 specifically includes the following steps: During training, the following function is used as the loss function: The first loss function: Where Loss1 is the first loss function value; b is the prediction box; b gt is the real frame; IoU(b,b gt ) is the intersection-over-union ratio of the predicted box and the true box; λ is a parameter that controls the degree of influence of shape difference; α is a parameter that controls the degree of weighting of geometric shape difference; ΔShape is the shape difference metric of the box, and ΔShape = |x pred -x gt |+|y pred -y gt |, x pred is the horizontal coordinate of the center of the prediction box, y pred is the ordinate of the center of the prediction box, x gt is the horizontal coordinate of the center of the real frame, y gt is the ordinate of the center of the real frame; The second loss function: Loss2=-((yy j+1 -yy)logS j +(yy-yy j )logS j+1 ) Where Loss2 is the second loss function value; yy j+1 is the label value rounded up; yy is the actual value of the label; yy j is the label value after rounding down; S j is the predicted value of the output; S j+1 is the output immediate prediction value; The third loss function: Where N is the total number of hardware defect categories; y i is the true label of the i-th defect category; p i is the predicted probability of the i-th defect category; The total loss function Loss is calculated using the following formula: Loss=λ1Loss1+λ2Loss2+λ3Loss3 Wherein λ1 is the first weight parameter; λ2 is the second weight parameter; λ3 is the third weight parameter; During training, the total loss function Loss is used to train the model.

9. A system for implementing the method for detecting defects in power transmission line fittings based on feature fusion according to any one of claims 1 to 8, characterized in that It includes a data acquisition module, a data processing module, a model building module, a model training module and a hardware detection module; the data acquisition module, the data processing module, the model building module, the model training module and the hardware detection module are connected in series in sequence; the data acquisition module is used to obtain defect image data information of power transmission line hardware, and upload the data information to the data processing module; The data processing module is used to pre-process the acquired data information according to the received data information to construct a training data set, and upload the data information to the model construction module; The model building module is used to build a primary model for power transmission line hardware defect detection based on the received data information, the convolution layer, the pooling layer and the connection layer, and upload the data information to the model training module; The model training module is used to train the constructed primary model for power transmission line fitting defect detection using the obtained training data set according to the received data information, obtain the power transmission line fitting defect detection model, and upload the data information to the fitting detection module; The hardware detection module is used to perform transmission line hardware defect detection on the target transmission line hardware based on feature fusion according to the received data information and the obtained transmission line hardware defect detection model.

Images