Transmission line hardware defect detection method and system based on feature fusion

Abstract

The application discloses a kind of based on feature fusion transmission line hardware defects detection method, including obtaining the defect image data information of transmission line hardware and pre-processing to construct training dataset;Based on convolution layer, pooling layer and connection layer, construct transmission line hardware defects detection primary model and training obtains transmission line hardware defects detection model;Using the transmission line hardware defects detection model obtained to target transmission line hardware is based on feature fusion transmission line hardware defects detection.The application also discloses a kind of system for realizing the based on feature fusion transmission line hardware defects detection method.The application is fused by the transmission line hardware image obtained information, and using the defect detection model constructed is detected by the image data of hardware;Therefore, the application not only can realize based on feature fusion transmission line hardware defects detection, but also higher reliability, better accuracy, efficiency is also higher.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of electrical automation, and particularly relates to a transmission line hardware defect detection method and system based on feature fusion. BACKGROUND

[0002] With the development of economy and technology and the improvement of people's living standards, electric energy has become an essential secondary energy in people's production and life, bringing endless convenience to people's production and life. Therefore, ensuring the stable and reliable supply of electric energy has become one of the most important tasks of the power system. The transmission line is an important part of the power system, and the transmission line hardware bears an important task in the transmission line. Therefore, the detection of the transmission line hardware is of great significance to the power system.

[0003] At present, the hardware detection scheme commonly used by the power system is still a manual detection scheme, that is, the power system maintenance personnel go to the scene to maintain the hardware of the transmission line. However, this manual maintenance scheme is not only time-consuming and laborious, low in safety, but also low in efficiency, and the accuracy and reliability are not high. At present, although there is a scheme of detecting hardware by using a drone, the drone detection process is also to use a drone to shoot pictures of the hardware, and then detect the hardware by using a manual detection method. This detection method still has the problems of poor reliability and poor accuracy. SUMMARY

[0004] One of the purposes of the present application is to provide a transmission line hardware defect detection method based on feature fusion, which has high reliability, good accuracy and high efficiency.

[0005] The second purpose of the present application is to provide a system for implementing the transmission line hardware defect detection method based on feature fusion.

[0006] The transmission line hardware defect detection method based on feature fusion provided by the present application comprises the following steps:

[0007] S1. Obtain defect image data information of the transmission line hardware;

[0008] S2. Preprocess the data information obtained in step S1 to construct a training data set;

[0009] S3. Construct a transmission line hardware defect detection primary model based on a convolution layer, a pooling layer and a connection layer;

[0010] S4. Train the transmission line hardware defect detection primary model constructed in step S3 by using the training data set obtained in step S2 to obtain a transmission line hardware defect detection model;

[0011] S5. Adopting the transmission line fitting defect detection model obtained in step S4, the target transmission line fitting is subjected to transmission line fitting defect detection based on feature fusion.

[0012] The convolution layer, the pooling layer and the connection layer based on step S3 are used to construct a transmission line fitting defect detection primary model, and specifically include the following steps:

[0013] The input image data is processed by the first convolution layer and the second convolution layer in sequence to extract defect features;

[0014] The defect features are processed by the first defect feature construction module, the third convolution layer, the second defect feature construction module, the fourth convolution layer, the third defect feature construction module, the fifth convolution layer and the sixth convolution layer in sequence to complete the construction of the defect features, and the constructed defect features are obtained;

[0015] The constructed defect features are processed by the global feature extraction module to extract global features;

[0016] After the global features are up-sampled, the output of the third defect feature construction module is spliced in the channel to obtain the first spliced feature, and the first spliced feature is used as the input of the first defect feature enhancement module; the output of the third defect feature construction module is processed by the first maximum pooling layer, and the output of the first defect feature enhancement module is fused by feature weighting, and then input to the first detection head for detection;

[0017] The first spliced feature is processed by the seventh convolution layer and up-sampling, and the output of the third convolution layer is spliced in the channel to obtain the second spliced feature, and the second spliced feature is used as the input of the second defect feature enhancement module;

[0018] The second spliced feature is processed by up-sampling, and the output of the second convolution layer is spliced in the channel to obtain the third spliced feature, and the third spliced feature is used as the input of the third defect feature enhancement module;

[0019] The third spliced feature is processed by the second maximum pooling layer, and the output of the second defect feature enhancement module is fused by feature weighting, and then input to the second detection head for detection;

[0020] The output of the third defect feature enhancement module is directly input to the third detection head for detection;

[0021] The first detection head, the second detection head and the third detection head are used for classification and bounding box regression processing to obtain the final fitting defect detection result.

[0022] The first convolutional layer is a 1*1 convolution with a step of 1; the second convolutional layer is a 3*3 convolution with a step of 2; the third convolutional layer is a 3*3 convolution with a step of 2; the fourth convolutional layer is a 3*3 convolution with a step of 2; the fifth convolutional layer is a 1*1 convolution with a step of 1; the sixth convolutional layer is a 3*3 convolution with a step of 2; the seventh convolutional layer is a 3*3 convolution with a step of 1; the first max-pooling layer is a max-pooling layer with a step of 2 and a pooling kernel size of 2; the second max-pooling layer is a max-pooling layer with a step of 2 and a pooling kernel size of 2; and the upsampling is a bilinear interpolation upsampling;

[0023] The feature weighting fusion specifically includes the following steps:

[0024] The feature weighting fusion is performed by using the following formula:

[0025]

[0026] In the formula, F out is the output feature after the feature weighting fusion; w i is a weight parameter; F i is the i-th input feature; and ε is a constant set to prevent the denominator from being 0.

[0027] The processing process of the first defect feature construction module, the second defect feature construction module and the third defect feature construction module specifically includes the following steps:

[0028] The first defect feature construction module, the second defect feature construction module and the third defect feature construction module have the same structure.

[0029] The defect feature construction module includes an eighth convolutional layer, a ninth convolutional layer, a Sigmoid activation function layer, a tenth convolutional layer, a first batch normalization layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a second batch normalization layer and a SiLU activation function layer.

[0030] After the input feature is processed by the eighth convolutional layer, the feature is divided into three paths: the first path feature is processed by the ninth convolutional layer and the Sigmoid activation function layer in sequence to obtain a first branch feature;

[0031] After the second path feature and the first path branch feature are spliced in the number of channels, the second path feature is processed by the tenth convolutional layer, the first batch normalization layer and the eleventh convolutional layer in sequence to obtain a second branch feature;

[0032] After the third path feature is processed by the twelfth convolutional layer, the third path feature is spliced with the second branch feature in the number of channels, and then the third path feature is processed by the thirteenth convolutional layer, the second batch normalization layer and the SiLU activation function layer in sequence to obtain an output feature of the defect feature construction module.

[0033] The eighth convolutional layer is a 3*3 convolutional layer with a step of 1; the ninth convolutional layer is a 1*1 depth separable convolutional layer with a step of 1; the tenth convolutional layer is a 3*3 convolutional layer with a step of 1; the eleventh convolutional layer is a 1*1 convolutional layer with a step of 1; the twelfth convolutional layer is a 3*3 convolutional layer with a step of 1; and the thirteenth convolutional layer is a 1*1 convolutional layer with a step of 1;

[0034] The sigmoid activation function layer adopts the following formula as a nonlinear activation function:

[0035]

[0036] In the formula, σ(x) is the processing result of the sigmoid activation function layer; x is an element in the feature processed by the sigmoid activation function layer.

[0037] The SiLU activation function layer adopts the following formula as a nonlinear activation function:

[0038]

[0039] In the formula, SiLU(x) is the processing result of the SiLU activation function layer; x is an element in the feature processed by the SiLU activation function layer.

[0040] The processing process of the global feature extraction module specifically includes the following steps:

[0041] The global feature extraction module includes a pooling layer, a fourteenth convolutional layer, an upsampling layer, a first dilated convolutional layer, a fifteenth convolutional layer, a second dilated convolutional layer, a sixteenth convolutional layer, a normalization layer, a GELU activation function layer and a seventeenth convolutional layer.

[0042] The input feature is divided into four paths:

[0043] The first path of the input feature is processed by the pooling layer, the fourteenth convolutional layer and the upsampling layer in sequence to obtain a first branch feature;

[0044] The second path of the input feature is processed by the first dilated convolutional layer to obtain a second branch feature;

[0045] The third path of the input feature is processed by the fifteenth convolutional layer and the second dilated convolutional layer to obtain a third branch feature;

[0046] The fourth path of the input feature is processed by the sixteenth convolutional layer to obtain a fourth branch feature;

[0047] The first branch feature, the second branch feature, the third branch feature and the fourth branch feature are spliced in the channel, and then processed by the normalization layer, the GELU activation function layer and the seventeenth convolutional layer in sequence to obtain the feature output by the global feature extraction module.

[0048] The pooling layer is an adaptive mean pooling layer with an output size of 2; the fourteenth convolutional layer is a 1*1 convolutional layer with a step size of 1; the up-sampling layer adopts bilinear interpolation up-sampling; the first dilated convolutional layer is a 3*3 dilated convolutional layer with a dilation rate of 6; the fifteenth convolutional layer is a 3*3 depth separable convolutional layer with a step size of 1; the second dilated convolutional layer is a 3*3 dilated convolutional layer with a dilation rate of 12; the sixteenth convolutional layer is a 1*1 convolutional layer with a step size of 1; the seventeenth convolutional layer is a 1*1 convolutional layer with a step size of 1;

[0049] The GELU activation function layer adopts the following formula as a nonlinear activation function:

[0050]

[0051] In the formula, GELU(x) is the processing result of the GELU activation function layer; x is an element in the feature processed by the GELU activation function layer.

[0052] The processing procedures of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module specifically include the following steps:

[0053] The structures of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module are all the same;

[0054] The defect feature enhancement module includes an eighteenth convolutional layer, a first feature fusion layer, a max pooling layer, a nineteenth convolutional layer, a first up-sampling layer, a second feature fusion layer, a second up-sampling layer, a twentieth convolutional layer, a GULU activation function layer, a twenty-first convolutional layer, a Sigmoid activation function layer, a summation layer, a normalization layer and a SiLU activation function layer;

[0055] The input feature is divided into two paths:

[0056] The first path of the input feature is processed through the eighteenth convolutional layer and the first feature fusion layer;

[0057] The second path of the input feature is processed through the max pooling layer, the nineteenth convolutional layer, the first up-sampling layer and the second feature fusion layer;

[0058] The output of the max pooling layer is processed through the second up-sampling layer, and then is spliced with the output of the eighteenth convolutional layer in the channel, and then is processed through the twentieth convolutional layer, the GULU activation function layer, the twenty-first convolutional layer and the Sigmoid activation function layer in turn to obtain the relative attention weight;

[0059] The relative attention weight is multiplied by a set first constant to obtain a first relative attention weight, and the first feature fusion layer is configured to perform pixel-level multiplication between the obtained first relative attention weight and the output of the eighteenth convolutional layer, and upload the result to the summation layer;

[0060] The relative attention weight is multiplied by a set second constant to obtain a second relative attention weight, and the second feature fusion layer is configured to perform pixel-level multiplication between the obtained second relative attention weight and the output of the first up-sampling layer, and upload the result to the summation layer;

[0061] The summation layer adds the received features, and then sequentially processes the result through the normalization layer and the SiLU activation function layer to obtain the output of the defect feature enhancement module.

[0062] The eighteenth convolutional layer is a 3*3 convolutional layer with a step of 1; the max-pooling layer is a max-pooling layer with a step of 2; the nineteenth convolutional layer is a 3*3 convolutional layer with a step of 1; the first up-sampling layer adopts bilinear interpolation up-sampling; the second up-sampling layer adopts bilinear interpolation up-sampling; the twentieth convolutional layer is a 1*1 convolutional layer with a step of 1; the twenty-first convolutional layer is a 3*3 convolutional layer with a step of 1;

[0063] The Sigmoid activation function layer adopts the following formula as a nonlinear activation function:

[0064]

[0065] In the formula, σ(x) is the processing result of the Sigmoid activation function layer; x is an element in the feature processed by the Sigmoid activation function layer;

[0066] The SiLU activation function layer adopts the following formula as a nonlinear activation function:

[0067]

[0068] In the formula, SiLU(x) is the processing result of the SiLU activation function layer; x is an element in the feature processed by the SiLU activation function layer;

[0069] The GELU activation function layer adopts the following formula as a nonlinear activation function:

[0070]

[0071] In the formula, GELU(x) is the processing result of the GELU activation function layer; x is an element in the feature processed by the GELU activation function layer.

[0072] The processing procedures of the first detection head, the second detection head and the third detection head specifically include the following steps:

[0073] The first detection head, the second detection head and the third detection head are the same in structure;

[0074] The detection head comprises a twenty-second convolutional layer, a twenty-third convolutional layer, a twenty-fourth convolutional layer, a first full connection layer, a twenty-fifth convolutional layer, a twenty-sixth convolutional layer, a twenty-seventh convolutional layer, a twenty-eighth convolutional layer, a twenty-ninth convolutional layer and a second full connection layer.

[0075] The input features are divided into two paths:

[0076] The first path of input features is sequentially processed by the twenty-second convolutional layer, the twenty-third convolutional layer, the twenty-fourth convolutional layer and the first full connection layer to obtain the center point coordinates of the prediction frame, the prediction frame width, the prediction frame height and the confidence value.

[0077] The second path of input features is sequentially processed by the twenty-fifth convolutional layer, the twenty-sixth convolutional layer, the twenty-seventh convolutional layer, the twenty-eighth convolutional layer, the twenty-ninth convolutional layer and the second full connection layer to obtain the defect category prediction result of the fitting.

[0078] The twenty-second convolutional layer is a 3*3 convolutional layer with a step of 1; the twenty-third convolutional layer is a 3*3 convolutional layer with a step of 1; the twenty-fourth convolutional layer is a 1*1 convolutional layer with a step of 1; the twenty-fifth convolutional layer is a 3*3 depth separable convolutional layer with a step of 1; the twenty-sixth convolutional layer is a 1*1 convolutional layer with a step of 1; the twenty-seventh convolutional layer is a 3*3 depth separable convolutional layer with a step of 1; the twenty-eighth convolutional layer is a 1*1 convolutional layer with a step of 1; and the twenty-ninth convolutional layer is a 1*1 convolutional layer with a step of 1.

[0079] The training in step S4 specifically comprises the following steps:

[0080] The first loss function:

[0081]

[0082] In the formula, Loss1 is the first loss function value; b is the prediction frame; b gt is the real frame; IoU(b,b gt ) is the intersection over union of the prediction frame and the real frame; λ is a parameter for controlling the influence degree of the shape difference; α is a parameter for controlling the weighting degree of the geometric shape difference; and ΔShape is the shape difference measurement value of the frame, and ΔShape = |x pred -x gt | + |y pred -y gt |, x pred is the horizontal coordinate of the center of the prediction frame, y pred is the vertical coordinate of the center of the prediction frame, x gt is the horizontal coordinate of the center of the real frame, and y gtthe longitudinal coordinate of the center of the real box;

[0083] the second loss function:

[0084] Loss2 = -((yy j+1 - yy) log S j + (yy - yy j ) log S j+1 )

[0085] wherein Loss2 is the value of the second loss function; yy j+1 is the label value after rounding up; yy is the actual value of the label; yy j is the label value after rounding down; S j is the predicted value of the output; S j+1 is the adjacent predicted value of the output;

[0086] the third loss function:

[0087]

[0088] wherein N is the total number of the fitting defect categories; y i is the real label of the i-th defect category; p i is the predicted probability of the i-th defect category;

[0089] The total loss function Loss is calculated by the following formula:

[0090] Loss = λ1Loss1 + λ2Loss2 + λ3Loss3

[0091] wherein λ1 is the first weight parameter; λ2 is the second weight parameter; and λ3 is the third weight parameter;

[0092] During training, the model is trained by using the total loss function Loss.

[0093] This invention also provides a system for implementing the feature fusion-based method for detecting defects in transmission line fittings, comprising a data acquisition module, a data processing module, a model building module, a model training module, and a fitting detection module; the data acquisition module, data processing module, model building module, model training module, and fitting detection module are connected in series; the data acquisition module is used to acquire defect image data information of transmission line fittings and upload the data information to the data processing module; the data processing module is used to preprocess the acquired data information according to the received data information to construct a training dataset and upload the data information to the model building module; the model... The construction module is used to build a primary model for detecting defects in transmission line fittings based on the received data, using convolutional layers, pooling layers, and connection layers, and then uploads the data to the model training module. The model training module is used to train the primary model for detecting defects in transmission line fittings using the received data and the obtained training dataset, resulting in a model for detecting defects in transmission line fittings, and then uploads the data to the fitting detection module. The fitting detection module is used to perform feature fusion-based defect detection on target transmission line fittings using the received data and the obtained model for detecting defects in transmission line fittings.

[0094] The present invention provides a method and system for detecting defects in transmission line fittings based on feature fusion. This method fuses information from acquired images of transmission line fittings and uses a constructed defect detection model to detect defects through the image data of the fittings. Therefore, the present invention not only enables defect detection of transmission line fittings based on feature fusion, but also has higher reliability, better accuracy, and higher efficiency. Attached Figure Description

[0095] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0096] Figure 2 This is a schematic diagram of the functional modules of the system of the present invention. Detailed Implementation

[0097] like Figure 1 The diagram shown is a flowchart of the method of the present invention: The method for detecting defects in transmission line fittings based on feature fusion disclosed in this invention includes the following steps:

[0098] S1. Obtain defect image data information of transmission line fittings;

[0099] S2. Preprocess the data obtained in step S1 to construct a training dataset;

[0100] In practice, data preprocessing includes sorting, classifying, cleaning, labeling, and image enhancement to obtain a dataset that meets the set requirements.

[0101] S3. Constructing a power transmission line fitting defect detection primary model based on the convolution layer, the pooling layer and the connection layer;

[0102] Specifically, the method comprises the following steps:

[0103] The input image data is processed by the first convolution layer and the second convolution layer in sequence to extract defect features;

[0104] The defect features are processed by the first defect feature construction module, the third convolution layer, the second defect feature construction module, the fourth convolution layer, the third defect feature construction module, the fifth convolution layer and the sixth convolution layer in sequence to complete the construction of the defect features and obtain constructed defect features;

[0105] The constructed defect features are processed by the global feature extraction module to extract global features;

[0106] After the global features are up-sampled, the global features are spliced with the output of the third defect feature construction module in the channel to obtain first spliced features, and the first spliced features are input into the first defect feature enhancement module; after the output of the third defect feature construction module is processed by the first max-pooling layer, the output of the third defect feature construction module is fused with the output of the first defect feature enhancement module by feature weighting, and the fused output is input into the first detection head for detection;

[0107] The first spliced features are processed by the seventh convolution layer and up-sampling, and then the first spliced features are spliced with the output of the third convolution layer in the channel to obtain second spliced features, and the second spliced features are input into the second defect feature enhancement module;

[0108] The second spliced features are up-sampled, and then the second spliced features are spliced with the output of the second convolution layer in the channel to obtain third spliced features, and the third spliced features are input into the third defect feature enhancement module;

[0109] After the third spliced features are processed by the second max-pooling layer, the third spliced features are fused with the output of the second defect feature enhancement module by feature weighting, and the fused output is input into the second detection head for detection;

[0110] The output of the third defect feature enhancement module is directly input into the third detection head for detection;

[0111] The first detection head, the second detection head and the third detection head are used for classification and bounding box regression processing to obtain final fitting defect detection results;

[0112] In the overall description of the primary model for detecting defects of power transmission line fittings, the first convolutional layer is a 1*1 convolution with a step of 1; the second convolutional layer is a 3*3 convolution with a step of 2; the third convolutional layer is a 3*3 convolution with a step of 2; the fourth convolutional layer is a 3*3 convolution with a step of 2; the fifth convolutional layer is a 1*1 convolution with a step of 1; the sixth convolutional layer is a 3*3 convolution with a step of 2; the seventh convolutional layer is a 3*3 convolution with a step of 1; the first max-pooling layer is a max-pooling layer with a step of 2 and a pooling kernel size of 2; the second max-pooling layer is a max-pooling layer with a step of 2 and a pooling kernel size of 2; and the upsampling is bilinear interpolation upsampling;

[0113] The feature weighting fusion specifically includes the following steps:

[0114] The feature weighting fusion is performed by using the following formula:

[0115]

[0116] In the formula, F out is the output feature after the feature weighting fusion; w i is a weight parameter; F i is the i-th input feature; and ε is a constant set to prevent the denominator from being 0;

[0117] In specific implementation:

[0118] The processing procedures of the first defect feature construction module, the second defect feature construction module and the third defect feature construction module specifically include the following steps:

[0119] The first defect feature construction module, the second defect feature construction module and the third defect feature construction module have the same structure.

[0120] The defect feature construction module includes an eighth convolutional layer, a ninth convolutional layer, a Sigmoid activation function layer, a tenth convolutional layer, a first batch normalization layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a second batch normalization layer and a SiLU activation function layer.

[0121] After the input feature is processed by the eighth convolutional layer, the feature is divided into three paths: the first path feature is processed by the ninth convolutional layer and the Sigmoid activation function layer in sequence to obtain a first branch feature;

[0122] After the second path feature and the first path branch feature are spliced in the channel number, the second path feature and the first path branch feature are processed by the tenth convolutional layer, the first batch normalization layer and the eleventh convolutional layer in sequence to obtain a second branch feature.

[0123] The third branch feature is processed by the twelfth convolutional layer, spliced with the second branch feature in the number of channels, and then sequentially processed by the thirteenth convolutional layer, the second batch normalization layer and the SiLU activation function layer to obtain the output feature of the defect feature construction module. The branch through which the third branch feature passes retains the original input feature, which can alleviate problems such as gradient disappearance and gradient explosion in the deep network training process, and also helps to enhance the robustness of the network.

[0124] Moreover, the splicing operation involved in the defect feature construction module also enables defects under different receptive fields to be fully utilized, effectively constructing the defect features of the fitting.

[0125] In the above description of the processing process of the first defect feature construction module, the second defect feature construction module and the third defect feature construction module: the eighth convolutional layer is a 3*3 convolutional layer with a step of 1; the ninth convolutional layer is a 1*1 depth separable convolutional layer with a step of 1; the tenth convolutional layer is a 3*3 convolutional layer with a step of 1; the eleventh convolutional layer is a 1*1 convolutional layer with a step of 1; the twelfth convolutional layer is a 3*3 convolutional layer with a step of 1; and the thirteenth convolutional layer is a 1*1 convolutional layer with a step of 1.

[0126] The Sigmoid activation function layer uses the following formula as a nonlinear activation function:

[0127]

[0128] In the formula, σ(x) is the processing result of the Sigmoid activation function layer; and x is an element in the feature processed by the Sigmoid activation function layer.

[0129] The SiLU activation function layer uses the following formula as a nonlinear activation function:

[0130]

[0131] In the formula, SiLU(x) is the processing result of the SiLU activation function layer; and x is an element in the feature processed by the SiLU activation function layer.

[0132] In specific implementation:

[0133] The processing process of the global feature extraction module specifically includes the following steps:

[0134] The global feature extraction module includes a pooling layer, a fourteenth convolutional layer, an upsampling layer, a first dilated convolutional layer, a fifteenth convolutional layer, a second dilated convolutional layer, a sixteenth convolutional layer, a normalization layer, a GELU activation function layer and a seventeenth convolutional layer.

[0135] The input feature is divided into four branches:

[0136] The first branch feature is obtained after the first input feature is sequentially processed by a pooling layer, a fourteenth convolutional layer and an up-sampling layer.

[0137] The second branch feature is obtained after the second input feature is processed by a first hollow convolutional layer.

[0138] The third branch feature is obtained after the third input feature is processed by a fifteenth convolutional layer and a second hollow convolutional layer.

[0139] The fourth branch feature is obtained after the fourth input feature is processed by a sixteenth convolutional layer.

[0140] The first branch feature, the second branch feature, the third branch feature and the fourth branch feature are spliced in the channel, and then sequentially processed by a normalization layer, a GELU activation function layer and a seventeenth convolutional layer to obtain the feature output by the global feature extraction module.

[0141] The global feature extraction module generates feature maps with multiple different hollow rates, which can process information of different receptive fields in parallel on the same feature map, thereby capturing context information of different scales. Meanwhile, the size is changed to 1*1 by adaptive mean pooling, and the features of different sizes are extracted by splicing the global features in the channel through up-sampling, thereby effectively extracting the global features of the gold fitting defects.

[0142] In the above description of the processing process of the global feature extraction module: the pooling layer is an adaptive mean pooling layer with an output size of 2; the fourteenth convolutional layer is a 1*1 convolutional layer with a step size of 1; the up-sampling layer adopts bilinear interpolation up-sampling; the first hollow convolutional layer is a 3*3 hollow convolutional layer with a dilation rate of 6; the fifteenth convolutional layer is a 3*3 depth separable convolutional layer with a step size of 1; the second hollow convolutional layer is a 3*3 hollow convolutional layer with a dilation rate of 12; the sixteenth convolutional layer is a 1*1 convolutional layer with a step size of 1; and the seventeenth convolutional layer is a 1*1 convolutional layer with a step size of 1.

[0143] The GELU activation function layer adopts the following formula as a nonlinear activation function:

[0144]

[0145] In the formula, GELU(x) is the processing result of the GELU activation function layer; and x is an element in the feature processed by the GELU activation function layer.

[0146] In specific implementation:

[0147] The processing process of the first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module specifically includes the following steps:

[0148] The first defect feature enhancement module, the second defect feature enhancement module and the third defect feature enhancement module are all the same in structure;

[0149] The defect feature enhancement module comprises an eighteenth convolutional layer, a first feature fusion layer, a max-pooling layer, a nineteenth convolutional layer, a first up-sampling layer, a second feature fusion layer, a second up-sampling layer, a twentieth convolutional layer, a GULU activation function layer, a twenty-first convolutional layer, a Sigmoid activation function layer, a summation layer, a normalization layer and a SiLU activation function layer;

[0150] The input features are divided into two paths:

[0151] The first path of input features is processed by the eighteenth convolutional layer and the first feature fusion layer;

[0152] The second path of input features is processed by the max-pooling layer, the nineteenth convolutional layer, the first up-sampling layer and the second feature fusion layer;

[0153] The output of the max-pooling layer is processed by the second up-sampling layer, and then concatenated with the output of the eighteenth convolutional layer in the channel, and then processed by the twentieth convolutional layer, the GULU activation function layer, the twenty-first convolutional layer and the Sigmoid activation function layer in turn to obtain the relative attention weight;

[0154] The relative attention weight is multiplied by a first constant to obtain a first relative attention weight, and the first feature fusion layer is configured to perform pixel-level multiplication between the obtained first relative attention weight and the output of the eighteenth convolutional layer, and upload the result to the summation layer;

[0155] The relative attention weight is multiplied by a second constant to obtain a second relative attention weight, and the second feature fusion layer is configured to perform pixel-level multiplication between the obtained second relative attention weight and the output of the first up-sampling layer, and upload the result to the summation layer;

[0156] The summation layer adds the received features, and then processes the result by the normalization layer and the SiLU activation function layer in turn to obtain the output of the defect feature enhancement module;

[0157] The defect feature enhancement module performs feature fusion with other branches through max-pooling, enhances the defect features of the fittings, and makes the network focus on the prominent features of the defects;

[0158] In the above description of the processing procedures of the first defect feature enhancement module, the second defect feature enhancement module, and the third defect feature enhancement module: the eighteenth convolutional layer is a 3*3 convolutional layer with a step size of 1; the max-pooling layer is a max-pooling layer with a step size of 2; the nineteenth convolutional layer is a 3*3 convolutional layer with a step size of 1; the first upsampling layer adopts bilinear interpolation upsampling; the second upsampling layer adopts bilinear interpolation upsampling; the twentieth convolutional layer is a 1*1 convolutional layer with a step size of 1; the twenty-first convolutional layer is a 3*3 convolutional layer with a step size of 1;

[0159] The sigmoid activation function layer adopts the following formula as a nonlinear activation function:

[0160]

[0161] In the formula, σ(x) is the processing result of the sigmoid activation function layer; x is an element in the feature processed by the sigmoid activation function layer.

[0162] The SiLU activation function layer adopts the following formula as a nonlinear activation function:

[0163]

[0164] In the formula, SiLU(x) is the processing result of the SiLU activation function layer; x is an element in the feature processed by the SiLU activation function layer.

[0165] The GELU activation function layer adopts the following formula as a nonlinear activation function:

[0166]

[0167] In the formula, GELU(x) is the processing result of the GELU activation function layer; x is an element in the feature processed by the GELU activation function layer.

[0168] In specific implementation:

[0169] The processing procedures of the first detection head, the second detection head, and the third detection head specifically include the following steps:

[0170] The first detection head, the second detection head, and the third detection head have the same structure.

[0171] The detection head includes a twenty-second convolutional layer, a twenty-third convolutional layer, a twenty-fourth convolutional layer, a first fully connected layer, a twenty-fifth convolutional layer, a twenty-sixth convolutional layer, a twenty-seventh convolutional layer, a twenty-eighth convolutional layer, a twenty-ninth convolutional layer, and a second fully connected layer.

[0172] The input feature is divided into two paths:

[0173] The first path input feature is sequentially processed by the twenty-second convolutional layer, the twenty-third convolutional layer, the twenty-fourth convolutional layer and the first full connection layer to obtain the center point coordinates of the prediction frame, the prediction frame width, the prediction frame height and the confidence value;

[0174] The second path input feature is sequentially processed by the twenty-fifth convolutional layer, the twenty-sixth convolutional layer, the twenty-seventh convolutional layer, the twenty-eighth convolutional layer, the twenty-ninth convolutional layer and the second full connection layer to obtain the defect category prediction result of the fitting;

[0175] In the description of the processing process of the above first detection head, second detection head and third detection head: the twenty-second convolutional layer is a 3*3 convolutional layer with a step of 1; the twenty-third convolutional layer is a 3*3 convolutional layer with a step of 1; the twenty-fourth convolutional layer is a 1*1 convolutional layer with a step of 1; the twenty-fifth convolutional layer is a 3*3 depth separable convolutional layer with a step of 1; the twenty-sixth convolutional layer is a 1*1 convolutional layer with a step of 1; the twenty-seventh convolutional layer is a 3*3 depth separable convolutional layer with a step of 1; the twenty-eighth convolutional layer is a 1*1 convolutional layer with a step of 1; and the twenty-ninth convolutional layer is a 1*1 convolutional layer with a step of 1;

[0176] Among them, the branch through which the second path input feature passes uses two convolution kernels of 3*3 depth separable convolution, which reduces the model complexity and improves the fitting defect detection performance;

[0177] S4. Using the training data set obtained in step S2, the transmission line fitting defect detection primary model constructed in step S3 is trained to obtain a transmission line fitting defect detection model;

[0178] The training specifically includes the following steps:

[0179] During the training, the following function is used as the loss function:

[0180] The first loss function is:

[0181]

[0182] In the formula, Loss1 is the first loss function value; b is the prediction frame; b gt is the real frame; IoU(b,b gt ) is the intersection over union of the prediction frame and the real frame; λ is a parameter for controlling the influence degree of shape difference; α is a parameter for controlling the weighting degree of geometric shape difference; and ΔShape is the shape difference measurement value of the frame, and ΔShape=|x pred -x gt |+|y pred -y gt |, x pred is the horizontal coordinate of the center of the prediction frame, and y predx gt is a horizontal coordinate of the real bounding box center; gt is a vertical coordinate of the real bounding box center;

[0183] The second loss function is:

[0184] Loss2 = -((yy j+1 - yy) log S j + (yy - yy j ) log S j+1 )

[0185] wherein Loss2 is a value of the second loss function; yy j+1 is a label value after rounding up; yy is an actual value of the label; yy j is a label value after rounding down; S j is a predicted value outputted; S j+1 is a nearby predicted value outputted;

[0186] The third loss function is:

[0187]

[0188] wherein N is a total number of the accessory defect categories; y i is a real label of the i-th defect category; p i is a predicted probability of the i-th defect category;

[0189] The total loss function Loss is calculated by using the following formula:

[0190] Loss = λ1Loss1 + λ2Loss2 + λ3Loss3

[0191] wherein λ1 is a first weight parameter; λ2 is a second weight parameter; and λ3 is a third weight parameter;

[0192] During the training, the model is trained by using the total loss function Loss;

[0193] S5. The transmission line accessory defect detection model obtained in step S4 is used to perform the transmission line accessory defect detection based on the feature fusion on the target transmission line accessory.

[0194] As Figure 2The system of the application is shown in the functional module diagram: the system for realizing the feature fusion-based transmission line hardware defect detection method disclosed in the application comprises a data acquisition module, a data processing module, a model construction module, a model training module and a hardware detection module; the data acquisition module, the data processing module, the model construction module, the model training module and the hardware detection module are sequentially connected in series; the data acquisition module is used for acquiring defect image data information of the transmission line hardware and uploading the data information to the data processing module; the data processing module is used for pre-processing the acquired data information according to the received data information, constructing a training data set and uploading the data information to the model construction module; the model construction module is used for constructing a transmission line hardware defect detection primary model based on a convolution layer, a pooling layer and a connection layer according to the received data information and uploading the data information to the model training module; the model training module is used for training the constructed transmission line hardware defect detection primary model by using the obtained training data set according to the received data information, obtaining a transmission line hardware defect detection model and uploading the data information to the hardware detection module; and the hardware detection module is used for performing feature fusion-based transmission line hardware defect detection on a target transmission line hardware by using the obtained transmission line hardware defect detection model according to the received data information.

Claims

1. A method for detecting defects in transmission line fittings based on feature fusion, comprising the following steps: S1. Obtain defect image data information of transmission line fittings; S2. Preprocess the data obtained in step S1 to construct a training dataset; S3. Construct a primary model for detecting defects in transmission line fittings based on convolutional layers, pooling layers, and connection layers; specifically including the following steps: The input image data is processed sequentially through the first and second convolutional layers to extract defect features; The defect features are then processed sequentially through the first defect feature construction module, the third convolutional layer, the second defect feature construction module, the fourth convolutional layer, the third defect feature construction module, the fifth convolutional layer, and the sixth convolutional layer to complete the construction of the defect features and obtain the constructed defect features. Defect features are constructed and processed by a global feature extraction module to extract global features. After upsampling, the global features are concatenated with the output of the third defect feature construction module on the channel to obtain the first concatenated feature, which is then used as the input of the first defect feature enhancement module. The output of the third defect feature construction module is processed by the first maximum pooling layer, and then fused with the output of the first defect feature enhancement module through feature weighting, and input into the first detection head for detection. The first splicing feature, after being passed through the seventh convolutional layer and upsampled, is spliced ​​with the output of the third convolutional layer on the channel to obtain the second splicing feature, which is then used as the input to the second defect feature enhancement module. The second splicing feature, after upsampling, is spliced ​​with the output of the second convolutional layer on the channel to obtain the third splicing feature, which is then used as the input to the third defect feature enhancement module. After the third splicing feature is processed by the second maximum pooling layer, it is fused with the output of the second defect feature enhancement module through feature weighting and then input into the second detection head for detection. The output of the third defect feature enhancement module is directly input into the third detection head for detection. The first, second, and third detection heads are used for classification and bounding box regression processing to obtain the final hardware defect detection results. The processing steps of the first defect feature construction module, the second defect feature construction module, and the third defect feature construction module specifically include the following steps: The first defect feature construction module, the second defect feature construction module, and the third defect feature construction module have the same structure; The defect feature construction module includes the eighth convolutional layer, the ninth convolutional layer, the Sigmoid activation function layer, the tenth convolutional layer, the first batch normalization layer, the eleventh convolutional layer, the twelfth convolutional layer, the thirteenth convolutional layer, the second batch normalization layer, and the SiLU activation function layer. The input features are processed by the eighth convolutional layer and then divided into three features: the first feature is processed by the ninth convolutional layer and the sigmoid activation function layer in sequence to obtain the first branch feature; The second feature and the first branch feature are concatenated in terms of channel number, and then processed sequentially through the tenth convolutional layer, the first batch normalization layer, and the eleventh convolutional layer to obtain the second branch feature. The third feature is processed by the twelfth convolutional layer, and then concatenated with the second feature in terms of channel number. It is then processed sequentially by the thirteenth convolutional layer, the second batch normalization layer, and the SiLU activation function layer to obtain the output feature of the defect feature construction module. S4. Using the training dataset obtained in step S2, train the primary model for detecting defects in transmission line fittings constructed in step S3 to obtain the transmission line fitting defect detection model. S5. Using the transmission line hardware defect detection model obtained in step S4, perform feature fusion-based transmission line hardware defect detection on the target transmission line hardware.

2. The method for detecting defects in transmission line fittings based on feature fusion according to claim 1, characterized in that... The processing steps of the global feature extraction module include the following: The global feature extraction module includes a pooling layer, a fourteenth convolutional layer, an upsampling layer, a first dilated convolutional layer, a fifteenth convolutional layer, a second dilated convolutional layer, a sixteenth convolutional layer, a normalization layer, a GELU activation function layer, and a seventeenth convolutional layer; The input features are divided into four paths: The first input features are processed sequentially through a pooling layer, a fourteenth convolutional layer, and an upsampling layer to obtain the first branch features; The second input feature is processed by the first dilated convolutional layer to obtain the second branch feature; The third input feature is processed by the fifteenth convolutional layer and the second dilated convolutional layer to obtain the third branch feature; The fourth input feature is processed by the sixteenth convolutional layer to obtain the fourth branch feature; The features of the first branch, the second branch, the third branch, and the fourth branch are concatenated on the channel and then processed sequentially through a normalization layer, a GELU activation function layer, and a seventeenth convolutional layer to obtain the features output by the global feature extraction module.

3. The method for detecting defects in transmission line fittings based on feature fusion according to claim 2, characterized in that... The processing steps of the first defect feature enhancement module, the second defect feature enhancement module, and the third defect feature enhancement module specifically include the following steps: The first defect feature enhancement module, the second defect feature enhancement module, and the third defect feature enhancement module all have the same structure; The defect feature enhancement module includes the eighteenth convolutional layer, the first feature fusion layer, the max pooling layer, the nineteenth convolutional layer, the first upsampling layer, the second feature fusion layer, the second upsampling layer, the twentieth convolutional layer, the GULU activation function layer, the twenty-first convolutional layer, the Sigmoid activation function layer, the summation layer, the normalization layer, and the SiLU activation function layer. The input features are divided into two paths: The first input features are processed through the eighteenth convolutional layer and the first feature fusion layer; The second input feature is processed through a max pooling layer, a nineteenth convolutional layer, a first upsampling layer, and a second feature fusion layer. The output of the max pooling layer is processed by the second upsampling layer, and then concatenated with the output of the eighteenth convolutional layer on the channel. After being processed by the twentieth convolutional layer, the GULU activation function layer, the twenty-first convolutional layer, and the Sigmoid activation function layer in sequence, the relative attention weights are obtained. The first relative attention weight is obtained by multiplying the relative attention weight by a set first constant. The first feature fusion layer is used to multiply the obtained first relative attention weight by the output of the eighteenth convolutional layer at the pixel level and upload the result to the summation layer. The relative attention weight is multiplied by a set second constant to obtain the second relative attention weight. The second feature fusion layer is used to multiply the obtained second relative attention weight with the output of the first upsampling layer at the pixel level and upload the result to the summing layer. The summation layer adds up the received features, and then processes the result sequentially through the normalization layer and the SiLU activation function layer to obtain the output of the defect feature enhancement module.

4. The method for detecting defects in transmission line fittings based on feature fusion according to claim 3, characterized in that... The processing steps of the first, second, and third detection heads specifically include the following steps: The first, second, and third detection heads have the same structure. The detection head includes a 22nd convolutional layer, a 23rd convolutional layer, a 24th convolutional layer, a first fully connected layer, a 25th convolutional layer, a 26th convolutional layer, a 27th convolutional layer, a 28th convolutional layer, a 29th convolutional layer, and a second fully connected layer; The input features are divided into two paths: The first input features are processed sequentially through the 22nd, 23rd, and 24th convolutional layers and the first fully connected layer to obtain the center point coordinates, width, height, and confidence value of the predicted bounding box. The second input features are processed sequentially through the 25th, 26th, 27th, 28th, and 29th convolutional layers and the second fully connected layer to obtain the defect category prediction results for the hardware. The 22nd convolutional layer is a 3x3 convolutional layer with a stride of 1; the 23rd convolutional layer is a 3x3 convolutional layer with a stride of 1; the 24th convolutional layer is a 1x1 convolutional layer with a stride of 1; the 25th convolutional layer is a 3x3 depthwise separable convolutional layer with a stride of 1; the 26th convolutional layer is a 1x1 convolutional layer with a stride of 1; the 27th convolutional layer is a 3x3 depthwise separable convolutional layer with a stride of 1; the 28th convolutional layer is a 1x1 convolutional layer with a stride of 1; and the 29th convolutional layer is a 1x1 convolutional layer with a stride of 1.

5. The method for detecting defects in transmission line fittings based on feature fusion according to claim 4, characterized in that... In the primary model for detecting defects in transmission line fittings: the first convolutional layer is a 1x1 convolution with a stride of 1; the second convolutional layer is a 3x3 convolution with a stride of 2; the third convolutional layer is a 3x3 convolution with a stride of 2; the fourth convolutional layer is a 3x3 convolution with a stride of 2; the fifth convolutional layer is a 1x1 convolution with a stride of 1; the sixth convolutional layer is a 3x3 convolution with a stride of 2; the seventh convolutional layer is a 3x3 convolution with a stride of 1; the first max pooling layer is a max pooling layer with a stride of 2 and a kernel size of 2; the second max pooling layer is a max pooling layer with a stride of 2 and a kernel size of 2; bilinear interpolation upsampling is used. The aforementioned feature weighted fusion specifically includes the following steps: The following formula is used for feature weighting fusion: In the formula The output features are the weighted and fused features. These are weight parameters; Let i be the i-th feature of the input; This is a set constant used to prevent the denominator from being zero; In the defect feature construction module: the eighth convolutional layer is a 3*3 convolutional layer with a stride of 1; the ninth convolutional layer is a 1*1 depthwise separable convolutional layer with a stride of 1; the tenth convolutional layer is a 3*3 convolution with a stride of 1; the eleventh convolutional layer is a 1*1 convolution with a stride of 1; the twelfth convolutional layer is a 3*3 convolution with a stride of 1; and the thirteenth convolutional layer is a 1*1 convolution with a length of 1. The Sigmoid activation function layer uses the following formula as the non-linear activation function: In the formula This is the result of processing the Sigmoid activation function layer; For elements in the features processed by the Sigmoid activation function layer; The SiLU activation function layer uses the following formula as the non-linear activation function: In the formula The result of processing the SiLU activation function layer; Elements in the features processed by the SiLU activation function layer; In the global feature extraction module: the pooling layer is an adaptive mean pooling layer with an output size of 2; the fourteenth convolutional layer is a 1*1 convolutional layer with a stride of 1; the upsampling layer uses bilinear interpolation upsampling; the first dilated convolutional layer is a 3*3 dilated convolutional layer with a dilation rate of 6; the fifteenth convolutional layer is a 3*3 depthwise separable convolutional layer with a stride of 1; the second dilated convolutional layer is a 3*3 dilated convolutional layer with a dilation rate of 12; the sixteenth convolutional layer is a 1*1 convolutional layer with a stride of 1; and the seventeenth convolutional layer is a 1*1 convolutional layer with a stride of 1. The GELU activation function layer uses the following formula as the non-linear activation function: In the formula The result of processing the GELU activation function layer; Elements in the features processed by the GELU activation function layer; In the defect feature enhancement module: the eighteenth convolutional layer is a 3*3 convolutional layer with a stride of 1; the max pooling layer is a max pooling layer with a stride of 2; the nineteenth convolutional layer is a 3*3 convolutional layer with a stride of 1; the first upsampling layer uses bilinear interpolation upsampling; the second upsampling layer uses bilinear interpolation upsampling; the twentieth convolutional layer is a 1*1 convolutional layer with a stride of 1; and the twenty-first convolutional layer is a 3*3 convolutional layer with a stride of 1. The Sigmoid activation function layer uses the following formula as the non-linear activation function: In the formula This is the result of processing the Sigmoid activation function layer; For elements in the features processed by the Sigmoid activation function layer; The SiLU activation function layer uses the following formula as the non-linear activation function: In the formula The result of processing the SiLU activation function layer; Elements in the features processed by the SiLU activation function layer; The GULU activation function layer uses the following formula as the non-linear activation function: In the formula The result of processing the GELU activation function layer; Elements in the features processed by the GELU activation function layer.

6. The method for detecting defects in transmission line fittings based on feature fusion according to claim 5, characterized in that... The training described in step S4 specifically includes the following steps: During training, the following function is used as the loss function: First loss function: In the formula This is the value of the first loss function; For prediction boxes; For true bounding boxes; This represents the intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box. Parameters used to control the degree of influence of shape differences; Parameters used to control the weighting of geometric differences; This is a measure of the shape difference of the box, and , The x-coordinate of the center of the prediction box. The ordinate of the center of the prediction box. The x-coordinate of the center of the true bounding box. The ordinate of the center of the true bounding box; Second loss function: In the formula This is the value of the second loss function; The label value is rounded up. The actual value of the label; The label value is rounded down. This is the predicted output value; This is the output of the nearest neighbor prediction value; Third loss function: In the formula This represents the total number of hardware defect categories; The true label for the i-th defect category; Let be the predicted probability of the i-th defect category; The total loss function is calculated using the following formula. : In the formula This is the first weight parameter; This is the second weighting parameter; This is the third weighting parameter; During training, the total loss function is used. Train the model.

7. A system for implementing the feature fusion-based defect detection method for transmission line fittings as described in any one of claims 1 to 6, characterized in that... It includes a data acquisition module, a data processing module, a model building module, a model training module, and a hardware inspection module; the data acquisition module, data processing module, model building module, model training module, and hardware inspection module are connected in series; the data acquisition module is used to acquire defect image data information of transmission line hardware and upload the data information to the data processing module; The data processing module is used to preprocess the acquired data information based on the received data information in order to build a training dataset and upload the data information to the model building module; The model building module is used to build a primary model for detecting defects in transmission line fittings based on the received data information, using convolutional layers, pooling layers, and connection layers, and upload the data information to the model training module. The model training module is used to train the primary model for detecting defects in transmission line fittings using the received data information and the obtained training dataset, thereby obtaining the transmission line fitting defect detection model, and uploading the data information to the fitting detection module. The fittings detection module is used to perform feature fusion-based defect detection on the target transmission line fittings based on the received data and the obtained transmission line fittings defect detection model.

Images