Foreign object detection method for transmission lines based on YOLOv9 and diffusion model

Abstract

The present invention belongs to the technical field of image target detection, and relates to a transmission line foreign body detection method based on YOLOv9 and a diffusion model. By acquiring a transmission line foreign body image and performing data annotation, a transmission line foreign body image data set is obtained; a Vincent graph diffusion model is used to perform multimodal data enhancement, and an image data set after data enhancement is obtained, and the data is divided into a training set and a verification set; a public data set is used to pre-train an improved YOLOv9 model; the training set is input into the pre-trained improved YOLOv9 model for training, and a transmission line foreign body detection model is obtained; and the transmission line foreign body detection model is configured on an edge device to realize transmission line foreign body detection. The present invention uses the Vincent graph diffusion model for multimodal data enhancement, expands the model training data volume, improves the YOLOv9 model, and improves the transmission line foreign body detection performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image target detection and relates to a method for detecting foreign objects in a power transmission line based on YOLOv9 and a diffusion model. Background Art

[0002] Various transmission lines are spread across residential areas, railway networks and other places. They are easily attached by foreign objects such as balloons, kites, and bird nests, thus affecting the power transmission of the transmission lines. In order to prevent these accidents, timely detection of foreign objects on the transmission lines is of utmost importance. If only traditional manual methods are used to check whether there are foreign objects on the transmission lines, it will be a huge project. With the development of deep learning in the field of target detection in computer vision, it is divided into two-stage target detection methods and single-stage target detection methods, both of which can be deployed on edge devices. It is more suitable to use target detection algorithms combined with edge devices to identify complex and numerous foreign objects on transmission lines.

[0003] Among them, although the two-stage target detection method has higher accuracy, it consumes more computing power and has slow reasoning. Since the edge device itself has low computing power, the reasoning speed and accuracy will be sharply reduced. If the reasoning speed is slow, foreign objects on the transmission line cannot be detected in time. If the detection accuracy is low, the foreign objects detected will be inaccurate or even unable to be detected, which will reduce the safety of the transmission line. Single-stage target detection methods, such as the YOLOv9 algorithm, have fast reasoning speed and lower computing power requirements compared to the two-stage target detection method due to its real-time and end-to-end characteristics. However, the use of the YOLOv9 algorithm deployed on edge devices still has the problem of insufficient computing power, resulting in poor reasoning effect; and the collection of foreign object data sets on transmission lines is cumbersome and the amount of data is small, so a large number of data sets need to be supplemented to train the model.

[0004] Since the probability of foreign objects appearing on power transmission lines is low and there are few cases, it is difficult to collect data sets, which will result in a small number of training data sets. The models of the YOLO series (including YOLOv9) are relatively complex. If the training data set is small, it will lead to model overfitting, so data enhancement is needed to strengthen the training of the model. However, traditional data enhancement methods such as cutting, pasting, and Mosaic will lose the authenticity of the image itself, making it impossible for the model to learn the real features; there is also a way to use generative adversarial networks (GANs) to generate images for data enhancement, but generative adversarial networks (GANs) are unstable when training models, and the images generated are likely to be uneven and lack diversity.

[0005] The Wensheng graph diffusion model can be used as a data enhancement method to improve the generalization ability and accuracy of the model due to its stability, realistic image generation, and the ability to generate other objects of the same category based on text labels. Therefore, how to design a lightweight real-time object detection network YOLOv9 and use the Wensheng graph diffusion model to generate realistic images to enhance data, so as to achieve good results in transmission line foreign body detection under low computing power, which is of great significance to transmission line safety.

[0006] Since the devices for real-time detection of foreign objects in power transmission lines are all edge devices, and the computing power of edge devices themselves is low, and the use of YOLO series (including YOLOv9) algorithms requires sufficient computing power while taking into account the inference speed, fps, and accuracy, it is crucial to make lightweight improvements to the models deployed on edge devices. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention proposes a transmission line foreign object detection method based on YOLOv9 and diffusion model.

[0008] The present invention is achieved through the following technical solutions.

[0009] The foreign body detection method for transmission lines based on YOLOv9 and diffusion model includes the following steps:

[0010] S1. Acquire the transmission line foreign body image and perform data annotation to obtain the transmission line foreign body image dataset;

[0011] S2. Use the Wensheng graph diffusion model to perform multimodal data enhancement to obtain the image dataset after data enhancement, and divide it into a training set and a validation set;

[0012] S3. Use the public dataset (COCO2017) to pre-train the improved YOLOv9 model and obtain the best pre-training weights;

[0013] S4, input the training set obtained in step S2 into the pre-trained improved YOLOv9 model for training, verify the verification set in each round, obtain the best weights and hyperparameters, and obtain a power transmission line foreign body detection model;

[0014] S5. Configure the power transmission line foreign object detection model on the edge device to implement power transmission line foreign object detection;

[0015] The improved YOLOv9 model includes a backbone network, a neck network, a head network and a programmable gradient information module; the backbone network, the neck network and the head network are connected in sequence to form a main branch, the programmable gradient information module is connected to the backbone network, the programmable gradient information module is used as an auxiliary branch, and the neck network adopts a high-level screening feature fusion pyramid (HS-FPN); the MobileNetv4-Small network is used as the backbone network, and the backbone network includes MNv4-1 module, MNv4-2 module, MNv4-3 module, MNv4-4 module and MNv4-5 module which are arranged in sequence. Modules, among which the MNv4-1 module contains a convolutional normalization module (convbn), the MNv4-2 module is composed of two consecutive convolutional normalization modules in series, the MNv4-3 module is composed of a first universal inverted bottleneck block (UIB1), three second universal inverted bottleneck blocks (UIB2) and another first universal inverted bottleneck block (UIB1) in series, the MNv4-4 module is composed of two consecutive first universal inverted bottleneck blocks (UIB1) and four universal inverted bottleneck blocks (UIB2) in series, and the MNv4-5 module is composed of two convolutional normalization modules (convbn) in series.

[0016] Further preferably, the text-image diffusion model includes a language model, a text encoder, an image encoder, a multimodal model of image-text pairing (CLIP), a variational autoencoder (VAE) and a U-Net network. The variational autoencoder (VAE) includes an encoder and a decoder. The power transmission line foreign body image dataset is processed by the language model, the text encoder and the image encoder to obtain corresponding text feature vectors and image feature vectors for training the multimodal model of image-text pairing. The text description of the foreign body and the text description of the background are input into the trained multimodal model of image-text pairing to generate a text encoding vector; the random matrix is ​​input into the encoder of the variational autoencoder (VAE) to generate a low-dimensional matrix of the latent space, the low-dimensional matrix of the latent space, the text encoding vector, the customized foreign body position information, the background initial information, and the noise intensity are input into the U-Net network to perform noise clipping and merging operations, as well as diffusion noise addition operations and reverse diffusion denoising operations to generate a low-dimensional image matrix, the low-dimensional image matrix is ​​input into the decoder of the variational autoencoder (VAE) for decoding, and a reconstructed image is generated. The reconstructed image and the original power transmission line foreign body image constitute a data enhanced image dataset.

[0017] Further preferably, when training a multimodal model for image-text pairing, text feature vectors and image feature vectors are constructed as image-text pairs, and the similarity comparison loss of the image-text pairs is calculated for training the multimodal model for image-text pairing.

[0018] Further preferably, the text information T consisting of the text description of the foreign object and the text description of the background is input into the trained multimodal model of image-text pairing, and the text encoding vector Z is generated by the multimodal model of image-text pairing. T ={t 1 , t 2 ,…,t n ,t bg}, where t 1 , t 2 ,…,t n are the text encoding vectors corresponding to the 1st, 2nd, …, nth text descriptions of foreign objects, respectively, t bg Vector of text code for background.

[0019] Further preferably, the process of generating the low-dimensional image matrix is:

[0020] Step 1: Given customized foreign body location information And fixed background initial information ,in are the location information of the 1st, 2nd, …, mth foreign objects respectively, m≤n, that is, a foreign object has at least one text description; the foreign object location information and the fixed background initial information are input into the U-Net network to generate the foreign object location noise information Z Lm ={z L1 , z L2 ,…,z Lm} and background noise information z bg , where z L1 , z L2 ,…,z Lm are the 1st, 2nd, ..., mth foreign body position noise information respectively; Then the foreign body position noise information is clipped and merged on the background noise information to obtain the noise containing all foreign body position noise information after clipping and merging :

[0021] ;

[0022] in, is the noise intensity;

[0023] Step 2: The encoder of the variational autoencoder (VAE) takes the input random matrix Mapping to the latent space obtains the low-dimensional matrix z of the latent space, ,in For a given random matrix The conditional probability distribution of the low-dimensional matrix 𝑧 in the latent space when , is the mean value generated by the encoder, is the standard deviation generated by the encoder, is the parameter of the encoder;

[0024] Step 3: Encode the text vector Z T , noise containing all foreign body position noise information The low-dimensional matrix of the latent space is input into the U-Net network, and the diffusion denoising process and the reverse diffusion denoising process are continuously performed to generate a low-dimensional image matrix.

[0025] Further optimization, the high-level screening feature fusion pyramid (HS-FPN) and three RepNCSPELAN4 feature fusion modules used in the neck network. The high-level screening feature fusion pyramid includes three coordinate attention mechanism modules and two selective feature fusion modules. First, the 8-fold downsampling features extracted by the MNv4-3 module, the 16-fold downsampling module extracted by the MNv4-4 module, and the 32-fold downsampling module extracted by the MNv4-5 module are respectively passed through the three coordinate attention mechanism modules to eliminate redundant data and compress features, and then they are respectively passed through 1×1 convolutional layers to change the number of channels to obtain low-level features, intermediate features, and high-level features, respectively.

[0026] The high-level features are fused through one of the RepNCSPELAN4 feature fusion modules to obtain the first output feature;

[0027] The high-level features and the intermediate features are fused through one of the selective feature fusion modules, and then the second output feature is obtained after feature fusion through a RepNCSPELAN4 feature fusion module;

[0028] The high-level features and low-level features are fused through another selective feature fusion module, and then the third output feature is obtained after feature fusion through a RepNCSPELAN4 feature fusion module.

[0029] Further optimization, a two-stage neural search architecture (Tu-NAS) is used for parameter design of the MobileNetv4-Small network to determine the optimal convolution kernel size and universal inverted bottleneck block (UIB) configuration respectively.

[0030] Further preferably, the convolution normalization module is further improved. The improved convolution normalization module includes ordinary convolution and depthwise separable convolution. First, the features of the image are extracted by ordinary convolution, and then the spatial features are extracted again by depthwise separable convolution. Then, the extracted feature channels are split into two parts, one part remains unchanged, and the other part is subjected to point convolution operation. Point convolution fuses the channel features, and then performs element-by-element multiplication operation through star operation to further fuse the features of the two branches, and then batch normalization is performed, and the h-swish activation function is used.

[0031] Further preferably, the 8-fold, 16-fold and 32-fold down-sampling features extracted from the backbone network are input into a programmable gradient information module, and the programmable gradient information module supplements the obtained classification and regression gradient information into the main branch.

[0032] Further preferably, the head network uses a focus loss function (Focaler-IoU) and introduces adjustment factors d and u:

[0033] ;

[0034] in, is the intersection-over-union ratio of the reconstructed real box and the predicted box, IoU is the intersection-over-union ratio, d and u are adjustment factors, and the adjustment factor value range is in the (0, 1) interval. The loss of the focal loss function (Focaler-IoU) for .

[0035] The present invention has the following advantages:

[0036] (1) The present invention proposes to use the Wensheng graph diffusion model for multimodal data enhancement, because the Wensheng graph diffusion model is very stable in training, the generated images are generally good and real, and there will be no situation of good and bad, and the desired images can be generated based on the input text, greatly increasing the diversity of images. The multimodal data enhancement of the present invention can generate real, logically laid out images without losing their real information, and the generation process is stable, the image quality is generally good, and the desired image can be obtained by inputting text and the real frame position information of foreign objects, making the image more diverse, and obtaining a data-enhanced image that conforms to the real frame position information, without the need for additional manual annotation, which is conducive to training the YOLOv9 model and improving its generalization ability.

[0037] (2) The present invention uses the MobileNetv4-Small network as the backbone network and uses a two-stage neural search architecture (Tu NAS) to search for a neural network architecture, obtaining a Pareto-optimal network structure that meets the needs of edge devices with low computing power, which can maximize the use of the computing power of the edge device. The convolution normalization module is improved, and the h-swish activation function is used to better alleviate the gradient disappearance and improve the generalization ability to a certain extent.

[0038] (3) The neck network uses a lighter high-level screening feature fusion pyramid (HS-FPN) to fuse features, and retains a non-anchor-based approach to decouple classification and regression, which can effectively reduce the redundancy of anchor boxes and improve the reasoning efficiency of the model. The bounding box regression loss function is improved using the focal loss function (Focaler-IoU), which makes it easier to calculate the loss, and introduces an adjustment factor to reduce the loss contribution of samples that are correctly detected, increase the loss contribution of difficult samples, and improve the utilization of lightweight networks.

[0039] (4) The head network does not use non-maximum suppression (NMS), that is, it uses the NMS-Free module. During training, it uses a parallel approach of assigning multiple prediction box branches to one real box (one-many branches) and assigning one prediction box branch to one real box (one-one branches) to enrich information. It uses a unified matching metric and further optimizes the NMS-Free module using programmable gradient information. During reasoning, it abandons the one-to-many branch and uses the one-to-one branch, which can speed up reasoning without losing too much information. This improves the end-to-end deployment capability, reasoning speed, and frames per second (fps). BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention is a flow chart of the method.

[0041] Figure 2 Schematic diagram of a multimodal model for training image-text pairing.

[0042] Figure 3 Processing flow chart for variational autoencoder (VAE) and U-Net network.

[0043] Figure 4 Schematic diagram of the improved YOLOv9 model.

[0044] Figure 5 Schematic diagram of the improved convolution normalization module.

[0045] Figure 6 This is the head network workflow diagram. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below with reference to the accompanying drawings.

[0047] like Figure 1 As shown, the transmission line foreign body detection method based on YOLOv9 and diffusion model includes the following steps:

[0048] S1. Acquire the transmission line foreign body image and perform data annotation to obtain a transmission line foreign body image dataset, wherein the data annotation at least annotates the position information of the real frame;

[0049] S2. Use the Wensheng graph diffusion model to perform multimodal data enhancement to obtain the image dataset after data enhancement, and divide it into a training set and a validation set;

[0050] S3. Use the public dataset (COCO2017) to pre-train the improved YOLOv9 model and obtain the best pre-training weights;

[0051] S4, input the training set obtained in step S2 into the pre-trained improved YOLOv9 model for training, verify the verification set in each round, obtain the best weights and hyperparameters, and obtain a power transmission line foreign body detection model;

[0052] S5. Configure the transmission line foreign object detection model on the edge device to implement transmission line foreign object detection.

[0053] In this embodiment, refer to Figure 2 and Figure 3 The text-image diffusion model includes a language model, a text encoder, an image encoder, a multimodal model of image-text pairing (CLIP), a variational autoencoder (VAE) and a U-Net network. The variational autoencoder (VAE) includes an encoder and a decoder. The foreign body image dataset of the power transmission line is processed by the language model, the text encoder and the image encoder to obtain the corresponding text feature vector and image feature vector, which are used to train the multimodal model of image-text pairing. The text description of the foreign body and the text description of the background are input into the trained multimodal model of image-text pairing to generate a text encoding vector; the random matrix The low-dimensional matrix of the latent space is generated by inputting the encoder of the variational autoencoder (VAE), and the low-dimensional matrix of the latent space, the text encoding vector, the customized foreign body position information, the background initial information, and the noise point intensity are input into the U-Net network for noise cropping and merging operations, as well as diffusion noise addition operations and reverse diffusion denoising operations to generate a low-dimensional image matrix. The low-dimensional image matrix is ​​input into the decoder of the variational autoencoder (VAE) for decoding to generate a reconstructed image. The reconstructed image and the original transmission line foreign body image constitute the image data set after data enhancement, which is divided into a training set and a validation set according to 8:2. Since the foreign body position information is added to the input of the U-Net network, the image generated later will generate customized and diversified transmission line foreign bodies on the foreign body position information at the time of input, which can simulate the real position information of foreign bodies on the transmission line, and since the foreign body and background are used for noise cropping and merging operations, the foreign body and the background can be more closely matched, making the image generated by data enhancement more realistic, and the generated image does not require additional manual data annotation, which greatly saves the time of data annotation.

[0054] Specifically, the process of training a multimodal model for image-text pairing is as follows: Figure 2As shown, the transmission line foreign body image is input into the language model (ChatGPT4.0) to generate text, and then the text is input into the text encoder to obtain the text feature vector u, and the transmission line foreign body image is input into the image encoder to obtain the image feature vector v. The text feature vector and the image feature vector are constructed as an image-text pair. A transmission line foreign body image may match multiple texts, and a text may also match multiple transmission line foreign body images. Then, the similarity contrast loss of the image-text pair is calculated to train the multimodal model of image-text pairing. The contrast loss of each image-text pair is calculated, that is, the loss from text to image. and image-to-text loss :

[0055] ;

[0056] ;

[0057] Where N is the number of image-text pairs, is the smoothing coefficient, i represents the index value of the text, j represents the index value of the transmission line foreign body image, k represents the transmission line foreign body image matching the i-th text, h represents the text matching the j-th transmission line foreign body image, and u i is the i-th text feature vector, v j is the jth image feature vector, v k is the image feature vector corresponding to the i-th text feature vector, u h is the text feature vector that matches the jth image feature vector.

[0058] The total loss is , in order to maximize the similarity of correctly paired image-text pairs and minimize the similarity of incorrect pairs, and finally obtain a trained multimodal model (CLIP) for image-text pairing that can match the power line foreign object dataset.

[0059] Specifically, in this embodiment, the text information T consisting of the text description of the foreign object and the text description of the background is input into the trained multimodal model of image-text pairing, and the text encoding vector Z is generated by the multimodal model of image-text pairing. T ={t 1 , t 2 ,…,t n ,t bg}, where t 1 , t 2 ,…,t n are the text encoding vectors corresponding to the 1st, 2nd, …, nth text descriptions of foreign objects, respectively, t bg Vector of text code for background.

[0060] Reference Figure 3 , the process of generating a low-dimensional image matrix is:

[0061] Step 1: Given customized foreign body location information And fixed background initial information ,in are the location information of the 1st, 2nd, …, mth foreign objects respectively, m≤n, that is, a foreign object has at least one text description; the foreign object location information and the fixed background initial information are input into the U-Net network to generate the foreign object location noise information Z Lm ={z L1 , z L2 ,…,z Lm} and background noise information z bg , where z L1 , z L2 ,…,z Lm are the 1st, 2nd, ..., mth foreign body position noise information respectively; Then the foreign body position noise information is clipped and merged on the background noise information to obtain the noise containing all foreign body position noise information after clipping and merging :

[0062] ;

[0063] in, is the noise intensity.

[0064] Step 2: The encoder of the variational autoencoder (VAE) takes the input random matrix Mapping to the latent space obtains the low-dimensional matrix z of the latent space, ,in For a given random matrix The conditional probability distribution of the low-dimensional matrix z in the latent space is, is the mean value generated by the encoder, is the standard deviation generated by the encoder, are the parameters of the encoder.

[0065] Step 3: Encode the text vector Z T , noise containing all foreign body position noise information The low-dimensional matrix of the latent space is input into the U-Net network, and the diffusion denoising process and the reverse diffusion denoising process are continuously performed to generate a low-dimensional image matrix.

[0066] The diffusion noise adding process is: ;

[0067] Among them, q is the probability distribution in the diffusion noise process, is the noise intensity, is a Gaussian distribution, is the identity matrix, is the low-dimensional matrix of the latent space at time t, is the low-dimensional matrix of the latent space at time t-1;

[0068] The reverse diffusion denoising process is: ;

[0069] in is the reverse diffusion model, The mean function of the back-diffusion model, is the covariance function of the back diffusion model, and t is the time.

[0070] In this embodiment, the low-dimensional image matrix is ​​input into the decoder of the variational autoencoder (VAE) for re-parameterization, and then the result obtained after the re-parameterization is decoded in the decoder. The decoder maps the potential low-dimensional image matrix back to the data space to generate a reconstructed image.

[0071] Since foreign object position noise information is artificially added to each foreign object to be generated as the input of the denoising part of the U-Net network, all foreign object position noise information is cropped and merged on the background noise information, and then input into the denoising part of the U-Net network. After continuous noise addition and denoising operations, a low-dimensional image matrix with foreign object position information is output, and then the low-dimensional image matrix is ​​restored using a decoder, so that a foreign object image that matches the input foreign object position information can be generated. Therefore, the artificially given foreign object position information can be directly used as the annotation of the real frame of the reconstructed image after data enhancement, and no manual annotation is required. Through this method of merging and cropping noise for each foreign object, the position layout of the foreign object can be simply and intuitively controlled without the need for additional complex operations. Even if many overlapping foreign object position information is given, since each foreign object has a separate position noise information, the authenticity of the generated foreign object can be guaranteed, and no artifacts will be generated around the foreign object generated in this way; the image after data enhancement can be better integrated with the background to achieve a more realistic effect, and dense multi-target foreign objects can be well generated.

[0072] Reference Figure 4 The improved YOLOv9 model described in this embodiment includes a backbone network, a neck network, a head network and a programmable gradient information module; the backbone network, the neck network and the head network are connected in sequence to form a main branch, the programmable gradient information module is connected to the backbone network, the programmable gradient information module is used as an auxiliary branch, and the neck network adopts a high-level screening feature fusion pyramid (HS-FPN).

[0073] The MobileNetv4-Small network is used as the backbone network. The backbone network includes MNv4-1 module, MNv4-2 module, MNv4-3 module, MNv4-4 module and MNv4-5 module, which are arranged in sequence. The MNv4-1 module contains a convolutional normalization module (convbn), the MNv4-2 module is composed of two consecutive convolutional normalization modules in series, the MNv4-3 module is composed of a first universal inverted bottleneck block (UIB1), three second universal inverted bottleneck blocks (UIB2) and another first universal inverted bottleneck block (UIB1) in series, the MNv4-4 module is composed of two consecutive first universal inverted bottleneck blocks (UIB1) and four universal inverted bottleneck blocks (UIB2) in series, and the MNv4-5 module is composed of two convolutional normalization modules (convbn) in series.

[0074] The high-level screening feature fusion pyramid (HS-FPN) and three RepNCSPELAN4 feature fusion modules used in the neck network. The high-level screening feature fusion pyramid includes three coordinate attention mechanism modules and two selective feature fusion modules. First, the 8-fold downsampling features extracted by the MNv4-3 module, the 16-fold downsampling module extracted by the MNv4-4 module, and the 32-fold downsampling module extracted by the MNv4-5 module are respectively passed through the three coordinate attention mechanism modules to eliminate redundant data and compress features. Then, they are respectively passed through 1×1 convolutional layers to change the number of channels to obtain low-level features, intermediate features, and high-level features respectively.

[0075] The high-level features are fused through one of the RepNCSPELAN4 feature fusion modules to obtain the first output feature;

[0076] The high-level features and the intermediate features are fused through one of the selective feature fusion modules, and then the second output feature is obtained after feature fusion through a RepNCSPELAN4 feature fusion module;

[0077] The high-level features and low-level features are fused through another selective feature fusion module, and then the third output feature is obtained after feature fusion through a RepNCSPELAN4 feature fusion module.

[0078] The head network includes three detection heads corresponding to three output features, and each detection head includes a classification branch and a regression branch.

[0079] In this embodiment, facing edge devices with limited computing resources, the backbone network adopts the MobileNetv4-Small network, and the two-stage neural search architecture (Tu-NAS) is used for the parameter design of the MobileNetv4-Small network, which focuses on determining the optimal convolution kernel size and the universal inverted bottleneck block (UIB) configuration. Compared with the original YOLOv9 network, the backbone network can reduce the parameters and computational complexity of the model with fewer lost features. For the convolution parameter setting, a two-stage neural search architecture (Tu-NAS) is used, in which the first stage is a coarse-grained search stage: focusing on determining the optimal convolution kernel size, and a fine-grained search stage: searching for the configuration of the deep convolution layer in the universal inverted bottleneck block (UIB), which can achieve Pareto optimality on edge devices with lower computing power. And appropriately reduce redundant universal inverted bottleneck blocks, reduce the number of parameters while improving parameter utilization.

[0080] Furthermore, this method further improves the convolution normalization module in the MobileNetv4-Small network. The improved convolution normalization module is shown in Figure 5 , including ordinary convolution and depth-wise separable convolution. First, the features of the image are extracted through ordinary convolution, and then the spatial features are extracted again using depth-wise separable convolution. Then the extracted feature channels are split into two parts, one part remains unchanged, and the other part is subjected to point convolution. Point convolution fuses the channel features, and then the star operation is used to perform element-by-element multiplication to further fuse the features of the two branches. The star operation can reduce the computational complexity and improve the performance of feature fusion. It can enable the module to obtain richer features without increasing the number of parameters and the amount of calculation. Then batch normalization is performed and the h-swish activation function is used. The h-swish activation function simulates the nonlinear function with a linear piecewise function, which can better alleviate the gradient disappearance and improve a certain generalization ability.

[0081] In this embodiment, the neck network (Neck) uses a high-level screening feature fusion pyramid (HS-FPN) for feature selection and fusion. First, the features of the input coordinate attention mechanism module are processed, first through global average pooling and global maximum pooling. The main purpose of maximum pooling is to extract the most relevant data from each channel, while average pooling is to evenly obtain all data from the features to minimize excessive losses. The resulting features are combined together. This step can greatly reduce the dimensionality of the features and reduce the dimensionality of the features, eliminate redundant data, compress features and reduce the number of parameters. Then, the Sigmoid activation function is used to determine the weight value of each channel, thereby obtaining the weight of each channel.

[0082] For feature fusion, the multi-scale feature maps generated by the backbone network present high-level features with rich semantic information, but the target location is relatively shallow. On the contrary, low-level features provide precise target locations but contain limited semantic information. Therefore, for the selected features, they are input into Figure 4 The Selective Feature Fusion module (SFF) in , which uses high-level features as weights to filter the basic semantic information embedded in low-level features, and then fuses the high-level features with the low-level features:

[0083] ;

[0084] ;

[0085] Among them, CA represents the operation of the coordinate attention mechanism module, BL is bilinear interpolation, is the transposed convolution, For advanced features, For low-level features, For extended advanced features, is the feature after filtering and fusion.

[0086] The selective feature fusion module is used to fuse the favorable information of the features, reduce the amount of redundant parameters, and no longer use the bottom-up feature fusion method in the previous YOLO algorithm, which can also obtain a better feature fusion effect. And the programmable gradient information module (PGI) of the original YOLOv9 model can still be kept as an auxiliary branch, and the 8-fold, 16-fold and 32-fold down-sampled features extracted from the backbone network are input into the programmable gradient information module. The features are passed through the convolution module, upsampling module, concat splicing module and RepNCSPELAN4 feature fusion convolution module in the programmable gradient information module, and the obtained classification and regression gradient information is added to the main branch. The number of redundant channels of the RepNCSPELAN4 feature fusion module in YOLOv9 is adjusted to improve the utilization of parameters and reduce a certain amount of parameters and calculations.

[0087] The head network retains the non-anchor-based approach to reduce the redundancy of anchor frames, and the detection head decouples classification and regression. Using the focal loss function (Focaler-IoU) can simplify the loss calculation. Introducing adjustment factors d and u will make the model pay more attention to samples that are difficult to detect, thereby improving the model accuracy. This embodiment uses the focal loss function (Focaler-IoU) for improvement, which can reduce the loss contribution to samples that are easy to detect and increase the contribution to samples that are difficult to detect. ;

[0088] in, is the intersection over union (IoU) of the reconstructed real box and the predicted box. IoU is the intersection over union ratio. Both d and u are adjustment factors. The adjustment factor ranges from (0 to 1). By simply adjusting the values ​​of d and u, Focus on different regression samples and focus on the loss function (Focaler-IoU) for .

[0089] The detection head does not use non-maximum suppression NMS, that is, the NMS-Free module is used, such as Figure 6 As shown in the figure, a consistent dual label assignment strategy is introduced during training, that is, one real box is assigned to multiple prediction box branches (one-many one-to-many branches) and one real box is assigned to one prediction box (one-one one-to-one branches). One real box is assigned to multiple prediction boxes. This strategy provides rich supervision signals and can optimize the backbone network and the neck network. One real box is assigned to one prediction box so that the network does not rely on NMS for post-processing, but it will reduce the accuracy. Therefore, these two branches are trained together during training. In order to achieve better joint training results, the two branches use a consistent matching metric to evaluate the consistency between the real box and the prediction box, so that the one-to-many branch can make up for the shortcomings of the one-to-one branch with insufficient supervision signals and low accuracy. During reasoning, one real box is discarded to assign multiple prediction box branches (one-many one-to-many branches), and only one real box is assigned to one prediction box branch (one-one one-to-one branch), which can speed up the model reasoning speed and increase the number of frames per second (fps). The unified matching metric is , Q indicates whether the predicted anchor point is within the instance, e is the classification score, and p represent the predicted box and the true box respectively, 𝛾 and 𝛿 are two hyperparameters that balance the influence of semantic prediction task and position regression task respectively, and the one-to-many branch index is , the one-to-one branch indicator is , They are two hyperparameters that affect the balanced semantic prediction task and position regression task of one-to-many branches, These are the two hyperparameters that affect the balanced semantic prediction task and the position regression task of the one-to-one branch. During training, the gap between the two indicators is continuously reduced, so that the two branches can obtain a unified matching metric.

[0090] The NMS-Free module is further improved by inputting the information of the one-to-many branch into the programmable gradient information, and using the programmable gradient information to supplement the lost information of its gradient, so that the one-to-many branch can obtain richer feature information, and artificially add perturbed gradient information to further improve the generalization ability, robustness and convergence speed of the bounding box regression loss function of the one-to-many branch, so that better supervision information can be integrated into the one-to-one branch, allowing the one-to-one branch to learn richer feature information, so that the one-to-one branch can achieve better accuracy improvement during reasoning.

[0091] The network structure configuration file YOLOv9-s.yaml in the original YOLOv9 code, that is, the YOLOv9-s model, is used as the baseline model for ablation experiment comparison with the improved model. The model is trained on a CentOS system machine configured with Intel(R) Xeon(R) Gold 6126CPU @ 2.60GHz, using an Nvidia-V100-16GB graphics card. The versions of the programming software, language, and framework used are: pycharm 2022, python 3.9, and torch 2.0.1; NVIDIA GeForce RTX 4060 Laptop GPU is used for inference.

[0092] Evaluation indicators:

[0093] Number of network layers: The total number of network layers of the model. The higher the number of layers, the more complex the model.

[0094] Parameter quantity: The size of the model, the higher the parameter, the larger the model.

[0095] Computational Amount: The amount of computation required for model reasoning, i.e. computing power.

[0096] Frames per second: The number of frames that the model can process per second. A higher frame rate means a faster inference speed.

[0097] mAP@0.5: When the intersection over union (IoU) is set to 0.5, the AP of all images in each category is calculated, and then the average of all categories is calculated, i.e. mAP@0.5.

[0098] mAP@0.5:0.95: represents the average precision at different intersection-over-union (IoU) thresholds (from 0.5 to 0.95, with a step size of 0.05) (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

[0099] Table 1

[0100]

[0101] The experimental results are shown in Table 1. The improved YOLOv9 model of the present invention and the original YOLOv9-s model are compared in terms of network parameters, parameter quantity, floating-point operations, frames per second, mAP@0.5 and mAP@0.5:0.95. The experiment shows that, when the number of network layers of the improved YOLOv9 model of the present invention is reduced by 15.3%, the number of parameters is reduced by 45%, the amount of calculation is reduced by 9.8%, and the number of frames per second is increased by 37.8%, the mAP@0.5 of four types of foreign objects, namely balloons, floating objects, bird nests and plastic bags, is only reduced by 0.2%, which is almost negligible, and the mAP@0.5:0.95 is only reduced by 2.1%. Comprehensively considering all indicators, it is proved that the experimental effect is improved. In the environment of real-time foreign object detection on power transmission lines, the four main foreign objects, namely balloons, floating objects, bird nests, and plastic bags, can be detected with high accuracy. In addition, due to the reduction in the number of network layers, the number of parameters and the amount of calculation, and the increase in the number of frames per second, the improved YOLOv9 model is more suitable for edge devices with insufficient computing power, as well as for this real-time detection application scenario. Only by quickly detecting foreign objects can we better protect power transmission lines and reduce hidden dangers.

[0102] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the present invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A foreign body detection method for power transmission lines based on YOLOv9 and diffusion model, characterized in that: The following steps are involved: S1. Acquire the transmission line foreign body image and perform data annotation to obtain the transmission line foreign body image dataset; S2. Use the Wensheng graph diffusion model to perform multimodal data enhancement to obtain an image dataset after data enhancement, and divide it into a training set and a validation set; S3. Use the public dataset to pre-train the improved YOLOv9 model to obtain the best pre-training weights; S4, input the training set obtained in step S2 into the pre-trained improved YOLOv9 model for training, verify the verification set in each round, obtain the best weights and hyperparameters, and obtain a power transmission line foreign body detection model; S5. Configure the power transmission line foreign object detection model on the edge device to implement power transmission line foreign object detection; The improved YOLOv9 model includes a backbone network, a neck network, a head network and a programmable gradient information module; the backbone network, the neck network and the head network are connected in sequence to form a main branch, the programmable gradient information module is connected to the backbone network, the programmable gradient information module is used as an auxiliary branch, and the neck network adopts a high-level screening feature fusion pyramid; the MobileNetv4-Small network is used as the backbone network, the backbone network includes MNv4-1 module, MNv4-2 module, MNv4-3 module, MNv4-4 module and MNv4-5 module arranged in sequence, wherein the MNv4-1 module includes a convolution normalization module, the convolution normalization module includes ordinary convolution and depth separable convolution, firstly, the image features are extracted by ordinary convolution, and then the spatial features are extracted again by depth separable convolution, and then the extracted feature channel is split into two parts, one part remains unchanged, and the other part is subjected to point convolution operation, the point convolution fuses the channel features, and then the features of the two branches are further fused by element-by-element multiplication operation, and then batch normalization is performed, and the h-swish activation function is used; The MNv4-2 module is composed of two consecutive convolutional normalization modules in series, the MNv4-3 module is composed of a first universal inverted bottleneck block, three second universal inverted bottleneck blocks and another first universal inverted bottleneck block in series, the MNv4-4 module is composed of two consecutive first universal inverted bottleneck blocks and four second universal inverted bottleneck blocks in series, and the MNv4-5 module is composed of two convolutional normalization modules in series; the neck network uses a high-level screening feature fusion pyramid and three RepNCSPELAN4 feature fusion modules. The high-level screening feature fusion pyramid includes three coordinate attention mechanism modules and two selective feature fusion modules. First, the 8-fold downsampling features extracted by the MNv4-3 module, the 16-fold downsampling module extracted by the MNv4-4 module, and the 32-fold downsampling module extracted by the MNv4-5 module are respectively subjected to three coordinate attention mechanism modules to eliminate redundant data and compress features, and then they are respectively subjected to 1×1 convolutional layers to change the number of channels to obtain low-level features, intermediate features, and high-level features, respectively; The high-level features are fused through one of the RepNCSPELAN4 feature fusion modules to obtain the first output feature; The high-level features and the intermediate features are fused through one of the selective feature fusion modules, and then the second output feature is obtained after feature fusion through a RepNCSPELAN4 feature fusion module; The high-level features and low-level features are fused through another selective feature fusion module, and then the third output feature is obtained after feature fusion through a RepNCSPELAN4 feature fusion module; The formula for the selective feature fusion module to fuse high-level features with low-level features is as follows: ； ； Among them, CA represents the operation of the coordinate attention mechanism module, BL is bilinear interpolation, is the transposed convolution, For advanced features, For low-level features, For extended advanced features, is the feature after filtering and fusion; The first output feature, the second output feature, and the third output feature are respectively input into the three detection heads of the head network for classification.

2. The method for detecting foreign objects in a power transmission line based on YOLOv9 and a diffusion model according to claim 1, characterized in that: The text-image diffusion model includes a language model, a text encoder, an image encoder, a multimodal model of image-text pairing, a variational autoencoder and a U-Net network. The variational autoencoder includes an encoder and a decoder. The transmission line foreign body image dataset is processed by the language model, the text encoder and the image encoder to obtain corresponding text feature vectors and image feature vectors for training the multimodal model of image-text pairing. The text description of the foreign body and the text description of the background are input into the trained multimodal model of image-text pairing to generate a text encoding vector; a random matrix is ​​input into the encoder of the variational autoencoder to generate a latent space low-dimensional matrix; the latent space low-dimensional matrix, the text encoding vector, the customized foreign body position information, the background initial information, and the noise intensity are input into the U-Net network to perform noise cropping and merging operations, as well as diffusion noise addition operations and reverse diffusion denoising operations to generate a low-dimensional image matrix; the low-dimensional image matrix is ​​input into the decoder of the variational autoencoder for decoding to generate a reconstructed image; the reconstructed image and the original transmission line foreign body image constitute a data enhanced image dataset.

3. The method for detecting foreign objects in a power transmission line based on YOLOv9 and a diffusion model according to claim 2, characterized in that: When training a multimodal model for image-text pairing, the text feature vector and the image feature vector are constructed as an image-text pair, and the similarity comparison loss of the image-text pair is calculated to train the multimodal model for image-text pairing.

4. The method for detecting foreign objects in a power transmission line based on YOLOv9 and a diffusion model according to claim 2, characterized in that: The text information T consisting of the text description of the foreign object and the text description of the background is input into the trained multimodal model of image-text pairing, and the text encoding vector Z is generated by the multimodal model of image-text pairing. T ={t1, t2,…,t n ,t bg }, where t1, t2,…,t n are the text encoding vectors corresponding to the 1st, 2nd, …, nth text descriptions of foreign objects, respectively, t bg Vector of text code for background.

5. The method for detecting foreign objects in a power transmission line based on YOLOv9 and a diffusion model according to claim 4, characterized in that: The process of generating a low-dimensional image matrix is: Step 1: Given customized foreign body location information And fixed background initial information ,in are the location information of the 1st, 2nd, …, mth foreign objects respectively, m≤n, that is, a foreign object has at least one text description; the foreign object location information and the fixed background initial information are input into the U-Net network to generate the foreign object location noise information Z Lm ={z L1 ,z L2 ,…,z Lm } and background noise information z bg , where z L1 , z L2 ,…,z Lm They are the 1st, 2nd, …, mth foreign body position noise information respectively; Then, the foreign object position noise information is cropped and merged on the background noise information to obtain the noise containing all foreign object position noise information after cropping and merging. : ； in, is the noise intensity; Step 2: The encoder of the variational autoencoder maps the input random matrix x to the latent space to obtain the low-dimensional matrix z of the latent space. ,in is the conditional probability distribution of the low-dimensional matrix 𝑧 in the latent space given a random matrix x, is the mean value generated by the encoder, is the standard deviation generated by the encoder, is the parameter of the encoder; Step 3: Encode the text vector Z T , noise containing all foreign body position noise information The low-dimensional matrix of the latent space is input into the U-Net network, and the diffusion denoising process and the reverse diffusion denoising process are continuously performed to generate a low-dimensional image matrix.

6. The method for detecting foreign objects in a power transmission line based on YOLOv9 and a diffusion model according to claim 1, characterized in that: A two-stage neural search architecture is used for the parameter design of the MobileNetv4-Small network to determine the optimal convolution kernel size and the general inverted bottleneck block configuration respectively.

7. The method for detecting foreign objects in a power transmission line based on YOLOv9 and a diffusion model according to claim 1, characterized in that: The 8x, 16x, and 32x downsampled features extracted from the backbone network are input into the programmable gradient information module, which then adds the obtained classification and regression gradient information to the main branch.

8. The method for detecting foreign objects in a power transmission line based on YOLOv9 and a diffusion model according to claim 1, characterized in that: The head network uses a focused loss function and introduces adjustment factors d and u: ； in, is the intersection over union ratio of the reconstructed real box and the predicted box, IoU is the intersection over union ratio, d and u are both adjustment factors, and the adjustment factor value range is in the interval (0, 1). The loss of the focus loss function for .

Images