Electric power image target detection method and system based on feature fusion and attention

Abstract

The invention discloses an electric power image target detection method and system based on feature fusion and attention. The method comprises the following steps: carrying out basic information setting on edge equipment; performing image acquisition on the to-be-detected power equipment, and preprocessing the acquired image by the edge equipment; the edge device performs defect detection by using a preset target detection model based on feature fusion and attention; and judging whether data acquisition is completed or not, if so, generating defect detection result statistical information of the to-be-detected power equipment, all defect categories and positions which can be identified by the model and the number of defects detected by each category, and otherwise, returning. According to the invention, the appropriate feature map is selected according to the clustering result, and then the combination of the four designed feature fusion modules is utilized to carry out adaptive feature fusion, so that the detection performance on the electric power data set with large target size difference is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and more specifically, relates to a method and system for power image target detection based on feature fusion and attention. Background Art

[0002] Due to its powerful processing power and adaptability, the neural network model has a wide range of application prospects in the power grid. For example, by analyzing historical data, the neural network model can learn the laws and characteristics of the energy transmission process of the power system, thereby optimizing the issuance of energy dispatch instructions and improving the efficiency of the power grid. The neural network model can quickly detect, locate and classify faults in the power system, reduce power outage time, and improve the reliability of the power grid. At present, using neural networks to detect defects in collected images in power scenarios is an important application.

[0003] Existing target detection methods are:

[0004] The patent with authorization announcement number CN116843649A discloses a method for intelligent defect detection of power transmission lines based on an improved YOLOv5 network. The YOLOv5 network is improved to construct a target detection network. The target detection network includes a backbone network, a path fusion network and an output module. The backbone network includes Foces, CBL convolution blocks, an improved cross-stage local network, a CBAM module, and a spatial pyramid pooling structure module; wherein the improved cross-stage local network converts the residual module into a ghost network, and the CBAM module processes the feature layer with a channel attention mechanism and a spatial attention mechanism respectively; the path fusion network uses a feature pyramid FPN+OAN structure, the FPN layer transmits and fuses the feature information of the high and low layers through upsampling, and the PAN layer splices the low-level features with the high-level features.

[0005] However, due to various factors such as the size of the equipment itself, the shooting angle and distance of the drone, the target sizes of different defect categories in the collected transmission line images vary greatly. The sizes of targets of the same category also vary greatly, which affects the detection effect of the target detection algorithm. Summary of the invention

[0006] In order to solve the deficiencies in the prior art, the present invention provides a power image target detection method based on feature fusion and attention.

[0007] The present invention adopts the following technical solution.

[0008] The first aspect of the present invention proposes a method for power image target detection based on feature fusion and attention, comprising the following steps:

[0009] S1, setting basic information for the edge device, where the basic information includes the device to which the power image belongs, the power image acquisition range, and the maximum input space size of the image that the edge device can receive;

[0010] S2, collecting images of the power equipment to be inspected, and the edge device preprocesses the collected images to be inspected;

[0011] S3, the edge device uses a preset target detection model based on feature fusion and attention to perform defect detection on the preprocessed image to be detected, wherein the target detection model includes a backbone network for feature extraction, an attention module and a feature fusion module for processing the extracted features, and a detector module for realizing target detection;

[0012] S4, determine whether the data collection is completed. If the collection is completed, generate statistical information on the defect detection results of the power equipment to be detected, including: basic information of the edge device settings, all defect categories and locations that the model can identify, and the number of defects detected in each category. Otherwise, return to step S2.

[0013] Preferably, in S2, the specific content of image preprocessing is:

[0014] Adjust the image size of the image to be detected to the maximum input space size (H, W) that the edge device can receive during the inference process, and normalize the image pixel value from the [0, 255] interval to the [0, 1] interval;

[0015] The image format of the image to be detected is adjusted and converted to the (N, C, H, W) format, where N represents the batch size, which is set to 1; C represents the number of channels in an image, which is 3, representing R, G, and B respectively; H represents the number of pixels in the vertical dimension of the image; and W represents the number of pixels in the horizontal dimension of the image.

[0016] Preferably, in S3, the backbone network is specifically:

[0017] The backbone network includes 6 convolution modules, 5 CSP bottleneck modules and 1 SPPF module; the preprocessed image passes through the first convolution module, the second convolution module, the first CSP bottleneck module, the third convolution module, the second CSP bottleneck module, the fourth convolution module, the third CSP bottleneck module, the fifth convolution module, the fourth CSP bottleneck module, and the first SPPF module in sequence. Each convolution module contains a convolution layer, batch normalization and activation function;

[0018] During the operation, the feature graph Feat1 obtained from the second CSP bottleneck module, the feature graph Feat2 obtained from the fourth convolution module, the feature graph Feat3 obtained from the fourth CSP bottleneck module, the feature graph Feat4 obtained from the fifth CSP bottleneck module and the feature graph Feat5 obtained from the SPPF module are obtained;

[0019] The spatial size of the feature map Feat1 is The feature map Feat2 space size is The feature map Feat3 space size is The feature map Feat4 space size is The feature map Feat5 space size is

[0020] Preferably, the feature map is selected by using Gaussian mixture clustering, and the length and width of the defects to be detected in the image are clustered using a Gaussian mixture clustering model. The input is the length and width of the defects to be detected in the image relative to the original image, which is defect length / image length, defect width / image width. After clustering, all defects are divided into five categories according to the length and width relative to the original image, and the clustering result is mapped back to the original image. The cluster center point is matched with the receptive field of the feature map output by each module in the backbone network to obtain the feature map output by the corresponding module in the backbone network, and the cluster center point matches the receptive field size of the module layer where the selected feature map is located;

[0021] The parameter estimation of the clustering model includes two steps:

[0022] Step E: Under given model parameters, calculate the posterior probability of each data point belonging to each Gaussian distribution;

[0023] Step M: Based on the posterior probability calculated in step E, update the parameters of each Gaussian distribution, including mean, covariance, and mixing coefficient, to maximize the likelihood function of the observed data;

[0024] The E-step-M-step is continuously iterated until the change of the model parameters is less than the set threshold or the set number of iterations is reached, and the algorithm converges.

[0025] Preferably, in S3, the feature fusion module for processing the extracted features includes a first feature fusion module to an eighth feature fusion module;

[0026] After the feature map Feat5 is input to the attention module, the attention module outputs the signal feature Feat after processing the coordinate attention and channel attention. 51 , respectively as the first input feature of the first feature fusion module 1 and the second input feature of the eighth feature fusion module 8;

[0027] The feature map Feat4 is used as the second input feature of the first feature fusion module, and the first feature fusion module 1 performs fusion output signal feature Feat 41 , signal feature Feat 41 As the first input feature of the second feature fusion module 2 and the second input feature of the seventh feature fusion module 7;

[0028] The feature map Feat3 is used as the second input feature of the second feature fusion module 2, and the second feature fusion module 2 performs feature fusion and outputs the signal feature Feat 31 As the first input feature of the third feature fusion module 3, and the second input feature of the sixth feature fusion module 6;

[0029] The feature map Feat2 is used as the second input feature of the third feature fusion module 3, and the third feature fusion module 3 performs feature fusion and outputs the signal feature Feat 21 As the first input feature of the fourth feature fusion module 4, and the second input feature of the fifth feature fusion module 5;

[0030] The feature map Feat1 is used as the second input feature of the fourth feature fusion module 4, and the feature fusion is performed to obtain the signal feature Feat 11 As the first input feature of the fifth feature fusion module 5;

[0031] The signal feature Feat output by the fifth feature fusion module 5 after feature fusion 22 As the first input feature of the sixth feature fusion module 6, the sixth feature fusion module 6 performs feature fusion and outputs a signal feature Feat 32 As the first input feature of the seventh feature fusion module 7, the seventh feature fusion module 7 performs feature fusion and outputs a signal feature Feat 42 As the first input feature of the eighth feature fusion module 8; the eighth feature fusion module 8 performs feature fusion and outputs a signal feature Feat 52 ;

[0032] The signal feature Feat 11 、Feat 22 、Feat 32 、Feat 42 、Feat 52 As input to the detector module.

[0033] Preferably, the attention module first converts the channel into 2 times through a convolution module with a convolution kernel size of 1x1 and a step size of 1, and then divides the input feature map from the channel into two parts in the channel dimension, the first part is retained, and the second part is processed by a coordinate attention submodule and a channel attention submodule, and the feature map obtained after processing and the feature map before processing are added point by point using an addition layer; after the coordinate attention and channel attention processing are completed, the obtained feature map and the feature map retained in the first part are connected in the channel dimension, and then feature fusion is performed through a convolution module with a convolution kernel size of 1x1 and a step size of 1.

[0034] Preferably, the coordinate attention submodule and the channel attention submodule are specifically:

[0035] Coordinate attention submodule: Through one-dimensional pooling in the horizontal and vertical directions, the corresponding feature maps containing horizontal and vertical information are obtained. After convolution and activation, the horizontal attention feature map and the vertical attention feature map are obtained. Then, the two attention feature maps are applied to the input feature map by multiplication to obtain the output feature map.

[0036] Channel attention submodule: Through one-dimensional pooling in the channel direction, the corresponding feature map containing channel direction information is obtained. After convolution and activation, the horizontal attention feature map and the vertical attention feature map are obtained. Then, both attention feature maps are applied to the input feature map by multiplication to obtain the output feature map.

[0037] Preferably, the first feature fusion module 1: connects the first input feature and the second input feature of the same size in the channel dimension using a connection layer, and then sequentially uses a convolution module with a convolution kernel size of 1x1 and a step size of 1, a grouped convolution module with a convolution kernel size of 3x3, a step size of 1 and a group number equal to the number of channels, and a convolution module with a convolution kernel size of 1x1 and a step size of 1 to fuse the features;

[0038] The second feature fusion module 2, the third feature fusion module 3, and the fourth feature fusion module 4 have the same structure. The spatial size of the first input feature is smaller than the spatial size of the second input feature. The upsampling layer is used to upsample the first input feature to the same spatial size as the second input feature. Then, the connection layer is used to connect the upsampled first input feature and the second input feature in the channel dimension. Finally, a convolution module with a convolution kernel size of 1x1 and a step size of 1, a grouped convolution module with a convolution kernel size of 3x3, a step size of 1 and a group number equal to the number of channels, and a convolution module with a convolution kernel size of 1x1 and a step size of 1 are used in sequence to fuse the features.

[0039] The fifth feature fusion module 5, the sixth feature fusion module 6, and the seventh feature fusion module 7 have the same structure; the spatial size of the first input feature is larger than the spatial size of the second input feature, and the first input feature is downsampled to the same spatial size as the second input feature using a downsampling layer, and then the upsampled first input feature and the second input feature are connected in the channel dimension using a connection layer, and finally, a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 are used in turn to fuse the features;

[0040] The eighth feature fusion module 8 first uses a connection layer to connect the first input feature and the second input feature of the same spatial size in the channel dimension, and finally uses a convolution module with a convolution kernel size of 1x1 and a step size of 1, and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to fuse the features.

[0041] Preferably, the specific process of target detection is: the feature map Feat obtained by the feature fusion module 11 、Feat 22 、Feat 32 、Feat 42 and Feat 52 , both use the convolution module with a convolution kernel size of 3x3, a step size of 1, and a channel number equal to the number of categories, and the convolution module with a convolution kernel size of 3x3, a step size of 1, and a channel number of 16x4 to perform convolution operations, and obtain two sets of feature maps, specifically: Feature map Feat for category prediction 1,cls 、Feat 2,cls 、Feat 3,cls 、Feat 4,cls and Feat 5,cls , and the feature map Feat of position prediction 1,pos 、Feat 2,pos 、Feat 3,pos 、Feat 4,pos and Feat 5,pos ;

[0042] The feature maps of category prediction and position prediction are flattened from two dimensions to one dimension in the spatial dimension, and then concatenated in the last dimension to obtain the feature map Feat cls And the feature map Feat' pos ; Feature map Feat cls Output for target category prediction;

[0043] The feature map Feat' pos The channel dimension is divided into 16 groups, and the weighted sum of the 16 groups of coordinate positions is obtained to obtain the feature map Feat pos, the weights from the first group to the last group are 1 to 16 respectively; the feature map is the output of the target position prediction, and then the prediction in lt and rb format is converted into the center point length and width format, and Feat cls Post-processing is performed together to finally obtain the target detection result.

[0044] Preferably, the post-processing is to filter out the detection results with confidence scores less than the threshold according to the confidence threshold, the confidence threshold is 0.001, and non-maximum suppression is performed on the remaining results, and the iou threshold is 0.5.

[0045] The second aspect of the present invention proposes a power image target detection system based on feature fusion and attention. Using the method described in the first aspect of the present invention, the power image target detection system includes an image acquisition module, an image preprocessing module, a defect detection module and a result statistics module:

[0046] An image acquisition module, which acquires images of the power equipment to be tested;

[0047] Image preprocessing module, adjusts the size and format of the image;

[0048] Defect detection module, which uses a preset feature fusion and attention-based object detection model for defect detection;

[0049] The result statistics module records the defect detection results and determines whether all data has been processed. If so, it generates defect detection result statistics.

[0050] The third aspect of the present invention proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the power image target detection method based on feature fusion and attention as described in the first aspect of the present invention.

[0051] The fourth aspect of the present invention proposes a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the power image target detection method based on feature fusion and attention described in the first aspect of the present invention is used.

[0052] The beneficial effect of the present invention is that, compared with the prior art,

[0053] This patent proposes a method and system for power image target detection based on feature fusion and attention. Through feature fusion and attention, the detection performance on power data sets with large target size differences is improved; the appropriate feature map is selected according to the clustering results, and then the combination of the designed four feature fusion modules is used to perform adaptive feature fusion, which is more suitable for processing data with large target size differences and adapting to different situations. The modular design is adopted to facilitate continuous updating and adding new components in the later stage. The attention module is used to embed the position information into the attention map so that the model can obtain information from a larger area and enhance the expression ability of the learned features. The combination of coordinate attention and channel attention is used in the attention module to extract attention from two dimensions: space and channel. Coordinate attention is used to consider the contextual information of the spatial coordinates in the input feature map, so as to enhance the network's sensitivity to spatial position. Channel attention is used to consider the relative importance between different feature channels, and use spatial position information to guide the network to learn richer spatial feature representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a detection flow chart of the present invention;

[0055] Figure 2 It is a target detection model framework based on feature fusion and attention;

[0056] Figure 3 This is the structural diagram of the attention module;

[0057] Figure 4 This is the structure diagram of the coordinate attention module;

[0058] Figure 5 This is the structure diagram of the channel attention module;

[0059] Figure 6 This is the structure diagram of the feature fusion module. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only embodiments of a part of the present invention, rather than all embodiments. Based on the spirit of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the protection scope of the present invention.

[0061] like Figure 1 As shown, Embodiment 1 of the present invention provides a method for detecting power image targets based on feature fusion and attention, comprising the following steps:

[0062] S1, setting basic information for the edge device, where the basic information includes the device to which the power image belongs, the power image acquisition range, and the maximum input space size of the image that the edge device can receive;

[0063] S2, collecting images of the power equipment to be inspected, and the edge device preprocesses the collected images to be inspected;

[0064] Preferably, in S2, the specific content of image preprocessing is:

[0065] Adjust the image size of the image to be detected to the maximum input space size (H, W) that the edge device can receive during the inference process, and normalize the image pixel value from the [0, 255] interval to the [0, 1] interval;

[0066] The image format of the image to be detected is adjusted and converted to the (N, C, H, W) format, where N represents the batch size, which is set to 1; C represents the number of channels in an image, which is 3, representing R, G, and B respectively; H represents the number of pixels in the vertical dimension of the image; and W represents the number of pixels in the horizontal dimension of the image.

[0067] S3, the edge device uses a preset target detection model based on feature fusion and attention to perform defect detection on the preprocessed image to be detected, wherein the target detection model includes a backbone network for feature extraction, an attention module and a feature fusion module for processing the extracted features, and a detector module for realizing target detection;

[0068] Preferably, in S3, the backbone network is specifically:

[0069] The backbone network includes 6 convolution modules, 5 CSP bottleneck modules and 1 SPPF module; the preprocessed image passes through the first convolution module, the second convolution module, the first CSP bottleneck module, the third convolution module, the second CSP bottleneck module, the fourth convolution module, the third CSP bottleneck module, the fifth convolution module, the fourth CSP bottleneck module, and the first SPPF module in sequence. Each convolution module contains a convolution layer, batch normalization and activation function;

[0070] During the operation, the feature graph Feat1 obtained from the second CSP bottleneck module, the feature graph Feat2 obtained from the fourth convolution module, the feature graph Feat3 obtained from the fourth CSP bottleneck module, the feature graph Feat4 obtained from the fifth CSP bottleneck module and the feature graph Feat5 obtained from the SPPF module are obtained;

[0071] The spatial size of the feature map Feat1 is The feature map Feat2 space size is The feature map Feat3 space size is The feature map Feat4 space size is The feature map Feat5 space size is

[0072] Specifically, the modules of the backbone network are:

[0073] The first convolution module has a convolution layer with 64 output channels, a convolution kernel size of 3x3, and a stride of 2.

[0074] The second convolution module has a convolution layer with 128 output channels, a convolution kernel size of 3x3, and a stride of 2.

[0075] The first CSP bottleneck module has a CSP bottleneck layer repeated 3 times, the number of output channels is 128, and the bottleneck layer structure uses residual connection;

[0076] The third convolution module has a convolution layer with an output channel number of 256, a convolution kernel size of 3x3, and a stride of 2;

[0077] The second CSP bottleneck module has a CSP bottleneck layer repeated 6 times, the number of output channels is 256, and the bottleneck layer structure uses residual connection;

[0078] The fourth convolution module has a convolution layer with an output channel number of 512, a convolution kernel size of 3x3, and a stride of 2;

[0079] The third CSP bottleneck module has a CSP bottleneck layer repeated 6 times, the number of output channels is 512, and the bottleneck layer structure uses residual connection;

[0080] The fifth convolution module has a convolution layer with an output channel number of 1024, a convolution kernel size of 3x3, and a stride of 2;

[0081] The fourth CSP bottleneck module has a CSP bottleneck layer repeated 3 times, the number of output channels is 1024, and the bottleneck layer structure uses residual connection;

[0082] The sixth convolution module has a convolution layer with an output channel number of 1024, a convolution kernel size of 3x3, and a stride of 2;

[0083] The fifth CSP bottleneck module has a CSP bottleneck layer repeated 3 times, the number of output channels is 1024, and the bottleneck layer structure uses residual connection;

[0084] The first SPPF module has an output channel number of 1024 and a pooling kernel size of 5x5;

[0085] The activation functions in the convolution modules are all SiLU activation functions.

[0086] Preferably, the five feature maps are selected by using Gaussian mixture clustering, and the length and width of the defects to be detected in the image are clustered using a Gaussian mixture clustering model. The input is the length and width of the defects to be detected in the image relative to the original image, which is defect length / image length, defect width / image width. After clustering, all defects are divided into five categories according to the length and width relative to the original image, and the clustering result is mapped back to the original image. The cluster center point is matched with the receptive field of the output feature map of each module in the backbone network to obtain the feature map output by the corresponding module in the backbone network, and the cluster center point matches the receptive field size of the module layer where the selected feature map is located;

[0087] The parameters of the clustering model are estimated using the following algorithm, which consists of two steps:

[0088] Step E: Under given model parameters, calculate the posterior probability of each data point belonging to each Gaussian distribution;

[0089] Step M: Based on the posterior probability calculated in step E, update the parameters of each Gaussian distribution, including mean, covariance, and mixing coefficient, to maximize the likelihood function of the observed data;

[0090] These two steps are iterated continuously until the change of model parameters is less than the set threshold or the set number of iterations is reached, and the algorithm converges.

[0091] Specifically, the threshold is set to 1e-3 and the number of iterations is set to 100.

[0092] It should be noted that Gaussian mixture model clustering (GMM) is a clustering algorithm widely used in statistical learning and machine learning. It assumes that the samples in the data set are a mixture of multiple Gaussian distributions (also called normal distributions), and each Gaussian distribution corresponds to a hidden category or cluster. The goal of GMM is to find the parameters (mean, covariance, and mixing coefficient) of the Gaussian distribution that best represents the data, so as to achieve data clustering. GMM believes that each data point in the data set may be generated by one of these Gaussian distributions, and the probability that each data point belongs to a Gaussian distribution can be obtained by calculating its corresponding posterior probability. GMM iteratively optimizes the parameters of these Gaussian distributions so that the model can best fit the data.

[0093] Preferably, in S3, the feature fusion module for processing the extracted features includes a first feature fusion module to an eighth feature fusion module;

[0094] After the feature map Feat5 is input to the attention module, the attention module outputs the signal feature Feat after processing the coordinate attention and channel attention. 51, respectively as the first input feature of the first feature fusion module 1 and the second input feature of the eighth feature fusion module 8;

[0095] The feature map Feat4 is used as the second input feature of the first feature fusion module, and the first feature fusion module 1 performs fusion output signal feature Feat 41 , signal feature Feat 41 As the first input feature of the second feature fusion module 2 and the second input feature of the seventh feature fusion module 7;

[0096] The feature map Feat3 is used as the second input feature of the second feature fusion module 2, and the second feature fusion module 2 performs feature fusion and outputs the signal feature Feat 31 As the first input feature of the third feature fusion module 3, and the second input feature of the sixth feature fusion module 6;

[0097] The feature map Feat2 is used as the second input feature of the third feature fusion module 3, and the third feature fusion module 3 performs feature fusion and outputs the signal feature Feat 21 As the first input feature of the fourth feature fusion module 4, and the second input feature of the fifth feature fusion module 5;

[0098] The feature map Feat1 is used as the second input feature of the fourth feature fusion module 4, and the feature fusion is performed to obtain the signal feature Feat 11 As the first input feature of the fifth feature fusion module 5;

[0099] The signal feature Feat output by the fifth feature fusion module 5 after feature fusion 22 As the first input feature of the sixth feature fusion module 6, the sixth feature fusion module 6 performs feature fusion and outputs a signal feature Feat 32 As the first input feature of the seventh feature fusion module 7, the seventh feature fusion module 7 performs feature fusion and outputs a signal feature Feat 42 As the first input feature of the eighth feature fusion module 8; the eighth feature fusion module 8 performs feature fusion and outputs a signal feature Feat 52 ;

[0100] The signal feature Feat 11 、Feat 22 、Feat 32 、Feat 42 、Feat 52 As input to the detector module.

[0101] Preferably, if Figure 3As shown in S3, the attention module is specifically as follows: first, the channel is converted to 2 times through a convolution module with a convolution kernel size of 1x1 and a stride of 1, and then the feature map is divided into two parts from the channel dimension. The first part is retained, and the second part is processed by the coordinate attention submodule and the channel attention submodule. The feature map obtained after processing and the feature map before processing are added point by point using the addition layer; after the coordinate attention and channel attention processing are completed, the obtained feature map and the feature map retained in the first part are connected in the channel dimension, and then a convolution module with a convolution kernel size of 1x1 and a stride of 1 is used for feature fusion; the highest semantic feature Feat5 obtained by feature extraction is input into the attention module for processing, and the feature map Feat is output. 51 .

[0102] Preferably, if Figure 4 and Figure 5 As shown, the coordinate attention submodule and the channel attention submodule are specifically:

[0103] Coordinate attention submodule: Through one-dimensional pooling in the horizontal and vertical directions, the corresponding feature maps containing horizontal and vertical information are obtained. After convolution and activation, the horizontal attention feature map and the vertical attention feature map are obtained. Then, the two attention feature maps are applied to the input feature map by multiplication to obtain the output feature map.

[0104] Channel attention submodule: Through one-dimensional pooling in the channel direction, the corresponding feature map containing channel direction information is obtained. After convolution and activation, the horizontal attention feature map and the vertical attention feature map are obtained. Then, both attention feature maps are applied to the input feature map by multiplication to obtain the output feature map.

[0105] It should be noted that the two attention feature maps are then applied to the input feature map by multiplication: the sizes of the two attention maps are b×c×h×1 and b×c×1×w, respectively, which are repeated w times in the fourth dimension and h times in the third dimension, respectively, to obtain two maps of size b×c×h×w, which are dot-multiplied with the feature map, and the processed feature map is output.

[0106] Specifically, the pooling, convolution and activation of the coordinate attention submodule and the channel attention submodule are as follows:

[0107] The pooling of the coordinate attention submodule is to aggregate the input feature map into four separate feature maps containing horizontal and vertical direction information along the vertical and horizontal directions respectively through one-dimensional average pooling and one-dimensional maximum pooling operations; the pooling of the channel attention submodule is to aggregate the input feature map into two separate feature maps containing channel direction information along the channel direction respectively through one-dimensional average pooling and one-dimensional maximum pooling operations;

[0108] The convolution and activation processes of the coordinate attention submodule and the channel attention submodule both use a convolution layer, a Relu activation layer, and a convolution layer in sequence to encode two feature maps with embedded horizontal information and two feature maps with embedded vertical information, respectively, and add the two feature maps in the horizontal direction after encoding point by point, and add the two feature maps in the vertical direction after encoding point by point, and perform Sigmoid activation on the two added feature maps to obtain the horizontal attention feature map and the vertical attention feature map; the convolution kernel size of the above-mentioned convolution layer is 1x1, and the step size is 1, but the parameters of the convolution layer used by the coordinate attention submodule and the channel attention submodule are different.

[0109] Specifically, the pooling uses adaptive pooling. When performing one-dimensional pooling in the horizontal direction, the pooling parameters are set to (1, None). The pooling size and compensation size are adaptively determined according to the feature map size. When the feature map size is b×c×h×w, the pooling size and step size are both (1, w), and the edge padding is 0; when performing one-dimensional pooling in the vertical direction, the pooling parameters are set to (None, 1). The pooling size and compensation size are adaptively determined according to the feature map size. When the feature map size is b×c×h×w, the pooling size and step size are both (h, 1), and the edge padding is 0.

[0110] Preferably, if Figure 6 As shown, in S3, the feature fusion module specifically includes:

[0111] First feature fusion module 1: The first input feature and the second input feature of the same size are connected in the channel dimension using a connection layer, and then a convolution module with a convolution kernel size of 1x1 and a stride of 1, a grouped convolution module with a convolution kernel size of 3x3, a stride of 1 and the number of groups equal to the number of channels, and a convolution module with a convolution kernel size of 1x1 and a stride of 1 are used in sequence to fuse the features;

[0112] The second feature fusion module 2, the third feature fusion module 3, and the fourth feature fusion module 4 have the same structure. The spatial size of the first input feature is smaller than the spatial size of the second input feature. The upsampling layer is used to upsample the first input feature to the same spatial size as the second input feature. Then, the connection layer is used to connect the upsampled first input feature and the second input feature in the channel dimension. Finally, a convolution module with a convolution kernel size of 1x1 and a step size of 1, a grouped convolution module with a convolution kernel size of 3x3, a step size of 1 and a group number equal to the number of channels, and a convolution module with a convolution kernel size of 1x1 and a step size of 1 are used in sequence to fuse the features.

[0113] The fifth feature fusion module 5, the sixth feature fusion module 6, and the seventh feature fusion module 7 have the same structure; the spatial size of the first input feature is larger than the spatial size of the second input feature, and the first input feature is downsampled to the same spatial size as the second input feature using a downsampling layer, and then the upsampled first input feature and the second input feature are connected in the channel dimension using a connection layer, and finally, a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 are used in turn to fuse the features;

[0114] The eighth feature fusion module 8 first uses a connection layer to connect the first input feature and the second input feature of the same spatial size in the channel dimension, and finally uses a convolution module with a convolution kernel size of 1x1 and a step size of 1, and a convolution module with a convolution kernel size of 3x3 and a step size of 1 to fuse the features.

[0115] Specifically, the upsampling process of feature module 2 is as follows:

[0116] The input feature map size is w×h, and the expected output feature map size is 2w×2h; for each position (i', j') in the output feature map, we first map the coordinates, and determine the corresponding position of (i', j') in the original feature map according to the upsampling ratio, to prevent (i', j') from not falling directly on the integer position of the original feature map, and map it to the nearest integer coordinate. The calculation formula is:

[0117]

[0118] I' out (i',j')=I in (i,j)

[0119] In the original feature map, I in (i,j) represents the pixel value at coordinate (i,j) in the input feature map, and I' in the output feature map out (i', j') represents the pixel value at the coordinate (i', j') on the output feature map; repeat the above steps until all target positions in the output feature map are filled.

[0120] The specific process of downsampling described in feature module 3 is as follows:

[0121]

[0122] Among them, I' in (i, j) represents the pixel value at the input feature map coordinate (i, j), I' out(i', j') represents the pixel value at the coordinate (i', j') on the output feature map; repeat the above steps until all target positions in the output feature map are filled.

[0123] Preferably, in S3, the detector module is specifically as follows: the detector module uses an anchored detection head for detection, which is divided into target position prediction and target category prediction; the target position prediction predicts the distances lt and rb from the upper left corner and lower right corner of the target to the center point of each feature position on the feature map in the original image, and the target category prediction predicts the target category closest to each feature position on the feature map.

[0124] Preferably, the specific process of target detection is: the feature map Feat obtained by the feature fusion module 11 、Feat 22 、Feat 32 、Feat 42 and Feat 52 , both use the convolution module with a convolution kernel size of 3x3, a step size of 1, and a channel number equal to the number of categories, and the convolution module with a convolution kernel size of 3x3, a step size of 1, and a channel number of 16x4 to perform convolution operations, and obtain two sets of feature maps, specifically: Feature map Feat for category prediction 1,cls 、Feat 2,cls 、Feat 3,cls 、Feat 4,cls and Feat 5,cls , and the feature map Feat of position prediction 1,pos 、Feat 2,pos 、Feat 3,pos 、Feat 4,pos and Feat 5,pos ; The feature map Feat predicted by the category 1,cls 、Feat 2,cls 、Feat 3,cls 、Feat 4,cls and Feat 5,cls Flatten the spatial dimension from two dimensions to one dimension, and concatenate in the last dimension to obtain the feature map Feat cls ; Similarly, the feature map Feat predicted by the position 1,pos 、Feat 2,pos 、Feat 3,pos 、Feat 4,pos and Feat 5,pos Flatten the spatial dimension from two dimensions to one dimension, and connect it in the last dimension to get the feature map Feat' pos ; Then the feature map Feat' pos The channel dimension is divided into 16 groups, and the weighted sum of the 16 groups of coordinate positions is obtained to obtain the feature map Featpos , the weights from the first group to the last group are 1 to 16 respectively; then the predictions in lt and rb formats are converted to the center point length and width format. The feature map is a four-channel feature map, and each data is left, top, right, and bottom. Left and top can determine the upper left corner of the target, and right and bottom can determine the lower right corner of the target, as the output of the target position prediction, and Feat cls Post-processing is performed together to finally obtain the target detection result, including the location of the target and its corresponding category.

[0125] Preferably, the post-processing is to filter out the detection results with confidence scores less than the threshold according to the confidence threshold (0.001), and perform non-maximum suppression on the remaining results, and the iou threshold is 0.5.

[0126] S4, determine whether the data collection is completed. If the collection is completed, generate statistical information on the defect detection results of the power equipment to be detected, including: basic information of the edge device settings, all defect categories and locations that the model can identify, and the number of defects detected in each category. Otherwise, return to step S2.

[0127] Embodiment 2 of the present invention provides a power image target detection system based on feature fusion and attention. Using the method described in Embodiment 1, the power image target detection system includes an image acquisition module, an image preprocessing module, a defect detection module and a result statistics module:

[0128] An image acquisition module, which acquires images of the power equipment to be tested;

[0129] Image preprocessing module, adjusts the size and format of the image;

[0130] Defect detection module, which uses a preset feature fusion and attention-based object detection model for defect detection;

[0131] The result statistics module records the defect detection results and determines whether all data has been processed. If so, it generates defect detection result statistics.

[0132] Embodiment 3 of the present invention proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the power image target detection method based on feature fusion and attention as described in Embodiment 1 of the present invention.

[0133] Embodiment 4 of the present invention proposes a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the power image target detection method based on feature fusion and attention described in Embodiment 1 of the present invention is used.

[0134] The beneficial effect of the present invention is that, compared with the prior art,

[0135] This patent proposes a method for power image target detection based on feature fusion and attention. This method improves the detection performance on power data sets with large target size differences through feature fusion and attention. According to the clustering results, a suitable feature map is selected, and then the combination of the designed four feature fusion modules is used to perform adaptive feature fusion, which is more suitable for processing data with large target size differences and adapting to different situations. The modular design is adopted to facilitate the continuous update and addition of new components in the later stage. The attention module is used to embed the position information into the attention map so that the model can obtain information of a larger area and enhance the expression ability of the learned features. The combination of coordinate attention and channel attention is used in the attention module to extract attention from two dimensions: space and channel. Coordinate attention is used to consider the contextual information of the spatial coordinates in the input feature map, so as to enhance the network's sensitivity to spatial position. Channel attention is used to consider the relative importance between different feature channels, and use spatial position information to guide the network to learn richer spatial feature representations.

[0136] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0137] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0138] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A power image target detection method based on feature fusion and attention, characterized in that: The following steps are involved: S1, setting basic information for the edge device, where the basic information includes the device to which the power image belongs, the power image acquisition range, and the maximum input space size of the image that the edge device can receive; S2, collecting images of the power equipment to be inspected, and the edge device preprocesses the collected images to be inspected; S3, the edge device uses a preset target detection model based on feature fusion and attention to perform defect detection on the preprocessed image to be detected, wherein the target detection model includes a backbone network for feature extraction, an attention module and a feature fusion module for processing the extracted features, and a detector module for realizing target detection; S4, determine whether the data collection is completed. If the collection is completed, generate statistical information on the defect detection results of the power equipment to be detected, including: basic information of the edge device settings, all defect categories and locations that the model can identify, and the number of defects detected in each category. Otherwise, return to step S2.

2. The method for power image target detection based on feature fusion and attention according to claim 1, characterized in that: In S2, the specific contents of image preprocessing are: Adjust the image size of the image to be detected to the maximum input space size (H, W) that the edge device can receive during the inference process, and normalize the image pixel value from the [0, 255] interval to the [0, 1] interval; The image format of the image to be detected is adjusted and converted to the (N, C, H, W) format, where N represents the batch size, which is set to 1; C represents the number of channels in an image, which is 3, representing R, G, and B respectively; H represents the number of pixels in the vertical dimension of the image; and W represents the number of pixels in the horizontal dimension of the image.

3. The method for power image target detection based on feature fusion and attention according to claim 2, characterized in that: In S3, the backbone network is as follows: The backbone network includes 6 convolution modules, 5 CSP bottleneck modules and 1 SPPF module; the preprocessed image passes through the first convolution module, the second convolution module, the first CSP bottleneck module, the third convolution module, the second CSP bottleneck module, the fourth convolution module, the third CSP bottleneck module, the fifth convolution module, the fourth CSP bottleneck module, and the first SPPF module in sequence. Each convolution module contains a convolution layer, batch normalization and activation function; During the operation, the feature map Feat1 obtained from the second CSP bottleneck module, the feature map Feat2 obtained from the fourth convolution module, the feature map Feat3 obtained from the fourth CSP bottleneck module, the feature map Feat4 obtained from the fifth CSP bottleneck module and the feature map Feat5 obtained from the SPPF module are obtained; The spatial size of the feature map Feat1 is The feature map Feat2 space size is The feature map Feat3 space size is The feature map Feat4 space size is The feature map Feat5 space size is 4. The method for detecting power image targets based on feature fusion and attention according to claim 3, characterized in that: The feature map is selected by using Gaussian mixture clustering. The Gaussian mixture clustering model is used to cluster the length and width of the defects to be detected in the image. The input is the length and width of the defects to be detected in the image relative to the original image, which is defect length / image length, defect width / image width. After clustering, all defects are divided into five categories according to the length and width relative to the original image. The clustering result is mapped back to the original image. The cluster center point is matched with the receptive field of the output feature map of each module in the backbone network to obtain the feature map output by the corresponding module in the backbone network. The cluster center point matches the receptive field size of the module layer where the selected feature map is located. The parameter estimation of the clustering model includes two steps: Step E: Under given model parameters, calculate the posterior probability of each data point belonging to each Gaussian distribution; Step M: Based on the posterior probability calculated in step E, update the parameters of each Gaussian distribution, including mean, covariance, and mixing coefficient, to maximize the likelihood function of the observed data; The E-step-M-step is continuously iterated until the change of the model parameters is less than the set threshold or the set number of iterations is reached, and the algorithm converges.

5. The method for power image target detection based on feature fusion and attention according to claim 3, characterized in that: In S3, the feature fusion module for processing the extracted features includes a first feature fusion module to an eighth feature fusion module; After the feature map Feat5 is input to the attention module, the attention module outputs the signal feature Feat after processing the coordinate attention and channel attention. 51 , respectively as the first input feature of the first feature fusion module (1) and the second input feature of the eighth feature fusion module (8); The feature map Feat4 is used as the second input feature of the first feature fusion module. The first feature fusion module (1) performs fusion output signal feature Feat 41 , signal feature Feat 41 as the first input feature of the second feature fusion module (2) and the second input feature of the seventh feature fusion module (7); The feature map Feat3 is used as the second input feature of the second feature fusion module (2). The second feature fusion module (2) performs feature fusion and outputs a signal feature Feat 31 as the first input feature of the third feature fusion module (3), and the second input feature of the sixth feature fusion module (6); The feature map Feat2 is used as the second input feature of the third feature fusion module (3). The third feature fusion module (3) performs feature fusion and outputs a signal feature Feat 21 as the first input feature of the fourth feature fusion module (4), and the second input feature of the fifth feature fusion module (5); The feature map Feat1 is used as the second input feature of the fourth feature fusion module (4), and feature fusion is performed to obtain the signal feature Feat 11 as the first input feature of the fifth feature fusion module (5); The fifth feature fusion module (5) performs feature fusion and outputs the signal feature Feat 22 As the first input feature of the sixth feature fusion module (6), the sixth feature fusion module (6) performs feature fusion and outputs a signal feature Feat 32 As the first input feature of the seventh feature fusion module (7), the seventh feature fusion module (7) performs feature fusion and outputs a signal feature Feat 42 as the first input feature of the eighth feature fusion module (8); The eighth feature fusion module (8) performs feature fusion and outputs signal features Feat 52 ; The signal feature Feat 11 、Feat 22 、Feat 32 、Feat 42 、Feat 52 As input to the detector module.

6. The method for power image target detection based on feature fusion and attention according to claim 5, characterized in that: The attention module first converts the channel into two times through a convolution module with a convolution kernel size of 1x1 and a stride of 1, and then divides the input feature map into two parts in the channel dimension. The first part is retained, and the second part is processed by the coordinate attention submodule and the channel attention submodule. The feature map obtained after processing and the feature map before processing are added point by point using the addition layer; After the coordinate attention and channel attention processing is completed, the obtained feature map is connected with the feature map retained in the first part in the channel dimension, and then the features are fused through a convolution module with a convolution kernel size of 1x1 and a step size of 1.

7. The method for power image target detection based on feature fusion and attention according to claim 6, characterized in that: The coordinate attention submodule and channel attention submodule are specifically: Coordinate attention submodule: Through one-dimensional pooling in the horizontal and vertical directions, the corresponding feature maps containing horizontal and vertical information are obtained. After convolution and activation, the horizontal attention feature map and the vertical attention feature map are obtained. Then, the two attention feature maps are applied to the input feature map by multiplication to obtain the output feature map. Channel attention submodule: Through one-dimensional pooling in the channel direction, the corresponding feature map containing channel direction information is obtained. After convolution and activation, the horizontal attention feature map and the vertical attention feature map are obtained. Then, both attention feature maps are applied to the input feature map by multiplication to obtain the output feature map.

8. A method for power image target detection based on feature fusion and attention according to claim 5 or 6, characterized in that: First feature fusion module (1): The first input feature and the second input feature of the same size are connected in the channel dimension using a connection layer, and then a convolution module with a convolution kernel size of 1x1 and a stride of 1, a grouped convolution module with a convolution kernel size of 3x3, a stride of 1 and a group number equal to the number of channels, and a convolution module with a convolution kernel size of 1x1 and a stride of 1 are used in sequence to fuse the features; The second feature fusion module (2), the third feature fusion module (3), and the fourth feature fusion module (4) have the same structure. The spatial size of the first input feature is smaller than the spatial size of the second input feature. The upsampling layer is used to upsample the first input feature to the same spatial size as the second input feature. Then, the connection layer is used to connect the upsampled first input feature and the second input feature in the channel dimension. Finally, a convolution module with a convolution kernel size of 1x1 and a step size of 1, a grouped convolution module with a convolution kernel size of 3x3, a step size of 1 and a group number equal to the number of channels, and a convolution module with a convolution kernel size of 1x1 and a step size of 1 are used in sequence to fuse the features. The fifth feature fusion module (5), the sixth feature fusion module (6), and the seventh feature fusion module (7) have the same structure; the spatial size of the first input feature is larger than the spatial size of the second input feature, and the first input feature is downsampled to the same spatial size as the second input feature using a downsampling layer, and then the upsampled first input feature and the second input feature are connected in the channel dimension using a connection layer, and finally, a convolution module with a convolution kernel size of 1x1 and a step size of 1 and a convolution module with a convolution kernel size of 3x3 and a step size of 1 are used in sequence to fuse the features; The eighth feature fusion module (8) first uses a connection layer to connect the first input feature and the second input feature of the same spatial size in the channel dimension, and finally uses a convolution module with a convolution kernel size of 1x1 and a stride of 1, and a convolution module with a convolution kernel size of 3x3 and a stride of 1 to fuse the features.

9. The method for power image target detection based on feature fusion and attention according to claim 8, characterized in that: The specific process of target detection is as follows: the feature map Feat obtained by the feature fusion module 11 、Feat 22 、Feat 32 、Feat 42 and Feat 52 , both use the convolution module with a convolution kernel size of 3x3, a step size of 1, and a channel number equal to the number of categories, and the convolution module with a convolution kernel size of 3x3, a step size of 1, and a channel number of 16x4 to perform convolution operations, and obtain two sets of feature maps, specifically: Feature map Feat for category prediction 1,cls 、Feat 2,cls 、Feat 3,cls 、Feat 4,cls and Feat 5,cls , and the feature map Feat of position prediction 1,pos 、Feat 2,pos 、Feat 3,pos 、Feat 4,pos and Feat 5,pos ; The feature maps of category prediction and position prediction are flattened from two dimensions to one dimension in the spatial dimension, and then concatenated in the last dimension to obtain the feature map Feat cls And the feature map Feat' pos ; Feature map Feat cls Output for target category prediction; The feature map Feat' pos The channel dimension is divided into 16 groups, and the weighted sum of the 16 groups of coordinate positions is obtained to obtain the feature map Feat pos , the weights from the first group to the last group are 1 to 16 respectively; the feature map is the output of the target position prediction, and then the prediction in lt and rb format is converted into the center point length and width format, and Feat cls Post-processing is performed together to finally obtain the target detection result.

10. The method for power image target detection based on feature fusion and attention according to claim 9, characterized in that: The post-processing is to filter out the detection results with confidence scores less than the threshold according to the confidence threshold, the confidence threshold is 0.001, and perform non-maximum suppression on the remaining results, and the iou threshold is 0.

5.

11. A power image target detection system based on feature fusion and attention, using the method described in claims 1-10, the power image target detection system comprises an image acquisition module, an image preprocessing module, a defect detection module and a result statistics module; characterized in that: An image acquisition module, which acquires images of the power equipment to be tested; Image preprocessing module, adjusts the size and format of the image; Defect detection module, which uses a preset feature fusion and attention-based object detection model for defect detection; The result statistics module records the defect detection results and determines whether all data has been processed. If so, it generates defect detection result statistics.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the power image target detection method based on feature fusion and attention as described in any one of claims 1 to 10.

13. A readable storage medium storing a computer program, wherein the computer program, when executed by a processor, uses the power image target detection method based on feature fusion and attention as described in any one of claims 1 to 10.

Images