Method, system, device and medium for detecting power facilities in unmanned aerial vehicle image

Abstract

The application belongs to the field of electric power automation, and discloses a kind of unmanned aerial vehicle image electric power facility detection method, system, equipment and medium, comprising: obtaining the unmanned aerial vehicle image to be detected;The unmanned aerial vehicle image is input into the preset pre-training high-precision electric power facility target detection model, and the target anchor frame of each electric power facility in the unmanned aerial vehicle image is labeled by the pre-training high-precision electric power facility target detection model, and the position information and classification result of the target anchor frame of each electric power facility are obtained;Wherein, the classification result is the type of electric power facility.Based on the high-performance target detection capability of the pre-training high-precision electric power facility target detection model, high-precision detection of various electric power facilities such as high-voltage towers, photovoltaic panels, oil storage tanks and chimneys is realized, the efficiency and accuracy of electric power facility detection are improved, which can help staff make correct decisions, and has important significance for the maintenance and management of electric power facilities.

Description

Technical Field

[0001] This invention belongs to the field of power automation and relates to a method, system, equipment and medium for detecting power facilities in UAV images. Background Technology

[0002] The detection of power facilities plays a crucial role in maintaining the safety of the power grid system. Large power facilities such as high-voltage towers, photovoltaic panels, oil tanks, and chimneys exhibit high spatial complexity, requiring extensive data acquisition equipment, such as drones, to obtain their source images. Therefore, research on the detection of power facility targets in drone imagery is of great significance. However, unlike natural scenes, the background of power facilities is complex, causing irrelevant background features to severely interfere with the feature information of the target objects. Furthermore, some power facilities are densely packed and small in size, making it difficult for drone imagery to effectively capture the feature distribution of such facilities. Simultaneously, the orientation of power facilities is arbitrary, further complicating the detection of power facilities in drone imagery.

[0003] In recent years, with the support of massive high-resolution image data and deep learning, a number of target detection technologies applied to images have emerged. In particular, target detection based on deep learning technology has developed rapidly. For example, target detection algorithms based on convolutional neural networks have been well applied in UAV imagery. However, due to the complexity of power facilities, such as high-voltage towers, photovoltaic panels, oil tanks and chimneys, there are serious background interference and densely arranged image problems. As a result, the efficiency and accuracy of existing ordinary convolutional neural networks in target detection of power facilities in UAV imagery are relatively low. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system, equipment and medium for detecting power facilities in UAV images.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] In a first aspect, the present invention provides a method for detecting power facilities in unmanned aerial vehicle (UAV) imagery, comprising:

[0007] Acquire images of the drone to be inspected;

[0008] The UAV imagery is input into a pre-trained high-precision power facility target detection model. The model is then used to annotate the target frames of each power facility in the UAV imagery and obtain the location information and classification results of the target frames of each power facility. The classification result is the type of power facility.

[0009] Optionally, the pre-trained high-precision power facility target detection model is obtained by training a high-precision power facility target detection model based on several labeled UAV images. The high-precision power facility target detection model includes a bidirectional feature fusion module, a feature extraction module, a regression refinement module, and a classification module.

[0010] The bidirectional feature fusion module is used to extract bidirectional fused multi-scale feature maps of UAV images through bidirectional fusion.

[0011] The extraction module is used to obtain classification task features and regression task features from bidirectional fused multi-scale feature maps through spatial attention network and channel attention network;

[0012] The regression refinement module is used to generate a regression task feature map based on the regression task features, and to use each feature point on the regression task feature map as each power facility and to label the initial anchor box of each power facility; and to iteratively optimize the initial anchor box of each power facility based on a preset regression loss function to obtain the target anchor box of each power facility and to obtain the location information of the target anchor box of each power facility.

[0013] The classification module is used to obtain the classification results of the target anchor frames for each power facility based on the characteristics of the classification task.

[0014] Optionally, the bidirectional feature fusion module includes a ResNet network, a top-down fusion network, and a bottom-up fusion network;

[0015] The top-down fusion network is as follows:

[0016]

[0017] The bottom-up convergence network is:

[0018]

[0019] Among them, C i (i∈3,4,5) represent the outputs of the ResNet network, specifically the outputs of the Conv3_x, Conv4_x, and Conv5_x layers, respectively; P i (i∈3,4,5,6,7) represents the multi-scale feature map output by the top-down fusion network, P i '(i∈3,4,5,6,7) represents the multi-scale feature map output by the bottom-up fusion network and serves as the bidirectional fusion multi-scale feature map; 1×1Conv represents a convolution operation with a stride of 1 and a kernel size of 1×1, and 3×3Conv represents a convolution operation with a stride of 1 and a kernel size of 3×3; 2×UpSample represents a 2x upsampling operation, and 2×DownSample represents a 2x downsampling operation.

[0020] Optionally, the step of obtaining classification task features and regression task features by means of a spatial attention network and a channel attention network to obtain bidirectional fused multi-scale feature maps includes:

[0021] The bidirectional fused multi-scale feature map is input into the spatial attention network to obtain the spatial attention response map;

[0022] The bidirectional fused multi-scale feature map is input into the channel attention network to obtain the channel attention response map;

[0023] The spatial attention response map and the channel attention response map are multiplied by tensors to obtain the tensor multiplication result;

[0024] The feature weights of the regression task are obtained by activating the tensor multiplication result through the regression task activation function;

[0025] The classification task feature weights are obtained by activating the tensor multiplication result using the classification task activation function;

[0026] The regression task feature weights are element-wise multiplied with the bidirectional fusion multi-scale feature map to obtain the first element-wise multiplication result, which is then summed with the residual of the bidirectional fusion multi-scale feature map to obtain the regression task features.

[0027] The classification task feature weights are element-wise multiplied with the bidirectional fusion multi-scale feature map to obtain the second element-wise multiplication result, which is then summed with the residual of the bidirectional fusion multi-scale feature map to obtain the classification task features.

[0028] The spatial attention network is as follows:

[0029] W ci =Sig(FC2(FC1(GAP(P) i '))))

[0030] Where GAP represents global average pooling, FC1 and FC2 represent fully connected layers, Sig represents the Sigmoid function, and W... ci Represents spatial attention response map;

[0031] The channel attention network is as follows:

[0032]

[0033] Among them, Conv 1×1 Represents a convolution with a kernel size of 1×1, Conv 3×3 This represents a convolution with a kernel size of 3×3. and These represent convolutions with a kernel size of 3×3 and dilatations of 1, 3, 5, and 7, respectively. `cat` represents the channel concatenation operation, and `W` represents...si Representative channel attention response diagram;

[0034] Regression task activation function f reg (x) is:

[0035] f reg (x) = 4x(1-x)

[0036] Classification task activation function f cls (x) is:

[0037]

[0038] Here, x represents the activation variable.

[0039] Optionally, the regression refinement module is specifically used for:

[0040] S101: Concatenate the regression task features along the channel dimension to obtain the regression task feature map;

[0041] S102: Treat each feature point in the feature map of the regression task as a power facility, and predefine a horizontal anchor box on each feature point as the initial anchor box for each power facility.

[0042] S103: Iteratively optimize the initial anchor frame of each power facility based on the preset regression loss function, and perform S104 and S105 once in each iteration until the preset number of iterations is reached, to obtain the target anchor frame of each power facility and obtain the position information of the target anchor frame of each power facility.

[0043] S104: Perform four convolution operations on the feature map of the regression task with a stride of 1 and a kernel size of 3×3, while keeping the number of feature channels unchanged;

[0044] S105: Perform a convolution operation with a stride of 1 and a kernel size of 3×3 on the regression task feature map output in step S104, and change the number of feature channels to 5.

[0045] Optionally, the iterative optimization of the initial anchor frame for each power facility based on a preset regression loss function includes:

[0046] For the nth optimization: the anchor frame offset of the initial anchor frame of each power facility is obtained based on the preset regression loss function; and based on the anchor frame offset of the initial anchor frame of each power facility, the initial anchor frame of each power facility after the nth optimization is obtained by the following formula:

[0047]

[0048] Where, x n y n wn h n and θ n These represent the x-axis coordinates, y-axis coordinates, width, height, and angle of the initial anchor frame after the nth optimization, respectively, where n is a preset constant. and These represent the x-axis offset, y-axis offset, width offset, height offset, and angle offset of the initial anchor frame center point during the nth optimization, respectively.

[0049] Optionally, the regression loss function is:

[0050]

[0051] Where N represents the initial number of anchor frames. Represents the future, Represents background, v represents the offset of the predicted bounding box relative to the preset anchor box. nj The value represents the offset of the ground truth bounding box relative to the preset anchor box; the IoU represents the intersection-over-union ratio of the predicted bounding box and the ground truth bounding box; x, y, w, h and θ represent the x-axis coordinate, y-axis coordinate, width, height and angle of the center point of the initial anchor box, respectively.

[0052] The loss function of the classification module is:

[0053]

[0054] Among them, L Focal (·) represents the focus loss function, t i p represents the classification result of target anchor box i. i The probability distribution of different classification results predicted for target anchor box i.

[0055] In a second aspect, the present invention provides a system for detecting power facilities in UAV imagery, comprising:

[0056] The data acquisition module is used to acquire images of the drone to be detected;

[0057] The model processing module is used to input UAV imagery into a pre-trained high-precision power facility target detection model, annotate the target anchor boxes of each power facility in the UAV imagery using the pre-trained high-precision power facility target detection model, and obtain the location information and classification results of the target anchor boxes of each power facility; wherein, the classification result is the power facility type.

[0058] Optionally, the pre-trained high-precision power facility target detection model is obtained by training a high-precision power facility target detection model based on several labeled UAV images. The high-precision power facility target detection model includes a bidirectional feature fusion module, a feature extraction module, a regression refinement module, and a classification module.

[0059] The bidirectional feature fusion module is used to extract bidirectional fused multi-scale feature maps of UAV images through bidirectional fusion.

[0060] The extraction module is used to obtain classification task features and regression task features from bidirectional fused multi-scale feature maps through spatial attention network and channel attention network;

[0061] The regression refinement module is used to generate a regression task feature map based on the regression task features, and to use each feature point on the regression task feature map as each power facility and to label the initial anchor box of each power facility; and to iteratively optimize the initial anchor box of each power facility based on a preset regression loss function to obtain the target anchor box of each power facility and to obtain the location information of the target anchor box of each power facility.

[0062] The classification module is used to obtain the classification results of the target anchor frames for each power facility based on the characteristics of the classification task.

[0063] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for detecting power facilities in UAV imagery.

[0064] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for detecting power facilities in UAV imagery.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] This invention discloses a method for detecting power facilities in UAV imagery. The method involves inputting UAV imagery into a pre-trained high-precision power facility target detection model. Based on this model, target bounding boxes for each power facility in the UAV imagery are annotated. Then, the location information and classification results of each power facility's target bounding boxes are determined, thus identifying the location and type of each power facility. Applying a high-performance pre-trained high-precision power facility target detection model to power facility detection effectively extracts useful information from UAV imagery and specifically enhances the classification and location information within the imagery. Ultimately, this achieves accurate location and type identification of each power facility in the UAV imagery, significantly improving the efficiency and accuracy of identification. This helps personnel make correct decisions and is of great significance for the maintenance and management of power facilities. Attached Figure Description

[0067] Figure 1 This is a flowchart of a method for detecting power facilities in UAV images according to an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram illustrating the principle of the high-precision power facility target detection model according to an embodiment of the present invention.

[0069] Figure 3 This is a block diagram of the detection system for power facilities in UAV images according to an embodiment of the present invention. Detailed Implementation

[0070] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0071] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0072] The present invention will now be described in further detail with reference to the accompanying drawings:

[0073] See Figure 1 and 2 In one embodiment of the present invention, a method for detecting power facilities in UAV images is provided, which can effectively detect various power facilities, such as high-voltage towers, photovoltaic panels, oil storage tanks and chimneys, and can effectively improve the efficiency and accuracy of target detection, thereby helping staff to make correct decisions, which is of great significance for the maintenance and management of power facilities.

[0074] Specifically, the method for detecting power facilities in the drone imagery includes the following steps:

[0075] S1: Acquire images of the drone to be detected.

[0076] S2: Input the UAV imagery into the pre-trained high-precision power facility target detection model, annotate the target anchor boxes of each power facility in the UAV imagery using the pre-trained high-precision power facility target detection model, and obtain the location information and classification results of the target anchor boxes of each power facility, where the classification result is the power facility type.

[0077] This invention discloses a method for detecting power facilities in UAV imagery. The method involves inputting UAV imagery into a pre-trained high-precision power facility target detection model. Based on this model, target bounding boxes for each power facility in the UAV imagery are annotated. Then, the location information and classification results of each power facility's target bounding boxes are determined, thus identifying the location and type of each power facility. Applying a high-performance pre-trained high-precision power facility target detection model to power facility detection effectively extracts useful information from UAV imagery and specifically enhances the classification and location information within the imagery. Ultimately, this achieves accurate location and type identification of each power facility in the UAV imagery, significantly improving the efficiency and accuracy of identification. This helps personnel make correct decisions and is of great significance for the maintenance and management of power facilities.

[0078] In this embodiment, the pre-trained high-precision power facility target detection model is obtained by training a high-precision power facility target detection model based on several labeled UAV images. The high-precision power facility target detection model includes a bidirectional feature fusion module, a feature extraction module, a regression refinement module, and a classification module.

[0079] The bidirectional feature fusion module extracts bidirectional fused multi-scale feature maps from UAV imagery through bidirectional fusion. The extraction module uses spatial attention and channel attention networks to obtain classification and regression task features from the bidirectional fused multi-scale feature maps. The regression refinement module generates regression task feature maps based on the regression task features, uses each feature point on the regression task feature map as a power facility, and labels the initial anchor boxes of each power facility. It also iteratively optimizes the initial anchor boxes of each power facility based on a preset regression loss function to obtain the target anchor boxes of each power facility and acquire their location information. The classification module obtains the classification results of the target anchor boxes of each power facility based on the classification task features. Both the regression refinement and classification modules can be built using convolutional neural networks.

[0080] In one possible implementation, the bidirectional feature fusion module includes a ResNet network, a top-down fusion network, and a bottom-up fusion network.

[0081] The top-down fusion network is as follows:

[0082]

[0083] The bottom-up convergence network is:

[0084]

[0085] Among them, C i (i∈3,4,5) represent the outputs of the ResNet network, specifically the outputs of the Conv3_x, Conv4_x, and Conv5_x layers, respectively; P i (i∈3,4,5,6,7) represents the multi-scale feature map output by the top-down fusion network, P i '(i∈3,4,5,6,7) represents the multi-scale feature map output by the bottom-up fusion network and serves as the bidirectional fusion multi-scale feature map; 1×1Conv represents a convolution operation with a stride of 1 and a kernel size of 1×1, and 3×3Conv represents a convolution operation with a stride of 1 and a kernel size of 3×3; 2×UpSample represents a 2x upsampling operation, and 2×DownSample represents a 2x downsampling operation.

[0086] Specifically, the UAV imagery is first input into a ResNet network for processing, and feature maps of different scales output from the Conv3_x, Conv4_x, and Conv5_x layers of the ResNet network are recorded, denoted as feature map C3, feature map C4, and feature map C5, respectively. Then, feature maps C3, C4, and C5 are fused through two convolutional layers in the lower layers of the top-down fusion network to obtain multi-level feature maps P3, P4, and P5. Simultaneously, feature map C3 is fused into smaller multi-level feature maps P6 and P7 through two convolutional layers in the higher layers of the top-down fusion network. Finally, the multi-level feature maps P3, P4, P5, P6, and P7 are fused from the bottom up using a bottom-up fusion network to obtain bidirectional fused multi-level feature maps P3', P4', P5', P6', and P7', which are then combined as the bidirectional fused multi-scale feature map P. i (i∈3,4,5,6,7).

[0087] Optionally, in this embodiment, bilinear interpolation is used as the specific method for the double upsampling operation, and convolution operation with a stride of 2 and a kernel size of 3×3 is used as the specific method for the double downsampling operation.

[0088] In one possible implementation, obtaining classification task features and regression task features from a bidirectional fused multi-scale feature map using a spatial attention network and a channel attention network includes: inputting the bidirectional fused multi-scale feature map into a spatial attention network to obtain a spatial attention response map; inputting the bidirectional fused multi-scale feature map into a channel attention network to obtain a channel attention response map; performing tensor multiplication on the spatial attention response map and the channel attention response map to obtain a tensor multiplication result; activating the tensor multiplication result using a regression task activation function to obtain regression task feature weights; activating the tensor multiplication result using a classification task activation function to obtain classification task feature weights; performing element-wise multiplication of the regression task feature weights with the bidirectional fused multi-scale feature map to obtain a first element-wise multiplication result, and performing residual summation with the bidirectional fused multi-scale feature map to obtain regression task features; performing element-wise multiplication of the classification task feature weights with the bidirectional fused multi-scale feature map to obtain a second element-wise multiplication result, and performing residual summation with the bidirectional fused multi-scale feature map to obtain classification task features.

[0089] The spatial attention network is as follows:

[0090]

[0091] Where GAP represents global average pooling, FC1 and FC2 represent fully connected layers, and Sig represents the sigmoid function. This represents the spatial attention response map.

[0092] The channel attention network is as follows:

[0093]

[0094] Among them, Conv 1×1 Represents a convolution with a kernel size of 1×1, Conv 3×3 This represents a convolution with a kernel size of 3×3. and These represent convolutions with a kernel size of 3×3 and dilatations of 1, 3, 5, and 7, respectively. `cat` represents a channel concatenation operation. Representative channel attention response diagram.

[0095] Regression task activation function f reg (x) is:

[0096] f reg (x) = 4x(1-x)

[0097] Classification task activation function f cls (x) is:

[0098]

[0099] Here, x represents the activation variable, which in practical applications is the result of tensor multiplication.

[0100] Therefore, the process of generating features for the regression task can be represented as:

[0101]

[0102] The process of generating features for a classification task is represented as follows:

[0103]

[0104] in, ⊙ represents tensor multiplication, ⊙ represents element-wise multiplication, and F represents a bidirectional fusion of multi-scale feature maps of UAV imagery.

[0105] In one possible implementation, the regression refinement module generates a regression task feature map based on the regression task features generated above. Each feature point on the regression task feature map is used as a power facility, and the initial anchor frame of each power facility is marked. Then, the initial anchor frame of each power facility is iteratively optimized based on a preset regression loss function to achieve multi-level adjustment of the anchor frame parameters and maximize the spatial alignment between the anchor frame and the target.

[0106] Specifically, the following detailed steps are included:

[0107] S101: Regression task features REG i (i∈3,4,5,6,7) are concatenated along the channel dimension to obtain the regression task feature map REG.

[0108] S102: Treat each feature point in the regression task feature map REG as a power facility, and predefine a horizontal anchor box on each feature point as the initial anchor box for each power facility.

[0109] S103: Iteratively optimize the initial anchor frame of each power facility based on the preset regression loss function, and perform S104 and S105 once in each iteration until the preset number of iterations is reached, to obtain the target anchor frame of each power facility and obtain the position information of the target anchor frame of each power facility.

[0110] S104: Perform four convolution operations on the regression task feature map REG with a stride of 1 and a kernel size of 3×3, while keeping the number of feature channels unchanged.

[0111] S105: Perform a convolution operation with a stride of 1 and a kernel size of 3×3 on the regression task feature map REG output in step S104, and change the number of feature channels to 5.

[0112] Specifically, the regression refinement module optimizes the anchor frame offset based on the regression loss function, thereby controlling the change of the initial anchor frame in the next iteration. The iterative optimization of the initial anchor frame of each power facility based on the preset regression loss function includes: For the nth optimization: obtaining the anchor frame offset of the initial anchor frame of each power facility based on the preset regression loss function; and obtaining the initial anchor frame of each power facility after the nth optimization using the following formula based on the anchor frame offset of the initial anchor frame of each power facility:

[0113]

[0114] Where, x n y n w n h n and θ n These represent the x-axis coordinates, y-axis coordinates, width, height, and angle of the initial anchor frame after the nth optimization, respectively, where n is a preset constant. and They represent the first n The initial anchor frame's center point x-axis offset, center point y-axis offset, width offset, height offset, and angle offset are calculated during the second optimization.

[0115] Specifically, the basis for regression refinement is that after the regression refinement module learns the anchor frame offset once, it can calculate the optimized anchor frame position coordinates based on the current anchor frame position coordinates. Therefore, in this system of equations, the anchor frame offset of the nth optimization and the anchor frame parameters of the (n-1)th optimization are known quantities, and the optimized anchor frame parameters of the nth optimization can be calculated according to the equations.

[0116] In one possible implementation, in order to further improve the training efficiency of the high-precision power facility target detection model and the accuracy of the pre-trained high-precision power facility target detection model in detecting power facilities, corresponding loss functions are designed for the regression refinement module and the classification module to supervise the regression task process and the classification task process, respectively.

[0117] Specifically, the loss function for the regression refinement module is:

[0118]

[0119] Where N represents the initial number of anchor frames. Represents the future, Represents background, v represents the offset of the predicted bounding box relative to the preset anchor box. nj The offset of the true bounding box relative to the preset anchor box is represented by IoU, which represents the intersection-over-union ratio (IoU) between the predicted and true bounding boxes. x, y, w, h, and θ represent the x-axis coordinate, y-axis coordinate, width, height, and angle of the center point of the initial anchor box, respectively. Here, the predicted bounding box is the bounding box generated for each target when the high-precision power facility target detection model outputs the data; the preset anchor box is the bounding box manually set for each target when inputting UAV imagery; and the true bounding box is the actual bounding box of each target in the UAV imagery, i.e., the training label of the high-precision power facility target detection model. Based on this design, the loss function of the regression refinement module can effectively avoid the boundary problems associated with rotated anchor boxes.

[0120] The loss function of the classification module is:

[0121]

[0122] Among them, L Focal (·) represents the focus loss function, t i p represents the classification result of target anchor box i. i This represents the probability distribution of different classification results predicted for target anchor box i. Based on this design, the loss function of the classification module can solve the problem of imbalanced positive and negative samples during training.

[0123] In summary, the high-precision power facility target detection model of this invention enhances the feature representation capability of the network based on the bidirectional feature fusion module, enabling the features to contain more useful information; the feature extraction module specifically enhances the classification and localization information in the features, providing more refined feature guidance for the regression refinement module and the classification module; at the same time, the regression refinement module adaptively learns the rotating anchor frame, continuously reducing the offset between the anchor frame and the target, ultimately improving the detection accuracy of power facilities.

[0124] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0125] See Figure 3 In another embodiment of the present invention, a detection system for power facilities in UAV images is provided, which can be used to implement the above-mentioned detection method for power facilities in UAV images. Specifically, the detection system for power facilities in UAV images includes a data acquisition module and a model processing module.

[0126] The data acquisition module is used to acquire images of the UAV to be detected; the model processing module is used to input the UAV images into a pre-trained high-precision power facility target detection model, annotate the target anchor boxes of each power facility in the UAV images through the pre-trained high-precision power facility target detection model, and obtain the location information and classification results of the target anchor boxes of each power facility; wherein, the classification result is the power facility type.

[0127] In one possible implementation, the pre-trained high-precision power facility target detection model is obtained by training a high-precision power facility target detection model based on several labeled UAV images.

[0128] The high-precision power facility target detection model includes a bidirectional feature fusion module, a feature extraction module, a regression refinement module, and a classification module. The bidirectional feature fusion module is used to extract bidirectional fused multi-scale feature maps from UAV images through bidirectional fusion. The extraction module is used to obtain classification task features and regression task features of the bidirectional fused multi-scale feature maps through spatial attention networks and channel attention networks. The regression refinement module is used to generate regression task feature maps based on regression task features, and to use each feature point on the regression task feature map as a power facility and label the initial anchor boxes of each power facility. It also iteratively optimizes the initial anchor boxes of each power facility based on a preset regression loss function to obtain the target anchor boxes of each power facility and obtain the position information of the target anchor boxes of each power facility. The classification module is used to obtain the classification results of the target anchor boxes of each power facility based on the classification task features.

[0129] In one possible implementation, the bidirectional feature fusion module includes a ResNet network, a top-down fusion network, and a bottom-up fusion network.

[0130] The top-down fusion network is as follows:

[0131]

[0132] The bottom-up convergence network is:

[0133]

[0134] Among them, C i (i∈3,4,5) represent the outputs of the ResNet network, specifically the outputs of the Conv3_x, Conv4_x, and Conv5_x layers, respectively; P i (i∈3,4,5,6,7) represents the multi-scale feature map output by the top-down fusion network, P i '(i∈3,4,5,6,7) represents the multi-scale feature map output by the bottom-up fusion network and serves as the bidirectional fusion multi-scale feature map; 1×1Conv represents a convolution operation with a stride of 1 and a kernel size of 1×1, and 3×3Conv represents a convolution operation with a stride of 1 and a kernel size of 3×3; 2×UpSample represents a 2x upsampling operation, and 2×DownSample represents a 2x downsampling operation.

[0135] In one possible implementation, obtaining classification task features and regression task features from a bidirectional fused multi-scale feature map using a spatial attention network and a channel attention network includes: inputting the bidirectional fused multi-scale feature map into a spatial attention network to obtain a spatial attention response map; inputting the bidirectional fused multi-scale feature map into a channel attention network to obtain a channel attention response map; performing tensor multiplication on the spatial attention response map and the channel attention response map to obtain a tensor multiplication result; activating the tensor multiplication result using a regression task activation function to obtain regression task feature weights; activating the tensor multiplication result using a classification task activation function to obtain classification task feature weights; performing element-wise multiplication of the regression task feature weights with the bidirectional fused multi-scale feature map to obtain a first element-wise multiplication result, and performing residual summation with the bidirectional fused multi-scale feature map to obtain regression task features; performing element-wise multiplication of the classification task feature weights with the bidirectional fused multi-scale feature map to obtain a second element-wise multiplication result, and performing residual summation with the bidirectional fused multi-scale feature map to obtain classification task features.

[0136] The spatial attention network is as follows:

[0137]

[0138] Where GAP represents global average pooling, FC1 and FC2 represent fully connected layers, and Sig represents the sigmoid function. This represents the spatial attention response map.

[0139] The channel attention network is as follows:

[0140]

[0141] Among them, Conv 1×1 Represents a convolution with a kernel size of 1×1, Conv 3×3 This represents a convolution with a kernel size of 3×3. and These represent convolutions with a kernel size of 3×3 and dilatations of 1, 3, 5, and 7, respectively. `cat` represents a channel concatenation operation. Representative channel attention response diagram.

[0142] Regression task activation function f reg (x) is:

[0143] f reg (x) = 4x(1-x)

[0144] Classification task activation function f cls (x) is:

[0145]

[0146] Here, x represents the activation variable.

[0147] In one possible implementation, the regression refinement module is specifically used for:

[0148] S101: Concatenate the regression task features along the channel dimension to obtain the regression task feature map.

[0149] S102: Treat each feature point in the feature map of the regression task as a power facility, and predefine a horizontal anchor box on each feature point as the initial anchor box for each power facility.

[0150] S103: Iteratively optimize the initial anchor frame of each power facility based on the preset regression loss function, and perform S104 and S105 once in each iteration until the preset number of iterations is reached, to obtain the target anchor frame of each power facility and obtain the position information of the target anchor frame of each power facility.

[0151] S104: Perform four convolution operations on the feature map of the regression task with a stride of 1 and a kernel size of 3×3, while keeping the number of feature channels unchanged.

[0152] S105: Perform a convolution operation with a stride of 1 and a kernel size of 3×3 on the regression task feature map output in step S104, and change the number of feature channels to 5.

[0153] In one possible implementation, the iterative optimization of the initial anchor frame of each power facility based on a preset regression loss function includes: for the nth optimization: obtaining the anchor frame offset of the initial anchor frame of each power facility based on the preset regression loss function; and obtaining the initial anchor frame of each power facility after the nth optimization based on the anchor frame offset of the initial anchor frame of each power facility using the following formula:

[0154]

[0155] Where, x n y n w n h n and θ n These represent the x-axis coordinates, y-axis coordinates, width, height, and angle of the initial anchor frame after the nth optimization, respectively, where n is a preset constant. and These represent the x-axis offset, y-axis offset, width offset, height offset, and angle offset of the initial anchor frame center point during the nth optimization, respectively.

[0156] In one possible implementation, when training the high-precision power facility target detection model, the loss function of the regression refinement module is:

[0157]

[0158] Where N represents the initial number of anchor frames. Represents the future, Represents background, v represents the offset of the predicted bounding box relative to the preset anchor box. nj The offset of the true bounding box relative to the preset anchor box is represented by IoU, which represents the intersection-union ratio between the predicted bounding box and the true bounding box.

[0159] The loss function of the classification module is:

[0160]

[0161] Among them, L Focal (·) represents the focus loss function, t i Let pi represent the classification result of target anchor box i, where pi represents the probability distribution of different classification results predicted for target anchor box i.

[0162] All relevant content of each step involved in the aforementioned embodiments of the method for detecting power facilities in UAV images can be referenced to the functional description of the corresponding functional module of the UAV image detection system in the embodiments of the present invention, and will not be repeated here.

[0163] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0164] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a method for detecting power facilities in UAV imagery.

[0165] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for detecting power facilities in UAV imagery in the above embodiments.

[0166] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting power facilities in UAV imagery, characterized in that, include: Acquire images of the drone to be inspected; The UAV imagery is input into a pre-trained high-precision power facility target detection model. The model is then used to annotate the target bounding boxes of each power facility in the UAV imagery and obtain the location information and classification results of the target bounding boxes of each power facility. The classification result is the type of power facility. The pre-trained high-precision power facility target detection model is obtained by training a high-precision power facility target detection model based on several labeled UAV images. The high-precision power facility target detection model includes a bidirectional feature fusion module, a feature extraction module, a regression refinement module, and a classification module. The bidirectional feature fusion module is used to extract bidirectional fused multi-scale feature maps of UAV images through bidirectional fusion. The extraction module is used to obtain classification task features and regression task features from bidirectional fused multi-scale feature maps through spatial attention network and channel attention network; The regression refinement module is used to generate a regression task feature map based on the regression task features, and to use each feature point on the regression task feature map as each power facility and to label the initial anchor box of each power facility; and to iteratively optimize the initial anchor box of each power facility based on a preset regression loss function to obtain the target anchor box of each power facility and to obtain the location information of the target anchor box of each power facility. The classification module is used to obtain the classification results of the target anchor frames of each power facility based on the characteristics of the classification task; The bidirectional feature fusion module includes a ResNet network, a top-down fusion network, and a bottom-up fusion network; The top-down fusion network is as follows: The bottom-up convergence network is: in, These represent the outputs of the ResNet network, specifically the outputs of the Conv3_x, Conv4_x, and Conv5_x layers of the ResNet network. This represents the multi-scale feature map output from the top-down fusion network. This represents the multi-scale feature map output from the bottom-up fusion network and serves as the bidirectional fusion multi-scale feature map. This represents a convolution operation with a stride of 1 and a kernel size of 1×1. This represents a convolution operation with a stride of 1 and a kernel size of 3×3. This represents a double upsampling operation. This represents a double downsampling operation; The process of obtaining classification task features and regression task features by using spatial attention networks and channel attention networks to fuse bidirectional multi-scale feature maps includes: The bidirectional fused multi-scale feature map is input into the spatial attention network to obtain the spatial attention response map; The bidirectional fused multi-scale feature map is input into the channel attention network to obtain the channel attention response map; The spatial attention response map and the channel attention response map are multiplied by tensors to obtain the tensor multiplication result; The feature weights of the regression task are obtained by activating the tensor multiplication result through the regression task activation function; The classification task feature weights are obtained by activating the tensor multiplication result using the classification task activation function; The regression task feature weights are element-wise multiplied with the bidirectional fusion multi-scale feature map to obtain the first element-wise multiplication result, which is then summed with the residual of the bidirectional fusion multi-scale feature map to obtain the regression task features. The classification task feature weights are element-wise multiplied with the bidirectional fusion multi-scale feature map to obtain the second element-wise multiplication result, which is then summed with the residual of the bidirectional fusion multi-scale feature map to obtain the classification task features. The spatial attention network is as follows: in, Represents global average pooling. and Represents a fully connected layer. Represents the Sigmoid function. Represents spatial attention response map; The channel attention network is as follows: in, This represents a convolution with a kernel size of 1×1. This represents a convolution with a kernel size of 3×3. , , and These represent convolutions with a kernel size of 3×3 and dilatations of 1, 3, 5, and 7, respectively. `cat` represents a channel concatenation operation. Representative channel attention response graph; Regression task activation function for: Classification task activation function for: in, x This represents the activation variable.

2. The method for detecting power facilities in UAV imagery according to claim 1, characterized in that, The regression refinement module is specifically used for: S101: Concatenate the regression task features along the channel dimension to obtain the regression task feature map; S102: Treat each feature point in the feature map of the regression task as a power facility, and predefine a horizontal anchor box on each feature point as the initial anchor box for each power facility. S103: Iteratively optimize the initial anchor frame of each power facility based on the preset regression loss function, and perform S104 and S105 once in each iteration until the preset number of iterations is reached, to obtain the target anchor frame of each power facility and obtain the position information of the target anchor frame of each power facility. S104: Perform four convolutions on the feature map of the regression task, with a stride of 1 and a kernel size of [missing value]. The convolution operation is performed while keeping the number of feature channels constant; S105: Perform a convolution on the regression task feature map output from step S104 with a stride of 1 and a kernel size of [missing value]. The convolution operation is performed, and the number of feature channels is changed to 5.

3. The method for detecting power facilities in UAV imagery according to claim 2, characterized in that, The initial anchor frame for each power facility is iteratively optimized based on a preset regression loss function, including: For the n The optimization step involves obtaining the initial anchor frame offset of each power facility based on a preset regression loss function; and then, based on the initial anchor frame offset of each power facility, obtaining the first optimization step using the following formula. n The initial anchor frames for each power facility after the second optimization: in, , , , and They represent the first n The center point of the initial anchor frame after secondary optimization x Axis coordinates, center point y Axis coordinates, width, height, and angle. n As a preset constant, , , , and They represent the first n The center point of the initial anchor frame during the second optimization. x Axis coordinate offset, center point y Axis coordinate offset, width offset, height offset, and angle offset.

4. The method for detecting power facilities in UAV imagery according to claim 1, characterized in that, The regression loss function is: in, N This represents the initial number of anchor frames. Represents the future, Represents background, This represents the offset of the predicted bounding box relative to the preset anchor box. This represents the offset of the actual bounding box relative to the preset anchor box. This represents the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. , , , and These represent the center points of the initial anchor frame, respectively. x Axis coordinates, center point y Axis coordinates, width, height, and angle; The loss function of the classification module is: in, (·) represents the focus loss function. t i Indicates the target anchor box i The classification results p i Represents the target anchor frame i The probability distribution of the predicted different classification results.

5. A detection system for power facilities in UAV imagery, characterized in that, include: The data acquisition module is used to acquire images of the drone to be detected; The model processing module is used to input UAV imagery into a pre-trained high-precision power facility target detection model, annotate the target anchor boxes of each power facility in the UAV imagery using the pre-trained high-precision power facility target detection model, and obtain the location information and classification results of the target anchor boxes of each power facility; wherein, the classification result is the power facility type; The pre-trained high-precision power facility target detection model is obtained by training a high-precision power facility target detection model based on several labeled UAV images. The high-precision power facility target detection model includes a bidirectional feature fusion module, a feature extraction module, a regression refinement module, and a classification module. The bidirectional feature fusion module is used to extract bidirectional fused multi-scale feature maps of UAV images through bidirectional fusion. The extraction module is used to obtain classification task features and regression task features from bidirectional fused multi-scale feature maps through spatial attention network and channel attention network; The regression refinement module is used to generate a regression task feature map based on the regression task features, and to use each feature point on the regression task feature map as each power facility and to label the initial anchor box of each power facility; and to iteratively optimize the initial anchor box of each power facility based on a preset regression loss function to obtain the target anchor box of each power facility and to obtain the location information of the target anchor box of each power facility. The classification module is used to obtain the classification results of the target anchor frames of each power facility based on the characteristics of the classification task; The bidirectional feature fusion module includes a ResNet network, a top-down fusion network, and a bottom-up fusion network; The top-down fusion network is as follows: The bottom-up convergence network is: in, These represent the outputs of the ResNet network, specifically the outputs of the Conv3_x, Conv4_x, and Conv5_x layers of the ResNet network. This represents the multi-scale feature map output from the top-down fusion network. This represents the multi-scale feature map output from the bottom-up fusion network and serves as the bidirectional fusion multi-scale feature map. This represents a convolution operation with a stride of 1 and a kernel size of 1×1. This represents a convolution operation with a stride of 1 and a kernel size of 3×3. This represents a double upsampling operation. This represents a double downsampling operation; The process of obtaining classification task features and regression task features by using spatial attention networks and channel attention networks to fuse bidirectional multi-scale feature maps includes: The bidirectional fused multi-scale feature map is input into the spatial attention network to obtain the spatial attention response map; The bidirectional fused multi-scale feature map is input into the channel attention network to obtain the channel attention response map; The spatial attention response map and the channel attention response map are multiplied by tensors to obtain the tensor multiplication result; The feature weights of the regression task are obtained by activating the tensor multiplication result through the regression task activation function; The classification task feature weights are obtained by activating the tensor multiplication result using the classification task activation function; The regression task feature weights are element-wise multiplied with the bidirectional fusion multi-scale feature map to obtain the first element-wise multiplication result, which is then summed with the residual of the bidirectional fusion multi-scale feature map to obtain the regression task features. The classification task feature weights are element-wise multiplied with the bidirectional fusion multi-scale feature map to obtain the second element-wise multiplication result, which is then summed with the residual of the bidirectional fusion multi-scale feature map to obtain the classification task features. The spatial attention network is as follows: in, Represents global average pooling. and Represents a fully connected layer. Represents the Sigmoid function. Represents spatial attention response map; The channel attention network is as follows: in, This represents a convolution with a kernel size of 1×1. This represents a convolution with a kernel size of 3×3. , , and These represent convolutions with a kernel size of 3×3 and dilatations of 1, 3, 5, and 7, respectively. `cat` represents a channel concatenation operation. Representative channel attention response diagram; Regression task activation function for: Classification task activation function for: in, x This represents the activation variable.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method for detecting power facilities in UAV imagery as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for detecting power facilities in UAV imagery as described in any one of claims 1 to 4.

Images