Knowledge-guided infrared power line intelligent detection method
By introducing knowledge embedding strategies and cognitive decoders in power line detection, using the prior information of power lines to build a KRNet model, the problem of the failure of the prior art to fully utilize the prior information of power lines is solved, and the power lines in complex contexts is realized Improved detection performance.
Patent Information
- Application Number
- CN202410732299.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-06
- Publication Date
- 2024-11-12
AI Technical Summary
The existing power line detection methods are mainly concentrated on segmentation detection methods, and fail to fully utilize multiple prior information of power lines, such as linearity, distribution characteristics and non-directionality, resulting in poor detection results.
Using a knowledge embedding strategy, combining cognitive decoder with RetinaNet model, linear aggregation branches and distribution perception branches are embedded, self-attention mechanism and morphological convolution are used to mine the prior features of power lines, and RACIoU loss function and binary cross entropy loss are introduced. Function, considering the non-directionality and rotation angle of the power line, a KRNet model is constructed for infrared power line detection.
In a complex context, the comprehensive perception of power lines is achieved, the detection accuracy and performance are improved, false alarms and missed reports are reduced, and the refinement level of power line detection is improved.
Smart Images

Figure CN118941495A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of intelligent detection, and in particular to a knowledge-guided infrared power line intelligent detection method. Background Art
[0002] In the field of low-altitude flight, aircraft such as drones and helicopters face many potential threats when performing increasingly numerous tasks. Among them, power lines are regarded as a non-significant target with extremely high danger due to their many characteristics. At the same time, infrared is widely used in low-altitude aircraft as an increasingly common sensing data. Therefore, accurate detection of power lines with infrared fusion images is crucial to ensure the safety of low-altitude flight.
[0003] Traditional power line image detection methods are divided into methods based on linear features and associated target detection. However, these methods have certain limitations in actual operation. It is not easy to manually adjust parameters to obtain the best results, and when the parameters are fixed, they tend to produce more false positives and false negatives on the entire dataset. Thanks to the explosive development of deep learning, the level of refined detection of power lines has been continuously improved. For example, the power line detection method based on the pyramid classification framework uses a convolutional neural network (CNN) to determine whether the patch contains a target, effectively reducing the occurrence of false alarms. In order to further improve the detection effect, some studies have combined well-designed feature fusion and deep learning methods. The cross-scene attention information fusion network based on pixel-level power line detection uses semantic and location information to make the model pay more attention to power lines rather than unexpected scenes. In addition, some scholars have proposed methods that use multi-scale and structured prior information to reduce noise interference and achieve accurate and efficient detection. However, existing research mainly focuses on segmentation detection methods, which often require expensive annotation costs. At the same time, due to the characteristics of the power line itself and the fact that these methods do not fully utilize multiple prior knowledge of the power line (such as linearity, distribution characteristics, and non-directionality, etc.), the performance improvement is limited. Summary of the invention
[0004] In view of the problem that the existing power line detection methods mainly focus on segmentation detection methods, but fail to fully utilize multiple prior information of power lines, such as linearity, distribution characteristics and non-directionality, and cannot achieve satisfactory detection results, the present invention provides a knowledge-guided infrared power line intelligent detection method, which adopts a knowledge embedding strategy, considers the prior information of power lines in model construction and training, realizes comprehensive perception of power lines in complex backgrounds, and thus improves detection accuracy.
[0005] The knowledge-guided infrared power line intelligent detection method of the present invention includes:
[0006] A detection model for classifying power lines from infrared images is established based on the RetinaNet model. The RetinaNet model is mainly composed of a ResNet50 backbone network and a feature pyramid network connected in sequence. The output of the feature pyramid network is respectively input into a regression subnetwork and a classification subnetwork to obtain corresponding bounding box regression results and classification results.
[0007] Wherein, a cognitive decoder is embedded after the feature pyramid network to form a KRNet model, wherein the cognitive decoder includes a linear aggregation branch and a distributed perception branch;
[0008] Based on the idea of self-attention mechanism, the cognitive decoder applies three convolutions to map the input feature X of dimension H×W×C to Three low-dimensional features: Q, K and V, where r represents the dimensionality reduction coefficient;
[0009] The three low-dimensional features are input into the distribution perception branch, which uses the bidirectional self-attention mechanism to explore the distribution characteristics of power lines, including the relationship between power lines and towers;
[0010] The three low-dimensional features are also input into the linear aggregation branch, and the shape convolution is used for effective linear feature mining;
[0011] Finally, the output of the distribution perception branch is used to adjust the output of the linear aggregation branch, and the result is convolved to obtain an output feature with a dimension of H×W×C and input into the regression sub-network and the classification sub-network;
[0012] During training, the bounding box regression result corresponding to the output of the regression subnetwork adopts the RACIoU loss function, which is customized for the inherent non-directionality of power lines and extends the existing CIoU loss by additionally considering the rotation angle;
[0013] Corresponding to the output of the classification subnetwork, the binary cross entropy function is used as the loss function, and both boundary regression and classification losses are incorporated;
[0014] The trained KRNet model is used to detect power lines from infrared images.
[0015] Furthermore, in the linear aggregation branch, Q, K, and V are connected in the channel dimension, and two different linear convolution blocks are used to amplify the prior linear characteristics. Finally, this information is aggregated and integrated with the distribution information. The process is specifically expressed as follows:
[0016] f mid =σ(F 1×3 ([Q,K,V])) H×W×C / r
[0017] f Aug. =σ(F3×1 (f mid )) H×W×C / r
[0018] Among them, f mid represents the result of the first convolutional block, f Aug. represents the output of the linear aggregation branch, σ represents the ReLU activation function, and F 1×3 and F 3×1 They represent convolutions with kernel shapes of 1×3 and 3×1 respectively.
[0019] Furthermore, in the distributed perception branch, the input Q, K, and V features are pooled in W dimension on the one hand, and in H dimension on the other hand. The results of these two pools are used to calculate the dot product attention in their respective directions as output. The specific process is shown as follows:
[0020] f X =softmax(GAP H (Q) GAP H (K) T )·GAP H (V)
[0021] f Y =softmax(GAP W (Q) GAP W (K) T )·GAP W (V)
[0022] Among them, GAP H Represents H×1 global average pooling, GAP W represents 1×W global average pooling, and the symbol · represents matrix multiplication. Represents element-wise multiplication.
[0023] Furthermore, the cognitive decoder uses f X and f Y To adjust f Aug. , and use 1×1 convolution to adjust the number of channels, restore the dimension to H×W×C, and finally use the sigmoid function to activate the output f out , the process is expressed as follows:
[0024]
[0025] Among them, f out represents the output of the cognitive module, F 1×1 Represents the convolution of the adjusted channels.
[0026] Furthermore, the RACIoU loss function describes the gap between the measured bounding box and the predicted bounding box from four aspects: overlap rate, center distance, shape, and rotation angle. The specific expressions are as follows:
[0027] L RACIoU =Φ overlap +Φ dis +Φ shape +Φ θ
[0028] Among them, Ф overlap is a differentiable RIoU loss, expressed as 1-ξ RIoU , Ф dis To measure the Euclidean distance between the center points of the bounding box and the predicted bounding box, it is normalized by the diagonal c of the enclosing rectangle, which is specifically expressed as:
[0029]
[0030] Among them, P gt To measure the center point of the bounding box, P pre is the center point of the predicted box boundary, ρ 2 (·) indicates Euclidean distance calculation;
[0031] Ф overlap When RIoU is high, the normalized difference between the aspect ratio of the bounding box and the predicted box is calculated, which is specifically expressed as:
[0032]
[0033] Among them, w gt 、h gt 、w pre and h pre Represent the long and short sides of the measured bounding box and the predicted bounding box respectively. When RIoU is less than 0.5, Φ shape It doesn’t work;
[0034]
[0035] Among them, θ gt and θ pre denote the actual angle and predicted angle of the bounding box, respectively.
[0036] Furthermore, the actual angle range is
[0037] Furthermore, when the difference between the predicted angle and the actual angle exceeds When shape Set to zero.
[0038] Furthermore, the binary cross entropy loss function is expressed as:
[0039]
[0040] Where N represents the number of positive samples; p i represents the predicted category probability of the i-th sample. If the i-th sample is a positive sample, then p i is equal to 1; otherwise, p i Then it is 0, L cls is the classification loss, L reg is the regression loss. i is the true value, T i is the predicted value and λ is the balancing hyperparameter.
[0041] The beneficial effect of the present invention lies in that a knowledge-guided rotation network KRNet is proposed, a new single-stage rotation detection method for power line detection in infrared images is implemented, a knowledge embedding strategy is utilized, and prior knowledge of power lines is considered in model construction and training, thereby achieving comprehensive perception of power lines in complex backgrounds and improving detection performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 Schematic diagram of the overall framework of the KRNet model proposed in the embodiment of the present invention;
[0043] Figure 2 Schematic diagram of infrared power line image characteristics in an embodiment of the present invention;
[0044] Figure 3 Schematic diagram of a cognitive decoder framework in an embodiment of the present invention;
[0045] Figure 4 is a schematic diagram of a processing process in a cognitive decoder in an embodiment of the present invention;
[0046] Figure 5 Schematic diagram of the RACIoU loss function in an embodiment of the present invention;
[0047] Figure 6 The figure shows the comparison of the detection results of different models on the PRDD dataset in the embodiments of the present invention. DETAILED DESCRIPTION
[0048] The following is combined with Figure 1-6 The present invention is further described in detail with reference to specific implementation cases.
[0049] The KRNet model in this example is built based on the RetinaNet model. The classic RetinaNet model for infrared power line detection is mainly composed of a ResNet50 backbone network, a feature pyramid network (FPN) and a decoupling head connected in sequence. The decoupling head includes a regression head subnetwork and a classification subnetwork. The RetinaNet model abandons the general idea of using top-level features for prediction, and instead obtains bottom-level features and high-level features from the feature extraction network (ResNet50 backbone network), and fuses the bottom-level features and high-level features in turn through FPN, and then predicts on different feature layers respectively. Finally, the classification subnetwork outputs the classification result, and the regression subnetwork outputs the bounding box regression result.
[0050] In addition, the power lines are easily confused in the complex background, and the power lines themselves in the infrared fusion image have Figure 2 The low radiation characteristics shown in (a) Figure 2 The abnormal aspect ratio characteristics shown in (b) and Figure 2 The non-directionality shown in (c) affects the performance of infrared power line detection.
[0051] To solve this problem, Figure 1 As shown in the figure, in this example, a cognitive decoder (LDPM) is embedded in the feature fusion stage of FPN to learn and understand the linear and distributed features of power lines, thereby integrating prior knowledge into the features. In addition, in the training of the model, corresponding to the bounding box output of the regression subnetwork, the RACIoU (Rotation-Aware Complete Intersection-over-Union) loss function is introduced in this example. This loss function is customized for the inherent non-directionality of power lines and extends the existing CIoU loss by additionally considering the rotation angle. On the other hand, corresponding to the output of the classification subnetwork, the binary cross entropy function is used as the loss function.
[0052] As attached Figure 3 As shown in the figure, the cognitive decoder uses the prior knowledge obtained in the linear aggregation branch (Linear Aug.) and the distribution awareness branch (Dist.Aware) to suppress the false detection caused by these problems. In the construction of this module, based on the idea of the self-attention mechanism, three convolutions are applied to map the input feature X of dimension H×W×C into three low-dimensional features: Q, K and V, each of which has a dimension of where represents the dimension reduction coefficient, which serves as the basis for the subsequent branching process. In the distribution perception branch, the bidirectional self-attention mechanism is used to explore the distribution characteristics of power lines, including the relationship between power lines and towers. The processing process of each low-dimensional feature in the cognitive decoder is as follows: Figure 3 As shown, the details are as follows:
[0053] Linear Aggregation Branch Module: This module is specially designed to handle the linear local details inherent in power lines. The process first connects Q, K, and V in the channel dimension, then uses two different linear convolution blocks to amplify the prior linear characteristics, and finally aggregates and integrates this information with the distribution information. The above process can be expressed as:
[0054] f mid =σ(F 1×3 ([Q,K,V])) H×W×C / r
[0055] f Aug. =σ(F 3×1 (f mid )) H×W×C / r
[0056] Among them, f mid Represents the result of the first convolutional block. Aug. represents the output of the linear aggregation branch, σ represents the ReLU activation function, and F 1×3 and F 3×1 They represent convolutions with kernel shapes of 1×3 and 3×1, respectively. This example design includes linear prior information in the shape of the convolution kernel. In order to solve the problem of non-directional power lines, this example designs two convolution kernels in different directions to achieve robust linear capture.
[0057] Distribution-aware module: This example considers the distribution relationship between power lines and towers and between power lines themselves as global information. To balance efficient computation and sensitivity to power line distribution, this example starts with bidirectional global average pooling applied to Q, K, and V, as Figure 3 As shown in the figure, the first input Q, K and V features are pooled in the X-axis direction (W dimension) and in the Y-axis direction (H dimension). The two pooled results are used to calculate the dot product attention in their respective directions. Finally, the output f from the linear aggregation is filtered using bidirectional attention. Aug. , the process can be expressed as:
[0058] f X =softmax(GAP H (Q) GAP H (K) T )·GAP H (V)
[0059] f Y =softmax(GAP W (Q) GAP W (K) T )·GAP W (V)
[0060]
[0061] Among them, GAP H Represents H×1 global average pooling, GAP W represents 1×W global average pooling, F 1×1 represents the convolution of the adjusted channel, and the symbol · represents matrix multiplication. represents element-wise multiplication, f out represents the output of the cognitive decoder, using the prior knowledge about the power line distribution f X and f Y To adjust the linear aggregation features, and use 1×1 convolution to adjust the number of channels, its dimension is restored to H×W×C, and finally use the sigmoid function to activate the output f out .
[0062] A cognitive decoder is embedded in the feature fusion stage to obtain the prior features of the power lines based on the known distribution and linear features. This method uses special shaped convolution to perform effective linear feature mining in the linear aggregation branch.
[0063] In order to improve RetinaNet’s ability to capture power line features, by reasonably and efficiently utilizing the prior knowledge of power lines, in addition to the designed cognitive module and RAIoU loss function, this example relies on the linear, distributed, and non-directional characteristics of power lines to enhance the feature representation of power lines, thereby achieving the purpose of accurately detecting power line targets.
[0064] In addition, in order to improve the regression accuracy, a priori constraint on the non-directionality of power lines is introduced in the designed RACIoU loss function.
[0065] The RACIoU loss function designed in this example extends the existing RIoU loss by additionally considering the rotation angle. This loss function is tailored to the inherent non-directionality of power lines, such as Figure 4 As shown in the figure, the loss function describes the gap between the measured bounding box and the predicted bounding box from four aspects: overlap rate, center distance, shape, and rotation angle. The specific expression is as follows:
[0066] L RACIoU =Φ overlap +Φ dis +Φ shape +Φ θ
[0067] Among them, Ф overlap is a differentiable RIoU loss, expressed as 1-ξ RIoU , Ф dis To measure the Euclidean distance between the center points of the bounding box and the predicted bounding box, normalize it by the diagonal c of the enclosing rectangle, which can be expressed as:
[0068]
[0069] Among them, P gt To measure the center point of the bounding box, P pre is the center point of the predicted box boundary, ρ 2 (·) indicates Euclidean distance calculation;
[0070] In addition, Ф overlap The calculation of RIoU is to measure the normalized difference between the aspect ratio of the bounding box and the predicted box when it is high, which can be expressed as:
[0071]
[0072] Among them, w gt 、h gt 、w pre and h pre Represent the long and short sides of the measured bounding box and the predicted bounding box respectively. When RIoU is less than 0.5, Φ shape It doesn’t work;
[0073] Although the rotation angle is Φ shape However, this example designs a more precise angle normalization constraint, which can be expressed as:
[0074]
[0075] Among them, θ gt and θ pre Represent the actual angle and predicted angle of the bounding box respectively. In this example, the actual angle range is In order to deal with the periodicity of the angle, a simple treatment is adopted here, that is, when the difference between the predicted angle and the actual angle exceeds When shape Set to zero, RACIoU will be Φ shape The soft constraint and Φ shape Combined to guide model fine-tuning.
[0076] For the output of the classification subnetwork, this example uses the binary cross entropy loss function format of the baseline, that is, boundary regression and classification loss, and the overall loss is expressed as:
[0077]
[0078] Where N represents the number of positive samples; p i represents the predicted category probability of the i-th sample. If the i-th sample is a positive sample, then p i is equal to 1; otherwise, p i Then it is 0, L cls is the classification loss, Lreg is the regression loss. i is the true value, T i is the predicted value and λ is the balancing hyperparameter.
[0079] Finally, if Figure 6 As shown, it is different from the Ground truth model ( Figure 6 (a)), R-Faster R-CNN model ( Figure 6 (b) ROI Transformer Model Figure 6 (c)), Oriented R-CNN model ( Figure 6 (d) R-FCOS model Figure 6 (e)) and CFA model ( Figure 6 (f)) and CIoU model ( Figure 6 (e)) In the comparison of the detection results obtained on the PRDD dataset, Figure 6 The detection results obtained by the KRNet model shown in (h) confirm that the proposed KRNet model is able to improve the detection performance of power lines.
Claims
1. A knowledge-guided infrared power line intelligent detection method, comprising: establishing a detection model for classifying power lines from infrared images based on a RetinaNet model, wherein the RetinaNet model is mainly composed of a ResNet50 backbone network and a feature pyramid network connected in sequence, and the output of the feature pyramid network is respectively input into a regression subnetwork and a classification subnetwork to obtain corresponding bounding box regression results and classification results; It is characterized in that A cognitive decoder is embedded after the feature pyramid network to form a KRNet model, wherein the cognitive decoder includes a linear aggregation branch and a distribution perception branch; Based on the idea of self-attention mechanism, the cognitive decoder applies three convolutions to map the input feature X of dimension H×W×C to Three low-dimensional features: Q, K and V, where r represents the dimensionality reduction coefficient; The three low-dimensional features are input into the distribution perception branch, which uses the bidirectional self-attention mechanism to explore the distribution characteristics of power lines, including the relationship between power lines and towers; The three low-dimensional features are also input into the linear aggregation branch, and the shape convolution is used for effective linear feature mining; Finally, the output of the distribution perception branch is used to adjust the output of the linear aggregation branch, and the result is convolved to obtain an output feature with a dimension of H×W×C and input into the regression sub-network and the classification sub-network; During training, the bounding box regression result corresponding to the output of the regression subnetwork adopts the RACIoU loss function, which is customized for the inherent non-directionality of power lines and extends the existing CIoU loss by additionally considering the rotation angle; Corresponding to the output of the classification subnetwork, the binary cross entropy function is used as the loss function, and both boundary regression and classification losses are incorporated; The trained KRNet model is used to detect power lines from infrared images.
2. The method according to claim 1, characterized in that In the linear aggregation branch, Q, K, and V are connected in the channel dimension, and two different linear convolution blocks are used to amplify the prior linear characteristics. Finally, this information is aggregated and integrated with the distribution information. The specific process is expressed as: f mid =σ(F 1×3 ([Q,K,V])) H×W×C / r f Aug. =σ(F 3×1 (f mid )) H×W×C / r Among them, f mid represents the result of the first convolutional block, f Aug. represents the output of the linear aggregation branch, σ represents the ReLU activation function, and F 1×3 and F 3×1 They represent convolutions with kernel shapes of 1×3 and 3×1 respectively.
3. The method according to claim 2, characterized in that In the distributed perception branch, the input Q, K, and V features are pooled in W dimension on the one hand, and in H dimension on the other hand. The two pooling results are used to calculate the dot product attention in their respective directions as output. The specific process is shown as follows: f X =softmax(GAP H (Q)·GAP H (K) T )·GAP H (V) f Y =softmax(GAP W (Q)·GAP W (K) T )·GAP W (V) Among them, GAP H Represents H×1 global average pooling, GAP W represents 1×W global average pooling, and the symbol · represents matrix multiplication. Represents element-wise multiplication.
4. The method according to claim 3, characterized in that The cognitive decoder uses f X and f Y To adjust f Aug. , and use 1×1 convolution to adjust the number of channels, restore the dimension to H×W×C, and finally use the sigmoid function to activate the output f out , the process is expressed as follows: Among them, f out represents the output of the cognitive module, F 1×1 Represents the convolution of the adjusted channels.
5. The method according to claim 1, characterized in that The RACIoU loss function describes the gap between the measured bounding box and the predicted bounding box from four aspects: overlap rate, center distance, shape, and rotation angle. The specific expressions are as follows: L RACIoU =Φ overlap +F dis +F shape +F θ Among them, Ф overlap is a differentiable RIoU loss, expressed as 1-ξ RIoU , Ф dis To measure the Euclidean distance between the center points of the bounding box and the predicted bounding box, it is normalized by the diagonal c of the enclosing rectangle, which is specifically expressed as: Among them, P gt To measure the center point of the bounding box, P pre is the center point of the predicted box boundary, ρ 2 (·) indicates Euclidean distance calculation; Ф overlap When RIoU is high, the normalized difference between the aspect ratio of the bounding box and the predicted box is calculated, which is specifically expressed as: Among them, w gt 、h gt 、w pre and h pre Represent the long and short sides of the measured bounding box and the predicted bounding box respectively. When RIoU is less than 0.5, Φ shape It doesn’t work; θ gt and θ pre denote the actual angle and predicted angle of the bounding box, respectively.
6. The method according to claim 5, characterized in that The actual angle range is 7. The method according to claim 5, characterized in that That is, when the difference between the predicted angle and the actual angle exceeds When shape Set to zero.
8. The method according to claim 5, characterized in that The binary cross entropy loss function is expressed as: Where N represents the number of positive samples; p i represents the predicted category probability of the i-th sample. If the i-th sample is a positive sample, then p i is equal to 1, otherwise, p i Then it is 0; L cls is the classification loss, L reg is the regression loss, t i is the true value, T i is the predicted value and λ is the balancing hyperparameter.
