UAV line patrol navigation method and system based on infrared image power line recognition

Abstract

The present invention proposes a drone line inspection navigation method and system based on infrared image power line recognition. The method comprises: mathematically abstracting overhead power lines as a navigation basis for drones flying along the power lines; randomly extracting from the acquired aerial images as training and test sets, and annotating the coordinate values ​​of each power line endpoint according to the image's default coordinate system; constructing a deep learning recognition model, the model comprising an encoding-decoding module and a prediction module, the prediction module comprising three fully convolutional network branches, which respectively predict the Gaussian distribution kernel of the power line center point, the offset of the power line center point relative to the true value, and the power line direction; clustering the power line center points and directions predicted by the deep learning recognition model to obtain the overall position and direction of the power line bundle, which is used to guide the autonomous navigation of the drone. The present invention can achieve end-to-end identification of power line center coordinates and azimuths, with high efficiency, and facilitates drone inspection navigation.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle (UAV) patrol navigation, and in particular relates to a UAV patrol navigation method and system based on infrared image power line recognition. Background Art

[0002] To ensure the reliability of transmission line systems and prevent widespread power outages caused by equipment damage, power operators typically conduct regular inspections of equipment and accessories in outdoor transmission line systems. This helps identify problems and potential safety hazards, facilitates timely maintenance, and ensures stable operation of the transmission system. Compared to the tasks of remote power line monitoring systems, transmission line inspections cover a much larger area, are more complex, and present a greater challenge. Power line inspections involve a wider variety of components, distributed across a complex environment, and are also subject to weather conditions. Early manual inspections, characterized by low efficiency and poor safety, have been gradually replaced by vision-based drone-based power line inspection solutions. A key challenge facing drone-based power line inspections is ensuring efficient inspections. To enable drones to more effectively identify and identify risks in critical infrastructure such as power lines and utility poles, a more effective navigation method involves flying the drone above and following the power lines. During flight, the drone's forward-facing camera tracks the poles when it detects them, adjusting the camera's angle to capture real-time footage. After completing its inspection mission, the drone can also return home along the power lines. Due to the complex distribution of power lines, in some areas, power line information is not accurately mapped, making it difficult to achieve accurate navigation based on GPS information, and it is difficult to ensure that the camera can clearly capture the power lines and their accessories. Furthermore, GPS information in some areas is weak, and signal loss may occur at times. Therefore, relying solely on GPS navigation is not very reliable. However, vision-based navigation along power lines relies on accurate identification of power lines and is more reliable and accurate than GPS navigation.

[0003] Therefore, identifying power lines is an essential task for autonomous nighttime drone inspections based on infrared images. If a drone can identify the location and orientation of power lines, it can adjust its flight direction and position in real time based on this information to maintain alignment with the power line's path. Numerous studies have been conducted on the problem of power line identification during drone inspections. For example, Bums et al. (1986) proposed a method for identifying power lines using edge detection and pixel clustering, Akinlar et al. (2011) proposed an edge contour detection method called EDlines, and Ceron et al. (2014) proposed a drone power line detection model based on circle search (CBS). Based on CBS, Ceron et al. (2018) proposed a power line identification method based on histogram of line gradients (HOS). However, these methods are not stable and reliable when processing drone aerial images against complex backgrounds. Summary of the Invention

[0004] In response to the deficiencies in the prior art, the present invention provides a method and system for drone line patrol navigation based on infrared image power line recognition, so as to achieve end-to-end identification of power line center coordinates and direction angles, with high efficiency, and facilitate the completion of drone patrol navigation. Since drones operate in complex environments, automatic identification and following of power lines is a challenging task. In order to solve this problem, the present invention proposes a new method based on deep convolutional neural networks. Unlike traditional methods, this method can automatically extract the features of power lines from complex backgrounds, and can effectively extract the features of power lines and predict the center and direction of each power line in real time; based on this, the present invention further proposes a clustering method for the center position and orientation of power lines, which aggregates the positions and orientations of all power lines into a total position and orientation, and uses it to guide the nighttime patrol navigation of drones.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A drone line inspection and navigation method based on infrared image power line recognition is characterized by comprising the following steps:

[0007] Step 1: Perform mathematical abstraction of the overhead power lines and use it as a navigation basis for the UAV to fly along the power lines;

[0008] Step 2: Randomly select several aerial photos as training and test sets, and mark the coordinate values ​​of each power line endpoint;

[0009] Step 3: Construct a deep learning recognition model and train the model using a training set and a test set; the deep learning recognition model extracts image features and generates a feature map, searches for regions with Gaussian kernel distribution characteristics on the feature map, selects the center of each region and ranks them according to the size of the center point value, selects the top point as the candidate power line center point, then screens the pixel values ​​of the candidate power line center points to obtain a preliminary predicted power line center point, searches for its offset from the actual power line center based on the coordinates of the preliminary predicted power line center point, and obtains the final predicted value of the power line center point coordinate after correcting the offset; the deep learning recognition model also predicts the direction of each power line;

[0010] Step 4: Cluster the power line center points and power line directions predicted by the deep learning recognition model to obtain the overall position and direction of the power line bundle, which is used to guide the autonomous navigation of the UAV.

[0011] To optimize the above technical solutions, specific measures taken also include:

[0012] Furthermore, in step 1, mathematically abstracting the overhead power lines includes averaging the endpoints of the power lines in the overhead view of the drone to obtain the center point of the power lines, and using the sine and cosine values ​​of the power line center point and the power line direction angle to represent the power lines.

[0013] Furthermore, step 2 also includes designing a data enhancement method based on the power line endpoints, and the data enhancement method includes arbitrary direction translation, arbitrary scale reduction, arbitrary angle rotation and arbitrary shear deformation.

[0014] Furthermore, in step 3, the deep learning recognition model ranks the points according to the size of the center point value, and selects up to the top 100 points as alternative power line center points; the deep learning recognition model screens the pixel values ​​of the alternative power line center points, and takes the alternative power line center points with pixel values ​​greater than or equal to the threshold as the preliminarily predicted power line center points.

[0015] Furthermore, the step 4 further includes: calculating the minimum bounding rectangle of all the power line center points predicted by the deep learning recognition model, and constructing a Gaussian kernel to describe the weight corresponding to the predicted power line according to the center position of the minimum bounding rectangle, (x i ,y i ) is the coordinate of the center point of the i-th power line, (x c ,y c ) is the center coordinate of the Gaussian kernel, then the weight value w corresponding to the i-th power line i Calculated according to the following formula:

[0016]

[0017] The weight distribution of the power lines is thus obtained, and the overall position and direction of the power line bundle are obtained by taking this weight distribution into consideration.

[0018] The present invention also proposes a UAV line patrol navigation system based on infrared image power line recognition, which is characterized by comprising: a mathematical abstraction module, a data annotation and enhancement module, a deep learning recognition model and a clustering module;

[0019] The mathematical abstraction module performs mathematical abstraction on the overhead power lines as a navigation basis for the UAV to fly along the power lines;

[0020] The data annotation and enhancement module randomly selects several aerial photos as training sets and test sets, annotates the coordinate values ​​of each power line endpoint, and performs data enhancement based on the power line endpoints;

[0021] The deep learning recognition model is trained using a training set and a test set to predict the center point and direction of the power line. The deep learning recognition model includes an encoding-decoding module and a prediction module. The encoding-decoding module extracts image features and generates a feature map required by the prediction module. The prediction module includes three fully convolutional network branches: the first branch predicts the Gaussian kernel of the power line center line and generates a heat map including the position of the power line center point and the Gaussian kernel distribution; the second branch predicts the offset between the center coordinates of the Gaussian kernel in the heat map and the actual power line center, and the offset is used to predict the power line center point; the third branch predicts the sine and cosine values ​​of the power line direction angle;

[0022] The clustering module clusters the power line center points and power line directions predicted by the deep learning recognition model to obtain the overall position and direction of the power line bundle, which is used to guide the autonomous navigation of the UAV.

[0023] Furthermore, the mathematical abstraction module performs mathematical abstraction on the overhead power lines, including: averaging the power line endpoints in the drone's overhead perspective to obtain the power line center point, and using the power line center point and the sine and cosine values ​​of the power line direction angle to represent the power line.

[0024] Furthermore, in the data annotation and enhancement module, the data enhancement methods include arbitrary direction translation, arbitrary scale reduction, arbitrary angle rotation and arbitrary shear deformation.

[0025] Furthermore, the input of the deep learning recognition model is a 512×512×3 infrared RGB image, which generates a feature map of size 128×128×3 after passing through the encoding-decoding module; the first branch is composed of a single-channel feature map, and the output heat map size is 128×128×1; the second branch size is 128×128×2.

[0026] Furthermore, in the deep learning recognition model, the first branch uses the focal loss function as the loss function:

[0027]

[0028] Where, L hm is the loss function of the first branch, Y xy is the true value, is the predicted value of the deep learning recognition model, α and β are hyperparameters in focal loss, and N is the number of samples;

[0029] The second and third branches use the L1 loss function:

[0030]

[0031]

[0032] Where, L off is the loss function of the second branch, L ori is the loss function of the third branch, Pos represents the positive sample; x and o y is the real center point coordinate of the power line, and is the coordinate of the center point of the power line predicted by the deep learning recognition model, θ p is the true direction angle of the electric line, is the direction angle of the power line predicted by the deep learning recognition model;

[0033] The final loss function of the deep learning recognition model is the weighted sum of the loss functions of the three branches:

[0034] L det =L hm +λL off +γL ori

[0035] Where, L det is the final loss function of the deep learning recognition model, and λ and γ are the weights of the power line center point offset loss function and the power line direction loss function, respectively.

[0036] The beneficial effects of the present invention are:

[0037] 1) The method proposed in this paper can achieve end-to-end identification of power line center coordinates and direction angles with high efficiency. Compared with traditional line detection and edge detection, the proposed power line center point and direction identification method has higher accuracy and robustness;

[0038] 2) When designing a deep learning-based power line center point and direction recognition model, the sine and cosine values ​​of the power line direction angle are innovatively used as the output of the deep learning model and activated through the tanh function, thus expanding the deep learning-based target detection method from the recognition of object target frames to the recognition of power line center points and directions, enabling the deep learning-based target detection model to realize the new function of detecting power line center points and directions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The center point and direction diagram of the power lines in the top view provided by the embodiment of the present invention.

[0040] Figures 2a to 2d An arbitrary horizontal displacement transformation diagram provided by an embodiment of the present invention, Figure 2a It's the original picture. Figure 2b The two endpoints of the power line are still within the horizontal boundary of the image after translation. Figure 2c One end point of the power line moves out of the left boundary of the image after the horizontal translation transformation. Figure 2d One end point of the power line moves out of the right boundary of the image after the horizontal translation transformation.

[0041] Figures 3a to 3d An arbitrary vertical displacement transformation diagram provided by an embodiment of the present invention, Figure 3a It's the original picture. Figure 3b The two endpoints of the power line are still within the upper and lower boundaries of the image after transformation. Figure 3c One end point of the power line is vertically translated beyond the upper boundary of the image. Figure 3d One end point of the power line exceeds the lower boundary of the image after vertical translation transformation.

[0042] Figure 4a and Figure 4b Any scale reduction diagram provided by the embodiment of the present invention, Figure 4a It's the original picture. Figure 4b are the scaled power lines.

[0043] Figures 5a to 5c The arbitrary angle rotation diagram provided by the embodiment of the present invention, Figure 5a It's the original picture. Figure 5b It is a rotated picture. Figure 5c It is a rotated image with changed dimensions.

[0044] Figure 6a and Figure 6b An arbitrary shear deformation diagram provided by an embodiment of the present invention, Figure 6a It's the original picture. Figure 6b It is a shear deformation picture.

[0045] Figure 7This is a structural diagram of a model for identifying the center point and direction of power lines provided by an embodiment of the present invention.

[0046] Figure 8 This is a Gaussian distribution diagram of power line weights provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will now be described in further detail with reference to the accompanying drawings.

[0048] Example 1

[0049] In this embodiment, the present invention proposes a drone line patrol navigation method based on infrared image power line recognition, which specifically includes the following steps:

[0050] Step 1: Look down at the mathematical abstraction of the power lines.

[0051] During a drone's nighttime patrol flight, power lines are usually in the drone's bird's-eye view. Therefore, for guiding the drone's navigation mission, infrared power lines can be approximated as straight lines, and the approximate center point and direction of the power lines can be obtained based on the endpoints of the straight lines. Figure 1 As shown, P 31 (x 31 ,y 31 ) and P 32 (x 32 ,y 32 ) are the two endpoints of a power line, then the power line can be approximated by a line segment formed by these two points; assuming that C3 is the midpoint of the approximated line segment and θ is the direction angle of this line, then the coordinates and direction of the center point of this power line can be expressed by equations (1), (2), (3) and (4):

[0052]

[0053]

[0054]

[0055]

[0056] For the power lines in the drone's bird's-eye view, the endpoints are averaged to obtain the approximate center point of the power line. The center point of the power line and the sine and cosine values ​​of the power line's direction angle are used to represent the power line, which serves as the navigation basis for the drone flying along the power line.

[0057] Step 2: Data annotation and enhancement.

[0058] 2,818 aerial images were obtained from the video published in the paper by Ceron et al. (https: / / youtu.be / pa4fqoGJmD) as research data, of which 2,297 and 521 images were randomly selected as training and test sets. This example annotated the coordinate values ​​of each power line endpoint according to the image's default coordinate system and saved them as a JSON file similar to the MS COCO format. Due to the small dataset, this example designed a series of data augmentation methods based on power line endpoints to fully utilize the existing data resources, reduce overfitting in deep learning model training, and improve model accuracy and robustness.

[0059] (1) Translation in any direction

[0060] The arbitrary translation of the power line can be decomposed into any horizontal translation and any vertical translation. The final translation result is the linear superposition of the two translations. According to the knowledge of homogeneous coordinate transformation, when point P(x, y) is translated in any direction to P′(x′y′), its corresponding coordinate change can be determined by equation (5):

[0061]

[0062] For any horizontal translation, the value is 0; Figures 2a to 2d shown.

[0063] Figure 2a A schematic diagram of an untransformed power line is shown, with a size of (w×h), where the two endpoints and the center point of the power line are P1(x1, y1), P2(x2, y2) and C(x c ,y c ). After the power lines are translated, the two endpoints are still within the horizontal boundary of the image, such as Figure 2b As shown in Figure 2, the transformed power line endpoints P′1(x′1, y′1) and P′2(x′2, y′2) can be obtained by equations (6), (7), (8) and (9) respectively:

[0064] x′1=x1+δx (6)

[0065] x′2=x2+δx (7)

[0066] y′1=y1 (8)

[0067] y′2=y2 (9)

[0068] Therefore, the coordinates of the center point of the transformed straight line can be calculated using equations (10) and (11):

[0069]

[0070]

[0071] For the second case, one end point of the power line moves out of the left boundary of the image after the horizontal translation transformation, such as Figure 2c As shown in Figure 2, the left endpoint of the power line should be replaced by the intersection of the power line and the left edge of the image. The endpoint of the transformed power line can be calculated according to equations (12), (13), (14) and (15):

[0072] x′1=x1+δx (12)

[0073] x′2=0 (13)

[0074] y′1=y1 (14)

[0075]

[0076] Correspondingly, in the third case, the endpoint of a power line exceeds the right boundary of the image after translation, as shown in Figure 2d As shown, the right endpoint of the translated power line is replaced by the intersection of the power line and the right boundary of the image. The endpoint of the transformed power line can be calculated by equations (16), (17), (18) and (19):

[0077] x′1=w (16)

[0078] x′2=x2+δx (17)

[0079]

[0080] y′2=y2 (19)

[0081] For any vertical translation transformation, there are also three cases that need to be analyzed. Figures 3a to 3d Several typical vertical variations are shown.

[0082] Figure 3a is the original image. For the first case of vertical translation transformation, the two endpoints of the power line are still within the upper and lower boundaries of the image after transformation, such as Figure 3b Therefore, the transformed power line endpoint coordinates in the first case can be determined using equations (20), (21), (22), and (23) respectively:

[0083] x′1=x1 (20)

[0084] x′2=x2 (21)

[0085] y′1=y1+δy ​​(22)

[0086] y′2=y2+δy (23)

[0087] In the second case, one end point of the power line exceeds the upper boundary of the image after vertical translation. Therefore, the upper end point of the transformed power line needs to be replaced by the intersection of the power line and the upper edge of the image, such as Figure 3c As shown. The endpoint coordinates of the transformed power lines can be calculated by equations (24), (25), (26) and (27) respectively:

[0088] x′1=x1 (24)

[0089]

[0090] y′1=y1+δy ​​(26)

[0091] y′2=0 (27)

[0092] When one end point of the power line exceeds the lower boundary of the image after vertical translation, then this change satisfies the third case, such as Figure 3d As shown, the transformed power line endpoint coordinates are calculated by equations (28), (29), (30) and (31):

[0093]

[0094] x′2=x2 (29)

[0095] y′1=h (30)

[0096] y′2=y2+δy (31)

[0097] Using this method, we can calculate the new endpoints of the power line after horizontal and vertical translation in different situations. The corresponding new power line center coordinates can be obtained by averaging the coordinates of the two new endpoints. At the same time, the sine and cosine values ​​of the power line's direction angle can be calculated using the coordinates of the two endpoints before the transformation.

[0098] (2) Reduce to any ratio

[0099] Due to the different heights of drones, the scales of power lines in the images obtained will also vary greatly. In order to generate data of different scales, this embodiment uses a data enhancement method of arbitrary size scaling, such as Figure 4a and Figure 4b shown.

[0100] Figure 4a is the original image. When the image is scaled by the scale factor s (s∈[0,1]), the scaled power lines are as follows Figure 4bAs shown in Figure 2, both ends of the power line are located within the image boundary after scaling. δh and δw are the vertical and horizontal distances between the border of the scaled image and the border of the original image, which can be calculated using equations (32) and (33):

[0101]

[0102]

[0103] Therefore, the coordinates of the endpoints of the transformed power lines in the original image coordinate system can be calculated using equations (34), (35), (36) and (37):

[0104]

[0105]

[0106]

[0107]

[0108] (3) Rotation at any angle

[0109] Since the drone has a yaw angle (yaw) with the direction of the power lines during aerial photography, and the value of the yaw angle may fluctuate within a certain range, this embodiment introduces an arbitrary angle rotation transformation to reduce the fluctuation of the model prediction accuracy caused by the change of the yaw angle. Figures 5a to 5c The process of arbitrary angle rotation transformation of electric lines is demonstrated.

[0110] Figure 5a is the original image. Generally speaking, after a point P(x, y) on the image is rotated by an angle, the corresponding point is P′(x′y′). Then this correspondence can be expressed by formula (38):

[0111]

[0112] In the formula, δ refers to the change, and θ is the direction angle of the line.

[0113] Figure 5b shows the electric lines after rotation, Figure 5c is a picture with changed specifications. Since the size of the image changes after rotation, when the rotated image and the image before rotation are aligned to the same coordinate system, the center of the rotated image is displaced relative to the original image. The sum of the displacement values ​​can be calculated using equations (39) and (40):

[0114]

[0115]

[0116] Therefore, the endpoint coordinates of the power lines after rotation transformation can be calculated according to equations (41), (42), (43) and (44):

[0117]

[0118]

[0119]

[0120]

[0121] (4) Arbitrary shear deformation

[0122] Shear deformation is a commonly used data enhancement method in image recognition. Its main function is to improve the robustness of the model to different perspectives. Figure 6a and Figure 6b A typical shear deformation process is shown.

[0123] For a point on an image, the shear transformation expression is shown in Equation (45), where m is the shear coefficient:

[0124]

[0125] Therefore, the coordinate values ​​of the power line endpoints after shear deformation can be obtained according to equations (46), (47), (48) and (49):

[0126]

[0127]

[0128] y′1=y1 (48)

[0129] y′2=y2 (49)

[0130] Step 3: Model establishment.

[0131] For the task of autonomous navigation along power lines, this embodiment proposes a power line center point and direction recognition model based on deep learning. This model adopts a compact single-step target recognition model structure based on an anchor-free frame architecture similar to CenterNet. Unlike general target detection models, this model can directly predict the center point and direction of the power line, while canceling the recognition of the object target frame. This model structure mainly includes an encoding-decoding module, whose function is to extract image features and generate feature maps required by the prediction module; a prediction module, whose main structure includes three fully convolutional network branches, which respectively predict the Gaussian distribution kernel of the power line center point, the offset of the power line center point relative to the true value, and the direction of the power line center point. Figure 7 The structure of the power line center point and direction recognition model proposed in this embodiment is demonstrated.

[0132] The model input is an infrared RGB image of size (512×512×3). After the encoding-decoding module, a feature map of size (128×128×3) is generated. This feature map is divided into three branches. The first branch consists of a single-channel feature map, which mainly generates a Gaussian distribution kernel for fitting the center of the object. Therefore, the pixel values ​​of this feature map are projected into a numerical space with a value range of [0, 1] through the sigmoid function. Assume that Y is the Gaussian distribution kernel mentioned above, is the coordinate of the center point of the Gaussian kernel, σ p is the standard deviation related to the target box size of the object, then any point (x, y) in this Gaussian kernel can be determined by formula (50):

[0133]

[0134] Where Y xy represents the Gaussian distribution kernel function.

[0135] Therefore, the task of the first branch of the model is to generate a heat map that is consistent with the true value distribution, where the center of the map with power lines has a Gaussian kernel distribution, while the value of the part without power lines is 0.

[0136] Because the heat map output by the model is sized (128×128×1), which is 1 / 4 the size of the original image, upsampling errors occur when the center coordinates of the Gaussian kernel corresponding to the heat map output by the model are restored to the original image. To accurately predict the location of the power line center, this model has a second branch of size (128×128×2). Its main function is to predict the offset between the center coordinates of the Gaussian kernel in the heat map and the actual power line center.

[0137] The model also has a third branch, which is used to predict the sine and cosine values ​​of the power line angle. Since the sine and cosine values ​​range from [-1, 1], this embodiment uses the tanh function to project each pixel value on the feature map output by the third branch of the model into the [-1, 1] region.

[0138] When training the deep learning recognition model, this embodiment uses a corresponding loss function to optimize the model parameters, thereby obtaining a high-precision power line recognition model. For the heat map branch (i.e., the first branch) output by the model, this embodiment uses a focal loss function for classification tasks, as shown in Equation (51):

[0139]

[0140] Where, L hm is the loss function of the Gaussian distribution kernel branch in the model that predicts the center point of the power line, Y xy is the true value, and is the predicted value of the model. α and β are hyperparameters in focal loss. The same settings as CornerNet are used, with α and β set to 2 and 4 respectively. For the power line center offset and power line direction angle prediction branches, this embodiment uses the L1 loss function. The specific calculation formulas are shown in Equations (52) and (53):

[0141]

[0142]

[0143] Where, L off is the loss function of the offset prediction branch (i.e., the second branch) of the center point of the power line relative to the true value, N is the number of samples; L ori It is the loss function of the direction branch (i.e. the third branch) of the predicted power line center point. Pos is the abbreviation of positive, which refers to the positive sample; x and o y is the true center point of the power line, and is the coordinate of the center point of the power line predicted by the model; θ p is the true direction angle of the electric line, and is the direction angle of the electric line predicted by the model.

[0144] Therefore, the final loss function of the model is the weighted sum of the loss functions of the three prediction branches, as shown in formula (54):

[0145] L det =L hm +λL off +γL ori (54)

[0146] Wherein, λ and γ are the weights of the power line center point offset loss function and the power line direction loss function, respectively. In this embodiment, the value of λ is set to 0.1 and the value of γ is set to 1.

[0147] Step 4: Prediction of the center point and direction angle of the power line.

[0148] Since the model used in this embodiment has three branches, when predicting the center point and direction angle of the power line, the model first searches for an area with Gaussian kernel distribution characteristics on the feature map of the power line center prediction branch and selects its center. For multiple Gaussian distribution kernels on the feature map, the model ranks them according to the value of their center points and selects up to the top 100 points as candidate power line center points; at the same time, the model screens the pixel values ​​corresponding to the candidate power line center points and selects the points that are greater than or equal to the threshold T. det The point is taken as the center point of the power line initially predicted by the model. In this embodiment, T det The commonly used empirical value of the target detection model is set to 0.33.

[0149] At the same time, the model searches for the offset of the corresponding coordinate value on the prediction map generated by the corresponding power line center offset prediction branch based on the coordinates of the center point of the initially predicted power line, and obtains the final predicted value of the power line center point coordinate after the offset correction.

[0150] In addition to predicting the center values ​​of power lines, the power line direction prediction branch proposed in this embodiment can also directly predict the sine and cosine values ​​of the direction angle of each power line, corresponding to the power line center points predicted by the other two branches. Therefore, the model proposed in this embodiment can directly predict all power line center points and their corresponding power line directions.

[0151] Since the power line patrol navigation task requires a general overview of the overall position and direction of the power line bundle, all the power line center points and directions predicted by the model need to be clustered to guide the autonomous navigation of the UAV. This embodiment proposes a customized power line center point and direction clustering method based on the characteristics of the power line identification task in the UAV line patrol. First, the minimum enclosing rectangle of all the power line center points predicted by the model is calculated. Then, according to the center position of the minimum enclosing rectangle, a Gaussian kernel is constructed to describe the weight corresponding to the predicted power line. If (x i ,y i ) is the coordinate of the center point of the i-th power line, (x c ,y c ) is the center coordinate of the Gaussian kernel, then the weight value w corresponding to the i-th power line i It can be calculated according to formula (55):

[0152]

[0153] The distribution of power line weights is as follows Figure 8 As shown, it can be seen that the design of this Gaussian kernel is to strengthen the influence of the power lines located in the central area of ​​the power line bundle on the final decision, while also taking into account the positions and directions of other power lines.

[0154] Example 2

[0155] In this embodiment, the present invention proposes a drone line inspection and navigation system based on infrared image power line recognition, which corresponds to the drone line inspection and navigation method based on infrared image power line recognition in Example 1. The drone line inspection and navigation system includes: a mathematical abstraction module, a data annotation and enhancement module, a deep learning recognition model, and a clustering module.

[0156] (1) Mathematical abstraction module: mathematically abstract the overhead power lines as a navigation basis for the UAV flying along the power lines.

[0157] Mathematical abstraction specifically includes averaging the endpoints of the power lines in the drone's bird's-eye view to obtain the center point of the power lines, and using the sine and cosine values ​​of the center point of the power lines and the direction angle of the power lines to represent the power lines.

[0158] (2) Data annotation and enhancement module: Randomly select several aerial photos as training sets and test sets, annotate the coordinate values ​​of each power line endpoint, and perform data enhancement based on the power line endpoints.

[0159] In the data annotation and enhancement module, data enhancement methods include arbitrary direction translation, arbitrary scale reduction, arbitrary angle rotation and arbitrary shear deformation.

[0160] (3) Deep learning recognition model: The model is trained using training and test sets to predict the center point and direction of power lines. The deep learning recognition model includes an encoding-decoding module and a prediction module. The encoding-decoding module extracts image features and generates feature maps required by the prediction module. The prediction module includes three fully convolutional network branches: the first branch predicts the Gaussian kernel of the power line center line and generates a heat map including the power line center point position and Gaussian kernel distribution; the second branch predicts the offset between the center coordinates of the Gaussian kernel in the heat map and the actual power line center. The offset is used to predict the power line center point; the third branch predicts the sine and cosine values ​​of the power line direction angle.

[0161] The input of the deep learning recognition model is a 512×512×3 infrared RGB image, which is passed through the encoding-decoding module to generate a feature map of size 128×128×3; the first branch consists of a single-channel feature map, and the output heat map size is 128×128×1; the second branch size is 128×128×2.

[0162] In the deep learning recognition model, the first branch uses the focal loss function as the loss function:

[0163]

[0164] Where, L hmis the loss function of the first branch, Y xy is the true value, is the predicted value of the deep learning recognition model, α and β are hyperparameters in focal loss, and N is the number of samples;

[0165] The second and third branches use the L1 loss function:

[0166]

[0167]

[0168] Where, L off is the loss function of the second branch, L ori is the loss function of the third branch, Pos represents the positive sample; x and o y is the real center point coordinate of the power line, and is the coordinate of the center point of the power line predicted by the deep learning recognition model, θ p is the true direction angle of the electric line, is the direction angle of the power line predicted by the deep learning recognition model;

[0169] The final loss function of the deep learning recognition model is the weighted sum of the loss functions of the three branches:

[0170] L det =L hm +λL off +γL ori

[0171] Where, L det is the final loss function of the deep learning recognition model, and λ and γ are the weights of the power line center point offset loss function and the power line direction loss function, respectively.

[0172] (4) Clustering module: Cluster the power line center points and power line directions predicted by the deep learning recognition model to obtain the overall position and direction of the power line bundle, which is used to guide the autonomous navigation of the UAV.

[0173] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A drone line patrol navigation method based on infrared image power line recognition, characterized in that: The steps include: Step 1: Perform mathematical abstraction of the overhead power lines and use it as a navigation basis for the UAV to fly along the power lines; Step 2: Randomly select several aerial photos as training and test sets, and mark the coordinate values ​​of each power line endpoint; Step 3: Construct a deep learning recognition model and train the model using a training set and a test set; the deep learning recognition model extracts image features and generates a feature map, searches for regions with Gaussian kernel distribution characteristics on the feature map, selects the center of each region and ranks them according to the size of the center point value, selects the top point as the candidate power line center point, then screens the pixel values ​​of the candidate power line center points to obtain a preliminary predicted power line center point, searches for its offset from the actual power line center based on the coordinates of the preliminary predicted power line center point, and obtains the final predicted value of the power line center point coordinate after correcting the offset; the deep learning recognition model also predicts the direction of each power line; Step 4: Clustering the power line center points and power line directions predicted by the deep learning recognition model to obtain the overall position and direction of the power line bundle, which is used to guide the autonomous navigation of the UAV; Step 4 also includes: calculating the minimum enclosing rectangle of all the power line center points predicted by the deep learning recognition model, and constructing a Gaussian kernel to describe the weight corresponding to the predicted power line according to the center position of the minimum enclosing rectangle, (x i ,y i ) is the coordinate of the center point of the i-th power line, (x c ,y c ) is the center coordinate of the Gaussian kernel, then the weight value w corresponding to the i-th power line i Calculated according to the following formula: The weight distribution of the power lines is thus obtained, and the overall position and direction of the power line bundle are obtained by taking this weight distribution into consideration.

2. The method for unmanned aerial vehicle line inspection and navigation based on infrared image power line recognition according to claim 1, characterized in that: In step 1, mathematically abstracting the overhead power lines includes averaging the endpoints of the power lines in the overhead view of the drone to obtain the center point of the power lines, and using the sine and cosine values ​​of the power line center point and the power line direction angle to represent the power lines.

3. The method for unmanned aerial vehicle line inspection and navigation based on infrared image power line recognition according to claim 1, characterized in that: The step 2 also includes designing a data enhancement method based on the power line endpoints, and the data enhancement method includes arbitrary direction translation, arbitrary scale reduction, arbitrary angle rotation and arbitrary shear deformation.

4. The method for unmanned aerial vehicle line inspection and navigation based on infrared image power line recognition according to claim 1, characterized in that: In step 3, the deep learning recognition model ranks the points according to the size of the center point value and selects up to the top 100 points as candidate power line center points; the deep learning recognition model screens the pixel values ​​of the candidate power line center points and takes the candidate power line center points with pixel values ​​greater than or equal to the threshold as the preliminarily predicted power line center points.

5. A drone line patrol navigation system based on infrared image power line recognition, used to implement the drone line patrol navigation method based on infrared image power line recognition as claimed in claim 1, characterized in that: include: Mathematical abstraction module, data annotation and enhancement module, deep learning recognition model and clustering module; The mathematical abstraction module performs mathematical abstraction on the overhead power lines as a navigation basis for the UAV to fly along the power lines; The data annotation and enhancement module randomly selects several aerial photos as training sets and test sets, annotates the coordinate values ​​of each power line endpoint, and performs data enhancement based on the power line endpoints; The deep learning recognition model is trained using a training set and a test set to predict the center point and direction of the power line. The deep learning recognition model includes an encoding-decoding module and a prediction module. The encoding-decoding module extracts image features and generates a feature map required by the prediction module. The prediction module includes three fully convolutional network branches: the first branch predicts the Gaussian kernel of the power line center line and generates a heat map including the position of the power line center point and the Gaussian kernel distribution; the second branch predicts the offset between the center coordinates of the Gaussian kernel in the heat map and the actual power line center, and the offset is used to predict the power line center point; the third branch predicts the sine and cosine values ​​of the power line direction angle; The clustering module clusters the power line center points and power line directions predicted by the deep learning recognition model to obtain the overall position and direction of the power line bundle, which is used to guide the autonomous navigation of the UAV.

6. The UAV line patrol navigation system based on infrared image power line recognition according to claim 5, characterized in that: The mathematical abstraction module mathematically abstracts the overhead power lines by averaging the power line endpoints in the drone's overhead view to obtain the power line center point, and uses the power line center point and the sine and cosine values ​​of the power line direction angle to represent the power line.

7. The UAV line patrol navigation system based on infrared image power line recognition according to claim 5, characterized in that: In the data annotation and enhancement module, the data enhancement methods include arbitrary direction translation, arbitrary scale reduction, arbitrary angle rotation and arbitrary shear deformation.

8. The UAV line patrol navigation system based on infrared image power line recognition according to claim 5, characterized in that: The input of the deep learning recognition model is a 512×512×3 infrared RGB image, which generates a feature map of size 128×128×3 after passing through the encoding-decoding module; the first branch consists of a single-channel feature map, and the output heat map size is 128×128×1; the second branch size is 128×128×2.

9. The UAV line patrol navigation system based on infrared image power line recognition according to claim 5, characterized in that: In the deep learning recognition model, the first branch uses the focal loss function as the loss function: Where, L hm is the loss function of the first branch, Y xy is the true value, is the predicted value of the deep learning recognition model, α and β are hyperparameters in focal loss, and N is the number of samples; The second and third branches use the L1 loss function: Where, L off is the loss function of the second branch, L ori is the loss function of the third branch, Pos represents the positive sample; x and o y is the real center point coordinate of the power line, and is the coordinate of the center point of the power line predicted by the deep learning recognition model, θ p is the true direction angle of the electric line, is the direction angle of the power line predicted by the deep learning recognition model; The final loss function of the deep learning recognition model is the weighted sum of the loss functions of the three branches: THE det =L hm +λL off +γL ori Where, L det is the final loss function of the deep learning recognition model, and λ and γ are the weights of the power line center point offset loss function and the power line direction loss function, respectively.

Images