An Unmanned Aerial Vehicle Infrared Autonomous Inspection Method for Transmission Lines Based on Optical Flow and Kalman Filtering

Abstract

The present invention discloses an infrared autonomous inspection method for transmission lines by drones based on optical flow and Kalman filtering, including: predicting the target box of the transmission line on the image by a target detection model; calculating the average optical flow vector of the pixel points within the target box; dividing the image into a target box area R1 and a non-target box area R2, obtaining the corner coordinates of the weighted average target box, and then calculating the center of the weighted average target box; obtaining the average optical flow vector [u c , v c of the pixel points within the target box of the current frame, and the center coordinates [x k , y k of the target box of the current frame predicted by Kalman filtering, and then performing weighted averaging on them to obtain the final center coordinates of the target box of the transmission line; using the PID algorithm to control the camera and continuously track the transmission line. The present invention fuses multiple information for transmission line tracking: tracking the transmission line by combining the target box predicted by the target detection model, the optical flow method, and Kalman filtering.

Description

Technical Field

[0001] The present invention belongs to the technical field of power transmission line inspection, and particularly relates to an unmanned aerial vehicle (UAV) infrared autonomous inspection method for power transmission lines based on optical flow and Kalman filtering. Background Art

[0002] In order to ensure the reliability of the power transmission line system and avoid problems such as large-area power outages caused by equipment damage in the power transmission line, power operation units usually adopt the method of inspection to regularly inspect the equipment and accessories in the outdoor power transmission line system, timely discover problems and potential safety hazards, so as to facilitate timely maintenance and ensure the stable operation of the power transmission system. Compared with the tasks of the power line remote monitoring system, the inspection scope of the power line inspection is very large, the tasks are more extensive, and the difficulty is greater. There are many types of components to be inspected in the power line inspection task, the distribution environment is relatively complex, and it is also affected by weather.

[0003] Early manual inspection has the characteristics of low efficiency and poor safety, and has gradually been replaced by the vision-based UAV power line inspection scheme. In the UAV power line inspection scheme, the main problem to be solved is how to ensure that the UAV can efficiently complete the inspection task. In order to enable the UAV to more effectively identify and conduct risk investigation on important facilities such as power lines and power transmission lines during flight, a more effective navigation method is to make the UAV fly above the power line and always keep following the power line. During the flight, the power line is inspected. When the front camera detects the power transmission line, the power transmission line is tracked, and the camera angle is adjusted for real-time shooting. After completing the inspection task, the UAV can also return along the power line. Due to the complex actual distribution of power lines, in some areas, the power line information is not accurately drawn into the navigation map, and it is difficult to achieve accurate navigation relying on GPS information, and it is difficult to ensure that the camera captures clear power lines and their accessories; at the same time, the GPS information in some areas is relatively weak, and there may sometimes be a situation of signal loss. Therefore, the reliability of relying solely on GPS navigation is not high. And the vision-based navigation along the power line relies on the accurate identification of the power line, so it is more reliable and accurate than GPS navigation.

[0004] When the UAV flies along the power line, it should have good power transmission line recognition ability. As the UAV approaches the power transmission line, the camera carried by the UAV will adjust its perspective following the power transmission line, so that the power transmission line can fall into the center of the camera's field of view as much as possible. Due to factors such as camera photographing and processing, the lag of the pan-tilt movement, the self-vibration of the UAV platform and the camera pan-tilt, and the influence of wind in the environment, the camera is prone to losing the target when tracking the power transmission line. Therefore, corresponding control algorithms are needed to optimize the control of the camera and enhance the robustness of power transmission line tracking. When the UAV flies over the top of the power transmission line, the shooting angle of the camera is already in the vertically downward direction, and the projection of the power transmission line in the camera has a certain difference from the image taken laterally. Subsequently, when the UAV flies forward after flying over the power transmission line, the camera still tracks the power transmission line to keep taking pictures until the power transmission line exceeds the shooting angle range of the camera.

[0005] Therefore, the power transmission line tracking technology based on infrared assistance is of great significance in the power line inspection work of UAVs. After the UAV identifies the power transmission line, it needs to continuously track the power transmission line during the subsequent flight and keep the shooting angle of the camera in a better position during the tracking process, so as to obtain more power transmission line photo data for further analysis and processing. Keeping track of the power transmission line can improve the work efficiency of UAV inspection and reduce the probability of missed inspection. Identifying power lines is an essential task in the autonomous navigation of UAVs for night power line inspection based on infrared images. If the UAV can identify the position and orientation of the power line, it can adjust its own flight direction and position in real time according to this information and keep consistent with the path of the power line. Regarding the problem of power line identification in UAV inspection, there have been many related studies. For example, Burns et al. (1986) proposed a method to identify power lines through edge detection and pixel clustering. Akinlar et al. (2011) proposed an edge contour line detection method called EDlines. Ceron et al. (2014) proposed a UAV power line detection model based on the circle search technique (CBS). Based on CBS, Ceron et al. (2018) further proposed a power line identification method based on the histogram of oriented segments (HOS). However, these methods are not stable and reliable enough when dealing with UAV aerial images in complex backgrounds. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an infrared autonomous inspection method for UAVs on transmission lines based on optical flow and Kalman filtering, a target detection method based on deep learning, which combines the optical flow method and the Kalman filtering algorithm to continuously track the position of the transmission line, so as to guide the camera to adjust the angle and take more sufficient pictures of the power line, and realize the automatic identification and following of the power line by the UAV in a complex environment.

[0007] To achieve the above technical objectives, the technical solution adopted by the present invention is as follows:

[0008] An infrared autonomous inspection method for UAVs on transmission lines based on optical flow and Kalman filtering, including:

[0009] Step S1: Obtain the virtual infrared dataset and the real virtual infrared dataset of the UAV on the transmission line, establish and train a target detection model for transmission line recognition based on deep learning, and predict the target box of the transmission line on the image;

[0010] Step S2: Use the optical flow method to calculate the average optical flow vector of the pixel points within the target box according to the information of the previous frame and the current frame;

[0011] Step S3: Divide the image area along the direction of the optical flow vector with the corner points of the target box, divide the image into the target box area R1 and the non-target box area R2, assign a larger weight W1 to the target box area R1, and assign a smaller weight W2 to the non-target box area R2. After obtaining the weighted average corner point coordinates of the target box, calculate the weighted average center of the target box;

[0012] Step S4: Obtain the average optical flow vector [u c , v c of the pixel points within the target box of the current frame, and the center coordinates [x k , y k of the target box of the current frame predicted by Kalman filtering, and then perform weighted averaging on them to obtain the center coordinates of the final target box of the transmission line;

[0013] Step S5: Based on the center coordinates of the target box of the transmission line, use a computer-based proportional-integral-derivative (PID) control algorithm to control the camera, so that the camera can automatically adjust the shooting angle during the process of the UAV flying over the transmission line and continuously track the transmission line.

[0014] To optimize the above technical solution, the specific measures taken also include:

[0015] The above-mentioned step S1 of establishing and training a target detection model for transmission line recognition based on deep learning specifically includes:

[0016] Step (1): Establish an object detection model for transmission line recognition based on deep learning. The object detection model is the transmission line recognition model:

[0017] The object detection model includes a deep learning-based encoding module and a decoding module;

[0018] The encoding module is responsible for extracting transmission line features, and the decoding module is responsible for restoring the target box information of the transmission line from the features;

[0019] The same encoding module and decoding module are used for the real transmission line dataset and the virtual transmission line dataset, and the weights are shared between them;

[0020] Step (2): Train the object detection model in an alternating training manner:

[0021] First, input the virtual infrared dataset into the object detection model to obtain a complete object detection model. The encoder module in it can achieve good feature extraction for the data in the virtual infrared dataset, and the decoding module can better restore the features extracted by the encoder to the center position and target box information of the transmission line. Subsequently, fix the decoder module of the object detection model, and train the object detection model with a small amount of real infrared dataset and various types of data augmentation techniques;

[0022] Step (3): Generate the center position and target box information of the transmission line through the previously fixed decoder module, and train the decoder module with the real infrared dataset and various data augmentation techniques to improve the generalization ability of the decoder module for real data. Alternately iterate these two processes, and use the virtual infrared dataset and the real infrared dataset respectively until the object detection model finally converges on the real infrared dataset.

[0023] The above object detection model also has a transfer learning layer, and the middle layer of the decoder is connected to the transfer learning layer through a cross-layer structure. During transfer learning training, the model first fixes the weights of the encoding module and the decoding module, and uses a mixture of the virtual infrared dataset and the real infrared dataset to perform various data augmentations on the real infrared dataset to achieve model training and iteration. Until the model converges, then open the weights of the encoder module and the decoder module to perform the final training of the model to obtain the best weights.

[0024] In the above step S2, for a frame of image, the object detection model based on deep learning predicts the target box of the transmission line on the image, and the coordinates of the upper left and lower right corner points are (x1, y1) and (x2, y2). Let I(x i , y i ) be the coordinates (x i , y i) A pixel point within the target box, and its corresponding optical flow vector is [u i , v i . Then, the average optical flow vector of the pixel points within the target box is calculated by Equation (10):

[0025]

[0026] In the above step S3, assume that the target detection model has identified a total of m target boxes, and the corner coordinates of their upper left and lower right corners are (x j1 , y j1 ) and (x j2 , y j2 ). Then, the corner coordinates of the upper left and lower right corners of the predicted target box of the target detection model, (x m1 + x m1 ) and (x m2 + x m2 ), are calculated by Equations (11), (12), (13), and (14):

[0027]

[0028]

[0029]

[0030]

[0031] After obtaining the corner coordinates of the weighted average target box, the center point of the weighted average target box is calculated according to Equations (15) and (16):

[0032]

[0033]

[0034] In the above step S4, the center coordinates of the transmission line target box are calculated according to Equations (17) and (18):

[0035]

[0036]

[0037] Among them, (x D , y D ) is the center coordinate of the target box obtained by weighted averaging the target box of the target detection model with the regional weights W1 and W2;

[0038] W D is the weight of the prediction result of the target detection model;

[0039] (xO , y O is the center coordinate value of the target box predicted by the optical flow method;

[0040] W O is the weight of the optical flow method prediction result;

[0041] (x K , y K ) is the center coordinate of the target box predicted by the Kalman filter algorithm;

[0042] W k is the weight of the Kalman filter algorithm prediction structure.

[0043] In the above step S5, the difference between the center horizontal and vertical coordinates of the transmission line target box obtained in step S4 and the true value is used as the PID closed-loop control term to perform real-time control on the pitch angle pitch and yaw angle yaw of the camera, so that the transmission line camera can fall into the central area of the camera's field of view during the process of the UAV flying over the transmission line.

[0044] The present invention has the following beneficial effects:

[0045] 1. The present invention fuses multi-information for transmission line tracking: through the target box predicted by the target detection model, the optical flow method and the Kalman filter are combined to track the transmission line. Compared with the traditional target tracking method, the present invention initializes the optical flow method and the Kalman filter with the target box coordinate prediction value of the deep learning model, and combines the optical flow method and the Kalman filter to enhance the robustness of target tracking, reduce the probability of target loss, and is more stable than using the optical flow method and the Kalman filter method alone;

[0046] 2. Compared with the target tracking method based on target detection, the method of target detection combined with optical flow and Kalman filter adopted by the present invention fully considers the correlation relationship between objects in consecutive frames, and effectively enhances the stability of transmission line tracking;

[0047] 3. Considering that during the line inspection flight of the UAV, the main attention needs to be concentrated on the transmission line on the flight path, the present invention introduces a transmission line attention mechanism based on the UAV flight path. Based on the identification of the transmission line of the UAV, the optical flow is calculated, and the flight direction of the UAV is estimated according to the optical flow. Through region division, different weights are assigned to the middle region and the two side regions along the flight direction of the UAV, and the identified target boxes are weighted and averaged according to the weights to reduce the influence of some interference factors in the non-central region on the transmission line tracking;

[0048] 5. The method proposed by the present invention is based on a simulator and a relatively small-scale real infrared dataset to achieve the training and optimization of the model, complete the accurate identification of other unknown transmission line data, and transfer the knowledge learned by the model from the virtual infrared dataset to the real infrared dataset based on the idea of meta-learning;

[0049] 6. A transmission line tracking method based on the transmission line identification model, combined with the optical flow method and the Kalman filter algorithm, performs real-time tracking of the transmission line according to the temporal relationship, enhances the coherence of the transmission line identification in time series, reduces misjudgment and missed judgment, and further improves the accuracy and robustness of the transmission line identification and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a target detection model diagram provided by an embodiment of the present invention;

[0051] Figure 2 It is a brightness diagram of pixel points in different frames provided by an embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of the Kalman filtering process provided by an embodiment of the present invention;

[0053] Figure 4 It is a diagram of a multi-information fusion transmission line tracking method provided by an embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of vision-based PID control provided by an embodiment of the present invention;

[0055] Figure 6 It is a flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The following further describes the embodiments of the present invention in detail with reference to the drawings.

[0057] As Figure 6 shown, an unmanned aerial vehicle (UAV) infrared autonomous inspection method for transmission lines based on optical flow and Kalman filtering according to the present invention, using a method of combining target detection with optical flow and Kalman filtering, fully considers the correlation relationship between objects in consecutive frames, effectively enhances the stability of transmission line tracking, has high efficiency, and is convenient for completing UAV inspection navigation, including:

[0058] Step S1: Obtain a virtual infrared dataset and a real virtual infrared dataset of the UAV for transmission lines, establish and train a target detection model for transmission line identification based on deep learning, and predict the target box of the transmission line on the image;

[0059] In step S1, a target detection model for transmission line recognition is established based on few-shot learning; based on a simulator and a relatively small-scale real infrared dataset, the target detection model is trained and optimized to accurately recognize other unknown transmission line data; based on the idea of meta-learning, the knowledge learned by the target detection model from the virtual infrared dataset is transferred to the real infrared dataset.

[0060] The specific steps of step S1 are as follows:

[0061] The Airsim simulation platform can be used to control the drone in the scene to fly along the power line and fly over the transmission line, which is very similar to real applications. By controlling the drone in the virtual environment to take pictures, a large number of real-scene images and semantic segmentation maps containing transmission lines can be obtained. A total of 9000 groups of transmission line image data are collected.

[0062] The idea of meta-learning is adopted, which is characterized by the model learning the basic knowledge in target recognition. For example, through the basic network, the basic features and position localization methods of the transmission line are learned from the virtual infrared dataset. Then, based on the learned knowledge, the model is fine-tuned according to the data in a small amount of real infrared datasets, so that the model can quickly adapt to new data and perform inference.

[0063] Few-shot learning has the characteristic of requiring a small amount of data and has very important research value in some fields where data acquisition is difficult.

[0064] The target detection model of the present invention is constructed based on the network structure of CenterNet, such as Figure 1 .

[0065] The model consists of an encoding module and a decoding module. The encoding module extracts the object features in the image, while the decoding module reorganizes the features in the image and finally outputs a heat map reflecting the center of the object. At the same time, two output layers of center offset and the length and width of the target box are generated. For the transmission line recognition task, the finally output heat map has only one layer. If the pixel value of a certain point on the heat map is higher than the pixel values of the surrounding 8 adjacent points, then this point is the potential center point of the object.

[0066] The same encoding module and decoding module are used for the real infrared dataset and the virtual infrared dataset, and they share weights. When training the network, the present invention refers to the characteristics of the generative adversarial network and introduces the mutual competition and corresponding relationship between the real infrared dataset and the virtual infrared dataset.

[0067] The object detection model includes a deep learning-based encoding module and a decoding module. The encoding module is responsible for extracting the characteristics of the transmission line, while the decoding module is responsible for restoring the target box information of the transmission line from the characteristics. During model training, a large number of virtual infrared datasets and a small number of real infrared datasets are used respectively. When training the model, an alternating training method is adopted. First, the virtual infrared dataset is input into the model, and a complete object detection model is obtained. The encoder module in it can achieve good feature extraction for the data in the virtual infrared dataset, and the decoding module can better restore the features extracted by the encoder into the center position and target box information of the transmission line. Subsequently, the decoder module of the model is fixed, and the model is trained with a small number of real infrared datasets and various types of data augmentation techniques. The purpose is to enable the encoder module in the object detection model to not only extract the characteristics of the transmission line in the virtual infrared dataset but also effectively extract the characteristics of the transmission line in the real infrared dataset. By sharing the decoder module, the encoder can generalize the data in the virtual and real infrared datasets. When the model is trained, the encoder module can accurately extract the characteristics of the transmission line in the virtual infrared dataset and the real dataset, and generate the center position and target box information of the transmission line through the previously fixed decoder module. To enhance the ability of the decoding module to generate the center and target box of the transmission line in the real infrared dataset, the weights of the encoder module are fixed at this time, and the decoder module is trained with the real infrared dataset and various data augmentation techniques to improve the generalization ability of the decoder module for real data. These two processes are iterated alternately, and virtual datasets and real infrared datasets are used respectively until the model finally converges on the real infrared dataset.

[0068] After the above training process is completed, the model has the ability to recognize the transmission lines in the virtual infrared dataset and the real infrared dataset.

[0069] To further improve the accuracy of the model, a transfer learning layer is proposed based on the existing model, and the intermediate layer of the decoder is connected to the transfer learning layer through a cross-layer structure.

[0070] The transfer learning layer is mainly composed of five convolutional layer modules. Each convolutional layer module contains a convolutional operation, a batch normalization operation, and an activation operation. A branch is introduced from the intermediate layer of the decoder in the model. The feature map is upsampled through a transposed convolution, and then fused with the feature map generated by the first convolutional layer module of the transfer learning layer in an element-wise addition manner.

[0071] During transfer learning training, the model first fixes the weights of the encoding module and the decoding module. By mixing the virtual infrared dataset and the real infrared dataset, various data augmentations are performed on the real infrared dataset, so that the real infrared dataset and the virtual infrared dataset are mixed into a batch of data in a 1:1 ratio, and the model is trained and iterated until the model converges. Then, the weights of the encoder and decoder modules are unlocked, and the model is finally trained to obtain the best weights.

[0072] Step S2: Using the dense optical flow method, calculate the dense optical flow based on the previous frame image and the current frame, and calculate the average optical flow of the pixel points within the target box;

[0073] The dense optical flow method performs dense matching on the image. By estimating the offsets of all pixel points, a dense optical flow field distribution is formed to meet more accurate target tracking tasks.

[0074] When studying how to use optical flow to track objects, the present invention is mainly based on the following three assumptions:

[0075] (1) The pixel brightness of the target in the image does not change significantly between adjacent frames;

[0076] (2) The time interval between adjacent frames is short enough, and the pixel points between adjacent frames do not change significantly in position;

[0077] (3) Adjacent pixel points on the image have similar motions;

[0078] Based on the first assumption, for a pixel point in the current frame whose coordinate value is (x, y), then at the next moment, it moves to and its coordinate value is updated to (x + dx, y + dy), as Figure 2 shown:

[0079] The brightness invariance of the pixel point between adjacent frames can be expressed by Equation (1):

[0080] I(x, y, t) = I(x + dx, y + dy, t + dt) (1)

[0081] Performing a Taylor series expansion on the right side of Equation (1) gives Equation (2):

[0082]

[0083] where I represents the brightness value of the pixel point, x, y, and t represent the horizontal and vertical coordinates and the time moment of the pixel point respectively, and ε is a higher-order infinitesimal that can be ignored. Rearranging (2) and dividing by dt gives Equation (3):

[0084]

[0085] Let It can be seen that \(u\) and \(v\) are the velocities of the pixel point in the \(x\)-axis and \(y\)-axis directions respectively. Let and Then Equation (3) can be written as the optical flow equation (4):

[0086] f x u + f y u + f t u = 0 (4)

[0087] \(u\) and \(v\) are solved by the dense optical flow method;

[0088] The dense optical flow method uses a method of dense matching of images. By estimating the offsets of all pixel points, a dense optical flow field distribution is formed to meet the more accurate target tracking task.

[0089] Similar to the sparse optical flow method, the objective function of the dense optical flow method is also to make the corresponding pixel points between adjacent frames after registration as accurate as possible.

[0090] Assume that in two consecutive frames of images: the previous frame image \(I_1\) and the current frame image \(I_2\), for the pixel point \(a\) with coordinates \((x a , y a ) in \(I_1\), there is a corresponding pixel point \(b\) in \(I_2\), and its coordinates are \((x b , y b ). The coordinates of \(a\) and \(b\) satisfy (5) and (6):

[0091] x b = x a + u (5)

[0092] y b = y a + v (6)

[0093] Then its optimization function:

[0094]

[0095] where \(\lambda\) is a proportionality coefficient to balance the influence of the two objective loss terms on the overall objective function.

[0096] Step S3: Divide the image region along the direction of the optical flow vector with the corner points of the target box, divide the image into the target box region \(R_1\) and the non-target box region \(R_2\), and assign a larger weight \(W_1\) to the target box region \(R_1\), while assign a smaller weight \(W_2\) to the non-target box region \(R_2\). After obtaining the corner point coordinates of the weighted average target box, calculate the center coordinates of the weighted average transmission line target box;

[0097] The Kalman Filter is a recursive filter, which is mainly a method for estimating the state of a dynamic system through a series of system input-output observation data that may contain noise. The Kalman Filter can predict the value at the next moment based on the measured values at different moments. Figure 3 The process of the Kalman Filter is shown as follows:

[0098] It can be seen that the main steps of the Kalman Filter method are to obtain the general estimate x at the current moment based on the optimal estimate x in the previous state k-1 and predict the optimal estimated value x at the current moment according to the current observation value x k-1 k-1 k-1 .

[0099] The Kalman Filter consists of two parts. The first part is the linear system state prediction equation, and the second part is the linear system observation equation, as shown in Equations (8) and (9) respectively.

[0100] x k = Ax k-1 + Bu k-1 + w k-1 (8)

[0101] z k = Hx k + v k (9)

[0102] Where x k and x k-1 are the state value target position vectors at the kth and (k - 1)th moments (x K , y K ), u k-1 is the system input at the (k - 1)th moment, z k is the observed value of the system at the kth moment, and w k-1 and v k are the system input noise and observation noise respectively. They are both Gaussian white noises that follow N(0, 1) and are independent of each other.

[0103] Based on the above method, a transmission line tracking method with multi-information fusion is proposed. Through the target bounding boxes predicted by the target detection model, the optical flow method and Kalman filter are combined to track the transmission line. Compared with traditional target tracking methods, the present invention initializes the optical flow method and Kalman filter with the predicted coordinate values of the target bounding boxes of the deep learning model, and combines the optical flow method and Kalman filter to enhance the robustness of target tracking, reduce the probability of target loss, and is more stable than using the optical flow method and Kalman filter method alone; compared with the target tracking method based on target detection, the method of combining target detection with optical flow and Kalman filter adopted by the present invention fully considers the correlation relationship of objects between consecutive frames, and effectively enhances the stability of transmission line tracking. Considering that during the line inspection flight of the unmanned aerial vehicle, the main attention needs to be concentrated on the transmission line on the flight path, the present invention introduces a transmission line attention mechanism based on the flight path of the unmanned aerial vehicle. Based on the identification of the transmission line by the unmanned aerial vehicle, the optical flow is calculated, and the flight direction of the unmanned aerial vehicle is estimated according to the optical flow. Through region division, different weights are assigned to the middle region and the two side regions along the flight direction of the unmanned aerial vehicle, and the identified target bounding boxes are weighted and averaged according to the weights to reduce the influence of some interference factors in the non-central region on the transmission line tracking.

[0104] As shown above, steps S1 - S2: For a frame of image, the target detection model based on deep learning will predict the target bounding box of the transmission line on the image, and the coordinates of the upper left and lower right corner points are (x1, y1) and (x2, y2). At the same time, the dense optical flow can be calculated based on the previous frame image and the current frame, and the average optical flow of the pixel points within the target bounding box is calculated.

[0105] Let I(x i ,y i ) be a pixel point with coordinates (x i ,y i ), and its corresponding optical flow vector is [u i ,v i . Then the average optical flow vector within the target bounding box can be calculated by Equation (10):

[0106]

[0107] After obtaining the average optical flow vector of the target bounding box, the present invention divides the image region along the direction of the optical flow vector of the corner points of the target bounding box, divides the image into the target bounding box region R1 and the non-target bounding box region R2, and assigns a larger weight W1 to the target bounding box region R1, while assigns a smaller weight W2 to the non-target bounding box region R2.

[0108] Assume that the target detection model has identified a total of m target bounding boxes, and the coordinates of the upper left and lower right corner points are (x j1 ,y j1 ) and (xj2 , y j2 ), then the corner coordinates (x m1 + x m1 ) and (x m2 + x m2 ) of the upper left and lower right corners of the predicted target box of the object detection model can be calculated by Equations (11), (12), (13), and (14):

[0109]

[0110]

[0111]

[0112]

[0113] After obtaining the corner coordinates of the weighted average target box, the center point of the weighted average target box can be calculated according to Equations (15) and (16):

[0114]

[0115]

[0116] Step S4: At the same time, based on the information of the previous frame and the current frame, the optical flow vector [u c , v c of the current frame and the Kalman filter prediction [x k , y k of the center point of the target box of the current frame can be obtained. The present invention uses a method of weighted averaging these information to obtain the center point of the final object target box, as Figure 4 shown:

[0117] The center coordinates of the final transmission line target box can be calculated according to Equations (17) and (18):

[0118]

[0119]

[0120] Among them, (x D , y D ) is the center coordinate of the target box obtained by weighted averaging the target box of the object detection model based on deep learning with the region weights W1 and W2, W D is the weight of the prediction result of the object detection model, (x O , y O ) is the center coordinate value of the target box predicted by the optical flow method, W O is the weight of the prediction result of the optical flow method, (xK , y K ) are the center coordinates of the target box predicted by the Kalman filter algorithm, and W k is its corresponding weight. The final predicted value of the center coordinates of the target box is the weighted average of the predicted values of the three methods of target recognition, optical flow method, and Kalman filter.

[0121] Step S5: Based on the above transmission line tracking method, the present invention uses a computer-based proportional-integral-derivative control (PID) method to control the camera in a simulation environment through the results of transmission line recognition and tracking, so that the camera can automatically adjust the shooting angle during the process of the drone flying over the transmission line and continuously track the transmission line. The main control elements and processes are as Figure 5 shown:

[0122] It can be seen that the present invention uses the differences between the center horizontal and vertical coordinates of the transmission line obtained by target recognition and tracking and the true values as the PID closed-loop control terms respectively, and controls the pitch angle and yaw angle of the camera in real time, so that the transmission line can fall into the central area of the camera's field of view.

[0123] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, several improvements and refinements made without departing from the principle of the present invention should be regarded as within the protection scope of the present invention.

Claims

1. An infrared autonomous inspection method for UAVs on transmission lines based on optical flow and Kalman filtering, characterized in that, Including: Step S1: Obtain the virtual infrared dataset of the transmission line UAV and the real virtual infrared dataset, establish and train an object detection model for transmission line recognition based on deep learning, predict the target box of the transmission line on the image, and obtain the target box coordinates (x D , y D ) output by the object detection model; Step S2: Calculate the average optical flow vector of pixel points within the target box using the optical flow method; Step S3: According to the average optical flow vector [u c , v c of the pixel points within the current frame target box, divide the image area along the direction of the optical flow vector from the corner points of the target box, divide the image into the target box area R1 and the non-target box area R2, and assign a larger weight W1 to the target box area R1, while assign a smaller weight W2 to the non-target box area R2. After obtaining the corner point coordinates of the weighted average target box, calculate the center of the weighted average target box, which is the center coordinate value (x O , y O ) predicted by the optical flow method; Step S4: The center coordinates [x k , y k of the target box in the current frame are predicted through Kalman filtering, and then the center coordinates of the target box of the transmission line are finally obtained by weighted averaging of the center coordinate values of the target box output by the object detection model and predicted by the optical flow method; Step S5: Based on the center coordinates of the final transmission line target box, use the PID algorithm to control the camera, enabling the camera to automatically adjust the shooting angle during the process of the drone flying over the transmission line and continuously track the transmission line; In step S3, assume that the target detection model identifies a total of m target bounding boxes, and the corner coordinates of the upper left corner and the lower right corner are (x j1 , y j1 ), and (x j2 , y j2 ). Then, the corner coordinates (x m1 , y m1 ) and (x m2 , y m2 ) of the predicted target bounding box of the target detection model are calculated by equations (11), (12), (13), and (14): After obtaining the corner coordinates of the weighted average target box, calculate the center point of the weighted average target box according to Equations (15) and (16): Obtained from equations (15) and (16) That is, the x coordinate value of the center of the target box predicted by the optical flow method (x O , y O ) is the x O , y O .

2. The method for infrared autonomous inspection of transmission lines by an unmanned aerial vehicle based on optical flow and Kalman filtering according to claim 1, wherein, The target detection model for transmission line recognition established and trained based on deep learning described in Step S1 specifically includes: Step (1): Establish a target detection model for transmission line recognition based on deep learning. The target detection model is the transmission line recognition model: The target detection model includes a deep learning-based encoding module and a decoding module; The encoding module is responsible for extracting transmission line features, and the decoding module is responsible for restoring the target box information of the transmission line from the features; Use the same encoding module and decoding module for the real transmission line dataset and the virtual transmission line dataset, and share the weights between them; Step (2): Train the target detection model in an alternating training manner: First, input the virtual infrared dataset into the target detection model and obtain a complete target detection model. The encoder module in it can achieve good feature extraction for the data in the virtual infrared dataset, and the decoding module can better restore the features extracted by the encoder to the center position and target box information of the transmission line. Subsequently, fix the decoder module of the target detection model and train the target detection model with a small amount of real infrared dataset and various types of data augmentation techniques; Step (3): Generate the center position and target box information of the transmission line through the previously fixed decoder module. Use the real infrared dataset and cooperate with various data augmentation techniques to train the decoder module to improve the generalization ability of the decoder module for real data. Alternately iterate these two processes, and use the virtual infrared dataset and the real infrared dataset respectively until the target detection model finally converges on the real infrared dataset, obtaining the target box coordinates (x D , y D ) output by the target detection model.

3. The method for infrared autonomous inspection of transmission lines by UAV based on optical flow and Kalman filtering according to claim 2, wherein The target detection model is also provided with a transfer learning layer, and the middle layer of the decoder is connected to the transfer learning layer through a cross-layer structure. During transfer learning training, the model first fixes the weights of the encoding module and the decoding module, uses a mixture of the virtual infrared dataset and the real infrared dataset, performs various data augmentations on the real infrared dataset, and realizes model training and iteration. After the model converges, then open the weights of the encoder module and the decoder module and perform the final training on the model to obtain the best weights.

4. A method for infrared autonomous inspection of transmission lines by an unmanned aerial vehicle based on optical flow and Kalman filtering according to claim 1, characterized in that, In the step S2, for a frame of image, the target detection model based on deep learning predicts the target box of the transmission line on the image, and the coordinates of the upper left corner and the lower right corner are (x1, y1) and (x2, y2). Let I(x i , y i ) be a pixel point in the target box with coordinates (x i , y i ), and its corresponding optical flow vector is [u i , v i . Then the average optical flow vector of the pixel points in the target box is calculated by Equation (10): The u and v obtained from Equation (10) form the average optical flow vector [u c , v c .

5. A method for infrared autonomous inspection of transmission lines by an unmanned aerial vehicle based on optical flow and Kalman filtering according to claim 1, characterized in that, In Step S4, the center coordinates of the transmission line target box are calculated according to Equations (17) and (18): Among them, (x D , y D ) is the center coordinate of the target box obtained by weighted averaging the target box of the target detection model with the regional weights W1 and W2, and is output by step S1 based on the deep learning target detection model; W D is the weight of the prediction result of the object detection model; (x O , y O ) is the predicted center coordinate value of the target box by the optical flow method; W O is the weight of the prediction result of the optical flow method; (x K , y K ) are the center coordinates of the target bounding box predicted by the Kalman filter algorithm; W k is the weight of the prediction structure of the Kalman filter algorithm.

6. The method for infrared autonomous inspection of transmission lines by an unmanned aerial vehicle based on optical flow and Kalman filtering according to claim 1, characterized in that In Step S5, use the differences between the horizontal and vertical coordinates of the center of the transmission line target box obtained in Step S4 and the real values as the PID closed-loop control terms to perform real-time control on the pitch angle and yaw angle of the camera, so that the transmission line camera can fall into the central area of the camera's field of view during the process of the drone flying over the transmission line.

Images