Control system for power transmission line monitoring device transporting and loading unmanned aerial vehicle

Abstract

The invention provides a control system for a power transmission line monitoring device transporting and loading unmanned aerial vehicle, which is used for accurate, efficient and autonomous operation of the power transmission line monitoring device transporting and loading unmanned aerial vehicle, adopts a control mode separated from an unmanned aerial vehicle control system, adopts an Arm + FPGA heterogeneous processor as a control core, and integrates various external communication interfaces and motor driving. An algorithm and edge calculation of multi-sensing (vision, laser and imu) data fusion are realized at an Arm Linux end, real-time operation control and data sensing acquisition are realized at an FPGA end, accurate estimation of the pose of an unmanned aerial vehicle can be realized by data fusion of vision and laser radar, efficient control of a motor of a hoisting mechanism is realized by operation control, an independent communication module is provided, and the system is convenient to use. And independent precise operation independent of an unmanned aerial vehicle controller is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the control of unmanned aerial vehicles, in particular to a control system for a power line monitoring device carrying unmanned aerial vehicle. BACKGROUND

[0002] The power line monitoring field has long been plagued by the problem of manual installation of monitoring devices and low automation level. The traditional solution deeply binds the monitoring device control with the unmanned aerial vehicle flight control, resulting in limited deployment of monitoring sensors due to the performance of the unmanned aerial vehicle master control, making it difficult to achieve real-time and precise control of the power line detection and complex actions of the unmanned aerial vehicle carrying mechanism. In the prior art, such as reference patent CN119269954A, although a high-precision perception power line measurement method based on radar and vision fusion is proposed, intelligent perception, automatic detection and prediction, and visual management of the power line are achieved, significantly improving the monitoring accuracy and operation efficiency, but there are deficiencies in independent precise operation of the unmanned aerial vehicle. The coupling degree of the control system and the unmanned aerial vehicle control system is high, and there is a lack of independent data transmission and autonomous operation capability. In addition, reference patent CN119600755A discloses a system for modeling and simulating monitoring of tower landslide disasters based on unmanned aerial vehicles, which improves the monitoring and early warning capability of tower landslide disasters, but has limitations in multi-sensor fusion control and data processing, and fails to achieve efficient and precise operation control. SUMMARY

[0003] The traditional solution deeply binds the monitoring device control with the unmanned aerial vehicle flight control, resulting in limited deployment of monitoring sensors due to the performance of the unmanned aerial vehicle master control, making it difficult to achieve real-time and precise control of the power line detection and complex actions of the unmanned aerial vehicle carrying mechanism. The present application proposes a control system for a power line monitoring device carrying unmanned aerial vehicle, which solves the problem of insufficient computing power of the master control chip and low maintainability and scalability of the entire system in the prior art.

[0004] The technical solution for achieving the purpose of the present application is as follows:

[0005] A control system for a power line monitoring device carrying unmanned aerial vehicle, comprising:

[0006] A control module for processing instructions, real-time control, deep learning and autonomous decision-making;

[0007] A remote terminal for remotely receiving unmanned aerial vehicle aerial video and images and controlling the unmanned aerial vehicle;

[0008] A wireless communication module for providing network access function, realizing information interaction between users and the system, and transmitting the video, images and other data collected by the unmanned aerial vehicle to the remote terminal through the network;

[0009] The flight control module includes a GPS positioning module and a host computer. The GPS positioning module is used to obtain the latitude and longitude information of the drone's location, and the host computer controls the drone's flight path based on the received remote sensing data.

[0010] The motor control module is used to receive signals from the wireless communication module, thereby controlling the motor.

[0011] The attitude calculation module estimates the attitude of the UAV by fusing LiDAR with vision.

[0012] The data processing module is used to process the data transmitted by the wireless communication module;

[0013] The storage module is used to store data locally;

[0014] The flight control module, data processing module, and storage module are connected to the host computer, which is connected to a remote terminal via a wireless communication module.

[0015] Furthermore, the host computer adopts a heterogeneous processor of ARM+FPGA.

[0016] Furthermore, the ARM of the heterogeneous processor includes NPU deep learning and autonomous decision-making algorithms.

[0017] Furthermore, the host integrates multiple external interfaces, through which the FPGA controls external motors.

[0018] Furthermore, the host integrates multiple sensor interfaces, and the ARM can access various sensors through these interfaces.

[0019] Furthermore, the host integrates a gigabit Ethernet port and a high-speed USB port, which are respectively connected to a high-definition camera and a LiDAR.

[0020] Furthermore, it also includes a computing unit running on ARM, which processes the received data and then sends it to the cloud platform via a wireless communication module.

[0021] Furthermore, it also includes an execution unit that processes logic control instructions on the FPGA, wherein the arithmetic processing unit and the execution unit exchange data through memory interaction.

[0022] Furthermore, the processing unit synthesizes the acquired high-definition images and laser data to achieve UAV attitude estimation and target detection, recognition, and classification based on deep learning.

[0023] Furthermore, the wireless communication module adopts a 4G / 5G network.

[0024] Furthermore, the motor control module uses RS485 / CAN bus or pulse to control the external driver.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] 1. Achieve precise, efficient, and autonomous operation. The system adopts a control method separate from the UAV control system, using an Arm+FPGA heterogeneous processor as the control core. On the Arm side, it can combine with an NPU for deep learning, and with multi-sensor data fusion and an independent communication module, it eliminates dependence on the UAV controller and achieves independent and precise operation.

[0027] 2. Improve attitude estimation accuracy. By fusing visual and lidar data and employing algorithms such as Kalman filtering, accurate attitude estimation of the UAV is achieved. Especially when the UAV's IMU is subjected to strong magnetic interference, it can still provide accurate attitude data, ensuring flight and operational stability.

[0028] 3. High-efficiency motor control. The motion control module can efficiently control the motor of the hoisting mechanism, supporting multiple control methods such as RS485 / CAN bus or pulse, and is compatible with various product models, enabling flexible configuration and precise drive.

[0029] 4. Enhance system autonomy and intelligence. Through independent communication modules (such as 5G / 4G), user-system information interaction is achieved, supporting remote terminal control, data transmission, and cloud platform interaction. Simultaneously, multi-sensor data fusion algorithms and edge computing are implemented on the Arm Linux platform, enabling target detection, recognition, and classification based on deep learning, thereby improving the system's intelligent decision-making capabilities.

[0030] Solving existing technical problems:

[0031] (1) It overcomes the problems of insufficient computing power of the main control chip, low system maintainability and scalability in traditional solutions, as well as the defects of high coupling between the control system and the UAV, lack of independent data transmission and autonomous operation capabilities in existing patents.

[0032] (2) The system can adjust the hardware configuration (such as external interface and sensor access) and software algorithm logic according to different task requirements to meet specific functional requirements, and has strong flexibility, scalability and adaptability.

[0033] (3) The hardware architecture integrates multiple communication interfaces and driver modules, and achieves efficient data exchange through memory interaction; the software process covers hardware initialization, data connectivity testing, closed-loop control and other links to ensure stable and reliable operation of the system. Attached Figure Description

[0034] Figure 1 This is a block diagram of the control system structure.

[0035] Figure 2 For control software flowchart.

[0036] Figure 3 This is a flowchart of the lidar sensor fusion ranging and pose output process. Detailed Implementation

[0037] Example 1

[0038] This invention proposes a control system for a UAV used to transport equipment for power transmission line monitoring. This system enables precise, efficient, and autonomous operation of the UAV. It employs a control approach separate from the UAV's control system, using a heterogeneous processor (Arm+FPGA) as the control core and integrating multiple external communication interfaces and motor drives. The system implements multi-sensor (vision, laser, IMU) data fusion algorithms and edge computing on the ArmLinux side, while real-time motion control and data sensing acquisition are achieved on the FPGA side. Data fusion from vision and laser radar enables accurate estimation of the UAV's pose. The motion control system efficiently controls the motors of the hoisting mechanism and includes an independent communication module, enabling independent and precise operation without relying on the UAV controller.

[0039] like Figure 1 As shown, the present invention provides a control system for a UAV used for transporting power transmission line monitoring devices, comprising: a power supply module 1, a central processing unit 3, and attitude estimation module 10, wireless module 9, peripheral sensor module 8, data acquisition and execution unit 5, memory unit 4, flight control module 7, and motor control module 6, all connected to the central processing unit 3. The attitude estimation module 10 includes a camera and a lidar, used for data fusion to acquire the attitude information of the UAV, and capable of target recognition and other calculations based on an ARM platform. The wireless module 9 is used to receive control data from a remote terminal. It can transmit image sensing data back to a remote terminal, and can also perform trajectory mapping and aerial photography; the motor control module 6 is used to control the motor of the transport device, and can support bus and pulse control to achieve flexible configuration; the central computing unit 3 is composed of an ARM processor, which includes an NPU to realize deep learning operations; the data acquisition and execution unit 5 is used to realize motor-assisted control of the transport mechanism and various sensor signals such as proximity switches; the attitude estimation module 10 realizes the UAV's attitude estimation by fusing visual and laser sensor data through Kalman filtering, and provides attitude data when the UAV's IMU is subjected to strong magnetic interference.

[0040] The central processing unit 3, memory unit 4, and data acquisition and execution unit 5 are integrated into the CPU hardware architecture. The data acquisition and execution unit 5, based on an FPGA chip, mainly implements the motor control function in the power transmission line monitoring device, including periodic task scheduling, pulse output service, IO signal acquisition service, CAN and RS485 bus output service. The motor control is compatible with various models of similar products. Its GPIO ports are directly brought out after isolation and protection, and wide voltage options are available. The memory unit 4 has 128MB of FLASH used to store the boot loader program and interactive data. The Ethernet port is configured with a gigabit RJ45 for connecting a high-definition camera or laser sensor; a high-speed USB for wireless communication; an RS232 and an RS485 for data access from other sensors; and an SPI port for data interaction with the flight controller, providing the flight controller with estimated UAV attitude.

[0041] like Figure 2 As shown, the implementation flow of the control system is as follows:

[0042] First, hardware initialization and module self-test are performed to conduct an initial check on the system hardware and each module to ensure that they can work normally.

[0043] Conduct data connectivity tests to verify whether data transmission between the controller and components such as flight controller, sensors, and memory is smooth.

[0044] The system performs pose estimation by fusing camera and laser data, and outputs the processed pose information to the flight controller for flight control-related decisions.

[0045] Perform other sensor data acquisition and logic execution, collect more sensor data and process it according to the set logic.

[0046] It wirelessly receives commands from a remote terminal and obtains control instructions from the remote terminal.

[0047] Based on instructions and system status, the motor is driven to move, while receiving feedback information from sensors to form a closed-loop control.

[0048] Based on the data collected by the sensors, including the pose data of the drone, the pose data of the detection device, the pose data of the drone relative to the power transmission line, and the encoder data of the device, deep learning and autonomous decision-making training can be performed on the NPU to control the autonomous movement of the transport device.

[0049] Combination Figure 3 The system uses a combination of camera and lidar sensors to detect the distance between a drone and a power transmission line. When the distance is less than 1 meter, it activates pose output. The specific scheme for camera and lidar sensor fusion ranging and pose output is as follows:

[0050] Point cloud data is acquired at a frequency of 10-30Hz, and each point cloud data can be represented as a combination of three-dimensional coordinates and reflection intensity:

[0051]

[0052] in, For spatial coordinates, Reflection intensity, This represents the number of point clouds.

[0053] The binocular camera acquires left and right views at frequencies above 30Hz. and Depth information is obtained through the principle of binocular parallax; parallax With depth The relationship is:

[0054]

[0055] in, For camera focal length, The baseline distance for the binocular camera.

[0056] For LiDAR, dynamic point cloud preprocessing is first performed, and a statistical filtering algorithm is used to calculate the mean of the local neighborhood (K nearest neighbors, K = 50) of the point cloud. and standard deviation Eliminate those that meet the requirements outliers

[0057]

[0058]

[0059] in, The distance from a point to the mean of its neighborhood. and These are points in the point cloud data. The voxel size and side length are dynamically adjusted based on the point cloud density. The calculation is as follows:

[0060]

[0061] in This represents the local point cloud density. , .

[0062] Next, we designed an adaptive point cloud convolutional network. Based on PointNet++, we designed an adaptive convolutional module that dynamically adjusts the convolutional kernel size according to the point cloud density. :

[0063]

[0064] in, Indicates rounding down. For the maximum convolution kernel size, Minimum density threshold This represents the density of a local region in the current point cloud. Global features of the point cloud are extracted using a multilayer perceptron (MLP) and max pooling operations. :

[0065]

[0066] For binocular cameras, feature extraction and semantic segmentation are performed. Feature extraction: The ORB feature algorithm is used to extract FAST corner points (corner response). ,filter (points), and calculate the BRIEF descriptor:

[0067]

[0068] in, For predefined sampling pairs, This represents the pixel grayscale value.

[0069] Semantic segmentation is performed on stereo images using U-Net, and the model is trained using the cross-entropy loss function.

[0070]

[0071] in, and For the image height and width, For the category predicted by the model, This is the true category.

[0072] Next, time synchronization and spatial registration are performed. Based on sensor timestamps, linear interpolation is used to align the LiDAR and stereo camera data. Then the dot cloud is aligned. for:

[0073]

[0074] Using hand-eye calibration to obtain the extrinsic parameter matrix of the LiDAR and camera (rotation matrix) Translation vector Transform the lidar point cloud to the camera coordinate system:

[0075]

[0076] Then, edge federated learning is performed to fuse the local model training with the aligned LiDAR features input from the UAV. Visual semantic features Fusion through a fully connected layer:

[0077]

[0078] in, This is the weight matrix. This is the bias vector.

[0079] The FedAvg algorithm is used for model parameter aggregation, and the global model parameter update formula is:

[0080]

[0081] in, These are the globally updated model parameters. For the i-th client (drone), the updated parameters are... Let be the number of samples for the i-th client. This represents the total number of samples.

[0082] The fusion weights of lidar and vision are dynamically adjusted according to environmental conditions. and :

[0083]

[0084] Regression prediction is performed on the fused feature F_{final}, and the distance prediction model is trained using the L2 loss function:

[0085]

[0086] in, To predict distance, This represents the actual distance.

[0087] Simultaneously, the pose is calculated and output based on the utility pole as a reference.

[0088] In summary, the present invention provides a control system for a UAV used to transport a power transmission line monitoring device, comprising a UAV hardware system and a software system.

[0089] The hardware system of this solution includes the drone body equipped with sensors such as cameras and lidar, and a remote terminal for the ground workstation.

[0090] The software system component of this solution includes algorithm programs and motion control programs.

[0091] In use, the online training algorithm program is installed on the central processing unit equipped with an NPU to train the deep learning model. After that, the image is preprocessed in real time and the sensor fusion of vision and laser is realized to achieve the pose estimation of the UAV.

[0092] The motion control program, implemented through logic and sensor data, achieves efficient motion control of the transport mechanism to accomplish power line inspection tasks.

[0093] Example 2

[0094] This embodiment uses a single high-performance processor (such as the NVIDIA Jetson series) instead of the Arm+FPGA architecture in Embodiment 1.

[0095] This is achieved using high-performance processors with integrated GPUs and CPUs (such as the NVIDIA Jetson AGX Orin), leveraging their multi-core CPUs and deep learning accelerators (such as Tensor Cores):

[0096] (1) CPU part: Processing algorithms such as multi-sensor data fusion (vision, laser, IMU), target detection and recognition.

[0097] (2) GPU / NPU part: Real-time pose estimation and deep learning inference (such as target detection model) are achieved through hardware acceleration, replacing the real-time operation and control function of FPGA.

[0098] The processor uses its built-in USB, Ethernet, CAN and other interfaces to connect to sensors (cameras, LiDAR) and motor drive modules.

[0099] Develop multithreaded programs under Linux systems, leveraging frameworks such as CUDA / OpenCL to accelerate sensor data fusion and machine learning algorithms. Ensure the real-time performance of motor control commands through a real-time operating system (RTOS) or priority scheduling mechanism (e.g., using the ROS2 framework for task scheduling).

[0100] The 5G / 4G wireless communication module is retained to enable remote data transmission and control, and to interact independently with the UAV flight control system via serial port or network interface.

[0101] This approach may have weaker real-time performance than FPGA (because FPGA processes data in parallel through hardware logic, while CPU / GPU relies on software scheduling), requiring optimization of algorithms and task priorities to ensure real-time motor control. Its anti-interference capability may also be inferior to FPGA (e.g., the stability of attitude data in strong magnetic environments), necessitating compensation through software filtering algorithms.

[0102] Example 3

[0103] This embodiment uses a heterogeneous architecture of DSP+MCU to replace Arm+FPGA in embodiment 1.

[0104] The DSP is responsible for real-time signal processing (such as LiDAR point cloud filtering and visual feature extraction) and attitude estimation algorithms (Kalman filtering, etc.), while the MCU is responsible for system control, sensor data acquisition (IMU, GPS), and motor drive logic (via PWM or bus protocols). Sensors and actuators are connected via interfaces such as SPI, I2C, and CAN, and data is cached using external memory (such as DDR3).

[0105] The DSP runs real-time signal processing algorithms, while the MCU runs system management and communication protocols (such as MQTT for cloud platform connections). Data interaction between the DSP and MCU is achieved through shared memory or interrupt mechanisms, such as transmitting attitude estimation results to the MCU for motor control. The vision + LiDAR fusion scheme can be retained, or replaced with millimeter-wave radar + vision (reducing costs and suitable for non-high-precision scenarios).

[0106] This solution has relatively weak deep learning inference capabilities (requiring a lightweight model or external NPU chip), which may reduce object detection accuracy. Its multi-task parallel processing capabilities are limited, and complex algorithms (such as 3D point cloud segmentation) may impact real-time performance.

[0107] Example 4

[0108] This embodiment adopts a purely software-defined solution (relying on cloud computing power + lightweight sensors), including:

[0109] Lightweight processors, such as the Raspberry Pi 4B (Arm architecture), are used to handle only data acquisition and communication, without performing complex calculations. A monocular camera, IMU, and GPS are used, employing visual SLAM algorithms (such as ORB-SLAM3) for pose estimation, replacing LiDAR. 5G and edge servers (such as local MEC nodes) are used to upload image data to the cloud for processing.

[0110] A lightweight data acquisition program runs locally, transmitting visual / IMU data to a cloud server via 5G. The cloud server utilizes a high-performance GPU cluster to perform multi-sensor fusion, target detection, and motor control command generation, which is then transmitted to the drone via 5G. After receiving the cloud commands, the local processor drives the motor via PWM or bus protocol, forming a closed loop of "cloud decision-making + local execution".

[0111] This solution relies on the stability of the 5G network, and network latency may cause control lag (such as affecting accurate operation if it exceeds 50ms).

[0112] When the network is disconnected, it cannot operate autonomously and loses its system independence, which does not meet the invention goal of "not relying on the drone controller".

[0113] In the above embodiments, the "high-performance single processor (such as NVIDIA Jetson)" and "millimeter-wave radar + vision fusion" solutions can achieve the core objective of this invention to a certain extent, but compromises must be made on real-time performance or accuracy. The "DSP + MCU architecture" and "pure cloud solution," due to limitations in computing power or autonomy, are only suitable for simplified scenarios. In contrast, the original Arm + FPGA heterogeneous architecture has comprehensive advantages in computing power allocation, real-time performance, and anti-interference capabilities, and remains the optimal solution. If cost reduction is required, a sensor optimization scheme replacing lidar with millimeter-wave radar can be considered, but it is necessary to verify whether the accuracy meets the requirements for power transmission line operations.

[0114] Addressing the functional and performance requirements of existing UAV applications for power transmission lines, this application's system design achieves control of the UAV hoisting device and can provide accurate attitude data even after the flight control gyroscope is subjected to strong magnetic interference. It features stability, reliability, high flexibility, and high scalability. The system's hardware configuration and software algorithm logic can be adjusted according to different mission requirements to achieve specific functional needs. The technological innovation of this application lies in not only improving the automation and operational accuracy of UAVs used for transporting power transmission line monitoring devices but also enhancing the system's autonomy and intelligence, providing an efficient, safe, and reliable solution for the field of power transmission line monitoring.

[0115] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0116] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A control system for a UAV used to transport power transmission line monitoring devices, characterized in that, It comprises: a control module for processing instructions, real-time control, deep learning and autonomous decision-making; a remote terminal for remotely receiving unmanned aerial vehicle aerial video and images and controlling the unmanned aerial vehicle; a wireless communication module for providing network access function, realizing information interaction between users and the system, and transmitting the video, images and other data collected by the unmanned aerial vehicle to the remote terminal through the network; a flight control module comprising a GPS positioning module and a host, the GPS positioning module being used to obtain the longitude and latitude information of the location of the unmanned aerial vehicle, and the host being used to control the flight route of the unmanned aerial vehicle according to the received remote sensing data; a motor control module for receiving signals from the wireless communication module to control the electric motor; a posture solving module for estimating the posture of the unmanned aerial vehicle through laser radar fusion vision; a data processing module for processing the data transmitted by the wireless communication module; a storage module for locally storing data; wherein the flight control module, data processing module and storage module are connected together with the host, and the host is connected with the remote terminal through the wireless communication module.

2. The control system for a power line monitoring device carrying drone according to claim 1, wherein, The host adopts an ARM+FPGA heterogeneous processor.

3. The control system for a power line monitoring device delivery drone of claim 2, wherein, The ARM of the heterogeneous processor comprises an NPU deep learning and autonomous decision-making algorithm.

4. The control system for a power line monitoring device delivery drone of claim 1, wherein, The host integrates multiple external interfaces, and the FPGA controls external motors through the interfaces.

5. The control system for a power line monitoring device delivery drone of claim 1, wherein, The host integrates multiple sensor interfaces, and the ARM accesses multiple sensors through the multiple sensor interfaces.

6. The control system for a power line monitoring device delivery drone of claim 1, wherein, The host integrates a gigabit Ethernet port and a high-speed USB port, which respectively access a high-definition camera and a laser radar.

7. The control system for a power line monitoring device carrying drone of claim 2, wherein, It further comprises an operation processing unit running on the ARM, which is used to perform operation processing on the received data and send the processed data to the cloud platform through the wireless communication module.

8. The control system for a power line monitoring device delivery drone of claim 7, wherein, It further comprises an execution unit for processing logical control instructions on the FPGA, and the operation processing unit and the execution unit exchange data through memory interaction.

9. The control system for a power line monitoring device delivery drone of claim 1, wherein, The operation processing unit synthesizes the collected high-definition images and laser data to realize unmanned aerial vehicle posture estimation and target detection, recognition and classification based on deep learning.

10. The control system for a power line monitoring device delivery drone of claim 1, wherein, The wireless communication module adopts 4G / 5G network.

11. The control system for a power line monitoring device delivery drone of claim 1, wherein, The motor control module adopts RS485 / CAN bus or pulse to control external drivers.

Images