UAV Inspection System for Substation Buildings Based on Machine Vision and Multi-Sensor Fusion

Abstract

The present invention proposes an unmanned aerial vehicle (UAV) inspection system for substation buildings based on machine vision and multi-sensor fusion, which includes a micro UAV with machine vision. The micro UAV takes off and lands at a helipad equipped with a wireless charging module and a power management module. The inspection system further includes edge devices provided on the helipad, and positioning identification codes that can be visually recognized by the micro UAV and are pasted beside the areas, instruments, and indicator lights to be inspected. The positioning identification codes are used to guide the micro UAV to reach the inspection targets. The edge devices include edge computing devices and edge control devices, which fuse the IMU attitude trajectory data transmitted back during UAV inspection and the coordinate data of UAV visual positioning to form fused positioning data. When the information for UAV visual positioning is lost, the edge devices actively intervene and obtain the UAV coordinates through the fused positioning data. The present invention can achieve three-dimensional and non-blind-spot coverage inspection of substation buildings.

Description

Technical Field

[0001] The present invention relates to the technical field of inspection of distribution substations, in particular to an unmanned aerial vehicle inspection system for distribution substations based on machine vision and multi-sensor fusion. Background Art

[0002] The power distribution system transports electricity from the transmission system to the required areas. As the central link, distribution substations are the core of regional power supply and play an important role in maintaining daily electricity use. Once management and operation and maintenance problems occur in distribution substations, huge economic losses and safety accidents will be caused, thus affecting the safe operation of the power system. Therefore, regular inspection of distribution substations has always been an essential task for maintaining the stable operation of distribution substations in the past few years.

[0003] The statistical report of the China Electricity Council shows that in 2022, the number of substations exceeded 30,000. At the same time, estimating based on an average of 1,000 distribution substations in each prefecture-level city, more than 300,000 distribution substations are owned by 297 prefecture-level and above cities (including 4 municipalities directly under the Central Government) across the country. And with the gradual expansion of the power grid scale, the number of distribution substations will continue to increase. Power companies generally require manual inspection operations for the distribution substations and switch rooms under their jurisdiction, with a monthly routine inspection once and a monthly infrared temperature measurement inspection once. During high-load periods and when there are special power supply guarantee requirements, the inspection frequency will be increased according to specific needs. There are many items in the manual routine inspection of distribution substations, and it takes an average of 40 minutes to inspect one site. At the same time, distribution substations are numerous and widely distributed, and during the inspection process, inspectors will spend a lot of time on the road.

[0004] In recent years, thanks to the development of microelectromechanical control technology and machine vision software and hardware technology, stable micro quadcopters have received extensive attention and have very promising application prospects. At the same time, the Internet of Things field is developing towards the direction of micro-level chips, low power consumption and low cost. Many sensors such as infrared array sensors, micro high-resolution vision sensors, and gas detection sensors have emerged, which are very suitable for being carried on micro quadcopters to expand the sensing range of unmanned aerial vehicles. Due to its small size, high flexibility, low cost, wide sensing range and other characteristics, micro unmanned aerial vehicles are very suitable for realizing fully autonomous automatic inspection in distribution substations with neatly arranged switch cabinets and fixed inspection routes, thus solving a series of problems brought by traditional manual inspection.

[0005] The current automatic inspection system for distribution substations mainly includes fixed monitoring cameras, limited fixed sensors, overhead rail-type inspection robots, mobile inspection robots on the ground, and unmanned aerial vehicles (UAVs) mounted on a mobile robot platform and connected by a telescopic cable for inspection at different heights. Among them, the monitoring method combining fixed monitoring cameras and limited sensors is affected by the installation location and cannot comprehensively cover all equipment without dead angles; the overhead rail robots require prior installation of inspection tracks, with poor flexibility, maintainability, and high costs; the mobile robots on the ground are limited by height and cannot inspect equipment beyond their preset height range, so the upper space of the distribution substation can only be inspected manually; the inspection UAVs mounted on a mobile robot platform and connected by a telescopic cable are large in size, complex in structure, and high in cost.

[0006] Therefore, in view of the above problems, there is an urgent need for a micro UAV autonomous inspection system for distribution substations. Summary of the Invention

[0007] The present invention proposes a UAV inspection system for distribution substations based on machine vision and multi-sensor fusion. The present invention does not rely on satellite data. The UAV can perform reciprocating inspection flights in an environment without satellite positioning signals (such as indoors), and the inspection route is accurate. It can also realize the navigation of the UAV's vertical flight. In the inspection task of the distribution substation, the system has functions of autonomous takeoff and landing, real-time positioning, wireless charging and power management, edge computing analysis, image recognition, meter reading, switch state recognition, infrared temperature measurement, toxic gas monitoring, intelligent alarm, video monitoring, data transmission back, and remote control.

[0008] The present invention adopts the following technical solutions.

[0009] A UAV inspection system for distribution substations based on machine vision and multi-sensor fusion. The inspection system includes one or more micro UAVs equipped with forward machine vision and top-down machine vision. Each micro UAV is equipped with a memory card and low-power environmental sensors of the required types according to the tasks it performs; the micro UAV takes off and lands at a helipad equipped with a wireless charging module and a power management module; the inspection system also includes edge devices provided on the helipad, and positioning identification codes that can be visually recognized by the micro UAV and are pasted beside the areas, meters, and indicators to be inspected; the positioning identification codes are used to guide the micro UAV to reach the inspection targets, including takeoff guide codes, landing guide codes, and inspection codes.

[0010] The edge device includes an edge computing device and an edge control device, which fuse the IMU attitude trajectory data transmitted back during the UAV inspection and the coordinate data of the UAV vision positioning to form fused positioning data; when the information for the UAV vision positioning is lost, the edge device actively intervenes, adaptively adjusts the weight information of the two through the fused positioning data, and eliminates outliers through a data prediction algorithm to obtain the accurate position coordinates of the UAV.

[0011] The usage method of the inspection system includes the following steps;

[0012] Step S1: When starting the inspection, the apron carrying the edge computing device first performs a self-check, analyzes the historical log information, the current UAV battery power, and the hardware status information. After confirmation, according to different inspection tasks issued by the upper computer, it selects and starts the micro UAV equipped with corresponding sensors according to the priority to start the inspection task;

[0013] Step S2: After the apron issues the inspection task, it commands the micro UAV to take off to the initial inspection altitude, and detects the takeoff and landing guidance code through the downward-looking machine vision camera to obtain the initial coordinate position of the UAV; and judges the position and attitude data of the UAV according to the attitude trajectory data of the inertial measurement unit (IMU) and the fusion positioning code.

[0014] Step S3: The UAV, according to the position information pre-stored in the inspection task and in cooperation with the positioning identification codes pasted at the key positions of the inspection route, obtains the position coordinates through real-time visual calculation on the UAV and transmits them back to the edge computing and control device. The edge device obtains the coordinate information of the current UAV relative to the starting point of the inspection route through the transformation of the position coordinates.

[0015] Step S4: When the UAV is performing inspection, the IMU attitude data of the UAV is synchronously transmitted back to the edge device to establish an odometer database, which is fused with the coordinate data obtained by the edge device through the UAV machine vision positioning. When the UAV machine vision positioning information is lost, the edge device actively intervenes, adaptively adjusts the weight information of the two, and eliminates outliers through a data prediction algorithm to obtain the accurate position coordinates of the UAV.

[0016] Step S5: The edge computing device combines the real-time calculated UAV position information and controls the attitude and movement of the UAV through a cascade PID algorithm. Guide the UAV to accurately reach the target to be detected;

[0017] Step S6: When the UAV reaches the inspection position, it uses the forward machine vision camera to record the inspection area image in real time, combines the video frame, time stamp and position information to form inspection data, and synchronously stores it in the on-board memory card;

[0018] Step S7: The drone repeats the positioning and inspection storage work of the above steps until it reaches the last inspection target and identifies the inspection end identification code;

[0019] Step S8: After completing the inspection of all targets to be inspected, the edge computing device performs backtracking based on the trajectory data in the historical log records combined with the real-time fusion positioning data, controls the drone to perform inspection again in the opposite direction according to the above steps, and generates a backup inspection video frame dataset until it returns to the initial position at takeoff, that is, the position of the takeoff and landing guidance code;

[0020] Step S9: After reaching the initial position, that is, the position of the apron, the drone uses machine vision from above to identify the positioning code in the apron, and the edge device performs real-time control and guidance on the drone to achieve precise landing of the drone at the base of the apron;

[0021] Step S10: After the drone has come to a stop, the apron turns on the power management through the contact electrodes at the base. The edge computing device sends a data reading instruction, and according to the time stamp, obtains the video frame data recorded during the inspection image and video stream shooting from the on-board memory card into the edge computing device.

[0022] In Step S10, the edge computing device identifies the readings of the instrument meters, switch states, and indicator light states in the inspection range through calculation. The method includes the following steps;

[0023] Step A1: The edge computing device reads key frames from the inspection image and video stream, and screens clear key picture frames according to the fuzzy detection algorithm;

[0024] Step A2: The key picture frames are sent into a pre-trained object detection model after image correction to locate the positions of the dial and switch in the image;

[0025] Step A3: In the operation of reading the meter readings, the positioned dial will use a semantic segmentation model to further segment the pointers and scales of each meter, and calculate the readings of each meter according to the relative position of the pointers and the known range;

[0026] Step A4: In the monitoring of the switch state, according to the key picture frames, the pre-trained object monitoring model directly returns the corresponding switch state category and the corresponding indicator light state.

[0027] In Step S10, the edge computing device judges outliers based on the calculated readings of the instrument meters, switch states, and indicator light states to automatically give an alarm; at the same time, it docks with the cloud control platform to summarize data in real time and perform remote visualization display; at the same time, the edge computing device receives remote control instructions in real time to control the next inspection task of the drone.

[0028] The edge device has a data interface for joint debugging with the automation devices in the substation building, so that the inspection system information can be interacted in real time and accurately. At the same time, according to the instruction release of the timing task, the inspection work of the UAV is repeated, so as to realize the unmanned and intelligent inspection and management of the substation building space.

[0029] When the environment of the substation building is abnormal, the edge computing and control device schedules different functional micro UAVs equipped with infrared temperature sensors and toxic gas sensors respectively according to the task sequence, conducts inspections according to the above inspection process respectively, stores the corresponding sensor data and uploads it to the edge computing device for calculation, and conducts intelligent alarms for corresponding tasks in case of data anomalies.

[0030] When the UAV is inspecting, the edge computing device uses an event-driven method to realize the event response control and resource management of the multi-task system. In the multi-task system driven by inspection events, tasks are assigned priorities in advance, and tasks with higher priorities will be executed first. During the inspection process, when an emergency event is marked in the interrupt, if the priority of the task corresponding to the event is high enough, it will be immediately responded to.

[0031] The fused positioning data is the fused positioning data of IMU and machine vision. The method of fusing IMU and machine vision is as follows: the edge computing device continuously receives the IMU attitude data and visual positioning coordinate data of the UAV, establishes an odometer according to the attitude data, calculates the coordinates of the UAV relative to the starting point of the inspection path according to dead reckoning, and fuses them with the visual positioning coordinates. However, due to the influence of cumulative error in the IMU sensor, that is, the cumulative error will increase with the increase of time. When the visual position information exists, with the increase of time, the confidence of the system in the visual positioning result will gradually increase, and at the same time the confidence in the IMU system will gradually decrease, as shown in the following formula:

[0032] Formula 1;

[0033] In x seconds, IMU will decay at a slope of 1 / x with the change of time until the moment before the stable visual coordinates are detected. At this time, the confidence of the inertial navigation is regarded as 0, and the inspection system no longer believes the positioning result of the IMU. When the stable visual coordinates are detected, the inspection system clears the data of the inertial navigation, clears the cumulative error, and repeats the above process until the navigation ends.

[0034] When the UAV locates through machine vision by positioning identification codes, the following method is adopted: the micro UAV extracts the distortion parameters of the camera used for machine vision by the on-board MCU, corrects the UAV camera and detects candidate markers;

[0035] When the machine vision of the drone is turned on, the drone MCU analyzes the camera images in real time to find squares that are candidates for markers, segments the markers using adaptive thresholding, then extracts the contours from the thresholded images, and discards those contours that are not convex or not close to squares. After candidate detection, it determines whether they are really markers by analyzing their internal codes, analyzes the number of black or white pixels in each cell to determine whether it is a white bit or a black bit, and determines whether the marker belongs to a specific dictionary through these bits; thus accurately judging the parameter information of the identification code. After that, coordinate transformation is performed to obtain the current coordinate information of the inspection drone relative to the identification code, and the coordinate information is transmitted to the edge computing device in a real-time wireless transmission manner.

[0036] The inspection system includes a front-end visualization display system developed based on Thing JS to realize the statistics and analysis of data of several power distribution substations in the urban area.

[0037] The present invention is a special device applicable to autonomous inspection in power distribution substations; compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. A newly proposed inspection system integrating a micro-drone and sensors solves the dead-angle problem and inspection problems at different heights existing in traditional inspections. Its flexible and mobile morphological characteristics can achieve three-dimensional and dead-angle-free coverage inspection of power distribution substations;

[0039] 2. The equipment system is simply deployed. Utilizing the morphological characteristics of the micro-drone and the spatial characteristics of the regular arrangement of power distribution substation equipment, it is very easy to plan and deploy inspection routes and migrate them to other power distribution substations;

[0040] 3. The positioning scheme of the micro-drone is mainly based on vision and supplemented by IMU odometer fusion positioning, with the characteristics of high positioning accuracy and no positioning dead angles, greatly ensuring the real-time position control of the drone;

[0041] 4. The micro-drone has the characteristics of strong mobility and fast moving speed, and only stays briefly at the position of the target to be inspected to record video stream data during the inspection process. Therefore, the inspection coherence of the whole system is strong, with high efficiency and fast speed;

[0042] 5. The cost of the whole system is low, the modular design has low coupling, is easy to maintain and replace, and is flexibly expandable. Multiple different micro-drones can carry corresponding sensors for different tasks respectively;

[0043] 6. The localized edge computing control device integrates four major functions: local drone management, real-time attitude control of inspection drones, analysis and processing of image data, and remote task reception. Data storage and analysis are both local, with high security and fast response speed;

[0044] 7. The intelligent helipad can obtain the battery status of the micro-unmanned aerial vehicle (UAV) in real time and manage the charging and discharging of the UAV through contact electrodes, greatly improving the endurance of the micro-UAV.

[0045] 8. The edge computing device has the characteristics of strong scalability, realizes the visual cloud display of key information, and can achieve real-time linkage with on-site automation devices through reserved data communication interfaces.

[0046] In the present invention, after the inspection of all targets to be inspected is completed, the edge computing device will perform backtracking based on the trajectory data in the historical log records combined with the real-time fusion positioning data, and control the UAV to perform inspection again in the opposite direction according to the above steps to generate a backup inspection video frame data set. In this process, since the UAV does not need to fly while shooting the positioning identification code, it can return at a faster flight speed, improving the operation efficiency and saving power. At the same time, due to the different technical methods used for the return flight control, the backup inspection video frame data set captured during the return flight can be used as the verification data for the inspection video.

[0047] In the present invention, since the UAV attitude data and sticker two-dimensional code information are used for positioning, it does not rely on satellite data. The UAV can perform reciprocating inspection flights in an environment without satellite positioning signals (such as indoors), and the inspection route is accurate. It can also realize the navigation of the UAV's vertical flight. The UAV can identify the instrument data vertically distributed in the cabinet during the vertical flight. Description of the Drawings

[0048] The following further details the present invention in conjunction with the drawings and specific embodiments:

[0049] Att Figure 1 is a schematic structural diagram of the micro-UAV in the present invention;

[0050] Att Figure 2 is a schematic diagram of the inspection system architecture of the present invention;

[0051] Att Figure 3 is a schematic diagram of the principle of IMU and vision fusion positioning in the present invention;

[0052] In the figure: the camera 101 for forward machine vision, the camera 102 for top-down machine vision, the infrared sensor 103, the extended communication interface 104, the contact charging electrode 105, the fuselage protective cover 106, the IMU and on-board computing and storage device 107. Detailed Embodiments

[0053] As shown in the figure, an unmanned aerial vehicle (UAV) inspection system for a distribution substation based on machine vision and multi-sensor fusion. The inspection system includes more than one micro UAV equipped with forward machine vision and downward machine vision. Each micro UAV is equipped with a memory card and low-power environmental sensors of the required types according to the tasks it performs. The micro UAV takes off and lands at a helipad equipped with a wireless charging module and a power management module. The inspection system also includes edge devices installed on the helipad, and positioning identification codes that can be visually recognized by the micro UAV and are pasted beside the areas, instruments, and indicator lights to be inspected. The positioning identification codes are used to guide the micro UAV to reach the inspection targets, including a takeoff guide code, a landing guide code, and an inspection code.

[0054] The edge devices include an edge computing device and an edge control device, which fuse the IMU attitude trajectory data transmitted back during UAV inspection and the coordinate data of UAV visual positioning to form fusion positioning data. When the information for UAV visual positioning is lost, the edge devices actively intervene, adaptively weight the information of the two through the fusion positioning data, and eliminate outliers through a data prediction algorithm to obtain the accurate position coordinates of the UAV.

[0055] The usage method of the inspection system includes the following steps;

[0056] Step S1: When starting the inspection, the helipad carrying the edge computing device first starts self-checking, analyzes the historical log information, the current battery power of the UAV, and the hardware status information. After confirming that there is no error, according to different inspection tasks issued by the upper computer, it selects and starts the micro UAV equipped with the corresponding sensors according to the priority to start the inspection task.

[0057] Step S2: After the helipad issues the inspection task, it commands the micro UAV to take off to the initial inspection altitude, and detects the takeoff and landing guide codes through the camera 102 of the downward machine vision to obtain the initial coordinate position of the UAV. And it judges the position and attitude data of the UAV according to the attitude trajectory data of the inertial measurement unit (IMU) and the fusion positioning code.

[0058] Step S3: The UAV, according to the position information pre-stored in the inspection task, cooperates with the positioning identification codes pasted at the key positions of the inspection route, visually calculates the position coordinates in real time on the UAV and transmits them back to the edge computing and control device. The edge device obtains the coordinate information of the current UAV relative to the starting point of the inspection route by transforming the position coordinates.

[0059] Step S4: During the UAV inspection, the IMU attitude data of the UAV is synchronously transmitted back to the edge device to establish an odometer database, which is fused with the coordinate data obtained by the edge device through UAV machine vision positioning. When the UAV machine vision positioning information is lost, the edge device actively intervenes, adapts the weight information of the two, and eliminates outliers through a data prediction algorithm to obtain the accurate position coordinates of the UAV;

[0060] Step S5: The edge computing device combines the real-time calculated UAV position information and controls the attitude and movement of the UAV through a cascade PID algorithm, guiding the UAV to accurately reach the target to be detected;

[0061] Step S6: When the UAV reaches the inspection position, the camera 101 of the forward machine vision records the images of the inspection area in real time. The video frames, timestamps and position information are combined to form inspection data, which are synchronously stored in the on-board memory card;

[0062] Step S7: The UAV repeats the positioning and inspection storage work of the above steps until it reaches the last inspection target and identifies the inspection end identification code;

[0063] Step S8: After completing the inspection of all targets to be inspected, the edge computing device performs a backtracking based on the trajectory data in the historical log record combined with the real-time fusion positioning data, and controls the UAV to perform another inspection in the reverse direction according to the above steps to generate a backup inspection video frame dataset until it returns to the initial position at takeoff, that is, the position of the takeoff and landing guidance code;

[0064] Step S9: After reaching the initial position, that is, the position of the apron, the UAV identifies the positioning code in the apron through top-down machine vision, and the edge device performs real-time control and guidance on the UAV to achieve accurate landing of the UAV at the base of the apron;

[0065] Step S10: After the UAV stops stably, the apron turns on the power management through the contact electrodes at the base. The edge computing device sends a data reading instruction and obtains the video frame data recorded by the inspection image video stream during the inspection from the on-board memory card to the edge computing device according to the timestamp.

[0066] In Step S10, the edge computing device identifies the readings of the instrument meters, switch states, and indicator light states in the inspection range through calculation. The method includes the following steps;

[0067] Step A1: The edge computing device reads key frames from the inspection image video stream and screens clear key picture frames according to the fuzzy detection algorithm;

[0068] Step A2: After image correction, the key picture frames are sent into a pre-trained target detection model to locate the positions of the dial and switch in the image;

[0069] Step A3: In the operation of reading the meter, the positioned dial will use a semantic segmentation model to further segment the pointers and scales of each meter, and calculate the readings of each meter according to the relative positions of the pointers and the known ranges.

[0070] Step A4: In the monitoring of the switch state, according to the key picture frames, directly return the corresponding switch state category and the corresponding indicator light state through a pre-trained target monitoring model.

[0071] In step S10, the edge computing device judges outliers based on the meter readings, switch states, and indicator light states of the calculation results to automatically give an alarm; at the same time, it connects to the cloud control platform to summarize data in real time and perform remote visual display; at the same time, the edge computing device receives remote control instructions in real time to control the next inspection task of the drone.

[0072] The edge device has a data interface for joint debugging with the automation equipment in the substation building, so that the inspection system information can be interacted in real time and without error; at the same time, according to the instruction release of the scheduled task, the inspection work of the drone is repeatedly executed, so as to realize the unmanned and intelligent inspection and management of the substation building space.

[0073] When the environment of the substation building is abnormal, the edge computing and control device schedules different functional micro-drones equipped with infrared temperature sensors and toxic gas sensors according to the task sequence, conducts inspections according to the above inspection process respectively, stores the corresponding sensor data and uploads it to the edge computing device for calculation, and gives intelligent alarms for corresponding tasks in case of data anomalies.

[0074] During the drone inspection, the edge computing device uses an event-driven method to realize the event response control and resource management of the multi-task system. In the event-driven multi-task system of the inspection, tasks are assigned priorities in advance, and tasks with higher priorities will be executed first; during the inspection process, when an emergency event is marked in the interrupt, if the priority of the task corresponding to the event is high enough, it will be immediately responded to.

[0075] The fused positioning data is the fused positioning data of IMU and machine vision; the method of fusing IMU and machine vision is as follows: the edge computing device continuously receives the IMU attitude data and visual positioning coordinate data of the drone, establishes an odometer according to the attitude data, calculates the coordinates of the drone relative to the starting point of the inspection path according to dead reckoning, and fuses them with the visual positioning coordinates. However, due to the influence of cumulative error in the IMU sensor, that is, the cumulative error will increase with the increase of time. When the visual position information exists, as time increases, the confidence of the system in the visual positioning result will gradually increase, and at the same time the confidence in the IMU system will gradually decrease, as shown in the following formula:

[0076] Formula 1;

[0077] Within x seconds, the IMU decays at a slope of 1 / x with the change of time until the moment before the stable visual coordinates are detected. At this time, the confidence level of the inertial navigation is regarded as 0, and the inspection system no longer believes the positioning result of the IMU. When the stable visual coordinates are detected, the inspection system clears the data of the inertial navigation, clears the accumulated error, and repeats the above process until the navigation ends.

[0078] When the UAV locates by machine vision through the positioning identification code, the following method is adopted: the micro UAV extracts the distortion parameters of the camera used for machine vision by the on-board MCU to correct the UAV camera and detect candidate markers;

[0079] When the machine vision of the UAV is turned on, the UAV MCU analyzes the camera image in real time to find the squares as marker candidates, segments the markers using an adaptive threshold, then extracts the contours from the threshold image, and discards those contours that are not convex or not close to a square. After candidate detection, it is determined whether they are really markers by analyzing their internal codes, analyzes the number of black or white pixels in each cell to determine whether it is a white bit or a black bit, and determines whether the marker belongs to a specific dictionary through these bits; thus accurately judging the parameter information of the identification code, and then performing coordinate transformation to obtain the current coordinate information of the inspection UAV relative to the identification code, and transmitting the coordinate information to the edge computing device in a real-time wireless transmission manner.

[0080] The inspection system includes a front-end visualization display system developed based on Thing JS to realize the statistics and analysis of data of several power distribution substations in the urban area.

[0081] Example 1:

[0082] In this example, the hardware devices of the system include several micro quadcopters (equipped with IMU, front-view and downward-view cameras, and communication interfaces connected to sensors), a smart wireless charging apron carrying edge computing devices, and several low-power environmental sensors carried by the micro quadcopters.

[0083] Figure 1A micro - drone device for inspection inside the distribution substation building is shown. The size of the micro - drone body is within 10 cm×10 cm. The micro - drone is equipped with a front - view camera 101, a downward vision sensor 102, an infrared sensor 103, an extended communication interface 104, a contact - type charging electrode 105, a fuselage protection cover 106, and an IMU and on - board computing and storage device 107. The MCU built into the micro - drone can achieve precise attitude control, can calculate the position data in real - time according to the positioning code, and transmit the position data and attitude data to the edge - side computing and control device in real - time. The edge - side computing and control device is the center of the whole system, with an edge - side computing power core and a computing and processing core.

[0084] Figure 2 The architecture diagram of the autonomous inspection system of the micro - drone in the distribution substation building is shown. The core technology of the present invention is a system specifically for autonomous inspection using a micro - drone in the distribution substation building environment. According to Figure 2 the system architecture diagram, the core technical solution of the specific distribution substation building inspection system will be further described:

[0085] The edge - side computing device uses an event - driven method to achieve event - response control and resource management of the multi - task system. In the event - driven multi - task system, tasks are assigned priorities in advance, and tasks with higher priorities are executed first. When an emergency event is marked in the interrupt, if the priority of the task corresponding to the event is high enough, it will be immediately responded to. Compared with the front - and - back - stage system, multi - task drive improves the real - time performance of the system, can customize the change of priorities according to task requirements, and effectively utilizes the limited computing power resources of the edge - side device;

[0086] When starting the inspection, the edge - side device activates the micro - drone of the corresponding task, self - checks the battery power data, attitude data, and sensor status data of the drone. After confirming that there is no error, it sends a take - off command. The drone detects the position of the initial - point identification code and starts the inspection according to the preset trajectory.

[0087] During the inspection process, when using machine vision positioning based on identification codes, the method is as follows: The on-board MCU of the micro UAV extracts the distortion parameters of the UAV camera, corrects the UAV camera, and detects candidate markers. When the vision is turned on, the UAV MCU will analyze the image in real time to find the squares that are candidates for markers, segment the markers using an adaptive threshold, then extract the contours from the threshold image, and discard those contours that are not convex or not close to squares. After candidate detection, it is determined whether they are really markers by analyzing their internal codes, analyzing the number of black or white pixels in each cell to determine whether it is a white bit or a black bit, and determining whether the marker belongs to a specific dictionary through these bits. Thus, the parameter information of the identification code is accurately judged, and then coordinate transformation is performed to obtain the coordinate information of the inspection UAV relative to the identification code, and the coordinate information is wirelessly transmitted in real time to the edge computing device.

[0088] When using the fusion positioning of IMU and machine vision, the method is as follows:

[0089] The edge computing device continuously receives the IMU attitude data and visual positioning coordinate data of the UAV. Figure 3 The following is the flow chart of fusion positioning. An odometer is established based on the attitude data, and the coordinates of the UAV relative to the starting point are calculated according to dead reckoning and fused with the visual positioning coordinates. However, due to the influence of cumulative errors in the IMU sensor, that is, the cumulative error will increase with the passage of time. When the visual position information exists, as time increases, the confidence of the system in the visual positioning result will gradually increase, and at the same time, the confidence in the IMU system will gradually decrease, as shown in the following formula:

[0090]

[0091] Within x seconds, the IMU will decay at a slope of 1 / x with the change of time until the moment before the stable visual coordinates are detected, and the confidence of the inertial navigation is basically 0. At this time, the system no longer believes the positioning result of the IMU. When stable visual coordinates are detected, the inertial navigation data will be completely cleared to empty the cumulative error, and the above process will be repeated until the navigation ends. At the same time, due to the influence of various measurement errors, the position will show fluctuations. The system filters the position information by the method of moving average to obtain the final position information of the moving device.

[0092] Before arriving at the target to be detected, the edge computing device sends a recording command to store the video stream data and sensor data, and continues to inspect according to the predetermined trajectory until it lands on the apron after the inspection is completed, and then transmits the data to the edge computing device.

[0093] Log storage and management: Real-time log storage of various operating parameters, attitude data, and position coordinates during the inspection of micro UAVs in distribution substations. Analyze the current state of the UAV through log information, and perform backtracking through log information during the return flight to assist the UAV in secondary return inspection.

[0094] Embodiment 2:

[0095] Monitoring of meter readings and switch states during the inspection process. The method is as follows: After the inspection video stream of the UAV is transmitted to the edge computing device through the extended communication interface 104, the video stream is first split into frames of images. Each frame of the image will be subjected to blurring judgment to screen out several relatively clear pictures in front of the target to be inspected. The images to be inspected are sent to a pre-trained target detection model to locate the positions of meters and switches in the images. In the meter readings, a semantic segmentation model is used to segment the pointers and scales of each meter, and finally, the readings of each meter are calculated based on the relative positions of the pointers and the known ranges. The meter data read from several frames is averaged to obtain the final meter readings. In the switch state monitoring, the pre-trained target detection model will return the corresponding switch state category and the corresponding indicator light state.

[0096] Environmental monitoring during the inspection process. The method is as follows: The inspection uses a micro UAV equipped with a low-resolution infrared sensor, a temperature and humidity sensor, and a gas sensor. Similarly, after returning to the intelligent apron with the edge computing device, the data is transmitted back and input into the trained classification and detection model to judge the abnormal state of the environment.

[0097] Intelligent wireless charging apron. At the end of the inspection, the UAV locates and guides to land on the intelligent apron according to the identification code. The apron performs contactless wireless charging (wireless induction charging through the contact charging electrode 105) and power management optimization on the UAV according to the log information and battery status information of the UAV.

[0098] At the same time, a set of front-end visualization display system is developed based on Thing JS to realize the statistics and analysis of data of several distribution substations in the urban area.

[0099] Joint debugging and linkage. According to the analysis data of the automatic inspection system and the parameter constraints of the upper computer, the automation equipment of the distribution substation is intelligently managed at fixed times and locations, improving the intelligent level of the distribution substation and saving energy, and ensuring real-time and error-free interaction of the information of the whole system.

[0100] The above implementation process further details the purpose, technical solution, and advantages of the present invention.

[0101] It should be understood that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An unmanned aerial vehicle inspection method for distribution substations based on machine vision and multi-sensor fusion, characterized in that: The inspection method includes an inspection system, and the inspection method includes the following steps; Step S1: When starting the inspection, the apron carrying the edge computing device first performs self-check, analyzes the historical log information, the current UAV battery power and the hardware status information. After confirmation, according to different inspection tasks issued by the host computer, the micro UAV equipped with corresponding sensors is selected to start the inspection task according to the priority; Step S2: After the apron issues the inspection task, it commands the micro UAV to take off to the initial inspection altitude, and detects the take-off and landing guidance code through the overhead machine vision camera to obtain the initial coordinate position of the UAV; And judge the position and attitude data of the UAV according to the attitude trajectory data of the inertial measurement unit IMU and the positioning code; Step S3: The UAV obtains the position coordinates in real-time visual solution on the UAV according to the position information pre-stored in the inspection task and the positioning identification code pasted at the key positions of the inspection route, and then transmits them back to the edge computing control device. The edge device obtains the coordinate information of the current UAV relative to the starting point of the inspection route by converting the position coordinates; Step S4: When the UAV is performing inspection, the IMU attitude data of the UAV is synchronously transmitted back to the edge device to establish an odometer database, which is fused with the coordinate data obtained by the edge device through the UAV machine vision positioning. When the UAV machine vision positioning information is lost, the edge device actively intervenes, adapts the weight information of the two, and eliminates outliers through the data prediction algorithm to obtain the accurate position coordinates of the UAV; Step S7: The edge computing device combines the real-time solved UAV position information and controls the attitude and movement of the UAV through the cascade PID algorithm; Guide the UAV to accurately reach the target to be detected; Step S6: When the UAV reaches the inspection position, the video of the inspection area is recorded in real-time through the forward machine vision camera. The video frames, timestamps and position information are combined to form inspection data, which are synchronously stored in the on-board memory card; Step S7: The UAV repeats the positioning and inspection storage work of the above steps until it reaches the last inspection target and recognizes the inspection end identification code; Step S8: After completing the inspection of all targets to be inspected, the edge computing device performs backtracking according to the trajectory data in the historical log record combined with the real-time fusion positioning data, controls the UAV to perform inspection again in the reverse direction according to the above steps, and generates a backup inspection video frame data set until it returns to the initial position at take-off, that is, the position of the take-off and landing guidance code; Step S9: When reaching the initial position, that is, the apron position, the UAV recognizes the positioning code in the apron through the overhead machine vision, and the edge device performs real-time control and guidance on the UAV to realize the accurate landing of the UAV on the base of the apron; Step S10: After the UAV stops stably, the apron turns on the power management through the contact electrode of the base; The edge computing device sends a data reading instruction, and according to the time stamp, obtains the video frame data recorded by the inspection image video stream during the inspection from the on-board memory card to the edge computing device.

2. The method for inspecting a substation building by an unmanned aerial vehicle based on machine vision and multi-sensor fusion according to claim 1, wherein: The inspection system includes more than one micro UAV with forward machine vision and downward machine vision. Each micro UAV is equipped with a memory card and low-power environmental sensors of the required types according to the tasks it performs. The micro UAV takes off and lands at a helipad equipped with a wireless charging module and a power management module. The inspection system also includes edge devices installed on the helipad, and positioning identification codes that can be visually recognized by the micro UAV and are pasted beside the areas, instruments, and indicator lights to be inspected. The positioning identification codes are used to guide the micro UAV to reach the inspection targets, including takeoff guide codes, landing guide codes, and inspection codes. The edge devices include edge computing devices and edge control devices, which fuse the IMU attitude trajectory data and the coordinate data of UAV visual positioning transmitted back during UAV inspection to form fused positioning data. When the information for UAV visual positioning is lost, the edge devices actively intervene, adaptively weight the two pieces of information through the fused positioning data, and eliminate outliers through a data prediction algorithm to obtain the accurate position coordinates of the UAV.

3. The method for inspecting a distribution substation building by an unmanned aerial vehicle based on machine vision and multi-sensor fusion according to claim 1, wherein: In step S10, the edge computing device identifies the readings of instrument meters, switch states, and indicator light states within the inspection range through calculation. The method includes the following steps; Step A1: The edge computing device reads key frames from the inspection image video stream and filters out clear key picture frames according to the fuzzy detection algorithm. Step A2: The key picture frames are sent to a pre-trained object detection model after image correction to locate the positions of the dials and switches in the image. Step A3: In the operation of reading meter readings, the positioned dials will use a semantic segmentation model to further segment the pointers and scales of each meter, and calculate the readings of each meter according to the relative positions of the pointers and the known ranges. Step A4: In the monitoring of switch states, according to the key picture frames, the pre-trained object monitoring model directly returns the corresponding switch state categories and corresponding indicator light states.

4. The method for unmanned aerial vehicle inspection of distribution substations based on machine vision and multi-sensor fusion according to claim 3, characterized in that: In step S10, the edge computing device judges outliers based on the calculated readings of instrument meters, switch states, and indicator light states to automatically give an alarm; at the same time, it connects to the cloud control platform to summarize data in real time and perform remote visualization display; at the same time, the edge computing device receives remote control instructions in real time to control the next inspection task of the UAV.

5. The method for inspecting a substation building by an unmanned aerial vehicle based on machine vision and multi-sensor fusion according to claim 3, wherein: The edge devices have data interfaces for joint debugging with the automation devices in the substation building, so that the information of the inspection system can be interacted in real time and accurately; at the same time, according to the instructions issued by the timing tasks, the inspection work of the UAV is repeated, so as to realize the unmanned and intelligent inspection and management of the substation building space.

6. The method for inspecting a distribution substation building by an unmanned aerial vehicle based on machine vision and multi-sensor fusion according to claim 2, wherein: When the environment of the substation building is abnormal, the edge computing and control device respectively schedules different functional micro UAVs equipped with infrared temperature sensors and toxic gas sensors according to the task sequence, and performs inspections according to the above inspection process respectively, stores the corresponding sensor data and uploads it to the edge computing device for calculation, and gives intelligent alarms for corresponding tasks in case of data anomalies.

7. The method for inspecting a substation building by an unmanned aerial vehicle based on machine vision and multi-sensor fusion according to claim 1, wherein: During the drone inspection, the edge computing device uses an event-driven approach to achieve event response control and resource management for the multi-task system. In the multi-task system driven by inspection events, tasks are pre-assigned priorities, and tasks with higher priorities are executed first; During the inspection process, when an emergency event is marked in the interrupt, if the priority of the task corresponding to the event is high enough, it will be immediately responded to.

8. The method for unmanned aerial vehicle inspection of distribution substations based on machine vision and multi-sensor fusion according to claim 1, wherein: The fused positioning data is the fused positioning data of IMU and machine vision; the method of fusing IMU and machine vision is as follows: the edge computing device continuously receives the IMU attitude data and visual positioning coordinate data of the drone, establishes an odometer based on the attitude data, calculates the coordinates of the drone relative to the starting point of the inspection path according to dead reckoning, and fuses them with the visual positioning coordinates; However, due to the influence of cumulative error in IMU, that is, the cumulative error will increase with time; When the visual position information exists, as time increases, the confidence of the system in the visual positioning result will gradually increase, and at the same time, the confidence in the IMU system will gradually decrease, as shown in the following formula: Formula 1; Within x seconds, IMU will decay at a slope of 1 / x with time until the moment before a stable visual coordinate is detected. At this time, the confidence of the inertial navigation is regarded as 0, and the inspection system no longer believes in the positioning result of IMU; when a stable visual coordinate is detected, the inspection system clears the inertial navigation data, clears the cumulative error, and repeats the above process until the navigation ends.

9. The method for inspecting a distribution substation building by an unmanned aerial vehicle based on machine vision and multi-sensor fusion according to claim 1, wherein: When the drone uses machine vision to locate by the positioning identification code, the following method is adopted: the micro-drone extracts the distortion parameters of the camera used for machine vision by the on-board MCU, corrects the drone camera to detect candidate marks; When the machine vision of the drone is turned on, the drone MCU performs real-time analysis on the camera image to find a square as a mark candidate, uses adaptive threshold segmentation to segment the mark, then extracts the contours from the threshold image, and discards those contours that are not convex or not close to a square. After candidate detection, determine whether they are really marks by analyzing their internal codes, analyze the number of black or white pixels in each cell to determine whether it is a white bit or a black bit, and determine whether the mark belongs to a specific dictionary through these bits; thus accurately judge the parameter information of the identification code, and then perform coordinate conversion to obtain the current coordinate information of the inspection drone relative to the identification code, and transmit the coordinate information to the edge computing device in a real-time wireless transmission manner.

10. The method for inspecting a distribution substation building by an unmanned aerial vehicle based on machine vision and multi-sensor fusion according to claim 5, wherein: The inspection system includes a front-end visualization display system developed based on Thing JS to achieve the statistics and analysis of data of several power distribution stations in the urban area.

Images