Operation site inspection system based on unmanned aerial vehicle and AI

Abstract

The invention discloses an operation site inspection system based on an unmanned aerial vehicle and AI, which comprises an unmanned aerial vehicle inspection module, an operation violation detection module, an operation violation behavior target tracking module and a remote early warning platform, and is characterized in that the unmanned aerial vehicle inspection module comprises an unmanned aerial vehicle, a vehicle nest and an edge computer, the unmanned aerial vehicle is used for inspecting and shooting an operation site, and the edge computer is used for carrying out remote early warning on the operation site; the machine nest provides a platform for centralized take-off and landing, charging, maintenance, control, video storage, video analysis and video transmission for the unmanned aerial vehicle, and the edge computer is connected with the machine nest and performs operation hidden danger intelligent identification analysis on videos shot by the unmanned aerial vehicle; the operation violation detection module and the operation violation behavior target tracking module are arranged in a machine nest or an edge computer, and the remote early warning platform carries out real-time video viewing, alarm recording, statistics and query, and the mobile terminal APP subscribes early warning information.

Description

Technical Field

[0001] The present invention relates to the field of electric power grid technology, and in particular to an operation site inspection system based on drones and AI. Background Art

[0002] In recent years, with the rapid development of social economy and the rapid growth of urban scale, the demand for electricity for life and industry has been increasing. Behind the large demand for electricity, a large national interconnected power grid with ultra-high voltage transmission lines (hereinafter referred to as transmission lines) as the main framework has emerged, and the scale of the power grid has also been increasing. It is precisely because of the rapid expansion of the power grid that the challenges and risks faced by the construction of transmission lines are also increasing day by day.

[0003] Substations are an important part of the power grid. They have many internal devices of various types and are difficult to operate and manage. Substation equipment is exposed to the natural environment for a long time. It not only has to bear the internal pressure of normal mechanical loads and power loads, but also has to withstand external damage such as dirt, lightning strikes, strong winds, and bird damage. These will cause the aging of its components. If the hidden dangers are not discovered and eliminated in time, accidents may occur, seriously threatening the safety of the power grid.

[0004] Therefore, substation inspection has become an important part of basic operation and maintenance work. Through the inspection of substation equipment, we can grasp the operating status of the equipment and changes in the surrounding environment, timely discover equipment defects and safety hazards, and put forward specific maintenance suggestions, so as to eliminate defects in time and prevent accidents, thereby ensuring the safety of power transmission and transformation equipment and the stability of the power system.

[0005] At present, the operation is supervised by collecting data from vehicle-mounted or individual camera, and transmitting it to the backend through 3G / 4G network, or copying the video content to the server through post-connection when the network is poor. Line status inspection The line status inspection is carried out by fixed camera monitoring on the tower, or by drone or manual control.

[0006] At present, there are many construction sites, and each monitoring post has to monitor a large number of cameras, which makes it impossible to achieve real-time monitoring of all the work sites at the same time. The contradiction between the high-intensity monitoring demand and the status of the monitoring personnel themselves. Personnel monitoring itself has shortcomings. People have limited energy. Video monitoring requires monitoring of many work points, long monitoring time, and heavy monitoring tasks. Monitoring personnel work in a state of staring at the screen and concentrating their energy. After a long time, fatigue will inevitably occur, and the monitoring efficiency will decrease, affecting the monitoring effect. The third is the contradiction that the lagging monitoring prompt cannot take effect on real-time violations in time. Although the current video monitoring can achieve real-time monitoring of a certain work point, due to the lack of a medium for direct communication with the work site, the violations and hidden dangers found by the video monitoring need to be handled by contacting the work site, which takes a certain amount of time, and there is a lag in the risk control of the site. On-site violations are usually uninterrupted behaviors of continuous operations, which are real-time and sudden. When violations are about to occur or are occurring, video monitoring personnel cannot prompt or stop the behavior of operators in time, and cannot play a role in pre-prevention. Summary of the invention

[0007] In order to overcome the deficiencies in the prior art, the present invention provides a work site inspection system based on drones and AI.

[0008] The present invention is achieved through the following technical solutions:

[0009] The on-site inspection system based on drones and AI includes:

[0010] The drone inspection module includes a drone, a machine nest, and an edge computer. The drone is used to inspect and photograph the work site. The machine nest provides a centralized platform for take-off and landing, charging, maintenance, control, video storage, video analysis, and video transmission for the drone. The edge computer is connected to the machine nest and performs intelligent identification and analysis of operational hazards on the videos taken by the drone.

[0011] An operation violation detection module is provided in the machine nest or the edge computer. A violation detection algorithm is established in the operation violation detection module to automatically identify violation behaviors from the perspective of the drone.

[0012] An operation violation target tracking module is provided in the machine nest or the edge computer. A target tracking algorithm is established in the operation violation target tracking module to detect and track operation violations.

[0013] The remote early warning platform performs real-time video viewing, alarm recording, statistics and query, and mobile terminal APP subscription to early warning information operations.

[0014] Furthermore, in the drone inspection module, the drone nest acts as a video transmission device, transmits the real-time video stream to the edge computer through the network port interface, and performs intelligent identification and analysis of operational hazards in the edge computer.

[0015] Furthermore, the data of intelligent identification and analysis of operational hazards performed by the edge computer transmits alarm information to the remote early warning platform via the 4G network.

[0016] Furthermore, in the operation violation detection module, data collection and annotation are performed on violation data related to the drone's perspective, and various known violations are sorted out, and samples of each type of violation are accurately annotated. YOLOv8 is used as the core model, and the network architecture of YOLOv8 is analyzed. The model for violation identification is trained and optimized, and NVIDIA's TensorRT SDK is used as the inference engine. The TensorRT SDK inference engine can accelerate the inference speed of the deep learning model by building an optimized runtime engine, and then through YOLOv8 model conversion and optimization, efficient model deployment is achieved to ensure that the drone can quickly and accurately identify violations.

[0017] Furthermore, in the operation violation target tracking module, YoloV8 target detection technology and BoT-SORT tracking technology are used to detect and track operation violations, and the tracking results are sent to the flight control module of the drone for real-time adjustment, so as to achieve more accurate analysis and rapid response to violations.

[0018] Furthermore, the tracking result is sent to the flight control module of the drone for real-time adjustment, including the adjustment of the gimbal, camera focal length, deflection angle, and horizontal and vertical movement distance.

[0019] Furthermore, in the remote early warning platform, the drone inspects the work site and sends video data to the edge computer through image transmission. The edge computer uses the AI ​​algorithms of YoloV8 and BoT-SORT to analyze the video data in real time to detect and track violations.

[0020] Furthermore, in the remote warning platform, the MQTT protocol is used to realize real-time transmission of warning information. The edge computer encapsulates the warning result into an MQTT message and publishes it to the MQTT server through the 4G network. The MQTT server routes the MQTT message to all MQTT clients that have subscribed to the corresponding topic.

[0021] Furthermore, the MQTT client includes a Web server, a smart bracelet APP or a mobile phone APP.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] Through the on-site inspection system of drones and AI, it is possible to quickly and effectively detect operational violations and illegal behaviors in a variety of situations and environments, transmit them to the required terminals in a timely manner, and obtain feedback in a timely manner, so as to promptly prompt or stop the operators and play a role in pre-emptive prevention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0025] The operation site inspection system based on drones and AI is characterized by:

[0026] The drone inspection module includes a drone, a machine nest, and an edge computer. The drone is used to inspect and photograph the work site. The machine nest provides a centralized platform for take-off and landing, charging, maintenance, control, video storage, video analysis, and video transmission for the drone. The edge computer is connected to the machine nest and performs intelligent identification and analysis of operational hazards on the videos taken by the drone.

[0027] The machine nest serves as a video transmission device, which transmits the real-time video stream to the edge computer through the network port interface, and performs intelligent identification and analysis of operational hazards in the edge computer. The data of the intelligent identification and analysis of operational hazards by the edge computer is transmitted as alarm information to the remote early warning platform through the 4G network.

[0028] An operation violation detection module is provided in the machine nest or the edge computer. A violation detection algorithm is established in the operation violation detection module to automatically identify violation behaviors from the perspective of the drone.

[0029] We collect and annotate the violation data related to the drone’s perspective, sort out various known violations, and accurately annotate samples of each type of violation. We use YOLOv8 as the core model, analyze the network architecture of YOLOv8, train and optimize the violation identification model, and use NVIDIA’s TensorRTSDK as the inference engine. The TensorRT SDK inference engine can accelerate the inference speed of deep learning models by building an optimized runtime engine, and then achieve efficient model deployment through YOLOv8 model conversion and optimization, ensuring that the drone can quickly and accurately identify violations.

[0030] An operation violation target tracking module is provided in the machine nest or the edge computer. A target tracking algorithm is established in the operation violation target tracking module to detect and track operation violations.

[0031] YoloV8 target detection technology and BoT-SORT tracking technology are used to detect and track operational violations, and the tracking results are sent to the flight control module of the drone for real-time adjustment, so as to achieve more accurate analysis and rapid response to violations. The tracking results are sent to the flight control module of the drone for real-time adjustment, and the adjustments include gimbal, camera focal length, deflection angle, horizontal and vertical movement distance.

[0032] The remote early warning platform performs real-time video viewing, alarm recording, statistics and query, and mobile terminal APP subscription to early warning information operations.

[0033] The drone inspects the work site and sends video data to the edge computer through image transmission. The edge computer uses the AI ​​algorithms of YoloV8 and BoT-SORT to analyze the video data in real time to detect and track violations.

[0034] In the remote warning platform, the MQTT protocol is used to realize the real-time transmission of warning information. The edge computer encapsulates the warning result into an MQTT message and publishes it to the MQTT server through the 4G network. The MQTT server routes the MQTT message to all MQTT clients that have subscribed to the corresponding topic. The MQTT client includes a Web server, a smart bracelet APP or a mobile phone APP.

[0035] In other embodiments, the drone inspection module includes a drone, a nest, and an edge computer. The drone is used to inspect and photograph the work site. The nest provides a centralized take-off and landing, charging, maintenance, control, video storage, video analysis, and video transmission platform for the drone. The edge computer is connected to the nest and performs intelligent identification and analysis of work hazards on the video taken by the drone.

[0036] The drone nest is used as a video transmission device to transmit the real-time video stream to the edge computer through the network port and other interfaces, and intelligent identification and analysis of operational hazards are performed in the edge computer.

[0037] The alarm information is actively transmitted to the server platform for display through the 4G network, and the operator's mobile phone APP and the wristband APP actively subscribe to the server middleware alarm information for reminders. The machine nest has strong environmental adaptability, can realize unattended operation, can remotely formulate flight plans, automatically execute tasks, and liberate complex labor. It has a highly integrated design and supports rapid deployment. It is suitable for situations where the inspection points are fixed.

[0038] In other embodiments, an operation violation detection module is provided in the machine nest or the edge computer, and a violation detection algorithm is established in the operation violation detection module to automatically identify the violation behavior from the perspective of the drone, and the algorithm is specifically for automatically identifying a variety of violations from the perspective of the drone. In the drone monitoring environment, due to the unique perspective and the changing environment, the accurate identification of violations presents a huge challenge. Therefore, we will conduct a systematic study to solve this problem.

[0039] First, detailed data collection and labeling will be carried out, especially for the data on violations related to drone vision. In order to obtain a high-quality training data set, various known violations will be comprehensively sorted out, and samples of each type of behavior will be accurately labeled. In addition, in order to ensure the quality of the data, necessary data preprocessing steps will be carried out, including data cleaning and data enhancement, so as to ensure the feasibility and effectiveness of subsequent model training.

[0040] In terms of model selection, YOLOv8 (You Only Look Once, version 8), which is a leader in the field of object detection, is used as the core model of this study. YOLOv8 is known for its excellent performance and flexibility. Its unique one-time detection mechanism and prior box strategy make it have excellent performance in both speed and accuracy. The network architecture of YOLOv8 will be deeply analyzed, with special attention paid to its newly introduced improved features, as well as how to train and optimize its model for violation identification.

[0041] In order to achieve high-performance real-time violation identification on edge devices, NVIDIA's TensorRT SDK is used as the inference engine. TensorRT is a library designed specifically for deep learning reasoning optimization. It can significantly accelerate the inference speed of deep learning models by building an optimized runtime engine. By converting and optimizing the YOLOv8 model, efficient model deployment can be achieved on edge devices (such as the computing platform of drones), thereby ensuring that drones can quickly and accurately identify violations on the edge side.

[0042] Not only does it achieve high-precision identification of violations, but it also takes into account the actual application and execution of the identification results. Specifically, once the system detects a violation, the relevant information of the behavior will be immediately sent to the platform for display, so that operators can review and further process it.

[0043] In addition, considering the importance of real-time tracking. In drone applications, once a violation is detected, it is often necessary to track the subject in real time to obtain more information. Therefore, the identified violation target is used as the prior frame of the target tracker to specify and start real-time tracking of the violation target.

[0044] In other embodiments, an operation violation target tracking module is provided in the machine nest or the edge computer, and a target tracking algorithm is established in the operation violation target tracking module to detect and track operation violations;

[0045] The study studies the target tracking algorithm based on UAV video operation violations. With the help of YoloV8 target detection technology and BoT-SORT tracking technology, the operation violations are detected and tracked, and the tracking results are sent to the UAV flight control module for real-time adjustment, including gimbal, camera focal length, deflection angle, horizontal and vertical movement distance, etc., to adjust to a more appropriate UAV perspective, so as to achieve more accurate analysis and rapid response to violations.

[0046] Before implementing the tracking task, the BoT-SORT model first needs to be trained with a large amount of training data. To ensure the accuracy of tracking in the drone shooting environment, data collection and labeling will be carried out on the violation data related to the drone perspective to build a high-quality data set. A comprehensive review of the currently known violation categories will be conducted, with the focus on data collection and labeling on violation data from the drone perspective. Based on the collected data, the violations will be labeled and necessary data preprocessing will be performed to ensure the feasibility and effectiveness of subsequent model training.

[0047] After such a training process, the model is able to accurately detect and track new video data. In actual use, we input the newly collected video data into the model, and the model will output the bounding box and identifier of each target object. This information will be used to adjust the flight parameters of the drone, optimize the drone's viewing angle, and improve the efficiency of monitoring violations.

[0048] First, in the drone inspection system, Yolov8, as a target detection model, uses its precise target recognition capabilities to accurately detect operational violations, detect all targets in the video frame, and output the bounding boxes and category labels of these targets. Once Yolov8 detects one or more abnormal targets (such as violations), these abnormal targets need to be passed to the BoT-SORT model for tracking. This process can be carried out through the following steps:

[0049] Abnormal target screening: In the results of Yolov8 detection, abnormal targets are screened out according to predefined abnormal behavior categories. For example, if a target is marked as "violation", it is considered an abnormal target.

[0050] Single target selection: You need to select one target to track among multiple abnormal targets. The selection criteria can be defined according to the actual situation. For example, you can select the abnormal target that appears first, or the abnormal target that occupies the largest area in the picture.

[0051] Abnormal target labeling: The bounding box and category label of the selected single abnormal target are passed to the tracking model. This target will be used as the input of the tracking model for subsequent tracking tasks.

[0052] Object tracking: When the tracking model receives an abnormal object, it will start tracking it. If the object disappears or is occluded in subsequent video frames, the tracking model will use the previously learned information to try to continue tracking the object.

[0053] Tracking result feedback: The output of the tracking model will be fed back to the flight control system of the drone. Based on this information, the drone can adjust the gimbal, camera focal length, deflection angle, horizontal and vertical movement distance and other parameters to optimize the drone's viewing angle and achieve accurate tracking of abnormal targets.

[0054] Secondly, the BoT-SORT tracking technology used is a new and robust tracker that combines the advantages of motion and surface information, using camera motion compensation and a more accurate Kalman filter state vector. Compared with the classic DeepSORT, it uses the length and width of the box instead of the ratio as the output of the KF state, which can provide more accurate tracking results. As for the method of using IOU as a similarity evaluation method commonly used in the SORT series, BoT-SORT has also been optimized and a simple and effective IOU is proposed to achieve a higher MOTA.

[0055] In addition, BoT-SORT also introduces the camera motion model CMC to solve the problem of lost tracking caused by drastic changes in the pixel position of the box in dynamic camera scenes. Even in static cameras, it can effectively eliminate small vibrations caused by factors such as wind. In addition, the model uses OpenCV's global motion compensation (GMC) to approximate the alignment between adjacent frames. When tracking and matching, the fusion method of IoU and Re ID is adopted. Using the backbone of resnetst50, the fastreid framework extracts ReID features, and then updates them frame by frame. The average features are extracted to calculate the similarity, which further improves the matching accuracy. The specific detection and tracking process is shown in the figure below:

[0056] Through such a system design, the tracking results can be fed back to the UAV's flight control. By adjusting the UAV's gimbal, camera focal length, deflection angle, horizontal and vertical movement distance and other parameters, the UAV's posture and viewing angle can be optimized to better analyze illegal targets and improve the efficiency of monitoring illegal operations.

[0057] In other embodiments, the remote warning platform performs real-time video viewing, alarm recording, statistics and query, and mobile terminal APP subscription to warning information operations.

[0058] With drones as the core, it is equipped with advanced edge computing technology and AI analysis algorithms, which can capture and process drone video data in real time to detect, track and warn of violations. The system also includes a remote warning platform that can perform real-time video viewing, alarm recording, statistics and query, and mobile terminal APP subscription of warning information.

[0059] In actual work, drones will inspect the work site and send video data to the edge computer through image transmission. The edge computer uses the AI ​​algorithms of YoloV8 and BoT-SORT to analyze the video data in real time, thereby detecting and tracking violations. Once a violation is found, the system will immediately generate an early warning and send the warning information to the MQTT server through the 4G network. The web front end and mobile terminal APP will notify the staff by subscribing to the early warning information.

[0060] In the remote warning platform, the MQTT protocol is used to realize the real-time transmission of warning information. The edge computer encapsulates the warning results into MQTT messages and publishes them to the MQTT server through the 4G network. The MQTT server is responsible for routing these messages to all clients that have subscribed to the corresponding topics.

[0061] As an MQTT client, the web server subscribes to topics related to warnings. Therefore, when new warning messages are published to the MQTT server, the web server will immediately receive these messages and display them on the web front end.

[0062] At the same time, the smart bracelet APP and mobile phone APP also serve as MQTT clients and subscribe to topics related to warnings. Therefore, they will also receive warning information in real time and remind operators to pay attention to safety through vibration and voice.

[0063] The advantage of using the MQTT protocol is that it is a lightweight, real-time message transmission protocol, which is very suitable for the transmission of warning information. Moreover, it adopts a publish / subscribe mode, which can realize one-to-many message transmission, so that all subscribers can receive warning information in time. This is of great significance to improving work safety.

[0064] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. The on-site inspection system based on drones and AI is characterized by: include: A drone inspection module, which includes a drone, a nest, and an edge computer. The drone is used to inspect and photograph the work site. The nest provides a centralized platform for take-off and landing, charging, maintenance, control, video storage, video analysis, and video transmission for the drone. The edge computer is connected to the nest and performs intelligent identification and analysis of work hazards on the video taken by the drone. An operation violation detection module, wherein the operation violation detection module is disposed in the machine nest or the edge computer, and a violation detection algorithm is established in the operation violation detection module to automatically identify violation behaviors from the perspective of the drone; An operation violation target tracking module, wherein the operation violation target tracking module is arranged in the machine nest or the edge computer, and a target tracking algorithm is established in the operation violation target tracking module to detect and track operation violation behaviors; The remote early warning platform performs real-time video viewing, alarm recording, statistics and query, and mobile terminal APP subscription to early warning information operations.

2. The work site inspection system based on drone and AI according to claim 1 is characterized in that: In the drone inspection module, the drone nest acts as a video transmission device, transmits the real-time video stream to the edge computer through the network port interface, and performs intelligent identification and analysis of operational hazards in the edge computer.

3. The work site inspection system based on drone and AI according to claim 2 is characterized in that: The data from the intelligent identification and analysis of operational hazards by the edge computer is transmitted via the 4G network as alarm information to the remote early warning platform.

4. The work site inspection system based on drone and AI according to claim 1 is characterized in that: In the operation violation detection module, data collection and annotation are performed on violation behavior data related to the drone's perspective, and various known violations are sorted out, and samples of each type of violation are accurately annotated. YOLOv8 is used as the core model, and the network architecture of YOLOv8 is analyzed. The model for violation identification is trained and optimized. NVIDIA's TensorRT SDK is used as the inference engine. The TensorRT SDK inference engine can accelerate the inference speed of the deep learning model by building an optimized runtime engine, and then through YOLOv8 model conversion and optimization, efficient model deployment is achieved to ensure that the drone can quickly and accurately identify violations.

5. The work site inspection system based on drone and AI according to claim 1 is characterized in that: In the operation violation target tracking module, YoloV8 target detection technology and BoT-SORT tracking technology are used to detect and track operation violations, and the tracking results are sent to the flight control module of the drone for real-time adjustment, so as to achieve more accurate analysis and rapid response to violations.

6. The work site inspection system based on drone and AI according to claim 5 is characterized in that: The tracking results are sent to the flight control module of the drone for real-time adjustment, including the adjustment of the gimbal, camera focal length, deflection angle, horizontal and vertical movement distance.

7. The work site inspection system based on drone and AI according to claim 1 is characterized in that: In the remote early warning platform, the drone inspects the work site and sends video data to the edge computer through image transmission. The edge computer uses the AI ​​algorithms of YoloV8 and BoT-SORT to analyze the video data in real time to detect and track violations.

8. The work site inspection system based on drone and AI according to claim 7 is characterized in that: In the remote warning platform, the MQTT protocol is used to realize the real-time transmission of warning information. The edge computer encapsulates the warning result into an MQTT message and publishes it to the MQTT server through the 4G network. The MQTT server routes the MQTT message to all MQTT clients that have subscribed to the corresponding topic.

9. The work site inspection system based on drone and AI according to claim 8 is characterized in that: The MQTT client includes a Web server, a smart bracelet APP or a mobile phone APP.