Intelligent electric power inspection unmanned aerial vehicle system based on multi-modal perception and method thereof

Abstract

The invention discloses an intelligent electric power inspection unmanned aerial vehicle system based on multi-mode perception and a method thereof, and relates to the technical field of unmanned aerial vehicle control and computer vision. The system comprises a multi-mode sensor group, an edge calculation unit and a flight control and communication module. The edge computing unit is provided with a VLA decision center, an improved YOLOv8-CA target detection engine and a dynamic arbitration engine. According to the method, natural language instructions are analyzed and tasks are planned through the VLA decision center, and manual pre-planning and three-dimensional modeling do not need to be depended on; high-precision positioning of components and defects is realized through a target detection engine; multi-level response is triggered based on detection confidence through a dynamic arbitration engine, double-model work is coordinated, and intelligent decisions such as emergency obstacle avoidance and infrared verification are achieved. According to the invention, multi-mode perception and double-model collaborative decision are fused, the problems of high cost, poor flexibility, small target leak detection and insufficient dynamic environment adaptability of traditional inspection are solved, and full-autonomous, high-reliability and sustainable-evolution intelligent electric power inspection is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle control and computer vision technology, and particularly relates to an intelligent power inspection unmanned aerial vehicle system based on multi-modal perception and a method thereof. BACKGROUND

[0002] Power inspection is an important link to ensure the safe and stable operation of the power grid. At present, unmanned aerial vehicle technology has been widely used in the inspection of power towers, wires, insulators and other power equipment. However, the existing intelligent power inspection unmanned aerial vehicle has low intelligence, and mainly relies on the following two modes in actual application, both of which have significant defects.

[0003] The first mode is completely dependent on manual operation. When performing tasks, the unmanned aerial vehicle basically completely relies on manual remote control by the pilot or manual pre-planned flight path. Since the power tower is very high, the surrounding wires are complex, and in the outdoor environment, the operator's naked eye may not be able to see the wires or may have visual blind spots, resulting in high flight risk. At the same time, the effect of manual operation depends heavily on the actual operation level and experience of the pilot, and a large number of experienced pilots need to be trained, which is very costly in terms of manpower, and the operation effect is difficult to standardize.

[0004] The second mode is an automatic inspection mode based on three-dimensional laser point cloud modeling. This method has been applied in recent years. The process is first to use a laser radar to model a three-dimensional point cloud of the power tower, and then to manually plan each flight point that needs to be photographed on the preset point cloud model, and the unmanned aerial vehicle automatically performs the task accordingly. Although this method reduces the real-time dependence on the pilot, it requires detailed three-dimensional modeling of each power tower in advance, and a large amount of manual work is required to edit and save the flight tasks in advance, which is very costly in terms of time and money. Especially for large-area power inspection, thousands of power towers need to be modeled in advance, which is a huge cost.

[0005] In addition, this mode has poor flexibility and can only perform pre-set tasks rigidly and cannot be flexible. For power towers without pre-set maps, or when the environment around the power tower changes due to construction, vegetation growth, etc., this method is completely useless and has a high safety risk and missed inspection risk.

[0006] Furthermore, the above existing technical solutions have not effectively solved the inherent shortcomings of traditional single artificial intelligence algorithms in power inspection:

[0007] 1. Poor dynamic environment adaptability: pure target detection models (such as YOLO) cannot understand complex semantic instructions such as "avoid left obstacles", and static detection models cannot cope with dynamic obstacles that suddenly appear during inspection, such as flying birds, floating plastic bags, and close-range overhead wires.

[0008] 2. High small target defect miss rate: The detection accuracy of general target detection models for small targets such as tower bolt rust and insulator fine cracks is insufficient, and the miss rate is usually greater than 15%, which is difficult to meet the stringent safety requirements of power inspection.

[0009] 3. Low multi-task coordination efficiency: Traditional artificial intelligence algorithms can usually only perform a single task, such as only defect detection or only path planning. For complex inspection instructions, tasks such as defect detection, real-time path planning, and dynamic obstacle avoidance need to develop multiple algorithms independently, and task switching requires human intervention, which cannot form an intelligent closed loop.

[0010] Therefore, there is an urgent need in the art for an intelligent power inspection UAV system that can break free from the dependence on artificial pre-planning and pre-3D modeling, and can intelligently adapt to complex dynamic environments, accurately identify small defects, and autonomously coordinate multiple tasks. SUMMARY

[0011] The technical problem to be solved by the present application is: In order to overcome the above technical problems, the present application provides an intelligent power inspection UAV system based on multi-modal perception and a method thereof.

[0012] The technical solution adopted by the present application to solve its technical problem is: An intelligent power inspection UAV system based on multi-modal perception, comprising:

[0013] A multi-modal sensor group for collecting perception data of power equipment and surrounding environment;

[0014] An edge computing unit deployed at the UAV end;

[0015] The edge computing unit is configured to run:

[0016] A target detection engine for processing the perception data, identifying power equipment components and defects, and outputting detection results including defect location and confidence;

[0017] A VLA (Visual-Linguistic-Action) decision hub, which is a model based on a large language model and fine-tuned in the field using LoRA technology, for analyzing natural language instructions and generating dynamic control signals for the UAV;

[0018] A dynamic arbitration engine for receiving the detection results output by the target detection engine and triggering a multi-level response strategy based on the confidence to coordinate the working states of the target detection engine and the VLA decision hub;

[0019] The dynamic arbitration engine is configured to trigger the VLA decision hub to execute different control instructions to coordinate the working states of the target detection engine and the VLA decision hub to realize autonomous inspection when the confidence meets different conditions.

[0020] The system further comprises a UAV platform and a flight control and communication module; the multi-modal sensor group is integrated on the UAV platform; the flight control and communication module is used for instruction execution and data transmission, and provides state information for UAV flight control.

[0021] The multi-modal sensor group comprises one or at least two of a visible light camera, an infrared thermal imager and a laser radar.

[0022] The target detection engine is an improved YOLO model based on an attention mechanism, the attention mechanism is a coordinate attention mechanism, multi-scale feature fusion FPN+PAN, and the improved YOLO model is quantized by INT8 after training and deployed on an edge computing unit.

[0023] The control signal of the UAV is a high-level task instruction for UAV control, which is converted into a bottom-level control signal by the UAV platform, and the control signal of the UAV comprises linear velocity, yaw angular velocity and gimbal control instructions.

[0024] The dynamic arbitration engine is configured to execute the following multi-level response strategies:

[0025] When the confidence is higher than or equal to a first threshold, the VLA decision hub is triggered to execute an emergency obstacle avoidance or key re-inspection instruction;

[0026] When the confidence is lower than the first threshold but higher than or equal to a second threshold, a multi-modal verification instruction is triggered to call multiple sensors in the multi-modal sensor group to jointly verify the detected target;

[0027] When the confidence is lower than the second threshold, the detection result is marked for subsequent review.

[0028] The multi-modal verification instruction comprises starting the infrared thermal imager to perform temperature verification on the suspected defect point.

[0029] The system further comprises a closed-loop optimization system for receiving data uploaded by the edge computing unit and performing incremental training and model optimization on the target detection engine and / or the VLA decision hub based on an active learning mechanism.

[0030] The application provides an intelligent unmanned aerial vehicle power inspection method based on multi-modal perception, which is applied to the system and has the characteristics that it includes the following steps:

[0031] Receiving and analyzing natural language inspection instructions;

[0032] Based on the natural language inspection instructions, the VLA decision hub plans an initial flight path;

[0033] During the flight inspection process, the target detection engine processes the perception data collected by the multi-modal sensor group in real time, and outputs detection results;

[0034] The dynamic arbitration engine dynamically decides and triggers corresponding response actions according to the confidence in the detection results, and the response actions include continuing cruising, approaching re-inspection, multi-modal verification, or emergency obstacle avoidance;

[0035] According to the response action, the unmanned aerial vehicle performs corresponding operations, and generates an inspection report.

[0036] The dynamic decision process further includes: when the output of the target detection engine conflicts with the decision history or environmental context information of the VLA decision hub, the decision with higher confidence is preferred, and the conflict data sample is returned to the closed-loop optimization system for model optimization.

[0037] Compared with the prior art, the intelligent power inspection unmanned aerial vehicle system and method based on multi-modal perception have the following remarkable beneficial effects:

[0038] 1. The application analyzes natural language instructions through the VLA decision hub, autonomously plans an inspection path and task, does not need to rely on manual remote control flight, and does not need to perform expensive laser radar three-dimensional point cloud modeling and manual waypoint planning on power towers in advance. This makes the deployment time of the inspection task reduced from "hour level" or even "day level" to "minute level", and eliminates the need for a large number of professional pilots and high three-dimensional modeling costs, thereby fundamentally reducing the comprehensive cost of power inspection.

[0039] 2. The application fuses the VLA decision hub and the special target detection engine, and introduces the dynamic arbitration engine for cooperation, effectively solving the three problems existing in the traditional scheme:

[0040] Dynamic environmental adaptability: the VLA model gives the unmanned aerial vehicle the ability to understand semantic instructions and perform spatio-temporal context reasoning, so that it can cope with sudden dynamic obstacles such as birds and plastic bags, instead of rigidly executing preset routes.

[0041] Small target defect detection precision: through the joint action of FPN+PAN multi-scale feature fusion and CA coordinate attention enhancement, the target detection engine of the application can maintain good detection capability for targets of various sizes while achieving a breakthrough in the detection precision of small target defects. After model training and INT8 quantization, the system is deployed on an edge computing unit, and the single-frame processing delay is < 50 ms.

[0042] Multi-task coordination efficiency: an intelligent closed loop of "perception-decision-execution-verification" is formed, which can autonomously complete the complete process from task analysis, global path planning, real-time defect detection, to dynamic obstacle avoidance and fine re-inspection, without the need for manual intervention to switch tasks, and the efficiency is extremely high.

[0043] 3. The dynamic arbitration engine of the application intelligently triggers a multi-level response strategy based on the confidence of target detection. This mechanism can effectively filter visual false positives caused by light, reflection, etc., and cross-verify through physical characteristics such as infrared temperature measurement to ensure the accuracy of defect recognition. At the same time, in the face of decision conflicts between models, the arbitration mechanism can select the optimal strategy to ensure the reliable operation of the system in complex and uncertain environments.

[0044] 4. The system of the application automatically returns low-confidence samples and conflict samples in the arbitration process to the training database through an active learning mechanism, and uses tools such as SAM for semi-automatic labeling for incremental training of the model. This enables the target detection model and the VLA decision hub to continuously evolve with the execution of the inspection task, adapt to new defect types and environmental changes, form a virtuous cycle of getting smarter with use, and maintain a long-term technical advantage. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is the architecture diagram of the intelligent power inspection unmanned aerial vehicle system provided by the embodiment of the application. DETAILED DESCRIPTION

[0046] The application will be further described in detail below in conjunction with the drawings and specific embodiments, but the embodiments of the application are not limited thereto.

[0047] Embodiment one: system hardware architecture

[0048] As shown in Figure 1 , the intelligent power inspection unmanned aerial vehicle system based on multi-modal perception coordination of the embodiment includes an unmanned aerial vehicle platform, a multi-modal sensor group, an edge computing unit, a flight control and communication module, and a closed-loop optimization system.

[0049] The multi-modal sensor group is integrated on the unmanned aerial vehicle platform and includes:

[0050] Visible light camera: used to capture high-definition images of the tower structure, with a resolution of ≥4K, providing high-quality visual input for the target detection engine.

[0051] Infrared thermal imager: optional, used to detect equipment overheating anomalies, with a temperature measurement accuracy of ±2℃. It is activated when the dynamic arbitration engine triggers a check.

[0052] Laser radar: optional, used to assist in distance measurement and three-dimensional environmental perception in extreme weather conditions such as heavy fog and heavy rain.

[0053] In this embodiment, the laser radar uses livox mid360, installed on the top of the UAV, with a 30° angle between the horizontal bottom surface of the radar and the horizontal plane. Laser radar calibration mainly solves two core problems:

[0054] Intrinsic calibration: determines the parameters of the laser radar itself, such as the vertical and horizontal angle of each laser beam, distance measurement deviation, etc. This is usually done by the manufacturer and is generally considered known in subsequent fusion.

[0055] Extrinsic calibration: determines the relative position and attitude relationship between the laser radar and other sensors (such as IMU, camera), i.e. a rotation matrix and a translation vector. This is the main work of multi-sensor fusion calibration, and high-precision extrinsic parameters are the premise of subsequent fusion algorithms, and calibration errors will directly affect the accuracy of positioning, mapping, and perception.

[0056] Laser radar and IMU calibration is a key link in this system. High-frequency data from the IMU can help correct the point cloud distortion caused by the laser radar during motion, while the precise pose estimation provided by the laser radar helps correct the drift of the IMU.

[0057] Main method: the current mainstream method is based on continuous time trajectory optimization. This method fits the IMU measurements into a continuous time trajectory using spline curves and other methods, and the point cloud scanned by the laser radar can be projected onto the same time through this trajectory, thereby correcting the point cloud distortion caused by the motion of the carrier. Then, by optimizing the geometric feature distance of "point to plane" or "point to line" (i.e. the idea of ICP algorithm), the extrinsic parameters between the laser radar and the IMU and the motion trajectory of the carrier are estimated.

[0058] The laser radar provides accurate three-dimensional geometric information, and the camera provides rich two-dimensional texture information. The calibration of the two is the basis for realizing true color point cloud and providing depth information for images, etc.

[0059] Target-based calibration: This is the most commonly used and highly accurate method. It establishes the correspondence between feature points (such as corner points and center points) using a specific calibration board (e.g., checkerboard, ArUco code, or 3D calibration board with a special pattern) simultaneously observed by both LiDAR and camera. By using data acquired from multiple poses, the optimal extrinsic parameter matrix is ​​solved, ensuring that when feature points in the LiDAR point cloud are projected onto the image, they overlap as much as possible with the feature points detected in the camera image.

[0060] Targetless calibration: This method does not rely on specially made calibration objects, but uses natural features in the environment (such as edges and contours) for calibration.

[0061] For example, extrinsic parameters can be optimized by extracting edge features from images and point clouds and establishing correspondences between them. Targetless calibration is more flexible and suitable for online operations, but its accuracy and robustness are generally inferior to target-based methods, and it is mostly used for fine-tuning extrinsic parameters or specific scenarios.

[0062] The edge computing unit is the core computing module deployed on the drone. It has a built-in high-performance GPU (Graphics Processing Unit) with a computing power of no less than 64 TOPS, preferably using an AI computing module with embedded GPU, such as the NVIDIA Jetson Orin series. This unit is responsible for carrying and running core algorithm models such as the YOLOv8-CA object detection engine, VLA decision center, and dynamic arbitration engine. The GPU has more than 24 GB of video memory and more than 6000 CUDA cores.

[0063] The flight control and communication module is a built-in module of the UAV, responsible for command execution and data transmission. Its uplink data rate is no less than 100Mbps, and it integrates a 9-axis IMU (Inertial Measurement Unit), including three axes each for acceleration, angular velocity, magnetic field heading, and provides status information for flight control.

[0064] The closed-loop optimization system is used for incremental training and optimization of the model.

[0065] Example 2: Workflow of the core software module

[0066] The core software module of this invention runs on an edge computing unit and includes an object detection engine, a VLA decision center, and a dynamic arbitration engine. Its complete workflow is as follows:

[0067] Step S201: Task initialization and path planning

[0068] Users can issue commands via natural language, such as: "Scan the large side insulator strings of the tension tower, focusing on checking for damage and dirt."

[0069] The VLA decision center is responsible for dynamic decision-making and task scheduling. By analyzing the instruction through the VLA decision center, key information is extracted, including the target power tower, core components, and defect type.

[0070] Subsequently, the VLA decision center outputs dynamic path planning, such as avoiding areas affected by wind speed, and target priority: insulators > bolts > conductors. Safety boundary settings are based on power safety regulations, allowing for the setting of minimum flight distances, such as 5m for shock prevention; shooting angles: such as a 45° oblique angle to the insulator string to avoid blind spots; and the generation of 3D waypoints: such as coordinates A→B→C→…, corresponding to standard shooting points on tension towers, with coordinate A corresponding to tower 1, B to tower 2, and so on.

[0071] Step S202: Cruise and coarse screening

[0072] The drone cruises along a planned path, while a visible light camera captures a 4K video stream.

[0073] The YOLOv8-CA target detection engine is used to focus on high-precision static target detection. It processes video frames of 4K video streams in real time at 30FPS to locate power tower components (insulators, bolts, etc.) and identify defects (rust, damage, etc.).

[0074] YOLOv8-CA-HMS is adopted, in which the CA attention mechanism and IoU-center point deviation are jointly matched to improve the detection accuracy of small defects in power equipment, such as insulator cracks and bolt corrosion.

[0075] Deployment method: After INT8 quantization, it runs on edge devices, such as Jetson Orin, with a single frame processing latency of <50ms.

[0076] When a potential defect (such as confidence level > 60%) or a high-risk target (such as a "hanging foreign object") is detected, the system sends the frame image and its location information to the dynamic arbitration engine.

[0077] Step S203: Detailed review and decision-making

[0078] The dynamic arbitration engine executes multi-level response strategies based on the received confidence level, such as emergency obstacle avoidance, initiation verification, and marking for investigation.

[0079] Step S204: Closed-loop processing and reporting

[0080] All defect information (location, image, infrared data, confidence level) is recorded in a structured manner, and the YOLOv8-CA target detection engine generates an inspection report.

[0081] Based on the findings of this inspection, the VLA decision-making center will dynamically optimize subsequent tasks, such as adding instructions like "add infrared scanning to insulators at subsequent tower locations" and avoiding routes like "mark the coordinates of the plastic bag so that subsequent drones can detour around it."

[0082] The following is a detailed implementation of the key algorithm modules:

[0083] YOLOv8-CA object detection engine:

[0084] This embodiment uses the YOLOv8 model architecture as its foundation and makes key improvements to adapt it to power line inspection scenarios. Its core structure comprises two key parts:

[0085] a) Multi-scale Feature Fusion Network (FPN+PAN):

[0086] YOLOv8 uses a PAN structure, which, combined with the standard FPN structure, creates a powerful feature pyramid.

[0087] FPN (Top-Down Path): Upsamples the deep, high-semantic feature map output by the backbone network and adds it element-wise to the shallow, high-resolution feature map. This process transfers rich semantic information from deep layers to shallow layers, enabling shallow features containing small objects to also possess strong class discrimination capabilities.

[0088] PAN (Bottom-Up Path): Building upon FPN, an additional bottom-up path is added to further transmit and enhance the precise location information of shallow features to deeper layers. This bidirectional architecture, combining top-down and bottom-up approaches, ensures that feature layers at different scales simultaneously possess both rich semantic information and precise location information.

[0089] b) Introduce the Coordinate Attention (CA) mechanism:

[0090] Building upon the multi-scale feature map output by the fused FPN+PAN structure, this embodiment further introduces a coordinate attention module. This module captures highly directional long-range dependencies by performing global pooling along both the height and width directions, and retains precise positional information when calculating attention. The CA module is embedded in the critical path of the FPN / PAN, adaptively weighting the fused features to significantly enhance the feature response to small target defects such as bolt corrosion and insulator micro-cracks, while suppressing irrelevant background noise.

[0091] Through the combined effect of FPN+PAN multi-scale feature fusion and CA coordinate attention enhancement, the target detection engine of this invention achieves a breakthrough improvement in the detection accuracy of small target defects while maintaining good detection capabilities for targets of various sizes. After model training, it is quantized by INT8 and deployed on edge computing units, with a single-frame processing latency of <50ms.

[0092] VLA Decision Center:

[0093] This embodiment uses a model based on the Llama-7B architecture as its foundation. It adapts to the power industry scenario through LoRA (Low-Rank Adaptation) fine-tuning technology (rank r=32) and utilizes a natural language command library specific to the power industry to support natural language command parsing, such as "prioritize scanning the tower top insulators." This model can fuse features extracted by a visual encoder (such as DINOv2) with language commands, directly outputting control signals for the drone, including obstacle avoidance paths, shooting angle adjustments, linear velocity, yaw rate, and gimbal control command linear velocity. The core of natural language commands in the power industry is to translate everyday human language into machine-understandable operations, thereby enabling more intelligent and efficient handling of various tasks. For example, for a power inspection drone, the operator can directly issue commands such as "Inspect the A-phase insulator string of tower 1, maintain a safe distance of 5 meters, and take clear photos." The drone can understand the semantics of the commands, autonomously plan its path, and complete the task without manual remote control or pre-setting complex waypoints. For such accurate language commands, thousands can be pre-trained and deployed on a large local model language library, and then the drone can be specifically trained to understand the actions corresponding to these commands.

[0094] Dynamic arbitration engine:

[0095] In this embodiment, the dynamic arbitration engine serves as the system's command and dispatch center. Its core function is to implement multi-level response strategies based on the confidence level output by the target detection engine, and to intelligently coordinate the work of the target detection engine and the VLA decision center to address detection results with different levels of certainty. Its core decision-making logic is summarized as follows:

[0096] High-confidence instant response: When the confidence level output by the target detection engine is higher than or equal to the first threshold (e.g., 0.85), the arbitration engine determines that it is a clear defect or a high-risk target and immediately triggers the VLA decision center to execute emergency obstacle avoidance or hovering commands to avoid collision risks.

[0097] Medium-confidence multimodal verification: When the confidence level is lower than the first threshold but higher than or equal to the second threshold (e.g., 0.6), the arbitration engine initiates the multimodal verification process. For example, it calls an infrared thermal imager to measure the temperature at the suspected location; if the detected temperature difference exceeds a preset threshold (e.g., 5°C), it is confirmed as a defect. Subsequently, the arbitration engine instructs the VLA decision center to generate an "approach re-inspection" command, controlling the drone to approach the target and perform multi-angle high-definition shooting to complete the refined diagnosis. During this process, the system performs multimodal data fusion: combining the defect location box of YOLO, the temperature measurement data of the infrared thermal imager, and real-time wind speed information, the VLA decision center controls the drone to hover precisely at the defect location, possessing dynamic wind resistance and attitude stabilization capabilities; and performs multi-angle shooting, such as obtaining global information from an orthogonal view and capturing local details from a 45° oblique view.

[0098] Low confidence recording and feedback: When the confidence level is lower than the second threshold, the arbitration engine marks the detection result and location information as "to be investigated" and sends it to the active learning queue of the closed-loop optimization system as a difficult sample for subsequent incremental training of the model.

[0099] In a preferred embodiment, the first threshold is set to 0.85, the second threshold is set to 0.6, and the infrared temperature difference threshold is 5°C. Those skilled in the art will understand that these thresholds can be adjusted within a range according to different inspection environments and requirements for false alarm / missed detection rates. For example, the first threshold can be selected between 0.75 and 0.95, the second threshold between 0.55 and 0.65, and the infrared temperature difference threshold between 4.5°C and 5.5°C.

[0100] Conflict resolution mechanism: When the output of the target detection engine conflicts with the decision history of the VLA decision center based on the environmental context, the arbitration engine prioritizes the decision from the source with higher confidence and sends back the conflicting data sample for closed-loop optimization of the system, thereby continuously improving the reliability of the system's decision-making in complex environments.

[0101] Because different artificial intelligence algorithm models have differences in semantic and perceptual cognition, conflicts between the two models will inevitably occur during the fusion process. In such cases, detection and judgment are necessary, for example:

[0102] YOLOv8-CA target detection engine: relies on local visual features, such as the rust texture of power towers, and is susceptible to false alarms due to interference from lighting and motion blur.

[0103] VLA Decision Center: It integrates spatiotemporal context, such as the equipment's historical status and flight trajectory, but may overlook subtle defects.

[0104] Typical conflict scenario: The YOLOv8-CA target detection engine misjudges metallic reflection as "insulator damage" with a confidence level of 65%, while the VLA decision center determines safety based on the equipment's intact record.

[0105] The solution is as follows:

[0106] 1. Confidence-based dynamically weighted arbitration

[0107] Rule design:

[0108] if YOLO.confidence > 85%: # High confidence defect

[0109] Forced VLA decision center re-examination (close-up imaging + infrared thermometry) [6,8](@ref)

[0110] elif 60% < YOLO.confidence ≤ 85%:

[0111] Initiate multimodal verification: Compare the infrared thermograms; if the temperature difference is greater than 5°C, it is confirmed as a defect [6](@ref).

[0112] else: # Low confidence

[0113] Record the location for subsequent batches to be rescanned.

[0114] Weighting: VLA's decision-making center accounts for 70% of the weight in path planning, while YOLO accounts for 80% in static defect detection. Weight refers to confidence level; it's the confidence level of the two decision-making methods when a conflict arises in a given scenario. This is determined and calculated in real-time by the onboard computer. The aforementioned rule design refers to the calculation logic, while the weights are pre-set. The weight of each decision-making method is set based on historical experience for different scenarios and can be continuously adjusted based on actual results.

[0115] 2. Spatiotemporal context verification

[0116] Historical status comparison: If the difference between the current power tower and the previous inspection data (such as the progress of bolt corrosion) is greater than 10%, YOLO will be used as the primary source.

[0117] Trajectory rationality analysis: The VLA decision center verifies target reachability through flight trajectory (e.g., if YOLO reports a defect at the top of the tower but the drone is at the bottom, it is determined to be a false detection).

[0118] 3. Optimization of closed-loop processing for conflicting samples

[0119] Automatic annotation reflow: Conflicting frames are saved to a database, which is named Active Learning Database in this embodiment. Training samples are generated through semi-supervised annotation (such as SAM segmentation).

[0120] Incremental Training Strategy: The closed-loop optimization system initiates an incremental training process periodically, for example, once a week or when a certain number of conflict samples accumulate. This process uses the collected hard sample dataset to unfreeze and train the head of the object detection engine model while keeping its backbone network weights fixed. The training cycle is approximately 10 epochs, in order to quickly adapt to new defect patterns while ensuring model stability.

[0121] Example 4: System Closed-Loop Optimization

[0122] The closed-loop optimization system is responsible for the continuous evolution of the model. It collects difficult samples from the edge (such as low-confidence samples and arbitration conflict samples), performs semi-automatic annotation using tools such as SAM (Segment Anything Model), and constructs an incremental training dataset. It regularly fine-tunes the YOLOv8-CA model and VLA decision center, and distributes the optimized model parameters to the drone, enabling the entire system to have continuous learning capabilities.

[0123] This invention achieves intelligent power inspection throughout the entire process through a four-layer architecture of multi-sensor perception, dual-model collaborative decision-making, dynamic arbitration, and closed-loop evolution, significantly improving the efficiency of power inspection, reducing costs, and redefining the technical standards for power inspection.

[0124] The deployment time for traditional inspection tasks includes the following components:

[0125] Data collection phase (several hours):

[0126] Equipment preparation and scheduling: A professional LiDAR scanning team and equipment need to be arranged to go to the site.

[0127] On-site scanning: Performing a detailed lidar scan on a single power tower requires acquiring data from multiple angles to ensure the integrity of the point cloud. This process itself takes 1-3 hours per tower, and even longer if the terrain is complex.

[0128] Data processing and modeling phase (hours to days, depending on the hardware configuration of the modeling server):

[0129] Point cloud denoising and registration: Merging scan data from multiple sites into a complete, noise-free 3D point cloud model of the power tower. This requires specialized software and operators and takes approximately 8-48 hours.

[0130] 3D model reconstruction: Although it is usually done using point clouds rather than manual modeling, generating a high-precision point cloud model that can be used for path planning still takes time. This process heavily relies on high-performance computers and skilled professionals and cannot be completed on-site.

[0131] Task planning phase (several hours):

[0132] Manual point cloud analysis: Engineers need to manually observe the 3D point cloud model to identify key components such as insulators, wires, and hardware.

[0133] Manual waypoint setting: For each component that needs to be inspected, the drone's flight waypoints, gimbal angle, shooting distance, and other parameters are manually set. This is an extremely meticulous and tedious task; a complex power tower may require setting hundreds of waypoints, which is very time-consuming.

[0134] Pilot training and scheduling (hidden costs):

[0135] Training a pilot capable of performing such complex tasks requires months of time and incurs significant training costs. Personnel scheduling itself also incurs time costs.

[0136] In summary, the traditional process is sequential; the next step cannot proceed until the previous one is completely completed. The entire process typically takes 1-2 days to deploy a new tower inspection task.

[0137] The core of the method of this invention is a closed loop of perception-decision-execution, which almost completely skips the preprocessing stage mentioned above.

[0138] Initial localization and perception (<1 minute)

[0139] Once the drone arrives at the work area, it does not need to know the precise three-dimensional coordinates of the power tower in advance. It scans the environment in real time using an onboard vision system (such as an RGB camera) and automatically identifies the approximate location of the power tower using a built-in object detection model (such as a trained power tower detector).

[0140] Natural Language Command Parsing and Autonomous Planning (1-3 minutes)

[0141] Command Examples

[0142] "Inspect the A-phase insulator string around its perimeter, maintaining a safe distance of 5 meters, and take clear photos."

[0143] VLA Central Workflow

[0144] Visual perception

[0145] As the drone approaches the power tower, a real-time video stream is input into the visual model, which identifies entities such as the "power tower," "phase A," and "insulator string."

[0146] Language comprehension

[0147] Natural language instructions are parsed into structured task objectives (e.g., target object = insulator string, action = orbit + take a picture, constraint = distance 5 meters).

[0148] Decision-making and planning (Action)

[0149] The VLA center aligns and fuses visual perception and language commands. It will:

[0150] Dynamic path planning: Calculates a flight path in real time that can safely circle the "A-phase insulator string" while maintaining a distance of approximately 5 meters. This requires no preset waypoints; the path is generated and adjusted in real time.

[0151] Active observation: If the current viewpoint is not ideal, the drone will autonomously adjust its position to obtain a better observation angle.

[0152] Closed-loop verification: Continuously judge whether the captured image meets the requirement of "clarity". If it does not meet the requirement, it will automatically adjust until the task is completed.

[0153] Task execution and data acquisition (synchronized with inspection time)

[0154] Once the planning is complete, the drones immediately begin autonomous flight and inspections. The deployment and execution phases are seamlessly integrated.

[0155] The minute-level deployment time of this invention mainly refers to steps 1 and 2 mentioned above, that is, the preparation time from when the UAV arrives at the site to when it "understands" the mission and begins autonomous flight. This time is mainly used for initialization, visual search, and target locking.

[0156] The drone based on the VLA (Vision-Language-Action) model of this invention possesses the ability to cope with sudden dynamic obstacles (such as birds or plastic bags). The core of this capability lies in its integration of semantic understanding, real-time perception, and dynamic decision-making into a cohesive whole. Compared to traditional drones that rely on preset rules or static maps, the VLA model enables the drone to truly "understand" what is happening in front of it, like an experienced pilot, and to respond intelligently and flexibly. A traditional drone might only detect a "moving pixel" ahead, but a VLA drone can identify it as "a bird" or "a plastic bag." This is thanks to its powerful vision-language alignment capability.

[0157] Accurate semantic recognition: Leveraging powerful visual foundational models such as YOLO, the model can accurately identify the semantic category of obstacles. Knowing whether an obstacle is a "bird" or a "plastic bag" is the first step in making informed decisions.

[0158] Trajectory prediction: The VLA model not only identifies static objects but also predicts their future trajectories within a short period. By analyzing consecutive frames of images and potentially combining algorithms such as Kalman filtering, the model can estimate the bird's flight direction and speed, thereby predicting its likely location.

[0159] Risk Assessment: Based on identification and prediction, the model performs rapid collision risk calculations. It comprehensively considers the type of obstacle, relative speed, distance, and its own flight status to determine whether a collision risk exists and its level. For example, the risk assessment for a bird rapidly crossing laterally is drastically different from that for a hovering plastic bag.

[0160] Based on the above understanding, the VLA model generates highly contextualized obstacle avoidance strategies, rather than mechanically executing the simple rule of "hovering when an obstacle is detected".

[0161] Dealing with fast-moving obstacles such as birds: For birds whose flight trajectories may be difficult to predict, the system tends to adopt a proactive and conservative avoidance strategy.

[0162] Proactive avoidance: Once a potential collision is predicted, a smooth detour path will be planned in advance, potentially maintaining a safe distance from the predicted trajectory of the bird.

[0163] Multiple alternative strategies: The system may generate multiple alternative strategies (such as emergency hovering, lateral maneuvering, vertical climb, and curved flight), and select the optimal solution based on the risk assessment level (high, medium, and low) to achieve a balance between safety and efficiency.

[0164] Dealing with lightweight floating objects such as plastic bags: The strategies are different for lightweight plastic bags whose movement is unpredictable with the wind.

[0165] Risk assessment and trajectory fine-tuning: If the plastic bag is determined to be lightweight and pose little threat, and its trajectory is relatively slow at the current wind speed, the system may not make large-scale maneuvers, but will instead make fine trajectory fine-tuning, such as slightly adjusting the speed or yaw angle, to safely pass by it.

[0166] Dynamic adjustment of sensor weights: When dealing with such obstacles, the system may dynamically adjust the confidence weights of different sensors. For example, visual sensors may be more reliable when tracking lightweight objects, while LiDAR may be less effective due to the low reflectivity of the object. The system will correspondingly reduce the weight of LiDAR data to avoid misjudgment.

[0167] After making a decision, the VLA model translates the high-level strategy into specific, executable low-level continuous control commands (such as velocity vectors and angular velocities), rather than just discrete waypoints. This end-to-end mapping capability makes the drone's obstacle avoidance maneuvers smoother and more natural. During the execution of the maneuver, the system continuously monitors environmental changes and its own state, forming a closed loop of perception-decision-action-verification. If the obstacle's behavior suddenly changes, such as a bird making a sharp turn, the model can re-evaluate and adjust its strategy in real time based on the new perception information.

[0168] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A multi-modal perception based intelligent power inspection drone system, characterized in that, Comprise: A multi-modal sensor group for collecting perception data of power equipment and surrounding environment; An edge computing unit deployed at the end of the UAV; The edge computing unit is configured to run: A target detection engine for processing the perception data, identifying power equipment components and defects, and outputting detection results including defect location and confidence; A VLA decision hub, which is a model based on a large language model and fine-tuned in the field using LoRA technology, for parsing natural language instructions and generating dynamic control signals for the UAV; A dynamic arbitration engine for receiving the detection results output by the target detection engine and triggering a multi-level response strategy based on the confidence to coordinate the working states of the target detection engine and the VLA decision hub; Wherein, the dynamic arbitration engine is configured to trigger the VLA decision hub to execute different control instructions when the confidence meets different conditions, in order to coordinate the working states of the target detection engine and the VLA decision hub, and realize autonomous inspection; the system realizes autonomous power inspection without pre-generating a three-dimensional point cloud model through the cooperation of the target detection engine, the VLA decision hub and the dynamic arbitration engine.

2. The intelligent power inspection drone system based on multi-modal perception of claim 1, wherein: It also includes a UAV platform and a flight control and communication module; the multi-modal sensor group is integrated on the UAV platform; the flight control and communication module is used for instruction execution and data transmission to provide state information for UAV flight control.

3. The intelligent power inspection drone system based on multi-modal perception of claim 1, wherein: The multi-modal sensor group includes one or at least two of a visible light camera, an infrared thermal imager and a laser radar.

4. The intelligent power inspection drone system based on multi-modal perception of claim 1, wherein: The target detection engine is an improved YOLO model based on an attention mechanism, the attention mechanism is a coordinate attention mechanism, and multi-scale feature fusion FPN+PAN is used; the improved YOLO model is quantized by INT8 after training and deployed on the edge computing unit.

5. The intelligent power inspection drone system based on multi-modal perception of claim 1, wherein: The control signal of the UAV is a high-level task instruction for controlling the UAV, which is converted into a bottom-level control signal by the UAV platform; the control signal of the UAV includes linear velocity, yaw angular velocity and gimbal control instructions.

6. The intelligent power inspection drone system based on multi-modal perception of claim 1, wherein: The dynamic arbitration engine is configured to execute the following multi-level response strategy: When the confidence is higher than or equal to a first threshold, trigger the VLA decision hub to execute emergency obstacle avoidance or key re-inspection instructions; When the confidence is lower than the first threshold but higher than or equal to a second threshold, trigger a multi-modal verification instruction to call multiple sensors in the multi-modal sensor group to jointly verify the detected target; When the confidence is lower than the second threshold, mark the detection result.

7. The intelligent power inspection drone system based on multi-modal perception as claimed in claim 6, wherein: The multi-modal verification instruction includes starting the infrared thermal imager to perform temperature verification on suspected defect points.

8. The intelligent power inspection drone system based on multi-modal perception of claim 1, wherein: The system also includes a closed-loop optimization system for receiving data uploaded by the edge computing unit and performing incremental training and model optimization on the target detection engine and / or the VLA decision hub based on an active learning mechanism. 9.A method for intelligent unmanned aerial vehicle power inspection based on multi-modal perception, applied to the system of any one of claims 1-8, characterized in that, The steps include: Receiving and parsing natural language inspection instructions; Based on the natural language inspection instructions, planning an initial flight path through the VLA decision hub; In the process of flight inspection, the target detection engine is used to process the perception data collected by the multi-modal sensor group in real time, and output detection results; The dynamic arbitration engine is used to dynamically decide and trigger corresponding response actions according to the confidence in the detection results, the response actions including continuing cruising, approaching re-inspection, multi-modal verification or emergency obstacle avoidance; According to the response actions, the UAV is controlled to perform corresponding operations, and an inspection report is generated.

10. The intelligent drone power inspection method based on multi-modal perception of claim 9, wherein: The dynamic decision-making process further includes: when the output of the target detection engine conflicts with the decision history or environmental context information of the VLA decision hub, the decision of the source with higher confidence is adopted preferentially, and the conflict data sample is returned to the closed-loop optimization system for model optimization.

Images